Serving data from the Marine Institute (Ireland)
Brought to you by the Marine Institute(Ireland)
|
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2026-07-31T21:48:50+00:00
Startup was at 2026-07-20T15:21:19+00:00
Last major LoadDatasets started 2m 36s ago and finished after 110 seconds.
nGridDatasets = 9
nTableDatasets = 77
nTotalDatasets = 86
n Datasets Failed To Load (in the last major LoadDatasets) = 6
ireland_marine_research_database_charts, imifeeaghco2, imiroughriverarms,
compass_neatl_hindcast_grid, IMI_Model_Stats, IMI_EATL_WAVE, (end)
ERROR: Duplicate datasetIDs in datasets.xml:
smartbay_metbuoy, intgn_weatherancillary
Unique users (since startup) n = 32234
Response Failed Time (since last major LoadDatasets) n = 3, median ~= 150 ms
Response Failed Time (since last Daily Report) n = 6239, median ~= 199 ms
Response Failed Time (since startup) n = 94792, median ~= 266 ms
Response Succeeded Time (since last major LoadDatasets) n = 11, median ~= 583 ms
Response Succeeded Time (since last Daily Report) n = 32354, median ~= 58 ms
Response Succeeded Time (since startup) n = 499935, median ~= 58 ms
TaskThread has finished 0 out of 0 tasks. Currently, no task is running.
TaskThread Failed Time (since last Daily Report) n = 0
TaskThread Failed Time (since startup) n = 0
TaskThread Succeeded Time (since last Daily Report) n = 0
TaskThread Succeeded Time (since startup) n = 0
Number of threads: Tomcat-waiting=48, inotify=4, other=55
MemoryInUse= 972 MB (highWaterMark= 4034 MB) (Xmx ~= 5888 MB)
Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open
timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files
---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- -----
2026-07-31T21:48:02+00:00 108s 15 6 86 533 ( 95) 126 ( 217) 0 0 50 4 55 190 0%
2026-07-31T21:32:47+00:00 92s 11 6 86 422 ( 51) 51 ( 150) 0 0 44 4 55 380 0%
2026-07-31T21:17:52+00:00 98s 12 6 86 685 ( 66) 65 ( 159) 0 0 46 4 55 187 0%
2026-07-31T21:02:48+00:00 94s 11 6 86 524 ( 119) 72 ( 142) 0 9 43 4 56 193 0%
2026-07-31T20:48:02+00:00 108s 15 6 86 549 ( 42) 141 ( 291) 0 0 52 4 55 176 0%
2026-07-31T20:32:42+00:00 90s 11 6 86 435 ( 37) 71 ( 109) 0 0 43 4 55 194 0%
2026-07-31T20:17:48+00:00 96s 12 6 86 526 ( 41) 59 ( 135) 0 0 40 4 56 189 0%
2026-07-31T20:02:45+00:00 94s 12 6 86 393 ( 44) 53 ( 139) 0 0 43 4 56 189 0%
2026-07-31T19:47:57+00:00 105s 18 7 85 531 ( 263) 111 ( 193) 0 4 51 4 55 193 0%
2026-07-31T19:32:44+00:00 92s 12 7 85 501 ( 65) 81 ( 115) 0 0 42 4 55 194 0%
2026-07-31T19:17:43+00:00 92s 13 7 85 650 ( 83) 60 ( 136) 0 4 53 4 55 195 0%
2026-07-31T19:02:44+00:00 92s 12 7 85 563 ( 239) 85 ( 127) 0 7 56 4 55 195 0%
2026-07-31T18:47:45+00:00 94s 14 7 85 427 ( 103) 105 ( 209) 0 0 44 4 56 193 0%
2026-07-31T18:33:01+00:00 109s 15 7 85 506 ( 93) 69 ( 147) 0 0 43 4 58 193 0%
2026-07-31T18:17:39+00:00 88s 14 7 85 749 ( 101) 64 ( 141) 0 9 42 4 77 243 0%
2026-07-31T18:02:40+00:00 89s 13 8 84 385 ( 42) 63 ( 128) 0 0 30 4 55 194 0%
2026-07-31T17:47:35+00:00 84s 13 7 85 459 ( 51) 124 ( 229) 0 0 30 4 55 193 0%
2026-07-31T17:32:42+00:00 92s 12 7 85 495 ( 54) 53 ( 154) 0 0 39 4 60 1033 0%
2026-07-31T17:17:59+00:00 110s 17 7 85 792 ( 83) 65 ( 136) 0 3 30 4 58 196 0%
2026-07-31T17:02:38+00:00 89s 12 7 85 445 ( 48) 57 ( 156) 0 0 41 4 55 193 0%
2026-07-31T16:47:37+00:00 88s 15 7 85 457 ( 77) 117 ( 221) 0 0 44 4 56 369 0%
2026-07-31T16:32:37+00:00 87s 13 7 85 409 ( 80) 52 ( 152) 0 0 33 4 55 192 0%
2026-07-31T16:17:36+00:00 87s 12 7 85 485 ( 63) 52 ( 135) 0 0 31 4 56 402 0%
2026-07-31T16:04:09+00:00 180s 28 7 85 453 ( 40) 56 ( 144) 0 0 34 4 55 269 0%
2026-07-31T15:47:50+00:00 101s 20 7 85 406 ( 47) 98 ( 218) 0 0 34 4 55 192 0%
2026-07-31T15:33:06+00:00 117s 31 7 85 448 ( 33) 58 ( 144) 0 0 37 4 57 454 0%
2026-07-31T15:17:34+00:00 85s 13 7 85 606 ( 40) 53 ( 158) 0 0 41 4 55 194 0%
2026-07-31T15:02:59+00:00 110s 15 7 85 513 ( 22) 51 ( 140) 0 0 39 4 58 464 0%
2026-07-31T14:47:33+00:00 84s 15 7 85 470 ( 75) 126 ( 315) 0 0 43 4 55 191 0%
2026-07-31T14:32:35+00:00 88s 13 8 84 565 ( 15) 53 ( 148) 0 0 45 4 55 178 0%
2026-07-31T14:17:36+00:00 88s 14 8 84 712 ( 44) 55 ( 160) 0 0 45 4 55 192 0%
2026-07-31T14:02:37+00:00 89s 15 8 84 478 ( 67) 61 ( 152) 0 0 47 4 55 198 0%
2026-07-31T13:47:55+00:00 107s 16 8 84 598 ( 68) 116 ( 282) 0 0 38 4 55 185 0%
2026-07-31T13:32:37+00:00 90s 13 8 84 477 ( 37) 78 ( 90) 0 0 36 4 57 207 0%
2026-07-31T13:17:33+00:00 86s 14 8 84 668 ( 37) 65 ( 150) 0 0 38 4 55 191 0%
2026-07-31T13:02:40+00:00 93s 14 8 84 454 ( 41) 42 ( 141) 0 0 38 4 55 196 0%
2026-07-31T12:47:49+00:00 102s 16 8 84 628 ( 95) 112 ( 276) 0 0 38 4 55 185 0%
2026-07-31T12:32:34+00:00 87s 15 8 84 503 ( 28) 65 ( 123) 0 0 36 4 57 192 0%
2026-07-31T12:17:34+00:00 87s 14 8 84 622 ( 57) 57 ( 153) 0 0 38 4 55 195 0%
2026-07-31T12:02:36+00:00 89s 14 8 84 685 ( 91) 56 ( 155) 0 0 38 4 55 190 0%
2026-07-31T11:47:35+00:00 88s 14 8 84 465 ( 45) 87 ( 234) 0 0 38 4 55 190 0%
2026-07-31T11:32:55+00:00 108s 15 8 84 503 ( 35) 49 ( 145) 0 0 42 4 55 194 0%
2026-07-31T11:17:38+00:00 92s 14 8 84 1038 ( 119) 72 ( 164) 0 0 43 4 55 198 0%
2026-07-31T11:02:36+00:00 89s 13 8 84 478 ( 40) 60 ( 159) 0 0 42 4 56 191 0%
2026-07-31T10:47:35+00:00 89s 16 8 84 418 ( 40) 96 ( 224) 0 0 42 4 57 187 0%
2026-07-31T10:32:33+00:00 87s 13 8 84 500 ( 17) 49 ( 140) 0 0 51 4 55 189 0%
2026-07-31T10:17:53+00:00 108s 18 8 84 783 ( 60) 76 ( 154) 0 0 52 4 56 189 0%
2026-07-31T10:02:36+00:00 91s 13 8 84 532 ( 96) 101 ( 138) 0 0 43 4 55 191 0%
2026-07-31T09:47:36+00:00 90s 15 8 84 489 ( 37) 104 ( 249) 0 0 43 4 55 192 0%
2026-07-31T09:32:35+00:00 90s 13 8 84 532 ( 22) 137 ( 2409) 0 0 46 4 55 196 0%
2026-07-31T09:17:48+00:00 103s 15 7 85 694 ( 50) 73 ( 164) 0 0 35 4 57 196 0%
2026-07-31T09:02:37+00:00 92s 12 7 85 748 ( 206) 621 ( 827) 4 0 33 4 56 414 0%
2026-07-31T08:47:36+00:00 91s 13 7 85 622 ( 168) 449 ( 784) 0 0 32 4 57 182 0%
2026-07-31T08:32:31+00:00 91s 12 7 85 488 ( 47) 63 ( 148) 0 0 33 4 55 190 0%
2026-07-31T08:17:44+00:00 104s 16 7 85 697 ( 70) 289 ( 843) 0 0 39 4 55 193 0%
2026-07-31T08:02:32+00:00 92s 12 7 85 630 ( 144) 477 ( 911) 12 0 38 4 56 250 0%
2026-07-31T07:47:27+00:00 88s 13 7 85 424 ( 70) 225 ( 474) 0 0 41 4 55 190 0%
2026-07-31T07:32:34+00:00 95s 12 7 85 513 ( 56) 120 ( 136) 0 0 40 4 57 197 0%
2026-07-31T07:17:27+00:00 88s 13 7 85 582 ( 47) 90 ( 152) 0 0 33 4 55 189 0%
2026-07-31T07:02:47+00:00 109s 15 7 85 571 ( 87) 208 ( 914) 0 0 32 4 56 383 0%
2026-07-31T06:47:37+00:00 99s 20 7 85 542 ( 89) 488 ( 1017) 2 0 32 4 55 181 0%
2026-07-31T06:32:38+00:00 100s 14 7 85 494 ( 35) 111 ( 196) 0 0 30 4 58 698 0%
2026-07-31T06:17:28+00:00 90s 14 7 85 661 ( 53) 131 ( 105) 0 0 33 4 55 186 0%
2026-07-31T06:02:40+00:00 103s 13 7 85 440 ( 32) 68 ( 118) 0 0 32 4 56 187 0%
2026-07-31T05:47:41+00:00 104s 16 7 85 343 ( 40) 95 ( 221) 0 0 27 4 55 186 0%
2026-07-31T05:32:22+00:00 86s 12 7 85 351 ( 23) 74 ( 150) 0 0 27 4 55 334 0%
2026-07-31T05:17:25+00:00 89s 13 7 85 566 ( 44) 86 ( 132) 0 0 27 4 55 192 0%
2026-07-31T05:02:31+00:00 95s 12 7 85 321 ( 39) 75 ( 94) 0 0 27 4 55 192 0%
2026-07-31T04:47:36+00:00 100s 16 7 85 340 ( 67) 98 ( 224) 0 0 27 4 57 804 0%
2026-07-31T04:32:23+00:00 87s 12 7 85 383 ( 29) 54 ( 128) 0 0 30 4 56 192 0%
2026-07-31T04:17:21+00:00 85s 13 7 85 546 ( 45) 53 ( 139) 0 0 34 4 55 193 0%
2026-07-31T04:02:25+00:00 89s 12 7 85 363 ( 43) 128 ( 1) 0 0 25 4 57 285 0%
2026-07-31T03:47:19+00:00 83s 13 7 85 317 ( 66) 101 ( 221) 0 0 25 4 55 194 0%
2026-07-31T03:32:42+00:00 107s 15 7 85 316 ( 25) 52 ( 147) 0 0 25 4 55 193 0%
2026-07-31T03:17:27+00:00 91s 13 7 85 519 ( 42) 55 ( 147) 0 0 25 4 55 193 0%
2026-07-31T03:02:25+00:00 90s 12 7 85 360 ( 45) 53 ( 136) 0 0 26 4 55 192 0%
2026-07-31T02:47:23+00:00 88s 14 7 85 381 ( 46) 97 ( 233) 0 0 27 4 55 202 0%
2026-07-31T02:32:42+00:00 106s 15 7 85 344 ( 61) 57 ( 127) 0 0 27 4 55 190 0%
2026-07-31T02:17:22+00:00 86s 15 7 85 536 ( 45) 55 ( 152) 0 0 28 4 55 191 0%
2026-07-31T02:02:33+00:00 98s 12 7 85 588 ( 39) 144 ( 20) 0 0 27 4 56 192 0%
2026-07-31T01:47:24+00:00 88s 13 7 85 383 ( 39) 93 ( 240) 0 0 27 4 55 192 0%
2026-07-31T01:32:38+00:00 103s 16 7 85 352 ( 75) 73 ( 102) 0 0 27 4 55 192 0%
2026-07-31T01:17:21+00:00 86s 14 8 84 467 ( 67) 66 ( 139) 0 0 26 4 57 201 0%
2026-07-31T01:02:29+00:00 94s 13 8 84 329 ( 36) 66 ( 123) 0 0 29 4 55 192 0%
2026-07-31T00:47:27+00:00 93s 14 8 84 342 ( 65) 96 ( 252) 0 0 30 4 55 193 0%
2026-07-31T00:32:21+00:00 87s 13 8 84 340 ( 39) 56 ( 130) 0 0 30 4 55 192 0%
2026-07-31T00:17:48+00:00 114s 19 8 84 546 ( 74) 78 ( 128) 0 0 29 4 56 210 0%
2026-07-31T00:02:35+00:00 101s 15 10 82 385 ( 54) 146 ( 1) 0 0 26 4 59 191 0%
2026-07-30T23:47:32+00:00 98s 18 10 82 357 ( 45) 108 ( 223) 0 0 31 4 55 195 0%
2026-07-30T23:32:41+00:00 107s 13 8 84 301 ( 40) 50 ( 139) 0 0 33 4 56 196 0%
2026-07-30T23:17:35+00:00 101s 14 8 84 483 ( 55) 55 ( 152) 0 0 35 4 55 192 0%
2026-07-30T23:02:57+00:00 123s 17 8 84 348 ( 49) 59 ( 144) 0 0 38 4 55 181 0%
2026-07-30T22:47:32+00:00 98s 14 8 84 400 ( 108) 109 ( 226) 0 0 38 4 55 189 0%
Major LoadDatasets Times Distribution (since last Daily Report):
n = 59, median ~= 90517 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 58
<= 5 min: 1
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Major LoadDatasets Times Distribution (since startup):
n = 1082, median ~= 97225 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 872
<= 5 min: 46
<= 10 min: 26
<= 20 min: 137
<= 1 hr: 1
> 1 hr: 0
Minor LoadDatasets Times Distribution (since last Daily Report):
n = 2, median ~= 1000 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 1
<= 2 s: 0
<= 5 s: 1
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Minor LoadDatasets Times Distribution (since startup):
n = 31, median ~= 3816 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 2
<= 500 ms: 1
<= 1 s: 1
<= 2 s: 0
<= 5 s: 19
<= 10 s: 4
<= 20 s: 1
<= 1 min: 0
<= 2 min: 3
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last major LoadDatasets):
n = 3, median ~= 150 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 3
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last Daily Report):
n = 6239, median ~= 199 ms
0 ms: 207
1 ms: 135
2 ms: 70
<= 5 ms: 109
<= 10 ms: 69
<= 20 ms: 80
<= 50 ms: 214
<= 100 ms: 237
<= 200 ms: 2026
<= 500 ms: 976
<= 1 s: 888
<= 2 s: 515
<= 5 s: 428
<= 10 s: 130
<= 20 s: 107
<= 1 min: 48
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since startup):
n = 94792, median ~= 266 ms
0 ms: 5907
1 ms: 4184
2 ms: 2114
<= 5 ms: 1739
<= 10 ms: 1322
<= 20 ms: 1576
<= 50 ms: 3892
<= 100 ms: 6040
<= 200 ms: 16688
<= 500 ms: 17888
<= 1 s: 13733
<= 2 s: 8180
<= 5 s: 7629
<= 10 s: 2806
<= 20 s: 671
<= 1 min: 264
<= 2 min: 45
<= 5 min: 94
<= 10 min: 18
<= 20 min: 2
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last major LoadDatasets):
n = 11, median ~= 583 ms
0 ms: 1
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 1
<= 20 ms: 0
<= 50 ms: 2
<= 100 ms: 0
<= 200 ms: 1
<= 500 ms: 0
<= 1 s: 3
<= 2 s: 1
<= 5 s: 1
<= 10 s: 0
<= 20 s: 1
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last Daily Report):
n = 32354, median ~= 58 ms
0 ms: 3358
1 ms: 1862
2 ms: 951
<= 5 ms: 1591
<= 10 ms: 1158
<= 20 ms: 2148
<= 50 ms: 4583
<= 100 ms: 3233
<= 200 ms: 3020
<= 500 ms: 3702
<= 1 s: 2069
<= 2 s: 1428
<= 5 s: 2134
<= 10 s: 915
<= 20 s: 185
<= 1 min: 16
<= 2 min: 1
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since startup):
n = 499935, median ~= 58 ms
0 ms: 62648
1 ms: 29813
2 ms: 13351
<= 5 ms: 19061
<= 10 ms: 10981
<= 20 ms: 29254
<= 50 ms: 76102
<= 100 ms: 53933
<= 200 ms: 43449
<= 500 ms: 65636
<= 1 s: 31021
<= 2 s: 20849
<= 5 s: 27159
<= 10 s: 12639
<= 20 s: 3141
<= 1 min: 649
<= 2 min: 211
<= 5 min: 26
<= 10 min: 3
<= 20 min: 5
<= 1 hr: 4
> 1 hr: 0
TaskThread Failed Time Distribution (since last Daily Report):
n = 0
TaskThread Failed Time Distribution (since startup):
n = 0
TaskThread Succeeded Time Distribution (since last Daily Report):
n = 0
TaskThread Succeeded Time Distribution (since startup):
n = 0
SgtMap topography nFromCache=51 nNotFromCache=301
GSHHS: nCached=100 of 100, nCoarse=1441, nSuccesses=1301, nTossed=649
NationalBoundaries: nCached=64 of 100, nCoarse=200, nSuccesses=281, nTossed=0
StateBoundaries: nCached=64 of 100, nCoarse=152, nSuccesses=281, nTossed=0
Rivers: nCached=64 of 100, nCoarse=177, nSuccesses=281, nTossed=0
bufferedImage isAccelerated=[unknown]
canonical map sizes: 87 + 748 + 285 + 1580 + 1026 + 1473 = 5199
canonicalStringHolder map sizes: 467 + 876 + 1080 + 525 + 210 + 316 = 3474
Number of threads: Tomcat-waiting=48, inotify=4, other=55
(format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus)
#1 Thread[AsyncFileHandlerWriter-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#2 Thread[Catalina-utility-1,1,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
java.lang.Thread.run(Thread.java:750)
#3 Thread[Catalina-utility-2,1,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
java.lang.Thread.run(Thread.java:750)
#4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#13 Thread[cluster1-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#14 Thread[cluster1-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#26 Thread[cluster2-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#27 Thread[cluster2-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#30 Thread[cluster3-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#31 Thread[cluster3-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#32 Thread[cluster3-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#33 Thread[cluster3-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#34 Thread[cluster3-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#35 Thread[cluster3-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#36 Thread[cluster3-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#37 Thread[cluster3-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#38 Thread[cluster3-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#39 Thread[cluster3-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#40 Thread[cluster3-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#41 Thread[cluster3-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#42 Thread[cluster3-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#43 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#44 Thread[Finalizer,8,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
#45 Thread[GC Daemon,2,system] TIMED_WAITING daemon
java.lang.Object.wait(Native Method)
sun.misc.GC$Daemon.run(GC.java:117)
#46 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:660)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:69)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:146)
java.lang.Thread.run(Thread.java:750)
#47 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:965)
java.lang.Thread.run(Thread.java:750)
#48 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:660)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:69)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:146)
java.lang.Thread.run(Thread.java:750)
#49 Thread[https-jsse-nio-8443-exec-4132,5,main] RUNNABLE daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.readInternal(IOBuffer.java:1003)
com.microsoft.sqlserver.jdbc.TDSChannel$ProxyInputStream.read(IOBuffer.java:993)
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464)
sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1350)
sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:966)
com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:2023)
com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6418)
com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7579)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:592)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:524)
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194)
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2979)
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:248)
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:223)
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery(SQLServerPreparedStatement.java:446)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.getDataForDapQuery(EDDTableFromDatabase.java:953)
gov.noaa.pfel.erddap.dataset.EDDTable.respondToDapQuery(EDDTable.java:3076)
gov.noaa.pfel.erddap.Erddap.doDap(Erddap.java:4428)
gov.noaa.pfel.erddap.Erddap.doGet(Erddap.java:569)
javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:201)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:146)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:57)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:170)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:146)
org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:340)
org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:167)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:170)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:146)
org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:140)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:170)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:146)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:88)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:534)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:129)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:86)
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:764)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:71)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:350)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:407)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:71)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:1344)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:2089)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:74)
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:976)
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:494)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
java.lang.Thread.run(Thread.java:750)
#50 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:965)
java.lang.Thread.run(Thread.java:750)
#51 Thread[Java2D Disposer,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
sun.java2d.Disposer.run(Disposer.java:148)
java.lang.Thread.run(Thread.java:750)
#52 Thread[main,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
org.apache.catalina.core.StandardServer.await(StandardServer.java:527)
org.apache.catalina.startup.Catalina.await(Catalina.java:951)
org.apache.catalina.startup.Catalina.start(Catalina.java:899)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:348)
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)
#53 Thread[Reference Handler,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#54 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon
java.lang.Thread.sleep(Native Method)
gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288)
#55 Thread[Signal Dispatcher,9,system] RUNNABLE daemon