Serving data from the Marine Institute (Ireland)
Brought to you by the Marine Institute(Ireland)
|
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2026-06-05T19:10:09+00:00
Startup was at 2026-05-17T04:59:56+00:00
Last major LoadDatasets started 1m 26s ago and is still running.
nGridDatasets = 9
nTableDatasets = 83
nTotalDatasets = 92
n Datasets Failed To Load (in the last major LoadDatasets) = 8
ireland_marine_research_database_charts, imiglenamongarms, imifeeaghco2, imiroughriverarms,
compass_neatl_hindcast_grid, IMI_Model_Stats, spiddal_obs_adcp, IMI_EATL_WAVE, (end)
Unique users (since startup) n = 42122
Response Failed Time (since last major LoadDatasets) n = 32, median ~= 611 ms
Response Failed Time (since last Daily Report) n = 3646, median ~= 546 ms
Response Failed Time (since startup) n = 110707, median ~= 375 ms
Response Succeeded Time (since last major LoadDatasets) n = 398, median ~= 82 ms
Response Succeeded Time (since last Daily Report) n = 17219, median ~= 139 ms
Response Succeeded Time (since startup) n = 632409, median ~= 93 ms
TaskThread has finished 0 out of 0 tasks. Currently, no task is running.
TaskThread Failed Time (since last Daily Report) n = 0
TaskThread Failed Time (since startup) n = 0
TaskThread Succeeded Time (since last Daily Report) n = 0
TaskThread Succeeded Time (since startup) n = 0
Number of threads: Tomcat-waiting=25, inotify=4, other=55
MemoryInUse= 432 MB (highWaterMark= 2543 MB) (Xmx ~= 5888 MB)
Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open
timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files
---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- -----
2026-06-05T18:55:12+00:00 89s 14 8 92 263 ( 186) 26 ( 350) 0 0 24 4 56 198 0%
2026-06-05T18:40:10+00:00 87s 14 8 92 307 ( 42) 73 ( 359) 0 0 30 4 55 208 0%
2026-06-05T18:25:27+00:00 104s 15 8 92 260 ( 109) 22 ( 545) 0 0 20 4 55 222 0%
2026-06-05T18:10:28+00:00 105s 14 8 92 319 ( 191) 34 ( 500) 0 0 30 4 55 221 0%
2026-06-05T17:55:11+00:00 88s 14 8 92 304 ( 73) 30 ( 300) 0 0 54 4 55 219 0%
2026-06-05T17:40:14+00:00 91s 14 8 92 412 ( 353) 104 ( 353) 0 0 43 4 55 206 0%
2026-06-05T17:25:26+00:00 104s 15 8 92 302 ( 94) 34 ( 350) 0 0 33 4 55 206 0%
2026-06-05T17:10:16+00:00 94s 14 8 92 358 ( 132) 35 ( 518) 0 0 30 4 55 211 0%
2026-06-05T16:55:14+00:00 92s 15 8 92 363 ( 149) 41 ( 113) 0 0 28 4 55 206 0%
2026-06-05T16:40:11+00:00 88s 14 8 92 290 ( 80) 74 ( 353) 0 0 28 4 56 252 0%
2026-06-05T16:25:25+00:00 103s 15 8 92 381 ( 170) 44 ( 500) 0 0 26 4 55 208 0%
2026-06-05T16:10:16+00:00 95s 14 8 92 448 ( 178) 52 ( 313) 0 0 31 4 55 210 0%
2026-06-05T15:55:08+00:00 86s 14 8 92 286 ( 57) 25 ( 568) 0 0 30 4 55 206 0%
2026-06-05T15:40:12+00:00 90s 14 8 92 397 ( 127) 104 ( 318) 0 0 35 4 56 205 0%
2026-06-05T15:25:25+00:00 104s 15 8 92 579 ( 330) 57 ( 911) 0 0 18 4 76 497 0%
2026-06-05T15:10:15+00:00 93s 13 8 92 430 ( 41) 50 ( 175) 0 0 32 4 57 773 0%
2026-06-05T14:55:16+00:00 95s 16 8 92 369 ( 177) 43 ( 804) 0 0 29 4 55 208 0%
2026-06-05T14:40:07+00:00 86s 16 8 92 391 ( 92) 101 ( 342) 0 0 30 4 55 207 0%
2026-06-05T14:25:07+00:00 86s 13 8 92 350 ( 119) 30 ( 750) 0 0 29 4 55 202 0%
2026-06-05T14:10:51+00:00 130s 16 8 92 520 ( 122) 80 ( 938) 0 0 30 4 54 224 0%
2026-06-05T13:55:09+00:00 89s 16 8 92 414 ( 99) 42 ( 400) 0 0 24 4 57 340 0%
2026-06-05T13:40:12+00:00 91s 20 8 92 397 ( 164) 84 ( 347) 0 0 28 4 56 253 0%
2026-06-05T13:25:08+00:00 88s 13 8 92 326 ( 233) 46 ( 571) 0 0 29 4 56 298 0%
2026-06-05T13:10:25+00:00 105s 14 8 92 370 ( 184) 59 ( 554) 0 0 32 4 55 188 0%
2026-06-05T12:55:56+00:00 135s 17 8 92 343 ( 145) 44 ( 200) 0 0 29 4 55 201 0%
2026-06-05T12:40:08+00:00 88s 15 8 92 312 ( 143) 96 ( 342) 0 0 22 4 55 206 0%
2026-06-05T12:25:08+00:00 88s 13 8 92 310 ( 155) 41 ( 150) 0 0 30 4 56 205 0%
2026-06-05T12:10:14+00:00 94s 15 8 92 341 ( 186) 67 ( 45) 0 0 22 4 55 212 0%
2026-06-05T11:55:24+00:00 104s 17 9 91 278 ( 200) 48 ( 643) 0 0 25 4 55 204 0%
2026-06-05T11:40:07+00:00 87s 16 9 91 546 ( 8) 86 ( 371) 0 0 26 4 55 207 0%
2026-06-05T11:25:06+00:00 86s 14 9 91 220 ( 39) 29 ( 88) 0 0 21 4 55 204 0%
2026-06-05T11:10:21+00:00 102s 14 9 91 312 ( 140) 50 ( 160) 0 0 21 4 56 231 0%
2026-06-05T10:55:11+00:00 91s 15 9 91 324 ( 246) 39 ( 425) 0 0 25 4 56 207 0%
2026-06-05T10:40:26+00:00 106s 19 9 91 289 ( 95) 78 ( 362) 0 0 22 4 55 203 0%
2026-06-05T10:25:06+00:00 87s 15 9 91 321 ( 147) 45 ( 307) 0 0 27 4 55 206 0%
2026-06-05T10:10:04+00:00 85s 14 9 91 333 ( 253) 85 ( 268) 0 0 25 4 55 207 0%
2026-06-05T09:55:09+00:00 90s 14 9 91 327 ( 183) 48 ( 500) 0 0 18 4 73 317 0%
2026-06-05T09:40:21+00:00 102s 18 9 91 325 ( 124) 87 ( 385) 0 0 23 4 55 207 0%
2026-06-05T09:25:35+00:00 116s 20 9 91 328 ( 82) 46 ( 440) 0 0 27 4 55 205 0%
2026-06-05T09:10:20+00:00 101s 17 9 91 377 ( 291) 363 ( 827) 0 0 20 4 55 208 0%
2026-06-05T08:55:06+00:00 87s 14 9 91 414 ( 290) 264 ( 886) 0 0 24 4 56 257 0%
2026-06-05T08:40:21+00:00 102s 18 9 91 422 ( 67) 87 ( 415) 0 0 30 4 55 207 0%
2026-06-05T08:25:15+00:00 92s 15 9 91 405 ( 167) 69 ( 933) 0 0 27 4 55 209 0%
2026-06-05T08:10:19+00:00 101s 16 9 91 514 ( 267) 341 ( 1088) 4 0 29 4 62 849 0%
2026-06-05T07:55:02+00:00 85s 14 9 91 306 ( 116) 106 ( 1125) 0 0 28 4 56 204 0%
2026-06-05T07:40:22+00:00 105s 19 9 91 348 ( 246) 141 ( 492) 0 0 27 4 56 189 0%
2026-06-05T07:25:05+00:00 89s 14 9 91 290 ( 174) 65 ( 483) 0 0 28 4 55 212 0%
2026-06-05T07:10:13+00:00 97s 14 9 91 372 ( 144) 67 ( 750) 0 19 71 4 56 206 0%
2026-06-05T06:55:04+00:00 88s 15 9 91 369 ( 196) 61 ( 486) 0 0 32 4 55 205 0%
2026-06-05T06:40:03+00:00 88s 14 9 91 330 ( 138) 82 ( 375) 0 0 28 4 55 202 0%
2026-06-05T06:25:37+00:00 121s 30 9 91 359 ( 164) 53 ( 750) 0 0 25 4 55 207 0%
2026-06-05T06:10:31+00:00 116s 19 9 91 389 ( 61) 53 ( 172) 0 0 30 4 55 206 0%
2026-06-05T05:55:03+00:00 88s 16 9 91 258 ( 206) 29 ( 646) 0 0 24 4 56 201 0%
2026-06-05T05:40:02+00:00 87s 15 9 91 309 ( 131) 96 ( 415) 0 0 38 4 56 206 0%
2026-06-05T05:25:03+00:00 88s 16 9 91 283 ( 142) 40 ( 864) 0 0 26 4 57 208 0%
2026-06-05T05:10:56+00:00 141s 39 9 91 365 ( 269) 60 ( 625) 0 0 20 4 55 204 0%
2026-06-05T04:55:01+00:00 86s 15 9 91 246 ( 89) 21 ( 523) 0 0 20 4 56 208 0%
2026-06-05T04:40:02+00:00 87s 14 9 91 282 ( 77) 76 ( 353) 0 0 19 4 56 208 0%
2026-06-05T04:25:08+00:00 93s 16 9 91 227 ( 76) 21 ( 563) 0 0 19 4 56 208 0%
2026-06-05T04:10:06+00:00 92s 14 9 91 288 ( 125) 37 ( 130) 0 0 20 4 55 261 0%
2026-06-05T03:55:19+00:00 105s 16 9 91 222 ( 135) 21 ( 659) 0 0 21 4 55 208 0%
2026-06-05T03:40:01+00:00 87s 15 9 91 234 ( 119) 66 ( 363) 0 0 20 4 56 208 0%
2026-06-05T03:25:03+00:00 89s 16 9 91 222 ( 174) 28 ( 727) 0 0 25 4 55 204 0%
2026-06-05T03:10:19+00:00 105s 14 9 91 294 ( 132) 52 ( 144) 0 0 22 4 55 210 0%
2026-06-05T02:55:16+00:00 102s 16 9 91 267 ( 185) 30 ( 773) 0 0 25 4 55 208 0%
2026-06-05T02:40:04+00:00 90s 14 9 91 326 ( 106) 69 ( 359) 0 0 19 4 56 207 0%
2026-06-05T02:25:14+00:00 100s 18 9 91 260 ( 137) 22 ( 591) 0 0 22 4 55 207 0%
2026-06-05T02:10:05+00:00 91s 14 9 91 390 ( 126) 47 ( 119) 0 0 22 4 55 204 0%
2026-06-05T01:55:06+00:00 92s 15 9 91 311 ( 153) 29 ( 175) 0 0 23 4 55 205 0%
2026-06-05T01:40:24+00:00 111s 17 9 91 276 ( 139) 75 ( 347) 0 0 21 4 55 204 0%
2026-06-05T01:25:01+00:00 88s 17 9 91 223 ( 172) 21 ( 525) 0 0 19 4 56 207 0%
2026-06-05T01:10:07+00:00 94s 14 9 91 317 ( 112) 39 ( 275) 0 0 28 4 55 207 0%
2026-06-05T00:55:02+00:00 89s 14 9 91 226 ( 108) 20 ( 556) 0 0 20 4 57 205 0%
2026-06-05T00:40:25+00:00 112s 17 9 91 311 ( 159) 71 ( 365) 0 0 24 4 55 207 0%
2026-06-05T00:25:01+00:00 88s 17 9 91 257 ( 181) 22 ( 556) 0 0 23 4 56 206 0%
2026-06-05T00:10:13+00:00 101s 14 9 91 319 ( 105) 44 ( 425) 0 0 24 4 56 205 0%
2026-06-04T23:55:03+00:00 90s 14 9 91 261 ( 140) 21 ( 525) 0 0 24 4 55 207 0%
2026-06-04T23:40:16+00:00 104s 16 9 91 300 ( 131) 74 ( 368) 0 0 24 4 55 205 0%
2026-06-04T23:24:59+00:00 87s 17 9 91 281 ( 173) 36 ( 778) 0 0 23 4 56 214 0%
2026-06-04T23:10:06+00:00 94s 14 9 91 316 ( 180) 60 ( 900) 0 0 24 4 55 209 0%
2026-06-04T22:55:00+00:00 88s 14 9 91 355 ( 298) 34 ( 700) 0 0 36 4 55 211 0%
2026-06-04T22:40:16+00:00 104s 16 9 91 324 ( 150) 88 ( 314) 0 0 30 4 56 210 0%
2026-06-04T22:24:59+00:00 87s 16 9 91 395 ( 111) 77 ( 186) 0 0 33 4 55 213 0%
2026-06-04T22:10:15+00:00 104s 17 9 91 398 ( 110) 71 ( 174) 0 0 33 4 55 224 0%
2026-06-04T21:54:59+00:00 88s 14 9 91 368 ( 92) 84 ( 168) 0 0 32 4 56 230 0%
2026-06-04T21:40:02+00:00 90s 14 9 91 349 ( 184) 131 ( 294) 0 0 33 4 56 227 0%
2026-06-04T21:25:25+00:00 113s 19 9 91 549 ( 896) 110 ( 158) 0 0 38 4 55 227 0%
2026-06-04T21:10:04+00:00 93s 17 9 91 2132 ( 1335) 140 ( 186) 0 0 64 4 56 230 0%
2026-06-04T20:54:58+00:00 86s 14 9 91 2285 ( 792) 81 ( 105) 0 0 62 4 58 317 0%
2026-06-04T20:40:09+00:00 97s 15 9 91 1858 ( 971) 105 ( 359) 0 0 63 4 56 230 0%
2026-06-04T20:24:58+00:00 86s 18 9 91 1795 ( 1368) 48 ( 320) 0 0 62 4 58 327 0%
2026-06-04T20:10:25+00:00 114s 22 9 91 2016 ( 1417) 71 ( 563) 0 0 62 4 58 231 0%
2026-06-04T19:54:59+00:00 88s 15 9 91 2015 ( 1417) 65 ( 475) 0 0 91 4 56 227 0%
Major LoadDatasets Times Distribution (since last Daily Report):
n = 47, median ~= 91333 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 45
<= 5 min: 2
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Major LoadDatasets Times Distribution (since startup):
n = 1880, median ~= 92395 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 1741
<= 5 min: 84
<= 10 min: 31
<= 20 min: 24
<= 1 hr: 0
> 1 hr: 0
Minor LoadDatasets Times Distribution (since last Daily Report):
n = 1, median ~= 3500 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 1
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Minor LoadDatasets Times Distribution (since startup):
n = 83, median ~= 3306 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 1
<= 200 ms: 6
<= 500 ms: 8
<= 1 s: 7
<= 2 s: 6
<= 5 s: 31
<= 10 s: 5
<= 20 s: 2
<= 1 min: 4
<= 2 min: 13
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last major LoadDatasets):
n = 32, median ~= 611 ms
0 ms: 0
1 ms: 0
2 ms: 1
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 1
<= 50 ms: 2
<= 100 ms: 2
<= 200 ms: 7
<= 500 ms: 1
<= 1 s: 9
<= 2 s: 5
<= 5 s: 2
<= 10 s: 2
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last Daily Report):
n = 3646, median ~= 546 ms
0 ms: 210
1 ms: 137
2 ms: 44
<= 5 ms: 33
<= 10 ms: 47
<= 20 ms: 75
<= 50 ms: 119
<= 100 ms: 102
<= 200 ms: 160
<= 500 ms: 804
<= 1 s: 991
<= 2 s: 425
<= 5 s: 313
<= 10 s: 149
<= 20 s: 35
<= 1 min: 1
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 1
> 1 hr: 0
Response Failed Time Distribution (since startup):
n = 110707, median ~= 375 ms
0 ms: 9352
1 ms: 5354
2 ms: 605
<= 5 ms: 934
<= 10 ms: 1338
<= 20 ms: 1341
<= 50 ms: 5520
<= 100 ms: 6317
<= 200 ms: 14048
<= 500 ms: 18103
<= 1 s: 20913
<= 2 s: 9629
<= 5 s: 10108
<= 10 s: 5238
<= 20 s: 1407
<= 1 min: 261
<= 2 min: 45
<= 5 min: 186
<= 10 min: 4
<= 20 min: 2
<= 1 hr: 1
> 1 hr: 1
Response Succeeded Time Distribution (since last major LoadDatasets):
n = 398, median ~= 82 ms
0 ms: 21
1 ms: 19
2 ms: 19
<= 5 ms: 23
<= 10 ms: 12
<= 20 ms: 22
<= 50 ms: 45
<= 100 ms: 60
<= 200 ms: 33
<= 500 ms: 66
<= 1 s: 28
<= 2 s: 23
<= 5 s: 22
<= 10 s: 5
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last Daily Report):
n = 17219, median ~= 139 ms
0 ms: 1091
1 ms: 1130
2 ms: 596
<= 5 ms: 778
<= 10 ms: 346
<= 20 ms: 750
<= 50 ms: 1836
<= 100 ms: 1414
<= 200 ms: 1701
<= 500 ms: 3004
<= 1 s: 1310
<= 2 s: 987
<= 5 s: 1480
<= 10 s: 713
<= 20 s: 70
<= 1 min: 8
<= 2 min: 0
<= 5 min: 0
<= 10 min: 1
<= 20 min: 1
<= 1 hr: 3
> 1 hr: 0
Response Succeeded Time Distribution (since startup):
n = 632409, median ~= 93 ms
0 ms: 68221
1 ms: 31120
2 ms: 13558
<= 5 ms: 26876
<= 10 ms: 18370
<= 20 ms: 29865
<= 50 ms: 75760
<= 100 ms: 60831
<= 200 ms: 65512
<= 500 ms: 66220
<= 1 s: 49746
<= 2 s: 48725
<= 5 s: 48990
<= 10 s: 25915
<= 20 s: 2111
<= 1 min: 447
<= 2 min: 105
<= 5 min: 18
<= 10 min: 8
<= 20 min: 8
<= 1 hr: 3
> 1 hr: 0
TaskThread Failed Time Distribution (since last Daily Report):
n = 0
TaskThread Failed Time Distribution (since startup):
n = 0
TaskThread Succeeded Time Distribution (since last Daily Report):
n = 0
TaskThread Succeeded Time Distribution (since startup):
n = 0
SgtMap topography nFromCache=51 nNotFromCache=427
GSHHS: nCached=100 of 100, nCoarse=3293, nSuccesses=1754, nTossed=870
NationalBoundaries: nCached=78 of 100, nCoarse=425, nSuccesses=348, nTossed=0
StateBoundaries: nCached=78 of 100, nCoarse=366, nSuccesses=348, nTossed=0
Rivers: nCached=78 of 100, nCoarse=371, nSuccesses=348, nTossed=0
bufferedImage isAccelerated=[unknown]
canonical map sizes: 101 + 748 + 284 + 1610 + 1041 + 1470 = 5254
canonicalStringHolder map sizes: 489 + 888 + 1110 + 556 + 205 + 305 = 3553
Number of threads: Tomcat-waiting=25, inotify=4, other=55
(format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus)
#1 Thread[AsyncFileHandlerWriter-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#2 Thread[Catalina-utility-1,1,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
java.lang.Thread.run(Thread.java:750)
#3 Thread[Catalina-utility-2,1,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
java.lang.Thread.run(Thread.java:750)
#4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#13 Thread[cluster1-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#14 Thread[cluster1-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#26 Thread[cluster2-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#27 Thread[cluster2-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#30 Thread[cluster3-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#31 Thread[cluster3-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#32 Thread[cluster3-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#33 Thread[cluster3-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#34 Thread[cluster3-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#35 Thread[cluster3-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#36 Thread[cluster3-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#37 Thread[cluster3-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#38 Thread[cluster3-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#39 Thread[cluster3-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#40 Thread[cluster3-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#41 Thread[cluster3-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#42 Thread[cluster3-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#43 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#44 Thread[Finalizer,8,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
#45 Thread[GC Daemon,2,system] TIMED_WAITING daemon
java.lang.Object.wait(Native Method)
sun.misc.GC$Daemon.run(GC.java:117)
#46 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:566)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:69)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:128)
java.lang.Thread.run(Thread.java:750)
#47 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:831)
java.lang.Thread.run(Thread.java:750)
#48 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:566)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:69)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:128)
java.lang.Thread.run(Thread.java:750)
#49 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:831)
java.lang.Thread.run(Thread.java:750)
#50 Thread[Java2D Disposer,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
sun.java2d.Disposer.run(Disposer.java:148)
java.lang.Thread.run(Thread.java:750)
#51 Thread[LoadDatasets,5,main] RUNNABLE daemon
com.cohort.array.RowComparatorIgnoreCase.compare(RowComparatorIgnoreCase.java:48)
java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
java.util.TimSort.sort(TimSort.java:234)
java.util.Arrays.sort(Arrays.java:1438)
com.cohort.array.PrimitiveArray.lowRank(PrimitiveArray.java:2881)
com.cohort.array.PrimitiveArray.rankIgnoreCase(PrimitiveArray.java:2869)
com.cohort.array.PrimitiveArray.sortIgnoreCase(PrimitiveArray.java:2923)
gov.noaa.pfel.coastwatch.pointdata.Table.sortIgnoreCase(Table.java:26570)
gov.noaa.pfel.coastwatch.pointdata.Table.leftToRightSortIgnoreCase(Table.java:26650)
gov.noaa.pfel.erddap.dataset.TableWriterDistinct.sortAndRemoveDuplicates(TableWriterDistinct.java:148)
gov.noaa.pfel.erddap.dataset.TableWriterDistinct.writeSome(TableWriterDistinct.java:84)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.getDataForDapQuery(EDDTableFromDatabase.java:1033)
gov.noaa.pfel.erddap.dataset.EDDTable.subsetVariablesDataTable(EDDTable.java:12453)
gov.noaa.pfel.erddap.dataset.EDDTable.distinctSubsetVariablesDataTable(EDDTable.java:12519)
gov.noaa.pfel.erddap.dataset.EDDTable.ensureValid(EDDTable.java:730)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.<init>(EDDTableFromDatabase.java:601)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.lowFromXml(EDDTableFromDatabase.java:248)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.fromXml(EDDTableFromDatabase.java:106)
gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:443)
gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:359)
#52 Thread[main,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
org.apache.catalina.core.StandardServer.await(StandardServer.java:522)
org.apache.catalina.startup.Catalina.await(Catalina.java:826)
org.apache.catalina.startup.Catalina.start(Catalina.java:774)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
#53 Thread[Reference Handler,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#54 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon
java.lang.Thread.sleep(Native Method)
gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288)
#55 Thread[Signal Dispatcher,9,system] RUNNABLE daemon