Serving data from the Marine Institute (Ireland)
Brought to you by the Marine Institute(Ireland)
|
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2026-09-26T07:37:45+00:00
Startup was at 2026-09-13T04:55:02+00:00
Last major LoadDatasets started 13m 38s ago and finished after 105 seconds.
nGridDatasets = 9
nTableDatasets = 75
nTotalDatasets = 84
n Datasets Failed To Load (in the last major LoadDatasets) = 8
ireland_marine_research_database_charts, imiglenamongarms, imifeeaghco2,
compass_neatl_hindcast_grid, IMI_Model_Stats, imifurnaceawqms, spiddal_obs_adcp, IMI_EATL_WAVE,
(end)
ERROR: Duplicate datasetIDs in datasets.xml:
smartbay_metbuoy, intgn_weatherancillary
Unique users (since startup) n = 40775
Response Failed Time (since last major LoadDatasets) n = 77, median ~= 1063 ms
Response Failed Time (since last Daily Report) n = 134, median ~= 164 ms
Response Failed Time (since startup) n = 194215, median ~= 81 ms
Response Succeeded Time (since last major LoadDatasets) n = 418, median ~= 57 ms
Response Succeeded Time (since last Daily Report) n = 1071, median ~= 48 ms
Response Succeeded Time (since startup) n = 862234, median ~= 61 ms
TaskThread has finished 0 out of 0 tasks. Currently, no task is running.
TaskThread Failed Time (since last Daily Report) n = 0
TaskThread Failed Time (since startup) n = 0
TaskThread Succeeded Time (since last Daily Report) n = 0
TaskThread Succeeded Time (since startup) n = 0
Number of threads: Tomcat-waiting=38, inotify=4, other=54
MemoryInUse= 1698 MB (highWaterMark= 4375 MB) (Xmx ~= 5888 MB)
Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open
timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files
---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- -----
2026-09-26T07:25:49+00:00 103s 16 8 84 653 ( 45) 57 ( 1) 0 0 37 4 57 265 0%
2026-09-26T07:10:48+00:00 101s 21 8 84 1037 ( 81) 108 ( 31) 0 0 39 4 54 345 0%
2026-09-26T06:55:44+00:00 98s 13 8 84 588 ( 75) 61 ( 295) 0 0 39 4 55 263 0%
2026-09-26T06:40:37+00:00 90s 16 8 84 551 ( 87) 62 ( 1) 0 0 52 4 55 272 0%
2026-09-26T06:25:55+00:00 108s 17 8 84 634 ( 159) 116 ( 400) 0 1 52 4 55 259 0%
2026-09-26T06:11:11+00:00 125s 24 7 85 829 ( 58) 49 ( 2) 0 0 36 4 56 245 0%
2026-09-26T05:55:38+00:00 92s 12 7 85 395 ( 48) 55 ( 0) 0 0 36 4 55 271 0%
2026-09-26T05:41:04+00:00 118s 17 7 85 479 ( 41) 55 ( 0) 0 0 36 4 55 323 0%
2026-09-26T05:26:11+00:00 125s 37 7 85 601 ( 64) 37 ( 35) 0 0 36 4 55 275 0%
2026-09-26T05:10:53+00:00 107s 15 7 85 958 ( 83) 68 ( 0) 0 0 35 4 56 327 0%
2026-09-26T04:55:32+00:00 86s 12 7 85 486 ( 58) 62 ( 0) 0 0 37 4 55 464 0%
2026-09-26T04:40:33+00:00 87s 13 7 85 475 ( 57) 30 ( 50) 0 0 39 4 55 268 0%
2026-09-26T04:25:33+00:00 87s 13 7 85 457 ( 60) 52 ( 1) 0 0 41 4 56 476 0%
2026-09-26T04:10:45+00:00 99s 12 7 85 738 ( 59) 69 ( 0) 0 0 45 4 55 266 0%
2026-09-26T03:55:56+00:00 110s 14 6 86 525 ( 49) 34 ( 40) 0 0 50 4 55 271 0%
2026-09-26T03:40:32+00:00 87s 12 6 86 617 ( 49) 51 ( 0) 0 0 49 4 56 408 0%
2026-09-26T03:25:34+00:00 89s 12 6 86 524 ( 70) 80 ( 1) 0 0 43 4 55 273 0%
2026-09-26T03:10:45+00:00 100s 11 6 86 806 ( 61) 42 ( 1) 0 0 25 4 55 272 0%
2026-09-26T02:55:38+00:00 93s 11 6 86 484 ( 39) 58 ( 0) 0 0 36 4 56 479 0%
2026-09-26T02:40:51+00:00 106s 16 6 86 470 ( 43) 56 ( 0) 0 0 36 4 56 322 0%
2026-09-26T02:25:36+00:00 91s 12 6 86 497 ( 47) 26 ( 35) 0 0 50 4 55 270 0%
2026-09-26T02:10:32+00:00 87s 11 6 86 998 ( 42) 64 ( 0) 0 0 61 4 55 272 0%
2026-09-26T01:55:34+00:00 89s 11 6 86 498 ( 44) 58 ( 0) 0 0 37 4 55 269 0%
2026-09-26T01:40:49+00:00 105s 15 6 86 632 ( 45) 49 ( 1) 0 0 35 4 57 270 0%
2026-09-26T01:25:31+00:00 87s 12 6 86 660 ( 29) 61 ( 0) 0 0 36 4 56 276 0%
2026-09-26T01:10:30+00:00 86s 11 6 86 804 ( 63) 80 ( 0) 0 0 33 4 59 359 0%
2026-09-26T00:55:31+00:00 87s 11 6 86 526 ( 73) 44 ( 4) 0 0 40 4 55 268 0%
2026-09-26T00:40:51+00:00 107s 14 6 86 593 ( 95) 81 ( 6) 0 0 52 4 55 269 0%
2026-09-26T00:25:33+00:00 89s 14 6 86 491 ( 63) 96 ( 3) 0 0 45 4 55 269 0%
2026-09-26T00:10:39+00:00 95s 11 6 86 906 ( 56) 72 ( 83) 0 0 37 4 56 441 0%
2026-09-25T23:55:32+00:00 89s 14 6 86 517 ( 47) 67 ( 1) 0 0 31 4 55 272 0%
2026-09-25T23:40:46+00:00 103s 14 6 86 507 ( 57) 59 ( 0) 0 0 32 4 55 266 0%
2026-09-25T23:25:29+00:00 86s 14 6 86 501 ( 46) 24 ( 10) 0 0 32 4 55 271 0%
2026-09-25T23:10:42+00:00 100s 13 6 86 896 ( 67) 60 ( 1) 0 0 40 4 55 266 0%
2026-09-25T22:55:28+00:00 86s 11 6 86 560 ( 52) 61 ( 0) 0 0 40 4 55 271 0%
2026-09-25T22:40:30+00:00 87s 13 6 86 510 ( 42) 25 ( 2) 0 0 41 4 55 271 0%
2026-09-25T22:25:44+00:00 102s 15 6 86 488 ( 45) 53 ( 0) 0 0 43 4 55 270 0%
2026-09-25T22:10:33+00:00 91s 11 6 86 927 ( 54) 57 ( 1) 0 0 45 4 55 369 0%
2026-09-25T21:55:26+00:00 84s 11 6 86 522 ( 67) 64 ( 75) 0 0 38 4 55 273 0%
2026-09-25T21:40:27+00:00 85s 12 6 86 527 ( 63) 235 ( 298) 0 0 38 4 55 270 0%
2026-09-25T21:25:31+00:00 89s 12 6 86 471 ( 57) 50 ( 0) 0 0 38 4 55 471 0%
2026-09-25T21:10:49+00:00 107s 15 6 86 964 ( 54) 58 ( 94) 0 0 38 4 55 270 0%
2026-09-25T20:55:27+00:00 85s 11 6 86 597 ( 38) 80 ( 1) 0 0 38 4 55 269 0%
2026-09-25T20:40:29+00:00 87s 13 6 86 530 ( 49) 52 ( 0) 0 0 37 4 56 761 0%
2026-09-25T20:25:28+00:00 87s 12 6 86 581 ( 61) 46 ( 4) 0 0 38 4 55 266 0%
2026-09-25T20:10:45+00:00 104s 14 6 86 945 ( 71) 55 ( 1) 0 0 37 4 56 325 0%
2026-09-25T19:55:28+00:00 87s 11 6 86 429 ( 60) 75 ( 29) 0 0 38 4 55 320 0%
2026-09-25T19:40:27+00:00 86s 12 6 86 563 ( 63) 77 ( 4) 0 0 38 4 55 272 0%
2026-09-25T19:25:28+00:00 87s 12 6 86 486 ( 90) 71 ( 1) 0 0 39 4 55 273 0%
2026-09-25T19:10:29+00:00 88s 11 6 86 1091 ( 77) 71 ( 1) 0 0 39 4 55 273 0%
2026-09-25T18:55:46+00:00 105s 14 6 86 686 ( 70) 62 ( 1) 0 0 40 4 55 269 0%
2026-09-25T18:40:27+00:00 86s 13 6 86 683 ( 110) 53 ( 0) 0 0 41 4 55 261 0%
2026-09-25T18:25:29+00:00 88s 12 6 86 609 ( 49) 45 ( 1) 0 0 42 4 56 421 0%
2026-09-25T18:10:32+00:00 91s 11 6 86 970 ( 90) 51 ( 4) 0 0 45 4 55 271 0%
2026-09-25T17:55:44+00:00 104s 14 6 86 634 ( 42) 58 ( 1) 0 0 45 4 55 438 0%
2026-09-25T17:40:28+00:00 88s 12 6 86 661 ( 42) 83 ( 24) 0 0 49 4 56 479 0%
2026-09-25T17:25:31+00:00 90s 12 6 86 863 ( 17) 53 ( 1) 0 0 49 4 56 470 0%
2026-09-25T17:10:41+00:00 101s 11 6 86 1245 ( 56) 109 ( 146) 0 0 50 4 58 270 0%
2026-09-25T16:55:28+00:00 88s 11 6 86 637 ( 95) 67 ( 150) 0 0 50 4 55 269 0%
2026-09-25T16:40:46+00:00 105s 15 6 86 603 ( 46) 59 ( 1) 0 0 44 4 55 272 0%
2026-09-25T16:25:28+00:00 88s 12 6 86 566 ( 67) 54 ( 0) 0 0 44 4 55 273 0%
2026-09-25T16:10:29+00:00 89s 11 6 86 895 ( 66) 66 ( 80) 0 0 46 4 55 270 0%
2026-09-25T15:55:33+00:00 94s 11 6 86 514 ( 49) 56 ( 1) 0 0 44 4 55 272 0%
2026-09-25T15:40:42+00:00 103s 15 6 86 561 ( 41) 67 ( 1) 0 0 45 4 56 463 0%
2026-09-25T15:25:26+00:00 87s 12 6 86 504 ( 53) 44 ( 10) 0 0 47 4 56 272 0%
2026-09-25T15:10:38+00:00 99s 11 6 86 1070 ( 70) 58 ( 1) 0 0 47 4 56 392 0%
2026-09-25T14:55:27+00:00 88s 11 6 86 556 ( 57) 71 ( 1) 0 0 36 4 69 490 0%
2026-09-25T14:40:42+00:00 103s 16 6 86 667 ( 48) 41 ( 5) 0 0 50 4 55 272 0%
2026-09-25T14:25:27+00:00 88s 12 6 86 659 ( 48) 60 ( 1) 0 0 50 4 55 315 0%
2026-09-25T14:10:40+00:00 101s 11 6 86 1005 ( 49) 74 ( 2) 0 0 49 4 55 269 0%
2026-09-25T13:55:27+00:00 88s 11 6 86 623 ( 39) 37 ( 24) 0 0 49 4 55 271 0%
2026-09-25T13:40:40+00:00 102s 20 6 86 694 ( 41) 84 ( 0) 0 0 46 4 57 775 0%
2026-09-25T13:25:43+00:00 105s 15 6 86 793 ( 41) 67 ( 1) 0 0 48 4 56 265 0%
2026-09-25T13:10:39+00:00 101s 11 6 86 1048 ( 71) 39 ( 15) 0 0 49 4 55 270 0%
2026-09-25T12:55:25+00:00 87s 11 6 86 482 ( 43) 68 ( 1) 0 0 49 4 55 267 0%
2026-09-25T12:40:25+00:00 87s 12 6 86 713 ( 55) 75 ( 2) 0 0 49 4 55 271 0%
2026-09-25T12:25:42+00:00 104s 17 6 86 597 ( 82) 59 ( 950) 0 0 49 4 55 269 0%
2026-09-25T12:10:35+00:00 98s 11 6 86 859 ( 65) 98 ( 5) 0 0 49 4 55 270 0%
2026-09-25T11:55:33+00:00 95s 11 6 86 506 ( 58) 80 ( 1) 0 0 49 4 55 269 0%
2026-09-25T11:40:36+00:00 98s 12 6 86 535 ( 56) 28 ( 20) 0 0 50 4 55 249 0%
2026-09-25T11:25:47+00:00 109s 13 6 86 521 ( 48) 44 ( 0) 0 0 50 4 55 399 0%
2026-09-25T11:11:04+00:00 127s 14 6 86 1216 ( 72) 77 ( 1) 0 0 50 4 55 270 0%
2026-09-25T10:55:38+00:00 101s 11 6 86 629 ( 70) 61 ( 58) 0 0 46 4 55 408 0%
2026-09-25T10:40:41+00:00 104s 12 6 86 830 ( 63) 71 ( 31) 0 0 46 4 55 271 0%
2026-09-25T10:25:40+00:00 103s 14 6 86 743 ( 40) 64 ( 0) 0 0 46 4 55 267 0%
2026-09-25T10:10:58+00:00 121s 16 7 85 998 ( 49) 44 ( 80) 0 0 45 4 55 272 0%
2026-09-25T09:55:36+00:00 100s 13 8 84 538 ( 91) 527 ( 956) 0 0 46 4 56 270 0%
2026-09-25T09:41:11+00:00 134s 14 8 84 1277 ( 73) 155 ( 1203) 0 0 46 4 56 265 0%
2026-09-25T09:25:44+00:00 107s 14 8 84 515 ( 44) 27 ( 58) 0 0 35 4 55 265 0%
2026-09-25T09:11:00+00:00 124s 16 8 84 1123 ( 72) 64 ( 0) 0 0 43 4 55 275 0%
2026-09-25T08:55:36+00:00 100s 13 8 84 566 ( 78) 119 ( 525) 0 0 43 4 55 271 0%
2026-09-25T08:40:39+00:00 103s 14 8 84 769 ( 102) 267 ( 758) 8 0 41 4 57 544 0%
2026-09-25T08:25:34+00:00 99s 14 8 84 606 ( 53) 142 ( 750) 0 0 43 4 55 278 0%
Major LoadDatasets Times Distribution (since last Daily Report):
n = 1, median ~= 90000 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 1
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Major LoadDatasets Times Distribution (since startup):
n = 1257, median ~= 93003 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 1
<= 10 s: 0
<= 20 s: 0
<= 1 min: 1
<= 2 min: 1139
<= 5 min: 111
<= 10 min: 2
<= 20 min: 2
<= 1 hr: 1
> 1 hr: 0
Minor LoadDatasets Times Distribution (since last Daily Report):
n = 0
Minor LoadDatasets Times Distribution (since startup):
n = 27, median ~= 1750 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 7
<= 500 ms: 2
<= 1 s: 0
<= 2 s: 6
<= 5 s: 10
<= 10 s: 2
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last major LoadDatasets):
n = 77, median ~= 1063 ms
0 ms: 7
1 ms: 7
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 1
<= 50 ms: 1
<= 100 ms: 3
<= 200 ms: 8
<= 500 ms: 5
<= 1 s: 6
<= 2 s: 8
<= 5 s: 9
<= 10 s: 18
<= 20 s: 3
<= 1 min: 1
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last Daily Report):
n = 134, median ~= 164 ms
0 ms: 28
1 ms: 17
2 ms: 1
<= 5 ms: 1
<= 10 ms: 2
<= 20 ms: 2
<= 50 ms: 3
<= 100 ms: 6
<= 200 ms: 11
<= 500 ms: 8
<= 1 s: 12
<= 2 s: 11
<= 5 s: 9
<= 10 s: 18
<= 20 s: 4
<= 1 min: 1
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since startup):
n = 194215, median ~= 81 ms
0 ms: 27366
1 ms: 13881
2 ms: 1551
<= 5 ms: 1613
<= 10 ms: 1066
<= 20 ms: 1288
<= 50 ms: 27121
<= 100 ms: 37427
<= 200 ms: 21389
<= 500 ms: 8457
<= 1 s: 14121
<= 2 s: 11069
<= 5 s: 21067
<= 10 s: 3787
<= 20 s: 1439
<= 1 min: 655
<= 2 min: 85
<= 5 min: 416
<= 10 min: 392
<= 20 min: 20
<= 1 hr: 5
> 1 hr: 0
Response Succeeded Time Distribution (since last major LoadDatasets):
n = 418, median ~= 57 ms
0 ms: 40
1 ms: 10
2 ms: 5
<= 5 ms: 17
<= 10 ms: 9
<= 20 ms: 42
<= 50 ms: 79
<= 100 ms: 48
<= 200 ms: 64
<= 500 ms: 36
<= 1 s: 45
<= 2 s: 9
<= 5 s: 12
<= 10 s: 2
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last Daily Report):
n = 1071, median ~= 48 ms
0 ms: 104
1 ms: 40
2 ms: 21
<= 5 ms: 42
<= 10 ms: 46
<= 20 ms: 101
<= 50 ms: 195
<= 100 ms: 119
<= 200 ms: 133
<= 500 ms: 126
<= 1 s: 106
<= 2 s: 12
<= 5 s: 24
<= 10 s: 2
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since startup):
n = 862234, median ~= 61 ms
0 ms: 70066
1 ms: 35098
2 ms: 19134
<= 5 ms: 29872
<= 10 ms: 21333
<= 20 ms: 59349
<= 50 ms: 171546
<= 100 ms: 112233
<= 200 ms: 97701
<= 500 ms: 109892
<= 1 s: 70009
<= 2 s: 20819
<= 5 s: 30967
<= 10 s: 10504
<= 20 s: 2559
<= 1 min: 809
<= 2 min: 271
<= 5 min: 42
<= 10 min: 15
<= 20 min: 10
<= 1 hr: 5
> 1 hr: 0
TaskThread Failed Time Distribution (since last Daily Report):
n = 0
TaskThread Failed Time Distribution (since startup):
n = 0
TaskThread Succeeded Time Distribution (since last Daily Report):
n = 0
TaskThread Succeeded Time Distribution (since startup):
n = 0
SgtMap topography nFromCache=28 nNotFromCache=146
GSHHS: nCached=100 of 100, nCoarse=2189, nSuccesses=959, nTossed=319
NationalBoundaries: nCached=57 of 100, nCoarse=231, nSuccesses=152, nTossed=0
StateBoundaries: nCached=48 of 100, nCoarse=192, nSuccesses=143, nTossed=0
Rivers: nCached=57 of 100, nCoarse=216, nSuccesses=152, nTossed=0
bufferedImage isAccelerated=[unknown]
canonical map sizes: 87 + 761 + 293 + 1582 + 1030 + 1484 = 5237
canonicalStringHolder map sizes: 11221 + 916 + 1159 + 1020 + 535 + 750 = 15601
Number of threads: Tomcat-waiting=38, inotify=4, other=54
(format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus)
#1 Thread[AsyncFileHandlerWriter-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#2 Thread[Catalina-utility-1,1,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
java.lang.Thread.run(Thread.java:750)
#3 Thread[Catalina-utility-2,1,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:80)
java.lang.Thread.run(Thread.java:750)
#4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#13 Thread[cluster1-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#14 Thread[cluster1-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#26 Thread[cluster2-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#27 Thread[cluster2-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#30 Thread[cluster3-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#31 Thread[cluster3-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#32 Thread[cluster3-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#33 Thread[cluster3-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#34 Thread[cluster3-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#35 Thread[cluster3-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#36 Thread[cluster3-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#37 Thread[cluster3-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#38 Thread[cluster3-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#39 Thread[cluster3-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#40 Thread[cluster3-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#41 Thread[cluster3-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#42 Thread[cluster3-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#43 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#44 Thread[Finalizer,8,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
#45 Thread[GC Daemon,2,system] TIMED_WAITING daemon
java.lang.Object.wait(Native Method)
sun.misc.GC$Daemon.run(GC.java:117)
#46 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:825)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:146)
java.lang.Thread.run(Thread.java:750)
#47 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1130)
java.lang.Thread.run(Thread.java:750)
#48 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:825)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:146)
java.lang.Thread.run(Thread.java:750)
#49 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1130)
java.lang.Thread.run(Thread.java:750)
#50 Thread[Java2D Disposer,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
sun.java2d.Disposer.run(Disposer.java:148)
java.lang.Thread.run(Thread.java:750)
#51 Thread[main,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
org.apache.catalina.core.StandardServer.await(StandardServer.java:527)
org.apache.catalina.startup.Catalina.await(Catalina.java:951)
org.apache.catalina.startup.Catalina.start(Catalina.java:899)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:348)
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)
#52 Thread[Reference Handler,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#53 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon
java.lang.Thread.sleep(Native Method)
gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288)
#54 Thread[Signal Dispatcher,9,system] RUNNABLE daemon