Serving data from the Marine Institute (Ireland)
Brought to you by the Marine Institute(Ireland)
|
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2025-12-31T12:48:20+00:00
Startup was at 2025-12-14T07:16:28+00:00
Last major LoadDatasets started 8m 41s ago and is still running.
nGridDatasets = 3
nTableDatasets = 77
nTotalDatasets = 80
n Datasets Failed To Load (in the last major LoadDatasets) = 11
BANTRY_PARTICLES, ireland_marine_research_database_charts, imifeeaghco2, IMI_IRISH_SHELF_SWAN_WAVE,
IMI_CONNEMARA_SWAN_WAVE, IMI_NEATL, IMI_CONN_3D, compass_neatl_hindcast_grid, IMI_Model_Stats,
IMI_CONN_2D, IMI_EATL_WAVE, (end)
Unique users (since startup) n = 22892
Response Failed Time (since last major LoadDatasets) n = 13, median ~= 2750 ms
Response Failed Time (since last Daily Report) n = 1260, median ~= 1003 ms
Response Failed Time (since startup) n = 46567, median ~= 788 ms
Response Succeeded Time (since last major LoadDatasets) n = 161, median ~= 276 ms
Response Succeeded Time (since last Daily Report) n = 5016, median ~= 161 ms
Response Succeeded Time (since startup) n = 307859, median ~= 154 ms
TaskThread has finished 0 out of 0 tasks. Currently, no task is running.
TaskThread Failed Time (since last Daily Report) n = 0
TaskThread Failed Time (since startup) n = 0
TaskThread Succeeded Time (since last Daily Report) n = 0
TaskThread Succeeded Time (since startup) n = 0
Number of threads: Tomcat-waiting=21, inotify=6, other=55
MemoryInUse= 1401 MB (highWaterMark= 2285 MB) (Xmx ~= 5888 MB)
Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open
timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files
---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- -----
2025-12-31T12:35:04+00:00 626s 18 11 80 607 ( 106) 10 ( 500) 0 0 20 4 55 207 0%
2025-12-31T12:20:03+00:00 625s 16 11 80 360 ( 6) 9 ( 92) 0 0 20 4 55 162 0%
2025-12-31T12:05:02+00:00 624s 17 11 80 107 ( 195) 9 ( 75) 0 0 20 4 55 173 0%
2025-12-31T11:50:17+00:00 639s 17 11 80 202 ( 235) 15 ( 2500) 0 0 25 4 55 167 0%
2025-12-31T11:35:01+00:00 623s 17 11 80 105 ( 90) 20 ( 300) 0 0 18 4 57 173 0%
2025-12-31T11:20:01+00:00 623s 17 11 80 228 ( 10) 9 ( 75) 0 0 19 4 56 171 0%
2025-12-31T11:05:00+00:00 622s 16 11 80 115 ( 88) 5 ( 92) 0 0 19 4 56 379 0%
2025-12-31T10:50:01+00:00 624s 16 11 80 209 ( 321) 7 ( 94) 0 0 18 4 57 226 0%
2025-12-31T10:35:18+00:00 641s 18 11 80 245 ( 396) 270 ( 863) 0 0 23 4 55 150 0%
2025-12-31T10:20:00+00:00 622s 16 11 80 197 ( 307) 27 ( 964) 0 0 19 4 56 164 0%
2025-12-31T10:05:00+00:00 623s 16 11 80 176 ( 189) 32 ( 3125) 0 0 25 4 56 165 0%
2025-12-31T09:49:59+00:00 622s 16 11 80 482 ( 6) 24 ( 1143) 0 0 20 4 55 232 0%
2025-12-31T09:35:16+00:00 638s 18 11 80 242 ( 482) 196 ( 1364) 2 0 18 4 57 558 0%
2025-12-31T09:20:01+00:00 624s 16 11 80 244 ( 332) 241 ( 1383) 0 0 18 4 57 165 0%
2025-12-31T09:05:43+00:00 666s 31 11 80 207 ( 283) 29 ( 1071) 0 0 19 4 56 155 0%
2025-12-31T08:50:02+00:00 625s 16 11 80 164 ( 98) 6 ( 75) 0 0 19 4 56 169 0%
2025-12-31T08:35:14+00:00 637s 18 11 80 166 ( 308) 24 ( 813) 0 0 18 4 57 209 0%
2025-12-31T08:19:59+00:00 623s 16 11 80 240 ( 423) 139 ( 895) 2 0 21 4 55 165 0%
2025-12-31T08:05:12+00:00 636s 21 11 80 106 ( 154) 43 ( 925) 0 0 20 4 55 167 0%
2025-12-31T07:50:57+00:00 680s 35 11 80 203 ( 215) 70 ( 2000) 0 0 20 4 55 163 0%
2025-12-31T07:35:02+00:00 625s 21 12 79 80 ( 142) 6 ( 200) 0 0 18 4 57 172 0%
2025-12-31T07:20:15+00:00 638s 22 12 79 170 ( 244) 56 ( 1000) 0 0 28 4 55 161 0%
2025-12-31T07:05:10+00:00 634s 18 12 79 79 ( 121) 8 ( 75) 0 0 20 4 55 171 0%
2025-12-31T06:50:00+00:00 624s 17 12 79 193 ( 196) 9 ( 125) 0 0 20 4 55 169 0%
2025-12-31T06:34:59+00:00 623s 17 12 79 107 ( 287) 28 ( 500) 0 0 20 4 55 146 0%
2025-12-31T06:20:15+00:00 639s 19 12 79 220 ( 292) 17 ( 47) 0 0 20 4 55 168 0%
2025-12-31T06:05:00+00:00 624s 17 12 79 100 ( 100) 8 ( 35) 0 0 18 4 57 313 0%
2025-12-31T05:50:19+00:00 643s 17 12 79 129 ( 66) 6 ( 50) 0 0 20 4 55 169 0%
2025-12-31T05:35:19+00:00 644s 17 12 79 857 ( 0) 6 ( 300) 0 0 20 4 55 177 0%
2025-12-31T05:20:12+00:00 637s 17 12 79 268 ( 0) 11 ( 175) 0 0 26 4 55 170 0%
2025-12-31T05:07:02+00:00 746s 19 12 79 275 ( 0) 46 ( 5833) 0 0 27 4 55 149 0%
2025-12-31T04:50:01+00:00 626s 19 13 78 313 ( 0) 10 ( 200) 0 0 12 4 70 279 0%
2025-12-31T04:35:00+00:00 625s 18 13 78 261 ( 0) 11 ( 75) 0 0 19 4 56 248 0%
2025-12-31T04:20:15+00:00 639s 19 13 78 284 ( 0) 12 ( 67) 0 0 20 4 55 164 0%
2025-12-31T04:04:59+00:00 624s 19 13 78 201 ( 0) 5 ( 150) 0 0 20 4 55 175 0%
2025-12-31T03:49:59+00:00 624s 18 13 78 201 ( 125) 8 ( 100) 0 0 19 4 56 175 0%
2025-12-31T03:34:58+00:00 623s 18 13 78 71 ( 136) 4 ( 100) 0 0 20 4 55 172 0%
2025-12-31T03:20:00+00:00 625s 18 13 78 125 ( 186) 7 ( 150) 0 0 20 4 55 167 0%
2025-12-31T03:05:14+00:00 639s 21 13 78 80 ( 114) 5 ( 125) 0 0 19 4 56 180 0%
2025-12-31T02:49:59+00:00 624s 18 13 78 170 ( 238) 7 ( 150) 0 0 20 4 55 173 0%
2025-12-31T02:35:01+00:00 626s 18 13 78 90 ( 123) 13 ( 75) 0 0 18 4 57 294 0%
2025-12-31T02:19:59+00:00 623s 18 13 78 161 ( 156) 7 ( 175) 0 0 20 4 55 170 0%
2025-12-31T02:04:51+00:00 642s 22 13 78 121 ( 43) 5 ( 75) 0 0 20 4 55 171 0%
2025-12-31T01:49:35+00:00 626s 18 13 78 173 ( 244) 9 ( 92) 0 0 20 4 55 171 0%
2025-12-31T01:34:34+00:00 626s 18 13 78 114 ( 57) 8 ( 50) 0 0 19 4 56 215 0%
2025-12-31T01:19:34+00:00 626s 18 13 78 158 ( 94) 7 ( 150) 0 0 20 4 56 173 0%
2025-12-31T01:04:46+00:00 637s 20 13 78 106 ( 85) 7 ( 88) 0 0 20 4 55 168 0%
2025-12-31T00:49:31+00:00 623s 18 13 78 174 ( 251) 9 ( 92) 0 0 20 4 55 178 0%
2025-12-31T00:34:32+00:00 624s 18 13 78 105 ( 161) 21 ( 2938) 0 0 20 4 55 172 0%
2025-12-31T00:19:35+00:00 627s 21 13 78 151 ( 188) 8 ( 50) 0 0 20 4 55 246 0%
2025-12-31T00:04:46+00:00 638s 19 13 78 97 ( 146) 8 ( 30) 0 0 20 4 55 168 0%
2025-12-30T23:49:33+00:00 625s 23 13 78 204 ( 243) 23 ( 2900) 0 0 20 4 55 170 0%
2025-12-30T23:34:32+00:00 624s 18 13 78 200 ( 67) 7 ( 63) 0 0 20 4 55 173 0%
2025-12-30T23:19:33+00:00 625s 18 13 78 176 ( 314) 23 ( 88) 0 0 27 4 55 157 0%
2025-12-30T23:04:32+00:00 624s 19 13 78 742 ( 0) 15 ( 2750) 0 0 26 4 55 167 0%
2025-12-30T22:49:47+00:00 639s 19 13 78 288 ( 36) 9 ( 88) 0 0 20 4 55 166 0%
2025-12-30T22:34:31+00:00 623s 20 13 78 177 ( 182) 18 ( 10) 0 0 20 4 55 186 0%
2025-12-30T22:19:32+00:00 624s 19 14 77 131 ( 197) 7 ( 75) 0 0 20 4 55 171 0%
2025-12-30T22:04:36+00:00 628s 19 14 77 99 ( 185) 20 ( 3000) 0 0 18 4 57 443 0%
2025-12-30T21:49:46+00:00 638s 19 13 78 181 ( 248) 10 ( 90) 0 0 20 4 55 174 0%
2025-12-30T21:34:31+00:00 624s 19 13 78 115 ( 73) 182 ( 1036) 0 0 21 4 55 174 0%
2025-12-30T21:19:30+00:00 623s 18 13 78 310 ( 696) 46 ( 3364) 0 0 42 4 55 163 0%
2025-12-30T21:04:34+00:00 627s 18 13 78 136 ( 300) 15 ( 1500) 0 0 20 4 55 174 0%
2025-12-30T20:49:45+00:00 638s 19 13 78 186 ( 217) 9 ( 94) 0 0 20 4 55 164 0%
2025-12-30T20:34:31+00:00 624s 19 13 78 105 ( 114) 9 ( 63) 0 0 20 4 55 168 0%
2025-12-30T20:19:34+00:00 627s 18 13 78 211 ( 338) 18 ( 2000) 0 0 29 4 55 155 0%
2025-12-30T20:04:30+00:00 623s 18 13 78 102 ( 252) 18 ( 4000) 0 0 20 4 55 187 0%
2025-12-30T19:49:46+00:00 639s 19 13 78 195 ( 215) 10 ( 100) 0 0 20 4 55 172 0%
2025-12-30T19:34:31+00:00 625s 19 13 78 80 ( 232) 13 ( 2500) 0 0 20 4 55 170 0%
2025-12-30T19:19:29+00:00 623s 20 13 78 154 ( 257) 9 ( 150) 0 0 20 4 55 175 0%
2025-12-30T19:04:29+00:00 622s 19 13 78 131 ( 464) 17 ( 4500) 0 0 26 4 55 155 0%
2025-12-30T18:49:33+00:00 627s 19 13 78 153 ( 245) 8 ( 100) 0 0 29 4 55 159 0%
2025-12-30T18:34:48+00:00 642s 21 13 78 185 ( 428) 13 ( 425) 0 0 27 4 56 213 0%
2025-12-30T18:19:33+00:00 626s 18 13 78 140 ( 183) 7 ( 88) 0 0 20 4 55 181 0%
2025-12-30T18:04:28+00:00 622s 18 13 78 111 ( 158) 6 ( 200) 0 0 20 4 55 176 0%
2025-12-30T17:49:28+00:00 622s 18 13 78 191 ( 228) 15 ( 2375) 0 0 29 4 55 158 0%
2025-12-30T17:34:46+00:00 640s 20 13 78 131 ( 269) 23 ( 5833) 1 0 33 4 56 202 0%
2025-12-30T17:19:32+00:00 626s 17 12 79 192 ( 253) 22 ( 400) 0 0 22 4 55 176 0%
2025-12-30T17:04:34+00:00 628s 18 12 79 159 ( 128) 11 ( 1500) 0 0 20 4 55 171 0%
2025-12-30T16:49:31+00:00 625s 18 13 78 218 ( 158) 7 ( 150) 0 0 20 4 55 146 0%
2025-12-30T16:34:42+00:00 636s 19 13 78 151 ( 193) 17 ( 183) 0 0 19 4 56 266 0%
2025-12-30T16:19:29+00:00 623s 19 13 78 225 ( 273) 29 ( 4250) 0 0 20 4 55 174 0%
2025-12-30T16:04:36+00:00 630s 18 13 78 87 ( 207) 12 ( 3500) 0 0 20 4 55 325 0%
2025-12-30T15:49:28+00:00 622s 18 13 78 190 ( 154) 9 ( 88) 0 0 20 4 55 168 0%
2025-12-30T15:34:34+00:00 629s 18 13 78 117 ( 194) 8 ( 200) 0 0 34 4 56 173 0%
2025-12-30T15:19:44+00:00 638s 20 13 78 140 ( 185) 11 ( 88) 0 0 20 4 55 171 0%
2025-12-30T15:04:33+00:00 627s 18 13 78 152 ( 95) 10 ( 1500) 0 0 20 4 55 174 0%
2025-12-30T14:49:28+00:00 623s 18 13 78 211 ( 291) 12 ( 350) 0 0 20 4 55 169 0%
2025-12-30T14:34:32+00:00 626s 20 13 78 154 ( 317) 15 ( 3071) 0 0 24 4 55 168 0%
2025-12-30T14:19:44+00:00 639s 21 13 78 239 ( 373) 13 ( 150) 0 0 20 4 55 171 0%
2025-12-30T14:04:30+00:00 624s 18 13 78 145 ( 169) 9 ( 275) 0 0 20 4 55 164 0%
2025-12-30T13:49:28+00:00 622s 18 13 78 265 ( 230) 20 ( 3000) 0 0 29 4 55 167 0%
2025-12-30T13:34:29+00:00 624s 18 13 78 130 ( 367) 14 ( 2857) 0 0 20 4 55 172 0%
Major LoadDatasets Times Distribution (since last Daily Report):
n = 22, median ~= 900000 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 22
<= 1 hr: 0
> 1 hr: 0
Major LoadDatasets Times Distribution (since startup):
n = 1652, median ~= 115488 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 1
<= 10 s: 0
<= 20 s: 0
<= 1 min: 1
<= 2 min: 891
<= 5 min: 21
<= 10 min: 27
<= 20 min: 710
<= 1 hr: 1
> 1 hr: 0
Minor LoadDatasets Times Distribution (since last Daily Report):
n = 1, median ~= 3500 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 1
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Minor LoadDatasets Times Distribution (since startup):
n = 43, median ~= 3550 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 2
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 4
<= 5 s: 30
<= 10 s: 5
<= 20 s: 0
<= 1 min: 0
<= 2 min: 2
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last major LoadDatasets):
n = 13, median ~= 2750 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 1
<= 50 ms: 2
<= 100 ms: 1
<= 200 ms: 0
<= 500 ms: 1
<= 1 s: 1
<= 2 s: 0
<= 5 s: 2
<= 10 s: 4
<= 20 s: 1
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last Daily Report):
n = 1260, median ~= 1003 ms
0 ms: 19
1 ms: 15
2 ms: 1
<= 5 ms: 5
<= 10 ms: 7
<= 20 ms: 11
<= 50 ms: 33
<= 100 ms: 51
<= 200 ms: 20
<= 500 ms: 86
<= 1 s: 381
<= 2 s: 318
<= 5 s: 228
<= 10 s: 66
<= 20 s: 13
<= 1 min: 3
<= 2 min: 2
<= 5 min: 1
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since startup):
n = 46567, median ~= 788 ms
0 ms: 3423
1 ms: 1852
2 ms: 67
<= 5 ms: 180
<= 10 ms: 238
<= 20 ms: 609
<= 50 ms: 2194
<= 100 ms: 3308
<= 200 ms: 1835
<= 500 ms: 3964
<= 1 s: 9760
<= 2 s: 8007
<= 5 s: 6611
<= 10 s: 2993
<= 20 s: 925
<= 1 min: 185
<= 2 min: 55
<= 5 min: 350
<= 10 min: 5
<= 20 min: 6
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last major LoadDatasets):
n = 161, median ~= 276 ms
0 ms: 3
1 ms: 6
2 ms: 3
<= 5 ms: 6
<= 10 ms: 1
<= 20 ms: 4
<= 50 ms: 20
<= 100 ms: 18
<= 200 ms: 3
<= 500 ms: 65
<= 1 s: 12
<= 2 s: 5
<= 5 s: 13
<= 10 s: 2
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last Daily Report):
n = 5016, median ~= 161 ms
0 ms: 472
1 ms: 223
2 ms: 128
<= 5 ms: 346
<= 10 ms: 206
<= 20 ms: 155
<= 50 ms: 395
<= 100 ms: 391
<= 200 ms: 313
<= 500 ms: 1074
<= 1 s: 551
<= 2 s: 454
<= 5 s: 206
<= 10 s: 87
<= 20 s: 15
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since startup):
n = 307859, median ~= 154 ms
0 ms: 32927
1 ms: 16762
2 ms: 8127
<= 5 ms: 15873
<= 10 ms: 5255
<= 20 ms: 8762
<= 50 ms: 24806
<= 100 ms: 29622
<= 200 ms: 21884
<= 500 ms: 65833
<= 1 s: 31325
<= 2 s: 24355
<= 5 s: 12866
<= 10 s: 7388
<= 20 s: 1653
<= 1 min: 217
<= 2 min: 134
<= 5 min: 21
<= 10 min: 13
<= 20 min: 36
<= 1 hr: 0
> 1 hr: 0
TaskThread Failed Time Distribution (since last Daily Report):
n = 0
TaskThread Failed Time Distribution (since startup):
n = 0
TaskThread Succeeded Time Distribution (since last Daily Report):
n = 0
TaskThread Succeeded Time Distribution (since startup):
n = 0
SgtMap topography nFromCache=25 nNotFromCache=206
GSHHS: nCached=100 of 100, nCoarse=994, nSuccesses=815, nTossed=341
NationalBoundaries: nCached=89 of 100, nCoarse=131, nSuccesses=205, nTossed=0
StateBoundaries: nCached=88 of 100, nCoarse=99, nSuccesses=204, nTossed=0
Rivers: nCached=93 of 100, nCoarse=113, nSuccesses=209, nTossed=0
bufferedImage isAccelerated=[unknown]
canonical map sizes: 267 + 744 + 272 + 1569 + 1009 + 1449 = 5310
canonicalStringHolder map sizes: 621 + 937 + 1153 + 581 + 244 + 392 = 3928
Number of threads: Tomcat-waiting=21, inotify=6, other=55
(format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus)
#1 Thread[AsyncFileHandlerWriter-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#2 Thread[Catalina-utility-1,1,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
java.lang.Thread.run(Thread.java:750)
#3 Thread[Catalina-utility-2,1,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
java.lang.Thread.run(Thread.java:750)
#4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#13 Thread[cluster1-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#14 Thread[cluster1-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#26 Thread[cluster2-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#27 Thread[cluster2-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#30 Thread[cluster3-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#31 Thread[cluster3-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#32 Thread[cluster3-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#33 Thread[cluster3-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#34 Thread[cluster3-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#35 Thread[cluster3-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#36 Thread[cluster3-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#37 Thread[cluster3-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#38 Thread[cluster3-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#39 Thread[cluster3-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#40 Thread[cluster3-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#41 Thread[cluster3-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#42 Thread[cluster3-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#43 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#44 Thread[Finalizer,8,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
#45 Thread[GC Daemon,2,system] TIMED_WAITING daemon
java.lang.Object.wait(Native Method)
sun.misc.GC$Daemon.run(GC.java:117)
#46 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
java.lang.Thread.run(Thread.java:750)
#47 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:811)
java.lang.Thread.run(Thread.java:750)
#48 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
java.lang.Thread.run(Thread.java:750)
#49 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:811)
java.lang.Thread.run(Thread.java:750)
#50 Thread[Java2D Disposer,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
sun.java2d.Disposer.run(Disposer.java:148)
java.lang.Thread.run(Thread.java:750)
#51 Thread[LoadDatasets,5,main] RUNNABLE daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345)
sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:745)
sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:680)
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1610)
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1515)
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
gov.noaa.pfel.coastwatch.util.SSR.getUrlConnBufferedInputStream(SSR.java:1681)
gov.noaa.pfel.coastwatch.util.SSR.getUrlConnBufferedInputStream(SSR.java:1596)
gov.noaa.pfel.coastwatch.util.SSR.getUrlBufferedInputStream(SSR.java:1860)
gov.noaa.pfel.coastwatch.util.SSR.getUrlResponseBytes(SSR.java:2055)
gov.noaa.pfel.erddap.dataset.EDDGridFromDap.<init>(EDDGridFromDap.java:357)
gov.noaa.pfel.erddap.dataset.EDDGridFromDap.fromXml(EDDGridFromDap.java:199)
gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:425)
gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:359)
#52 Thread[main,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
org.apache.catalina.core.StandardServer.await(StandardServer.java:524)
org.apache.catalina.startup.Catalina.await(Catalina.java:829)
org.apache.catalina.startup.Catalina.start(Catalina.java:777)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
#53 Thread[Reference Handler,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#54 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon
java.lang.Thread.sleep(Native Method)
gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288)
#55 Thread[Signal Dispatcher,9,system] RUNNABLE daemon