Serving data from the Marine Institute (Ireland)
Brought to you by the Marine Institute(Ireland)
|
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2022-11-27T07:50:35+00:00
Startup was at 2022-11-17T07:16:55+00:00
Last major LoadDatasets started 2m 13s ago and is still running.
nGridDatasets = 10
nTableDatasets = 58
nTotalDatasets = 68
n Datasets Failed To Load (in the last major LoadDatasets) = 4
IMINewportBuoys, imiglenamongarms, imifeeaghco2, ireland_marine_research_database_charts, (end)
Unique users (since startup) n = 3369
Response Failed Time (since last major LoadDatasets) n = 46, median ~= 4455 ms
Response Failed Time (since last Daily Report) n = 153, median ~= 3958 ms
Response Failed Time (since startup) n = 27004, median ~= 465 ms
Response Succeeded Time (since last major LoadDatasets) n = 202, median ~= 473 ms
Response Succeeded Time (since last Daily Report) n = 626, median ~= 474 ms
Response Succeeded Time (since startup) n = 137419, median ~= 96 ms
TaskThread has finished 0 out of 0 tasks. Currently, no task is running.
TaskThread Failed Time (since last Daily Report) n = 0
TaskThread Failed Time (since startup) n = 0
TaskThread Succeeded Time (since last Daily Report) n = 0
TaskThread Succeeded Time (since startup) n = 0
Number of threads: Tomcat-waiting=24, inotify=3, other=55
MemoryInUse= 249 MB (highWaterMark= 1799 MB) (Xmx ~= 5888 MB)
Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open
timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files
---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- -----
2022-11-27T07:37:31+00:00 249s 14 4 68 225 ( 373) 58 ( 3600) 0 0 10 3 54 91 0%
2022-11-27T07:21:52+00:00 211s 10 4 68 199 ( 672) 49 ( 3950) 0 0 10 3 54 86 0%
2022-11-27T07:07:28+00:00 246s 11 4 68 159 ( 410) 35 ( 6250) 0 0 10 3 54 89 0%
2022-11-27T06:51:54+00:00 212s 12 4 68 177 ( 446) 37 ( 5962) 0 0 9 3 55 93 0%
2022-11-27T06:37:26+00:00 245s 13 4 68 152 ( 667) 36 ( 6000) 0 0 10 3 54 92 0%
2022-11-27T06:21:56+00:00 214s 10 4 68 168 ( 565) 39 ( 5536) 0 0 10 3 54 91 0%
2022-11-27T06:06:50+00:00 208s 10 4 68 177 ( 413) 36 ( 6071) 0 0 10 3 54 92 0%
2022-11-27T05:51:50+00:00 208s 12 4 68 173 ( 539) 37 ( 5893) 0 0 10 3 54 91 0%
2022-11-27T05:37:21+00:00 240s 13 4 68 173 ( 717) 47 ( 4250) 0 0 10 3 54 92 0%
2022-11-27T05:21:55+00:00 214s 12 4 68 172 ( 467) 39 ( 5577) 0 0 10 3 54 93 0%
2022-11-27T05:06:56+00:00 216s 11 4 68 165 ( 463) 35 ( 6346) 0 0 10 3 54 93 0%
2022-11-27T04:51:52+00:00 211s 12 4 68 174 ( 571) 37 ( 5833) 0 0 10 3 54 91 0%
2022-11-27T04:37:35+00:00 254s 11 4 68 171 ( 583) 36 ( 6154) 0 0 10 3 54 93 0%
2022-11-27T04:22:05+00:00 225s 12 4 68 204 ( 221) 38 ( 5714) 0 0 10 3 54 90 0%
2022-11-27T04:07:05+00:00 224s 10 4 68 199 ( 191) 36 ( 6154) 0 0 10 3 54 92 0%
2022-11-27T03:52:05+00:00 224s 12 4 68 173 ( 560) 37 ( 5962) 0 0 10 3 54 91 0%
2022-11-27T03:37:43+00:00 262s 12 4 68 148 ( 473) 37 ( 6250) 0 0 10 3 54 93 0%
2022-11-27T03:21:56+00:00 216s 9 4 68 76 ( 1250) 24 ( 6111) 0 0 10 3 54 92 0%
2022-11-27T03:12:33+00:00 553s 12 4 68 177 ( 776) 38 ( 6071) 0 0 10 3 54 86 0%
2022-11-27T02:57:20+00:00 540s 12 3 69 157 ( 688) 33 ( 6607) 0 0 10 3 54 90 0%
2022-11-27T02:43:05+00:00 585s 11 3 69 157 ( 844) 40 ( 5417) 0 0 10 3 75 86 0%
2022-11-27T02:27:20+00:00 540s 13 3 69 170 ( 763) 36 ( 6071) 0 0 10 3 54 89 0%
2022-11-27T02:12:03+00:00 523s 12 3 69 164 ( 889) 44 ( 5000) 0 0 10 3 54 82 0%
2022-11-27T01:57:16+00:00 536s 11 3 69 135 ( 438) 27 ( 4700) 0 0 10 3 54 91 0%
2022-11-27T01:43:17+00:00 597s 10 3 69 229 ( 958) 51 ( 6029) 0 0 30 3 54 77 0%
2022-11-27T01:24:57+00:00 397s 9 3 69 187 ( 1167) 52 ( 5526) 0 0 10 3 54 89 0%
2022-11-27T01:06:51+00:00 212s 8 3 69 116 ( 200) 27 ( 7188) 0 0 10 3 54 93 0%
2022-11-27T00:55:16+00:00 416s 13 3 69 185 ( 330) 49 ( 8393) 0 0 19 3 54 89 0%
2022-11-27T00:40:30+00:00 430s 9 3 69 86 ( 1250) 34 ( 8571) 0 0 10 3 54 90 0%
2022-11-27T00:28:34+00:00 615s 9 3 69 214 ( 482) 112 ( 10000) 0 0 35 3 54 81 0%
2022-11-27T00:06:54+00:00 215s 9 3 69 177 ( 194) 47 ( 7500) 0 0 10 3 54 93 0%
2022-11-26T23:52:13+00:00 234s 12 3 69 172 ( 575) 36 ( 6154) 0 0 10 3 54 91 0%
2022-11-26T23:37:44+00:00 265s 11 3 69 202 ( 806) 52 ( 3846) 0 0 10 3 54 92 0%
2022-11-26T23:21:55+00:00 216s 9 3 69 186 ( 688) 50 ( 3667) 0 0 10 3 54 91 0%
2022-11-26T23:06:54+00:00 215s 10 3 69 181 ( 568) 46 ( 5556) 0 0 10 3 54 93 0%
2022-11-26T22:52:05+00:00 226s 13 3 69 195 ( 335) 36 ( 6250) 0 0 10 3 54 93 0%
2022-11-26T22:37:30+00:00 251s 11 3 69 202 ( 529) 47 ( 3833) 0 0 10 3 54 93 0%
2022-11-26T22:21:47+00:00 209s 9 3 69 169 ( 768) 43 ( 5156) 0 0 24 3 54 89 0%
2022-11-26T22:06:52+00:00 214s 9 3 69 198 ( 429) 58 ( 5000) 0 0 15 3 54 93 0%
2022-11-26T21:51:50+00:00 212s 12 3 69 151 ( 488) 36 ( 6000) 0 0 10 3 54 91 0%
2022-11-26T21:37:14+00:00 236s 10 3 69 193 ( 486) 44 ( 4455) 0 0 10 3 54 93 0%
2022-11-26T21:21:51+00:00 212s 12 3 69 162 ( 938) 50 ( 3800) 0 0 10 3 54 87 0%
2022-11-26T21:06:53+00:00 214s 9 3 69 232 ( 750) 53 ( 5156) 0 0 10 3 54 93 0%
2022-11-26T20:51:50+00:00 212s 13 3 69 170 ( 1000) 47 ( 6406) 0 0 32 3 54 92 0%
2022-11-26T20:37:19+00:00 241s 10 3 69 170 ( 475) 36 ( 6250) 0 0 10 3 54 92 0%
2022-11-26T20:21:45+00:00 207s 9 3 69 194 ( 636) 49 ( 4550) 0 0 10 3 54 93 0%
2022-11-26T20:06:48+00:00 210s 10 3 69 168 ( 333) 42 ( 5000) 0 0 10 3 54 93 0%
2022-11-26T19:51:48+00:00 210s 11 3 69 173 ( 450) 36 ( 6071) 0 0 9 3 55 95 0%
2022-11-26T19:37:26+00:00 249s 11 3 69 181 ( 516) 41 ( 4833) 0 0 10 3 54 93 0%
2022-11-26T19:21:45+00:00 207s 9 3 69 147 ( 688) 40 ( 5714) 0 0 10 3 54 92 0%
2022-11-26T19:07:03+00:00 226s 9 3 69 204 ( 700) 49 ( 3950) 0 0 10 3 54 93 0%
2022-11-26T18:52:07+00:00 229s 13 3 69 186 ( 647) 42 ( 5357) 0 0 10 3 54 92 0%
2022-11-26T18:37:21+00:00 244s 10 3 69 189 ( 463) 46 ( 4625) 0 0 16 3 54 90 0%
2022-11-26T18:21:50+00:00 213s 10 3 69 234 ( 1467) 74 ( 4625) 0 0 10 3 54 86 0%
2022-11-26T18:06:52+00:00 215s 9 3 69 259 ( 1607) 80 ( 6190) 0 10 48 3 57 238 0%
2022-11-26T17:51:48+00:00 211s 11 3 69 146 ( 200) 26 ( 5500) 0 0 10 3 54 92 0%
2022-11-26T17:38:19+00:00 302s 10 3 69 201 ( 589) 48 ( 6667) 0 0 35 3 54 82 0%
2022-11-26T17:21:43+00:00 206s 11 3 69 175 ( 455) 39 ( 5536) 0 0 10 3 54 91 0%
2022-11-26T17:06:59+00:00 222s 9 3 69 183 ( 297) 35 ( 6458) 0 0 10 3 54 92 0%
2022-11-26T16:51:51+00:00 214s 12 3 69 156 ( 650) 42 ( 5313) 0 0 17 3 54 85 0%
2022-11-26T16:37:37+00:00 260s 12 4 68 174 ( 329) 36 ( 6154) 0 0 10 3 54 91 0%
2022-11-26T16:21:50+00:00 214s 11 4 68 146 ( 480) 44 ( 4500) 0 0 10 3 54 92 0%
2022-11-26T16:07:11+00:00 235s 12 5 67 191 ( 285) 42 ( 5000) 0 0 11 3 56 83 0%
2022-11-26T15:52:03+00:00 227s 13 5 67 168 ( 667) 37 ( 5893) 0 0 10 3 54 92 0%
2022-11-26T15:37:58+00:00 282s 13 5 67 267 ( 203) 42 ( 5417) 0 0 10 3 54 90 0%
2022-11-26T15:21:56+00:00 220s 12 5 67 160 ( 405) 38 ( 5667) 0 0 10 3 54 90 0%
2022-11-26T15:07:01+00:00 224s 11 4 68 221 ( 883) 69 ( 3300) 0 0 10 3 54 91 0%
2022-11-26T14:51:57+00:00 221s 12 4 68 251 ( 443) 59 ( 3038) 0 0 10 3 54 91 0%
2022-11-26T14:37:53+00:00 277s 12 4 68 209 ( 491) 39 ( 5179) 0 0 10 3 54 92 0%
2022-11-26T14:22:10+00:00 234s 10 4 68 207 ( 516) 40 ( 5333) 0 0 10 3 54 94 0%
2022-11-26T14:07:13+00:00 237s 12 4 68 196 ( 431) 47 ( 4250) 0 0 10 3 54 92 0%
2022-11-26T13:52:09+00:00 233s 13 4 68 211 ( 350) 48 ( 4100) 0 0 10 3 54 92 0%
2022-11-26T13:37:27+00:00 251s 11 4 68 212 ( 194) 39 ( 5536) 0 0 10 3 54 92 0%
2022-11-26T13:21:57+00:00 222s 9 3 69 190 ( 383) 44 ( 5000) 0 0 10 3 54 91 0%
2022-11-26T13:07:03+00:00 227s 10 3 69 199 ( 370) 41 ( 5167) 0 0 10 3 54 92 0%
2022-11-26T12:52:01+00:00 226s 13 3 69 225 ( 732) 58 ( 3500) 0 0 10 3 54 90 0%
2022-11-26T12:37:34+00:00 258s 11 4 68 179 ( 268) 36 ( 6071) 0 0 10 3 54 75 0%
2022-11-26T12:21:59+00:00 223s 10 3 69 191 ( 803) 50 ( 5833) 0 0 13 3 54 92 0%
2022-11-26T12:07:13+00:00 237s 10 3 69 212 ( 226) 36 ( 5385) 0 0 10 3 54 91 0%
2022-11-26T11:52:10+00:00 235s 12 3 69 172 ( 500) 40 ( 5714) 0 0 10 3 54 95 0%
2022-11-26T11:37:53+00:00 278s 11 3 69 192 ( 330) 37 ( 5893) 0 0 10 3 54 92 0%
2022-11-26T11:22:18+00:00 243s 9 3 69 211 ( 389) 43 ( 5156) 0 0 10 3 54 89 0%
2022-11-26T11:07:18+00:00 243s 9 3 69 204 ( 223) 37 ( 5962) 0 0 10 3 54 90 0%
2022-11-26T10:52:10+00:00 235s 12 3 69 190 ( 200) 35 ( 6167) 0 0 10 3 54 92 0%
2022-11-26T10:37:30+00:00 256s 11 3 69 200 ( 550) 48 ( 5882) 0 0 15 3 54 92 0%
2022-11-26T10:21:51+00:00 217s 9 3 69 169 ( 488) 43 ( 5167) 0 0 10 3 54 92 0%
2022-11-26T10:06:57+00:00 223s 9 3 69 191 ( 318) 36 ( 6071) 0 0 18 3 54 92 0%
2022-11-26T09:52:42+00:00 268s 12 3 69 217 ( 691) 65 ( 2844) 0 0 10 3 54 92 0%
2022-11-26T09:38:04+00:00 289s 13 4 68 193 ( 303) 44 ( 4571) 0 0 3 3 81 130 0%
2022-11-26T09:22:47+00:00 273s 10 4 68 208 ( 806) 63 ( 3676) 0 0 14 3 54 90 0%
2022-11-26T09:07:02+00:00 228s 9 3 69 171 ( 177) 35 ( 4167) 0 0 13 3 54 87 0%
2022-11-26T08:53:44+00:00 330s 32 3 69 229 ( 732) 44 ( 5882) 0 0 31 3 57 85 0%
2022-11-26T08:37:38+00:00 264s 16 4 68 192 ( 417) 39 ( 4813) 0 0 15 3 54 92 0%
Major LoadDatasets Times Distribution (since last Daily Report):
n = 2, median ~= 210000 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 2
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Major LoadDatasets Times Distribution (since startup):
n = 962, median ~= 233473 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 0
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 0
<= 1 s: 0
<= 2 s: 0
<= 5 s: 0
<= 10 s: 0
<= 20 s: 0
<= 1 min: 0
<= 2 min: 0
<= 5 min: 763
<= 10 min: 171
<= 20 min: 28
<= 1 hr: 0
> 1 hr: 0
Minor LoadDatasets Times Distribution (since last Daily Report):
n = 0
Minor LoadDatasets Times Distribution (since startup):
n = 154, median ~= 1367 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 9
<= 100 ms: 15
<= 200 ms: 9
<= 500 ms: 2
<= 1 s: 2
<= 2 s: 109
<= 5 s: 4
<= 10 s: 0
<= 20 s: 2
<= 1 min: 0
<= 2 min: 1
<= 5 min: 1
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last major LoadDatasets):
n = 46, median ~= 4455 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 0
<= 50 ms: 4
<= 100 ms: 0
<= 200 ms: 0
<= 500 ms: 7
<= 1 s: 1
<= 2 s: 2
<= 5 s: 11
<= 10 s: 14
<= 20 s: 7
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since last Daily Report):
n = 153, median ~= 3958 ms
0 ms: 0
1 ms: 0
2 ms: 0
<= 5 ms: 0
<= 10 ms: 0
<= 20 ms: 2
<= 50 ms: 13
<= 100 ms: 2
<= 200 ms: 0
<= 500 ms: 18
<= 1 s: 9
<= 2 s: 9
<= 5 s: 36
<= 10 s: 42
<= 20 s: 22
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Failed Time Distribution (since startup):
n = 27004, median ~= 465 ms
0 ms: 5498
1 ms: 2114
2 ms: 61
<= 5 ms: 108
<= 10 ms: 154
<= 20 ms: 384
<= 50 ms: 2974
<= 100 ms: 627
<= 200 ms: 272
<= 500 ms: 1483
<= 1 s: 1222
<= 2 s: 1364
<= 5 s: 4266
<= 10 s: 4005
<= 20 s: 2273
<= 1 min: 133
<= 2 min: 57
<= 5 min: 9
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last major LoadDatasets):
n = 202, median ~= 473 ms
0 ms: 11
1 ms: 7
2 ms: 5
<= 5 ms: 10
<= 10 ms: 2
<= 20 ms: 6
<= 50 ms: 25
<= 100 ms: 17
<= 200 ms: 8
<= 500 ms: 11
<= 1 s: 14
<= 2 s: 12
<= 5 s: 29
<= 10 s: 42
<= 20 s: 3
<= 1 min: 0
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since last Daily Report):
n = 626, median ~= 474 ms
0 ms: 22
1 ms: 24
2 ms: 13
<= 5 ms: 36
<= 10 ms: 11
<= 20 ms: 11
<= 50 ms: 57
<= 100 ms: 58
<= 200 ms: 39
<= 500 ms: 46
<= 1 s: 42
<= 2 s: 46
<= 5 s: 96
<= 10 s: 112
<= 20 s: 9
<= 1 min: 4
<= 2 min: 0
<= 5 min: 0
<= 10 min: 0
<= 20 min: 0
<= 1 hr: 0
> 1 hr: 0
Response Succeeded Time Distribution (since startup):
n = 137419, median ~= 96 ms
0 ms: 11052
1 ms: 9777
2 ms: 4316
<= 5 ms: 8457
<= 10 ms: 1233
<= 20 ms: 2486
<= 50 ms: 13196
<= 100 ms: 19638
<= 200 ms: 9729
<= 500 ms: 11674
<= 1 s: 10983
<= 2 s: 10909
<= 5 s: 11624
<= 10 s: 9846
<= 20 s: 2273
<= 1 min: 178
<= 2 min: 42
<= 5 min: 3
<= 10 min: 2
<= 20 min: 0
<= 1 hr: 1
> 1 hr: 0
TaskThread Failed Time Distribution (since last Daily Report):
n = 0
TaskThread Failed Time Distribution (since startup):
n = 0
TaskThread Succeeded Time Distribution (since last Daily Report):
n = 0
TaskThread Succeeded Time Distribution (since startup):
n = 0
SgtMap topography nFromCache=31 nNotFromCache=59
GSHHS: nCached=100 of 100, nCoarse=1196, nSuccesses=443, nTossed=245
NationalBoundaries: nCached=46 of 100, nCoarse=115, nSuccesses=94, nTossed=0
StateBoundaries: nCached=46 of 100, nCoarse=99, nSuccesses=94, nTossed=0
Rivers: nCached=46 of 100, nCoarse=105, nSuccesses=94, nTossed=0
bufferedImage isAccelerated=[unknown]
canonical map sizes: 69 + 686 + 250 + 1540 + 987 + 1422 = 4954
canonicalStringHolder map sizes: 283 + 624 + 718 + 279 + 178 + 282 = 2364
Number of threads: Tomcat-waiting=24, inotify=3, other=55
(format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus)
#1 Thread[AsyncFileHandlerWriter-159413332,5,main] TIMED_WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:159)
#2 Thread[Catalina-utility-1,1,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:750)
#3 Thread[Catalina-utility-2,1,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:750)
#4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#13 Thread[cluster1-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#14 Thread[cluster1-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:286)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#26 Thread[cluster2-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#27 Thread[cluster2-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#30 Thread[cluster3-connection-reaper-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#31 Thread[cluster3-nio-worker-0,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#32 Thread[cluster3-nio-worker-1,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#33 Thread[cluster3-nio-worker-2,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#34 Thread[cluster3-nio-worker-3,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#35 Thread[cluster3-nio-worker-4,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#36 Thread[cluster3-nio-worker-5,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#37 Thread[cluster3-nio-worker-6,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:192)
io.netty.channel.epoll.Native.epollWait(Native.java:185)
io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#38 Thread[cluster3-nio-worker-7,5,main] RUNNABLE
io.netty.channel.epoll.Native.epollWait0(Native Method)
io.netty.channel.epoll.Native.epollWait(Native.java:176)
io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281)
io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#39 Thread[cluster3-reconnection-0,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#40 Thread[cluster3-reconnection-1,5,main] WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#41 Thread[cluster3-scheduled-task-worker-0,5,main] TIMED_WAITING
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#42 Thread[cluster3-timeouter-0,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
#43 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
#44 Thread[Finalizer,8,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
#45 Thread[GC Daemon,2,system] TIMED_WAITING daemon
java.lang.Object.wait(Native Method)
sun.misc.GC$Daemon.run(GC.java:117)
#46 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:540)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106)
java.lang.Thread.run(Thread.java:750)
#47 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:750)
#48 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:540)
org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78)
org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106)
java.lang.Thread.run(Thread.java:750)
#49 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:787)
java.lang.Thread.run(Thread.java:750)
#50 Thread[Java2D Disposer,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
sun.java2d.Disposer.run(Disposer.java:148)
java.lang.Thread.run(Thread.java:750)
#51 Thread[LoadDatasets,5,main] RUNNABLE daemon
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:2023)
com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6465)
com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6368)
com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6346)
com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6647)
com.microsoft.sqlserver.jdbc.SimpleInputStream.read(SimpleInputStream.java:302)
com.microsoft.sqlserver.jdbc.SimpleInputStream.read(SimpleInputStream.java:264)
com.microsoft.sqlserver.jdbc.SimpleInputStream.getBytes(SimpleInputStream.java:349)
com.microsoft.sqlserver.jdbc.DDC.convertStreamToObject(DDC.java:660)
com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3773)
com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:247)
com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:190)
com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2041)
com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2027)
com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2489)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.getDataForDapQuery(EDDTableFromDatabase.java:988)
gov.noaa.pfel.erddap.dataset.EDDTable.subsetVariablesDataTable(EDDTable.java:12453)
gov.noaa.pfel.erddap.dataset.EDDTable.distinctSubsetVariablesDataTable(EDDTable.java:12519)
gov.noaa.pfel.erddap.dataset.EDDTable.ensureValid(EDDTable.java:730)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.<init>(EDDTableFromDatabase.java:601)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.lowFromXml(EDDTableFromDatabase.java:248)
gov.noaa.pfel.erddap.dataset.EDDTableFromDatabase.fromXml(EDDTableFromDatabase.java:106)
gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:443)
gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:359)
#52 Thread[main,5,main] TIMED_WAITING
java.lang.Thread.sleep(Native Method)
org.apache.catalina.core.StandardServer.await(StandardServer.java:563)
org.apache.catalina.startup.Catalina.await(Catalina.java:864)
org.apache.catalina.startup.Catalina.start(Catalina.java:810)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345)
org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:476)
#53 Thread[Reference Handler,10,system] WAITING daemon
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
#54 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon
java.lang.Thread.sleep(Native Method)
gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288)
#55 Thread[Signal Dispatcher,9,system] RUNNABLE daemon