![]() |
Marine Institute ERDDAP Server
Serving data from the Marine Institute (Ireland) |
Brought to you by the Marine Institute(Ireland) |
Current time is 2025-02-13T08:38:18+00:00 Startup was at 2025-02-07T07:16:09+00:00 Last major LoadDatasets started 6m 6s ago and finished after 75 seconds. nGridDatasets = 9 nTableDatasets = 70 nTotalDatasets = 79 n Datasets Failed To Load (in the last major LoadDatasets) = 7 ireland_marine_research_database_charts, imiglenamongarms, imifeeaghco2, imiroughriverarms, compass_neatl_hindcast_grid, IMI_Model_Stats, IMI_EATL_WAVE, (end) Unique users (since startup) n = 2465 Response Failed Time (since last major LoadDatasets) n = 18, median ~= 400 ms Response Failed Time (since last Daily Report) n = 1142, median ~= 1077 ms Response Failed Time (since startup) n = 32485, median ~= 453 ms Response Succeeded Time (since last major LoadDatasets) n = 25, median ~= 22 ms Response Succeeded Time (since last Daily Report) n = 1138, median ~= 247 ms Response Succeeded Time (since startup) n = 75337, median ~= 167 ms TaskThread has finished 0 out of 0 tasks. Currently, no task is running. TaskThread Failed Time (since last Daily Report) n = 0 TaskThread Failed Time (since startup) n = 0 TaskThread Succeeded Time (since last Daily Report) n = 0 TaskThread Succeeded Time (since startup) n = 0 Number of threads: Tomcat-waiting=19, inotify=4, other=48 MemoryInUse= 1045 MB (highWaterMark= 3510 MB) (Xmx ~= 5888 MB) Major LoadDatasets Time Series: MLD Datasets Loaded Requests (median times in ms) Number of Threads MB Open timestamp time nTry nFail nTotal nSuccess (median) nFail (median) memFail tooMany tomWait inotify other inUse Files ---------------------------- ----- ----------------- ------------------------------------------------ --------------------- ----- ----- 2025-02-13T08:33:22+00:00 71s 12 7 79 168 ( 262) 218 ( 740) 0 0 20 4 49 108 0% 2025-02-13T08:18:38+00:00 86s 15 7 79 137 ( 68) 127 ( 528) 2 0 19 4 50 201 0% 2025-02-13T08:03:24+00:00 73s 12 7 79 232 ( 156) 104 ( 911) 0 0 21 4 49 106 0% 2025-02-13T07:48:27+00:00 75s 15 7 79 237 ( 563) 472 ( 1350) 1 0 35 4 50 90 0% 2025-02-13T07:33:31+00:00 79s 15 7 79 204 ( 357) 149 ( 497) 0 0 35 4 50 101 0% 2025-02-13T07:18:35+00:00 84s 15 7 79 135 ( 383) 54 ( 500) 0 0 20 4 49 107 0% 2025-02-13T07:03:23+00:00 71s 14 7 79 95 ( 66) 30 ( 338) 0 0 20 4 49 106 0% 2025-02-13T06:48:23+00:00 72s 12 7 79 91 ( 49) 28 ( 440) 0 0 20 4 49 102 0% 2025-02-13T06:33:22+00:00 71s 15 7 79 125 ( 196) 144 ( 482) 0 0 25 4 49 104 0% 2025-02-13T06:18:24+00:00 73s 14 7 79 144 ( 300) 30 ( 340) 0 0 20 4 49 105 0% 2025-02-13T06:03:39+00:00 88s 13 7 79 86 ( 49) 28 ( 350) 0 0 20 4 49 105 0% 2025-02-13T05:48:21+00:00 71s 12 7 79 88 ( 55) 29 ( 330) 0 0 20 4 49 104 0% 2025-02-13T05:33:22+00:00 72s 15 7 79 95 ( 97) 62 ( 350) 0 0 20 4 49 107 0% 2025-02-13T05:18:24+00:00 73s 14 7 79 95 ( 48) 32 ( 320) 0 0 20 4 49 107 0% 2025-02-13T05:03:33+00:00 83s 13 7 79 77 ( 68) 32 ( 300) 0 0 20 4 49 105 0% 2025-02-13T04:48:21+00:00 71s 13 7 79 79 ( 54) 28 ( 340) 0 0 20 4 49 108 0% 2025-02-13T04:33:23+00:00 72s 14 7 79 94 ( 167) 87 ( 354) 0 0 20 4 49 107 0% 2025-02-13T04:18:22+00:00 72s 14 7 79 89 ( 89) 30 ( 340) 0 0 20 4 49 104 0% 2025-02-13T04:03:21+00:00 71s 12 7 79 93 ( 79) 29 ( 341) 0 0 20 4 49 107 0% 2025-02-13T03:48:40+00:00 90s 13 7 79 86 ( 58) 28 ( 340) 0 0 20 4 49 104 0% 2025-02-13T03:33:21+00:00 71s 12 7 79 93 ( 81) 62 ( 353) 0 0 20 4 49 105 0% 2025-02-13T03:18:23+00:00 73s 16 7 79 80 ( 67) 31 ( 330) 0 0 20 4 49 105 0% 2025-02-13T03:03:21+00:00 71s 14 7 79 100 ( 65) 28 ( 340) 0 0 20 4 49 107 0% 2025-02-13T02:48:32+00:00 83s 18 7 79 95 ( 135) 62 ( 350) 0 0 20 4 49 104 0% 2025-02-13T02:33:22+00:00 72s 13 7 79 89 ( 88) 62 ( 357) 0 0 20 4 49 108 0% 2025-02-13T02:18:26+00:00 77s 16 7 79 155 ( 34) 30 ( 340) 0 0 20 4 49 108 0% 2025-02-13T02:03:27+00:00 78s 12 7 79 96 ( 46) 28 ( 340) 0 0 20 4 49 108 0% 2025-02-13T01:48:23+00:00 74s 12 7 79 84 ( 47) 28 ( 350) 0 0 20 4 49 108 0% 2025-02-13T01:33:36+00:00 87s 14 7 79 86 ( 96) 62 ( 347) 0 0 20 4 49 107 0% 2025-02-13T01:18:25+00:00 76s 15 7 79 85 ( 66) 29 ( 350) 0 0 20 4 49 108 0% 2025-02-13T01:03:29+00:00 80s 14 7 79 110 ( 100) 17 ( 250) 0 0 20 4 49 108 0% 2025-02-13T00:48:25+00:00 76s 12 7 79 93 ( 139) 13 ( 88) 0 0 20 4 49 108 0% 2025-02-13T00:33:33+00:00 84s 12 7 79 125 ( 203) 86 ( 357) 0 0 20 4 49 108 0% 2025-02-13T00:18:41+00:00 93s 15 7 79 146 ( 58) 21 ( 750) 0 0 20 4 49 104 0% 2025-02-13T00:03:34+00:00 85s 12 7 79 101 ( 79) 14 ( 50) 0 0 20 4 49 104 0% 2025-02-12T23:48:29+00:00 80s 14 7 79 132 ( 258) 14 ( 50) 0 0 20 4 49 104 0% 2025-02-12T23:33:32+00:00 84s 12 7 79 112 ( 194) 47 ( 355) 0 0 20 4 49 107 0% 2025-02-12T23:18:39+00:00 91s 18 7 79 112 ( 164) 14 ( 100) 0 0 20 4 49 107 0% 2025-02-12T23:03:31+00:00 83s 13 8 78 179 ( 388) 41 ( 3833) 0 0 20 4 49 106 0% 2025-02-12T22:48:28+00:00 80s 15 8 78 152 ( 332) 20 ( 1429) 0 0 33 4 50 91 0% 2025-02-12T22:33:24+00:00 76s 14 8 78 118 ( 207) 81 ( 352) 0 0 20 4 49 104 0% 2025-02-12T22:18:41+00:00 93s 17 8 78 100 ( 86) 15 ( 88) 0 0 20 4 49 105 0% 2025-02-12T22:03:32+00:00 85s 13 8 78 148 ( 200) 16 ( 83) 0 0 20 4 49 104 0% 2025-02-12T21:48:19+00:00 71s 16 8 78 106 ( 269) 48 ( 364) 0 0 20 4 49 106 0% 2025-02-12T21:33:18+00:00 70s 13 8 78 130 ( 154) 13 ( 75) 0 0 20 4 49 170 0% 2025-02-12T21:18:38+00:00 91s 16 8 78 96 ( 138) 14 ( 100) 0 0 20 4 49 106 0% 2025-02-12T21:03:19+00:00 72s 13 8 78 176 ( 169) 135 ( 445) 0 0 20 4 49 103 0% 2025-02-12T20:48:18+00:00 70s 13 8 78 146 ( 350) 215 ( 410) 0 0 20 4 49 105 0% 2025-02-12T20:33:20+00:00 73s 15 8 78 125 ( 142) 30 ( 3200) 0 0 20 4 49 106 0% 2025-02-12T20:18:32+00:00 85s 16 8 78 203 ( 265) 18 ( 1500) 0 0 36 4 49 106 0% 2025-02-12T20:03:24+00:00 76s 13 8 78 207 ( 261) 25 ( 1625) 0 0 20 4 49 107 0% 2025-02-12T19:48:19+00:00 72s 13 8 78 89 ( 165) 19 ( 1300) 0 0 20 4 49 107 0% 2025-02-12T19:33:17+00:00 70s 15 8 78 154 ( 200) 54 ( 388) 0 0 20 4 49 107 0% 2025-02-12T19:18:31+00:00 85s 16 8 78 102 ( 118) 7 ( 35) 0 0 20 4 49 107 0% 2025-02-12T19:03:17+00:00 70s 13 8 78 103 ( 113) 8 ( 35) 0 0 20 4 49 108 0% 2025-02-12T18:48:18+00:00 71s 13 8 78 141 ( 277) 46 ( 306) 0 0 20 4 49 107 0% 2025-02-12T18:33:18+00:00 72s 13 8 78 106 ( 171) 42 ( 320) 0 0 20 4 49 107 0% 2025-02-12T18:18:20+00:00 74s 18 8 78 100 ( 155) 7 ( 38) 0 0 20 4 49 106 0% 2025-02-12T18:03:32+00:00 86s 14 8 78 125 ( 166) 13 ( 29) 0 0 20 4 49 105 0% 2025-02-12T17:48:17+00:00 71s 13 8 78 109 ( 93) 7 ( 38) 0 0 20 4 49 107 0% 2025-02-12T17:33:17+00:00 71s 13 8 78 115 ( 145) 47 ( 330) 0 0 20 4 49 104 0% 2025-02-12T17:18:19+00:00 73s 17 8 78 119 ( 142) 7 ( 41) 0 0 20 4 49 104 0% 2025-02-12T17:03:17+00:00 71s 13 8 78 166 ( 263) 18 ( 200) 0 0 20 4 49 107 0% 2025-02-12T16:48:30+00:00 84s 14 8 78 131 ( 221) 9 ( 28) 0 0 20 4 49 106 0% 2025-02-12T16:33:19+00:00 74s 13 8 78 165 ( 86) 102 ( 342) 0 0 27 4 49 107 0% 2025-02-12T16:18:16+00:00 70s 13 7 79 173 ( 388) 12 ( 44) 0 0 20 4 49 107 0% 2025-02-12T16:04:19+00:00 134s 37 7 79 148 ( 92) 10 ( 40) 0 0 20 4 49 106 0% 2025-02-12T15:48:28+00:00 82s 13 7 79 126 ( 100) 18 ( 37) 0 0 20 4 49 104 0% 2025-02-12T15:33:15+00:00 70s 12 7 79 171 ( 190) 48 ( 320) 0 0 20 4 49 105 0% 2025-02-12T15:18:19+00:00 74s 15 7 79 178 ( 70) 13 ( 75) 0 0 20 4 49 105 0% 2025-02-12T15:03:18+00:00 72s 13 7 79 198 ( 96) 14 ( 100) 0 0 20 4 49 100 0% 2025-02-12T14:48:18+00:00 73s 14 7 79 191 ( 270) 10 ( 50) 0 0 27 4 49 104 0% 2025-02-12T14:33:27+00:00 82s 15 7 79 178 ( 322) 122 ( 388) 0 0 27 4 49 104 0% 2025-02-12T14:18:36+00:00 92s 20 9 77 130 ( 100) 14 ( 500) 0 0 26 4 49 107 0% 2025-02-12T14:05:23+00:00 198s 23 16 70 134 ( 142) 8 ( 35) 0 0 20 4 49 105 0% 2025-02-12T13:50:20+00:00 195s 21 16 70 158 ( 367) 24 ( 3500) 0 0 20 4 49 103 0% 2025-02-12T13:35:22+00:00 198s 23 16 70 192 ( 786) 54 ( 336) 0 0 20 4 49 107 0% 2025-02-12T13:20:29+00:00 204s 22 16 70 262 ( 52) 21 ( 41) 0 0 27 4 49 107 0% 2025-02-12T13:05:18+00:00 194s 24 16 70 279 ( 37) 22 ( 7500) 0 0 39 4 49 107 0% 2025-02-12T12:50:21+00:00 196s 21 16 70 208 ( 281) 17 ( 275) 0 0 20 4 49 107 0% 2025-02-12T12:35:21+00:00 197s 22 16 70 258 ( 50) 102 ( 350) 0 0 26 4 71 90 0% 2025-02-12T12:20:20+00:00 196s 22 16 70 305 ( 1870) 20 ( 2000) 0 0 62 4 49 107 0% 2025-02-12T12:05:43+00:00 219s 25 16 70 282 ( 1821) 28 ( 833) 0 0 30 4 49 103 0% 2025-02-12T11:50:19+00:00 195s 21 16 70 254 ( 1818) 30 ( 5000) 0 13 20 4 49 107 0% 2025-02-12T11:35:21+00:00 197s 22 16 70 208 ( 1074) 85 ( 354) 0 11 19 4 104 92 0% 2025-02-12T11:20:22+00:00 198s 23 16 70 258 ( 879) 43 ( 350) 0 0 33 4 52 640 0% 2025-02-12T11:05:43+00:00 220s 27 16 70 303 ( 2065) 28 ( 436) 0 0 23 4 49 107 0% 2025-02-12T10:50:54+00:00 231s 26 23 63 235 ( 1417) 37 ( 490) 0 0 38 4 49 77 0% 2025-02-12T10:35:54+00:00 231s 26 23 63 151 ( 808) 300 ( 610) 0 0 20 4 49 85 0% 2025-02-12T10:20:56+00:00 232s 28 23 63 211 ( 671) 145 ( 951) 0 0 22 4 50 73 0% 2025-02-12T10:05:55+00:00 231s 28 23 63 154 ( 388) 95 ( 513) 0 0 23 4 49 86 0% 2025-02-12T09:50:55+00:00 231s 26 23 63 310 ( 616) 271 ( 993) 0 0 40 4 50 223 0% 2025-02-12T09:35:54+00:00 231s 27 23 63 394 ( 788) 124 ( 468) 0 0 19 4 50 159 0% Major LoadDatasets Times Distribution (since last Daily Report): n = 6, median ~= 90000 ms 0 ms: 0 1 ms: 0 2 ms: 0 <= 5 ms: 0 <= 10 ms: 0 <= 20 ms: 0 <= 50 ms: 0 <= 100 ms: 0 <= 200 ms: 0 <= 500 ms: 0 <= 1 s: 0 <= 2 s: 0 <= 5 s: 0 <= 10 s: 0 <= 20 s: 0 <= 1 min: 0 <= 2 min: 6 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Major LoadDatasets Times Distribution (since startup): n = 582, median ~= 91765 ms 0 ms: 0 1 ms: 0 2 ms: 0 <= 5 ms: 0 <= 10 ms: 0 <= 20 ms: 0 <= 50 ms: 0 <= 100 ms: 0 <= 200 ms: 0 <= 500 ms: 0 <= 1 s: 0 <= 2 s: 0 <= 5 s: 0 <= 10 s: 0 <= 20 s: 0 <= 1 min: 3 <= 2 min: 544 <= 5 min: 35 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Minor LoadDatasets Times Distribution (since last Daily Report): n = 0 Minor LoadDatasets Times Distribution (since startup): n = 25, median ~= 3227 ms 0 ms: 0 1 ms: 0 2 ms: 0 <= 5 ms: 0 <= 10 ms: 0 <= 20 ms: 0 <= 50 ms: 0 <= 100 ms: 0 <= 200 ms: 4 <= 500 ms: 3 <= 1 s: 1 <= 2 s: 0 <= 5 s: 11 <= 10 s: 0 <= 20 s: 5 <= 1 min: 0 <= 2 min: 1 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Response Failed Time Distribution (since last major LoadDatasets): n = 18, median ~= 400 ms 0 ms: 0 1 ms: 0 2 ms: 0 <= 5 ms: 0 <= 10 ms: 0 <= 20 ms: 0 <= 50 ms: 2 <= 100 ms: 0 <= 200 ms: 1 <= 500 ms: 9 <= 1 s: 0 <= 2 s: 5 <= 5 s: 1 <= 10 s: 0 <= 20 s: 0 <= 1 min: 0 <= 2 min: 0 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Response Failed Time Distribution (since last Daily Report): n = 1142, median ~= 1077 ms 0 ms: 6 1 ms: 3 2 ms: 0 <= 5 ms: 2 <= 10 ms: 2 <= 20 ms: 10 <= 50 ms: 60 <= 100 ms: 24 <= 200 ms: 10 <= 500 ms: 251 <= 1 s: 182 <= 2 s: 274 <= 5 s: 227 <= 10 s: 53 <= 20 s: 33 <= 1 min: 5 <= 2 min: 0 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Response Failed Time Distribution (since startup): n = 32485, median ~= 453 ms 0 ms: 503 1 ms: 330 2 ms: 17 <= 5 ms: 26 <= 10 ms: 50 <= 20 ms: 239 <= 50 ms: 3271 <= 100 ms: 502 <= 200 ms: 938 <= 500 ms: 12306 <= 1 s: 3670 <= 2 s: 5463 <= 5 s: 3321 <= 10 s: 846 <= 20 s: 615 <= 1 min: 185 <= 2 min: 83 <= 5 min: 78 <= 10 min: 38 <= 20 min: 2 <= 1 hr: 2 > 1 hr: 0 Response Succeeded Time Distribution (since last major LoadDatasets): n = 25, median ~= 22 ms 0 ms: 2 1 ms: 1 2 ms: 1 <= 5 ms: 4 <= 10 ms: 1 <= 20 ms: 3 <= 50 ms: 9 <= 100 ms: 1 <= 200 ms: 2 <= 500 ms: 1 <= 1 s: 0 <= 2 s: 0 <= 5 s: 0 <= 10 s: 0 <= 20 s: 0 <= 1 min: 0 <= 2 min: 0 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Response Succeeded Time Distribution (since last Daily Report): n = 1138, median ~= 247 ms 0 ms: 38 1 ms: 63 2 ms: 31 <= 5 ms: 43 <= 10 ms: 23 <= 20 ms: 58 <= 50 ms: 149 <= 100 ms: 53 <= 200 ms: 90 <= 500 ms: 133 <= 1 s: 110 <= 2 s: 132 <= 5 s: 89 <= 10 s: 53 <= 20 s: 65 <= 1 min: 8 <= 2 min: 0 <= 5 min: 0 <= 10 min: 0 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 Response Succeeded Time Distribution (since startup): n = 75337, median ~= 167 ms 0 ms: 3641 1 ms: 5810 2 ms: 1628 <= 5 ms: 1970 <= 10 ms: 935 <= 20 ms: 2231 <= 50 ms: 11481 <= 100 ms: 5713 <= 200 ms: 6404 <= 500 ms: 9924 <= 1 s: 5740 <= 2 s: 8966 <= 5 s: 5446 <= 10 s: 3398 <= 20 s: 1312 <= 1 min: 497 <= 2 min: 218 <= 5 min: 21 <= 10 min: 2 <= 20 min: 0 <= 1 hr: 0 > 1 hr: 0 TaskThread Failed Time Distribution (since last Daily Report): n = 0 TaskThread Failed Time Distribution (since startup): n = 0 TaskThread Succeeded Time Distribution (since last Daily Report): n = 0 TaskThread Succeeded Time Distribution (since startup): n = 0 SgtMap topography nFromCache=44 nNotFromCache=72 GSHHS: nCached=100 of 100, nCoarse=971, nSuccesses=754, nTossed=295 NationalBoundaries: nCached=100 of 100, nCoarse=132, nSuccesses=213, nTossed=9 StateBoundaries: nCached=89 of 100, nCoarse=111, nSuccesses=202, nTossed=0 Rivers: nCached=100 of 100, nCoarse=127, nSuccesses=213, nTossed=6 bufferedImage isAccelerated=[unknown] canonical map sizes: 242 + 723 + 262 + 1555 + 994 + 1440 = 5216 canonicalStringHolder map sizes: 43405 + 761 + 971 + 521 + 213 + 370 = 46241 Number of threads: Tomcat-waiting=19, inotify=4, other=48 (format: #threadNumber Thread[threadName,threadPriority,threadGroup] threadStatus) #1 Thread[AsyncFileHandlerWriter-1,5,main] WAITING daemon sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492) java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) #2 Thread[Catalina-utility-1,1,main] WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) java.lang.Thread.run(Thread.java:750) #3 Thread[Catalina-utility-2,1,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) java.lang.Thread.run(Thread.java:750) #4 Thread[cluster1-connection-reaper-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #5 Thread[cluster1-nio-worker-0,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #6 Thread[cluster1-nio-worker-1,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #7 Thread[cluster1-nio-worker-2,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #8 Thread[cluster1-nio-worker-3,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #9 Thread[cluster1-nio-worker-4,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #10 Thread[cluster1-nio-worker-5,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #11 Thread[cluster1-nio-worker-6,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #12 Thread[cluster1-nio-worker-7,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #13 Thread[cluster1-reconnection-0,5,main] WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #14 Thread[cluster1-reconnection-1,5,main] WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #15 Thread[cluster1-scheduled-task-worker-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #16 Thread[cluster1-timeouter-0,5,main] TIMED_WAITING java.lang.Thread.sleep(Native Method) io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569) io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #17 Thread[cluster2-connection-reaper-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #18 Thread[cluster2-nio-worker-0,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #19 Thread[cluster2-nio-worker-1,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #20 Thread[cluster2-nio-worker-2,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #21 Thread[cluster2-nio-worker-3,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #22 Thread[cluster2-nio-worker-4,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #23 Thread[cluster2-nio-worker-5,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #24 Thread[cluster2-nio-worker-6,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait0(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:176) io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:281) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:351) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #25 Thread[cluster2-nio-worker-7,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #26 Thread[cluster2-reconnection-0,5,main] WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #27 Thread[cluster2-reconnection-1,5,main] WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #28 Thread[cluster2-scheduled-task-worker-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #29 Thread[cluster2-timeouter-0,5,main] TIMED_WAITING java.lang.Thread.sleep(Native Method) io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569) io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #30 Thread[cluster306-connection-reaper-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #31 Thread[cluster306-nio-worker-0,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #32 Thread[cluster306-nio-worker-1,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #33 Thread[cluster306-nio-worker-2,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #34 Thread[cluster306-nio-worker-3,5,main] RUNNABLE io.netty.channel.epoll.Native.epollWait(Native Method) io.netty.channel.epoll.Native.epollWait(Native.java:192) io.netty.channel.epoll.Native.epollWait(Native.java:185) io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:290) io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:347) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #35 Thread[cluster306-scheduled-task-worker-0,5,main] TIMED_WAITING sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #36 Thread[cluster306-timeouter-0,5,main] TIMED_WAITING java.lang.Thread.sleep(Native Method) io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569) io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) #37 Thread[FileHandlerLogFilesCleaner-1,5,main] WAITING daemon sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) #38 Thread[Finalizer,8,system] WAITING daemon java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188) #39 Thread[GC Daemon,2,system] TIMED_WAITING daemon java.lang.Object.wait(Native Method) sun.misc.GC$Daemon.run(GC.java:117) #40 Thread[http-nio-8080-Acceptor,5,main] RUNNABLE daemon sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545) org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71) org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129) java.lang.Thread.run(Thread.java:750) #41 Thread[http-nio-8080-Poller,5,main] RUNNABLE daemon sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:809) java.lang.Thread.run(Thread.java:750) #42 Thread[https-jsse-nio-8443-Acceptor,5,main] RUNNABLE daemon sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249) org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:545) org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71) org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129) java.lang.Thread.run(Thread.java:750) #43 Thread[https-jsse-nio-8443-Poller,5,main] RUNNABLE daemon sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:809) java.lang.Thread.run(Thread.java:750) #44 Thread[Java2D Disposer,10,system] WAITING daemon java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) sun.java2d.Disposer.run(Disposer.java:148) java.lang.Thread.run(Thread.java:750) #45 Thread[main,5,main] TIMED_WAITING java.lang.Thread.sleep(Native Method) org.apache.catalina.core.StandardServer.await(StandardServer.java:524) org.apache.catalina.startup.Catalina.await(Catalina.java:825) org.apache.catalina.startup.Catalina.start(Catalina.java:773) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:345) org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473) #46 Thread[Reference Handler,10,system] WAITING daemon java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) java.lang.ref.Reference.tryHandlePending(Reference.java:191) java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) #47 Thread[RunLoadDatasets,5,main] TIMED_WAITING daemon java.lang.Thread.sleep(Native Method) gov.noaa.pfel.erddap.RunLoadDatasets.run(RunLoadDatasets.java:288) #48 Thread[Signal Dispatcher,9,system] RUNNABLE daemon