[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-12-ea+12) - Build # 23313 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23313/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseSerialGC

15 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest

Error Message:
44 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest: 1) Thread[id=685, name=qtp1309702070-685, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 2) Thread[id=740, name=Scheduler-124753301, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 3) Thread[id=636, name=qtp1179819950-636, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 4) Thread[id=653, name=qtp567423036-653-acceptor-***@790c5eb0-***@3059f8ce{SSL,[ssl, http/1.1]}{127.0.0.1:41425}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 5) Thread[id=650, name=qtp567423036-650, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 6) Thread[id=683, name=qtp1309702070-683, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 7) Thread[id=680, name=qtp1309702070-680, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 8) Thread[id=684, name=qtp1309702070-684, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 9) Thread[id=638, name=qtp1179819950-638, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 10) Thread[id=651, name=qtp567423036-651, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 11) Thread[id=676, name=MetricsHistoryHandler-204-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 12) Thread[id=678, name=qtp1309702070-678, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 13) Thread[id=655, name=qtp567423036-655, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 14) Thread[id=682, name=qtp1309702070-682-acceptor-***@1ef8f7b5-***@44afc45e{SSL,[ssl, http/1.1]}{127.0.0.1:42365}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 15) Thread[id=712, name=coreContainerWorkExecutor-198-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 16) Thread[id=644, name=Scheduler-1641907467, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 17) Thread[id=752, name=updateExecutor-261-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 18) Thread[id=738, name=Scheduler-1626290695, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 19) Thread[id=711, name=Thread-112, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/java.lang.Object.wait(Native Method) at ***@12-ea/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1899) 20) Thread[id=733, name=coreContainerWorkExecutor-207-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 21) Thread[id=690, name=Thread-111, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/java.lang.Object.wait(Native Method) at ***@12-ea/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1899) 22) Thread[id=679, name=qtp1309702070-679, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 23) Thread[id=652, name=qtp567423036-652, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 24) Thread[id=637, name=qtp1179819950-637, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 25) Thread[id=640, name=qtp1179819950-640-acceptor-***@69c626f5-***@7c161655{SSL,[ssl, http/1.1]}{127.0.0.1:44953}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525) at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 26) Thread[id=656, name=qtp567423036-656, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 27) Thread[id=730, name=SolrRrdBackendFactory-219-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 28) Thread[id=686, name=Scheduler-1373336164, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 29) Thread[id=649, name=qtp567423036-649, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 30) Thread[id=709, name=SolrRrdBackendFactory-212-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 31) Thread[id=643, name=qtp1179819950-643, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 32) Thread[id=731, name=MetricsHistoryHandler-220-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 33) Thread[id=641, name=qtp1179819950-641, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 34) Thread[id=753, name=recoveryExecutor-262-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 35) Thread[id=654, name=qtp567423036-654, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 36) Thread[id=691, name=coreContainerWorkExecutor-196-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 37) Thread[id=639, name=qtp1179819950-639, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 38) Thread[id=681, name=qtp1309702070-681, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method) at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 39) Thread[id=710, name=MetricsHistoryHandler-213-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 40) Thread[id=732, name=Thread-113, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/java.lang.Object.wait(Native Method) at ***@12-ea/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1899) 41) Thread[id=675, name=SolrRrdBackendFactory-203-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 42) Thread[id=642, name=qtp1179819950-642, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 43) Thread[id=657, name=Scheduler-1010320365, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835) 44) Thread[id=739, name=Scheduler-1047015637, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest] at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method) at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at ***@12-ea/java.lang.Thread.run(Thread.java:835)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 44 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest:
1) Thread[id=685, name=qtp1309702070-685, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
2) Thread[id=740, name=Scheduler-124753301, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
3) Thread[id=636, name=qtp1179819950-636, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
4) Thread[id=653, name=qtp567423036-653-acceptor-***@790c5eb0-***@3059f8ce{SSL,[ssl, http/1.1]}{127.0.0.1:41425}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277)
at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
5) Thread[id=650, name=qtp567423036-650, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
6) Thread[id=683, name=qtp1309702070-683, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
7) Thread[id=680, name=qtp1309702070-680, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
8) Thread[id=684, name=qtp1309702070-684, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
9) Thread[id=638, name=qtp1179819950-638, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
10) Thread[id=651, name=qtp567423036-651, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
11) Thread[id=676, name=MetricsHistoryHandler-204-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
12) Thread[id=678, name=qtp1309702070-678, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
13) Thread[id=655, name=qtp567423036-655, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
14) Thread[id=682, name=qtp1309702070-682-acceptor-***@1ef8f7b5-***@44afc45e{SSL,[ssl, http/1.1]}{127.0.0.1:42365}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277)
at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
15) Thread[id=712, name=coreContainerWorkExecutor-198-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
16) Thread[id=644, name=Scheduler-1641907467, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
17) Thread[id=752, name=updateExecutor-261-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
18) Thread[id=738, name=Scheduler-1626290695, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
19) Thread[id=711, name=Thread-112, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/java.lang.Object.wait(Native Method)
at ***@12-ea/java.lang.Object.wait(Object.java:328)
at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1899)
20) Thread[id=733, name=coreContainerWorkExecutor-207-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
21) Thread[id=690, name=Thread-111, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/java.lang.Object.wait(Native Method)
at ***@12-ea/java.lang.Object.wait(Object.java:328)
at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1899)
22) Thread[id=679, name=qtp1309702070-679, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
23) Thread[id=652, name=qtp567423036-652, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
24) Thread[id=637, name=qtp1179819950-637, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
25) Thread[id=640, name=qtp1179819950-640-acceptor-***@69c626f5-***@7c161655{SSL,[ssl, http/1.1]}{127.0.0.1:44953}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:525)
at ***@12-ea/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:277)
at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
26) Thread[id=656, name=qtp567423036-656, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
27) Thread[id=730, name=SolrRrdBackendFactory-219-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
28) Thread[id=686, name=Scheduler-1373336164, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
29) Thread[id=649, name=qtp567423036-649, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
30) Thread[id=709, name=SolrRrdBackendFactory-212-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
31) Thread[id=643, name=qtp1179819950-643, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
32) Thread[id=731, name=MetricsHistoryHandler-220-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at ***@12-ea/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
33) Thread[id=641, name=qtp1179819950-641, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
34) Thread[id=753, name=recoveryExecutor-262-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
35) Thread[id=654, name=qtp567423036-654, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
36) Thread[id=691, name=coreContainerWorkExecutor-196-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at ***@12-ea/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at ***@12-ea/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at ***@12-ea/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
37) Thread[id=639, name=qtp1179819950-639, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
38) Thread[id=681, name=qtp1309702070-681, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/sun.nio.ch.EPoll.wait(Native Method)
at ***@12-ea/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at ***@12-ea/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at ***@12-ea/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$166/0x0000000100387440.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at ***@12-ea/java.lang.Thread.run(Thread.java:835)
39) Thread[id=710, name=MetricsHistoryHandler-213-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderWithPullReplicasTest]
at ***@12-ea/jdk.internal.misc.Unsafe.park(Native Method)
at ***@12-ea/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at ***@12-ea/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123

[...truncated too long message...]

] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46015_solr
[junit4] 2> 104724 INFO (zkCallback-1077-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 104724 INFO (jetty-closer-1042-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 104724 INFO (jetty-closer-1042-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [***@5e4a0931: rootName = solr_46015, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / ***@7301ea96
[junit4] 2> 104724 INFO (jetty-closer-1042-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 104725 INFO (jetty-closer-1042-thread-2) [ ] o.a.s.c.Overseer Overseer (id=72161558582788102-127.0.0.1:34949_solr-n_0000000000) closing
[junit4] 2> 104725 INFO (closeThreadPool-1090-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72161558582788102-127.0.0.1:34949_solr-n_0000000000) closing
[junit4] 2> 104727 INFO (jetty-closer-1042-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ***@500aad1b{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 104728 INFO (jetty-closer-1042-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped ***@7814b2fd{/solr,null,UNAVAILABLE}
[junit4] 2> 104728 INFO (jetty-closer-1042-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 104729 INFO (jetty-closer-1042-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ***@23c6b1dc{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 104729 INFO (jetty-closer-1042-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped ***@139c3d25{/solr,null,UNAVAILABLE}
[junit4] 2> 104729 INFO (jetty-closer-1042-thread-2) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 104730 INFO (TEST-CloudMLTQParserTest.testMLTQParser-seed#[A95595E15BB29190]) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 104730 ERROR (TEST-CloudMLTQParserTest.testMLTQParser-seed#[A95595E15BB29190]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 104730 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 3 /solr/collections/mlt-collection/terms/shard2
[junit4] 2> 3 /solr/collections/mlt-collection/terms/shard1
[junit4] 2> 2 /solr/security.json
[junit4] 2> 2 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 13 /solr/collections/mlt-collection/state.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 104731 INFO (TEST-CloudMLTQParserTest.testMLTQParser-seed#[A95595E15BB29190]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43165
[junit4] 2> 104731 INFO (TEST-CloudMLTQParserTest.testMLTQParser-seed#[A95595E15BB29190]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43165
[junit4] 2> 104731 INFO (TEST-CloudMLTQParserTest.testMLTQParser-seed#[A95595E15BB29190]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testMLTQParser
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudMLTQParserTest -Dtests.method=testMLTQParser -Dtests.seed=A95595E15BB29190 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=teo-UG -Dtests.timezone=Asia/Damascus -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 3.39s J2 | CloudMLTQParserTest.testMLTQParser <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:34949/solr/mlt-collection: Error from server at https://127.0.0.1:34949/solr/mlt-collection_shard1_replica_n1: Error completing MLT request. Could not fetch document with id [17]
[junit4] > at __randomizedtesting.SeedInfo.seed([A95595E15BB29190:9AE5DC1A29318F2A]:0)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
[junit4] > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:983)
[junit4] > at org.apache.solr.search.mlt.CloudMLTQParserTest.testMLTQParser(CloudMLTQParserTest.java:102)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:835)
[junit4] 2> 104732 INFO (SUITE-CloudMLTQParserTest-seed#[A95595E15BB29190]-worker) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 104732 INFO (SUITE-CloudMLTQParserTest-seed#[A95595E15BB29190]-worker) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:43165
[junit4] 2> 104732 INFO (SUITE-CloudMLTQParserTest-seed#[A95595E15BB29190]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 43165
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.search.mlt.CloudMLTQParserTest_A95595E15BB29190-002
[junit4] 2> Dec 09, 2018 2:20:13 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 8 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {_version_=PostingsFormat(name=Direct), _root_=PostingsFormat(name=Direct), lowerfilt_u=PostingsFormat(name=Direct), id=Lucene50(blocksize=128), lowerfilt1_u=Lucene50(blocksize=128)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1441, maxMBSortInHeap=7.584417788666809, sim=Asserting(***@72307a97), locale=teo-UG, timezone=Asia/Damascus
[junit4] 2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=117217992,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderWithPullReplicasTest, CloudMLTQParserTest, CloudMLTQParserTest]
[junit4] Completed [10/10 (4!)] on J2 in 31.80s, 9 tests, 4 errors <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1572: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1099: There were test failures: 10 suites, 50 tests, 4 suite-level errors, 7 errors, 27 ignored [seed: A95595E15BB29190]

Total time: 1 minute 47 seconds

[repro] Setting last failure code to 256

[repro] Failures:
[repro] 2/5 failed: org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest
[repro] 2/5 failed: org.apache.solr.search.mlt.CloudMLTQParserTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=df84a3c9815a85aa140e5013b44488f45de9a203, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #23312
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2