Elasticsearch Outage
All three nodes have died.
Tail of 10...21
[2020-02-18T12:17:56,706][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [La8d-al] fatal error in thread [elasticsearch[La8d-al][index][T#7]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236) ~[?:1.8.0_212]
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) ~[?:1.8.0_212]
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) ~[?:1.8.0_212]
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) ~[?:1.8.0_212]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2039) ~[jackson-core-2.8.11.jar:2.8.11]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeNumber(UTF8JsonGenerator.java:882) ~[jackson-core-2.8.11.jar:2.8.11]
at org.elasticsearch.common.xcontent.json.JsonXContentGenerator.writeNumber(JsonXContentGenerator.java:256) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:511) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:507) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.lambda$static$9(XContentBuilder.java:97) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder$$Lambda$22/564160838.write(Unknown Source) ~[?:?]
at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:815) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.value(XContentBuilder.java:915) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.unknownValue(XContentBuilder.java:824) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:888) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.common.xcontent.XContentBuilder.map(XContentBuilder.java:870) ~[elasticsearch-x-content-6.6.2.jar:6.6.2]
at org.elasticsearch.action.index.IndexRequest.source(IndexRequest.java:343) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.UpdateHelper.prepareUpdateScriptRequest(UpdateHelper.java:277) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:96) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:76) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:177) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:170) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:67) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:252) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:248) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:308) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:687) ~[elasticsearch-6.6.2.jar:6.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) ~[elasticsearch-6.6.2.jar:6.6.2]
[2020-02-18T12:18:48,199][ERROR][o.e.ExceptionsHelper ] [La8d-al] fatal error
at org.elasticsearch.ExceptionsHelper.lambda$maybeDieOnAnotherThread$2(ExceptionsHelper.java:264)
at java.util.Optional.ifPresent(Optional.java:159)
at org.elasticsearch.ExceptionsHelper.maybeDieOnAnotherThread(ExceptionsHelper.java:254)
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:61)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:856)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)
[2020-02-18T12:18:48,200][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [La8d-al] fatal error in thread [Thread-3725998], exiting
java.lang.OutOfMemoryError: Java heap space
Tail of 10...22
[2020-02-18T12:24:49,085][WARN ][o.e.a.b.TransportShardBulkAction] [j_L53Ds] [[minds-metrics-02-2020][0]] failed to perform indices:data/write/bulk[s] on replica [minds-metrics-02-2020][0], node[La8d-alqRgGIYyncoXAFjA], [R], s[STARTED], a[id=uX8ae762T7K7LXg3C34tXw]
org.elasticsearch.transport.NodeDisconnectedException: [La8d-al][10.0.5.21:9300][indices:data/write/bulk[s][r]] disconnected
[2020-02-18T12:24:49,086][ERROR][o.e.i.e.Engine ] [j_L53Ds] [minds-metrics-02-2020][0] already closed by tragic event on the index writer
java.lang.OutOfMemoryError: Java heap space
[2020-02-18T12:24:49,008][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [j_L53Ds] fatal error in thread [elasticsearch[j_L53Ds][write][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space
[2020-02-18T12:24:49,087][WARN ][o.e.a.b.TransportShardBulkAction] [j_L53Ds] [[minds-metrics-02-2020][0]] failed to perform indices:data/write/bulk[s] on replica [minds-metrics-02-2020][0], node[La8d-alqRgGIYyncoXAFjA], [R], s[STARTED], a[id=uX8ae762T7K7LXg3C34tXw]
org.elasticsearch.transport.NodeDisconnectedException: [La8d-al][10.0.5.21:9300][indices:data/write/bulk[s][r]] disconnected
[2020-02-18T12:24:49,088][WARN ][o.e.a.b.TransportShardBulkAction] [j_L53Ds] [[minds-metrics-02-2020][0]] failed to perform indices:data/write/bulk[s] on replica [minds-metrics-02-2020][0], node[La8d-alqRgGIYyncoXAFjA], [R], s[STARTED], a[id=uX8ae762T7K7LXg3C34tXw]
org.elasticsearch.transport.NodeDisconnectedException: [La8d-al][10.0.5.21:9300][indices:data/write/bulk[s][r]] disconnected
added scoped labels
changed the description
- Owner
added scoped label and automatically removed label
Please register or sign in to reply