Skip to main content
 首页 » 编程设计

python之ElasticSearch大量索引导致OOM

2024年11月01日9zdz8207

我一直在为这个问题苦苦挣扎,现在我决定就该主题发布一个问题以寻求一些建议。

我正在测试随着时间的推移索引大量数据的环境。基本上每天我都会索引来自各个网站的日志和相关文档。

我希望每个网站有一个索引,以便有更好的逻辑划分并能够过滤查询并获得更快的响应。

我预计每个索引每天的数据流量约为 1Gb。

我正在使用 2x80Gb SSD 的 AWS 实例上测试环境,CPU 供应商:Intel,CPU 型号:Xeon (2500 MHz),CPU 总逻辑内核数:8

我有 1 个节点、x 个索引和每个索引 1 个分片(用于测试环境),并推送从维基百科文章中获取的文档,将它们随机分布在 200 个或更多索引上。

我正在从一个队列中索引文档,单个客户端的速度约为 200/s。这只是为了测试。当我能够解决 OOM 问题时,我将增加客户端数量和节点数量。

我尝试了 500k 文档和 200 个索引,这很好,如果我尝试使用 500k 和 300 个索引,它会引发讨厌的 OOM 错误异常。 1m 个文档和 100 个索引也会引发 OOM 错误。
我现在正在运行 1m 文档和 200 个索引的测试。

我尝试更改分片号,我也尝试使用这些参数
index.merge.policy.max_merged_segment:2g
index.merge.policy.segments_per_tier:[我尝试了 5(也减少了最大数量) - 25]
index.merge.policy.max_merge_at_once:8

我的设置是(使用/_nodes/settings,os,process,jvm?pretty 命令检索)

{  
  "cluster_name" : "elasticsearch",  
  "nodes" : {  
    "QBk8YzISQPu-VVMnKvhEmQ" : {  
      "name" : "",  
      "transport_address" : "",  
      "host" : "",  
      "ip" : "",  
      "version" : "1.1.0",  
      "build" : "2181e11",  
      "http_address" : "",  
      "settings" : {  
        "path" : {  
          "data" : "/mnt/db/se_data/elasticsearch/",  
          "logs" : "/mnt/db/searchengines/elasticsearch-1.1.0/logs",  
          "home" : "/mnt/db/searchengines/elasticsearch-1.1.0"  
        },  
        "cluster" : {  
          "name" : "elasticsearch"  
        },  
        "index" : {  
          "number_of_shards" : "1"  
        },  
        "foreground" : "yes",  
        "name" : "",  
        "max-open-files" : "true"  
      },  
      "os" : {  
        "refresh_interval" : 1000,  
        "available_processors" : 8,  
        "cpu" : {  
          "vendor" : "Intel",  
          "model" : "Xeon",  
          "mhz" : 2500,  
          "total_cores" : 8,  
          "total_sockets" : 8,  
          "cores_per_socket" : 32,  
          "cache_size_in_bytes" : 25600  
        },  
        "mem" : {  
          "total_in_bytes" : 31502180352  
        },  
        "swap" : {  
          "total_in_bytes" : 3071995904  
        }  
      },  
      "process" : {  
        "refresh_interval" : 1000,  
        "id" : 20155,  
        "max_file_descriptors" : 64000,  
        "mlockall" : false  
      },  
      "jvm" : {  
        "pid" : 20155,  
        "version" : "1.7.0_51",  
        "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",  
        "vm_version" : "24.51-b03",  
        "vm_vendor" : "Oracle Corporation",  
        "start_time" : 1398962098305,  
        "mem" : {  
          "heap_init_in_bytes" : 268435456,  
          "heap_max_in_bytes" : 10667687936,  
          "non_heap_init_in_bytes" : 24313856,  
          "non_heap_max_in_bytes" : 136314880,  
          "direct_max_in_bytes" : 10667687936  
        },  
        "gc_collectors" : [ "ParNew", "ConcurrentMarkSweep" ],  
        "memory_pools" : [ "Code Cache", "Par Eden Space", "Par Survivor Space", "CMS Old Gen", "CMS Perm Gen" ]  
      }  
    }  
  }  
}  

以下是 JVM 设置: -Xms256m -Xmx10g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Delasticsearch -Des.foreground=yes
文件打开的最大数量设置为 65k(这是我之前遇到的问题.. 允许打开的文件数量仅为 4k)

使用 bigdesk 检查内存消耗,我看到提交/使用的堆内存在达到崩溃点时不断增长。我认为这会导致 OOM 错误,即使它在消耗达到 ~4-6Gb 并达到 10Gb 的最大分配 java 堆时崩溃。

会不会是 GC 问题?我应该按照本文中的设置进行调整吗?

如果我在每个索引上插入一个 flush() 命令,比如说,10k 个插入的文档,它是否有助于减少内存使用?

索引数量是否过多?我应该改变方法吗?将来自不同网站的更多日志放在一个索引中似乎是最合理的解决方案,不是吗?

PS:我正在使用和一个python脚本来测试环境,带有elasticsearch for python模块(我应该使用pyelasticsearch来代替吗?它应该只是一个功能更丰富的模块,对吧?)。

如果您需要更多信息,请告诉我。
感谢您的时间!

编辑:
来自 ES 日志文件的堆栈跟踪如下 - 尝试在 300 索引上加载 500k 文档。
[2014-05-01 00:16:49,526][INFO ][node                     ] [Captain Wings] stopping ... 
[2014-05-01 00:16:49,767][WARN ][index.shard.service      ] [Captain Wings] [index_220][0] Failed to perform scheduled engine refresh 
org.elasticsearch.index.engine.RefreshFailedEngineException: [index_220][0] Refresh failed 
    at org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:725) 
    at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:469) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:920) 
    [...] 
    at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.close(CompressingStoredFieldsWriter.java:138) 
    [...] 
    ... 5 more 
    Suppressed: java.io.FileNotFoundException: _75.fdx 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter.close(CompressingStoredFieldsIndexWriter.java:205) 
        ... 24 more 
[2014-05-01 00:16:49,804][WARN ][index.merge.scheduler    ] [Captain Wings] [index_65][0] failed to merge 
java.io.FileNotFoundException: _7n_es090_0.tim 
    at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
    at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81) 
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140) 
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371) 
    at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389) 
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) 
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119) 
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716) 
    at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122) 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89) 
    at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930) 
    at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47) 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:744) 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip 
        ... 26 more 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587) 
        ... 23 more 
        Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos 
            ... 28 more 
[2014-05-01 00:16:49,807][WARN ][index.engine.internal    ] [Captain Wings] [index_65][0] failed engine 
org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7n_es090_0.tim 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:92) 
    at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930) 
    at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47) 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:744) 
Caused by: java.io.FileNotFoundException: _7n_es090_0.tim 
    at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
    at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81) 
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140) 
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371) 
    at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389) 
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) 
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119) 
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716) 
    at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122) 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89) 
    ... 9 more 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip 
        ... 26 more 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587) 
        ... 23 more 
        Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos 
            ... 28 more 
[2014-05-01 00:16:49,807][WARN ][index.shard.service      ] [Captain Wings] [index_65][0] Failed to perform scheduled engine optimize/merge 
org.elasticsearch.index.engine.OptimizeFailedEngineException: [index_65][0] Optimize failed 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:936) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:744) 
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7n_es090_0.tim 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:93) 
    at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930) 
    at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47) 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926) 
    ... 4 more 
Caused by: java.io.FileNotFoundException: _7n_es090_0.tim 
    at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
    at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81) 
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140) 
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371) 
    at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389) 
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) 
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119) 
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716) 
    at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122) 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89) 
    ... 9 more 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.tip 
        ... 26 more 
    Suppressed: java.io.FileNotFoundException: _7n_es090_0.doc 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587) 
        ... 23 more 
        Suppressed: java.io.FileNotFoundException: _7n_es090_0.pos 
            ... 28 more 
[2014-05-01 00:16:49,913][WARN ][cluster.action.shard     ] [Captain Wings] [index_65][0] sending failed shard for [index_65][0], node[Fj243PXdSNGtSm_jLNl9hQ], [P], s[STARTED], indexUUID [v6ZQJKpTRbiZZ5e6Q6ENyw], reason [engine failure, message [MergeException[java.io.FileNotFoundException: _7n_es090_0.tim]; nested: FileNotFoundException[_7n_es090_0.tim]; ]] 
[2014-05-01 00:16:49,913][WARN ][cluster.action.shard     ] [Captain Wings] [index_65][0] received shard failed for [index_65][0], node[Fj243PXdSNGtSm_jLNl9hQ], [P], s[STARTED], indexUUID [v6ZQJKpTRbiZZ5e6Q6ENyw], reason [engine failure, message [MergeException[java.io.FileNotFoundException: _7n_es090_0.tim]; nested: FileNotFoundException[_7n_es090_0.tim]; ]] 
[2014-05-01 00:17:03,453][WARN ][index.merge.scheduler    ] [Captain Wings] [index_221][0] failed to merge 
java.io.FileNotFoundException: _7e_es090_0.tim 
    at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
    at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81) 
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140) 
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371) 
    at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389) 
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) 
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119) 
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716) 
    at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122) 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89) 
    at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930) 
    at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47) 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:744) 
    Suppressed: java.io.FileNotFoundException: _7e_es090_0.tip 
        ... 26 more 
    Suppressed: java.io.FileNotFoundException: _7e_es090_0.doc 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587) 
        ... 23 more 
        Suppressed: java.io.FileNotFoundException: _7e_es090_0.pos 
            ... 28 more 
[2014-05-01 00:17:03,454][WARN ][index.engine.internal    ] [Captain Wings] [index_221][0] failed engine 
org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: _7e_es090_0.tim 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:92) 
    at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936) 
    at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930) 
    at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47) 
    at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926) 
    at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:744) 
Caused by: java.io.FileNotFoundException: _7e_es090_0.tim 
    at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
    at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
    at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
    at org.apache.lucene.util.IOUtils.closeWhileHandlingException(IOUtils.java:81) 
    at org.apache.lucene.codecs.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1140) 
    at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$BloomFilteredFieldsConsumer.close(BloomFilterPostingsFormat.java:371) 
    at org.elasticsearch.index.codec.postingsformat.Elasticsearch090PostingsFormat$1.close(Elasticsearch090PostingsFormat.java:61) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsConsumerAndSuffix.close(PerFieldPostingsFormat.java:86) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:163) 
    at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:154) 
    at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
    at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:389) 
    at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106) 
    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119) 
    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716) 
    at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122) 
    at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89) 
    ... 9 more 
    Suppressed: java.io.FileNotFoundException: _7e_es090_0.tip 
        ... 26 more 
    Suppressed: java.io.FileNotFoundException: _7e_es090_0.doc 
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:261) 
        at org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63) 
        at org.elasticsearch.index.store.Store$StoreIndexOutput.close(Store.java:611) 
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:140) 
        at org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter.close(Lucene41PostingsWriter.java:587) 
        ... 23 more 
        Suppressed: java.io.FileNotFoundException: _7e_es090_0.pos 
            ... 28 more 
[2014-05-01 00:17:06,961][INFO ][node                     ] [Captain Wings] stopped 
[2014-05-01 00:17:06,961][INFO ][node                     ] [Captain Wings] closing ... 
[2014-05-01 00:17:06,966][INFO ][node                     ] [Captain Wings] closed 

编辑:另一次运行以存储超过 100 个索引的 3m 文档。这是我在 ES 崩溃时从 BigDesk 截取的截图:


这是堆栈跟踪(使用外部 repo ,因为它太大而无法粘贴)
http://m.uploadedit.com/b034/1399053538319.txt

它在大约 500k 文档处停止,我正在检查为该进程打开的文件描述符,它们小于 5000。JVM -Xmx 的堆内存为 20gb 和 5 个碎片......

无论如何,在 1 个索引中存储 3m 文档就可以了。

编辑:
ulimit -a 输出:
core file size          (blocks, -c) 0 
data seg size           (kbytes, -d) unlimited 
scheduling priority             (-e) 0 
file size               (blocks, -f) unlimited 
pending signals                 (-i) 240150 
max locked memory       (kbytes, -l) 64 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 1024 
pipe size            (512 bytes, -p) 8 
POSIX message queues     (bytes, -q) 819200 
real-time priority              (-r) 0 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 1024 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited 

/etc/security/limits.conf 文件中的文件数已更改为 64000。此处不显示。我正在监视/proc//fd/文件夹中打开文件的数量,它超过了 1024(我认为 100 个索引大约是 6k),但远低于 64k。

编辑:
崩溃前 http://54.227.158.137:9200/_nodes/stats/indices?pretty 的结果: { "cluster_name" : "elasticsearch", "nodes" : { "t5FjNo1xQbCk97Qqv0w2sQ" : { "timestamp" : 1399395162423, "name" : "Dementia", "transport_address" : "inet[/10.69.21.196:9300]", "host" : "ip-10-69-21-196", "ip" : [ "inet[/10.69.21.196:9300]", "NONE" ], "indices" : { "docs" : { "count" : 433533, "deleted" : 0 }, "store" : { "size_in_bytes" : 10837028821, "throttle_time_in_millis" : 1971123 }, "indexing" : { "index_total" : 435229, "index_time_in_millis" : 1651288, "index_current" : 0, "delete_total" : 0, "delete_time_in_millis" : 0, "delete_current" : 0 }, [...] "merges" : { "current" : 4, "current_docs" : 574, "current_size_in_bytes" : 10968915, "total" : 13935, "total_time_in_millis" : 9056598, "total_docs" : 635889, "total_size_in_bytes" : 15384667490 }, "refresh" : { "total" : 161452, "total_time_in_millis" : 9013814 }, "flush" : { "total" : 1000, "total_time_in_millis" : 730342 }, "warmer" : { "current" : 0, "total" : 176035, "total_time_in_millis" : 18410 }, "filter_cache" : { "memory_size_in_bytes" : 0, "evictions" : 0 }, "id_cache" : { "memory_size_in_bytes" : 0 }, "fielddata" : { "memory_size_in_bytes" : 0, "evictions" : 0 }, "percolate" : { "total" : 0, "time_in_millis" : 0, "current" : 0, "memory_size_in_bytes" : -1, "memory_size" : "-1b", "queries" : 0 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 36091, "memory_in_bytes" : 76230702 }, "translog" : { "operations" : 80160, "size_in_bytes" : 1070283 } } } } }

请您参考如下方法:

您是否调整了现场数据缓存?它默认为无界,并且有这么多索引,内存不足也就不足为奇了。 http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-fielddata.html

更新,这显然解决了问题:

OOM 发生在内存映射文件中。这种类型的内存是在常规 JVM 堆之外分配的,也超出了垃圾回收的范围。看看这个线程:groups.google.com/forum/#!topic/elasticsearch/4Nj_HUl78KA对于一些可能有效的事情。显然linux默认限制了可用的内存,你应该能够删除这个限制。