本文共 6867 字，大约阅读时间需要 22 分钟。
忙忙碌碌中2016就要过去，借这篇博客小小总结一下。像往年一样每年都是很忙很忙，今年尤其如此，哈哈哈！适逢年末，这一年可以总结的东西有很多，比如：开始读了这本专业书《Modern Authentication with Azure Active Directory for Web applications》，只可惜还没有完全读完，来年仍需要加把劲，多花些时间在读书，少些时间在手机上！此外，自从搬到28楼后，投身了一项新的群众体育运动 - 桌球，积极健身好好工作。
默认的Elasticsearch Azure插件只支持向一个Azure存储账号(storage account)写入/读出集群快照(snapshot)数据，索引的快照数据是以 block blob的形式存储在Azure存储账号的blob中的，我在另一篇博客《》中分析这部分的代码。这个限制对于大型Elasticsearch集群（例如：数据量很大TB， 数据节点>30）而言，会导致过载单一的storage account以至于snapshot失败或者PARTIAL失败，在Elasticsearch的日志文件或者快照状态信息中会看到IndexShardSnapshotFailedException，如下面的例子所示：
"state": "PARTIAL", "start_time": "2016-09-21T00:10:09.180Z", "start_time_in_millis": 1474416609180, "end_time": "2016-09-21T02:14:36.642Z", "end_time_in_millis": 1474424076642, "duration_in_millis": 7467462,
"failures": [ { "node_id": "SVT4jVpiTVmiH8K7ctWrOQ", "index": "my_index_20160913d", "reason": "IndexShardSnapshotFailedException[[my_index_20160913d][1] Failed to perform snapshot (index files)]; nested: IOException; nested: StorageException[The server encountered an unknown failure: ]; nested: IOException[Error writing to server]; ", "shard_id": 1, "status": "INTERNAL_SERVER_ERROR" }, ... ]
[2016-07-22 01:27:36,168][WARN ][snapshots ] [ESNode-ElasticSearchData_IN_53] [[myindex.2016_07_10][4]] [snapshot:001008] failed to create snapshot org.elasticsearch.index.snapshots.IndexShardSnapshotFailedException: [myindex.2016_07_10][4] Failed to perform snapshot (index files) at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$SnapshotContext.snapshot(BlobStoreIndexShardRepository.java:509) at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.snapshot(BlobStoreIndexShardRepository.java:140) at org.elasticsearch.index.snapshots.IndexShardSnapshotAndRestoreService.snapshot(IndexShardSnapshotAndRestoreService.java:85) at org.elasticsearch.snapshots.SnapshotsService$5.run(SnapshotsService.java:871) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:643) at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:444) at com.microsoft.azure.storage.blob.BlobOutputStream.access$000(BlobOutputStream.java:53) at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:388) at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:385) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: com.microsoft.azure.storage.StorageException: The server encountered an unknown failure: at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:101) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:199) at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1006) at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:978) at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:438)
... 9 more
Caused by: java.io.IOException: Error writing to server at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:666) at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:678) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:119) ... 12 more
通过扩展Elasticsearch Azure Plugin，可以让它支持读/写snapshot数据到多个storage account，从而将Lucene索引数据文件平均分配到多个storage account中，这样就避免到了过载一个storage account而导致snapshot失败。下表总结了将一个包含3777G，185847个Lucene文件的集群snapshot写入到9个Azure存储账户的结果，不难看出，扩展的Azure Plugin将快照数据基本上平均（文件个数和数据大小）写入到配置的9个账户中。基于这个思路，改造了Elasticsearch Azure Plugin支持配置多个storage accounts并读/写快照数据。可以在到这个扩展的Azure Plugin，目前支持的Elasticsearch版本有：
- 1.7.X
- 2.4.1
- 2.4.2
- 2.4.3
- 5.2.1 (包含了Bug#修复代码)
| Azure存储账号 | 文件数量 | 文件大小 |
| esstorage1 | 20634 | 420 G |
| esstorage2 | 20597 | 419 G |
| esstorage3 | 20549 | 418 G |
| esstorage4 | 20559 | 416 G |
| esstorage5 | 20637 | 420 G |
| esstorage6 | 20726 | 420 G |
| esstorage7 | 20769 | 424 G |
| esstorage8 | 20697 | 420 G |
| esstorage9 | 20679 | 417 G |
此外，Elasticsearch支持读写snapshot到 类型的storage accounts。通过写入snapshot到primary location，然后在secondary location恢复数据，可以在不同地域的数据中心之间建立互为备份的Elasticsearch集群，如下图所示：
Elasticsearch 1.7.X 版本插件配置及命令
Elasticsearch.yml cloud.azure.storage.account: [storageaccount1,storageaccount2,storageaccount3] cloud.azure.storage.key: [key1, key2, key3] Commands #1: define repository PUT _snapshot/plugintest160921 { "type": "azure", "settings": { "account": "storageaccount1,storageaccount2,storageaccount3", "container": "plugintest160921" } } #2: take snapshot PUT _snapshot/plugintest160921/backup0921?wait_for_completion=true { } #3: restore POST _snapshot/plugintest160921/backup0921/_restore?wait_for_completion=true { "ignore_unavailable": "true", "include_global_state": false } #4: define repository for secondary PUT _snapshot/plugintest160921 { "type": "azure", "settings": { "account": "storageaccount1,storageaccount2,storageaccount3", "container": "plugintest160921", "location_mode": "secondary_only" } } Elasticsearch 2.4.X 版本插件配置及命令
Elasticsearch.yml cloud.azure.storage.my_account1.account: storageaccount1 cloud.azure.storage.my_account1.key: key1 cloud.azure.storage.my_account1.default: true cloud.azure.storage.my_account2.account: storageaccount2 cloud.azure.storage.my_account2.key: key2 cloud.azure.storage.my_account2.default: true cloud.azure.storage.my_account3.account: storageaccount3 cloud.azure.storage.my_account3.key: key3 cloud.azure.storage.my_account3.default: true Commands #1: define repository PUT _snapshot/plugintest160921 { "type": "azure", "settings": { "account": "my_account1,my_account2,my_account3", "container": "plugintest160921" } } #2: take snapshot PUT _snapshot/plugintest160921/backup0921?wait_for_completion=true { } #3: restore POST _snapshot/plugintest160921/backup0921/_restore?wait_for_completion=true { "ignore_unavailable": "true", "include_global_state": false } #4: define repository for secondary PUT _snapshot/plugintest160921 { "type": "azure", "settings": { "account": "my_account1,my_account2,my_account3", "container": "plugintest160921", "location_mode": "secondary_only" } }
创建了Pull Request#给Elasticsearch，希望能够集成这个功能到Elasticsearch 5.X或者6.X中去。但Elasticsearch团队对snapshot/restore的未来实现有其他的考虑，所以我的PR未被接受。