Both sides previous revisionPrevious revisionNext revision | Previous revision |
wiki:elasticsearch_troubleshooting [2022/02/18 13:17] – [See also] add graylog antisa | wiki:elasticsearch_troubleshooting [2023/02/28 15:02] (current) – [Unable to write to elasticsearch] add graylog index link antisa |
---|
Then you might have to recalculate the index ranges (''System > Indices > index set > Maintenance > Recalculate index ranges'') and/or manually rotate the write index (''System > Indices > index set > Maintenance > Rotate active write index'') | Then you might have to recalculate the index ranges (''System > Indices > index set > Maintenance > Recalculate index ranges'') and/or manually rotate the write index (''System > Indices > index set > Maintenance > Rotate active write index'') |
| |
====== Tested on ====== | |
| |
| ===== Tested on ===== |
* Graylog 3.3.16 | * Graylog 3.3.16 |
* Debian 9.13 Stretch | * Debian 9.13 Stretch |
| |
| ===== Unable to write to elasticsearch ===== |
| Trying to GET some data from the ES works but POST does not. The issue could be that the ES was put into read-only mode. It does this if the free space on server starts getting low. In that case you'll get this warning: |
| |
| <code> |
| [4:39 PM] { |
| "error" : { |
| "root_cause" : [ |
| { |
| "type" : "cluster_block_exception", |
| "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" |
| } |
| ], |
| "type" : "cluster_block_exception", |
| "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];" |
| }, |
| "status" : 403 |
| } |
| </code> |
| |
| If you are running in docker you might see a less useful message like: |
| <code> |
| [2022-04-21T13:26:04,269][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ddbAopn] low disk watermark [85%] exceeded on [ddbAopnMTL2VKLZs_zM6bQ][ddbAopn][/usr/share/elasticsearch/data/nodes/0] free: 117.5gb[12.9%], replicas will not be assigned to this node |
| </code> |
| |
| |
| Free some disk space for example delete an old index (see howto for [[wiki:graylog_troubleshooting#elasticsearch_nodes_disk_usage_above_low_watermark|graylog index management]]) |
| |
| curl -X DELETE -u undefined:$ESPASS "localhost:9200/my-index?pretty" |
| |
| and run this: |
| curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' |
| |
| You can also change the watermark threshold e.g. |
| |
| |
| curl -X PUT -u undefined:$ESPASS "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' |
| { |
| "transient": { |
| "cluster.routing.allocation.disk.watermark.low": "100gb", |
| "cluster.routing.allocation.disk.watermark.high": "50gb", |
| "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", |
| "cluster.info.update.interval": "1m" |
| } |
| }' |
| |
| [[https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html|Check]] the docs for more info. |
| |
| ===== snapshot missing exception ===== |
| If you get an error like: |
| |
| "snapshot_missing_exception" |
| |
| Delete the snapshot repo |
| |
| curl -X DELETE -u undefined:$ESPASS "localhost:9200/_snapshot/es_backup?pretty" |
| |
| and try listing again. |
| |
| ===== index ... is the write index for the datastream ===== |
| When trying to delete the index like |
| |
| curl -XDELETE 'localhost:9200/.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001?pretty' |
| |
| you get |
| |
| <code> |
| { |
| "error" : { |
| "root_cause" : [ |
| { |
| "type" : "illegal_argument_exception", |
| "reason" : "index [.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted" |
| } |
| ], |
| "type" : "illegal_argument_exception", |
| "reason" : "index [.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted" |
| }, |
| "status" : 400 |
| } |
| |
| </code> |
| |
| you need to rollover to the new index, e.g. |
| |
| curl -s -X POST "localhost:9200/.logs-deprecation.elasticsearch-default/_rollover" |
| |
| and run delete command again. |
| ===== Tested on ===== |
| * Debian 10 |
| * Elastic search docker container ver. 6.8.16 |
| |
====== See also ====== | ====== See also ====== |
* [[wiki:graylog_troubleshooting|Graylog troubleshooting]] | * [[wiki:graylog_troubleshooting|Graylog troubleshooting]] |
| * [[wiki:elasticsearch_commands|Elasticsearch commands]] |
| * [[wiki:kibana_troubleshooting|Kibana troubleshooting]] |
====== References ====== | ====== References ====== |
* https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/ | * https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/ |
* https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html#reason-unassigned | * https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html#reason-unassigned |
* https://discuss.elastic.co/t/restore-of-elasticsearch-data-fails-with-corruptindexexception-checksum-failed-hardware-problem/261619/3 | * https://discuss.elastic.co/t/restore-of-elasticsearch-data-fails-with-corruptindexexception-checksum-failed-hardware-problem/261619/3 |
| * https://stackoverflow.com/questions/50609417/elasticsearch-error-cluster-block-exception-forbidden-12-index-read-only-all |
| * https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html |
| |