Differences

This shows you the differences between two versions of the page.

--- wiki:elasticsearch_troubleshooting [2022/01/12 15:29] – created antisa
+++ wiki:elasticsearch_troubleshooting [2024/07/23 11:28] (current) – add opensearch curl error antisa
@@ Line 1: / Line 1: @@
-{{tag>troubleshooting elasticsearch graylog}}
+{{tag>troubleshooting elasticsearch opensearch graylog}}
-====== Elasticsearch troubleshooting ======
+====== Elasticsearch/Opensearch troubleshooting ======
 If graylog stops showing message streams it could be an issue with indexes.
@@ Line 88: / Line 88: @@
 Then you might have to recalculate the index ranges (''System > Indices > index set > Maintenance > Recalculate index ranges'') and/or manually rotate the write index (''System > Indices > index set > Maintenance > Rotate active write index'')
-====== Tested on ======
+===== Tested on =====
   * Graylog 3.3.16
   * Debian 9.13 Stretch
-====== See also ======
+===== Unable to write to elasticsearch =====
+Trying to GET some data from the ES works but POST does not. The issue could be that the ES was put into read-only mode. It does this if the free space on server starts getting low. In that case you'll get this warning:
+<code>
+[4:39 PM] {
+  "error" : {
+    "root_cause" : [
+      {
+        "type" : "cluster_block_exception",
+        "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
+      }
+    ],
+    "type" : "cluster_block_exception",
+    "reason" : "blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"
+  },
+  "status" : 403
+}
+</code>
+If you are running in docker you might see a less useful message like:
+<code>
+[2022-04-21T13:26:04,269][INFO ][o.e.c.r.a.DiskThresholdMonitor] [ddbAopn] low disk watermark [85%] exceeded on [ddbAopnMTL2VKLZs_zM6bQ][ddbAopn][/usr/share/elasticsearch/data/nodes/0] free: 117.5gb[12.9%], replicas will not be assigned to this node
+</code>
+Free some disk space for example delete an old index (see howto for [[wiki:graylog_troubleshooting#elasticsearch_nodes_disk_usage_above_low_watermark|graylog index management]])
+  curl -X DELETE -u undefined:$ESPASS "localhost:9200/my-index?pretty"
+ and run this:
+  curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'
+You can also change the watermark threshold e.g.
+  curl -X PUT -u undefined:$ESPASS "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
+  {
+  "transient": {
+    "cluster.routing.allocation.disk.watermark.low": "100gb",
+    "cluster.routing.allocation.disk.watermark.high": "50gb",
+    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
+    "cluster.info.update.interval": "1m"
+  }
+  }'
+[[https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html|Check]] the docs for more info.
+===== snapshot missing exception =====
+If you get an error like:
+  "snapshot_missing_exception"
+Delete the snapshot repo
+  curl -X DELETE -u undefined:$ESPASS "localhost:9200/_snapshot/es_backup?pretty"
+and try listing again.
+===== index ... is the write index for the datastream =====
+When trying to delete the index like
+  curl -XDELETE 'localhost:9200/.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001?pretty'
+you get
+<code>
+{
+  "error" : {
+    "root_cause" : [
+      {
+        "type" : "illegal_argument_exception",
+        "reason" : "index [.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"
+      }
+    ],
+    "type" : "illegal_argument_exception",
+    "reason" : "index [.ds-.logs-deprecation.elasticsearch-default-2022.11.15-000001] is the write index for data stream [.logs-deprecation.elasticsearch-default] and cannot be deleted"
+  },
+  "status" : 400
+}
+</code>
+you need to rollover to the new index, e.g.
+  curl -s -X POST "localhost:9200/.logs-deprecation.elasticsearch-default/_rollover"
+and run delete command again.
+===== curl (52) empty reply from server =====
+Happened with OpenSearch docker compose installation trying this:
+  curl -u admin:Antekante_1 -XGET "http://localhost:9200/_cluster/health?pretty"
+It needs the certificate file in the command but if you are testing, easiest is just to disable the ssl. Add the following line in ''docker-compose.yml''
+<code>
+  - plugins.security.ssl.http.enabled=false
+</code>
+and rerun
+  docker-compose up -d
+===== Tested on =====
+  * Debian 10
+  * Elastic search docker container ver. 6.8.16
+====== See also ======
+  * [[wiki:graylog_troubleshooting|Graylog troubleshooting]]
+  * [[wiki:elasticsearch_commands|Elasticsearch commands]]
+  * [[wiki:kibana_troubleshooting|Kibana troubleshooting]]
 ====== References ======
   * https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/
   * https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html#reason-unassigned
   * https://discuss.elastic.co/t/restore-of-elasticsearch-data-fails-with-corruptindexexception-checksum-failed-hardware-problem/261619/3
+  * https://stackoverflow.com/questions/50609417/elasticsearch-error-cluster-block-exception-forbidden-12-index-read-only-all
+  * https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html