antisaWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{tag>elasticsearch elk opensearch logging}}

====== Rollover opensearch/elasticsearch indexes ======


===== Datastream indexes =====
In index management create a new policy. Most important is to match the index pattern with the name of your datastreams e.g. below we see a regex that will match datastreams as defined in //Index Management > Data streams//

{{ :wiki:screenshots:opensearch_state_idx_pattern.png?600 |}}

In the policy you can define what will be done with the matching indexes, in this case the backing indexes of those datastreams. Here's an example JSON policy that changes state and rolls over to a new backing index after 20GB:

<code json>
{
    "id": "rollover_20GB",
    "seqNo": 36234,
    "primaryTerm": 1,
    "policy": {
        "policy_id": "rollover_20GB",
        "description": "Rollover to a new index in datastream after the backing index reaches 20GB.",
        "last_updated_time": 1725353927700,
        "schema_version": 21,
        "error_notification": null,
        "default_state": "check index size",
        "states": [
            {
                "name": "check index size",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "rollover",
                        "conditions": {
                            "min_size": "20gb"
                        }
                    }
                ]
            },
            {
                "name": "rollover",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_size": "20gb",
                            "copy_alias": false
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "logstash-nginx-access-logs-live*"
                ],
                "priority": 10,
                "last_updated_time": 1725021614800
            }
        ]
    }
}
</code>

The datastreams rely on a template (see [[wiki:logstash_setup#nginx_logs_using_datastreams|]]) which should have the advanced setting set to the name of the backing index e.g.

  "index.plugins.index_state_management.rollover_alias": ".ds-logstash-nginx-*"

{{ :wiki:screenshots:2024:opensearch_template_advanced.png?600 |}}

If you now go to //Policy managed indexes// you should see the indices from the data streams:

{{ :wiki:screenshots:2024:opensearch_policy1.png?600 |}}

**1** and **2** shows the state the index is in defined by whatever the names are defined in above policy

Here we see a successful rollover to a new index

{{ :wiki:screenshots:2024:opensearch_policy_rollover_success.png?600 |}}

Also if you go to //Data streams// and into one of the indexes you should the policy attached there

{{ :wiki:screenshots:2024:opensearch_index_managed_by_policy.png?600 |}}

Any subsequent index should also be managed automatically now.

<WRAP center round important 60%>
The [[https://opensearch.org/docs/2.15/im-plugin/data-streams/#step-5-rollover-a-data-stream|docs]] mention that "The ISM policy is applied to the backing indexes at the time of their creation. When you associate a policy to a data stream, it only affects the future backing indexes of that data stream." meaning you will need to manually rollever the indexes the first time to apply the policy -> Select the datastreams then //Actions -> Roll over//
</WRAP>

====== Tested on ======
  * Opensearch 2.15

====== See also ======
  * [[wiki:opensearch_installation|Opensearch installation]]
====== References ======
  * https://opensearch.org/docs/2.15/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover