Differences

This shows you the differences between two versions of the page.

--- wiki:logstash_setup [2024/08/27 13:00] – created antisa
+++ wiki:logstash_setup [2024/10/17 13:05] (current) – [nginx logs using regular indices] add info about multiple file inputs antisa
@@ Line 10: / Line 10: @@
 ===== Examples =====
-==== nginx logs ====
+==== nginx logs using regular indices ====
 <code>
 input {
@@ Line 46: / Line 46: @@
 </code>
+<WRAP center round tip 60%>
+You can also use multiple file inputs like so:
+<code>
+input {
+   file { path => [
+     "/var/log/nginx/nginx_logs*_access.log",
+     "/var/log/nginx/some_other_web*_access.log"
+     ]
+   }
+}
+...
+</code>
+</WRAP>
 Above we're using a grok pattern named **NGINX_ACCESS** stored in patterns directory. Example of pattern:
@@ Line 167: / Line 179: @@
 ''referer=(%{URI:referer}|-)'' construct means the referer in this case might be empty (''-'').
+==== nginx logs using datastreams ====
+This is perhaps a better approach, using [[https://opensearch.org/docs/2.15/im-plugin/data-streams/|datastreams]] as this will do some automatic rollover to new index.
+Change the logstash ''output'' part to this
+<code>
+...
+output {
+  opensearch {
+    hosts       => "https://{{ opensearch_host }}:9200"
+    user        => "logstash"
+    password    => "mypassword"
+    ssl_certificate_verification => false
+    action      => "create"
+    index       => "whatever-pattern-abc"
+  }
+}...
+</code>
+''index'' needs to be set to the name of the datastream you defined in Opensearch and add ''action'' directive.
+In Opensearch you need to create a datastream, but first you need to create a template.
+**1.** Go to Index Management > Templates > Create template
+{{ :wiki:screenshots:opensearch_create_template1.png?800 |}}
+**2.** Add template name, select type "Data streams" and put in the Time field (@timestamp in this example). Index pattern should match the pattern name for which logstash user has rights to write to and this will be the name of datastream used later.
+{{ :wiki:screenshots:opensearch_create_template2.png?800 |}}
+**3.** You can add an alias if you want, replicas are set to 0 here to save some space.
+{{ :wiki:screenshots:opensearch_create_template3.png?800 |}}
+**4.** In field mappings you need to map the fields sent by logstash. Easiest is to c/p the json into json editor from existing index, like the one that would be created by logstash using the regular index (see above).
+{{ :wiki:screenshots:opensearch_create_template4.png?800 |}}
+**5.** When creating datastream the name must match the pattern from step 2 above, but it doesn't have to be exactly the same (so here it should be "whatever-pattern-abc" to match logstash config). The rest should be autofilled.
+{{ :wiki:screenshots:opensearch_create_datastream1.png?800 |}}
 ====== Tested on ======
-  *
+  * logstash-8.14.3
+  * Opensearch 2.15
 ====== See also ======
+  * [[wiki:rollover_opensearch_elasticsearch_indexes|Rollover opensearch/elasticsearch indexes]]
   * [[wiki:opensearch_installation|Opensearch installation]]