Differences

This shows you the differences between two versions of the page.

--- wiki:sentry_troubleshooting [2024/08/26 09:21] – created antisa
+++ wiki:sentry_troubleshooting [2026/05/27 08:16] (current) – [Postgres DB getting too big] antisa
@@ Line 3: / Line 3: @@
 ====== Sentry troubleshooting ======
 ===== Postgres DB getting too big =====
-One option is to truncate the nodestore_node table. Enter the DB:
+One option is to truncate the nodestore_node table. nodestore stores the raw, unindexed parts of the data of an event. For example all event breadcrumbs and contexts are stored there. This means you will lose the details when clicking on any event in "Issues" tab. This is probably OK since you can still receive new events.
+Enter the DB:
   psql -U postgres
@@ Line 13: / Line 15: @@
   postgres=# TRUNCATE nodestore_node ;
-====== Tested on ======
+===== Kafka too big =====
+Try adapting the KAFKA_* env variables to this:
+  KAFKA_LOG_RETENTION_BYTES: 53687091200
+  KAFKA_LOG_SEGMENT_BYTES: 1073741824
+  KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
+  KAFKA_LOG_SEGMENT_DELETE_DELAY_MS: 60000
+Example from docker-compose file:
+<code>
+...
+  kafka:
+    <<: *restart_policy
+    depends_on:
+      zookeeper:
+        <<: *depends_on-healthy
+    image: "confluentinc/cp-kafka:5.5.7"
+    environment:
+      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
+      KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:9092"
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
+      KAFKA_OFFSETS_TOPIC_NUM_PARTITIONS: "1"
+      KAFKA_LOG_RETENTION_HOURS: "24"
+      KAFKA_LOG_RETENTION_BYTES: 53687091200
+      KAFKA_LOG_SEGMENT_BYTES: 1073741824
+      KAFKA_LOG_RETENTION_CHECK_INTERVAL_MS: 300000
+      KAFKA_LOG_SEGMENT_DELETE_DELAY_MS: 60000
+      KAFKA_MESSAGE_MAX_BYTES: "50000000" #50MB or bust
+      KAFKA_MAX_REQUEST_SIZE: "50000000" #50MB on requests apparently too
+      CONFLUENT_SUPPORT_METRICS_ENABLE: "false"
+      KAFKA_LOG4J_LOGGERS: "kafka.cluster=WARN,kafka.controller=WARN,kafka.coordinator=WARN,kafka.log=WARN,kafka.server=WARN,kafka.zookeeper=WARN,state.change.logger=WARN"
+      KAFKA_LOG4J_ROOT_LOGLEVEL: "WARN"
+      KAFKA_TOOLS_LOG4J_LOGLEVEL: "WARN"
+    ulimits:
+      nofile:
+        soft: 4096
+        hard: 4096
+    volumes:
+      - "sentry-kafka:/var/lib/kafka/data"
+      - "sentry-kafka-log:/var/lib/kafka/log"
+      - "sentry-secrets:/etc/kafka/secrets"
+    healthcheck:
+      <<: *healthcheck_defaults
+      test: ["CMD-SHELL", "nc -z localhost 9092"]
+      interval: 10s
+      timeout: 10s
+      retries: 30
+...
+</code>
+==== Tested on =====
   * sentry_version: 23.8.0 (docker)
   * PostgreSQL 14.5
+===== Can't start some containers =====
+<code | >
+ERROR: for memcached  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: for symbolicator-cleanup  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: for smtp  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: for clickhouse  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: for zookeeper  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: for postgres  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
+ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
+If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
+</code>
+Just run it with higher timeout value:
+  COMPOSE_HTTP_TIMEOUT=120 docker-compose ps
+==== Tested on =====
+  * Sentry 20.6.037a7530 (docker)
 ====== See also ======
   * [[wiki:sentry_install|Sentry install]]
 ====== References ======
+  * https://forum.sentry.io/t/restrict-kafka-disk-usage/9838/5
+  * https://kafka.apache.org/documentation/#topicconfigs