Skip to content

Configuration

Rules and annotations guide for controlling log interception

General Configuration

Auditty Edge can be configured through Helm values and the ConfigMap. Most fields — including rules, enable_rule_annotations, and suppression policy settings — are cluster-wide and hot-reload within ~10–60s of a helm upgrade (Edge watches its ConfigMap file on disk); no pod restart needed. A few startup-only values (e.g. edge_api.service_url) do require a restart to take effect.

Key Configuration Fields:

  • hivemind_api_key (required): API key for connecting to Auditty control plane
  • secretRef: Store API key in a Kubernetes Secret instead of ConfigMap
  • edgeApi.enabled (default: true): Deploy Edge API for K8s API calls and metrics
  • cluster_name (required): Identifier for this cluster in Auditty — Edge will not start without this
  • org_name (optional): Organization name for display or licensing
  • org_unit (optional): Business unit or team name
  • env_name (optional): Environment identifier (dev, staging, prod). When set, Auditty surfaces an Environment filter across all dashboard, metrics, insights, and reports pages — letting you slice data by environment independently from cluster
  • enable_rule_annotations (optional): Enable per-workload annotation rules

Looking for suppression policy settings?

Everything under policy.suppression — summary format and thresholds, per-namespace/workload overrides, promote_numeric_fields, the Total-Recall search_index (search on any value directly in your platform), and outlier passthrough — is documented with full config examples in the Suppression guide. All of it is hot-reloadable through the same ConfigMap.

Example: Exclude monitoring namespace while intercepting all others:

configMap:
  rules:
    - name: "skip-monitoring"
      action: skip
      scope:
        namespace: "monitoring"
    - name: "intercept-all-others"
      action: intercept
      scope:
        namespace: "all"

Auto-Rollout on Config Change (optional)

Edge hot-reloads most configuration at runtime, but a few values (e.g. edge_api.service_url) are read once at startup. Set configMap.rollOnConfigChange: true to stamp a config checksum onto the edge DaemonSet and edge-api Deployment pod templates so any ConfigMap change automatically rolls the pods on the next helm upgrade — no manual restart, replacing the kustomize ConfigMap-hash trick. Defaults to false.

Edge API Configuration

Auditty Edge deploys two components: the DaemonSet (log interception; runs as root with scoped capabilities, not privileged mode) and Edge API (non-privileged, cluster-level service). Edge API has RBAC permissions to query the K8s API for workload resolution. And it sends metrics to the Auditty backend.

Edge API Benefits:

  • Accurate workload names: Resolves via K8s owner references (Deployment, StatefulSet, etc.)
  • Efficient K8s API: Uses informers with local cache for fast lookups
  • Efficient metrics: Batches and compresses metrics before sending to Auditty
# Edge API configuration (values.yaml)
edgeApi:
  enabled: true           # Set to false to disable Edge API
  replicas: 1             # Single replica sufficient for most clusters (holds per-cluster state — do not scale beyond 1)
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "768Mi"
      cpu: "1000m"

Air-Gapped Mode:

Setting edgeApi.enabled: false disables the Edge API component: no K8s API calls, no centralized workload-name resolution. Suppression works perfectly well without it. Metrics still flow directly to Auditty unless you also set hivemind_api_url to "disabled" — that is the setting for true air-gapped/no-egress operation.

API Key Configuration

The Auditty API key is a public JWT used for identification, not a secret credential. It can be safely stored in a ConfigMap like any other configuration data, or in a Kubernetes Secret if preferred.

Option 1: ConfigMap (simple)

configMap:
  hivemind_api_key: "your-api-key-here"

Option 2: Secret Reference

Store the API key in a Kubernetes Secret for better security practices.

# First, create the secret:
kubectl create secret generic auditty-secret \
  --namespace auditty \
  --from-literal=hivemind_api_key=your-api-key-here

# Then reference it in values.yaml:
secretRef:
  enabled: true
  secretName: "auditty-secret"
  apiKeyName: "hivemind_api_key"

Rules Overview

Auditty Edge uses rules to control which logs are intercepted and how they're processed. Rules come from three sources, merged at runtime:

  • ConfigMap rules (cluster-wide): Managed by platform/DevOps teams in the Helm values
  • Annotation rules (per-workload): Managed by application teams directly on their Deployments
  • Fleet rules (Auditty-authored): Created on the Fleet Rules page and delivered to every Edge node automatically — no ConfigMap edit or pod restart

This guide shows common patterns for configuring log interception across your cluster.

Enabling Annotation Rules

To enable annotation rules, set the feature flag in your ConfigMap:

configMap:
  enable_rule_annotations: true

Enabling Fleet Rules

Fleet rules close the loop between Auditty and Edge: rules authored on the Fleet Rules page (or generated by "Sync Monitors" with "Apply to Fleet") are delivered to every Edge node without a polling interval to wait out — a saved rule is hot-swapped into the active rule set within about a second, and an unchanged fleet costs zero network payload. Enable delivery in your Edge config:

configMap:
  remote_rules:
    enabled: true
    poll_interval: "60s"   # default
  • Fail-safe: a fetch or validation failure keeps the last-known-good fleet rules — local ConfigMap and annotation rules are never affected
  • Local overrides remote: a fleet rule whose name matches a local ConfigMap or annotation rule is ignored on that node — the local definition always wins. Define a local rule with the same name to pin or veto any fleet rule
  • Overrides are visible: the Fleet Rules page shows "overridden locally on N nodes" per rule (with the node, environment, the winning local action, and whether it came from ConfigMap or annotation) — a vetoed rule is never silently believed to be active
  • Connected Fleet: each node reports whether it is consuming fleet rules (enabled, applied version, last success/error) keyed by cluster + environment + node — the page shows converged / stale / not consuming, and warns loudly when enabled rules have zero confirmed consumers (including when no node reports status yet)
  • Rollback: disabling or deleting a rule in Auditty removes it from the fleet within seconds
  • Validated twice: rules are checked at authoring time (Auditty) and again on every node before applying — one bad rule can never freeze the fleet
  • Audited: every create, update, enable/disable, and delete is recorded in the Audit Log

Rule Actions

File-Level Actions (Control Interception)

intercept

Start monitoring logs from this workload

skip

Don't monitor logs from this workload (highest priority)

Line-Level Actions (Control Processing)

preserve

Keep these logs, never suppress them

suppress

Drop these logs to reduce noise

Common Patterns

Pattern 1: Enable Namespace, Allow Opt-Out

Scenario: DevOps team enables logging for entire production namespace, but allows teams to opt out specific workloads.

ConfigMap (cluster-wide):

configMap:
  enable_rule_annotations: true
  rules:
    - name: "intercept-production"
      action: intercept
      scope:
        namespace: "production"
      description: "Monitor all production workloads"

Annotation (opt-out specific workload):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-workload
  namespace: production
spec:
  template:
    metadata:
      annotations:
        auditty.ai/rule.skip-me: |-
          name: skip-test-workload
          action: skip
          description: "Exclude this workload from monitoring"

Result: All production workloads are monitored except test-workload (skip > intercept).

Pattern 2: Workload Opt-In (No Default Interception)

Scenario: No default monitoring - only workloads that explicitly opt-in are intercepted.

ConfigMap:

rules: []

Annotation (opt-in):

metadata:
  annotations:
    auditty.ai/rule.monitor-me: |-
      name: intercept-this-workload
      action: intercept
      description: "Monitor this specific workload"

Result: Only workloads with intercept annotations are monitored. This gives teams full control.

Note: Skip rules have higher precedence than intercept rules, so a ConfigMap skip cannot be overridden by an annotation intercept.

Pattern 3: Preserve Critical Errors by Content

Use case: Always keep error logs containing "database" or "payment".

rules:
  - name: "preserve-critical-errors"
    action: preserve
    scope:
      namespace: "production"
    match:
      any:
        - entryIncludes: "database connection failed"
        - entryIncludes: "payment processing error"
        - entryRegex: "ERROR.*timeout"
    description: "Never suppress critical errors"

Pattern 4: Preserve All High-Severity Logs

Use case: Ensure all ERROR-level and above logs are always preserved in production, regardless of other rules. This makes the built-in high-severity default explicit and extends it to include warnings.

rules:
  - name: "preserve-high-severity"
    action: preserve
    scope:
      namespaceIncludes: "prod"
    match:
      severity: "warning"
    description: "Never suppress warnings and above in production"

Note: Auditty already preserves ERROR+ logs by default (without any rule). Use a preserve rule with severity when you want to extend that protection to lower levels (like warning) or make the behavior explicit in your configuration.

Pattern 5: Suppress Noise

Use case: Drop debug logs and health check spam.

rules:
  - name: "suppress-debug"
    action: suppress
    scope:
      namespaceIncludes: "prod"
    match:
      entryIncludes: "DEBUG"
    description: "Suppress debug logs in production"

  - name: "suppress-health-checks"
    action: suppress
    match:
      entryRegex: "GET /health.*200 OK"
    description: "Drop successful health check logs"

Annotation Format

Annotations follow the pattern: auditty.ai/rule.<identifier>

Note: The name and description fields are optional.

Single Rule

Full rule with optional fields

Minimal Rule

Action only (simplest form)

Single Rule
annotations:
  auditty.ai/rule.my-rule: |-
    name: descriptive-name        # Optional
    action: skip                  # Required
    description: "Why this rule exists"  # Optional
Minimal Rule (action only)
annotations:
  auditty.ai/rule.skip-me: |-
    action: skip

Multiple Conditions (OR logic)

Matches if any condition is true

Multiple Conditions (AND logic)

Matches only if all conditions are true

Multiple Conditions (OR logic)
auditty.ai/rule.errors: |-
  name: preserve-errors
  action: preserve
  match:
    any:
      - entryIncludes: "ERROR"
      - entryIncludes: "FATAL"
      - entryIncludes: "CRITICAL"
Multiple Conditions (AND logic)
auditty.ai/rule.specific: |-
  name: suppress-specific-noise
  action: suppress
  match:
    all:
      - entryIncludes: "cache"
      - entryIncludes: "warning"
Preserve by Severity (per-workload override)
auditty.ai/rule.keep-errors: |-
  name: preserve-all-errors
  action: preserve
  match:
    severity: "error"

Override example: If a platform-wide ConfigMap rule suppresses all DEBUG logs (action: suppress, match: entryIncludes: "DEBUG"), an application team can use an annotation preserve rule with severity: "error" to ensure their errors are never suppressed — even if they contain the word "DEBUG". Preserve always has higher precedence than suppress.

Match Operators

entryIncludes (Simple String Match)

Fast, case-sensitive substring matching.

match:
  entryIncludes: "connection failed"

entryRegex (Pattern Matching)

For complex patterns. Use when entryIncludes isn't sufficient.

match:
  entryRegex: "HTTP [45][0-9]{2}"  # Matches HTTP 4xx/5xx errors

any (OR logic)

Matches if any condition is true.

match:
  any:
    - entryIncludes: "ERROR"
    - entryIncludes: "WARN"

all (AND logic)

Matches only if all conditions are true.

match:
  all:
    - entryIncludes: "database"
    - entryIncludes: "timeout"

severity (Hierarchical Level)

Matches logs at this severity level and above. Severity is auto-detected from all major log formats (JSON, logfmt, brackets, Python, Java, PHP, Ruby, Rust, and more).

match:
  severity: "warning"  # Matches warning, error, critical, fatal, etc.

Severity hierarchy (lowest to highest):

trace → debug → info → notice → warning → error → critical → fatal/panic → alert → emergency

Setting severity: "warning" matches warning, error, critical, fatal, panic, alert, and emergency. Setting severity: "error" matches error, critical, fatal, panic, alert, and emergency. Common shorthands (warn, err, crit, emerg) are also recognized.

stream (K8s Output Stream)

Matches logs from a specific container output stream. Accepts "stdout" or "stderr".

match:
  stream: "stderr"  # Match logs written to stderr

Scope Patterns

Rules are scoped by namespace, workload (the pod's owning controller — Deployment, StatefulSet, DaemonSet, or CronJob — resolved from Kubernetes owner references), and container. Each supports an exact form and an Includes (substring) form, and they can be combined and composed with the boolean any/all/not operators.

Exact Match

Match exact namespace/workload/container names

Substring Match

Match namespaces/workloads/containers by substring

Container Scope

Target a single container within a workload — fully central, no annotations

Exact Match
scope:
  namespace: "production"
  workload: "api-server"
  container: "app"          # Optional: target one container in the pod
Substring Match
scope:
  namespaceIncludes: "prod"      # Matches prod-us, prod-eu
  workloadIncludes: "api-"       # Matches api-server, api-worker
  containerIncludes: "proxy"     # Matches istio-proxy, envoy-proxy

Container-Level Scoping (cluster-wide)

Cluster-wide ConfigMap rules can target an individual container with container / containerIncludes — no pod annotations required, so all rule configuration stays in the platform-managed ConfigMap. The container is resolved on each node from the pod log path (no extra Kubernetes API calls), and rules that don't use container scope are completely unaffected.

rules:
  # Suppress chatty sidecar logs only — leave the app container untouched
  - name: "suppress-sidecar-noise"
    action: suppress
    scope:
      namespace: "payments"
      workload: "api"
      container: "sidecar"
    match:
      severity: "info"

Rule Precedence

When multiple rules match, actions are evaluated in this order (higher precedence wins):

  1. preserve (highest) - Keep logs (cannot be overridden)
  2. skip - Don't monitor this workload
  3. intercept - Monitor this workload
  4. suppress (lowest) - Drop logs

Key Points:

  • Higher precedence rules override lower ones
  • A ConfigMap skip rule cannot be overridden by an annotation intercept rule (skip > intercept)
  • A ConfigMap suppress rule can be overridden by an annotation preserve rule (preserve > suppress)

Example: If ConfigMap has suppress: "DEBUG" and annotation has preserve: "DEBUG", the logs are preserved because preserve has higher precedence.

Complete Example

ConfigMap Configuration

configMap:
  enable_rule_annotations: true
  cluster_name: "prod-us-west"

  rules:
    # Enable production namespace
    - name: "monitor-production"
      action: intercept
      scope:
        namespace: "production"

    # Skip staging
    - name: "skip-staging"
      action: skip
      scope:
        namespace: "staging"

    # Preserve all high-severity logs
    - name: "preserve-errors"
      action: preserve
      match:
        severity: "error"

    # Suppress debug in production
    - name: "suppress-debug"
      action: suppress
      scope:
        namespace: "production"
      match:
        entryIncludes: "DEBUG"

Deployment with Annotations

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  template:
    metadata:
      annotations:
        # Preserve payment-specific errors
        auditty.ai/rule.payment-errors: |-
          name: preserve-payment-errors
          action: preserve
          match:
            any:
              - entryIncludes: "payment failed"
              - entryIncludes: "transaction declined"
              - entryRegex: "card.*invalid"

        # Suppress noisy cache warnings
        auditty.ai/rule.cache-noise: |-
          name: suppress-cache-warnings
          action: suppress
          match:
            all:
              - entryIncludes: "cache"
              - entryIncludes: "miss"

Best Practices

  1. Start broad, refine narrow: Use ConfigMap for cluster-wide rules, annotations for workload-specific tuning
  2. Use entryIncludes when possible: It's faster than regex
  3. Preserve first, suppress later: When testing, use preserve to ensure you don't lose important logs
  4. Document your rules: Always include a clear description
  5. Validate in non-prod: Test annotation rules in staging before production

Troubleshooting

Annotations Not Working?

1. Annotation rules require both enable_rule_annotations: true and edgeApi.enabled: true (Edge API resolves owner/pod annotations). Check both:

kubectl get configmap auditty-config -n auditty -o yaml | grep -E "enable_rule_annotations|enabled"

2. Verify RBAC permissions — this is granted to the Edge API service account, not the DaemonSet:

kubectl auth can-i get deployments --as=system:serviceaccount:auditty:auditty-edge-api

3. Check Edge logs for errors:

kubectl logs -n auditty -l app.kubernetes.io/name=auditty-edge --tail=50 | grep -i annotation

Rule Not Matching?

  • Ensure entryIncludes string is exact (case-sensitive)
  • Test regex patterns with online tools before deploying
  • Check scope matches the namespace/workload correctly
  • Remember: annotations only affect the workload they're attached to

Summary

  • ConfigMap rules: Cluster-wide defaults managed by platform team
  • Annotation rules: Per-workload overrides managed by app teams
  • Fleet rules: Auditty-authored rules delivered to all nodes automatically
  • Precedence: preserve > skip > intercept > suppress (regardless of source)
  • Distributed control: Teams can preserve their logs even if platform suppresses them