Vault (Log Archive)

What Is Vault?

Vault is Edge's log archiving feature. When enabled, Edge writes every suppressed event (the ones replaced by a summary) to cloud storage — giving you a complete, cost-effective archive of exactly the log lines you no longer see inline. Non-suppressed events are unaffected: they already reach your normal destination unchanged, so Vault does not duplicate them. Suppressed logs can be recovered instantly using the backfill link included in every suppression summary.

Enabling Vault

Add the vault configuration under the configMap section in your values.yaml:

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "s3"
      bucket: "my-log-archive"
      prefix: "logs"              # Optional: object key prefix
      region: "us-east-1"         # AWS region (default: us-east-1)

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "gcs"
      bucket: "my-log-archive"
      prefix: "logs"

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "azure"
      bucket: "my-log-container"   # Azure Blob Storage container name
      prefix: "logs"
      azure_storage_account: "mystorageaccount"

S3-Compatible Storage (MinIO)

Vault also supports any S3-compatible storage backend:

configMap:
  forwarder:
    vault:
      enabled: true
      provider: "s3"
      bucket: "my-log-archive"
      s3_endpoint: "https://minio.internal:9000"
      s3_force_path_style: true    # Required for MinIO
      region: "us-east-1"

Retrieving Archived Logs

Every suppression summary written by Edge includes a backfill_url field. Click the link to open the Replay Viewer with all original suppressed logs for that pattern and time window.

A backfill_url never dangles. Edge is fail-closed by default (require_archive): if its durable-write path to the Vault is unhealthy — a cloud-storage outage, an expired license, or a full retry spool — it stops suppressing and passes raw lines straight through to your log platform until durability is confirmed again. So Auditty never emits a summary (and never asks you to trust a backfill link) for data it could not durably archive. Combined with the on-disk retry spool and off-node write-ahead log, this makes the suppress→archive→backfill path lossless end to end.

{
  "message": "Health check passed",
  "auditty": {
    "suppressed": 100,
    "window": { "start": "2026-01-15T09:00:00Z", "end": "2026-01-15T09:05:00Z" },
    "duration": "5m0s",
    "fingerprint": "a1b2c3d4",
    "backfill_url": "https://<your-hivemind-url>/backfill/replay?token=<TOKEN>&sig=<SIG>"
  }
}

Programmatic / Agentic Backfill API (JSON)

Raw (including suppressed) archived logs are available as JSON for scripts, agents, and compliance tooling — no browser required. For automated/agentic workflows use the versioned public API under /api/v1/backfill/*. These endpoints return JSON only and carry a stability guarantee (see below) — build against them rather than the HTML backfill page or the UI’s internal routes.

Authentication is the token + sig pair already embedded in every suppression summary’s backfill_url (HMAC-signed, tenant-scoped, capability-style). No session or separate API key is required, so an agent can go straight from a Datadog/Splunk record to its originals.

Read flow — poll status until ready, then page through the records:

# token + sig come from the summary's backfill_url
GET /api/v1/backfill/status?token=<TOKEN>&sig=<SIG>
{ "status": "ready" }            # retry while "loading"

GET /api/v1/backfill/page?token=<TOKEN>&sig=<SIG>&page=1&sort=desc
{
  "records": [
    {
      "timestamp": "2026-01-01T00:00:01.123Z",
      "namespace": "payments",
      "workload": "api",
      "fingerprint": "a1b2c3d4",
      "node_id": "node-1",
      "source_uri": "...",
      "suppressed": true,
      "log_line": "Health check passed",
      "preserved_json": "{...}"
    }
  ],
  "totalCount": 100,
  "hasMore": false,
  "page": 1,
  "pageSize": 100,
  "queryTimeMs": 7
}
# paginate while hasMore === true

Restore flow — list targets, then replay the originals back into your platform:

GET /api/v1/backfill/integrations?token=<TOKEN>&sig=<SIG>
[ { "id": 12, "platform": "datadog", "name": "DD - prod" } ]

POST /api/v1/backfill/replay
{ "token": "<TOKEN>", "sig": "<SIG>", "integrationId": 12 }

{ "totalSent": 100, "totalFailed": 0, "durationMs": 842 }
# already replayed (idempotent): { "alreadyReplayed": true, "totalSent": 100, ... }
# pass "force": true to re-send

Alternative — by fingerprint (no summary needed). To query the archive for any fingerprint and date range, use GET /api/vault/logs?fingerprint=…&startDate=YYYY-MM-DD&page=1. This call is authenticated with your Auditty session (not a backfill token) and returns the same record shape, with { status: "loading", retryAfterMs } (HTTP 202) while a cold fingerprint warms up.

Field Predicates: Filtering by Value

The summary's ids object indexes only searchable correlation keys — bare magnitudes and measurements (latencies.kong, bytes_sent) are archived losslessly but aren't listed there. A field predicate lets you filter a backfill down to only the rows where a specific field in preserved_json matches a value or comparison — no need to promote the field first, and no need to page through every archived record by hand.

In the Replay Viewer, use the filter box above the table: enter a field name (dotted paths like latencies.kong work directly — no need to escape the dot), pick an operator, and a value.

latencies.kong > 100    # only rows where the kong measurement exceeded 100

This works with the paginated table view and the time histogram alike, and with the JSON API (GET /api/v1/backfill/page) once the predicate is baked into the token you pass.

Global Search by Log Text

The Vault Explorer’s search box accepts free text, not just fingerprint hashes. Type a fragment of a log line (“connection refused”, an error code, a service name) and Auditty searches across all fingerprints, returning the patterns whose logs contain it — each with its representative log line, namespace/workload, and last-seen date. Click a match to open its archived logs in the Replay Viewer, where a field predicate can narrow further.

Vault Explorer (Auditty UI)

Beyond following backfill links from summaries, you can browse all archived logs directly in Auditty. Navigate to Vault in the Auditty dashboard:

Replay to Platform

The Replay Viewer lets you send archived logs directly back to your logging platform. Supported platforms include Datadog, Splunk, Elasticsearch, Coralogix, Grafana Loki, New Relic, Sumo Logic, and any custom endpoint via Webhook. This is useful when you need the suppressed logs re-ingested for investigation or compliance.

Query the Archive with SQL (Open Table Format)

The Vault is plain Parquet in your bucket under a Hive-style partition layout (tenant/year/month/day/hour) — open by construction, no proprietary format. In Vault Explorer, click Query with SQL, pick your engine, and copy the generated setup statement for your registered bucket. The full archive (suppressed originals included) is queryable with SQL, Spark, or any Parquet-native tool, with partition pruning and no Auditty components in the read path.

• Amazon Athena — partition projection: no crawler, no MSCK REPAIR, new vault files are queryable the moment they land
• Trino / Presto — hive connector external table
• Apache Iceberg (Spark SQL) — creates an Iceberg table and registers the existing vault files in place via add_files (no copy, no rewrite)

Raw Archival Sink (self-managed, full-fidelity)

Some teams need a complete, untouched copy of every log line in storage they control — independent of Auditty — for compliance or long-term retention. The raw archival sink is an optional, node-local sink that writes the complete pre-suppression stream (every line, suppressed or not) to plain-text files alongside the reduced stream that flows to your observability backend.

configMap:
  forwarder:
    raw_archive:
      enabled: true
      directory: /var/log/auditty/raw   # node-local base dir (mount as a hostPath/volume)
      rotation_max_size: 104857600       # 100Mi; rotated files are plain text (not gzipped)
      rotation_max_files: 3              # node-local buffer depth per source

The directory layout mirrors your source paths, so one recursive glob captures every stream. Point your downstream agent at it:

[sources.auditty_raw]
type    = "file"
include = ["/var/log/auditty/raw/**/*.log"]

Summary

• Vault archives every suppressed log line to cloud storage — non-suppressed events already reach your normal destination and are not duplicated into Vault
• Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob Storage, and S3-compatible backends
• Instant recovery via backfill links in suppression summaries or Auditty Vault Explorer
• Suppression summaries include a signed backfill URL to view original logs instantly
• One-click replay: Send archived logs directly to your logging platform from the Replay Viewer
• Suppress with confidence: every suppressed original is archived to your own storage — no silent loss in normal operation, with bounded durability during outages (retry spool, drop-oldest at its cap)
• Automatic optimization: Archived files are consolidated in the background for fast retrieval — no configuration needed
• Raw archival sink (optional): Write the complete pre-suppression stream to node-local plain-text files for a downstream agent to ship to storage you control — independently of Auditty