Dumpling documentation

Dumpling is a streaming anonymizer for plain SQL dumps. It supports PostgreSQL (pg_dump plain format), SQLite (.dump), and SQL Server / MSSQL (SSMS / mssql-scripter plain SQL output). For PostgreSQL custom-format or directory-format archives (e.g. Heroku pg:backups:download), Dumpling auto-detects them when --format postgres (default) and invokes pg_restore -f -—see PostgreSQL archives and compressed inputs in the configuration guide.

New here? Start with Getting started: generate a draft policy with scaffold-config, review and add secrets, run Dumpling, then tighten with lint-policy and optional CI flags.

This documentation covers the operating model for day-to-day use:

• why Dumpling is designed the way it is (Design rationale), including alternatives and trade-offs,
• how to build and run Dumpling locally,
• how to configure transformation behavior safely (including keep, column_cases, and predicate operators),
• how dump seals, --no-seal, and --report provide audit evidence,
• how CI validates quality before changes merge,
• and how maintainers produce tagged releases.

Documentation quality gate

The mdBook site is built in CI as follows:

• Pull requests: the Docs (PR) workflow runs mdbook build when docs-related paths change (no deploy).
• main: the Docs workflow builds and deploys to GitHub Pages when docs-related paths change.

This keeps the docs in a continuously deployable state instead of drifting from the codebase.

Getting started

This page is the shortest path from zero to a first successful run. For strategy details, row filters, dump seals (--no-seal), the JSON --report sidecar, and CI patterns, continue with the configuration guide and the repository README.md.

Prerequisites

• Rust stable toolchain (rustup recommended). The repo includes rust-toolchain.toml (stable + rustfmt + clippy) so CI and local cargo stay aligned.
• cargo on your PATH

Optional: run ./scripts/setup-dev.sh once from the repo root — it installs toolchain components, cargo fetch, and a pinned mdBook under .tools/ for the same docs build CI uses.

Build

cargo build --release
./target/release/dumpling --help

Python / pip (dumpling-cli)

pip install dumpling-cli
dumpling --help

First anonymization

1. Generate a draft policy (recommended) — From your project root (or anywhere you keep config):

   dumpling scaffold-config -i dump.sql -o .dumplingconf
   

   This beta subcommand streams the dump once and writes inferred [rules] from SQL column names (CREATE TABLE, INSERT, and PostgreSQL COPY column lists). It does not require an existing Dumpling config in the current directory (optional config is only merged for pg_restore / keep-original defaults). Heuristics are English-oriented; output is draft only—review and edit every rule, add a top-level salt (for hashing) and any ${…} secret placeholders before production use.

   Useful flags:

   • --infer-json-paths — Keep up to five sampled rows per table (reservoir) and suggest nested JSON rules as column.path.leaf.
   • --max-json-depth — Cap JSON walking depth when using --infer-json-paths (default 24).
   • --format — postgres (default), sqlite, or mssql.
   • --pg-restore-path / --pg-restore-arg — Optional pg_restore binary and extra arguments when --input is a PostgreSQL custom-format or directory-format archive (auto-detected with --format postgres); see PostgreSQL archives and compressed inputs.

   Run dumpling scaffold-config --help for the full flag list.

2. Or start from the example policy — Copy .dumplingconf.example to .dumplingconf (or merge under [tool.dumpling] in pyproject.toml) and author [rules] by hand. Set environment variables for salt and any ${…} references.

3. Align rules with your dump (manual path only) — If you skipped scaffold-config, use CREATE TABLE, COPY … (…), and INSERT INTO … (…) lines to name [rules."table"] or [rules."schema.table"] keys. Trim to the tables you care about first.

4. Run Dumpling — dumpling -i dump.sql -o sanitized.sql (add -c path if the config is not in the default search path). Use dumpling --check -i dump.sql when you only want to know whether anything would change. Output is prefixed with a dump-seal comment by default; pass --no-seal for stdin/stdout pipelines, and --report file.json for an audit sidecar (see Dump seal and JSON report).

5. Tighten the policy — Run dumpling lint-policy on your config (catches unsalted hashes, inconsistent domains, uncovered [sensitive_columns], and invalid regex predicates). When you are ready for stricter gates, add [sensitive_columns] and use --strict-coverage, --report, and --scan-output as described in the configuration guide and the repository README.md. For selective passthrough, see keep under column_cases and the not_* predicate operators in the README.

PostgreSQL custom-format archives

If your input is a PostgreSQL custom-format file or directory-format folder (not plain SQL), use --format postgres (default): Dumpling auto-detects the archive and runs pg_restore -f - (needs pg_restore from PostgreSQL client tools). Gzip-wrapped plain SQL is streamed without a temp file; ZIP (or gzip wrapping PGDMP) uses a temp extract that is cleaned up afterward. See PostgreSQL archives and compressed inputs in the configuration guide.

Test locally (contributors)

cargo fmt --all -- --check
cargo clippy --all-targets --all-features
cargo test --all-targets --all-features

Design rationale

This page explains why Dumpling is shaped the way it is: the problems it optimizes for, the alternatives we considered, and the strengths that fall out of those choices. It is written for operators and contributors who want the “why,” not only the “how.”

If you want to run Dumpling today, start with Getting started. Configuration details live in the configuration guide.

The problem Dumpling solves

Teams need realistic, shareable database snapshots for staging, demos, support reproduction, and analytics sandboxes — without shipping production PII.

That requirement sounds simple until you add constraints common in real pipelines:

• dumps are often multi-gigabyte and must run on modest CI runners;
• foreign keys and “same person across tables” must stay consistent after rewrite;
• a missing or empty policy must be a loud failure, not a silent passthrough;
• compliance and security teams want evidence that a given file was processed under a known policy;
• the tool must fit batch/CI workflows, not only interactive DBA sessions.

Dumpling is a streaming, file-based anonymizer for plain SQL dumps. It never connects to a live database. That single constraint drives most of the design below.

Design pillars (what “good” means here)

Alternative approaches — and why Dumpling’s model wins

Other ways to “get a scrubbed database” exist. They solve overlapping problems under different trade-offs. Dumpling deliberately picks the static dump niche and pushes that model as far as it can go.

1. Live-database anonymization (connect and UPDATE)

How it works: Point a tool at Postgres/MySQL, scan tables, rewrite rows in place or into a clone.

Strengths of that approach

• Can use live schema introspection and database constraints.
• Feels familiar to DBAs who already operate against a running instance.

Why Dumpling prefers files instead

• Blast radius: a live connection needs credentials and network path to data. A dump file can be processed on an isolated runner with no DB access.
• Repeatability: the same dump + same policy + same seed/profile → reproducible output. Live UPDATE jobs are harder to pin as an artifact.
• Pipeline fit: CI already moves artifacts (backup downloads, pg_dump outputs). File in → file out matches that shape.
• Blast-free rehearsal: you can re-run anonymization until the policy is right without mutating a shared database.

Live anonymizers remain useful when you must scrub an already-restored environment. Dumpling is better when the unit of work is a dump you will restore later.

2. In-database views / dynamic masking

How it works: Keep production data; expose masked views or session-level masking policies to lower-privilege users.

Strengths of that approach

• No separate sanitized copy to store.
• Masking can follow RBAC and change with the live schema.

Why Dumpling still exists beside masking

• Masking does not produce a portable snapshot for contractors, demos, or offline analytics.
• Many “we need a DB” workflows require a full restore with fake-but-shaped data, not a live view into prod.
• Dumpling’s output is a normal SQL dump: restore anywhere, no vendor masking stack required.

Use dynamic masking for day-to-day least privilege. Use Dumpling when you need a sanitized artifact.

3. Ad-hoc scripts (Python/sed/one-off SQL)

How it works: Engineers write custom parsers or regex rewrites for each dump shape.

Strengths of that approach

• Maximum flexibility for one weird table.
• No new tool to learn for a tiny team.

Why a purpose-built tool is better

• SQL dumps are hostile: multi-line INSERTs, COPY … FROM stdin, quoting, \N NULLs, escaped quotes — regexes rot quickly.
• Consistency bugs are silent: mismatched email domains across FK-linked tables break restores and tests in confusing ways. Dumpling’s domain cache exists specifically to prevent that class of bug.
• Safety defaults: scripts default to “do nothing special”; Dumpling defaults to fail-closed, coverage gates, and residual PII scanning.
• Shared policy: TOML in-repo is reviewable in PRs; tribal scripts are not.

Scripts are fine for a one-time migration. They are a poor long-term anonymization platform.

4. “Anonymize after restore” in application code

How it works: Load prod dump into staging, then run app factories / Faker in the ORM to overwrite columns.

Strengths of that approach

• Reuses application domain knowledge and factories.
• Easy to keep formats that your app already validates.

Why Dumpling prefers transform-before-restore

• You still briefly hold raw PII on disk and in the DB during restore — often the compliance pain point.
• App factories rarely cover every table (audit logs, JSON blobs, legacy schemas).
• Dumpling runs before restore, so staging never sees original values for ruled columns.
• Domain mapping works across tables that may not share an ORM model graph.

Application-level faking remains useful for generating synthetic fixtures from scratch. Dumpling is better for scrubbing an existing dump.

5. Heavyweight commercial data platforms

How it works: Enterprise suites with discovery UI, connectors, and policy engines across warehouses and DBs.

Strengths of that approach

• Broad connector coverage and vendor support contracts.
• Often includes discovery/classification UIs for large orgs.

Why Dumpling is often the better fit anyway

• Operational weight: many teams only need “scrub this pg_dump in CI.”
• Inspectable policy: plain TOML + open-source Rust beats opaque rule UIs when auditors ask “what exactly ran?”
• Cost and lock-in: Dumpling is a single binary / pip install dumpling-cli with no control plane.
• Determinism and seals: dump seals and --report give artifact-level provenance without a SaaS sidecar.

If you already run an enterprise platform for cross-system discovery, Dumpling can still be the sharp tool for SQL dump CI gates.

6. Fully random replacement (no domains)

How it works: Every cell gets an independent random value.

Why Dumpling supports this — but makes domains first-class

Independent randomness is fine for isolated columns. It breaks:

• foreign keys (users.id ↔ orders.user_id);
• natural keys reused across tables (email, external IDs);
• human debugging (“why doesn’t this order belong to anyone?”).

Dumpling’s domain option keeps a deterministic map: same input → same output within a named bucket, optionally with unique_within_domain. That is the difference between “technically anonymized” and “still a usable relational database.”

Summary: choosing an approach

Strengths of the Dumpling codebase (easy tour)

These are the concrete capabilities that make the pillars real.

Offline and multi-format input

• PostgreSQL plain SQL, plus auto-detect of custom/directory archives via pg_restore.
• SQLite .dump and SQL Server plain scripts via --format.
• Gzip streamed in-process; ZIP / nested archives handled with careful temp materialization and cleanup.

You sanitize artifacts, not production sockets.

Streaming SQL state machine

SqlStreamProcessor walks modes (Pass, InInsert, InCopy, InCreateTable) so huge VALUES lists and COPY bodies never require loading the whole dump. Quoting and parenthesis depth are tracked so statement boundaries stay correct.

Rich, allowlisted strategies

From cheap clears (null, redact, blank, empty JSON containers) to realistic fakes (email, name, payment_card, faker, date/time fuzz), plus conditional keep under column_cases when some rows must retain the original value. Config only carries string identifiers; new generators ship in Dumpling releases (faker_dispatch), never as eval’d user code.

Referential integrity via domains

Optional domain + in-memory mapping cache (and optional uniqueness retries) keeps related columns coherent after rewrite. SQL NULL stays NULL — no fabricated FK targets for missing values.

Row filters and conditional cases

• row_filters retain/delete whole rows before transforms.
• Optional [[row_filters."<parent>".cascade]] links keep child rows only when their FK matches a retained parent PK (explicit, shallow, parent-before-child in the dump)—avoids orphan-trimmed graphs without live FK discovery.
• column_cases apply first-match-wins strategies per row (including keep for allowlist exceptions, or cases-only scrub with keep-by-omission).
• Predicate operators include positive and negating forms (not_like / not_ilike / not_regex / not_iregex); invalid Rust regex patterns fail closed at config load.
• JSON path rules (dot or Django-style __) reach into json / jsonb text, including list-of-object shapes.

Schema-aware string lengths

CREATE TABLE parsing extracts varchar(N) / char(N) limits so generated strings truncate to fit — fewer restore failures from oversized fakes.

Safety and evidence built in

• Fail-closed config discovery.
• Dump seal comment fingerprinting policy + transform options (skippable with --no-seal for pipes).
• --report JSON audit sidecar (hashes, flags, coverage, scan outcomes).
• --strict-coverage against [sensitive_columns].
• Residual PII scan (email / SSN / PAN / token patterns) with fail thresholds.
• lint-policy for unsalted hashes, inconsistent domains, uncovered sensitive columns, empty rule tables, and invalid regex predicates.
• --security-profile hardened: OS CSPRNG + HMAC constructions when adversarial risk is in scope.

Contributor-friendly engineering

• Single Rust crate, inline tests, clippy/fmt as CI gates.
• Clear module boundaries (settings → sql → transform / filter / scan / report).
• Docs as mdBook with PR build checks so the rationale stays next to the code.

What we deliberately do not do

These omissions are intentional, not unfinished work:

How the pieces fit at runtime

  dump file / archive / stdin
            │
            ▼
   resolve input (detect format, decompress, pg_restore if needed)
            │
            ▼
   load & validate TOML policy (secrets → env, fail closed if missing)
            │
            ▼
   stream lines → parse INSERT/COPY rows
            │
            ├─ row filters (retain / delete)
            ├─ column_cases (first match) else rules
            ├─ apply strategy (random or domain-deterministic)
            └─ render cells (respect quoting + varchar limits)
            │
            ▼
   optional residual scan on the write path
            │
            ▼
   dump seal + sanitized SQL out   and/or   --report JSON sidecar

Each stage exists because an alternative (load-all parsing, silent missing config, independent random FKs, “trust the rewrite with no scan”) failed real operational needs.

Further reading

• Getting started — shortest path to a first run
• Configuration guide — seals, hardened profile, reports, strategies
• CI guardrails and policy linting — gates and audit evidence
• Repository README.md — strategy catalog and usage cheat sheet
• AGENTS.md — architecture notes for contributors

Configuration guide

Dump format

Use --format to declare the SQL dialect of your input file:

Example:

dumpling --format sqlite -i data.db.sql -o anonymized.sql
dumpling --format mssql  -i backup.sql  -o anonymized.sql

PostgreSQL archives and compressed inputs

Heroku PGBackups and many pipelines ship pg_dump custom format (-Fc), directory-format dumps, or gzip/ZIP-wrapped files. Dumpling’s SQL engine still expects plain text at the parser; anything else is normalized first.

Custom-format and directory dumps (auto-detected)

With --format postgres (default), Dumpling detects:

• Custom-format files (magic PGDMP at the start of the file), and
• Directory-format folders (a toc.dat beside table blobs),

then runs pg_restore -f - (script to stdout inside the process — no database) and pipes the result through the same anonymizer as a normal plain-SQL file. Detection from --input is automatic.

Requirements: PostgreSQL client tools on PATH (pg_restore), or --pg-restore-path.

Extra pg_restore arguments:

• CLI: --pg-restore-arg (repeatable), e.g. --pg-restore-arg=--no-owner --pg-restore-arg=--no-acl
• Config (optional): [pg_restore] — CLI overrides these when you pass path or args:

[pg_restore]
path = "/usr/bin/pg_restore"
args = ["--no-owner", "--no-acl"]

Gzip and ZIP wrappers

• Gzip (.gz) whose decompressed payload is plain SQL: decompressed in-process (streamed); no temporary dump file.
• ZIP containing a single dump file (or a single .sql when multiple files exist), gzip wrapping PGDMP, or gzip wrapping an inner ZIP: Dumpling writes under the system temp directory and removes those paths when the run completes (including after errors — cleanup runs on drop).

--in-place is rejected when Dumpling had to materialize a temp file for compression or when the resolved input is a PostgreSQL archive decoded via pg_restore (use --output or stdout).

Keeping inputs and --check

After a fully successful run, Dumpling removes the --input archive path (single file or directory-format folder) by default. To keep it:

• --keep-original, or
• keep_original = true at the top level of .dumplingconf / [tool.dumpling] (merged with CLI; --keep-original cannot be used with --in-place).

--check with a PostgreSQL archive requires an effective keep-original (CLI or config); otherwise the default deletion would remove the dump before you iterate on policy.

Examples (e.g. after heroku pg:backups:download):

dumpling -i latest.dump -c .dumplingconf -o anonymized.sql

Dry run while keeping the downloaded file:

dumpling --keep-original --check -i latest.dump -c .dumplingconf

Configuration sources

Configuration can be loaded from:

1. --config <path> (highest precedence)
2. .dumplingconf in the current working directory
3. [tool.dumpling] in pyproject.toml

If no configuration is found, Dumpling fails closed by default and exits non-zero. Error output includes every checked location. If you intentionally want a no-op run, pass --allow-noop.

Dump seal (on by default)

Successful runs that write output prefix the stream with one SQL comment:

-- dumpling-seal: v=3 version=<semver> profile=<standard|hardened> sha256=<64 hex chars>

The sha256 fingerprints Dumpling version, the active security profile, a stable encoding of the resolved policy (rules, row filters, column cases, sensitive columns, output scan, global salt), and runtime options that affect transforms: --format and the effective --seed / DUMPLING_SEED in standard profile (null in hardened, where seeds are ignored).

Incoming seals. If the input already starts with a matching seal, Dumpling copies the rest of the file through unchanged. A seal that does not match (stale policy, different flags, or older v=) is dropped and the dump is re-processed so you do not get two seal lines. --strict-coverage cannot be combined with a matching seal (table definitions are not scanned in passthrough mode).

--check writes no output, so it emits no seal line.

--no-seal

Skip writing the comment. Use this for stdin/stdout pipelines where a leading SQL comment is unwanted:

cat dump.sql | dumpling --no-seal --report report.json > sanitized.sql

Incoming seal lines are still recognized (match → pass the body through; stale → strip and re-process). --report still records the policy fingerprint as seal_sha256. See JSON report (audit sidecar) and Audit evidence.

JSON report (audit sidecar)

--report <file> writes a JSON sidecar for the run. Use it as compliance evidence next to (or instead of) a dump-seal comment.

Coverage arrays (sensitive_columns_*), output_scan, per-table counts, and change events are in the same file. input_sha256 is over the SQL stream Dumpling actually processed (decoded pg_restore output, decompressed gzip, and so on), not necessarily the raw archive on disk.

See Audit evidence for archival and verification examples.

Hardened security profile

For adversarial risk environments — where an internal or external actor may have partial auxiliary data — use --security-profile hardened:

dumpling --security-profile hardened -i dump.sql -o sanitized.sql

What changes in hardened mode

Why this matters

• Non-predictable output: xorshift64* is seeded from system time, which is guessable. The OS CSPRNG cannot be predicted from timing alone.
• Proper keyed hashing: SHA-256(key || data) is vulnerable to length-extension attacks and weak as a MAC. HMAC-SHA-256 uses the salt as a genuine cryptographic key, providing provable PRF security.
• Domain separation: HMAC construction ensures outputs from one salt/key cannot be confused with another.

Key management guidance

Configure a per-environment secret via an env-backed reference to prevent key leakage:

# .dumplingconf
salt = "${DUMPLING_HMAC_KEY}"

[rules."public.users"]
ssn = { strategy = "hash", as_string = true }
email = { strategy = "email", domain = "users" }

export DUMPLING_HMAC_KEY="$(openssl rand -base64 32)"
dumpling --security-profile hardened -i dump.sql -o sanitized.sql

Key rotation: Changing DUMPLING_HMAC_KEY will produce entirely different pseudonyms for all salted/domain-mapped columns. If you rely on referential consistency across separately-processed dumps (e.g., snapshots over time), keep the same key or re-anonymize all related dumps together. Rotate keys when:

• A key may have been compromised.
• You intentionally want to break prior referential linkability.

Report metadata

The JSON --report always includes the active security profile ("standard" or "hardened"). Full sidecar fields are documented under JSON report (audit sidecar).

{
  "security_profile": "hardened",
  "total_rows_processed": 1000
}

Faker strategy and the fake crate

When you use strategy = "faker" with faker = "module::Type", those names align with the Rust fake crate’s faker modules (for example name::FirstName ↔ fake::faker::name::raw::FirstName). Use the upstream docs to discover available generators and options:

• docs.rs — fake (crate overview)
• docs.rs — fake::faker (all faker submodules)
• GitHub — cksac/fake-rs (source + README)

Dumpling only exposes a subset wired in src/faker_dispatch.rs; unsupported module::Type pairs fail at config load.

Anonymization strategies

Strategy names and per-strategy options (min, scale, as_string, faker, …) are documented in the repository README under Anonymization strategies (each strategy lists only the keys it accepts, plus Choosing a strategy for when to prefer cheap vs realistic transforms, and Cross-cutting options for domain, unique_within_domain, and as_string).

Highlights that sit alongside the older clears and fakes:

• blank / empty_array / empty_object — cheap clears that preserve SQL NULL when the source is NULL (blank → ''; the empty JSON strategies emit unquoted [] / {}).
• keep — leave the cell unchanged; only under [column_cases] (a default [rules] keep is rejected). Pair with a scrubbing default for allowlists.
• decimal / payment_card — bounded numeric shapes and Luhn-valid synthetic PANs.

Conditional column_cases

For each cell, Dumpling evaluates column_cases in declaration order (first matching when wins), then falls back to a [rules] default when present. If nothing matches and there is no default / JSON path rule, the cell is left unchanged (keep-by-omission). That pattern is supported for selective anonymization (scrub matching rows; keep the rest). For the inverse shape (default scrub + allowlist exceptions), use the explicit keep strategy under column_cases. See the README section Conditional per-column cases for selection semantics and both cookbook examples.

The sections below expand on row-filter predicates, JSON path rules, and secret references.

Baseline config template

salt = "${DUMPLING_GLOBAL_SALT}"

[rules."public.users"]
email = { strategy = "hash", salt = "${env:DUMPLING_USERS_EMAIL_SALT}", as_string = true }
full_name = { strategy = "faker", faker = "name::Name" }
notes = { strategy = "blank" }

[sensitive_columns]
"public.users" = ["employee_number", "tax_id"]

[output_scan]
enabled_categories = ["email", "ssn", "pan", "token"]
default_threshold = 0
default_severity = "high"
fail_on_severity = "low"
sample_limit_per_category = 5

[output_scan.thresholds]
email = 0
ssn = 0
pan = 0
token = 0

[output_scan.severities]
email = "medium"
ssn = "high"
pan = "critical"
token = "high"

[row_filters."public.users"]
retain = [
  { column = "country", op = "eq", value = "US" },
  { column = "profile.plan", op = "eq", value = "gold" }
]
delete = [
  { column = "is_admin", op = "eq", value = "true" },
  { column = "devices__platform", op = "eq", value = "android" }
]

# Default scrub + keep exceptions (staff allowlist)
[[column_cases."public.users".email]]
when.any = [
  { column = "email", op = "ilike", value = "%@myco.com" },
]
strategy = { strategy = "keep" }

Secret references

Dumpling supports secret substitution in string config fields using two providers:

Example using both providers:

salt = "${DUMPLING_GLOBAL_SALT}"

[rules."public.users"]
ssn   = { strategy = "hash", salt = "${env:DUMPLING_USERS_SSN_SALT}" }
email = { strategy = "hash", salt = "${file:/run/secrets/dumpling_email_salt}" }

Behavior:

• Missing env references fail fast at startup with a non-zero exit and an actionable error including the config path.
• Missing or empty file references fail fast with a non-zero exit and an actionable error.
• Plaintext salt values are accepted for backward compatibility, but Dumpling prints a warning because plaintext secrets are insecure.
• Unknown providers fail startup with a list of supported providers.

Environment-variable secrets (CI / local dev)

# local development
export DUMPLING_GLOBAL_SALT='local-dev-salt'
export DUMPLING_USERS_EMAIL_SALT='users-email-salt'
dumpling --input dump.sql --check

# CI environment (values injected from your secret manager)
export DUMPLING_GLOBAL_SALT="$CI_DUMPLING_GLOBAL_SALT"
export DUMPLING_USERS_EMAIL_SALT="$CI_DUMPLING_USERS_EMAIL_SALT"
dumpling --input dump.sql --check --strict-coverage --report coverage.json

File-mounted secrets (Docker / Kubernetes)

The file: provider reads the secret value from a file on disk and trims trailing newlines. This is the natural format for Docker Swarm secrets (/run/secrets/<name>), Kubernetes mounted secrets, and HashiCorp Vault Agent injected files.

Docker Swarm — declare a secret and mount it into the service:

# docker-compose.yml
secrets:
  dumpling_hmac_key:
    external: true

services:
  anonymizer:
    image: your-image
    secrets:
      - dumpling_hmac_key
    environment:
      - DUMPLING_CONFIG=/app/.dumplingconf

# .dumplingconf
salt = "${file:/run/secrets/dumpling_hmac_key}"

Kubernetes — mount a Secret as a volume:

# deployment.yaml (excerpt)
volumes:
  - name: dumpling-secrets
    secret:
      secretName: dumpling-keys
volumeMounts:
  - name: dumpling-secrets
    mountPath: /run/secrets
    readOnly: true

# .dumplingconf
salt = "${file:/run/secrets/hmac_key}"

HashiCorp Vault Agent — inject secrets as files using the template stanza:

# vault-agent.hcl (excerpt)
template {
  contents     = "{{ with secret \"secret/dumpling\" }}{{ .Data.data.hmac_key }}{{ end }}"
  destination  = "/run/secrets/dumpling_hmac_key"
}

# .dumplingconf
salt = "${file:/run/secrets/dumpling_hmac_key}"

Row filters and predicates

[row_filters."table"] can retain (OR: keep only if at least one predicate matches) and delete (drop if any predicate matches, evaluated after retain). The same predicate operators appear in column_cases when.any / when.all.

Cascade retain (related rows)

Optional [[row_filters."<parent>".cascade]] entries keep child rows whose foreign key matches a retained parent primary key:

[row_filters."public.listing_order"]
retain = [{ column = "status", op = "eq", value = "open" }]

[[row_filters."public.listing_order".cascade]]
child_table = "public.listing_orderitem"
child_fk = "order_id"
parent_pk = "id"

Parent table data must appear before child data in the dump. Cascade filtering is additive to any local retain/delete on the child. NULL child FKs are dropped. Full FK-graph discovery and multi-hop cascades are out of scope.

format = "datetime" / "date" also applies to eq / neq. Threshold strings are validated at config load.

Invalid or unsupported regex patterns fail config load and surface as invalid-regex-predicate in dumpling lint-policy. For “scrub unless allowlisted domain” policies, prefer a default scrub plus a keep case, or positive not_ilike / not_iregex cases — see the README Conditional per-column cases and Row filtering sections.

Nested JSON targeting is supported in predicate column values via either:

• dot notation (payload.profile.tier)
• Django-style separators (payload__profile__tier)

When a JSON path traverses an array, Dumpling checks each element (useful for list-of-dicts JSON structures).

JSON path rules (json / jsonb columns)

You can anonymise values inside a text column that holds JSON using the same path syntax as row-filter predicates, but on [rules] keys:

• Dot notation: "payload.profile.email" = { strategy = "email", domain = "orders_email", as_string = true }
• Django-style: "payload__profile__email" = { strategy = "hash", salt = "${env:ORDER_SECRET_SALT}", as_string = true }

The part before the first dot or __ is the SQL column name; the rest is the path inside the parsed JSON document. Use quoted keys in TOML when the name contains dots. For a given table, you can use either path-level rules for a column or one whole-column rule for that column’s base name, not both (Dumpling rejects the conflict at startup). If a path is missing in a given row, that rule is skipped for that row. When only path rules apply (no whole-column rule), the rest of the JSON is left unchanged. Path rules are applied in longest-path-first order. column_cases still match the SQL column name only; use when predicates with nested column paths to branch on JSON content.

On JSON path leaves that must stay typed empty containers, use empty_array / empty_object instead of null or blank.

Safety recommendations

• Prefer deterministic runs in CI by passing --seed (or DUMPLING_SEED).
• Keep fail-closed behavior enabled in CI/CD; avoid --allow-noop unless a no-op run is explicitly intended.
• Treat new or changed anonymization rules as code changes and require review.
• Keep table/column names lowercase in config to avoid case-mismatch surprises.
• table_options are no longer supported; define explicit rules and optional conditional column_cases instead.
• Use --strict-coverage --report <file> --check in CI so uncovered sensitive columns fail the build.

Strict sensitive coverage

--strict-coverage enforces explicit policy coverage for sensitive columns.

• Sensitive columns are detected by:
  1. built-in column-name patterns, and
  2. explicit per-table lists under [sensitive_columns].
• A sensitive column is considered covered only if it has an explicit rules or column_cases entry (including JSON path rules whose base name is that column, e.g. payload.x.y covers payload).
• If uncovered sensitive columns are found, Dumpling exits non-zero.

When --report is enabled, coverage fields are added to JSON output:

• sensitive_columns_detected
• sensitive_columns_covered
• sensitive_columns_uncovered

The same file is the audit sidecar (dumpling_version, checksums, seal_sha256, gate flags, scan outcomes). run_id and started_at differ on every invocation; fingerprint fields are deterministic. With --no-seal the dump has no seal line, but seal_sha256 is still in the report. See Audit evidence.

Example CI gate:

dumpling --input dump.sql --check --strict-coverage --report coverage.json

Residual output scanning

Enable output scanning with:

dumpling --input dump.sql --scan-output --report scan_report.json

Add fail gates with:

dumpling --input dump.sql --check --scan-output --fail-on-findings --report scan_report.json

Output scanning inspects transformed output for common sensitive categories:

• email
• ssn
• pan (Luhn-validated card-like numbers)
• token (common secret/token formats)

When --report is set, report JSON includes an output_scan object with per-category:

• category
• count
• threshold
• severity
• sample_locations (line + column + snippet where available)

CI guardrails and policy linting

Dumpling ships with a built-in policy linter (lint-policy) and a reference GitHub Actions workflow so you can catch anonymization policy regressions in pull requests before they reach production.

dumpling lint-policy

dumpling lint-policy                          # auto-discover config
dumpling lint-policy --config .dumplingconf   # explicit config path
dumpling lint-policy --allow-noop             # treat missing config as empty (no violations)

The command loads your configuration, runs a set of policy checks, prints any violations to stderr, and exits:

Checks performed

Recommended CI setup

Minimal (policy lint only)

# .github/workflows/policy-lint.yml
name: Policy Lint

on:
  pull_request:
  push:
    branches: [main]

jobs:
  policy-lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo build --release --locked
      - run: ./target/release/dumpling lint-policy

This single job will block merges whenever a policy violation is introduced.

Production-ready: lint + strict coverage + PII scan

Combine lint-policy with Dumpling's other CI gates for defence in depth:

name: Anonymization CI

on:
  pull_request:
  push:
    branches: [main]

jobs:
  policy-lint:
    name: Policy lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo build --release --locked
      - name: Lint anonymization policy
        run: ./target/release/dumpling lint-policy

  anonymize-and-scan:
    name: Anonymize + residual PII scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - uses: Swatinem/rust-cache@v2
      - run: cargo build --release --locked

      - name: Anonymize with strict coverage enforcement
        run: |
          ./target/release/dumpling \
            --strict-coverage \
            --scan-output \
            --fail-on-findings \
            --report report.json \
            -i dump.sql \
            -o sanitized.sql

      - name: Upload anonymization report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: anonymization-report
          path: report.json

Gating on report diff against a baseline

To detect increases in risk findings across PRs, store a baseline report as a CI artifact on your main branch and compare against it in PRs:

- name: Download baseline report
  uses: dawidd6/action-download-artifact@v6
  with:
    workflow: ci.yml
    branch: main
    name: anonymization-report
    path: baseline/
  continue-on-error: true   # first run has no baseline yet

- name: Fail if findings increased
  run: |
    BASELINE=$(jq '.output_scan.total_findings // 0' baseline/report.json 2>/dev/null || echo 0)
    CURRENT=$(jq '.output_scan.total_findings' report.json)
    echo "Baseline findings: $BASELINE  Current findings: $CURRENT"
    if [ "$CURRENT" -gt "$BASELINE" ]; then
      echo "ERROR: residual PII findings increased from $BASELINE to $CURRENT"
      exit 1
    fi

Audit evidence

Keep report.json next to the sanitized dump (or keep the report alone when using --no-seal). Together they answer what policy and Dumpling version transformed which input, without reconstructing the run from CI logs.

Default (seal on the dump). The first line of the SQL output and seal_sha256 in the sidecar share the same digest:

SEAL=$(sed -n '1s/.*sha256=//p' sanitized.sql | tr -d '[:space:]')
REPORT=$(jq -r '.seal_sha256' report.json)
test -n "$SEAL" && test "$SEAL" = "$REPORT"

--no-seal (streaming / no comment on the SQL). There is no seal line to grep. Confirm the sidecar still has a 64-character digest and that flags.no_seal is true:

jq -e '.flags.no_seal == true and (.seal_sha256 | length == 64)' report.json

Useful fields for a compliance review:

input_sha256 is over the SQL byte stream Dumpling actually processed (decoded pg_restore output, decompressed gzip, and so on), not necessarily the raw archive file on disk.

Example production invocation (file output with dump seal):

dumpling \
  --strict-coverage \
  --scan-output \
  --fail-on-findings \
  --report report.json \
  -i dump.sql \
  -o sanitized.sql

Streaming without a dump-seal comment:

cat dump.sql | dumpling --no-seal --report report.json > sanitized.sql

Archive report.json with the sanitized SQL. To confirm a later re-run used the same policy, compare seal_sha256 (and config_sha256); do not expect run_id or started_at to match. Full field notes: JSON report (audit sidecar).

Tips

• Run dumpling lint-policy locally before opening a PR to catch violations early: cargo run -- lint-policy.
• Treat error-severity violations as mandatory fixes; warning-severity violations are advisory but should be reviewed.
• If you intentionally use hash without a salt (e.g. for non-sensitive low-cardinality fields), add a salt = "${ENV_VAR}" at the global level to suppress the unsalted-hash warning globally.
• In hardened security profile environments, a global salt is required anyway (--security-profile hardened will error without it), so unsalted-hash warnings become informational.
• Prefer keep under column_cases (or not_ilike / not_iregex) for allowlist-style exceptions; do not rely on regex look-around — it is rejected at config load and by invalid-regex-predicate.

Release process

This project uses tag-driven releases.

Release policy

• Versioning follows Semantic Versioning (MAJOR.MINOR.PATCH).
• Every release is tied to an immutable git tag (vX.Y.Z).
• The release workflow runs quality checks before publishing artifacts.

Maintainer checklist

1. Ensure main is green in CI.

2. Update Cargo.toml and pyproject.toml versions and CHANGELOG.md.

3. Open and merge a release preparation PR.

4. Create and push a tag from main:

   git tag -a vX.Y.Z -m "Release vX.Y.Z"
   git push origin vX.Y.Z
   

5. Verify the Release GitHub Actions workflow passes.

6. Validate uploaded artifacts and checksums from the GitHub Release page.

7. Announce the release with upgrade notes and rollback guidance.

Python package publishing (PyPI/TestPyPI)

Dumpling is configured for Python packaging via maturin (pyproject.toml). The Python distribution name is dumpling-cli, while the installed CLI command is still dumpling.

The Publish GitHub Actions workflow (.github/workflows/publish.yml) builds cross-platform wheels and an sdist, then publishes:

• Automatically to PyPI when a v*.*.* tag is pushed.
• Manually to PyPI or TestPyPI via workflow_dispatch.

Environment setup

• Configure a GitHub environment named pypi for trusted publishing.
• Optionally configure testpypi for manual pre-release validation.

Local dry-run commands

maturin build --release
maturin sdist

Rollback guidance

• If a release is faulty, create a new patch release (for example v1.2.4) that reverts or fixes the issue.
• Avoid deleting published tags; treat tags as immutable history.