We redact API keys before they reach the LLM. Here's how.

The problem in 100 words

If you use a coding agent, your API keys have probably already been sent to a model provider. Not because you pasted them. Because the agent ran cat .env, read a docker-compose.yml with a database URL in it, or captured a stack trace that echoed an Authorization header. Tool output goes back into the conversation, and the conversation is the request body.

Once it leaves, a prompt is a record on someone else's infrastructure. Retention windows apply. A training opt-out is not deletion. So we built a layer that swaps secrets for placeholders before the request leaves the process. This post is what it does, what it measured, and where it stops.

Why "I'm careful" doesn't hold

We used to think of secret leaks as a copy-paste problem. Then we watched what agents actually do in a session. Three paths, all from our own code and changelog:

1. Tool results are the richest source. On the Anthropic wire, tool_result.content carries shell output, file contents and the output of env. The agent didn't ask for the key. It asked for the config, and the key came along. We flag this field as the highest-yield location in wire_anthropic.py, for good reason.

2. Editor completions. Our inline completion endpoint sent the surrounding buffer to the model. Until 2026-08-19, that buffer was not scrubbed. If your cursor was two lines below STRIPE_SECRET=, the key was in the request. That leak is fixed in completions_routes.py and documented there as a leak, not as a feature.

3. Our own repo. During a measurement run of the new detection rules against UltraNOS itself, every hit was a real secret and there were zero false positives. The hits sat in project files that a session reads automatically, which is exactly the point: nobody pasted them, they were simply there.

None of these are exotic. The OWASP Top 10 for LLM applications lists sensitive information disclosure as a standing category, and Truffle Security has written about models teaching developers to hardcode keys in the first place. What we didn't find anywhere was a description of a concrete operational path that replaces secrets before sending. So here is ours.

What cloak does, step by step

cloak is a Python package with no runtime dependencies. In UltraNOS it runs in-process, immediately before the HTTP request is serialized. There is a bundled proxy called cloakd; we deliberately don't use it, because an in-process gate can fail closed in a way a sidecar can't.

Detection

Five categories: secret, secret_block, credential, email, person. All are on by default.

Detection runs as a fixed eight-stage pipeline, and the order is the precedence: existing placeholders, allowlist, key blocks, provider prefixes, structural credentials, vault values, email, and finally context-free forms. As of 2026-09-07 the rule set counts:

Rule group	Count	Examples
Provider prefixes	27	sk-ant-…, ghp_…, glpat-…, xoxb-…, AKIA…, sk_live_…
Structural credentials	13	URL userinfo, Authorization headers, KEY=value, connection strings, .netrc, curl -u, JWTs
Key blocks	2	PEM, PuTTY private keys
Allowlist reservations	8	PEM public keys, SSH pubkeys, digests, data: URIs, SCM remotes, passwd lines, port mappings
Context-free forms	6	mysql -p, redis-cli -a, --password, openssl -passin, opaque high-entropy tokens
Email	1	email.addr

That's 57 rule specs, 49 of which replace and 8 of which reserve, plus a vault detector for values you add by hand. The prefix rules for nano-gpt and Requesty landed on 2026-09-03.

Two design constraints matter more than the count. First, every regex uses bounded quantifiers. No .*. This is a ReDoS brake, and cloak doctor measures each rule against a corpus with a 2,000 ms budget per scan. Second, entropy is a gate, not a detector. Shannon bits per character are only checked after a context anchor matched, with thresholds between 2.0 and 4.0 depending on the rule. The only context-free rules live in one file with their own justification and their own negative corpus, because that's the place false positives come from.

On the false-positive side there are explicit deny lists: documentation example values behind a real prefix, pure placeholders like <token> or $VAR, weak passwords, and code references. When two matches overlap, the more sensitive category wins over the longer match. That last rule was a bug fix on 2026-08-19; before it, the longer match won regardless of category.

Placeholders

The model sees CLOAK_SECRET_…, CLOAK_CREDENTIAL_…, CLOAK_EMAIL_…, CLOAK_PERSON_… or CLOAK_SECRET_BLOCK_…, each followed by twelve hex characters. The longest token is 31 characters.

The ID is random, from secrets.token_hex, not a hash of the value. Stability comes from the vault: it keeps an index HMAC over pepper, category and value, finds the existing entry, and hands back the same placeholder. So the same key gets the same placeholder across turns and across sessions, process-wide, without the placeholder revealing anything about the key.

Because tool use is on by default, the vault stores the ciphertext too, AES-256-GCM with a stdlib fallback. This is worth stating plainly: the "one-way storage" promise in the package README only holds if you disable use. With defaults, secrets are stored encrypted, because the next step needs them.

Re-substitution at tool calls

This is the part that makes the whole thing usable rather than merely safe. When the model calls a tool with a placeholder as an argument, say curl -H "Authorization: Bearer CLOAK_SECRET_…", the gate resolves the placeholder immediately before execution. The tool gets the real value. The conversation history that goes back to the model keeps the placeholder. The model gets the placeholder; the shell gets the secret.

And it runs every round. Each tool result is scrubbed again before the next request, so a key read via read_file or dumped by Bash is replaced, provided a pattern matches.

Responses stream back through a reverse translation that holds back at most 32 characters at chunk boundaries, in case a placeholder is split across two chunks. Secrets stay as placeholders in what a human sees. Emails and person names are shown in clear text again, which is a display decision, not a transport one.

The model also gets an English notice, [cloak] Privacy layer active …, with three rules. Only in enforce, only when the body actually carries a placeholder, and you can turn it off with CLOAK_NOTICE=0.

Fail-closed, and why that's the right default

The backend flag ULTRANOS_FEATURE_CLOAK defaults to off. When it is on and the package is missing, or the runtime won't open, the gate raises CloakNichtVerfuegbar and the turn fails with a readable error. No provider catches it. On the CLI path, the CLI process is never even started. For editor completions, you get no completion rather than an unscrubbed one.

We learned this the hard way. The first version, delivered by an agent that reported "library done", defaulted to audit. Audit mode counts and passes the text through byte-identical with HTTP 200. That is worse than no proxy, because it looks like protection. We rolled it back on 2026-08-19 and rebuilt the wire layer the same day. enforce has been the default since, with or without a config file.

enforce also has hard limits: a node over 1 MiB raises LimitExceeded, a detector exception propagates instead of being swallowed, unknown config keys are errors, and file permissions on the key file and store are checked but never silently repaired.

Every path, with dates

The gate sits in every provider we ship:

• Anthropic API wire (2026-08-19), including each tool round. Scanned: system, every messages[].content text, tool_use.input, tool_result.content, tools[].description, metadata.
• OpenAI wire (2026-08-19). Scanned: message content for all roles including role="tool", tool_calls[].function.arguments (parsed as JSON and reserialized), tool descriptions.
• Ollama (2026-08-24). It was the last HTTP provider without the gate.
• Claude Code CLI paths (2026-08-31): claude and claude-instanz. Prompt and system prompt are scrubbed, including on --resume.
• Custom providers inherit from the Anthropic and OpenAI classes, so they get the gate for free.

Live proof from production on 2026-08-19, on the OpenAI wire via a third-party gateway: we sent a GitHub test key, the model answered with the placeholder, and the per-turn log read treffer {secret/key.github: 2}, platzhalter 1, ungescannt 0. That log line exists for every turn, and it never contains clear text.

Verify, don't trust us

GET /api/health on any UltraNOS instance returns a cloak field with three possible shapes: {aktiv: false}, {aktiv: true, bereit: false, fehler}, or {aktiv, bereit: true, profil, kategorien, ersetzt}. Note that bereit: false is also the lazy-init state before the first turn, not necessarily a failure. The API section describes the endpoint.

Measured on 2026-09-07 against the hosted instance:

cloak: {aktiv: true, bereit: true, profil: "enforce",
        kategorien: [credential, email, person, secret, secret_block],
        ersetzt: true}

What a request looks like from the provider's side, with a made-up key shape:

You:      Deploy with ANTHROPIC_API_KEY=sk-ant-… and mail ops@example.org when done.
Provider: Deploy with ANTHROPIC_API_KEY=CLOAK_SECRET_… and mail CLOAK_EMAIL_… when done.

There is also a per-turn audit copy, GET /api/privacy/transparenz/{session}, listing every replacement with category, rule ID, placeholder, length, a two-character fingerprint, location and context, next to the scrubbed prompt and the raw response. The route is mounted and gated behind the feature flag since 2026-09-03. It is not wired to a producer yet. The functions that would fill it are called only from tests. So today the route returns empty lists or 404, and the frontend hides the section. We're naming it as an open construction site, not offering it as evidence. The only evidence today is the health endpoint and the per-turn log.

Test coverage, for the record: the cloak suite is 128 tests (73 core, 26 for the context-free rules, 29 for the wire layer). On the backend side: 17 for the CLI paths, 4 for the tool-loop remainder, 32 plus 74 for the audit copy and its leak checks, 11 for the routes. Latency of the scrubbing path itself is not measured. We have the rule budget and a 0.035 s figure for listing a full audit ring, and that's it.

What cloak does NOT do

This is the section HN reads first, so we put the sharpest edge at the top.

The hosted instance runs the CLI path today. As of 2026-09-07, the active providers on ultranos.de are two claude-instanz accounts and the operator-paid access. All three go through Claude Code's CLI. On that path, only the prompt and system prompt are scrubbed. The CLI executes tools itself, over its own connection to Anthropic, and its tool results do not pass through cloak. A placeholder inside a CLI tool call is not resolved either; it just fails harmlessly. The full coverage described above, tool results every round and re-substitution at execution, applies to the API paths you get when you bring your own Anthropic, OpenAI or Ollama key. That's why our marketing copy says "on every path" and the absolute "never" phrasing is blocked by a frontend test.

The rest of the list:

• Pattern-based. A secret without a known prefix, without an assignment context, and with entropy below 4.0 bits per character passes through. A password on its own line with no anchor is not recognized.
• Not scanned: images, PDFs, base64 payloads, thinking and redacted_thinking blocks (signed, so untouchable; we count them as unscanned_leaves), stop_sequences, tool names and input schemas.
• Names in running text are not detected. A naive pattern produced 594 false hits in our own docs. Names only get replaced if you add them to the vault with cloak add person.
• Phone numbers, IBANs, card numbers produce findings but no replacement. Hostnames, IPs and paths have a detector that isn't in the pipeline.
• Storage: emails and person names sit in clear text in the message store because they are revealed for display. Secrets show as placeholders after a reload.
• Memory extraction has its own coarse net and explicitly does not replace cloak. Sub-agents and cross-chat go through the same providers, which should mean they're covered, but we haven't written a test that proves it.
• No sync, no export, no TTL. Lose the key file and the vault entries are unreadable.
• The package README still says the default is audit and that tool_use.input isn't resolved. Both are stale.

Who this isn't for

If you run local models with no network path, you have nothing to scrub. If your organization already has an enterprise contract with zero-retention terms and audited that, your lever is the contract, not a regex layer. And if your threat model includes secrets in screenshots or PDFs, cloak won't help, by design.

Where it fits: you use hosted models, you let agents read your filesystem, and you'd rather the shell had the key than the provider. Self-hosting is optional and available on request (the repo is not public); the same gate runs on the hosted instance at ultranos.de and on your own box. The German original of this post is at /blog/api-keys-im-prompt-schuetzen/, and the rest of the blog at /blog/.

FAQ

Does the model ever see the key?

On the API paths, not if a pattern matches: the placeholder goes out, the real value is resolved only at tool execution, and the history keeps the placeholder. On the CLI path, the prompt is scrubbed but the CLI's own tool results are not. And if no pattern matches, the value goes through in clear text. We don't claim "never".

What about passwords in terminal output?

On the API paths, every tool result is scrubbed before the next request, so mysql -p…, redis-cli -a …, --password …, connection strings and Authorization headers in captured output are replaced. A bare password with no surrounding context is not recognized. On the CLI path, terminal output from the CLI's own tools isn't scanned at all.

Does it work with my own API key?

Yes, and that's where coverage is best. Bring an Anthropic, OpenAI-compatible or Ollama endpoint and you get the full wire-layer treatment, including tool rounds and re-substitution. Your provider key itself is configuration, not prompt content, so it isn't part of what gets scrubbed.

Is this GDPR compliant?

That's not something a vendor certifies about itself, and we won't. What we can state: the server is in Nuremberg, a data processing agreement under Art. 28 GDPR is available, and API keys, credentials and emails are replaced before the request leaves, with the limits listed above. Prompts still go to whichever model provider you chose. Check their retention terms in the Anthropic docs or OpenAI docs.

Can I self-host it?

Self-hosting UltraNOS is optional and available on request (the repo is not public); the cloak gate is part of the backend, so yes. Set ULTRANOS_FEATURE_CLOAK=1, install the package, and check /api/health for bereit: true and profil: "enforce". If the package is missing, turns fail loudly rather than sending clear text, which is the behaviour you want to see on first boot.

Your keys stay yours — the rest runs in the browser.

UltraNOS is the agentic web IDE: parallel AI agents, editor, git and terminal. cloak replaces API keys and passwords before a prompt reaches the provider. Hosted in Germany, 14 days free.

Try 14 days free →