Files
98736457bd feat(web-search): add AI/ML API Toolbox as an opt-in backend (#1189)
* feat(aimlapi): shared service key and attribution for aimlapi.com services

Adds the pieces the Toolbox and MCP integrations share:

- internal/aimlapi: X-AIMLAPI-Source (agent/zero), the fixed Toolbox and
  MCP URLs, and an exact-hostname check (https, default port, no
  userinfo) for api/tools/mcp.aimlapi.com before the key or attribution
  headers are attached. The partner id still comes from ResolvePartnerID.
- internal/config: AIMLAPIServiceKey picks the key from the aimlapi
  profile (or AIMLAPI_API_KEY without one), reading the credential
  store, and withholds it when any aimlapi-keyed profile points at a
  proxy base URL or AIMLAPI_INFERENCE_URL is overridden. Keys that are
  not a single token are refused.

* feat(web-search): AI/ML API Toolbox as an opt-in web_search backend

ZERO_WEBSEARCH_PROVIDER=aimlapi runs the Toolbox tool web/search at the
fixed https://tools.aimlapi.com/v1/tools URL. It is selected only by that
setting; an AI/ML API key alone never enables it or registers web_search.

- The key comes from config.LoadAIMLAPIServiceKey on every search; a
  missing, proxy-profile or invalid key fails before any request.
  ZERO_WEBSEARCH_BASE_URL and ZERO_WEBSEARCH_API_KEY are ignored.
- The key and attribution headers are attached only after
  aimlapi.IsAttributedURL accepts the URL, and redirects are refused.
- Every run carries max_cost_usd 0.025 and a fresh idempotency key;
  limit maps to max_results and domains to include_domains, with the
  local domain filter kept.
- A queued run (202) is polled until it ends, honouring
  next_poll_hint_ms clamped to 250 ms..5 s. This backend gets a 30 s
  budget with wait_ms 20 s; the generic backends keep 10 s.
- Non-2xx responses, is_error runs, failed/stopped/timed_out runs and
  malformed bodies are errors carrying the error code and request id,
  with the key scrubbed.

* fix(aimlapi): preserve store errors and isolate Windows config tests

* fix(web-search): redact results and preserve domain errors

* docs(web-search): clarify minimum search charge

* fix(aimlapi): reject Unicode whitespace in service keys

* fix(aimlapi): report store reads after proxy eligibility checks

* fix(web-search): keep plain no-results for unscoped backends

Only a backend that applied the domain allowlist itself (the AI/ML API
Toolbox) should turn an empty result into the "no results matched
domains" error. A generic backend never saw the allowlist, so an empty
result from it is an ordinary "No results for query", as before this
branch.

Also clear ZERO_WEBSEARCH_PROVIDER in the unconfigured registry test:
ZERO_WEBSEARCH_PROVIDER=aimlapi alone now registers web_search, so an
exported value flipped the test's oracle.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(aimlapi): resolve the Toolbox key without an active provider

A user config that lists several providers but marks none active (or
names a removed one) makes Resolve return ErrNoActiveProvider together
with the normalized profiles. The app keeps running on that partial
config, but LoadAIMLAPIServiceKey stopped on the error, so web_search
failed even with an aimlapi profile or AIMLAPI_API_KEY set. Look the
key up in the partial config instead; the proxy check still scans every
resolved profile, and all other resolve errors are still returned.

Also state the key rules in the README as implemented: user config
only (project .zero providers are not used), which aimlapi profile is
used, and that any aimlapi profile or any profile reading
AIMLAPI_API_KEY on a non-canonical base URL withholds the key.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(aimlapi): withhold the Toolbox key for dropped proxy profiles

Resolve drops a non-active profile it cannot normalize (an unknown or
unsupported catalog entry, a missing base URL, ...), so the proxy check
never saw it. A dropped profile that reads AIMLAPI_API_KEY from a proxy
still shows the key belongs to that proxy, yet the key was sent to
tools.aimlapi.com. LoadAIMLAPIServiceKey now also checks the configured
user profiles, merged the same way Resolve merges them, and withholds
the key when a dropped profile carries it and is not explicitly on the
canonical base URL.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* test(web-search): pin the generic backend route in BASE_URL registration tests

defaultSearchBackend now checks ZERO_WEBSEARCH_PROVIDER=aimlapi before
ZERO_WEBSEARCH_BASE_URL. TestWebSearchRegisteredInCoreNetworkTools and
TestCoreNetworkToolsExposeSafetyMetadata set only the base URL and inherit
the provider from the process environment, so with an exported
ZERO_WEBSEARCH_PROVIDER=aimlapi they took the Toolbox route and still passed
even if generic base-URL registration broke. Clear the provider so they
exercise the generic route they describe, matching
TestCoreNetworkToolsOmitWebSearchWhenUnconfigured.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(redaction): redact caller-supplied secrets split by invisible characters

RedactString replaced Options.ExtraSecretValues only as contiguous text,
and the gap-tolerant pass from #1067 covers only the built-in credential
shapes. An opaque key such as the AI/ML API service key, echoed by a
service with a NUL or zero-width space inside it, matched neither pass.
Both fragments then reached Toolbox web_search errors (before clipping)
and successful results sent to the model and transcript.

Match each caller-supplied secret with runs of the same invisible
separators allowed between its characters, right after the contiguous
replacement. Separators inside the match are covered; those beside it
stay. Text without a separator takes the old path unchanged.

Regressions: split explicit values in the redaction package, and split
keys in the Toolbox HTTP error, run error and successful-result paths.
Raised by Copilot on #1189.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(redaction): claim each caller secret whole or split before shorter ones

RedactString replaced every ExtraSecretValues entry contiguously, longest
first, and only then ran the split match. A shorter secret inside a
longer one that was split by an invisible character was replaced first,
so "abc\x00def" with secrets abcdef and abc became "[REDACTED]\x00def"
and the rest of the longer secret stayed visible.

Run the contiguous and split match together per value, longest first,
so a longer value is claimed in either form before any shorter one.
Regressions cover a shorter secret at the start, end and middle of a
split longer one, and alongside it. Raised by CodeRabbit on #1189.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(redaction): merge overlapping caller-secret occurrences before replacing

ExtraSecretValues were replaced one occurrence at a time, so each
replacement could consume bytes another occurrence needed. With secret
abcabc, "abc\x00abcabc" became "abc\x00[REDACTED]". A key whose first and
last characters match lost all but one character the same way
("xY7q​[REDACTED]"), and two partly overlapping values left the
rest of one visible. Reordering the passes cannot fix this: the
contiguous and regexp replacements are both non-overlapping.

Find every occurrence of every value, whole or split by invisible
separators, in one separator-free copy of the text (linear KMP, all
overlapping starts), map the spans back, merge overlapping ones, and
replace each merged span once. Occurrences that only touch stay
separate replacements; separators beside a match stay; tab, newline
and carriage return are still not gaps. This honours the existing
TestRedactString_OverlappingExtraSecrets contract for both forms.
Raised by CodeRabbit on #1189.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(web-search): keep the request URL out of Toolbox transport errors

When http.Client.Do failed, the backend scrubbed err.Error() from the
*url.Error, which quotes the request URL. A polling URL carries the run
id the Toolbox returned, percent-escaped, so a key echoed in that id
appeared as "Lp9%00Xr4" or "Lp9%E2%80%8BXr4". The key scrubber matches
the key whole or split by raw invisible separators, not percent-escapes,
so the key stayed recoverable in the web_search error.

Report only the cause inside the *url.Error. The message prefix, the
redirect refusal text and key scrubbing are unchanged. The regression
fails a polling GET after a queued run whose id carries a NUL- or
ZWSP-split key and checks the output has neither the polling URL nor
the key. Raised by CodeRabbit on #1189.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

* fix(redaction): always redact a caller secret as written

Once the text held any invisible separator, redactExtraSecretValues
matched each caller secret only with separators removed from both. That
replaced the contiguous match instead of adding to it: a secret made
only of separators compacted to nothing and was skipped, and separators
at either end of a secret fell outside its match. RedactString("a\x00b")
with secret "\x00" returned the text unchanged, where main replaced it.

In the separator path, also record every occurrence of the secret as
written, then merge spans as before. Text without a separator takes the
unchanged path. Raised by CodeRabbit on #1189.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

---------

Co-authored-by: Stan <[email protected]>
Co-authored-by: Hugo <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
2026-11-04 18:24:09 -08:00

675 lines
26 KiB
Go

package redaction
import (
"fmt"
"regexp"
"regexp/syntax"
"sort"
"strings"
"unicode"
"unicode/utf8"
)
// A CREDENTIAL IS STILL A CREDENTIAL WITH AN INVISIBLE BYTE IN THE MIDDLE.
//
// The shape matchers (openaiKeyPattern, textSecretPatterns) all describe a
// CONTIGUOUS run of body characters, and none of their character classes admits
// a control or format character. So a single NUL, ESC or zero-width space
// dropped into the body of a key ends the match, and RedactString emits the two
// fragments untouched. Whoever reads that output next, a terminal that eats the
// escape, a log viewer, a JSON consumer that strips control bytes, or a person
// copying the text, sees the original key back, because nothing about those
// characters was ever displayed. Reported in #969 for NUL and ESC; every other
// Cc and Cf character behaves the same way, which is why this is written against
// the classes rather than against the two bytes named there.
//
// HOW: each shape is rewritten so that a run of separators may sit between any
// two characters it consumes, and matched against the ORIGINAL text. Nothing is
// removed before matching, so there are no offsets to map back, and the leading
// \b of every shape is judged by the regexp engine against the real neighbour:
// a key written after a NUL starts at a boundary, exactly as the unsplit key
// does, and a key glued to the end of a word does not.
//
// An earlier version removed the separators first and matched the compacted
// copy. That lost the difference between a separator inside a credential and one
// next to it, and the fixes for that (check the boundary against the source
// afterwards, map offsets back) left a hole: in "xAKIA<NUL>AKIAIOSF<SOH>ODNN7..."
// the leftmost compacted match began at the decoy AKIA, was then rejected for
// the "x" in front of it, and had already consumed the start of the real key, so
// the real key was never considered. Resuming the scan one byte after each
// rejection would close that, but it is quadratic for the greedy shapes on text
// like "ask-ask-ask-..." with a single NUL in it. Keeping the boundary inside
// the pattern lets RE2 do it in one linear pass.
//
// This pass runs AFTER the contiguous matchers, over their output, and never
// across a replacement. A credential that is whole is claimed by its own strict
// match, so two whole credentials with a separator between them stay two
// replacements with the separator still between them. Two split credentials do
// too: see spansFor.
//
// A match never starts or ends on a separator, so the separators on either side
// of a credential stay where they were: "<NUL>AKIA..." becomes
// "<NUL>[REDACTED]", as it does for the unsplit key.
//
// ONE KNOWN LIMIT, from running the contiguous matchers first. When the part of
// a split key BEFORE its first separator is already a complete credential on
// its own, the contiguous matcher claims that part and stops at the separator,
// and what follows is judged by itself. That is the rest of the key, which is no
// credential alone, so it stays visible: a fragment of the key, never the whole
// of it, because the part in front is replaced. For the same reason a credential
// glued straight onto that rest is left, where the unsplit key would have run on
// and swallowed it. Carrying the contiguous match across the separator instead
// would bring back what running first exists to prevent, since nothing tells the
// rest of a key from prose written after a whole one: that prose would be eaten,
// and a following JWT would lose its header.
//
// CALLER-SUPPLIED SECRETS (Options.ExtraSecretValues) get the same treatment
// from redactExtraSecretValues. Such a value is often opaque, matching no shape
// here, so without its own gap-tolerant match a separator inside it left both
// fragments in the output. Unlike a shape, the value is known exactly and has no
// word boundary to judge, so it is found in a copy with the separators removed
// and mapped back. RedactString runs it first, before any shape matcher, so a
// caller that clips or reformats the result afterwards never sees a fragment.
//
// WHAT THIS DELIBERATELY DOES NOT TREAT AS A SEPARATOR: tab, newline and
// carriage return. Those three are real text structure rather than invisible
// filler, so a credential broken across a line boundary is still two visible
// fragments here and is not covered. Treating them as gaps would also let a
// match span a line break and collapse unrelated lines into one replacement. A
// credential split that way is visible to whoever reads it; the ones handled
// here are not. Invalid UTF-8 is out for the same reason: the engine reads an
// invalid byte as U+FFFD, which is drawn.
// splitSecretSeparator reports a character that can sit inside a credential
// without being seen. Cc covers the C0 and C1 controls (NUL, ESC, DEL, NEL, CSI)
// and Cf the format characters (zero-width space and joiner, word joiner, BOM,
// soft hyphen). Tab, newline and carriage return are excluded: see above.
func splitSecretSeparator(r rune) bool {
switch r {
case '\t', '\n', '\r':
return false
}
return unicode.Is(unicode.Cc, r) || unicode.Is(unicode.Cf, r)
}
// containsSplitSeparator is the cheap gate in front of the gap-tolerant pass, so
// text with nothing invisible in it, which is nearly all text, costs one scan and
// no allocation. ASCII is settled from the byte alone; only a byte that could
// begin a multi-byte rune needs decoding, and the C1 and Cf blocks all start
// with a lead byte at or above 0xC2.
func containsSplitSeparator(value string) bool {
for i := 0; i < len(value); i++ {
b := value[i]
if b < utf8.RuneSelf {
if b < 0x20 || b == 0x7f {
if b == '\t' || b == '\n' || b == '\r' {
continue
}
return true
}
continue
}
r, width := utf8.DecodeRuneInString(value[i:])
if splitSecretSeparator(r) {
return true
}
i += width - 1
}
return false
}
// stripSplitSeparators returns value with every splitSecretSeparator removed.
// Used only on a matched credential, to judge it by its visible characters.
func stripSplitSeparators(value string) string {
var builder strings.Builder
builder.Grow(len(value))
for offset := 0; offset < len(value); {
r, width := utf8.DecodeRuneInString(value[offset:])
if !splitSecretSeparator(r) {
builder.WriteString(value[offset : offset+width])
}
offset += width
}
return builder.String()
}
// splitSeparatorClass is splitSecretSeparator as a regexp character class,
// generated from the same Unicode tables and the same three exclusions so the
// class and the predicate cannot disagree.
var splitSeparatorClass = func() string {
var class strings.Builder
class.WriteByte('[')
emit := func(lo, hi rune) {
fmt.Fprintf(&class, `\x{%x}-\x{%x}`, lo, hi)
}
for _, table := range []*unicode.RangeTable{unicode.Cc, unicode.Cf} {
visit := func(lo, hi, stride uint32) {
for r := lo; r <= hi; r += stride {
start := rune(r)
if !splitSecretSeparator(start) {
continue
}
end := start
if stride == 1 {
for next := rune(r) + 1; uint32(next) <= hi && splitSecretSeparator(next); next++ {
end = next
}
r = uint32(end)
}
emit(start, end)
}
}
for _, r := range table.R16 {
visit(uint32(r.Lo), uint32(r.Hi), uint32(r.Stride))
}
for _, r := range table.R32 {
visit(r.Lo, r.Hi, r.Stride)
}
}
class.WriteByte(']')
return class.String()
}()
// splitOpenAIPattern and splitTextPatterns are the shape matchers with a run of
// separators allowed between every two characters they consume, derived from
// the originals rather than written out a second time so the two lists cannot
// drift apart.
var (
splitOpenAIPattern = gapTolerant(openaiKeyPattern)
splitTextPatterns = gapTolerantAll(textSecretPatterns)
)
// splitShape is one credential shape as the split pass uses it.
type splitShape struct {
// free finds the shape anywhere, leftmost first. anchored matches it only
// where the text begins, and whole only when it spans the entire text.
free, anchored, whole *regexp.Regexp
// prefix is the literal every match of the shape begins with.
prefix string
// accept is the test a match has to pass to be a credential, or nil when
// every match is one.
accept func(match string) bool
}
var splitShapes = func() []splitShape {
shapes := []splitShape{newSplitShape(openaiKeyPattern, splitOpenAIPattern, func(match string) bool {
return openAIShapeIsSecret(stripSplitSeparators(match))
})}
for i, pattern := range textSecretPatterns {
shapes = append(shapes, newSplitShape(pattern, splitTextPatterns[i], nil))
}
return shapes
}()
func newSplitShape(original, free *regexp.Regexp, accept func(string) bool) splitShape {
return splitShape{
free: free,
anchored: regexp.MustCompile(`\A(?:` + free.String() + `)`),
whole: regexp.MustCompile(`\A(?:` + free.String() + `)\z`),
prefix: leadingLiteral(original),
accept: accept,
}
}
// leadingLiteral is the literal every match of pattern begins with, read past a
// leading \b, or "" when the pattern does not open with one.
func leadingLiteral(pattern *regexp.Regexp) string {
tree, err := syntax.Parse(pattern.String(), syntax.Perl)
if err != nil {
panic(fmt.Sprintf("redaction: parse %q: %v", pattern.String(), err))
}
nodes := []*syntax.Regexp{tree}
if tree.Op == syntax.OpConcat {
nodes = tree.Sub
}
for _, node := range nodes {
if node.Op == syntax.OpWordBoundary {
continue
}
if node.Op == syntax.OpLiteral {
return string(node.Rune)
}
return ""
}
return ""
}
// maxRunOnCandidates bounds the anchored matches tried inside one match, so a
// match full of lookalike starts still costs a fixed number of them.
const maxRunOnCandidates = 8
// spansFor turns one match of the shape into the stretches to replace.
//
// A MATCH CAN RUN ON INTO THE NEXT CREDENTIAL OF ITS OWN SHAPE. Gaps are allowed
// inside a match, and a separator between two credentials looks exactly like one
// inside a credential, so a greedy body runs through it into whatever follows. A
// credential of another shape is still found, because every shape is matched on
// its own and the union is replaced. One of the SAME shape is not: matches of one
// pattern never overlap, so the second credential cannot start inside the first
// one's match. Most bodies swallow it whole, so nothing leaks, but a JWT stops at
// the next token's first dot, and two split JWTs side by side left the second
// one's payload and signature in the clear. And a match the filter rejects hides
// whatever it ran into, so kebab-case prose in front of a split key took the key
// down with it. Reported by @jatmn.
//
// So when a match runs through a run of separators into a position where the
// same shape matches again and reaches at least as far, that credential is taken
// on its own, and the first is cut at the separators if it is a complete match by
// itself. The separators between the two then stay outside both replacements, as
// they do between two whole credentials, and each piece is filtered on its own.
// When the first is not complete alone, it keeps the whole of its match and the
// union joins the two into one replacement.
//
// ONE STEP PER MATCH, NOT THE WHOLE CHAIN. The second credential's own match can
// run on into a third, and following that from here would walk the rest of a
// chain once for every match inside it, which is quadratic: a megabyte of split
// JWTs ran for minutes. The scan's own next match starts inside the second
// credential and takes the step after it, so the chain is still covered, and in
// a chain of three or more the markers after the first pair can join into one.
func (shape splitShape) spansFor(text string, start, end int) [][]int {
next, gap, nextEnd := shape.runOn(text, start, end)
if next < 0 {
return shape.keep(nil, text, start, end)
}
var spans [][]int
if shape.whole.MatchString(text[start:gap]) {
spans = shape.keep(spans, text, start, gap)
} else {
spans = shape.keep(spans, text, start, end)
}
return shape.keep(spans, text, next, nextEnd)
}
func (shape splitShape) keep(spans [][]int, text string, start, end int) [][]int {
if shape.accept != nil && !shape.accept(text[start:end]) {
return spans
}
return append(spans, []int{start, end})
}
// runOn looks inside the match [start, end) for where it ran through a run of
// separators into another credential of the same shape. It returns where that
// credential begins, where the separator run in front of it begins, and where the
// credential's own match ends, or next = -1 when there is none.
//
// Latest first, because a match reaches into the next credential only as far as
// its own pattern allows. Only where the text after the separators opens with the
// shape's literal prefix, which is where a match of the shape can begin; the
// separator in front supplies the word boundary, so matching from there judges it
// the same as the whole text would.
func (shape splitShape) runOn(text string, start, end int) (next, gap, nextEnd int) {
tried := 0
for at := end; at > start && tried < maxRunOnCandidates; {
r, size := utf8.DecodeLastRuneInString(text[start:at])
if !splitSecretSeparator(r) {
at -= size
continue
}
candidate := at
for at > start {
r, size = utf8.DecodeLastRuneInString(text[start:at])
if !splitSecretSeparator(r) {
break
}
at -= size
}
if !hasGappedPrefix(text[candidate:], shape.prefix) {
continue
}
tried++
if match := shape.anchored.FindStringIndex(text[candidate:]); match != nil && candidate+match[1] >= end {
return candidate, at, candidate + match[1]
}
}
return -1, 0, 0
}
// hasGappedPrefix reports whether text opens with prefix, allowing a run of
// separators between its characters the way the shapes themselves do. A plain
// prefix test missed a token split inside its first few characters, and the run
// that swallowed it then went unnoticed.
func hasGappedPrefix(text, prefix string) bool {
at := 0
for index, want := range prefix {
if index > 0 {
for at < len(text) {
r, size := utf8.DecodeRuneInString(text[at:])
if !splitSecretSeparator(r) {
break
}
at += size
}
}
if at >= len(text) {
return false
}
r, size := utf8.DecodeRuneInString(text[at:])
if r != want {
return false
}
at += size
}
return true
}
func gapTolerantAll(patterns []*regexp.Regexp) []*regexp.Regexp {
tolerant := make([]*regexp.Regexp, 0, len(patterns))
for _, pattern := range patterns {
tolerant = append(tolerant, gapTolerant(pattern))
}
return tolerant
}
// gapTolerant rewrites pattern so that a run of separators may appear before
// every character it consumes except the first. Zero-width assertions, the
// leading \b in particular, are left exactly where they were.
func gapTolerant(pattern *regexp.Regexp) *regexp.Regexp {
tree, err := syntax.Parse(pattern.String(), syntax.Perl)
if err != nil {
panic(fmt.Sprintf("redaction: parse %q: %v", pattern.String(), err))
}
separator, err := syntax.Parse(splitSeparatorClass, syntax.Perl)
if err != nil {
panic(fmt.Sprintf("redaction: parse separator class: %v", err))
}
consumed := false
return regexp.MustCompile(withGaps(tree, separator, &consumed).String())
}
// withGaps inserts the separator run before each consuming atom. consumed
// tracks whether anything has been consumed yet on this path, because the first
// character of a match must be a real one: a leading gap would let a match start
// on a separator and swallow it.
func withGaps(node, separator *syntax.Regexp, consumed *bool) *syntax.Regexp {
switch node.Op {
case syntax.OpLiteral:
parts := make([]*syntax.Regexp, 0, len(node.Rune))
for _, r := range node.Rune {
atom := &syntax.Regexp{Op: syntax.OpLiteral, Rune: []rune{r}, Flags: node.Flags}
parts = append(parts, gapBefore(atom, separator, consumed))
}
if len(parts) == 1 {
return parts[0]
}
return &syntax.Regexp{Op: syntax.OpConcat, Sub: parts}
case syntax.OpCharClass, syntax.OpAnyChar, syntax.OpAnyCharNotNL:
return gapBefore(node, separator, consumed)
case syntax.OpConcat, syntax.OpCapture:
for i := range node.Sub {
node.Sub[i] = withGaps(node.Sub[i], separator, consumed)
}
return node
case syntax.OpAlternate:
start, after := *consumed, *consumed
for i := range node.Sub {
branch := start
node.Sub[i] = withGaps(node.Sub[i], separator, &branch)
after = after || branch
}
*consumed = after
return node
case syntax.OpStar, syntax.OpPlus, syntax.OpQuest, syntax.OpRepeat:
// Every shape opens with a literal prefix, so a repetition is never the
// first thing a match consumes. If one ever is, its first iteration would
// need no gap and the rest would; fail loudly rather than guess.
if !*consumed {
panic("redaction: gap-tolerant rewrite reached a repetition before any literal")
}
node.Sub[0] = withGaps(node.Sub[0], separator, consumed)
return node
case syntax.OpWordBoundary, syntax.OpNoWordBoundary, syntax.OpBeginLine, syntax.OpEndLine,
syntax.OpBeginText, syntax.OpEndText, syntax.OpEmptyMatch:
return node
default:
panic(fmt.Sprintf("redaction: gap-tolerant rewrite does not handle %v", node.Op))
}
}
func gapBefore(atom, separator *syntax.Regexp, consumed *bool) *syntax.Regexp {
if !*consumed {
*consumed = true
return atom
}
gap := &syntax.Regexp{Op: syntax.OpStar, Sub: []*syntax.Regexp{separator}}
return &syntax.Regexp{Op: syntax.OpConcat, Sub: []*syntax.Regexp{gap, atom}}
}
// openAIShapeIsSecret is the digit/kebab-case filter RedactString applies to an
// openaiKeyPattern match, named so both callers state the same rule once.
func openAIShapeIsSecret(match string) bool {
if knownOpenAIKeyPrefix(match) || secretMatchHasDigit(match) {
return true
}
return !strings.Contains(strings.TrimPrefix(match, "sk-"), "-")
}
// redactExtraSecretValues replaces every occurrence of every caller-supplied
// secret, whole or with runs of invisible separators between its characters.
//
// All occurrences, overlapping ones included, are found in the same text and
// overlapping spans are merged before anything is replaced. Replacing one at a
// time let each replacement consume bytes another occurrence needed: a shorter
// value inside a split longer one, a periodic value ("abcabc" in
// "abc<NUL>abcabc"), a value whose first and last characters match, or two
// values that overlap all left a fragment visible. Occurrences that only touch
// stay separate replacements, as they always were.
//
// Separators inside a match are covered by its replacement; a split match never
// starts or ends on one, so those beside it stay, unless the value as written
// starts or ends with them. Tab, newline and carriage return are not separators
// (see above).
func redactExtraSecretValues(value string, secrets []string, replacement string) string {
haystack, offsets := value, []int(nil)
if containsSplitSeparator(value) {
haystack, offsets = compactSplitSeparators(value)
}
var spans [][2]int
for _, secret := range secrets {
if strings.TrimSpace(secret) == "" {
continue
}
needle := secret
if offsets != nil {
// The value as written always matches, as it did before split
// matching existed. The compacted match cannot stand in for it: a
// value made only of separators compacts to nothing, and separators
// at either end of a value fall outside its compacted match.
if strings.Contains(value, secret) {
for _, at := range overlappingIndexes(value, secret) {
spans = append(spans, [2]int{at, at + len(secret)})
}
}
needle = stripSplitSeparators(secret)
}
if needle == "" || !strings.Contains(haystack, needle) {
continue
}
for _, at := range overlappingIndexes(haystack, needle) {
start, end := at, at+len(needle)
if offsets != nil {
start, end = offsets[start], offsets[end-1]+1
}
spans = append(spans, [2]int{start, end})
}
}
if len(spans) == 0 {
return value
}
sort.Slice(spans, func(i, j int) bool { return spans[i][0] < spans[j][0] })
var out strings.Builder
out.Grow(len(value))
written := 0
for i := 0; i < len(spans); {
start, end := spans[i][0], spans[i][1]
for i++; i < len(spans) && spans[i][0] < end; i++ {
end = max(end, spans[i][1])
}
out.WriteString(value[written:start])
out.WriteString(replacement)
written = end
}
out.WriteString(value[written:])
return out.String()
}
// compactSplitSeparators returns value with every splitSecretSeparator removed,
// and for each byte of the result the offset of that byte in value.
func compactSplitSeparators(value string) (string, []int) {
var builder strings.Builder
builder.Grow(len(value))
offsets := make([]int, 0, len(value))
for offset := 0; offset < len(value); {
r, width := utf8.DecodeRuneInString(value[offset:])
if !splitSecretSeparator(r) {
builder.WriteString(value[offset : offset+width])
for i := 0; i < width; i++ {
offsets = append(offsets, offset+i)
}
}
offset += width
}
return builder.String(), offsets
}
// overlappingIndexes returns the start of every occurrence of needle in
// haystack, overlapping ones included, in one linear Knuth-Morris-Pratt pass:
// restarting a search one byte after each match is quadratic for a periodic
// value over text full of it.
func overlappingIndexes(haystack, needle string) []int {
fallback := make([]int, len(needle))
for i, k := 1, 0; i < len(needle); i++ {
for k > 0 && needle[i] != needle[k] {
k = fallback[k-1]
}
if needle[i] == needle[k] {
k++
}
fallback[i] = k
}
var starts []int
for i, k := 0, 0; i < len(haystack); i++ {
for k > 0 && haystack[i] != needle[k] {
k = fallback[k-1]
}
if haystack[i] == needle[k] {
k++
}
if k == len(needle) {
starts = append(starts, i-len(needle)+1)
k = fallback[k-1]
}
}
return starts
}
// redactControlSplitSecrets replaces credentials whose body is interrupted by
// invisible characters. It is a no-op for text containing none of them, which is
// nearly all text, so the contiguous path keeps its behavior exactly.
//
// It runs over the contiguous matchers' output and stops at every replacement
// they left behind: those mark text already accounted for, and a match reaching
// across one would be reaching out of one credential and into the next.
//
// It runs at most twice. The second pass exists for a credential glued straight
// onto the end of another split one: it has no word boundary of its own until
// the first is replaced, and the replacement then supplies one, the same way the
// contiguous shapes cascade into each other. Two passes and no more keeps the
// cost linear; a longer chain of glued split credentials is left as the
// contiguous matchers leave a chain of glued whole ones.
func redactControlSplitSecrets(value, replacement string) string {
if !containsSplitSeparator(value) {
return value
}
once := redactSplitRegions(value, replacement, true)
if once == value {
return value
}
// Ungated: the credential the second pass exists for is often contiguous
// itself (a whole JWT glued onto a split AWS key), so its region can hold no
// separator at all once the first pass has cut the text around it.
return redactSplitRegions(once, replacement, false)
}
func redactSplitRegions(value, replacement string, gated bool) string {
if replacement == "" {
return redactSplitRegion(value, replacement, 0, gated)
}
regions := strings.Split(value, replacement)
if len(regions) == 1 {
return redactSplitRegion(value, replacement, 0, gated)
}
previous := byte(0)
for i, region := range regions {
regions[i] = redactSplitRegion(region, replacement, previous, gated)
previous = replacement[len(replacement)-1]
}
return strings.Join(regions, replacement)
}
// redactSplitRegion runs every gap-tolerant shape over one stretch of text with
// no replacement inside it and replaces the UNION of what they found.
//
// Independently, not one after another. An unbounded body with gaps allowed will
// run on across a separator into whatever follows, and when that is the start of
// another credential it takes that credential's prefix with it: in
// "sk-ant-…<ESC>build<ESC>eyJ….eyJ….sig" the Anthropic shape stops only at the
// JWT's first dot. Applied in sequence, the JWT shape then finds its header gone
// and the payload and signature are left in the clear. Matched against the same
// text, the JWT shape still finds the whole token, and the union covers both.
//
// previous is the byte immediately before the region in the original, or zero at
// the start of the string. It is lent to the engine in front of the region so a
// leading \b is judged against the real neighbour, and a match is never allowed
// to begin on it.
func redactSplitRegion(value, replacement string, previous byte, gated bool) string {
if gated && !containsSplitSeparator(value) {
return value
}
text, lead := value, 0
if previous != 0 {
text, lead = string([]byte{previous})+value, 1
}
var spans [][]int
for _, shape := range splitShapes {
for _, match := range shape.free.FindAllStringIndex(text, -1) {
if match[0] >= lead {
spans = append(spans, shape.spansFor(text, match[0], match[1])...)
}
}
}
return spliceSpans(text, spans, replacement)[lead:]
}
// spliceSpans replaces every named range of value with replacement, merging
// ranges that overlap so one stretch of text is never replaced twice.
func spliceSpans(value string, spans [][]int, replacement string) string {
if len(spans) == 0 {
return value
}
sort.Slice(spans, func(i, j int) bool {
if spans[i][0] != spans[j][0] {
return spans[i][0] < spans[j][0]
}
return spans[i][1] > spans[j][1]
})
var builder strings.Builder
builder.Grow(len(value))
written := 0
for _, span := range spans {
if span[0] < written {
// Overlaps the stretch just replaced: extend it rather than emit a
// second marker, which would claim two credentials where there was one
// stretch of text.
if span[1] > written {
written = span[1]
}
continue
}
builder.WriteString(value[written:span[0]])
builder.WriteString(replacement)
written = span[1]
}
builder.WriteString(value[written:])
return builder.String()
}