mirror of
https://github.com/BishopFox/joro.git
synced 2026-11-07 10:25:57 -10:00
Updated documentation
This commit is contained in:
1 parent
682077f92b
commit
d32a04b5a0
1 file changed
+6
-5
@@ -339,13 +339,14 @@ Tracked via `go.mod` / `go.sum` only — repo does **not** vendor (see "no vendo
|
||||
- **`Finding.ID` IS the dedupe identity**: `hex(sha256(ruleID ‖ host ‖ groupDim))[:32]`, containing no request ID, timestamp, or seq. Rescanning reproduces byte-identical IDs, so rescan is idempotent, the frontend live-merge is a plain map upsert, and findings reload with no ID remapping. `groupDim` is **per-rule** via `Rule.GroupBy`: `host` (headers/cookies/fingerprints), `url` (panels, listings, interesting files), `evidence` (secrets/PII — one finding per distinct value, hashing the *raw* match so two secrets sharing a prefix never collide and the key never carries the secret).
|
||||
- **Every finding funnels through `Store.Upsert`**, which increments `Count` only for an unseen `requestID:part:offset` (`occSeen`, capped at 512) — without that check a rescan doubles every count. Triage (`FalsePositive`, `Notes`, severity override) is preserved across upsert, and **rescan never clears first**.
|
||||
- **Three check kinds, one output path.** Declarative regex rules (the only kind operators may create), named Go **post-filters** (`Rule.PostFilters` → `postfilters.go`: `luhn`, `iban97`, `ssn`, `denylist`, `notVersionString`, `notHTML`, …), and built-in **analyzers** (`Kind: "analyzer"`) for absence/relational logic a regex cannot express. Analyzers are rules whose behavior is a func looked up by ID, so they inherit toggling, severity override, and grouping. All converge on `Engine.newFinding`, so redaction and truncation cannot be bypassed.
|
||||
- **Rule copy follows one convention, documented in the `rules_builtin.go` header and pinned by `rules_meta_test.go`.** Names are **sentence case** (not Title Case) with no trailing period, shaped `<subject> <state>` — `exposed`, `disclosed`, `missing`, `present`, `enabled`, `referenced`; the generic descriptor in a console name (`manager`, `console`, `admin`) stays lowercase so the product name is the only capitalized part. A rule about one specific header is named for that header (`Strict-Transport-Security missing`); a check spanning several mechanisms is named for the concept (`Clickjacking protection missing`). Where a product name is canonically lowercase, **reword so a capitalized word leads** (`Apache htpasswd file exposed`) rather than misspelling it. **Every rule carries a distinct non-empty `Description`** — the field is `omitempty` with no server-side validation and the frontend renders it conditionally with no fallback, so an empty one silently yields a name with no subtitle. Front-load it: the rules table truncates to a single line. When adding a rule family via a helper closure, **give the helper a `desc` parameter** — `fingerprint`/`stack`/`urlFile` originally had none, which is how 88 rules shipped description-less.
|
||||
- **Patterns are RE2: no lookahead, no lookbehind, no backreferences.** Negative conditions belong in a post-filter. RE2 also caps explicit repeat counts at **1000**, and Go's `(?m)$` matches only immediately before `\n` — so a class excluding `\r` can never reach `$` on a CRLF-framed header line (header patterns end at the character class, with no anchor). `Engine.applyRegexRule` trims the captured value, since a stray `\r` would land in both the evidence and the grouping hash.
|
||||
- **`Rule.Literal` prescreen**: a case-folded `bytes.Contains` before the regex, which is what makes ~167 rules per response affordable. A wrong `Literal` **silently disables the rule** (`TestBuiltinLiteralsAreConsistent`). `builtinRules()` is a **function**, not a package var, so nothing can alias the shipped table; **every shipped rule is enabled** (no present-but-disabled tier; `TestNoBuiltinShipsDisabled`), so operator changes are a `disabled` set plus a `severityOverride` map, and `PUT`/`DELETE` on a built-in returns 403.
|
||||
- **`Rule.Literal` prescreen**: a case-folded `bytes.Contains` before the regex, which is what makes ~167 rules per response affordable. A wrong `Literal` **silently disables the rule**. `builtinRules()` is a **function**, not a package var, so nothing can alias the shipped table; **every shipped rule is enabled** (no present-but-disabled tier), so operator changes are a `disabled` set plus a `severityOverride` map, and `PUT`/`DELETE` on a built-in returns 403.
|
||||
- **Info findings are hidden by default, not disabled, and severity visibility is one field.** 88 of the 167 rules are Info, so `emptyDetectFilter` (`web/src/stores/detectStore.ts`) defaults `severities` to the four non-Info bands; the rules still run and their findings are still deduplicated, counted in `summary.bySeverity`, and persisted. `DetectFilter.severities` is the **visible set**, and an **empty array means no severity filter at all** — the same meaning `MultiSelectDropdown`'s "Any"/Clear and an absent `severity` param carry. There is **no separate "show info" control** and the frontend does not use `minSeverity`. The Severity dropdown and the count-bar chips write that one field; a chip **reverses one band** rather than isolating it, so `toggleSeverityChip` materializes `SEVERITY_ORDER` before subtracting. Turning off the last visible chip lands on empty and reveals everything. The default is not persisted. While a band is hidden the chips sum higher than the table's total.
|
||||
- **Evidence offsets are absolute within the rendered document, and redaction is a display control.** `Finding.EvidenceOffset` is an offset into the raw request/response the UI renders — named by `EvidencePart` — not into the buffer the rule matched against, with `-1` meaning no faithful mapping exists (a decompressed body, or a URL match). `Message.absoluteOffset` is the single translation point. `EvidenceLength` is the length of the *matched region*, which for an analyzer is not `len(Evidence)` — that field holds a synthesized description. `TestEvidenceOffsetLandsOnTheMatch` asserts the round trip (`respRaw[off:off+len] == match`). **A byte offset is not a CodeMirror document position**: `EditorState.create` splits the doc on `/\r\n?|\n/` and rejoins with `\n`, so every CRLF collapses to one character and an HTTP body offset is short by one per header line. `byteOffsetToPos` (`web/src/pages/Detect.tsx`) subtracts the preceding CRLFs; a lone `\r` costs nothing, since it is also a split point but becomes one `\n`. The plugin then **verifies** the translated range against the finding's own evidence and falls back to a text search when it does not match; verification is skipped for analyzers, whose evidence is a description rather than a slice. As on the server in `Store.Upsert`, `evidenceOffset`/`evidenceLength`/`evidencePart` are **one value** and `detectStore.upsertFindings` must move them together, or a host-grouped merge points a stale length at a fresh offset. `Finding.RawEvidence` stores the unmasked value for the 48 rules that redact and is persisted in the project file: **redaction guards against shoulder-surfing and screen sharing, not data at rest**. `Finding.Evidence` keeps the mask, which is what the findings table renders; the detail pane starts masked on every selection and the eye toggle reveals per finding.
|
||||
- **Severity follows a fixed rubric, pinned by `TestSeverityRubric`.** Info = not exploitable on its own (exposed panels, missing headers, information disclosure, analytics keys, identifiers that are not credentials); Low = an exposed config file; Medium = the catch-all; High = account credentials, sensitive API keys, DB connection strings, low-level PII; Critical = high-grade PII (national ID or payment card) or severe compromise alone (RCE, auth bypass, a served DB dump). **Every credential is High and none is Critical**, and a rule that merely detects a *surface* is Info. Critical is a literal 15-ID allow-list in the test, which also asserts `Info >= Critical * 3` and that fewer than half the library sits at High or above. Severity is orthogonal to confidence, and both are operator-overridable per rule and per finding.
|
||||
- **Evidence offsets are absolute within the rendered document, and redaction is a display control.** `Finding.EvidenceOffset` is an offset into the raw request/response the UI renders — named by `EvidencePart` — not into the buffer the rule matched against, with `-1` meaning no faithful mapping exists (a decompressed body, or a URL match). `Message.absoluteOffset` is the single translation point. `EvidenceLength` is the length of the *matched region*, which for an analyzer is not `len(Evidence)` — that field holds a synthesized description. The round trip that must hold is `respRaw[off:off+len] == match`. **A byte offset is not a CodeMirror document position**: `EditorState.create` splits the doc on `/\r\n?|\n/` and rejoins with `\n`, so every CRLF collapses to one character and an HTTP body offset is short by one per header line. `byteOffsetToPos` (`web/src/pages/Detect.tsx`) subtracts the preceding CRLFs; a lone `\r` costs nothing, since it is also a split point but becomes one `\n`. The plugin then **verifies** the translated range against the finding's own evidence and falls back to a text search when it does not match; verification is skipped for analyzers, whose evidence is a description rather than a slice. As on the server in `Store.Upsert`, `evidenceOffset`/`evidenceLength`/`evidencePart` are **one value** and `detectStore.upsertFindings` must move them together, or a host-grouped merge points a stale length at a fresh offset. `Finding.RawEvidence` stores the unmasked value for the 48 rules that redact and is persisted in the project file: **redaction guards against shoulder-surfing and screen sharing, not data at rest**. `Finding.Evidence` keeps the mask, which is what the findings table renders; the detail pane starts masked on every selection and the eye toggle reveals per finding.
|
||||
- **Severity follows a fixed rubric.** Info = not exploitable on its own (exposed panels, missing headers, information disclosure, analytics keys, identifiers that are not credentials); Low = an exposed config file; Medium = the catch-all; High = account credentials, sensitive API keys, DB connection strings, low-level PII; Critical = high-grade PII (national ID or payment card) or severe compromise alone (RCE, auth bypass, a served DB dump). **Every credential is High and none is Critical**, and a rule that merely detects a *surface* is Info. Critical is a deliberately short list (~15 rules); the library's shape is `Info >= Critical * 3`, with fewer than half of all rules at High or above. Severity is orthogonal to confidence, and both are operator-overridable per rule and per finding.
|
||||
- **Bodies are not reliably plaintext.** `transport.go` sets `DisableCompression`, so Go neither requests gzip nor decodes it, `stripHopHeaders` leaves `Content-Encoding`, and the h2 path passes `Decompress: false` — so `parse.go` decompresses via the exported `proxy.TryDecompress`. Brotli/zstd are undecodable without a new dep, so those bodies are marked unscannable and **counted** (`skippedEncoded`) rather than silently skipped. `.js`/`.css` are **not** in `SkipExtensions` — the inverse of History's default exclude list.
|
||||
- **Content-type gating is a blacklist for data rules, a whitelist for format rules.** `Rule.ExcludeContentTypes` is checked before `ContentTypes` and wins outright. The seven ungated PII rules exclude `ctCSS` rather than whitelisting the formats that may carry PII, so they stay live everywhere else. **JavaScript is never excluded, and there is no `ctJS` constant**; `TestNoRuleExcludesJavaScript` pins it. `contentTypeKeyword` already returned `"css"`/`"js"`, so this added no classifier logic.
|
||||
- **Content-type gating is a blacklist for data rules, a whitelist for format rules.** `Rule.ExcludeContentTypes` is checked before `ContentTypes` and wins outright. The seven ungated PII rules exclude `ctCSS` rather than whitelisting the formats that may carry PII, so they stay live everywhere else. **JavaScript is never excluded, and there is no `ctJS` constant**. `contentTypeKeyword` already returned `"css"`/`"js"`, so this added no classifier logic.
|
||||
- **Cursor resets are load-bearing.** `Store.Clear()` zeroes `nextSeq` and `LoadItems()` rewrites it, so a stale high cursor means `SinceSeq` returns nothing and **detection silently stops for the session with no error**. Reset at all four sites: `handleClearRequests`, `resetLiveProjectState` (→ 0, via `resetDetectLiveState`), and `applyProjectConfig` (→ `store.LastSeq()`, **not** 0, so restored history isn't re-reported). `SetMaxSize` needs nothing.
|
||||
- **Findings persist in the project file** (schema **v5**), gated on `Config.PersistFindings` (default on), capped at 2,000 highest-severity-first with all triaged findings always kept: false-positive marks and notes cannot be recomputed, and "recompute from `RequestHistory`" fails whenever `saveHistory` is off. `liveStateSignature` includes the findings store's `atomic.Uint64` **revision counter**, since an FP toggle or note edit changes no count. Custom detect **rules** join `sharedConfigPayload`; **findings do not** (per-operator triage, and evidence can reference live secrets).
|
||||
- **Proxy-mode API enforces same-origin requests** (`internal/api/originguard.go`). `originGuard` rejects state-changing requests (and the `/ws` upgrade) unless `Sec-Fetch-Site` is `same-origin`/`none` and any `Origin` matches the host, plus a loopback/`--bind` `Host` allowlist. No `Access-Control-Allow-Origin` header is set (the SPA and plugin iframes are same-origin; the proxy→teamserver path is a non-browser Go client). Non-browser local tooling (no `Sec-Fetch-Site`/`Origin`) is allowed. The Host whitelist can be extended with `--allowed-host` (comma-separated or repeatable, hostname-only comparison) for setups that reach the loopback UI under a non-loopback Host, such as an SSH tunnel entry address. This only relaxes the Host check; the same-origin CSRF check is untouched. Listener/team-server mode uses `team.AuthMiddleware`'s bearer token instead (no origin guard).
|
||||
@@ -427,7 +428,7 @@ To add a new theme: create `web/src/themes/<name>.css` with all `--color-*` vari
|
||||
|
||||
## Testing
|
||||
|
||||
Limited automated coverage (`go test ./...` — `internal/callback/listeners_test.go`, `internal/proxy/statusfilter_test.go`, `internal/proxy/store_test.go`, and the `internal/detect` suite, where `parse_test.go` pins the assumptions about raw capture bytes); otherwise manual verification:
|
||||
Limited automated coverage (`go test ./...` — `internal/callback/listeners_test.go`, `internal/proxy/statusfilter_test.go`, and `internal/detect/rules_meta_test.go`, which pins the rule-copy conventions: every built-in has a distinct non-empty `Description`, names are sentence case with no trailing period, IDs and names are unique, and the analyzer registry resolves in both directions); otherwise manual verification:
|
||||
|
||||
1. `go build ./...` compiles cleanly, `go test ./...` passes
|
||||
2. `./joro` prints `Proxy listening on :8080` and `UI available at http://localhost:9090`
|
||||
|
||||
Reference in new issue
Block a user