mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-08 01:48:20 -10:00
tooling(split): split tooling v2 (S01/W0): journalled mover, re-export resolver, G11 preflight, widen v2, strict+file counters
Scripts only under scripts/split/. No Rust, no manifest, no budget touched. Base00be3d2e4f. tui strict production [M] 634,761 -> 634,761, file level [F] 943,686 -> 943,686, largest SCC 79 -> 79 (partition_report --non-test: [79, 8, 1, 1, 1]). Nothing was compiled: this is python; the rewrites it produces were checked with rustfmt parse/format, module_graph.py, the path-keyed budget scripts and counts, on a scratch `git init` export of this base (never the shared tree). A compile of a rewritten tree is the first Lane A window. What changed and why (reasoning lives here, not in comments): move-modules.py (rewritten). R2.5: the old tool exited 0 in a tree without .git after "moved 6 module(s)" (git failures were check=False), and its dry-run headline (354 files / 3,487 lines) counted 12 files / 212 lines inside the moved set that a real run never rewrites. - --dest-crate/--src-crate replace every hard-coded runtime literal; --json; --require-clean (dirty set from `git status --porcelain -z` plus files younger than --quiet-minutes, default 15); --apply-cfg (flips only `#[cfg(test)]` items that code outside the moved set uses; never touches cfg(not(test)) forks, which are an FD-04 decision); --restore <journal>. - No git mv (the shared index is not touched): plain file operations. Every write is compare-and-swap (stat+sha against the bytes the plan was computed from, temp file, os.replace) with a pre-image journal under CW/backups/; a change between plan and write refuses before writing (exit 3); a change or SIGTERM/SIGHUP/SIGINT during the write phase restores every pre-image and removes created files and directories (exit 4). git/git grep failures and a missing or foreign .git are fatal (exit 2). --require-clean refusals exit 5. - Rewritten Rust goes through `rustfmt --edition 2024 --emit stdout` on stdin per file; a file whose base text is not rustfmt-stable (the 17 include!d test_cases_*.rs fragments) is left exactly as rewritten, so no foreign line is reflowed. - Coverage: plain and grouped `use crate::{..}` (3 sites), alias, unqualified `<m>::` in the source lib (5 sites; cfg(test)-only use gets a cfg(test) import), `<dest>::` -> `crate::` inside moved files (code only), `codewhale_tui::<m>` in every other Rust file (cli config_keys 15 lines), sweep of path strings (plain, `tui/src/..`, segmented `"tui" / "src" / "<m>"`: this adds scripts/check_orcarouter_contract.py, which the old sweep missed) with vendor/, crates/config/, scripts/split/, CHANGELOG* and web/data/*.generated.json skipped; --derive-facts regenerates web/data (journalled pre-images). - Reports (and plans, never applies): residue edges from moved files to staying modules, super-root residue, manifest needs (deps, features, consumer crates), cfg sites by class. reexports.py (--resolve-reexports): resolves `pub use` facade names (globs through the upstream sources and the build.rs-generated constants via catalog_corrections.json / provider_descriptors.json) and rewrites `crate::config::NAME` and grouped lists to the upstream path. preflight.py (G11): whole-repo include_str!/include_bytes!/include!/#[path] resolution against the old-to-new map, features.toml owners, path strings the sweep cannot fix, `../src/<m>` references, CI test identities; exit 1 on residue. widen.py: --dest-crate/--src-crate/--packages, signal-safe lints restore (SIGTERM/SIGHUP/SIGINT and atexit; a manifest left capped by a killed run is restored first), atomic writes, --prepass static widening, unused_imports removed with the compiler's machine-applicable suggestions. count_lines.py promoted from the plan's scratch (identical [M]) with the [F] counter beside it. testsplit.py: function-level move/stay classifier with call closure. partition_report/closure_report --non-test; module_graph --tui-src/ --runtime-src and a regex-driven lexer (11.0 s -> 2.5 s over 1,585 files, 0 mismatches against the old lex_mask on every tracked .rs). Acceptance (scratch export of00be3d2e4f, config settings config_persistence config_keys oauth codex_model_cache): - dry run: staying callers 342 files / 3,275 lines + inside the moved set 12 / 212 = the old tool's 354 / 3,487; grouped 3; self paths 105 + 5 in comments = 110; other crates 2 files / 16 lines (15 cli + 1 runtime doc comment); sweep 31 files (+15 skipped); 81 cfg sites (53 test items, 11 test mods, 17 not(test) forks = the brief's 17); 9 s. - real rewrite: 403 paths, 18.6 s, counts identical to the dry run line for line; 334 of 351 changed .rs files rustfmt-stable (the 17 others are the base-unstable fragments); no `crate::<moved>` left in tui, no `codewhale_tui::<moved>` in cli; module_graph.py --check refuses this premature move with 43 residue edges (it is the closedness gate working). - closed module (remote_setup) real move: module_graph --check PASS with the unchanged baseline, check-blocking-calls-budget PASS (543 sites, 165 files; the sweep re-keyed the path-keyed JSON), dead-code budget PASS 209, check-provider-registry PASS, G11 unresolved 0. - guards on the scratch repo: change between plan and write -> exit 3 nothing written; change at write 201 of 403 -> exit 4 and `git status` back to only the concurrent edit; SIGTERM at 201 of 403 -> exit 4, status empty; dirty file and a file touched < 15 min -> exit 5; no .git -> exit 2. - widen.py: SIGTERM and SIGHUP while a compile round runs -> exit 130 and crates/runtime/Cargo.toml back to `[lints] workspace = true`; a capped manifest from an earlier run is restored at start; with a fake cargo JSON stream the E0603 widening, --prepass and the unused-import removal applied. - reexports: 226 config names (213 via models) -> 1,109 direct lines in 142 files and 68 grouped lists in 60 files outside the cluster, 166 files; the ProviderKind figure matches the plan exactly (339 lines / 95 files); the plan's 1,189 / 190 and 89 / 75 are not reproduced and likely include the cluster's own files; module_graph --check unchanged afterwards. - preflight on the cluster: residue 8 = lib.rs:20230 include_str, tests/ integration/main.rs:16 (path + relative), 5 skip-listed mentions in crates/config (catalog_corrections.json:4584 and models_dev.bundled.json:11769 key a source receipt `crates/tui/src/config/models.rs`), vendor picker.rs. - testsplit: config/tests.rs 407 tests (378 move, 29 stay), scope_tests.rs 8 (0 move: needs provider_readiness/fleet), whole cluster 687 tests (649 move, 38 stay); count_lines [M]/[F]: tui 634,761/943,686, cli 20,204/33,761, runtime 9,526/15,121, config 29,415/35,830, app-server 7,572/13,095. Not proven: no cargo build of any rewritten tree; widen.py's compile loop ran only against a fake cargo; --apply-cfg flipping is planned, not applied or compiled; Windows/FreeBSD cfg code; whether any test reads the source-receipt keys in crates/config assets as paths; the re-export resolver does not rewrite relative `config::NAME` uses (none found at this base); SCC/edge counts were not re-run after a rewrite (the cargo-free G2 stays Lane A's). Evidence: /Volumes/VIXinSSD/CW/artifacts/crate-split-w0-20261025/proof-w0.log and guard-demos.log. Dirty work of other writers in the checkout was left untouched (about 20 tui/core/models files, window W1). Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
This commit is contained in:
CodeWhale BotandClaude Sonnet 5.5 committed
1 parent
00be3d2e4f
commit
3b35f098f7
10 files changed
+2569
-178
No files matched your search
@@ -13,11 +13,16 @@ lines in its reverse closure (what a change to it can rebuild).
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
_spec = importlib.util.spec_from_file_location("count_lines", Path(__file__).with_name("count_lines.py"))
|
||||
cl = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(cl)
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
@@ -70,6 +75,11 @@ def main() -> None:
|
||||
parser.add_argument("--sort", choices=["name", "own", "closure", "reverse"], default="closure")
|
||||
parser.add_argument("--top", type=int, default=0)
|
||||
parser.add_argument("--tests", action="store_true", help="count test files too")
|
||||
parser.add_argument(
|
||||
"--non-test",
|
||||
action="store_true",
|
||||
help="size crates by strict production lines (inline cfg(test) code excluded) instead of file-level lines",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
meta = metadata()
|
||||
@@ -88,8 +98,13 @@ def main() -> None:
|
||||
|
||||
own = {}
|
||||
for name, pkg in packages.items():
|
||||
src = Path(pkg["manifest_path"]).parent / "src"
|
||||
own[name] = rust_lines(src, args.tests) if src.is_dir() else 0
|
||||
crate_dir = Path(pkg["manifest_path"]).parent
|
||||
src = crate_dir / "src"
|
||||
if args.non_test:
|
||||
_, res = cl.classify_crate(str(crate_dir))
|
||||
own[name] = sum(prod for (prod, _t, _o) in res.values())
|
||||
else:
|
||||
own[name] = rust_lines(src, args.tests) if src.is_dir() else 0
|
||||
|
||||
rows = []
|
||||
for name in packages:
|
||||
|
||||
@@ -0,0 +1,409 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Production/test line counter for the crate split (read-only).
|
||||
|
||||
python3 scripts/split/count_lines.py [crates_dir] [--json out.json] [--crate tui]
|
||||
|
||||
Two counters are printed for every crate and gated side by side (FD-07):
|
||||
|
||||
* [M] strict production: physical lines of production code, test files and
|
||||
`cfg(test)`-only items excluded;
|
||||
* [F] file level: physical lines of every non-test file, inline `cfg(test)`
|
||||
modules included (the design document's method; [F] = [M] + inline test).
|
||||
|
||||
A file is TEST when any of:
|
||||
A. a path component is `tests`
|
||||
B. the basename contains `tests` (tests.rs, foo_tests.rs, tests_foo.rs)
|
||||
C. it is declared by a `#[cfg(test)]`-gated `mod x;` (resolved to x.rs or
|
||||
x/mod.rs next to the declaring module, recursively everything below)
|
||||
D. it carries an inner `#![cfg(test)]` attribute
|
||||
E. a string-literal `include!` from test code pulls it in
|
||||
Inside non-test files, an item annotated with a test-only cfg
|
||||
(`#[cfg(test)]`, `#[cfg(all(test, ..))]`, `#[cfg(any(test, ..))]`,
|
||||
`#[cfg(feature = "test-support")]`) is TEST up to the end of its braces/semicolon.
|
||||
`benches/`, `examples/` and `build.rs` are counted as OTHER (neither).
|
||||
Lines are physical lines (newline count). Accuracy on tui is about +/-2%.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
TOKEN = re.compile(
|
||||
r"""//[^\n]*"""
|
||||
r"""|/\*.*?\*/"""
|
||||
r"""|b?r(?P<h>\#*)".*?"(?P=h)"""
|
||||
r"""|b?"(?:\\.|[^"\\])*\""""
|
||||
r"""|'(?:\\.[^']*|[^\\'])'""",
|
||||
re.S,
|
||||
)
|
||||
|
||||
|
||||
def blank(s: str) -> str:
|
||||
return re.sub(r"[^\n]", " ", s)
|
||||
|
||||
|
||||
def mask(src: str):
|
||||
def comments_only(m):
|
||||
t = m.group(0)
|
||||
if t.startswith("//") or t.startswith("/*"):
|
||||
return blank(t)
|
||||
return t
|
||||
|
||||
def everything(m):
|
||||
t = m.group(0)
|
||||
if t.startswith("//") or t.startswith("/*"):
|
||||
return blank(t)
|
||||
return t[0] + blank(t[1:-1]) + t[-1] if len(t) >= 2 else t
|
||||
|
||||
return TOKEN.sub(comments_only, src), TOKEN.sub(everything, src)
|
||||
|
||||
|
||||
CFG_OPEN = re.compile(r"#\s*\[\s*cfg\s*\(")
|
||||
CFG_TOK = re.compile(r'\s*(?:(?P<id>[A-Za-z_][A-Za-z0-9_]*)|(?P<str>"[^"]*")|(?P<p>[(),=]))')
|
||||
|
||||
|
||||
def cfg_possible(expr: str):
|
||||
toks = [m for m in CFG_TOK.finditer(expr)]
|
||||
pos = [0]
|
||||
|
||||
def peek():
|
||||
return toks[pos[0]] if pos[0] < len(toks) else None
|
||||
|
||||
def parse():
|
||||
t = toks[pos[0]]
|
||||
pos[0] += 1
|
||||
name = t.group("id")
|
||||
n = peek()
|
||||
if n is not None and n.group("p") == "(":
|
||||
pos[0] += 1
|
||||
kids = []
|
||||
while True:
|
||||
n = peek()
|
||||
if n is None:
|
||||
break
|
||||
if n.group("p") == ")":
|
||||
pos[0] += 1
|
||||
break
|
||||
if n.group("p") == ",":
|
||||
pos[0] += 1
|
||||
continue
|
||||
kids.append(parse())
|
||||
if name == "not":
|
||||
return {not v for v in kids[0]} if kids else {True, False}
|
||||
if name == "all":
|
||||
r = set()
|
||||
if all(True in k for k in kids):
|
||||
r.add(True)
|
||||
if any(False in k for k in kids):
|
||||
r.add(False)
|
||||
return r or {False}
|
||||
if name == "any":
|
||||
r = set()
|
||||
if any(True in k for k in kids):
|
||||
r.add(True)
|
||||
if all(False in k for k in kids):
|
||||
r.add(False)
|
||||
return r or {False}
|
||||
return {True, False}
|
||||
if n is not None and n.group("p") == "=":
|
||||
pos[0] += 1
|
||||
v = peek()
|
||||
pos[0] += 1
|
||||
if name == "feature" and v is not None and v.group("str") == '"test-support"':
|
||||
return {False}
|
||||
return {True, False}
|
||||
if name == "test":
|
||||
return {False}
|
||||
return {True, False}
|
||||
|
||||
try:
|
||||
return parse()
|
||||
except Exception:
|
||||
return {True, False}
|
||||
|
||||
|
||||
INCLUDE = re.compile(r'\binclude!\s*\(\s*"([^"]+)"\s*\)')
|
||||
PATH_ATTR = re.compile(r'#\s*\[\s*path\s*=\s*"([^"]+)"\s*\]')
|
||||
INNER_CFG = re.compile(r"#\s*!\s*\[\s*cfg\s*\(\s*test\s*\)\s*\]")
|
||||
ATTR_CLOSE = re.compile(r"\s*\]")
|
||||
ATTR = re.compile(r"\s*#\s*\[[^\]]*\]")
|
||||
MODDECL = re.compile(r"\s*(?:pub(?:\([^)]*\))?\s+)?mod\s+(\w+)\s*;")
|
||||
BR = re.compile(r"[{}]")
|
||||
|
||||
|
||||
ITEM_KW = re.compile(
|
||||
r"(?:pub(?:\s*\([^)]*\))?\s+)?(?:async\s+|unsafe\s+|const\s+|extern\s+(?:\"[^\"]*\"\s+)?)*"
|
||||
r"(?:mod|fn|impl|use|struct|enum|trait|type|static|const|macro_rules|extern|let|union)\b"
|
||||
)
|
||||
FALLBACK = []
|
||||
|
||||
|
||||
def item_end(m2: str, j: int) -> int:
|
||||
depth = 0
|
||||
n = len(m2)
|
||||
k = j
|
||||
comma_ends = not ITEM_KW.match(m2, j) and not m2.startswith("{", j)
|
||||
if comma_ends:
|
||||
FALLBACK.append(j)
|
||||
while k < n:
|
||||
c = m2[k]
|
||||
if c in "([":
|
||||
depth += 1
|
||||
elif c in ")]":
|
||||
depth -= 1
|
||||
elif depth <= 0 and c == ";":
|
||||
return k
|
||||
elif depth <= 0 and comma_ends and c == ",":
|
||||
return k
|
||||
elif depth <= 0 and c == "{":
|
||||
d = 0
|
||||
for b in BR.finditer(m2, k):
|
||||
if b.group(0) == "{":
|
||||
d += 1
|
||||
else:
|
||||
d -= 1
|
||||
if d == 0:
|
||||
return b.start()
|
||||
return n - 1
|
||||
k += 1
|
||||
return n - 1
|
||||
|
||||
|
||||
def analyse_file(path: str, src: str):
|
||||
"""returns (test_ranges[(a,b) char offsets], cfg_test_mod_names, whole_file_test)"""
|
||||
m1, m2 = mask(src)
|
||||
if INNER_CFG.search(m2) or any(
|
||||
INNER_CFG.match(m1, mm.start()) for mm in re.finditer(r"#\s*!", m2)
|
||||
):
|
||||
return [], [], True, [(m.start(), m.group(1)) for m in INCLUDE.finditer(m1) if m2[m.start()] == "i"]
|
||||
ranges = []
|
||||
mods = []
|
||||
covered = -1
|
||||
spans = []
|
||||
for m in CFG_OPEN.finditer(m1):
|
||||
if m2[m.start()] != "#":
|
||||
continue
|
||||
depth = 1
|
||||
k = m.end()
|
||||
while k < len(m1) and depth:
|
||||
if m1[k] == "(":
|
||||
depth += 1
|
||||
elif m1[k] == ")":
|
||||
depth -= 1
|
||||
k += 1
|
||||
close = ATTR_CLOSE.match(m1, k)
|
||||
if not close:
|
||||
continue
|
||||
expr = m1[m.end():k - 1]
|
||||
if not re.search(r"\btest\b|test-support", expr):
|
||||
continue
|
||||
if True in cfg_possible(expr):
|
||||
continue
|
||||
spans.append((m.start(), close.end()))
|
||||
for ms, me in spans:
|
||||
if ms <= covered:
|
||||
continue
|
||||
j = me
|
||||
path_attr = None
|
||||
while True:
|
||||
a = ATTR.match(m2, j)
|
||||
if not a:
|
||||
break
|
||||
pm = PATH_ATTR.match(m1, j + (len(m1[j:]) - len(m1[j:].lstrip())))
|
||||
if pm:
|
||||
path_attr = pm.group(1)
|
||||
j = a.end()
|
||||
md = MODDECL.match(m2, j)
|
||||
if md:
|
||||
mods.append(("path:" + path_attr) if path_attr else md.group(1))
|
||||
end = md.end()
|
||||
else:
|
||||
while j < len(m2) and m2[j].isspace():
|
||||
j += 1
|
||||
end = item_end(m2, j)
|
||||
ranges.append((ms, end))
|
||||
covered = end
|
||||
incs = [(m.start(), m.group(1)) for m in INCLUDE.finditer(m1) if m2[m.start()] == "i"]
|
||||
return ranges, mods, False, incs
|
||||
|
||||
|
||||
def read(path):
|
||||
with open(path, encoding="utf-8", errors="replace") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def collect(crate_dir):
|
||||
files = []
|
||||
for dp, dn, fn in os.walk(crate_dir):
|
||||
dn[:] = [d for d in dn if d not in ("target", "node_modules", ".git")]
|
||||
for f in fn:
|
||||
if f.endswith(".rs"):
|
||||
files.append(os.path.join(dp, f))
|
||||
return sorted(files)
|
||||
|
||||
|
||||
def classify_crate(crate_dir, tui_style=False):
|
||||
files = collect(crate_dir)
|
||||
info = {}
|
||||
for p in files:
|
||||
src = read(p)
|
||||
rel = os.path.relpath(p, crate_dir)
|
||||
info[p] = dict(rel=rel, src=src, nl=src.count("\n") + (0 if src.endswith("\n") or not src else 1))
|
||||
test_files = set()
|
||||
other_files = set()
|
||||
cfg_mods = {}
|
||||
for p, d in info.items():
|
||||
rel = d["rel"]
|
||||
parts = rel.split("/")
|
||||
base = parts[-1]
|
||||
if parts[0] in ("benches", "examples") or rel == "build.rs":
|
||||
other_files.add(p)
|
||||
continue
|
||||
if "tests" in parts[:-1] or "tests" in base:
|
||||
test_files.add(p)
|
||||
for p, d in info.items():
|
||||
if p in other_files:
|
||||
continue
|
||||
ranges, mods, whole, incs = analyse_file(p, d["src"])
|
||||
d["ranges"] = ranges
|
||||
d["incs"] = incs
|
||||
if whole:
|
||||
test_files.add(p)
|
||||
base = os.path.basename(p)
|
||||
stem = base[:-3]
|
||||
mdir = os.path.dirname(p) if base in ("mod.rs", "lib.rs", "main.rs") else os.path.join(os.path.dirname(p), stem)
|
||||
for name in mods:
|
||||
if name.startswith("path:"):
|
||||
cands = (os.path.normpath(os.path.join(os.path.dirname(p), name[5:])),)
|
||||
else:
|
||||
cands = (os.path.join(mdir, name + ".rs"), os.path.join(mdir, name, "mod.rs"))
|
||||
for cand in cands:
|
||||
if cand in info:
|
||||
test_files.add(cand)
|
||||
changed = True
|
||||
while changed:
|
||||
changed = False
|
||||
for p, d in info.items():
|
||||
for off, rel in d.get("incs", []):
|
||||
tgt = os.path.normpath(os.path.join(os.path.dirname(p), rel))
|
||||
if tgt not in info or tgt in test_files:
|
||||
continue
|
||||
if p in test_files or any(a <= off <= b for a, b in d.get("ranges", [])):
|
||||
test_files.add(tgt)
|
||||
changed = True
|
||||
for p in list(test_files):
|
||||
base = os.path.basename(p)
|
||||
stem = base[:-3]
|
||||
if base == "mod.rs":
|
||||
sub = os.path.dirname(p) + os.sep
|
||||
else:
|
||||
sub = os.path.join(os.path.dirname(p), stem) + os.sep
|
||||
for q in info:
|
||||
if q.startswith(sub) and q not in test_files and q not in other_files:
|
||||
test_files.add(q)
|
||||
changed = True
|
||||
res = {}
|
||||
for p, d in info.items():
|
||||
nl = d["nl"]
|
||||
if p in other_files:
|
||||
res[p] = (0, 0, nl)
|
||||
elif p in test_files:
|
||||
res[p] = (0, nl, 0)
|
||||
else:
|
||||
src = d["src"]
|
||||
tl = 0
|
||||
for a, b in d.get("ranges", []):
|
||||
tl += src.count("\n", a, b) + 1
|
||||
tl = min(tl, nl)
|
||||
res[p] = (nl - tl, tl, 0)
|
||||
return info, res
|
||||
|
||||
|
||||
def summarise(crates_dir):
|
||||
out = {}
|
||||
tui_detail = None
|
||||
for c in sorted(os.listdir(crates_dir)):
|
||||
cd = os.path.join(crates_dir, c)
|
||||
if not os.path.isfile(os.path.join(cd, "Cargo.toml")):
|
||||
continue
|
||||
info, res = classify_crate(cd)
|
||||
prod = sum(v[0] for v in res.values())
|
||||
test = sum(v[1] for v in res.values())
|
||||
other = sum(v[2] for v in res.values())
|
||||
inline_test = sum(tl for (pl, tl, ol) in res.values() if pl or ol)
|
||||
big = []
|
||||
for p, d in info.items():
|
||||
if res[p][0] or (res[p][1] == 0 and res[p][2] == 0):
|
||||
for a, b in d.get("ranges", []):
|
||||
big.append((d["src"].count("\n", a, b) + 1, d["rel"], " ".join(d["src"][a:b + 1].split())[:90]))
|
||||
big.sort(reverse=True)
|
||||
out[c] = dict(production_lines=prod, file_level_lines=prod + inline_test, test_lines=test,
|
||||
other_lines=other, inline_cfg_test_lines=inline_test, files=len(res),
|
||||
inline_ranges=len(big), inline_top=big[:6])
|
||||
if c == "tui":
|
||||
tui_detail = (cd, info, res)
|
||||
tui_mods = {}
|
||||
if tui_detail:
|
||||
cd, info, res = tui_detail
|
||||
src_root = os.path.join(cd, "src") + os.sep
|
||||
mods = defaultdict(lambda: dict(production_lines=0, file_level_lines=0, test_lines=0, files=0, biggest=[]))
|
||||
for p, (pl, tl, ol) in res.items():
|
||||
if not p.startswith(src_root):
|
||||
continue
|
||||
rel = p[len(src_root):]
|
||||
parts = rel.split("/")
|
||||
if len(parts) == 1:
|
||||
stem = parts[0][:-3]
|
||||
mod = {"lib": "<lib.rs>", "main": "<main.rs>"}.get(stem, stem)
|
||||
else:
|
||||
mod = parts[0]
|
||||
m = mods[mod]
|
||||
m["production_lines"] += pl
|
||||
m["file_level_lines"] += pl + (tl if (pl or ol) else 0)
|
||||
m["test_lines"] += tl
|
||||
m["files"] += 1
|
||||
if pl:
|
||||
m["biggest"].append((pl, rel))
|
||||
for m in mods.values():
|
||||
m["biggest"] = sorted(m["biggest"], reverse=True)[:3]
|
||||
tui_mods = dict(mods)
|
||||
out["tui"]["production_outside_src"] = sum(v[0] for p, v in res.items() if not p.startswith(src_root))
|
||||
return dict(crates=out, tui_modules=tui_mods, fallback_uses=len(FALLBACK))
|
||||
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
|
||||
repo = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
ap.add_argument("crates_dir", nargs="?", default=os.path.join(repo, "crates"))
|
||||
ap.add_argument("--json", metavar="FILE", help="write the full result as JSON ('-' for stdout)")
|
||||
ap.add_argument("--crate", action="append", help="print only these crates")
|
||||
ap.add_argument("--top", type=int, default=0, help="also list the N largest tui modules")
|
||||
args = ap.parse_args()
|
||||
result = summarise(args.crates_dir)
|
||||
if args.json:
|
||||
text = json.dumps(result, indent=1)
|
||||
if args.json == "-":
|
||||
print(text)
|
||||
else:
|
||||
with open(args.json, "w", encoding="utf-8") as fh:
|
||||
fh.write(text + "\n")
|
||||
rows = result["crates"]
|
||||
names = args.crate or sorted(rows, key=lambda n: -rows[n]["production_lines"])
|
||||
print(f"{'crate':22}{'[M] prod':>11}{'[F] file':>11}{'test':>11}{'inline test':>13}{'files':>7}")
|
||||
for n in names:
|
||||
r = rows[n]
|
||||
print(f"{n:22}{r['production_lines']:>11,}{r['file_level_lines']:>11,}{r['test_lines']:>11,}{r['inline_cfg_test_lines']:>13,}{r['files']:>7}")
|
||||
tot_m = sum(r["production_lines"] for r in rows.values())
|
||||
tot_f = sum(r["file_level_lines"] for r in rows.values())
|
||||
print(f"{'all crates':22}{tot_m:>11,}{tot_f:>11,}")
|
||||
if args.top and result["tui_modules"]:
|
||||
print("\nlargest tui modules ([M] / [F]):")
|
||||
for name, m in sorted(result["tui_modules"].items(), key=lambda kv: -kv[1]["production_lines"])[: args.top]:
|
||||
print(f" {name:28}{m['production_lines']:>9,}{m['file_level_lines']:>10,}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -101,18 +101,33 @@ def is_ui(module: str) -> bool:
|
||||
|
||||
_RAW_STR = re.compile(r'b?r(#*)"')
|
||||
_CHAR = re.compile(r"'(\\.[^']*|[^\\'])'")
|
||||
_START = re.compile(r'//|/\*|b?r#*"|b?"|\'')
|
||||
_BLOCK = re.compile(r'/\*|\*/')
|
||||
_STR_STOP = re.compile(r'[\\"]')
|
||||
_NOT_NL = re.compile(r'[^\n]')
|
||||
|
||||
|
||||
def lex_mask(src: str) -> str:
|
||||
out = list(src)
|
||||
i, n = 0, len(src)
|
||||
n = len(src)
|
||||
out = []
|
||||
last = 0
|
||||
i = 0
|
||||
|
||||
def blank(a: int, b: int) -> None:
|
||||
for k in range(a, min(b, n)):
|
||||
if out[k] != "\n":
|
||||
out[k] = " "
|
||||
def blank(a, b):
|
||||
nonlocal last
|
||||
b = min(b, n)
|
||||
if a >= b:
|
||||
return
|
||||
out.append(src[last:a])
|
||||
out.append(_NOT_NL.sub(" ", src[a:b]))
|
||||
last = b
|
||||
|
||||
while i < n:
|
||||
search = _START.search
|
||||
while True:
|
||||
m = search(src, i)
|
||||
if not m:
|
||||
break
|
||||
i = m.start()
|
||||
c = src[i]
|
||||
if src.startswith("//", i):
|
||||
j = src.find("\n", i)
|
||||
@@ -123,22 +138,23 @@ def lex_mask(src: str) -> str:
|
||||
if src.startswith("/*", i):
|
||||
depth, j = 1, i + 2
|
||||
while j < n and depth:
|
||||
if src.startswith("/*", j):
|
||||
mm = _BLOCK.search(src, j)
|
||||
if not mm:
|
||||
j = n
|
||||
break
|
||||
if mm.group(0) == "/*":
|
||||
depth += 1
|
||||
j += 2
|
||||
elif src.startswith("*/", j):
|
||||
depth -= 1
|
||||
j += 2
|
||||
else:
|
||||
j += 1
|
||||
depth -= 1
|
||||
j = mm.end()
|
||||
blank(i, j)
|
||||
i = j
|
||||
continue
|
||||
prev_ident = i > 0 and (src[i - 1].isalnum() or src[i - 1] == "_")
|
||||
m = _RAW_STR.match(src, i, i + 10)
|
||||
if m and not prev_ident:
|
||||
end = '"' + m.group(1)
|
||||
j = src.find(end, i + len(m.group(0)))
|
||||
rm = _RAW_STR.match(src, i, i + 10)
|
||||
if rm and not prev_ident:
|
||||
end = '"' + rm.group(1)
|
||||
j = src.find(end, i + len(rm.group(0)))
|
||||
j = n if j < 0 else j + len(end)
|
||||
blank(i + 1, j - 1)
|
||||
i = j
|
||||
@@ -146,17 +162,24 @@ def lex_mask(src: str) -> str:
|
||||
if c == '"' or (c == "b" and i + 1 < n and src[i + 1] == '"' and not prev_ident):
|
||||
j = i + (2 if c == "b" else 1)
|
||||
while j < n and src[j] != '"':
|
||||
j += 2 if src[j] == "\\" else 1
|
||||
sm = _STR_STOP.search(src, j)
|
||||
if not sm:
|
||||
j = n
|
||||
break
|
||||
j = sm.start()
|
||||
if src[j] == "\\":
|
||||
j += 2
|
||||
blank(i + 1, j)
|
||||
i = j + 1
|
||||
continue
|
||||
if c == "'":
|
||||
m = _CHAR.match(src, i, i + 12)
|
||||
if m:
|
||||
blank(i + 1, i + len(m.group(0)) - 1)
|
||||
i += len(m.group(0))
|
||||
cm = _CHAR.match(src, i, i + 12)
|
||||
if cm:
|
||||
blank(i + 1, i + len(cm.group(0)) - 1)
|
||||
i += len(cm.group(0))
|
||||
continue
|
||||
i += 1
|
||||
out.append(src[last:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
@@ -695,9 +718,11 @@ def main(argv: list[str] | None = None) -> int:
|
||||
"--baseline-ref",
|
||||
help="git revision whose committed baseline the current one may not exceed (CI: the PR base)",
|
||||
)
|
||||
parser.add_argument("--tui-src", type=Path, default=TUI_SRC, help="source directory of the crate being split")
|
||||
parser.add_argument("--runtime-src", type=Path, default=RUNTIME_SRC, help="source directory of the destination crate")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
report = build_report()
|
||||
report = build_report(args.tui_src.resolve(), args.runtime_src.resolve())
|
||||
if args.report:
|
||||
json.dump(
|
||||
{"closure": report.closure, "totals": totals(report.counts), "counts": report.counts,
|
||||
|
||||
+736
-137
@@ -1,177 +1,776 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Move top-level TUI modules into crates/runtime (the runtime/TUI split).
|
||||
"""Move top-level modules from one workspace crate into another (crate split).
|
||||
|
||||
For each named module this:
|
||||
python3 scripts/split/move-modules.py --dry-run --json config oauth
|
||||
python3 scripts/split/move-modules.py --require-clean config oauth
|
||||
python3 scripts/split/move-modules.py --resolve-reexports --dry-run config
|
||||
python3 scripts/split/move-modules.py --restore <journal-dir>
|
||||
|
||||
1. `git mv`s `crates/tui/src/<m>.rs` and/or `crates/tui/src/<m>/` into
|
||||
`crates/runtime/src/`;
|
||||
2. deletes its `mod <m>;` / `pub mod <m>;` line from the TUI `lib.rs` and adds
|
||||
`pub mod <m>;` to the runtime `lib.rs` (sorted);
|
||||
3. rewrites every `crate::<m>` path in `crates/tui/src` to
|
||||
`codewhale_runtime::<m>` and, when the TUI `lib.rs` uses `<m>::` unqualified,
|
||||
adds `use codewhale_runtime::<m>;` there;
|
||||
4. sweeps every string occurrence of the old path across the repository
|
||||
(`crates/tui/src/<m>` and `tui/src/<m>`), found with `git grep`.
|
||||
CHANGELOG files are history and are left alone.
|
||||
For each named module, moves `crates/<src>/src/<m>.rs` and `<m>/` into
|
||||
`crates/<dest>/src/` (default tui -> runtime) by plain file operations, never
|
||||
`git mv`, so the shared index is not touched; and rewrites:
|
||||
|
||||
It changes no item visibility; `scripts/split/widen.py` does that, driven by
|
||||
the compiler.
|
||||
* the `mod <m>;` line (removed from the source lib, `pub mod <m>;` added to the
|
||||
destination lib, sorted);
|
||||
* every `crate::<m>` in the staying source files, plain paths and grouped
|
||||
`use crate::{..}` lists, to `<dest crate>::<m>`; unqualified `<m>::` in the
|
||||
source lib gets a `use <dest crate>::<m>;`;
|
||||
* every `<src crate>::<m>` in any other Rust file of the repository;
|
||||
* `<dest crate>::` to `crate::` inside the moved files;
|
||||
* path strings in tracked text files (`crates/<src>/src/<m>`, `<src>/src/<m>`
|
||||
and segmented `"tui" / "src" / "<m>"`), except `vendor/`, `crates/config/`,
|
||||
`CHANGELOG*` and `web/data/*.generated.json` (regenerate those with
|
||||
`--derive-facts`).
|
||||
|
||||
Usage:
|
||||
python3 scripts/split/move-modules.py <module> [<module> ...]
|
||||
python3 scripts/split/move-modules.py --dry-run <module> ...
|
||||
Safety: git failures are fatal and the root must hold `.git`; `--require-clean`
|
||||
refuses when any file of the rewrite set is dirty or younger than
|
||||
`--quiet-minutes`; writes are compare-and-swap with a pre-image journal under
|
||||
`CW/backups/` and any change between plan and write restores everything;
|
||||
rewritten Rust is formatted in memory with `rustfmt --edition 2024` on stdin.
|
||||
|
||||
`--resolve-reexports` does not move anything: for each facade module it
|
||||
rewrites `crate::<facade>::NAME` to the upstream path that `pub use` hides.
|
||||
Manifest and `cfg` planning is report-only; `--apply-cfg` flips the
|
||||
`#[cfg(test)]` items that code outside the moved set uses to
|
||||
`any(test, feature = "test-support")`. It never touches `cfg(not(test))`
|
||||
behaviour forks (test isolation is a decision, not a rewrite).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
TUI_SRC = REPO_ROOT / "crates" / "tui" / "src"
|
||||
RUNTIME_SRC = REPO_ROOT / "crates" / "runtime" / "src"
|
||||
TUI_LIB = TUI_SRC / "lib.rs"
|
||||
RUNTIME_LIB = RUNTIME_SRC / "lib.rs"
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import splitlib as sl
|
||||
from splitlib import REPO_ROOT, Change, Fingerprint, SplitError
|
||||
|
||||
mg = sl.load_module_graph()
|
||||
|
||||
MUST_SWEEP = (".gitattributes", "scripts/command-migration-topology.json", "docs/features.toml")
|
||||
GENERATED_BY_DERIVE_FACTS = (
|
||||
"web/lib/facts.generated.ts",
|
||||
"web/data/features.generated.json",
|
||||
"web/data/agent-docs.generated.json",
|
||||
)
|
||||
BUILTIN_STD = {"crate", "self", "super", "std", "core", "alloc", "Self"}
|
||||
|
||||
|
||||
def git(*args: str, check: bool = True) -> str:
|
||||
return subprocess.run(
|
||||
["git", *args], cwd=REPO_ROOT, capture_output=True, text=True, check=check
|
||||
).stdout
|
||||
def log(msg: str) -> None:
|
||||
print(msg, file=sys.stderr)
|
||||
|
||||
|
||||
def module_paths(module: str) -> list[Path]:
|
||||
paths = [p for p in (TUI_SRC / f"{module}.rs", TUI_SRC / module) if p.exists()]
|
||||
if not paths:
|
||||
raise SystemExit(f"move-modules: no crates/tui/src/{module}.rs or {module}/")
|
||||
return paths
|
||||
def first_segment(masked_entry: str) -> str | None:
|
||||
m = re.match(r"\s*(?:::)?([A-Za-z_]\w*)", masked_entry)
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def remove_mod_line(text: str, module: str) -> tuple[str, bool]:
|
||||
pattern = re.compile(rf"^(?:pub(?:\([^)]*\))?\s+)?mod\s+{re.escape(module)}\s*;\n", re.M)
|
||||
match = pattern.search(text)
|
||||
if not match:
|
||||
return text, False
|
||||
# Refuse to drop a mod line that carries attributes (cfg, path): that
|
||||
# module needs a human decision, not a mechanical move.
|
||||
before = text[: match.start()].rstrip("\n").rsplit("\n", 1)[-1].strip()
|
||||
if before.startswith("#["):
|
||||
raise SystemExit(f"move-modules: `mod {module};` carries an attribute ({before}); move it by hand")
|
||||
return text[: match.start()] + text[match.end() :], True
|
||||
def split_entries(text: str, masked: str, a: int, b: int) -> list[tuple[str, str]]:
|
||||
entries, depth, start = [], 0, a
|
||||
for i in range(a, b):
|
||||
ch = masked[i]
|
||||
if ch == "{":
|
||||
depth += 1
|
||||
elif ch == "}":
|
||||
depth -= 1
|
||||
elif ch == "," and depth == 0:
|
||||
entries.append((text[start:i], masked[start:i]))
|
||||
start = i + 1
|
||||
entries.append((text[start:b], masked[start:b]))
|
||||
return [(t.strip(), m.strip()) for t, m in entries if m.strip()]
|
||||
|
||||
|
||||
def add_runtime_mod(text: str, module: str) -> str:
|
||||
line = f"pub mod {module};\n"
|
||||
if line in text:
|
||||
def attribute_lines_above(text: str, pos: int, indent: str) -> str:
|
||||
lines = text[:pos].split("\n")
|
||||
if lines and lines[-1] == "":
|
||||
lines = lines[:-1]
|
||||
block: list[str] = []
|
||||
while lines and re.match(r"^\s*#\[.*\]\s*$", lines[-1]):
|
||||
block.append(lines.pop().strip())
|
||||
return "".join(f"{indent}{line}\n" for line in reversed(block))
|
||||
|
||||
|
||||
class Stats:
|
||||
def __init__(self) -> None:
|
||||
self.plain: dict[str, int] = {}
|
||||
self.comment_hits = 0
|
||||
self.string_hits: list[str] = []
|
||||
self.grouped: list[dict] = []
|
||||
self.unqualified: list[dict] = []
|
||||
self.self_sites = 0
|
||||
self.self_skipped = 0
|
||||
self.self_files = 0
|
||||
self.residue = collections.defaultdict(list)
|
||||
self.super_root: list[dict] = []
|
||||
|
||||
|
||||
def split_groups(text: str, rel: str, bases: list[str], mods: set[str], dest_ident: str, stats: Stats) -> str:
|
||||
base_alt = "|".join(re.escape(b) for b in bases)
|
||||
group_re = re.compile(rf"(?m)^(?P<indent>[ \t]*)(?P<vis>pub(?:\([^)]*\))?[ \t]+)?use[ \t]+(?P<base>{base_alt})::\{{")
|
||||
if not group_re.search(text):
|
||||
return text
|
||||
lines = text.splitlines(keepends=True)
|
||||
mods = [i for i, l in enumerate(lines) if re.match(r"pub mod \w+;\n", l)]
|
||||
if not mods:
|
||||
return text.rstrip("\n") + "\n\n" + line
|
||||
insert = mods[-1] + 1
|
||||
for i in mods:
|
||||
if lines[i] > line:
|
||||
insert = i
|
||||
break
|
||||
lines.insert(insert, line)
|
||||
return "".join(lines)
|
||||
|
||||
|
||||
def rewrite_callers(modules: list[str], dry_run: bool) -> list[str]:
|
||||
alt = "|".join(re.escape(m) for m in modules)
|
||||
pattern = re.compile(rf"\bcrate::({alt})(?![A-Za-z0-9_])")
|
||||
touched: list[str] = []
|
||||
for rel in git("ls-files", "crates/tui/src", check=False).split():
|
||||
if not rel.endswith(".rs"):
|
||||
masked = mg.lex_mask(text)
|
||||
out, pos, changed = [], 0, False
|
||||
for m in group_re.finditer(masked):
|
||||
if m.start() < pos:
|
||||
continue
|
||||
path = REPO_ROOT / rel
|
||||
text = path.read_text(encoding="utf-8")
|
||||
updated = pattern.sub(lambda m: f"codewhale_runtime::{m.group(1)}", text)
|
||||
if updated != text:
|
||||
touched.append(rel)
|
||||
if not dry_run:
|
||||
path.write_text(updated, encoding="utf-8")
|
||||
return touched
|
||||
|
||||
|
||||
def import_unqualified(text: str, modules: list[str]) -> str:
|
||||
needed = [
|
||||
m
|
||||
for m in modules
|
||||
if re.search(rf"(?<![\w:]){re.escape(m)}::", text)
|
||||
and f"use codewhale_runtime::{m};" not in text
|
||||
]
|
||||
if not needed:
|
||||
open_i = m.end() - 1
|
||||
close_i = mg.match_brace(masked, open_i)
|
||||
semi = masked.find(";", close_i)
|
||||
if semi < 0:
|
||||
continue
|
||||
entries = split_entries(text, masked, open_i + 1, close_i)
|
||||
moved = [t for t, mk in entries if first_segment(mk) in mods]
|
||||
if not moved:
|
||||
continue
|
||||
kept = [t for t, mk in entries if first_segment(mk) not in mods]
|
||||
indent, vis, base = m.group("indent"), m.group("vis") or "", m.group("base")
|
||||
line = masked.count("\n", 0, m.start()) + 1
|
||||
stats.grouped.append({"file": rel, "line": line, "moved": [first_segment(mk) for t, mk in entries if t in moved]})
|
||||
moved_stmt = f"{indent}{vis}use {dest_ident}::{{{', '.join(moved)}}};"
|
||||
if kept:
|
||||
attrs = attribute_lines_above(text, m.start(), indent)
|
||||
new = f"{indent}{vis}use {base}::{{{', '.join(kept)}}};\n{attrs}{moved_stmt}"
|
||||
else:
|
||||
new = moved_stmt
|
||||
out.append(text[pos : m.start()])
|
||||
out.append(new)
|
||||
pos = semi + 1
|
||||
changed = True
|
||||
if not changed:
|
||||
return text
|
||||
anchor = re.search(r"^use codewhale_runtime::.*\n", text, re.M) or re.search(r"^use .*\n", text, re.M)
|
||||
lines = "".join(f"use codewhale_runtime::{m};\n" for m in sorted(needed))
|
||||
return text[: anchor.start()] + lines + text[anchor.start() :]
|
||||
out.append(text[pos:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def sweep_paths(module: str, dry_run: bool) -> list[str]:
|
||||
patterns = [
|
||||
(f"crates/tui/src/{module}", f"crates/runtime/src/{module}"),
|
||||
(f"tui/src/{module}", f"runtime/src/{module}"),
|
||||
]
|
||||
touched: list[str] = []
|
||||
for old, new in patterns:
|
||||
# The module as a whole path segment: `elapsed.rs`, `elapsed/`,
|
||||
# `elapsed::`, never `elapsed_time.rs`; `tui/src/x` only when it is not
|
||||
# already the tail of `crates/tui/src/x`.
|
||||
regex = re.compile(
|
||||
rf"(?<![A-Za-z0-9_/]){re.escape(old)}(?![A-Za-z0-9_])"
|
||||
if not old.startswith("crates/")
|
||||
else rf"{re.escape(old)}(?![A-Za-z0-9_])"
|
||||
)
|
||||
for rel in git("grep", "-l", "-F", old, check=False).split():
|
||||
if Path(rel).name.startswith("CHANGELOG"):
|
||||
def rewrite_paths(text: str, rel: str, bases: list[str], mods: set[str], dest_ident: str, stats: Stats) -> str:
|
||||
if not any(b + "::" in text for b in bases):
|
||||
return text
|
||||
text = split_groups(text, rel, bases, mods, dest_ident, stats)
|
||||
alt = "|".join(re.escape(m) for m in sorted(mods))
|
||||
base_alt = "|".join(re.escape(b) for b in bases)
|
||||
pat = re.compile(rf"(?<![\$\w])(?:{base_alt})::({alt})(?![A-Za-z0-9_])")
|
||||
hits = list(pat.finditer(text))
|
||||
if not hits:
|
||||
return text
|
||||
lines = set()
|
||||
for m in hits:
|
||||
lineno = text.count("\n", 0, m.start()) + 1
|
||||
lines.add(lineno)
|
||||
prefix = text[text.rfind("\n", 0, m.start()) + 1 : m.start()]
|
||||
if prefix.lstrip().startswith(("//", "*", "/*")) or "//" in prefix:
|
||||
stats.comment_hits += 1
|
||||
elif prefix.count('"') % 2 == 1:
|
||||
stats.string_hits.append(f"{rel}:{lineno}")
|
||||
stats.plain[rel] = len(lines)
|
||||
return pat.sub(lambda m: f"{dest_ident}::{m.group(1)}", text)
|
||||
|
||||
|
||||
def add_unqualified_imports(text: str, rel: str, mods: set[str], dest_ident: str, stats: Stats) -> str:
|
||||
masked = mg.lex_mask(text)
|
||||
prod = mg.strip_cfg_test(masked, [])
|
||||
alt = "|".join(re.escape(m) for m in sorted(mods))
|
||||
pat = re.compile(rf"(?<![\w:$])({alt})::")
|
||||
seen: dict[str, str] = {}
|
||||
for m in pat.finditer(masked):
|
||||
name = m.group(1)
|
||||
if f"use {dest_ident}::{name};" in text:
|
||||
continue
|
||||
line = masked.count("\n", 0, m.start()) + 1
|
||||
in_prod = prod[m.start() : m.end()] == m.group(0)
|
||||
stats.unqualified.append({"file": rel, "line": line, "module": name, "production": in_prod})
|
||||
if in_prod or name not in seen:
|
||||
seen[name] = "prod" if in_prod else "test"
|
||||
elif seen.get(name) != "prod":
|
||||
seen[name] = "test"
|
||||
if not seen:
|
||||
return text
|
||||
anchor = re.search(r"(?m)^use .*", masked)
|
||||
if not anchor:
|
||||
raise SplitError(f"{rel}: no top-level `use` to anchor imports for {sorted(seen)}")
|
||||
block = ""
|
||||
for name in sorted(seen):
|
||||
gate = "" if seen[name] == "prod" else "#[cfg(test)]\n"
|
||||
block += f"{gate}use {dest_ident}::{name};\n"
|
||||
return text[: anchor.start()] + block + text[anchor.start() :]
|
||||
|
||||
|
||||
def rewrite_moved(
|
||||
text: str, rel: str, dest_ident: str, resident: set[str], src_ident: str, mods: set[str], stats: Stats
|
||||
) -> str:
|
||||
masked = mg.lex_mask(text)
|
||||
pat = re.compile(rf"(?<![\w:$]){re.escape(dest_ident)}::")
|
||||
out, pos, sites = [], 0, 0
|
||||
for m in pat.finditer(text):
|
||||
if masked[m.start() : m.end()] != m.group(0):
|
||||
stats.self_skipped += 1
|
||||
continue
|
||||
out.append(text[pos : m.start()])
|
||||
out.append("crate::")
|
||||
pos = m.end()
|
||||
sites += 1
|
||||
if sites:
|
||||
out.append(text[pos:])
|
||||
text = "".join(out)
|
||||
stats.self_sites += sites
|
||||
stats.self_files += 1
|
||||
masked = mg.lex_mask(text)
|
||||
for m in re.finditer(r"(?<![\$\w])crate::(\w+)", masked):
|
||||
target = m.group(1)
|
||||
if target not in resident:
|
||||
line = masked.count("\n", 0, m.start()) + 1
|
||||
stats.residue[target].append(f"{rel}:{line}")
|
||||
for m in re.finditer(r"\buse\s+crate::\{", masked):
|
||||
close = mg.match_brace(masked, m.end() - 1)
|
||||
for _, path in mg.use_leaves(masked[m.end() : close]):
|
||||
if path[0] not in resident and path[0] not in BUILTIN_STD:
|
||||
line = masked.count("\n", 0, m.start()) + 1
|
||||
stats.residue[path[0]].append(f"{rel}:{line}")
|
||||
for pos_, name in mg.super_root_targets(masked, rel.split("/src/", 1)[-1]):
|
||||
if name not in resident and name not in BUILTIN_STD and name in mods | {"lib"}:
|
||||
stats.super_root.append({"file": rel, "line": masked.count("\n", 0, pos_) + 1, "target": name})
|
||||
return text
|
||||
|
||||
|
||||
class Mover:
|
||||
def __init__(self, args: argparse.Namespace) -> None:
|
||||
self.args = args
|
||||
self.mods = list(args.modules)
|
||||
self.modset = set(self.mods)
|
||||
self.src, self.dest = args.src_crate, args.dest_crate
|
||||
self.src_dir = f"crates/{self.src}/src"
|
||||
self.dest_dir = f"crates/{self.dest}/src"
|
||||
self.src_ident = sl.crate_ident(self.src)
|
||||
self.dest_ident = sl.crate_ident(self.dest)
|
||||
self.texts: dict[str, str] = {}
|
||||
self.fps: dict[str, Fingerprint] = {}
|
||||
self.new: dict[str, str | None] = {}
|
||||
self.stats = Stats()
|
||||
self.moves: dict[str, str] = {}
|
||||
self.other_crate_files: list[str] = []
|
||||
self.sweep_files: list[str] = []
|
||||
self.sweep_skipped: list[str] = []
|
||||
self.sweep_segmented = 0
|
||||
self.cfg_sites: list[dict] = []
|
||||
self.manifest: dict = {}
|
||||
self.extra_changes: list[str] = []
|
||||
|
||||
def load(self, rel: str) -> str:
|
||||
if rel in self.texts:
|
||||
return self.texts[rel]
|
||||
path = REPO_ROOT / rel
|
||||
with open(path, "rb") as fh:
|
||||
st = os.fstat(fh.fileno())
|
||||
data = fh.read()
|
||||
self.fps[rel] = Fingerprint(st.st_size, st.st_mtime_ns, st.st_ino, sl.sha256_bytes(data))
|
||||
self.texts[rel] = data.decode("utf-8")
|
||||
return self.texts[rel]
|
||||
|
||||
def plan_moves(self) -> None:
|
||||
for m in self.mods:
|
||||
files = sl.tracked_files(f"{self.src_dir}/{m}.rs", f"{self.src_dir}/{m}")
|
||||
files = [f for f in files if f == f"{self.src_dir}/{m}.rs" or f.startswith(f"{self.src_dir}/{m}/")]
|
||||
if not files:
|
||||
raise SplitError(f"no tracked {self.src_dir}/{m}.rs or {m}/")
|
||||
disk = REPO_ROOT / self.src_dir / m
|
||||
if disk.is_dir():
|
||||
tracked = set(files)
|
||||
loose = [
|
||||
str(p.relative_to(REPO_ROOT))
|
||||
for p in disk.rglob("*")
|
||||
if p.is_file() and str(p.relative_to(REPO_ROOT)) not in tracked
|
||||
]
|
||||
if loose:
|
||||
raise SplitError(f"untracked files under {self.src_dir}/{m}: {loose[:5]}")
|
||||
for f in files:
|
||||
dest = self.dest_dir + f[len(self.src_dir) :]
|
||||
if (REPO_ROOT / dest).exists():
|
||||
raise SplitError(f"destination {dest} already exists")
|
||||
self.moves[f] = dest
|
||||
|
||||
def resident_modules(self) -> set[str]:
|
||||
out = set(self.modset)
|
||||
base = REPO_ROOT / self.dest_dir
|
||||
for p in base.iterdir():
|
||||
if p.is_dir():
|
||||
out.add(p.name)
|
||||
elif p.suffix == ".rs" and p.stem not in ("lib", "main"):
|
||||
out.add(p.stem)
|
||||
return out
|
||||
|
||||
def run_plan(self) -> None:
|
||||
sl.require_git_repo()
|
||||
src_lib, dest_lib = f"{self.src_dir}/lib.rs", f"{self.dest_dir}/lib.rs"
|
||||
if not (REPO_ROOT / dest_lib).is_file():
|
||||
raise SplitError(f"{dest_lib} does not exist")
|
||||
self.plan_moves()
|
||||
resident = self.resident_modules()
|
||||
all_src = [f for f in sl.tracked_files(self.src_dir) if f.endswith(".rs")]
|
||||
staying = [f for f in all_src if f not in self.moves]
|
||||
bases = ["crate", self.src_ident]
|
||||
|
||||
lib_text = self.load(src_lib)
|
||||
for m in self.mods:
|
||||
lib_text = self.remove_mod_line(lib_text, m)
|
||||
dest_text = self.load(dest_lib)
|
||||
for m in self.mods:
|
||||
dest_text = self.add_dest_mod(dest_text, m)
|
||||
self.new[dest_lib] = dest_text
|
||||
|
||||
for rel in staying:
|
||||
text = lib_text if rel == src_lib else self.load(rel)
|
||||
new = rewrite_paths(text, rel, bases, self.modset, self.dest_ident, self.stats)
|
||||
if rel == src_lib:
|
||||
new = add_unqualified_imports(new, rel, self.modset, self.dest_ident, self.stats)
|
||||
masked_needed = "super::" in new and any(m in new for m in self.mods)
|
||||
if masked_needed:
|
||||
masked = mg.lex_mask(new)
|
||||
for pos_, name in mg.super_root_targets(masked, rel[len(self.src_dir) + 1 :]):
|
||||
if name in self.modset:
|
||||
self.stats.super_root.append({"file": rel, "line": masked.count("\n", 0, pos_) + 1, "target": name})
|
||||
if new != self.load(rel):
|
||||
self.new[rel] = new
|
||||
|
||||
internal_lines = internal_files = 0
|
||||
pat = re.compile(rf"(?<![\$\w])crate::({'|'.join(re.escape(m) for m in self.mods)})(?![A-Za-z0-9_])")
|
||||
for src_rel, dest_rel in self.moves.items():
|
||||
if not src_rel.endswith(".rs"):
|
||||
self.new[dest_rel] = self.load(src_rel)
|
||||
continue
|
||||
path = REPO_ROOT / rel
|
||||
text = path.read_text(encoding="utf-8")
|
||||
updated = regex.sub(new, text)
|
||||
text = self.load(src_rel)
|
||||
hit_lines = {text.count("\n", 0, m.start()) + 1 for m in pat.finditer(text)}
|
||||
if hit_lines:
|
||||
internal_files += 1
|
||||
internal_lines += len(hit_lines)
|
||||
self.new[dest_rel] = rewrite_moved(
|
||||
text, dest_rel, self.dest_ident, resident, self.src_ident, self.modset, self.stats
|
||||
)
|
||||
self.internal = (internal_files, internal_lines)
|
||||
|
||||
self.rewrite_other_crates(set(staying) | set(self.moves) | {dest_lib})
|
||||
self.sweep()
|
||||
self.plan_cfg()
|
||||
self.plan_manifests()
|
||||
|
||||
def remove_mod_line(self, text: str, module: str) -> str:
|
||||
pattern = re.compile(rf"^(?:pub(?:\([^)]*\))?\s+)?mod\s+{re.escape(module)}\s*;[ \t]*(?://[^\n]*)?\n", re.M)
|
||||
match = pattern.search(text)
|
||||
if not match:
|
||||
raise SplitError(f"no `mod {module};` line in {self.src_dir}/lib.rs")
|
||||
before = text[: match.start()].rstrip("\n").rsplit("\n", 1)[-1].strip()
|
||||
if before.startswith("#["):
|
||||
raise SplitError(f"`mod {module};` carries an attribute ({before}); move it by hand")
|
||||
return text[: match.start()] + text[match.end() :]
|
||||
|
||||
@staticmethod
|
||||
def add_dest_mod(text: str, module: str) -> str:
|
||||
line = f"pub mod {module};\n"
|
||||
if line in text:
|
||||
raise SplitError(f"destination lib already declares `pub mod {module};`")
|
||||
lines = text.splitlines(keepends=True)
|
||||
mods = [i for i, l in enumerate(lines) if re.match(r"pub mod \w+;\n", l)]
|
||||
if not mods:
|
||||
return text.rstrip("\n") + "\n\n" + line
|
||||
insert = mods[-1] + 1
|
||||
for i in mods:
|
||||
if lines[i] > line:
|
||||
insert = i
|
||||
break
|
||||
lines.insert(insert, line)
|
||||
return "".join(lines)
|
||||
|
||||
def rewrite_other_crates(self, already: set[str]) -> None:
|
||||
base = self.src_ident
|
||||
files: set[str] = set()
|
||||
for m in self.mods:
|
||||
files.update(sl.grep_files(f"{base}::{m}", "*.rs"))
|
||||
files = {f for f in files if f not in already and not f.startswith("vendor/")}
|
||||
bases = [base]
|
||||
for rel in sorted(files):
|
||||
text = self.load(rel)
|
||||
new = rewrite_paths(text, rel, bases, self.modset, self.dest_ident, self.stats)
|
||||
if new != text:
|
||||
self.new[rel] = new
|
||||
self.other_crate_files.append(rel)
|
||||
|
||||
def sweep(self) -> None:
|
||||
regexes = sl.sweep_regexes(self.src, self.dest, self.mods)
|
||||
patterns = []
|
||||
for m in self.mods:
|
||||
patterns += [f"{self.src}/src/{m}"]
|
||||
candidates: set[str] = set()
|
||||
for pat in patterns:
|
||||
candidates.update(sl.grep_files(pat))
|
||||
for quote in ('"', "'"):
|
||||
candidates.update(
|
||||
sl.grep_files(f"{quote}{self.src}{quote}", "*.py", "*.mjs", "*.js", "*.ts", "*.sh", "*.rs")
|
||||
)
|
||||
for rel in sorted(candidates):
|
||||
if sl.sweep_skipped(rel):
|
||||
self.sweep_skipped.append(rel)
|
||||
continue
|
||||
if rel in self.moves:
|
||||
key, text = self.moves[rel], self.new[self.moves[rel]]
|
||||
elif rel in self.new:
|
||||
key, text = rel, self.new[rel]
|
||||
else:
|
||||
key = rel
|
||||
try:
|
||||
text = self.load(rel)
|
||||
except UnicodeDecodeError:
|
||||
self.sweep_skipped.append(rel)
|
||||
continue
|
||||
updated = text
|
||||
for rx, repl in regexes:
|
||||
updated = rx.sub(repl, updated)
|
||||
if updated != text:
|
||||
touched.append(rel)
|
||||
if not dry_run:
|
||||
path.write_text(updated, encoding="utf-8")
|
||||
return sorted(set(touched))
|
||||
self.new[key] = updated
|
||||
self.sweep_files.append(rel)
|
||||
self.sweep_files = sorted(set(self.sweep_files))
|
||||
self.sweep_skipped = sorted(set(self.sweep_skipped))
|
||||
regenerated = [p for p in GENERATED_BY_DERIVE_FACTS if (REPO_ROOT / p).exists()]
|
||||
self.derive_needed = any(f.startswith(("docs/", "web/")) for f in self.sweep_files)
|
||||
self.derive_outputs = regenerated
|
||||
|
||||
def plan_cfg(self) -> None:
|
||||
item_re = re.compile(
|
||||
r"(?:pub(?:\([^)]*\)\s+)?)?(?:async\s+|unsafe\s+|const\s+|extern\s+(?:\"[^\"]*\"\s+)?)*"
|
||||
r"(?P<kind>mod|fn|struct|enum|trait|type|static|const|use|impl|macro_rules!)\s*(?P<name>\w+)?"
|
||||
)
|
||||
sites: list[dict] = []
|
||||
wanted: set[str] = set()
|
||||
edits_by_file: dict[str, list[tuple[int, int, dict]]] = {}
|
||||
for dest_rel in sorted(self.moves.values()):
|
||||
if not dest_rel.endswith(".rs"):
|
||||
continue
|
||||
text = self.new[dest_rel]
|
||||
if "cfg(" not in text:
|
||||
continue
|
||||
masked = mg.lex_mask(text)
|
||||
for m in re.finditer(r"#\[cfg\(", masked):
|
||||
depth, k = 1, m.end()
|
||||
while k < len(text) and depth:
|
||||
depth += {"(": 1, ")": -1}.get(masked[k], 0)
|
||||
k += 1
|
||||
if masked[k : k + 1] != "]":
|
||||
continue
|
||||
expr = text[m.end() : k - 1].strip()
|
||||
atoms = set(re.findall(r"[A-Za-z_][\w-]*", expr)) | set(re.findall(r'"([^"]*)"', expr))
|
||||
if "test" not in atoms and "test-support" not in atoms:
|
||||
continue
|
||||
line = text.count("\n", 0, m.start()) + 1
|
||||
j = k + 1
|
||||
while True:
|
||||
skip = re.match(r"\s*(?:#\[[^\]]*\]|///[^\n]*|//![^\n]*)", text[j:])
|
||||
if not skip:
|
||||
break
|
||||
j += skip.end()
|
||||
j += len(masked[j:]) - len(masked[j:].lstrip())
|
||||
im = item_re.match(masked, j)
|
||||
kind = im.group("kind") if im else "?"
|
||||
name = im.group("name") if im else None
|
||||
names: list[str] = []
|
||||
if kind == "impl":
|
||||
brace = masked.find("{", j)
|
||||
names = re.findall(r"\bfn\s+(\w+)", masked[j : mg.match_brace(masked, brace)]) if brace >= 0 else []
|
||||
name = "impl{" + ",".join(names[:4]) + ("..." if len(names) > 4 else "") + "}"
|
||||
elif kind == "use":
|
||||
stmt_end = masked.find(";", j)
|
||||
names = [leaf for leaf, _ in mg.use_leaves(masked[j + 3 : stmt_end])]
|
||||
name = ",".join(names[:4])
|
||||
elif name:
|
||||
names = [name]
|
||||
if expr == "test":
|
||||
cls = "test_mod" if kind == "mod" else "test_item"
|
||||
elif expr == "not(test)":
|
||||
cls = "fork_not_test"
|
||||
else:
|
||||
cls = "compound"
|
||||
site = {"file": dest_rel, "line": line, "cfg": expr, "class": cls, "item": f"{kind} {name or ''}".strip(), "used_outside": False, "_names": names}
|
||||
sites.append(site)
|
||||
if cls == "test_item":
|
||||
wanted.update(names)
|
||||
edits_by_file.setdefault(dest_rel, []).append((m.start(), k + 1, site))
|
||||
used: set[str] = set()
|
||||
if wanted:
|
||||
rx = re.compile(r"\b(?:" + "|".join(re.escape(n) for n in sorted(wanted)) + r")\b")
|
||||
moved_dests = set(self.moves.values())
|
||||
for f in sl.tracked_files("crates", "tests", "pet"):
|
||||
if not f.endswith(".rs") or f in self.moves or f in moved_dests:
|
||||
continue
|
||||
try:
|
||||
used.update(rx.findall(self.load(f)))
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
for site in sites:
|
||||
site["used_outside"] = any(n in used for n in site.pop("_names"))
|
||||
self.cfg_sites = sites
|
||||
if self.args.apply_cfg:
|
||||
for dest_rel, edits in edits_by_file.items():
|
||||
text = self.new[dest_rel]
|
||||
for a_, b_, site in sorted(edits, reverse=True):
|
||||
if site["used_outside"]:
|
||||
text = text[:a_] + '#[cfg(any(test, feature = "test-support"))]' + text[b_:]
|
||||
self.new[dest_rel] = text
|
||||
|
||||
def plan_manifests(self) -> None:
|
||||
def deps_of(crate_dir: str) -> tuple[set[str], set[str], set[str]]:
|
||||
data = tomllib.loads((REPO_ROOT / "crates" / crate_dir / "Cargo.toml").read_text(encoding="utf-8"))
|
||||
normal, dev, feats = set(), set(), set(data.get("features", {}))
|
||||
def collect(tbl, into):
|
||||
for k, v in tbl.get("dependencies", {}).items():
|
||||
into.add(k.replace("-", "_"))
|
||||
if isinstance(v, dict) and "package" in v:
|
||||
into.add(v["package"].replace("-", "_"))
|
||||
for k, v in tbl.get("build-dependencies", {}).items():
|
||||
into.add(k.replace("-", "_"))
|
||||
collect(data, normal)
|
||||
for tbl in data.get("target", {}).values():
|
||||
collect(tbl, normal)
|
||||
for k in data.get("dev-dependencies", {}):
|
||||
dev.add(k.replace("-", "_"))
|
||||
return normal, dev, feats
|
||||
|
||||
src_normal, src_dev, _ = deps_of(self.src)
|
||||
dest_normal, dest_dev, dest_feats = deps_of(self.dest)
|
||||
used_prod: set[str] = set()
|
||||
used_any: set[str] = set()
|
||||
features: set[str] = set()
|
||||
for dest_rel in self.moves.values():
|
||||
if not dest_rel.endswith(".rs"):
|
||||
continue
|
||||
text = self.new[dest_rel]
|
||||
masked = mg.lex_mask(text)
|
||||
prod = mg.strip_cfg_test(masked, [])
|
||||
roots = set(re.findall(r"(?<![\w:$])([a-z_][a-z0-9_]*)::", masked))
|
||||
prod_roots = set(re.findall(r"(?<![\w:$])([a-z_][a-z0-9_]*)::", prod))
|
||||
is_test_file = mg.is_test_path(dest_rel)
|
||||
used_any |= roots
|
||||
if not is_test_file:
|
||||
used_prod |= prod_roots
|
||||
features |= set(re.findall(r'feature\s*=\s*"([^"]+)"', text))
|
||||
resident_names = self.resident_modules()
|
||||
wanted_prod = {r for r in used_prod if r in src_normal and r not in resident_names and r not in BUILTIN_STD}
|
||||
wanted_any = {r for r in used_any if (r in src_normal or r in src_dev) and r not in resident_names and r not in BUILTIN_STD}
|
||||
self.manifest = {
|
||||
"dest_missing_normal": sorted(wanted_prod - dest_normal),
|
||||
"dest_missing_dev_or_normal": sorted((wanted_any - wanted_prod) - dest_normal - dest_dev),
|
||||
"features_used_not_declared_in_dest": sorted(f for f in features if f not in dest_feats),
|
||||
"features_used": sorted(features),
|
||||
}
|
||||
missing_consumers = []
|
||||
for rel in self.other_crate_files:
|
||||
parts = rel.split("/")
|
||||
if parts[0] != "crates" or len(parts) < 3:
|
||||
continue
|
||||
crate = parts[1]
|
||||
if crate == self.dest:
|
||||
continue
|
||||
normal, dev, _ = deps_of(crate)
|
||||
ident = self.dest_ident
|
||||
kind = "dev" if len(parts) > 2 and parts[2] in ("tests", "benches", "examples") else "normal"
|
||||
have = ident in normal or (kind == "dev" and ident in dev)
|
||||
if not have:
|
||||
missing_consumers.append({"crate": crate, "file": rel, "needs": kind})
|
||||
self.manifest["consumers_missing_dependency"] = sorted(
|
||||
{(c["crate"], c["needs"]) for c in missing_consumers}
|
||||
)
|
||||
|
||||
def rewrite_set(self) -> list[str]:
|
||||
paths = set(self.new) | set(self.moves)
|
||||
return sorted(paths)
|
||||
|
||||
def check_clean(self) -> dict:
|
||||
rewrite = self.rewrite_set()
|
||||
existing = [p for p in rewrite if (REPO_ROOT / p).exists()]
|
||||
dirty = sorted(sl.dirty_paths() & set(rewrite))
|
||||
young = [(p, round(a, 1)) for p, a in sl.recently_modified(existing, self.args.quiet_minutes)]
|
||||
return {"dirty": dirty, "recent": young}
|
||||
|
||||
def changes(self) -> list[Change]:
|
||||
out: list[Change] = []
|
||||
fmt_items: dict[str, tuple[str | None, str]] = {}
|
||||
final: dict[str, str | None] = dict(self.new)
|
||||
for src in self.moves:
|
||||
final[src] = None
|
||||
for rel, text in final.items():
|
||||
if text is None or not rel.endswith(".rs") or self.args.no_format:
|
||||
continue
|
||||
base_src = next((s for s, d in self.moves.items() if d == rel), rel)
|
||||
base = self.texts.get(base_src)
|
||||
if base is not None and base == text:
|
||||
continue
|
||||
fmt_items[rel] = (base, text)
|
||||
if fmt_items:
|
||||
formatted, unstable = sl.format_many(fmt_items)
|
||||
self.unformatted_base = unstable
|
||||
for rel, text in formatted.items():
|
||||
final[rel] = text
|
||||
else:
|
||||
self.unformatted_base = []
|
||||
for rel, text in sorted(final.items()):
|
||||
if rel in self.moves.values():
|
||||
src = next(s for s, d in self.moves.items() if d == rel)
|
||||
out.append(Change(rel, text, None, mode_from=src))
|
||||
else:
|
||||
out.append(Change(rel, text, self.fps.get(rel)))
|
||||
return out
|
||||
|
||||
def report(self, clean: dict | None) -> dict:
|
||||
s = self.stats
|
||||
internal_files, internal_lines = self.internal
|
||||
staying = {r: n for r, n in s.plain.items() if r.startswith(self.src_dir + "/")}
|
||||
other = {r: n for r, n in s.plain.items() if not r.startswith(self.src_dir + "/")}
|
||||
residue = {k: {"sites": len(v), "first": v[:3]} for k, v in sorted(s.residue.items(), key=lambda kv: -len(kv[1]))}
|
||||
return {
|
||||
"modules": self.mods,
|
||||
"src_crate": self.src,
|
||||
"dest_crate": self.dest,
|
||||
"files_moved": len(self.moves),
|
||||
"caller_rewrites": {
|
||||
"files": len(staying),
|
||||
"lines": sum(staying.values()),
|
||||
"in_comments": s.comment_hits,
|
||||
"in_string_literals": s.string_hits,
|
||||
},
|
||||
"other_crate_rewrites": {"files": len(other), "lines": sum(other.values())},
|
||||
"internal_refs_not_rewritten": {"files": internal_files, "lines": internal_lines},
|
||||
"legacy_dry_run_equivalent": {
|
||||
"files": len(staying) + internal_files,
|
||||
"lines": sum(staying.values()) + internal_lines,
|
||||
},
|
||||
"grouped_imports": s.grouped,
|
||||
"unqualified_in_source_lib": s.unqualified,
|
||||
"self_paths": {"sites": s.self_sites, "files": s.self_files, "skipped_in_comments_or_strings": s.self_skipped},
|
||||
"other_crate_files": self.other_crate_files,
|
||||
"sweep": {
|
||||
"files": self.sweep_files,
|
||||
"skipped": self.sweep_skipped,
|
||||
"must_sweep_covered": {p: (p in self.sweep_files) for p in MUST_SWEEP},
|
||||
"run_derive_facts": self.derive_needed,
|
||||
"derive_facts_outputs": self.derive_outputs,
|
||||
},
|
||||
"residue_moved_to_staying": residue,
|
||||
"super_root_residue": s.super_root,
|
||||
"manifest_plan": self.manifest,
|
||||
"cfg_plan": {
|
||||
"applied": bool(self.args.apply_cfg),
|
||||
"sites": len(self.cfg_sites),
|
||||
"by_class": dict(collections.Counter(c["class"] for c in self.cfg_sites)),
|
||||
"used_outside": sum(1 for c in self.cfg_sites if c["used_outside"] and c["class"] == "test_item"),
|
||||
"detail": self.cfg_sites,
|
||||
},
|
||||
"rewrite_set": self.rewrite_set(),
|
||||
"clean": clean,
|
||||
}
|
||||
|
||||
|
||||
def run_mover(args: argparse.Namespace) -> int:
|
||||
mover = Mover(args)
|
||||
mover.run_plan()
|
||||
clean = mover.check_clean()
|
||||
if args.require_clean and (clean["dirty"] or clean["recent"]):
|
||||
lines = [f"dirty: {p}" for p in clean["dirty"]] + [f"modified {a} min ago: {p}" for p, a in clean["recent"]]
|
||||
raise SplitError("--require-clean refused; the rewrite set overlaps live work:\n " + "\n ".join(lines), 5)
|
||||
changes = None if args.dry_run else mover.changes()
|
||||
rep = mover.report(clean)
|
||||
if args.json:
|
||||
json.dump(rep, sys.stdout, indent=1)
|
||||
print()
|
||||
else:
|
||||
print_report(rep, args.dry_run)
|
||||
if args.dry_run:
|
||||
return 0
|
||||
jdir = Path(args.journal_dir) if args.journal_dir else sl.default_journal_dir("move-" + "-".join(args.modules)[:40])
|
||||
sl.apply_changes(changes, jdir, args.pause_after_plan, args.pause_mid_write, log)
|
||||
log(f"applied {len(changes)} change(s); journal {jdir}")
|
||||
if args.derive_facts:
|
||||
derive_facts(jdir)
|
||||
return 0
|
||||
|
||||
|
||||
def derive_facts(jdir: Path) -> None:
|
||||
before = {p: (REPO_ROOT / p).read_bytes() if (REPO_ROOT / p).exists() else None for p in GENERATED_BY_DERIVE_FACTS}
|
||||
proc = subprocess.run(["node", "web/scripts/derive-facts.mjs"], cwd=REPO_ROOT, capture_output=True, text=True)
|
||||
if proc.returncode != 0:
|
||||
log(f"derive-facts failed: {proc.stderr.strip()[:500]}; generated files may be stale")
|
||||
return
|
||||
pre = jdir / "derive-pre"
|
||||
pre.mkdir(exist_ok=True)
|
||||
changed = []
|
||||
for p, old in before.items():
|
||||
new = (REPO_ROOT / p).read_bytes() if (REPO_ROOT / p).exists() else None
|
||||
if new != old:
|
||||
changed.append(p)
|
||||
if old is not None:
|
||||
(pre / p.replace("/", "__")).write_bytes(old)
|
||||
log("derive-facts regenerated: " + (", ".join(changed) if changed else "nothing changed") + " (add these to the rewrite set; pre-images in " + str(pre) + ")")
|
||||
|
||||
|
||||
def print_report(rep: dict, dry: bool) -> None:
|
||||
cr = rep["caller_rewrites"]
|
||||
print(f"modules: {' '.join(rep['modules'])} {rep['src_crate']} -> {rep['dest_crate']} files moved: {rep['files_moved']}{' (dry run)' if dry else ''}")
|
||||
print(f"callers rewritten: {cr['files']} files, {cr['lines']} lines (comments {cr['in_comments']}, string literals {len(cr['in_string_literals'])})")
|
||||
ir = rep["internal_refs_not_rewritten"]
|
||||
leg = rep["legacy_dry_run_equivalent"]
|
||||
print(f"inside the moved set (kept as crate::): {ir['files']} files, {ir['lines']} lines; old-tool dry-run equivalent: {leg['files']} files, {leg['lines']} lines")
|
||||
print(f"grouped imports split: {len(rep['grouped_imports'])}; unqualified in source lib: {len(rep['unqualified_in_source_lib'])}")
|
||||
sp = rep["self_paths"]
|
||||
print(f"self paths rewritten to crate::: {sp['sites']} in {sp['files']} files (skipped in comments/strings: {sp['skipped_in_comments_or_strings']})")
|
||||
oc = rep["other_crate_rewrites"]
|
||||
print(f"other crates' files rewritten: {oc['files']} files, {oc['lines']} lines")
|
||||
sw = rep["sweep"]
|
||||
print(f"path sweep: {len(sw['files'])} files ({len(sw['skipped'])} skipped by the skip list); derive-facts needed: {sw['run_derive_facts']}")
|
||||
print(f"residue moved->staying: {sum(v['sites'] for v in rep['residue_moved_to_staying'].values())} sites in {len(rep['residue_moved_to_staying'])} modules; super-root residue: {len(rep['super_root_residue'])}")
|
||||
print(f"cfg sites: {rep['cfg_plan']['sites']} {rep['cfg_plan']['by_class']}; manifest: {json.dumps(rep['manifest_plan'])}")
|
||||
c = rep["clean"]
|
||||
print(f"rewrite set: {len(rep['rewrite_set'])} paths; dirty overlap: {len(c['dirty'])}; recent (<min): {len(c['recent'])}")
|
||||
|
||||
|
||||
def restore(path: str) -> int:
|
||||
sl.require_git_repo()
|
||||
problems = sl.rollback(Path(path))
|
||||
for p in problems:
|
||||
log(f"restore problem: {p}")
|
||||
log(f"restored pre-images from {path}")
|
||||
return 1 if problems else 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
parser.add_argument("modules", nargs="+")
|
||||
parser.add_argument("modules", nargs="*")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--require-clean", action="store_true")
|
||||
parser.add_argument("--quiet-minutes", type=float, default=15.0)
|
||||
parser.add_argument("--dest-crate", default="runtime")
|
||||
parser.add_argument("--src-crate", default="tui")
|
||||
parser.add_argument("--apply-cfg", action="store_true")
|
||||
parser.add_argument("--resolve-reexports", action="store_true")
|
||||
parser.add_argument("--journal-dir")
|
||||
parser.add_argument("--pause-after-plan", type=float, default=0.0, help="seconds to wait between plan and first write (lets a test change a file)")
|
||||
parser.add_argument("--pause-mid-write", type=float, default=0.0, help="seconds to wait once half the writes are done (exercises the restore path)")
|
||||
parser.add_argument("--restore", metavar="JOURNAL_DIR")
|
||||
parser.add_argument("--no-format", action="store_true")
|
||||
parser.add_argument("--derive-facts", action="store_true")
|
||||
args = parser.parse_args(argv)
|
||||
if args.restore:
|
||||
return restore(args.restore)
|
||||
if not args.modules:
|
||||
parser.error("name at least one module")
|
||||
if args.resolve_reexports:
|
||||
import reexports
|
||||
|
||||
if not RUNTIME_LIB.is_file():
|
||||
raise SystemExit("move-modules: crates/runtime/src/lib.rs does not exist yet")
|
||||
tui_lib = TUI_LIB.read_text(encoding="utf-8")
|
||||
runtime_lib = RUNTIME_LIB.read_text(encoding="utf-8")
|
||||
for module in args.modules:
|
||||
paths = module_paths(module)
|
||||
tui_lib, found = remove_mod_line(tui_lib, module)
|
||||
if not found:
|
||||
raise SystemExit(f"move-modules: no `mod {module};` line in crates/tui/src/lib.rs")
|
||||
runtime_lib = add_runtime_mod(runtime_lib, module)
|
||||
if not args.dry_run:
|
||||
for path in paths:
|
||||
git("mv", str(path.relative_to(REPO_ROOT)), str((RUNTIME_SRC / path.name).relative_to(REPO_ROOT)))
|
||||
if not args.dry_run:
|
||||
tui_lib = import_unqualified(tui_lib, args.modules)
|
||||
TUI_LIB.write_text(tui_lib, encoding="utf-8")
|
||||
RUNTIME_LIB.write_text(runtime_lib, encoding="utf-8")
|
||||
for rel in rewrite_callers(args.modules, args.dry_run):
|
||||
print(f"callers rewritten: {rel}")
|
||||
swept: set[str] = set()
|
||||
for module in args.modules:
|
||||
swept.update(sweep_paths(module, args.dry_run))
|
||||
for rel in sorted(swept):
|
||||
print(f"path sweep: {rel}")
|
||||
print(f"moved {len(args.modules)} module(s){' (dry run)' if args.dry_run else ''}")
|
||||
return 0
|
||||
return reexports.run(args, sl, mg)
|
||||
return run_mover(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
sys.exit(sl.run_main(main))
|
||||
@@ -27,11 +27,29 @@ mg = importlib.util.module_from_spec(_spec)
|
||||
sys.modules["module_graph"] = mg
|
||||
_spec.loader.exec_module(mg)
|
||||
|
||||
_cl_spec = importlib.util.spec_from_file_location("count_lines", Path(__file__).with_name("count_lines.py"))
|
||||
cl = importlib.util.module_from_spec(_cl_spec)
|
||||
_cl_spec.loader.exec_module(cl)
|
||||
|
||||
ROOT = mg.ROOT_ITEMS
|
||||
|
||||
|
||||
def analyse() -> dict:
|
||||
tui, runtime, refs = mg.load_graph()
|
||||
def strict_lines(src_dirs: list[Path]) -> collections.Counter:
|
||||
strict: collections.Counter = collections.Counter()
|
||||
for src in src_dirs:
|
||||
crate_dir = src.parent
|
||||
if not crate_dir.is_dir():
|
||||
continue
|
||||
_, res = cl.classify_crate(str(crate_dir))
|
||||
for path, (prod, _test, _other) in res.items():
|
||||
rel = Path(path).relative_to(src) if Path(path).is_relative_to(src) else None
|
||||
if rel is not None:
|
||||
strict[mg.top_module(rel.as_posix())] += prod
|
||||
return strict
|
||||
|
||||
|
||||
def analyse(tui_src: Path = mg.TUI_SRC, runtime_src: Path = mg.RUNTIME_SRC, non_test: bool = False) -> dict:
|
||||
tui, runtime, refs = mg.load_graph(tui_src, runtime_src)
|
||||
resident = set(runtime.modules)
|
||||
modules = (tui.modules | runtime.modules) | {ROOT}
|
||||
prod_lines: collections.Counter = collections.Counter()
|
||||
@@ -42,6 +60,7 @@ def analyse() -> dict:
|
||||
module = mg.top_module(rel)
|
||||
bucket = test_lines if mg.file_is_test(rel, exact, prefixes, crate.production_files) else prod_lines
|
||||
bucket[module] += src.count("\n") + 1
|
||||
strict = strict_lines([tui_src, runtime_src]) if non_test else None
|
||||
out: dict[str, collections.Counter] = collections.defaultdict(collections.Counter)
|
||||
inbound: dict[str, collections.Counter] = collections.defaultdict(collections.Counter)
|
||||
for ref in refs:
|
||||
@@ -76,6 +95,7 @@ def analyse() -> dict:
|
||||
"resident": m in resident,
|
||||
"prod_lines": prod_lines[m],
|
||||
"test_lines": test_lines[m],
|
||||
**({"strict_lines": strict[m]} if strict is not None else {}),
|
||||
"out": len(out[m]),
|
||||
"in": len(inbound[m]),
|
||||
"scc": len(comp_members[c]),
|
||||
@@ -163,17 +183,20 @@ def strongly_connected(graph: dict[str, set[str]]) -> dict[str, int]:
|
||||
def render(result: dict, only: str | None) -> str:
|
||||
rows = [r for r in result["rows"] if not only or r["module"] == only]
|
||||
rows.sort(key=lambda r: (r["ui"], r["wave"], -r["prod_lines"]))
|
||||
has_strict = any("strict_lines" in r for r in rows)
|
||||
strict_head = f"{'strict':>9}" if has_strict else ""
|
||||
lines = [
|
||||
f"largest strongly connected components: {result['scc_sizes']}",
|
||||
f"largest SCC ({len(result['largest_scc'])} modules): {', '.join(result['largest_scc'])}",
|
||||
"",
|
||||
f"{'module':<26}{'wave':>5}{'prod':>9}{'test':>9}{'out':>5}{'in':>5}{'scc':>5} {'root':>5} blockers (module:refs)",
|
||||
f"{'module':<26}{'wave':>5}{'prod':>9}{strict_head}{'test':>9}{'out':>5}{'in':>5}{'scc':>5} {'root':>5} blockers (module:refs)",
|
||||
]
|
||||
for r in rows:
|
||||
tag = "UI " if r["ui"] else ("rt " if r["resident"] else " ")
|
||||
blockers = ", ".join(f"{d}:{n}" for d, n in r["blockers"][:5])
|
||||
strict_cell = f"{r.get('strict_lines', 0):>9}" if has_strict else ""
|
||||
lines.append(
|
||||
f"{tag}{r['module']:<23}{r['wave']:>5}{r['prod_lines']:>9}{r['test_lines']:>9}"
|
||||
f"{tag}{r['module']:<23}{r['wave']:>5}{r['prod_lines']:>9}{strict_cell}{r['test_lines']:>9}"
|
||||
f"{r['out']:>5}{r['in']:>5}{r['scc']:>5} {r['to_root']:>5} {blockers}"
|
||||
)
|
||||
free = [r for r in result["rows"] if not r["ui"] and r["wave"] == 0 and not r["resident"] and r["to_root"] == 0]
|
||||
@@ -186,8 +209,11 @@ def main() -> int:
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--module")
|
||||
parser.add_argument("--scc-edges", type=int, metavar="N", help="print the N heaviest edges inside the largest cycle")
|
||||
parser.add_argument("--non-test", action="store_true", help="add a strict production column (inline cfg(test) code excluded)")
|
||||
parser.add_argument("--tui-src", type=Path, default=mg.TUI_SRC, help="source directory of the crate being split")
|
||||
parser.add_argument("--runtime-src", type=Path, default=mg.RUNTIME_SRC, help="source directory of the destination crate")
|
||||
args = parser.parse_args()
|
||||
result = analyse()
|
||||
result = analyse(args.tui_src, args.runtime_src, args.non_test)
|
||||
if args.scc_edges:
|
||||
print(scc_edges(result, args.scc_edges, 5))
|
||||
return 0
|
||||
|
||||
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env python3
|
||||
"""G11 path-coupling preflight for the crate split (whole repository, cargo free).
|
||||
|
||||
python3 scripts/split/preflight.py config oauth settings # one window
|
||||
python3 scripts/split/preflight.py --json config oauth
|
||||
python3 scripts/split/preflight.py # inventory only
|
||||
|
||||
Resolves every coupling that `cargo check` cannot see against the old-to-new
|
||||
path map of the named modules (`crates/<src>/src/<m>.rs` and `<m>/` move to
|
||||
`crates/<dest>/src/`):
|
||||
|
||||
* `include_str!` / `include_bytes!` / `include!` and `#[path]` anywhere in the
|
||||
repository (`crates`, `pet`, `tests`, ...): the target must still resolve
|
||||
from the includer, whether one, both or neither moves;
|
||||
* `docs/features.toml` owner paths (`tests/feature_registry.rs` requires them
|
||||
to exist);
|
||||
* path strings in tracked text files (`crates/tui/src/<m>`, `tui/src/<m>`,
|
||||
segmented `"tui" / "src" / "<m>"`): fixed by the mover's sweep, except in the
|
||||
sweep's skip list, which are reported as residue;
|
||||
* manifest-relative and `../src/<m>` references from inside the source crate
|
||||
(tests, scripts, assets) that no sweep rewrites;
|
||||
* CI test identities (`<m>::` names in workflows and scripts).
|
||||
|
||||
Exit status is 1 when any residue remains, so the run is a gate; the residue
|
||||
lines say which file to edit by hand for the window.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import splitlib as sl
|
||||
from splitlib import REPO_ROOT
|
||||
|
||||
INCLUDE_RE = re.compile(r"\b(include_str|include_bytes|include)!\s*\(")
|
||||
PATH_ATTR_RE = re.compile(r'#\s*\[\s*path\s*=\s*"([^"]+)"\s*\]')
|
||||
STRING_RE = re.compile(r'"((?:\\.|[^"\\])*)"')
|
||||
|
||||
|
||||
def balanced(text: str, open_i: int) -> int:
|
||||
depth = 0
|
||||
for i in range(open_i, len(text)):
|
||||
if text[i] == "(":
|
||||
depth += 1
|
||||
elif text[i] == ")":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return i
|
||||
return len(text) - 1
|
||||
|
||||
|
||||
class Preflight:
|
||||
def __init__(self, mods: list[str], src: str, dest: str):
|
||||
self.mods, self.src, self.dest = mods, src, dest
|
||||
self.map: dict[str, str] = {}
|
||||
src_dir = f"crates/{src}/src"
|
||||
for m in mods:
|
||||
for rel in sl.tracked_files(f"{src_dir}/{m}.rs", f"{src_dir}/{m}"):
|
||||
if rel == f"{src_dir}/{m}.rs" or rel.startswith(f"{src_dir}/{m}/"):
|
||||
self.map[rel] = f"crates/{dest}/src" + rel[len(src_dir) :]
|
||||
self.rows: list[dict] = []
|
||||
|
||||
def add(self, kind: str, status: str, file: str, line: int, detail: str, residue: bool) -> None:
|
||||
self.rows.append({"kind": kind, "status": status, "file": file, "line": line, "detail": detail, "residue": residue})
|
||||
|
||||
def resolve(self, includer: str, literal: str, manifest_dir: str | None = None) -> str:
|
||||
base = manifest_dir if manifest_dir is not None else os.path.dirname(includer)
|
||||
return os.path.normpath(os.path.join(base, literal)) if not literal.startswith("/") else literal
|
||||
|
||||
def includes(self) -> None:
|
||||
files = set()
|
||||
for pat in ("include_str!", "include_bytes!", "include!(", "#[path", "# [path"):
|
||||
files.update(sl.grep_files(pat, "*.rs"))
|
||||
for rel in sorted(files):
|
||||
if rel.startswith("vendor/"):
|
||||
continue
|
||||
try:
|
||||
text = sl.read_text(rel)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
masked = None
|
||||
crate_dir = None
|
||||
parts = rel.split("/")
|
||||
if parts[0] == "crates" and len(parts) > 2:
|
||||
crate_dir = "/".join(parts[:2])
|
||||
entries: list[tuple[int, str, str | None, str]] = []
|
||||
for m in INCLUDE_RE.finditer(text):
|
||||
prefix = text[text.rfind("\n", 0, m.start()) + 1 : m.start()]
|
||||
if prefix.lstrip().startswith("//"):
|
||||
continue
|
||||
end = balanced(text, m.end() - 1)
|
||||
args = text[m.end() : end]
|
||||
lits = STRING_RE.findall(args)
|
||||
line = text.count("\n", 0, m.start()) + 1
|
||||
if "OUT_DIR" in args:
|
||||
self.add("include", "generated-out-dir", rel, line, args.strip()[:90], False)
|
||||
continue
|
||||
if not lits:
|
||||
self.add("include", "dynamic", rel, line, args.strip()[:90], False)
|
||||
continue
|
||||
if "CARGO_MANIFEST_DIR" in args and crate_dir:
|
||||
entries.append((line, lits[-1].lstrip("/"), crate_dir, m.group(1)))
|
||||
else:
|
||||
entries.append((line, "".join(lits), None, m.group(1)))
|
||||
for m in PATH_ATTR_RE.finditer(text):
|
||||
line = text.count("\n", 0, m.start()) + 1
|
||||
entries.append((line, m.group(1), None, "path"))
|
||||
for line, literal, manifest_dir, kind in entries:
|
||||
target = self.resolve(rel, literal, manifest_dir)
|
||||
self.classify_include(rel, line, kind, literal, target)
|
||||
|
||||
def classify_include(self, rel: str, line: int, kind: str, literal: str, target: str) -> None:
|
||||
includer_moves, target_moves = rel in self.map, target in self.map
|
||||
if not (REPO_ROOT / target).exists() and not target_moves:
|
||||
self.add("include", "unresolvable-today", rel, line, f"{kind}!({literal}) -> {target}", False)
|
||||
return
|
||||
if not includer_moves and not target_moves:
|
||||
self.add("include", "unaffected", rel, line, f"{kind} {literal}", False)
|
||||
elif includer_moves and target_moves:
|
||||
new_inc, new_tgt = self.map[rel], self.map[target]
|
||||
if os.path.normpath(os.path.join(os.path.dirname(new_inc), literal)) == new_tgt:
|
||||
self.add("include", "both-move-relative-kept", rel, line, f"{kind} {literal}", False)
|
||||
else:
|
||||
self.add("include", "BROKEN-relative-changed", rel, line, f"{kind} {literal}", True)
|
||||
elif target_moves:
|
||||
new_tgt = self.map[target]
|
||||
hint = os.path.relpath(new_tgt, os.path.dirname(rel))
|
||||
self.add("include", "BROKEN-target-moved", rel, line, f"{kind} {literal} -> {hint}", True)
|
||||
else:
|
||||
new_inc = self.map[rel]
|
||||
if os.path.normpath(os.path.join(os.path.dirname(new_inc), literal)) == target:
|
||||
self.add("include", "includer-moves-relative-kept", rel, line, f"{kind} {literal}", False)
|
||||
else:
|
||||
hint = os.path.relpath(target, os.path.dirname(new_inc))
|
||||
self.add("include", "BROKEN-includer-moved", rel, line, f"{kind} {literal} -> {hint}", True)
|
||||
|
||||
def features_owners(self) -> None:
|
||||
path = "docs/features.toml"
|
||||
if not (REPO_ROOT / path).is_file():
|
||||
return
|
||||
swept = not sl.sweep_skipped(path)
|
||||
for lineno, line in enumerate(sl.read_text(path).splitlines(), 1):
|
||||
m = re.match(r'\s*owner\s*=\s*"([^"]+)"', line)
|
||||
if not m:
|
||||
continue
|
||||
owner = m.group(1)
|
||||
if owner in self.map:
|
||||
self.add("features-owner", "moves-and-swept" if swept else "moves-NOT-swept", path, lineno, f"{owner} -> {self.map[owner]}", not swept)
|
||||
elif not (REPO_ROOT / owner).exists():
|
||||
self.add("features-owner", "missing-today", path, lineno, owner, True)
|
||||
|
||||
def strings(self) -> None:
|
||||
if not self.mods:
|
||||
return
|
||||
regexes = sl.sweep_regexes(self.src, self.dest, self.mods)
|
||||
cands: set[str] = set()
|
||||
for m in self.mods:
|
||||
cands.update(sl.grep_files(f"{self.src}/src/{m}"))
|
||||
for q in ('"', "'"):
|
||||
cands.update(sl.grep_files(f"{q}{self.src}{q}", "*.py", "*.mjs", "*.js", "*.ts", "*.sh", "*.rs"))
|
||||
for rel in sorted(cands):
|
||||
try:
|
||||
text = sl.read_text(rel)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
hits = [(rx, mo) for rx, _ in regexes for mo in rx.finditer(text)]
|
||||
if not hits:
|
||||
continue
|
||||
first = min(text.count("\n", 0, mo.start()) + 1 for _, mo in hits)
|
||||
if sl.SWEEP_SKIP_GENERATED.match(rel):
|
||||
self.add("path-string", "generated-run-derive-facts", rel, first, f"{len(hits)} hits", False)
|
||||
elif Path(rel).name.startswith("CHANGELOG"):
|
||||
self.add("path-string", "history-left-alone", rel, first, f"{len(hits)} hits", False)
|
||||
elif sl.sweep_skipped(rel):
|
||||
snippet = text.splitlines()[first - 1].strip()[:90]
|
||||
self.add("path-string", "SKIPPED-BY-SWEEP", rel, first, f"{len(hits)} hits: {snippet}", True)
|
||||
else:
|
||||
self.add("path-string", "swept", rel, first, f"{len(hits)} hits", False)
|
||||
|
||||
def relative_refs(self) -> None:
|
||||
if not self.mods:
|
||||
return
|
||||
alt = "|".join(re.escape(m) for m in self.mods)
|
||||
parent = re.compile(rf"(?:\.\./)+src/({alt})(?:\.rs|/)")
|
||||
manifest = re.compile(rf"""["']/?src/({alt})(?:\.rs|/)""")
|
||||
base = f"crates/{self.src}"
|
||||
for rel in sl.tracked_files(base):
|
||||
if rel.startswith(f"{base}/src/"):
|
||||
continue
|
||||
try:
|
||||
text = sl.read_text(rel)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
use_manifest = rel.endswith(".rs") and "CARGO_MANIFEST_DIR" in text
|
||||
for rx in (parent, manifest) if use_manifest else (parent,):
|
||||
for mo in rx.finditer(text):
|
||||
line = text.count("\n", 0, mo.start()) + 1
|
||||
self.add("relative-ref", "UNRESOLVED-relative", rel, line, mo.group(0), True)
|
||||
|
||||
def test_identities(self) -> None:
|
||||
if not self.mods:
|
||||
return
|
||||
alt = "|".join(re.escape(m) for m in self.mods)
|
||||
pat = re.compile(rf"(?<![\w:$])({alt})::\w")
|
||||
for rel in sl.grep_files("::", ".github", "scripts/*.sh"):
|
||||
try:
|
||||
text = sl.read_text(rel)
|
||||
except UnicodeDecodeError:
|
||||
continue
|
||||
for mo in pat.finditer(text):
|
||||
line = text.count("\n", 0, mo.start()) + 1
|
||||
ctx = text[text.rfind("\n", 0, mo.start()) + 1 : text.find("\n", mo.end())].strip()[:110]
|
||||
self.add("test-identity", "REVIEW", rel, line, ctx, True)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
parser.add_argument("modules", nargs="*")
|
||||
parser.add_argument("--src-crate", default="tui")
|
||||
parser.add_argument("--dest-crate", default="runtime")
|
||||
parser.add_argument("--json", action="store_true")
|
||||
parser.add_argument("--list", action="store_true", help="print every row, not only the residue")
|
||||
args = parser.parse_args(argv)
|
||||
sl.require_git_repo()
|
||||
pf = Preflight(args.modules, args.src_crate, args.dest_crate)
|
||||
pf.includes()
|
||||
pf.features_owners()
|
||||
pf.strings()
|
||||
pf.relative_refs()
|
||||
pf.test_identities()
|
||||
residue = [r for r in pf.rows if r["residue"]]
|
||||
summary = collections.Counter((r["kind"], r["status"]) for r in pf.rows)
|
||||
if args.json:
|
||||
json.dump(
|
||||
{
|
||||
"modules": args.modules,
|
||||
"files_in_move_map": len(pf.map),
|
||||
"summary": {f"{k}:{s}": n for (k, s), n in sorted(summary.items())},
|
||||
"residue_count": len(residue),
|
||||
"residue": residue,
|
||||
"rows": pf.rows,
|
||||
},
|
||||
sys.stdout,
|
||||
indent=1,
|
||||
)
|
||||
print()
|
||||
else:
|
||||
print(f"G11 path-coupling preflight: modules {' '.join(args.modules) or '(inventory)'}; {len(pf.map)} files in the move map")
|
||||
for (kind, status), n in sorted(summary.items()):
|
||||
print(f" {kind:15} {status:28} {n}")
|
||||
shown = pf.rows if args.list else residue
|
||||
for r in shown:
|
||||
print(f" {'RESIDUE' if r['residue'] else 'ok '} {r['kind']}:{r['status']} {r['file']}:{r['line']} {r['detail']}")
|
||||
print(f"unresolved: {len(residue)}")
|
||||
return 1 if residue else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(sl.run_main(main))
|
||||
@@ -0,0 +1,319 @@
|
||||
"""`move-modules.py --resolve-reexports`: retire a re-export facade.
|
||||
|
||||
For each facade module F (for example `config`) in the source crate, finds the
|
||||
names its `pub use` lines re-export from other crates, enumerates glob
|
||||
re-exports from the upstream sources (generated `include!(OUT_DIR/..)` modules
|
||||
through their JSON data), and rewrites `crate::F::NAME` in the staying files to
|
||||
the upstream path. Grouped `use crate::F::{..}` lists are split by upstream
|
||||
parent. Relative `F::NAME` uses are reported, not rewritten.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def flatten_groups_impl(mg, text: str, facades: list[str]) -> str:
|
||||
masked = mg.lex_mask(text)
|
||||
fac = "|".join(re.escape(f) for f in facades)
|
||||
group_re = re.compile(rf"(?m)^(?P<indent>[ \t]*)(?P<vis>pub(?:\([^)]*\))?[ \t]+)?use[ \t]+crate::\{{")
|
||||
out, pos = [], 0
|
||||
for m in group_re.finditer(masked):
|
||||
if m.start() < pos:
|
||||
continue
|
||||
open_i = m.end() - 1
|
||||
close_i = mg.match_brace(masked, open_i)
|
||||
semi = masked.find(";", close_i)
|
||||
depth, start, ents = 0, open_i + 1, []
|
||||
for i in range(open_i + 1, close_i):
|
||||
ch = masked[i]
|
||||
depth += (ch == "{") - (ch == "}")
|
||||
if ch == "," and depth == 0:
|
||||
ents.append((text[start:i].strip(), masked[start:i].strip()))
|
||||
start = i + 1
|
||||
ents.append((text[start:close_i].strip(), masked[start:close_i].strip()))
|
||||
ents = [(t, k) for t, k in ents if k]
|
||||
split = [t for t, k in ents if re.match(rf"(?:{fac})::", k)]
|
||||
if not split:
|
||||
continue
|
||||
kept = [t for t, k in ents if not re.match(rf"(?:{fac})::", k)]
|
||||
indent, vis = m.group("indent"), m.group("vis") or ""
|
||||
stmts = [f"{indent}{vis}use crate::{t};" for t in split]
|
||||
if kept:
|
||||
stmts.insert(0, f"{indent}{vis}use crate::{{{', '.join(kept)}}};")
|
||||
out.append(text[pos : m.start()])
|
||||
out.append("\n".join(stmts))
|
||||
pos = semi + 1
|
||||
if not out:
|
||||
return text
|
||||
out.append(text[pos:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def run(args, sl, mg) -> int:
|
||||
flatten_groups = lambda text, facades: flatten_groups_impl(mg, text, facades)
|
||||
REPO = sl.REPO_ROOT
|
||||
sl.require_git_repo()
|
||||
src = args.src_crate
|
||||
src_dir = f"crates/{src}/src"
|
||||
facades = list(args.modules)
|
||||
crate_dirs = {sl.crate_ident(p.name): p.name for p in (REPO / "crates").iterdir() if (p / "Cargo.toml").is_file()}
|
||||
warnings: list[str] = []
|
||||
|
||||
def generated_names(file_name: str, owner_dir: str) -> set[str]:
|
||||
base = REPO / "crates" / owner_dir
|
||||
names: set[str] = set()
|
||||
if file_name == "catalog_constants.rs":
|
||||
rev = json.loads((base / "assets/catalog_corrections.json").read_text(encoding="utf-8"))["reviewed"]
|
||||
names |= set(rev["constants"]) | set(rev["numeric_refs"]) | set(rev["groups"])
|
||||
build = (base / "build.rs").read_text(encoding="utf-8")
|
||||
names |= set(re.findall(r'\(\s*"([A-Z][A-Z0-9_]+_MODELS)"\s*,\s*"[a-z-]+"\s*,', build))
|
||||
names |= {"compatibility_alias", "completion_names"}
|
||||
elif file_name == "provider_defaults.rs":
|
||||
data = json.loads((base / "assets/provider_descriptors.json").read_text(encoding="utf-8"))
|
||||
names |= set(data["constant_refs"]) | set(data["compatibility_constants"])
|
||||
else:
|
||||
warnings.append(f"cannot enumerate generated names of {file_name} in crates/{owner_dir}")
|
||||
return names
|
||||
|
||||
def module_text(crate_ident: str, segs: tuple[str, ...]):
|
||||
owner = crate_dirs.get(crate_ident)
|
||||
if owner is None:
|
||||
return None, None
|
||||
mod_dir = REPO / "crates" / owner / "src"
|
||||
text = (mod_dir / "lib.rs").read_text(encoding="utf-8")
|
||||
for seg in segs:
|
||||
masked = mg.lex_mask(text)
|
||||
m = re.search(rf"(?m)^\s*(?:pub(?:\([^)]*\))?\s+)?mod\s+{re.escape(seg)}\s*(;|\{{)", masked)
|
||||
if not m:
|
||||
return None, owner
|
||||
if m.group(1) == "{":
|
||||
text = text[m.end() : mg.match_brace(masked, m.end() - 1)]
|
||||
else:
|
||||
path = next((c for c in (mod_dir / f"{seg}.rs", mod_dir / seg / "mod.rs") if c.is_file()), None)
|
||||
if path is None:
|
||||
return None, owner
|
||||
text = path.read_text(encoding="utf-8")
|
||||
mod_dir = mod_dir / seg
|
||||
return text, owner
|
||||
|
||||
def pub_names(crate_ident: str, segs: tuple[str, ...]) -> set[str]:
|
||||
text, owner = module_text(crate_ident, segs)
|
||||
if text is None:
|
||||
warnings.append(f"cannot find {crate_ident}::{'::'.join(segs)}")
|
||||
return set()
|
||||
masked = mg.lex_mask(text)
|
||||
names: set[str] = set()
|
||||
depth = 0
|
||||
for m in re.finditer(r"[{}]|(?m:^)[ \t]*pub(?:\([^)]*\))?\s+(?:async\s+|unsafe\s+|const\s+)*(?:const|static|fn|struct|enum|type|trait|mod|union)\s+(\w+)|(?m:^)[ \t]*pub\s+use\s+([^;]*);", masked):
|
||||
tok = m.group(0)
|
||||
if tok == "{":
|
||||
depth += 1
|
||||
elif tok == "}":
|
||||
depth -= 1
|
||||
elif depth == 0:
|
||||
if m.group(1):
|
||||
names.add(m.group(1))
|
||||
elif m.group(2) is not None:
|
||||
for bound, path in mg.use_leaves(m.group(2)):
|
||||
if bound != "*":
|
||||
names.add(bound)
|
||||
for inc in re.finditer(r'include!\s*\(\s*concat!\s*\(\s*env!\s*\(\s*"OUT_DIR"\s*\)\s*,\s*"/([\w.]+)"\s*\)\s*\)', text):
|
||||
names |= generated_names(inc.group(1), owner)
|
||||
return names
|
||||
|
||||
exports: dict[str, dict[str, tuple[str, ...]]] = {}
|
||||
|
||||
def facade_files(facade: str):
|
||||
out = {}
|
||||
for rel in sl.tracked_files(f"{src_dir}/{facade}.rs", f"{src_dir}/{facade}"):
|
||||
if not rel.endswith(".rs") or mg.is_test_path(rel):
|
||||
continue
|
||||
sub = rel[len(src_dir) + 1 :]
|
||||
parts = sub[:-3].split("/")
|
||||
if parts[-1] == "mod":
|
||||
parts = parts[:-1]
|
||||
out["::".join(parts)] = rel
|
||||
return out
|
||||
|
||||
local_children: dict[str, set[str]] = {}
|
||||
for facade in facades:
|
||||
files = facade_files(facade)
|
||||
for modpath, rel in files.items():
|
||||
text = sl.read_text(rel)
|
||||
masked = mg.lex_mask(text)
|
||||
ex = exports.setdefault(modpath, {})
|
||||
child_names = {k.split("::")[-1] for k in files if k.startswith(modpath + "::") and k.count("::") == modpath.count("::") + 1}
|
||||
local_children[modpath] = child_names
|
||||
for m in re.finditer(r"(?m)^(?P<attrs>(?:[ \t]*#\[[^\n]*\]\s*\n)*)pub(?:\([^)]*\))?[ \t]+use[ \t]+", masked):
|
||||
if "cfg(test)" in m.group("attrs"):
|
||||
continue
|
||||
end = masked.find(";", m.end())
|
||||
for bound, path in mg.use_leaves(masked[m.end() : end]):
|
||||
head = path[0]
|
||||
if head in ("self", "super", "crate") or head in child_names:
|
||||
continue
|
||||
if bound == "*":
|
||||
for name in pub_names(head, path[1:-1]):
|
||||
ex[name] = path[:-1] + (name,)
|
||||
else:
|
||||
ex[bound] = path
|
||||
for modpath in sorted(files, key=lambda p: -p.count("::")):
|
||||
text = sl.read_text(files[modpath])
|
||||
masked = mg.lex_mask(text)
|
||||
for m in re.finditer(r"(?m)^pub(?:\([^)]*\))?[ \t]+use[ \t]+(\w+)::\*[ \t]*;", masked):
|
||||
child = f"{modpath}::{m.group(1)}"
|
||||
if child in exports:
|
||||
exports[modpath].update(exports[child])
|
||||
|
||||
alt_by_module = {
|
||||
mp: re.compile(
|
||||
r"(?<![\$\w])crate::" + mp.replace("::", "::") + r"::(" + "|".join(re.escape(n) for n in sorted(ex, key=lambda n: -len(n))) + r")(?![A-Za-z0-9_])"
|
||||
)
|
||||
for mp, ex in exports.items()
|
||||
if ex
|
||||
}
|
||||
|
||||
all_src = [f for f in sl.tracked_files(src_dir) if f.endswith(".rs")]
|
||||
facade_rels = set()
|
||||
for facade in facades:
|
||||
facade_rels |= set(facade_files(facade).values())
|
||||
new: dict[str, str] = {}
|
||||
fps: dict[str, object] = {}
|
||||
|
||||
def read(rel: str) -> str:
|
||||
p = REPO / rel
|
||||
fps[rel] = sl.Fingerprint.of(p)
|
||||
return p.read_text(encoding="utf-8")
|
||||
|
||||
direct_lines = direct_files = 0
|
||||
grouped = []
|
||||
relative_uses = collections.Counter()
|
||||
direct_files_set = set()
|
||||
for rel in all_src:
|
||||
if rel in facade_rels:
|
||||
continue
|
||||
text = read(rel)
|
||||
if "crate::" not in text and not any(f + "::" in text for f in facades):
|
||||
continue
|
||||
out = flatten_groups(text, facades)
|
||||
masked = mg.lex_mask(out)
|
||||
for facade in facades:
|
||||
group_re = re.compile(rf"(?m)^(?P<indent>[ \t]*)(?P<vis>pub(?:\([^)]*\))?[ \t]+)?use[ \t]+crate::(?P<path>{re.escape(facade)}(?:::\w+)*)::\{{")
|
||||
pieces, pos = [], 0
|
||||
for m in group_re.finditer(masked):
|
||||
if m.start() < pos:
|
||||
continue
|
||||
ex = exports.get(m.group("path").replace("::", "::"))
|
||||
if not ex:
|
||||
continue
|
||||
open_i = m.end() - 1
|
||||
close_i = mg.match_brace(masked, open_i)
|
||||
semi = masked.find(";", close_i)
|
||||
depth, start, ents = 0, open_i + 1, []
|
||||
for i in range(open_i + 1, close_i):
|
||||
ch = masked[i]
|
||||
depth += (ch == "{") - (ch == "}")
|
||||
if ch == "," and depth == 0:
|
||||
ents.append((out[start:i].strip(), masked[start:i].strip()))
|
||||
start = i + 1
|
||||
ents.append((out[start:close_i].strip(), masked[start:close_i].strip()))
|
||||
ents = [(t, k) for t, k in ents if k]
|
||||
moved, kept = [], []
|
||||
for t, k in ents:
|
||||
name = re.match(r"(\w+)", k)
|
||||
(moved if name and name.group(1) in ex else kept).append((t, name.group(1) if name else None))
|
||||
if not moved:
|
||||
continue
|
||||
by_parent = collections.defaultdict(list)
|
||||
for t, name in moved:
|
||||
up = ex[name]
|
||||
rest = t[len(name) :]
|
||||
by_parent["::".join(up[:-1])].append(up[-1] + rest)
|
||||
indent, vis = m.group("indent"), m.group("vis") or ""
|
||||
stmts = []
|
||||
if kept:
|
||||
stmts.append(f"{indent}{vis}use crate::{m.group('path')}::{{{', '.join(t for t, _ in kept)}}};")
|
||||
for parent, leaves in by_parent.items():
|
||||
body = leaves[0] if len(leaves) == 1 else "{" + ", ".join(leaves) + "}"
|
||||
stmts.append(f"{indent}{vis}use {parent}::{body};")
|
||||
pieces.append(out[pos : m.start()])
|
||||
pieces.append("\n".join(stmts))
|
||||
pos = semi + 1
|
||||
grouped.append(f"{rel}:{masked.count(chr(10), 0, m.start()) + 1}")
|
||||
if pieces:
|
||||
pieces.append(out[pos:])
|
||||
out = "".join(pieces)
|
||||
masked = mg.lex_mask(out)
|
||||
lines = set()
|
||||
|
||||
def sub(mp, rx):
|
||||
nonlocal out
|
||||
def repl(mo):
|
||||
lines.add(out.count("\n", 0, mo.start()) + 1)
|
||||
return "::".join(exports[mp][mo.group(1)])
|
||||
out = rx.sub(repl, out)
|
||||
|
||||
for mp in sorted(alt_by_module, key=lambda p: -p.count("::")):
|
||||
sub(mp, alt_by_module[mp])
|
||||
if lines:
|
||||
direct_lines += len(lines)
|
||||
direct_files_set.add(rel)
|
||||
for facade in facades:
|
||||
if re.search(rf"\buse\s+crate::{re.escape(facade)}\s*(;|as\b)|crate::\{{[^}}]*\b{re.escape(facade)}\b[^:]", masked):
|
||||
ex = exports.get(facade, {})
|
||||
for mo in re.finditer(rf"(?<![\w:$]){re.escape(facade)}::(\w+)", masked):
|
||||
if mo.group(1) in ex:
|
||||
relative_uses[rel] += 1
|
||||
if out != text:
|
||||
new[rel] = out
|
||||
sweep = []
|
||||
report = {
|
||||
"facades": facades,
|
||||
"reexported_names": {mp: len(ex) for mp, ex in exports.items()},
|
||||
"direct_lines": direct_lines,
|
||||
"direct_files": len(direct_files_set),
|
||||
"grouped_lists": len(grouped),
|
||||
"grouped_files": len({g.split(":")[0] for g in grouped}),
|
||||
"files_rewritten": len(new),
|
||||
"relative_uses_not_rewritten": {"sites": sum(relative_uses.values()), "files": len(relative_uses)},
|
||||
"warnings": warnings,
|
||||
"upstream_paths": {mp: {n: "::".join(p) for n, p in sorted(ex.items())} for mp, ex in exports.items()},
|
||||
}
|
||||
if args.json:
|
||||
json.dump({**report, "rewrite_set": sorted(new)}, sys.stdout, indent=1)
|
||||
print()
|
||||
else:
|
||||
print(f"re-exported names: {report['reexported_names']}")
|
||||
print(f"direct crate::F::NAME: {direct_lines} lines in {len(direct_files_set)} files; grouped use lists split: {len(grouped)} in {report['grouped_files']} files")
|
||||
print(f"relative F::NAME uses (not rewritten): {report['relative_uses_not_rewritten']}; files rewritten: {len(new)}")
|
||||
for w in warnings:
|
||||
print(f"warning: {w}")
|
||||
if args.dry_run:
|
||||
return 0
|
||||
if args.require_clean:
|
||||
dirty = sorted(sl.dirty_paths() & set(new))
|
||||
young = sl.recently_modified(sorted(new), args.quiet_minutes)
|
||||
if dirty or young:
|
||||
raise sl.SplitError(
|
||||
"--require-clean refused; the rewrite set overlaps live work:\n "
|
||||
+ "\n ".join([f"dirty: {p}" for p in dirty] + [f"modified {a:.1f} min ago: {p}" for p, a in young]),
|
||||
5,
|
||||
)
|
||||
changes, fmt = [], {}
|
||||
originals = {rel: read(rel) if rel not in fps else None for rel in new}
|
||||
originals = {rel: (sl.REPO_ROOT / rel).read_text(encoding="utf-8") for rel in new}
|
||||
if not args.no_format:
|
||||
formatted, _ = sl.format_many({rel: (originals[rel], text) for rel, text in new.items()})
|
||||
else:
|
||||
formatted = new
|
||||
for rel, text in sorted(formatted.items()):
|
||||
changes.append(sl.Change(rel, text, fps[rel]))
|
||||
jdir = Path(args.journal_dir) if args.journal_dir else sl.default_journal_dir("reexports-" + "-".join(facades))
|
||||
sl.apply_changes(changes, jdir, args.pause_after_plan, args.pause_mid_write)
|
||||
print(f"applied {len(changes)} change(s); journal {jdir}", file=sys.stderr)
|
||||
return 0
|
||||
@@ -0,0 +1,412 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shared safety layer for the crate-split tools.
|
||||
|
||||
Everything here exists so a mechanical rewrite on the shared checkout can fail
|
||||
visibly and leave the tree as it found it:
|
||||
|
||||
* git failures are fatal, and the repository root must itself hold `.git`
|
||||
(a `git archive` export would otherwise silently do nothing);
|
||||
* every write is a compare-and-swap against the bytes the plan was computed
|
||||
from, through a temp file and `os.replace`, with a pre-image journal;
|
||||
* any exception or SIGTERM/SIGHUP/SIGINT during the apply phase restores the
|
||||
journalled pre-images in reverse order;
|
||||
* rewritten Rust goes through `rustfmt --edition 2024 --emit stdout` on stdin,
|
||||
one file at a time, so no `mod` child and no neighbouring writer's file is
|
||||
reformatted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
class SplitError(SystemExit):
|
||||
def __init__(self, message: str, code: int = 2):
|
||||
self.message = f"split: {message}"
|
||||
self.exit_code = code
|
||||
super().__init__(self.message)
|
||||
|
||||
def __str__(self) -> str:
|
||||
return self.message
|
||||
|
||||
|
||||
def run_main(fn) -> int:
|
||||
try:
|
||||
return fn()
|
||||
except SplitError as exc:
|
||||
print(exc.message, file=sys.stderr)
|
||||
return exc.exit_code
|
||||
|
||||
|
||||
def git(*args: str, input_text: str | None = None) -> str:
|
||||
proc = subprocess.run(
|
||||
["git", *args], cwd=REPO_ROOT, capture_output=True, text=True, input=input_text
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise SplitError(f"`git {' '.join(args)}` failed ({proc.returncode}): {proc.stderr.strip()}")
|
||||
return proc.stdout
|
||||
|
||||
|
||||
def require_git_repo() -> None:
|
||||
if not (REPO_ROOT / ".git").exists():
|
||||
raise SplitError(f"{REPO_ROOT} has no .git; refusing to run on a tree git cannot see")
|
||||
top = Path(git("rev-parse", "--show-toplevel").strip()).resolve()
|
||||
if top != REPO_ROOT.resolve():
|
||||
raise SplitError(f"git toplevel {top} is not {REPO_ROOT}; refusing to run")
|
||||
|
||||
|
||||
def tracked_files(*pathspecs: str) -> list[str]:
|
||||
out = git("ls-files", "-z", "--", *pathspecs)
|
||||
return [p for p in out.split("\0") if p]
|
||||
|
||||
|
||||
def grep_files(pattern: str, *pathspecs: str, fixed: bool = True) -> list[str]:
|
||||
args = ["grep", "-I", "-l", "-z"]
|
||||
if fixed:
|
||||
args.append("-F")
|
||||
args += ["-e", pattern, "--", *(pathspecs or (".",))]
|
||||
proc = subprocess.run(["git", *args], cwd=REPO_ROOT, capture_output=True, text=True)
|
||||
if proc.returncode not in (0, 1):
|
||||
raise SplitError(f"`git {' '.join(args)}` failed ({proc.returncode}): {proc.stderr.strip()}")
|
||||
return [p for p in proc.stdout.split("\0") if p]
|
||||
|
||||
|
||||
def dirty_paths() -> set[str]:
|
||||
out = git("status", "--porcelain=v1", "-z", "--untracked-files=all")
|
||||
paths: set[str] = set()
|
||||
entries = out.split("\0")
|
||||
i = 0
|
||||
while i < len(entries):
|
||||
entry = entries[i]
|
||||
i += 1
|
||||
if len(entry) < 4:
|
||||
continue
|
||||
status, path = entry[:2], entry[3:]
|
||||
paths.add(path)
|
||||
if status[0] in "RC":
|
||||
if i < len(entries) and entries[i]:
|
||||
paths.add(entries[i])
|
||||
i += 1
|
||||
return paths
|
||||
|
||||
|
||||
def recently_modified(rels: list[str], minutes: float) -> list[tuple[str, float]]:
|
||||
if minutes <= 0:
|
||||
return []
|
||||
now = time.time()
|
||||
out = []
|
||||
for rel in rels:
|
||||
try:
|
||||
age = (now - (REPO_ROOT / rel).stat().st_mtime) / 60.0
|
||||
except OSError:
|
||||
continue
|
||||
if age < minutes:
|
||||
out.append((rel, age))
|
||||
return out
|
||||
|
||||
|
||||
def sha256_bytes(data: bytes) -> str:
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Fingerprint:
|
||||
size: int
|
||||
mtime_ns: int
|
||||
ino: int
|
||||
sha: str
|
||||
|
||||
@staticmethod
|
||||
def of(path: Path) -> "Fingerprint":
|
||||
st = path.stat()
|
||||
return Fingerprint(st.st_size, st.st_mtime_ns, st.st_ino, sha256_bytes(path.read_bytes()))
|
||||
|
||||
def matches(self, path: Path) -> bool:
|
||||
try:
|
||||
st = path.stat()
|
||||
except OSError:
|
||||
return False
|
||||
if (st.st_size, st.st_mtime_ns, st.st_ino) == (self.size, self.mtime_ns, self.ino):
|
||||
return True
|
||||
return st.st_size == self.size and sha256_bytes(path.read_bytes()) == self.sha
|
||||
|
||||
|
||||
def read_text(rel: str) -> str:
|
||||
return (REPO_ROOT / rel).read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def rustfmt_text(text: str, label: str) -> str:
|
||||
proc = subprocess.run(
|
||||
["rustfmt", "--edition", "2024", "--emit", "stdout"],
|
||||
cwd=REPO_ROOT,
|
||||
input=text,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise SplitError(f"rustfmt failed on {label}: {proc.stderr.strip()[:600]}")
|
||||
return proc.stdout
|
||||
|
||||
|
||||
def format_many(items: dict[str, tuple[str | None, str]], workers: int = 8) -> tuple[dict[str, str], list[str]]:
|
||||
"""Format rewritten Rust in memory.
|
||||
|
||||
`items` maps a label to (base_text_or_None, new_text). A file whose base
|
||||
text is not already rustfmt-stable is left exactly as rewritten, so a
|
||||
rewrite never reflows lines it did not change; the labels are returned.
|
||||
"""
|
||||
result: dict[str, str] = {}
|
||||
unstable: list[str] = []
|
||||
|
||||
def one(label: str) -> tuple[str, str, bool]:
|
||||
base, new = items[label]
|
||||
formatted = rustfmt_text(new, label)
|
||||
if base is None:
|
||||
return label, formatted, False
|
||||
if rustfmt_text(base, label) != base:
|
||||
return label, new, True
|
||||
return label, formatted, False
|
||||
|
||||
with ThreadPoolExecutor(max_workers=workers) as pool:
|
||||
for label, text, was_unstable in pool.map(one, sorted(items)):
|
||||
result[label] = text
|
||||
if was_unstable:
|
||||
unstable.append(label)
|
||||
return result, unstable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Change:
|
||||
rel: str
|
||||
text: str | None # None deletes the file
|
||||
base: Fingerprint | None = None # None means the file must not exist yet
|
||||
mode_from: str | None = None # copy permission bits from this tracked path
|
||||
|
||||
|
||||
class Interrupted(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class Journal:
|
||||
def __init__(self, root: Path):
|
||||
self.root = root
|
||||
self.pre = root / "pre"
|
||||
self.ops: list[dict] = []
|
||||
self.created_dirs: list[str] = []
|
||||
self.pre.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def _flush(self) -> None:
|
||||
tmp = self.root / "manifest.json.tmp"
|
||||
tmp.write_text(
|
||||
json.dumps({"ops": self.ops, "created_dirs": self.created_dirs}, indent=1), encoding="utf-8"
|
||||
)
|
||||
os.replace(tmp, self.root / "manifest.json")
|
||||
|
||||
def record(self, op: dict, pre_bytes: bytes | None) -> None:
|
||||
if pre_bytes is not None:
|
||||
name = f"{len(self.ops):06d}"
|
||||
(self.pre / name).write_bytes(pre_bytes)
|
||||
op["pre"] = name
|
||||
self.ops.append(op)
|
||||
self._flush()
|
||||
|
||||
def note_dir(self, rel: str) -> None:
|
||||
self.created_dirs.append(rel)
|
||||
self._flush()
|
||||
|
||||
|
||||
def default_journal_dir(label: str) -> Path:
|
||||
stamp = time.strftime("%Y%m%d-%H%M%S")
|
||||
backups = REPO_ROOT.parent / "backups"
|
||||
if not backups.is_dir():
|
||||
raise SplitError(f"{backups} does not exist; pass --journal-dir")
|
||||
return backups / f"crate-split-{label}-{stamp}"
|
||||
|
||||
|
||||
def rollback(journal_root: Path, repo: Path = REPO_ROOT) -> list[str]:
|
||||
manifest = json.loads((journal_root / "manifest.json").read_text(encoding="utf-8"))
|
||||
problems: list[str] = []
|
||||
for op in reversed(manifest["ops"]):
|
||||
path = repo / op["path"]
|
||||
try:
|
||||
if op["op"] in ("write", "delete") and "pre" in op:
|
||||
data = (journal_root / "pre" / op["pre"]).read_bytes()
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = path.with_name(path.name + ".split-restore")
|
||||
tmp.write_bytes(data)
|
||||
if "mode" in op:
|
||||
os.chmod(tmp, op["mode"])
|
||||
os.replace(tmp, path)
|
||||
elif op["op"] == "write":
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
except OSError as exc:
|
||||
problems.append(f"{op['path']}: {exc}")
|
||||
for rel in reversed(manifest["created_dirs"]):
|
||||
try:
|
||||
(repo / rel).rmdir()
|
||||
except OSError:
|
||||
pass
|
||||
return problems
|
||||
|
||||
|
||||
def apply_changes(
|
||||
changes: list[Change],
|
||||
journal_dir: Path,
|
||||
pause_after_plan: float = 0.0,
|
||||
pause_mid_write: float = 0.0,
|
||||
log=lambda msg: print(msg, file=sys.stderr),
|
||||
) -> None:
|
||||
"""Compare-and-swap every change in, or restore every pre-image and raise."""
|
||||
journal_dir.mkdir(parents=True, exist_ok=False)
|
||||
journal = Journal(journal_dir)
|
||||
if pause_after_plan > 0:
|
||||
log(f"plan frozen; writing in {pause_after_plan:g}s (journal {journal_dir})")
|
||||
time.sleep(pause_after_plan)
|
||||
|
||||
stale = []
|
||||
for ch in changes:
|
||||
path = REPO_ROOT / ch.rel
|
||||
if ch.base is None:
|
||||
if path.exists() and ch.text is not None:
|
||||
stale.append(f"{ch.rel}: appeared after the plan was made")
|
||||
elif not ch.base.matches(path):
|
||||
stale.append(f"{ch.rel}: changed after the plan was made")
|
||||
if stale:
|
||||
shutil.rmtree(journal_dir)
|
||||
raise SplitError("compare-and-swap refused, nothing written:\n " + "\n ".join(stale), 3)
|
||||
|
||||
def on_signal(signum, _frame):
|
||||
raise Interrupted(f"signal {signum}")
|
||||
|
||||
previous = {}
|
||||
for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
|
||||
previous[sig] = signal.signal(sig, on_signal)
|
||||
try:
|
||||
for index, ch in enumerate(changes):
|
||||
if pause_mid_write > 0 and index == len(changes) // 2:
|
||||
log(f"{index} of {len(changes)} written; pausing {pause_mid_write:g}s (test hook)")
|
||||
time.sleep(pause_mid_write)
|
||||
path = REPO_ROOT / ch.rel
|
||||
if ch.base is not None and not ch.base.matches(path):
|
||||
raise SplitError(f"{ch.rel} changed while writing", 3)
|
||||
if ch.base is None and ch.text is not None and path.exists():
|
||||
raise SplitError(f"{ch.rel} appeared while writing", 3)
|
||||
pre = path.read_bytes() if ch.base is not None else None
|
||||
mode = path.stat().st_mode & 0o7777 if ch.base is not None else None
|
||||
if ch.text is None:
|
||||
journal.record({"op": "delete", "path": ch.rel, "mode": mode}, pre)
|
||||
path.unlink()
|
||||
continue
|
||||
op = {"op": "write", "path": ch.rel, "post_sha": sha256_bytes(ch.text.encode("utf-8"))}
|
||||
if mode is not None:
|
||||
op["mode"] = mode
|
||||
journal.record(op, pre)
|
||||
parent = path.parent
|
||||
missing = []
|
||||
while not parent.exists():
|
||||
missing.append(parent)
|
||||
parent = parent.parent
|
||||
for d in reversed(missing):
|
||||
d.mkdir()
|
||||
journal.note_dir(str(d.relative_to(REPO_ROOT)))
|
||||
tmp = path.with_name(path.name + ".split-tmp")
|
||||
tmp.write_bytes(ch.text.encode("utf-8"))
|
||||
if mode is None and ch.mode_from:
|
||||
try:
|
||||
mode = (REPO_ROOT / ch.mode_from).stat().st_mode & 0o7777
|
||||
except OSError:
|
||||
mode = None
|
||||
if mode is not None:
|
||||
os.chmod(tmp, mode)
|
||||
os.replace(tmp, path)
|
||||
except BaseException as exc:
|
||||
log(f"aborting: {exc}; restoring pre-images from {journal_dir}")
|
||||
for sig, handler in previous.items():
|
||||
signal.signal(sig, signal.SIG_IGN)
|
||||
problems = rollback(journal_dir)
|
||||
for sig, handler in previous.items():
|
||||
signal.signal(sig, handler)
|
||||
for ch in changes:
|
||||
stray = (REPO_ROOT / ch.rel).with_name(Path(ch.rel).name + ".split-tmp")
|
||||
if stray.exists():
|
||||
stray.unlink()
|
||||
if problems:
|
||||
log("restore problems:\n " + "\n ".join(problems))
|
||||
(journal_dir / "ROLLED_BACK").write_text(str(exc) + "\n", encoding="utf-8")
|
||||
if isinstance(exc, SplitError):
|
||||
raise
|
||||
raise SplitError(f"{exc}; tree restored from the journal", 4) from exc
|
||||
finally:
|
||||
for sig, handler in previous.items():
|
||||
signal.signal(sig, handler)
|
||||
(journal_dir / "COMMITTED").write_text(f"{len(changes)} change(s)\n", encoding="utf-8")
|
||||
|
||||
|
||||
def load_module_graph():
|
||||
path = Path(__file__).with_name("module_graph.py")
|
||||
spec = importlib.util.spec_from_file_location("module_graph", path)
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
sys.modules.setdefault("module_graph", mod)
|
||||
spec.loader.exec_module(mod)
|
||||
return mod
|
||||
|
||||
|
||||
def crate_ident(crate_dir: str) -> str:
|
||||
manifest = (REPO_ROOT / "crates" / crate_dir / "Cargo.toml").read_text(encoding="utf-8")
|
||||
in_package = False
|
||||
for line in manifest.splitlines():
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("["):
|
||||
in_package = stripped == "[package]"
|
||||
continue
|
||||
if in_package and stripped.startswith("name"):
|
||||
return stripped.split("=", 1)[1].strip().strip('"').replace("-", "_")
|
||||
raise SplitError(f"no [package] name in crates/{crate_dir}/Cargo.toml")
|
||||
|
||||
|
||||
SWEEP_SKIP_PREFIXES = ("vendor/", "crates/config/", "scripts/split/")
|
||||
SWEEP_SKIP_GENERATED = re.compile(r"^web/data/[^/]+\.generated\.json$")
|
||||
|
||||
|
||||
def sweep_skipped(rel: str) -> bool:
|
||||
return (
|
||||
rel.startswith(SWEEP_SKIP_PREFIXES)
|
||||
or Path(rel).name.startswith("CHANGELOG")
|
||||
or bool(SWEEP_SKIP_GENERATED.match(rel))
|
||||
)
|
||||
|
||||
|
||||
def sweep_regexes(src: str, dest: str, mods: list[str]) -> list[tuple[re.Pattern, object]]:
|
||||
"""Path-string rewrites shared by the mover and the preflight."""
|
||||
out: list[tuple[re.Pattern, object]] = []
|
||||
for m in mods:
|
||||
em = re.escape(m)
|
||||
out.append((re.compile(rf"crates/{src}/src/{em}(?![A-Za-z0-9_])"), f"crates/{dest}/src/{m}"))
|
||||
out.append((re.compile(rf"(?<![A-Za-z0-9_/]){src}/src/{em}(?![A-Za-z0-9_])"), f"{dest}/src/{m}"))
|
||||
segmented = re.compile(
|
||||
rf"""(?P<a>(["'])){src}(?P=a)(?P<s1>\s*(?:,|/)\s*)(?P<b>(["']))src(?P=b)(?P<s2>\s*(?:,|/)\s*)(?P<c>(["'])){em}(?:\.rs)?(?P=c)"""
|
||||
)
|
||||
out.append(
|
||||
(
|
||||
segmented,
|
||||
lambda mo, dest=dest, src=src: mo.group(0).replace(
|
||||
f"{mo.group('a')}{src}{mo.group('a')}", f"{mo.group('a')}{dest}{mo.group('a')}", 1
|
||||
),
|
||||
)
|
||||
)
|
||||
return out
|
||||
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Function-level test partitioner for a module move (S09 pre-split helper).
|
||||
|
||||
python3 scripts/split/testsplit.py config settings oauth config_persistence
|
||||
python3 scripts/split/testsplit.py config --file <tests file under the source crate> --json
|
||||
python3 scripts/split/testsplit.py config --emit-dir /tmp/split-out
|
||||
|
||||
For the files of the named modules that hold tests (`tests.rs`, `*_tests.rs`
|
||||
and the inline `mod tests` of the module files), classifies every top-level
|
||||
item as `move` or `stay`:
|
||||
|
||||
* an item STAYS when it names a staying source-crate module (`crate::M` with M
|
||||
neither moved nor already resident in the destination) or an identifier that
|
||||
a `use crate::M::..` import binds from one;
|
||||
* staying propagates through the in-file call graph: a test that calls a
|
||||
helper that stays, stays with it.
|
||||
|
||||
`#[test]` functions are counted individually, including those inside a nested
|
||||
test module (which is classified as a whole). `--emit-dir` writes
|
||||
`<stem>.move.rs` and `<stem>.stay.rs`: every `use` is kept in both, support
|
||||
items needed by both sides are duplicated and listed as `shared`, so the
|
||||
caller decides whether to widen them with `test-support` instead.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import splitlib as sl
|
||||
|
||||
mg = sl.load_module_graph()
|
||||
|
||||
TEST_ATTR = re.compile(r"#\[\s*(?:tokio::test|test|rstest|async_std::test)\b")
|
||||
ITEM_NAME = re.compile(r"\b(?:fn|struct|enum|const|static|type|trait|mod|union)\s+(\w+)")
|
||||
|
||||
|
||||
def items_of(text: str, masked: str, start: int, end: int):
|
||||
out, depth, item_start, i = [], 0, start, start
|
||||
while i < end:
|
||||
c = masked[i]
|
||||
if c == "{":
|
||||
depth += 1
|
||||
elif c == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
out.append((item_start, i + 1))
|
||||
item_start = i + 1
|
||||
elif c == ";" and depth == 0:
|
||||
out.append((item_start, i + 1))
|
||||
item_start = i + 1
|
||||
i += 1
|
||||
return [(a, b) for a, b in out if masked[a:b].strip()]
|
||||
|
||||
|
||||
def analyse_file(rel: str, stay_modules: set[str], inline_body: tuple[int, int] | None = None):
|
||||
text = sl.read_text(rel)
|
||||
masked = mg.lex_mask(text)
|
||||
a, b = inline_body or (0, len(text))
|
||||
spans = items_of(text, masked, a, b)
|
||||
items = []
|
||||
imports: dict[str, str] = {}
|
||||
for sa, sb in spans:
|
||||
code = masked[sa:sb]
|
||||
stripped = code.strip()
|
||||
head = re.sub(r"(?s)^(?:\s*#\[[^\]]*\])*\s*", "", stripped)
|
||||
kind = "use" if re.match(r"(?:pub(?:\([^)]*\))?\s+)?use\b", head) else "item"
|
||||
name_m = ITEM_NAME.search(head) if kind == "item" else None
|
||||
item = {
|
||||
"a": sa, "b": sb, "kind": kind, "name": name_m.group(1) if name_m else None,
|
||||
"tests": len(TEST_ATTR.findall(code)), "lines": text.count("\n", sa, sb) + 1,
|
||||
"tokens": set(re.findall(r"[A-Za-z_]\w*", code)),
|
||||
"direct": set(re.findall(r"(?<![\w$])crate::(\w+)", code)) & stay_modules,
|
||||
}
|
||||
if kind == "use":
|
||||
body = head[head.index("use") + 3 : head.rindex(";")] if ";" in head else head
|
||||
for bound, path in mg.use_leaves(body):
|
||||
if path[0] == "crate" and len(path) > 1:
|
||||
imports[bound] = path[1]
|
||||
items.append(item)
|
||||
defined = {it["name"]: i for i, it in enumerate(items) if it["name"]}
|
||||
for it in items:
|
||||
it["via_import"] = {imports[t] for t in it["tokens"] if t in imports} & stay_modules
|
||||
it["deps"] = {defined[t] for t in it["tokens"] if t in defined and t != it["name"]}
|
||||
it["stay"] = bool(it["direct"] or it["via_import"])
|
||||
it["why"] = sorted(it["direct"] | it["via_import"])
|
||||
changed = True
|
||||
while changed:
|
||||
changed = False
|
||||
for i, it in enumerate(items):
|
||||
if not it["stay"]:
|
||||
for d in it["deps"]:
|
||||
if items[d]["stay"]:
|
||||
it["stay"] = True
|
||||
it["why"] = [f"via {items[d]['name']}"] + items[d]["why"][:2]
|
||||
changed = True
|
||||
break
|
||||
return text, items
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
ap.add_argument("modules", nargs="+")
|
||||
ap.add_argument("--src-crate", default="tui")
|
||||
ap.add_argument("--dest-crate", default="runtime")
|
||||
ap.add_argument("--file", action="append", help="classify only these files")
|
||||
ap.add_argument("--json", action="store_true")
|
||||
ap.add_argument("--emit-dir")
|
||||
args = ap.parse_args(argv)
|
||||
sl.require_git_repo()
|
||||
src_dir = REPO = sl.REPO_ROOT / "crates" / args.src_crate / "src"
|
||||
dest_dir = sl.REPO_ROOT / "crates" / args.dest_crate / "src"
|
||||
moved = set(args.modules)
|
||||
|
||||
def top_modules(d: Path) -> set[str]:
|
||||
return {p.name if p.is_dir() else p.stem for p in d.iterdir() if p.is_dir() or p.suffix == ".rs"} - {"lib", "main"}
|
||||
|
||||
stay_modules = top_modules(src_dir) - moved - top_modules(dest_dir)
|
||||
files = args.file or []
|
||||
inline: dict[str, tuple[int, int]] = {}
|
||||
if not files:
|
||||
for m in args.modules:
|
||||
for rel in sl.tracked_files(f"crates/{args.src_crate}/src/{m}.rs", f"crates/{args.src_crate}/src/{m}"):
|
||||
if not rel.endswith(".rs"):
|
||||
continue
|
||||
name = Path(rel).name
|
||||
if name == "tests.rs" or name.endswith(("_tests.rs", "_test.rs")) or name.startswith("test_cases"):
|
||||
files.append(rel)
|
||||
else:
|
||||
text = sl.read_text(rel)
|
||||
masked = mg.lex_mask(text)
|
||||
mo = re.search(r"#\[cfg\(test\)\]\s*mod\s+tests\s*\{", masked)
|
||||
if mo:
|
||||
files.append(rel)
|
||||
inline[rel] = (mo.end(), mg.match_brace(masked, mo.end() - 1))
|
||||
report = []
|
||||
for rel in files:
|
||||
text, items = analyse_file(rel, stay_modules, inline.get(rel))
|
||||
tests = [it for it in items if it["tests"]]
|
||||
move_tests = sum(it["tests"] for it in tests if not it["stay"])
|
||||
stay_tests = sum(it["tests"] for it in tests if it["stay"])
|
||||
blockers = collections.Counter(w for it in tests if it["stay"] for w in it["why"][:1])
|
||||
helpers_stay = [it["name"] for it in items if it["stay"] and not it["tests"] and it["name"]]
|
||||
report.append(
|
||||
{
|
||||
"file": rel,
|
||||
"inline": rel in inline,
|
||||
"tests_total": move_tests + stay_tests,
|
||||
"tests_move": move_tests,
|
||||
"tests_stay": stay_tests,
|
||||
"stay_helpers": helpers_stay,
|
||||
"blocked_by": dict(blockers.most_common()),
|
||||
"lines_move": sum(it["lines"] for it in items if not it["stay"]),
|
||||
"lines_stay": sum(it["lines"] for it in items if it["stay"]),
|
||||
}
|
||||
)
|
||||
if args.emit_dir:
|
||||
out = Path(args.emit_dir)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
uses = [it for it in items if it["kind"] == "use"]
|
||||
stem = Path(rel).stem
|
||||
for side, want in (("move", False), ("stay", True)):
|
||||
needed: set[int] = set()
|
||||
body = [i for i, it in enumerate(items) if it["kind"] != "use" and it["stay"] == want and it["tests"]]
|
||||
stack = list(body)
|
||||
while stack:
|
||||
i = stack.pop()
|
||||
if i in needed:
|
||||
continue
|
||||
needed.add(i)
|
||||
stack.extend(items[i]["deps"])
|
||||
chosen = [it for it in uses] + [items[i] for i in sorted(needed)]
|
||||
(out / f"{stem}.{side}.rs").write_text("".join(text[it["a"] : it["b"]] for it in chosen), encoding="utf-8")
|
||||
total = {k: sum(r[k] for r in report) for k in ("tests_total", "tests_move", "tests_stay")}
|
||||
if args.json:
|
||||
json.dump({"stay_modules": sorted(stay_modules), "files": report, "total": total}, sys.stdout, indent=1)
|
||||
print()
|
||||
else:
|
||||
print(f"staying source-crate modules considered: {len(stay_modules)}")
|
||||
for r in report:
|
||||
print(f"{r['file']}: {r['tests_total']} tests, move {r['tests_move']}, stay {r['tests_stay']}; stay helpers {len(r['stay_helpers'])}; blocked by {dict(list(r['blocked_by'].items())[:5])}")
|
||||
print(f"total: {total['tests_total']} tests, move {total['tests_move']}, stay {total['tests_stay']}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(sl.run_main(main))
|
||||
+139
-11
@@ -19,23 +19,37 @@ compiler names:
|
||||
It never bulk-rewrites `pub(crate)`. Every widened item is printed; items in
|
||||
security-relevant modules are flagged for human review.
|
||||
|
||||
While the loop runs, the runtime manifest's `[lints] workspace = true` is
|
||||
While the loop runs, the destination manifest's `[lints] workspace = true` is
|
||||
commented out (lints capped to warnings), because an item that is dead inside
|
||||
the runtime would otherwise fail the runtime build under `warnings = "deny"`
|
||||
before cargo ever reached the TUI's privacy errors. The stanza is restored
|
||||
when the loop ends, and a final check runs with it.
|
||||
the destination would otherwise fail its build under `warnings = "deny"`
|
||||
before cargo ever reached the consumer's privacy errors. The stanza is
|
||||
restored on exit, on SIGTERM, SIGHUP and SIGINT (build-lock.py forwards the
|
||||
first two), and a manifest left capped by a killed earlier run is restored
|
||||
before anything else; a final check runs with the stanza back.
|
||||
|
||||
`--prepass` widens, before the first compile, every `pub(crate)`/`pub(super)`
|
||||
declaration in the destination whose name also appears in the consumer crates,
|
||||
which saves most of the compile rounds (privacy errors only show when the
|
||||
consumer is compiled). `unused_imports` warnings left by the move are removed
|
||||
with the compiler's own machine-applicable suggestions, because
|
||||
`warnings = "deny"` turns each one into a hard error.
|
||||
|
||||
Usage:
|
||||
python3 scripts/split/widen.py [--cargo "scripts/dev-cargo.sh"] [--max-rounds N]
|
||||
python3 scripts/split/widen.py [--dest-crate runtime] [--src-crate tui]
|
||||
[--packages codewhale-runtime,codewhale-tui,codewhale-cli]
|
||||
[--cargo "scripts/dev-cargo.sh"] [--max-rounds N] [--prepass]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import atexit
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
@@ -43,6 +57,22 @@ from pathlib import Path
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
RUNTIME_SRC = REPO_ROOT / "crates" / "runtime" / "src"
|
||||
RUNTIME_MANIFEST = REPO_ROOT / "crates" / "runtime" / "Cargo.toml"
|
||||
CONSUMER_DIRS = ["tui", "cli"]
|
||||
COMMON_NAMES = {
|
||||
"new", "fmt", "from", "into", "default", "len", "get", "set", "build", "name", "id", "path", "run",
|
||||
"load", "save", "parse", "take", "text", "kind", "value", "data", "state", "config", "error",
|
||||
}
|
||||
|
||||
|
||||
class Interrupted(BaseException):
|
||||
pass
|
||||
|
||||
|
||||
def configure(dest: str, src: str) -> None:
|
||||
global RUNTIME_SRC, RUNTIME_MANIFEST, CONSUMER_DIRS
|
||||
RUNTIME_SRC = REPO_ROOT / "crates" / dest / "src"
|
||||
RUNTIME_MANIFEST = REPO_ROOT / "crates" / dest / "Cargo.toml"
|
||||
CONSUMER_DIRS = [src, "cli"] if src != "cli" else ["cli"]
|
||||
LINTS_ON = "[lints]\nworkspace = true\n"
|
||||
LINTS_CAPPED = "# [lints] capped by scripts/split/widen.py\n# workspace = true\n"
|
||||
PRIVACY_CODES = {"E0603", "E0624", "E0616", "E0451"}
|
||||
@@ -149,10 +179,82 @@ def apply(edits: dict[tuple[Path, int], str | None]) -> list[str]:
|
||||
else:
|
||||
lines[line_no - 1] = new
|
||||
report.append(f"widened {rel}:{line_no}: {new.strip()}{flag}")
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
atomic_write(path, "\n".join(lines).encode("utf-8"))
|
||||
return report
|
||||
|
||||
|
||||
def atomic_write(path: Path, data: bytes) -> None:
|
||||
tmp = path.with_name(path.name + ".widen-tmp")
|
||||
tmp.write_bytes(data)
|
||||
os.chmod(tmp, path.stat().st_mode & 0o7777)
|
||||
os.replace(tmp, path)
|
||||
|
||||
|
||||
def unused_import_suggestions(messages: list[dict]) -> dict[Path, list[tuple[int, int, str]]]:
|
||||
out: dict[Path, set[tuple[int, int, str]]] = collections.defaultdict(set)
|
||||
for msg in messages:
|
||||
if (msg.get("code") or {}).get("code") != "unused_imports":
|
||||
continue
|
||||
for child in msg.get("children", []):
|
||||
for span in child.get("spans", []):
|
||||
if span.get("suggested_replacement") is None or span.get("suggestion_applicability") != "MachineApplicable":
|
||||
continue
|
||||
path = (REPO_ROOT / span["file_name"]).resolve()
|
||||
if str(path).startswith(str(REPO_ROOT / "crates")):
|
||||
out[path].add((span["byte_start"], span["byte_end"], span["suggested_replacement"]))
|
||||
return {p: sorted(v) for p, v in out.items()}
|
||||
|
||||
|
||||
def apply_suggestions(suggestions: dict[Path, list[tuple[int, int, str]]]) -> list[str]:
|
||||
report = []
|
||||
for path, spans in suggestions.items():
|
||||
data = path.read_bytes()
|
||||
last = len(data) + 1
|
||||
applied = 0
|
||||
for start, end, repl in sorted(spans, reverse=True):
|
||||
if end > last:
|
||||
continue
|
||||
data = data[:start] + repl.encode("utf-8") + data[end:]
|
||||
last = start
|
||||
applied += 1
|
||||
atomic_write(path, data)
|
||||
report.append(f"unused import: {applied} removal(s) in {path.relative_to(REPO_ROOT)}")
|
||||
return report
|
||||
|
||||
|
||||
def static_prepass() -> list[str]:
|
||||
decl = re.compile(
|
||||
r"^[ \t]*pub\((?:crate|super)\)[ \t]+(?:async\s+|unsafe\s+|const\s+)*(?:(?:fn|struct|enum|const|static|type|trait|mod)[ \t]+([A-Za-z_]\w*))",
|
||||
re.M,
|
||||
)
|
||||
tok = re.compile(r"[A-Za-z_]\w*")
|
||||
consumer_tokens: set[str] = set()
|
||||
for crate in CONSUMER_DIRS:
|
||||
for path in (REPO_ROOT / "crates" / crate).rglob("*.rs"):
|
||||
if "target" in path.parts:
|
||||
continue
|
||||
consumer_tokens.update(tok.findall(path.read_text(encoding="utf-8", errors="replace")))
|
||||
edits: dict[tuple[Path, int], str | None] = {}
|
||||
skipped = 0
|
||||
for path in RUNTIME_SRC.rglob("*.rs"):
|
||||
lines = path.read_text(encoding="utf-8").split("\n")
|
||||
for i, line in enumerate(lines):
|
||||
m = decl.match(line)
|
||||
if not m:
|
||||
continue
|
||||
name = m.group(1)
|
||||
if name in COMMON_NAMES or (len(name) < 6 and "_" not in name):
|
||||
skipped += 1
|
||||
continue
|
||||
if name in consumer_tokens:
|
||||
new = widen_line(line, name)
|
||||
if new is not None:
|
||||
edits[(path, i + 1)] = new
|
||||
log = apply(edits) if edits else []
|
||||
log.append(f"prepass: {len(edits)} declaration(s) widened, {skipped} common/short name(s) left to the compiler")
|
||||
return log
|
||||
|
||||
|
||||
def plan_edits(messages: list[dict]) -> tuple[dict, list[str]]:
|
||||
edits: dict[tuple[Path, int], str | None] = {}
|
||||
unresolved: list[str] = []
|
||||
@@ -217,31 +319,57 @@ def set_lints(capped: bool) -> None:
|
||||
text = RUNTIME_MANIFEST.read_text(encoding="utf-8")
|
||||
old, new = (LINTS_ON, LINTS_CAPPED) if capped else (LINTS_CAPPED, LINTS_ON)
|
||||
if old in text:
|
||||
RUNTIME_MANIFEST.write_text(text.replace(old, new), encoding="utf-8")
|
||||
atomic_write(RUNTIME_MANIFEST, text.replace(old, new).encode("utf-8"))
|
||||
|
||||
|
||||
def install_signal_handlers() -> None:
|
||||
def handler(signum, _frame):
|
||||
raise Interrupted(signal.Signals(signum).name)
|
||||
|
||||
for sig in (signal.SIGTERM, signal.SIGHUP, signal.SIGINT):
|
||||
signal.signal(sig, handler)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
parser.add_argument("--cargo", default="scripts/dev-cargo.sh")
|
||||
parser.add_argument("--max-rounds", type=int, default=12)
|
||||
parser.add_argument("--packages", default="codewhale-runtime,codewhale-tui")
|
||||
parser.add_argument("--dest-crate", default="runtime")
|
||||
parser.add_argument("--src-crate", default="tui")
|
||||
parser.add_argument("--packages", help="comma separated; default codewhale-<dest>,codewhale-<src>")
|
||||
parser.add_argument("--extra", default="--lib --tests")
|
||||
parser.add_argument("--prepass", action="store_true", help="widen declarations the consumers name before the first compile")
|
||||
parser.add_argument("--keep-unused-imports", action="store_true", help="do not apply the compiler's unused-import removals")
|
||||
args = parser.parse_args(argv)
|
||||
configure(args.dest_crate, args.src_crate)
|
||||
cargo = shlex.split(args.cargo)
|
||||
packages = args.packages.split(",")
|
||||
packages = (args.packages or f"codewhale-{args.dest_crate},codewhale-{args.src_crate}").split(",")
|
||||
extra = shlex.split(args.extra)
|
||||
|
||||
if LINTS_CAPPED in RUNTIME_MANIFEST.read_text(encoding="utf-8"):
|
||||
print(f"{RUNTIME_MANIFEST.relative_to(REPO_ROOT)} was left capped by an earlier run; restoring it first", flush=True)
|
||||
set_lints(capped=False)
|
||||
|
||||
log: list[str] = []
|
||||
install_signal_handlers()
|
||||
atexit.register(lambda: set_lints(capped=False) if RUNTIME_MANIFEST.exists() else None)
|
||||
set_lints(capped=True)
|
||||
try:
|
||||
if args.prepass:
|
||||
log.extend(static_prepass())
|
||||
for round_no in range(1, args.max_rounds + 1):
|
||||
messages = run_cargo(cargo, packages, extra)
|
||||
edits, unresolved = plan_edits(messages)
|
||||
if not edits:
|
||||
suggestions = {} if args.keep_unused_imports else unused_import_suggestions(messages)
|
||||
if not edits and not suggestions:
|
||||
break
|
||||
changes = apply(edits)
|
||||
changes = apply(edits) if edits else []
|
||||
changes += apply_suggestions(suggestions)
|
||||
log.extend(changes)
|
||||
print(f"round {round_no}: {len(changes)} edit(s)", flush=True)
|
||||
except Interrupted as exc:
|
||||
print(f"interrupted by {exc}; manifest restored, edits so far stay in the tree", flush=True)
|
||||
return 130
|
||||
finally:
|
||||
set_lints(capped=False)
|
||||
for line in log:
|
||||
|
||||
Reference in new issue
Block a user