Compare commits

...
Author SHA1 Message Date
l0ng-ai ee0952e32e test(evals): measure Java code intelligence as the user sees it
`pnpm eval` drives the production workbench in Chromium against a release
`openidea-server` and jdtls, and judges each scenario on what the user sees: time to
feedback, to the first result and to the final one, and the results against
`evals/java/golden/`. Scenarios cover navigation, usages, implementations, gutter
markers, code lenses, diagnostics, completion, quick documentation and rename, in
`fixtures/java-app` and a pinned spring-petclinic, warm and cold. The report
(`target/eval/report.json` / `.md`) names each failure's kind, a repro command and the
server commands in the slowest run's window; `--save-baseline` compares later runs.

- The frontend gets `data-*` hooks the scenarios read: usage / chooser rows carry their
  location, the usages popup its loading state and scope, a hint whether it is progress
  and what it answers.
- `smoke/tauriShim.ts`: the in-page Tauri runtime shared by the jsdom smoke tests and
  the evals; `perf/lib/bundle.ts` builds the perf bundle for both.
- `.github/tool-versions.env` pins the language-server versions for Language smoke and
  the evals.
- `.agents/skills/eval`: the measure → fix → prove loop for agents.
2026-11-11 22:39:59 +08:00
Vadim LazukandInf1mum a0c3249368 fix(linux): keep the first-map hint under Mutter's auto-maximize share (#2520)
Wayland has no primary monitor, so the first-map size hint fell back to scale 1 with no work-area limit and requested the saved physical size as logical. On a HiDPI output the first mapped area exceeded Mutter's 80% share and the window opened maximized before the saved geometry was restored. The hint now converts with the largest output scale and shrinks proportionally to stay within the tightest output's share; X11 keeps its primary-monitor path.

Fixes #2400

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 22:29:05 +08:00
l0ng-ai 757412cb73 fix(security): reach content roots in plugin quick fixes and keep diagrams local
- A plugin quick fix may change or open files in the project's content
  roots (a project window group's members, extra Project Structure
  folders), still never git's own files; before, it was disabled for
  every file outside the primary root.
- `git init` (New Project › Create Git repository) uses the configured
  git executable again.
- Mermaid output is cleaned before it is shown: remote image sources,
  external `url()`, `@import` and address strings in its CSS are
  removed, so `style`/`classDef` values and HTML labels fetch nothing.
- The reply image e2e test loads a web image only after its button is
  clicked; the post-checkout hook test runs in a trusted scope.
- Capability docs list `project.write` as high risk.
2026-11-11 22:18:48 +08:00
Fabio Nunes Cerqueira a497178b3e fix(security): close the findings of GHSA-442p-gxfg-qxv9
- Restricted Mode git: reads run without a signature verifier, hooks
  (including config-defined ones), an external diff, filters or a lazy
  fetch, also inside submodules; every repository and every git starts
  restricted until the project is trusted.
- project.write and plugin quick fixes cannot reach git's own files under
  any spelling or through links; project.write is a high-risk grant.
- Built-in web server: each served root has its own port and token; a
  file outside the project is served with its folder minus dot names, or
  alone when that folder is the home folder or above it; no fixed port.
- Images, references, Markdown links and language-server links reach no
  network share other than the one already open.
- Marketplace: entries whose id or version no manifest may carry are
  dropped; an index read over the network names nothing on this machine;
  network reads are bounded in size and time.
- Agent replies load remote images on a click; the Markdown preview and
  Mermaid configuration no longer fetch from other hosts.
2026-11-11 22:18:48 +08:00
ntbang0901 8b179992e8 feat(profiler): profile allocations and live objects of JVM runs (#2650)
The Profiler gets two more modes for JVM runs through async-profiler: allocations (where the program allocates, in bytes) and live objects (what is still alive at the end). A live-object profile collects garbage first: run executors get a beforeStop hook, bounded to 12 s, that runs a GC before the stop signal; a program that ends by itself says that unreachable objects may show as live.

Part 3 of #2217.

Co-authored-by: ntbang0901 <[email protected]>
2026-11-11 22:13:38 +08:00
wangjunboit 85541b3663 feat(vcs): show a diff in a separate window (#2231)
Changes trees get Show Diff in New Window: the diff opens in its own native window (labelled diff-*, with the same capability as comparison windows), with the full diff toolbar, editable working-copy side and a save prompt for unsaved edits on close. Showing the same diff again reuses its window.

Co-authored-by: wangjunboit <[email protected]>
2026-11-11 22:07:04 +08:00
l0ng-ai 0418a2ce01 fix(java): explain a Maven build plugin that doesn't resolve when m2e only logs it (#2745)
When a build plugin such as maven-resources-plugin can't be resolved
(e.g. the default version is missing from a company mirror), m2e fails
the import's last step and every build but puts no marker on pom.xml,
so Problems stays empty and the "Maven project could not be configured"
notification never appeared. The notification now also fires for these
logged failures, once per plugin whichever way jdtls reports it.
2026-11-11 21:39:00 +08:00
l0ng-ai c8639b38ba fix(libraries): write only SDKs in angle brackets under External Libraries (#2743)
Every External Libraries group was shown as < name >, so a Maven or Gradle project listed hundreds of bracketed dependencies. As in IntelliJ IDEA, only SDKs (JDK, Python, Ruby, Go, Rust) keep the brackets; dependency groups such as Maven: g:a:v are shown as they are.

Refs #1977
2026-11-11 21:34:48 +08:00
Vadim LazukandInf1mum f04658612f fix(plugins): plugins reach a project window group's member files (#2739)
In a project window group plugins only saw the container: projects() and project-of() reported the container root, read/write/find refused member paths, Find in Files dropped member matches and the document indexer skipped member documents. The plugin host now treats the members like the editor and language servers do: projects() lists them, project-of() resolves to the deepest member, the sandbox path checks accept paths inside any member (files outside every root stay refused), and find-files, symbol search and the indexer cover member files. No WIT change.

Fixes #2506

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:31:23 +08:00
Qingandqtvbwfn d5cb379cae feat(services): show only services the user added (#2740)
The Services tool window listed services nobody added: an injected Docker connection, Spring Boot by default and every configuration that had ever run. Now only added types, connections and registries are listed, removal takes effect at once, an untouched injected Docker connection is dropped from older docker.json files, and the detail pane has a single scrollbar with console controls only (Run/Debug/Stop stay on the window toolbar).

Co-authored-by: qtvbwfn <[email protected]>
2026-11-11 21:28:41 +08:00
Muuandmuutot e3a818b607 feat(color-scheme): group Python color attributes as PyCharm does (#2735)
The Python page of Settings › Editor › Color Scheme listed its attributes flat. They now follow PyCharm's tree (String › f-string, Docstring, Braces and Operators, Functions, Parameters, Special names…), in its order; only display names change, so .icls files and existing overrides keep working.

Co-authored-by: muutot <[email protected]>
2026-11-11 21:26:31 +08:00
Vadim LazukandInf1mum 0b1872937a fix(lsp): hint when Vue 2 Options API template navigation lacks strictFunctionTypes (#2728)
Vue 2.7's Options API types only infer methods and computed into this under strictFunctionTypes; without it template go-to-definition and hover return nothing. When vue-language-server serves a Vue 2 project whose jsconfig/tsconfig lacks the option (and doesn't extend a base), the IDE now shows a warning with the fix.

Fixes #1163

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:25:01 +08:00
Vadim LazukandInf1mum f11cd361a7 fix(color-scheme): import .xml scheme files and tolerate real-world IDEA variants (#2731)
The import chooser only offered .icls, so schemes distributed as .xml (Material Theme UI) could not be picked. It now offers .xml too, and the parser accepts a BOM, an encoding declaration, a DOCTYPE, a scheme wrapped in a settings export and a scheme without a name (named after its file); a rejected file shows the reason.

Fixes #2714

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:23:38 +08:00
Vadim LazukandInf1mum 7034fe1640 fix(debugger): offer to reinstall Delve when it is too old for the Go toolchain (#2730)
A Delve older than the Go toolchain refuses the launch with "Version of Delve is too old for Go version …". The debugger now recognises it and offers Reinstall and Debug, which runs the managed go install …/dlv@latest and starts the session again.

Fixes #2721

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:23:03 +08:00
Vadim LazukandInf1mum fc264d3f5c fix(ui): stop double-clicking tree rows from selecting label text (#2726)
The second press of a double-click started the browser's word selection, which in the virtualized tree spilled onto the rows an expanding folder had just inserted. Tree cancels the default of repeated presses; the first press still selects and dblclick still activates.

Fixes #2713

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:22:02 +08:00
Vadim LazukandInf1mum 8e90341b3a feat(build-tools): a Skip Tests toggle for the Gradle tool window (#2734)
The Gradle tool window gets an optional Skip Tests toggle (off by default, per project): task runs and saved configurations started from the tree get -x test, unless the command line already excludes test or the build has no test task (where -x test would fail the run). A run that skips tests is not marked as a test run.

Fixes #2688

Co-authored-by: Inf1mum <[email protected]>
2026-11-11 21:20:58 +08:00
l0ng-ai 3644956755 feat(terminal): activate the project's conda environment in new terminals (#2742)
Activate virtualenv only handled venvs, so a conda project interpreter left new terminals in (base). New terminals now run conda activate <prefix> through the installation's shell hook (cmd, PowerShell, bash, zsh, POSIX sh, fish), finding the installation from the env itself, the parent of envs/, CONDA_EXE or PATH.

Fixes #2697
2026-11-11 21:18:24 +08:00
l0ng-ai 90e7e3162e feat(mcp): run the IDE's tools without confirmation in brave mode (#2741)
Settings › Agents › IDE MCP Server gets Run tools without confirmation (brave mode): builds, run configurations and the other tools that change a project start without asking, so an unattended agent isn't left waiting on a confirmation. Read from the IDE's own settings only; the project layer already refuses mcpServer.*.

Fixes #2660
2026-11-11 21:17:24 +08:00
Yann 833ae4c3c4 feat(editor): add and remove UTF-8 BOM (#2717)
The encoding status menu of a UTF-8 file gets Add UTF-8 BOM / Remove UTF-8 BOM, and the status bar shows "UTF-8 BOM" for a file that has one. The change marks the document modified and is written on save (at once with auto-save).

Co-authored-by: YannABC <[email protected]>
2026-11-11 21:00:48 +08:00
Elmir Abkerimov 584baa3f71 feat(editor): code vision position per hint and a "no usages" switch (#2716)
Settings › Editor › Inlay Hints › Code Vision gets a Position for Usages and for Code author — Top (a row above the declaration, the default) or Right (after the end of the declaration's first line) — placed per hint, and Show "no usages" (on by default) to leave the usages hint off declarations nothing uses. The options sit under the hint they belong to.

Co-authored-by: w4terlaw <[email protected]>
2026-11-11 20:58:46 +08:00
Elmir Abkerimov 98010e6513 fix(editor): start every selected row at the text's left edge (#2724)
CodeMirror starts selection rows that run through a line start at the left padding rounded down to whole pixels, and a row that starts at a character at the real padding; with the 2.5px inset the two were half a pixel apart, a visible step on Retina. Rows that start at the real edge now move to the rounded one, and the sliver drawn on a line the selection ends at the start of is dropped after the rows are grown.

Co-authored-by: w4terlaw <[email protected]>
2026-11-11 20:56:38 +08:00
Muuandmuutot a3640f655f fix(perf): name language server memory rows by their workspace root (#2702)
The memory indicator labelled every language server row with the project folder, so with additional content roots the servers of each root showed as identical rows. Each row now names the server's own workspace root.

Co-authored-by: muutot <[email protected]>
2026-11-11 20:56:07 +08:00
Shohruh Ismoilovandshohruh-genius 8aaa7daf83 fix(project): keep workspace project icons stable on removal (#2699)
Removing a member from a workspace reset every detected project type, so the remaining members briefly showed folder icons and fell back to name order. The cached types of members still in the same workspace are kept while the rescan runs.

Co-authored-by: shohruh-genius <[email protected]>
2026-11-11 20:54:16 +08:00
Bin 6a9a970df4 fix(editor): use JSX comments for expression children (#2703)
Ctrl+/ on a JSX expression child such as {label} read the JavaScript comment tokens at the opening brace; it now wraps the line in the surrounding JSX comment pair, {/* … */}, and takes it back out.

Co-authored-by: ffgenius <[email protected]>
2026-11-11 20:53:53 +08:00
l0ng-ai a309b7e6fa fix(java): import Gradle builds on the JVM the IDE's Gradle runs use (#2736)
With Gradle JVM at Project JDK the IDE's Gradle runs use the project JDK (or an installed JDK the Gradle version supports), but the jdtls import ran Gradle on jdtls's own JDK 21, so Gradle < 8.5 failed to import and no dependency resolved. The import now takes JAVA_HOME and GRADLE_USER_HOME from gradle_build_env for Gradle builds.

Refs #2681
2026-11-11 20:52:49 +08:00
Zhu Minhaoandzmh69695328 6971936f78 feat(java): run jdtls's Gradle import with the build settings (#2701)
jdtls's Gradle import ran on its own defaults (project wrapper only, ~/.gradle, the server's JDK), so a project without gradlew was never imported and one whose caches live in a configured Gradle user home resolved no dependencies. Settings › Build Tools › Gradle now reach java.import.gradle.*: home with the wrapper disabled for a local distribution, plus user.home and java.home, resolved like the IDE's own Gradle runs and merged into the existing settings.

Co-authored-by: zmh69695328 <[email protected]>
2026-11-11 20:49:28 +08:00
l0ng-ai 9d781a2f83 fix(agents): start and delete tasks without waiting on slow git work (#2732)
Starting a task in the local checkout recorded the already-changed files through the full change listing (diff, numstat, a read of every untracked file, quadratic lookups); the snapshot now uses the status alone and the listing uses hashed lookups. Deleting a task now drops it from the list before its worktree folder is deleted.

Refs #2667, #2666
2026-11-11 20:46:13 +08:00
l0ng-ai 3260e5e224 fix(vcs): open the source file on F4 in the changes trees (#2729)
F4 in the Commit tool window opened the selected file, then the tree handled the same key as Enter and opened the diff preview, which took the focus. Tree takes an onJumpToSource handler for F4 / Cmd+Down; the commit and Git Log changed-files trees use it to open the file itself.

Fixes #2696
2026-11-11 20:38:27 +08:00
l0ng-ai ed575a00d9 docs: show sponsor QR codes at the same size with labels (#2725) 2026-11-11 20:33:59 +08:00
l0ng-ai 2b41e61ce9 docs: add Alipay and WeChat Pay sponsor QR codes (#2723) 2026-11-11 20:29:16 +08:00
l0ng-ai 908740b3f2 feat(plugins): a marketplace that helps pick plugins (#2718)
* feat(plugins): a marketplace that helps pick plugins

Rows show each plugin's description, publisher, downloads and an inline
Install / Update button; plugins without an icon get a tinted initial.
While browsing, the list opens with suggestions for the open project and
the maintainers' picks, then all community plugins by downloads; the
official plugins come last as examples. The first plugin is selected so
the details are never empty, and they show release notes, the homepage
and why a plugin is suggested.

The index gains optional homepage, notes, downloads, featured and suggest
fields; openidea-plugin index fills homepage and notes from the manifest.

* fix(plugins): plugin icons fill their tile like the initials

* feat(plugins): drop the Preview badge from the plugins page

* feat(plugins): open the plugins page on Available
2026-11-11 20:13:28 +08:00
l0ng-ai e702e4da7b chore: v0.1.42 2026-11-11 19:31:09 +08:00
arookieofc 91023de207 chore(ci): route heap dump dialogs through the platform wrapper; clippy 1.99 and Windows test fixes (#2711)
Heap dump open/save use the platform dialog wrapper (GTK fallback on Linux). ide-heap allows clippy 1.99's large_enum_variant on the hot bit table. dialog.test compares paths with forward slashes on Windows.
2026-11-11 19:30:15 +08:00
l0ng-ai 02ee2880fe fix: translate New Project popup and error text; green CI on main (#2715)
Translate the project popup actions and the non-empty target error; translate the JDK field label. Fix clippy in ide-jdwp, formatting, RUSTSEC-2026-0486 (rustix 1.1.6) and agent E2E tests that expected the removed floating Quote button.
2026-11-11 19:25:09 +08:00
l0ng-ai 5a316e81d6 fix: Compile File sees its module's classes; list external agent sessions per project off the main thread (#2707)
Compile File passed javac only the dependencies and upstream modules, so any file using
another class of its own module failed with "cannot find symbol". The module's own
output folders now come first on the classpath, and its source folders (generated ones
included; main ones for a test) are the -sourcepath with -implicit:none, so classes that
were never built resolve from source while only the file itself is written.

Listing sessions started outside OpenIDEA parsed every Claude Code transcript of every
project on a synchronous command, which runs on the main thread and froze the window
(1.2 GB of transcripts here). It now reads only the transcript folders of the open
project, keeps sessions whose working directory is in the project, reads Codex rollouts
line by line, and runs on the blocking pool; importing does the same.
2026-11-11 18:22:28 +08:00
l0ng-ai 7c2ca4d061 fix: build Maven again when outputs were deleted; main menu in a separate toolbar (#2706)
Build Project / Build Module skipped Maven when no input changed, even after the
modules' target/classes were deleted outside the IDE (mvn clean in a terminal).
The up-to-date record now also lists the output folders that existed after the
build, and a build is skipped only while they all still exist.

The "Always show the main menu" toggle duplicated Main menu › Merged with main
toolbar, showed on macOS where it does nothing, and kept the titles in a title
bar too narrow for them. It is replaced by a third Main menu choice, Separate
toolbar: the menu titles get their own row below the title bar and never
collapse into the hamburger button.
2026-11-11 18:09:01 +08:00
ntbang0901 8d09479a65 feat(debugger): heap snapshots of a debugged JVM (#2649)
Capture a heap dump of the debugged JVM over the attach listener (GC.heap_dump, after checking free disk space and the JDK's rules for suspended VMs) or open an .hprof file, and browse it in an editor tab: classes with shallow and retained sizes, biggest objects, dominators, the path from an object to a GC root, and a comparison of two snapshots. A snapshot captured at a stop adds a Retained Size column to that stop's Memory tab.

New crate ide-heap: HotSpot hprof parser (bounds-checked; unfinished or crafted dumps are refused), object graph, dominator tree (semi-NCA), retained sizes and GC-root paths. Snapshots live in private directories removed with their window; an opened dump is analyzed from a private copy so changes to the user's file can't affect the mapped data.

Co-authored-by: ntbang0901 <[email protected]>
2026-11-11 17:43:07 +08:00
Bin 8fafde10b4 fix(editor): keep tab actions at the pane edge (#2682)
The trailing slot of the editor tab strip had flex: none, so the split / more buttons followed the last tab. An automatic left margin keeps them at the right edge of the pane without changing how wrapped tabs share space.

Co-authored-by: ffgenius <[email protected]>
2026-11-11 17:35:16 +08:00
Muhammad N. Fadhil b1cd917a96 fix(gradle): process resources before launches from the language server's classpath (#2678)
The Java language server copies resources unchanged into Buildship's bin/<source set>, so launches with "Build and run using: OpenIDEA" saw unfiltered resources (e.g. `version: @version@`). The Gradle sync now records which source sets Gradle changes resources for (filters, expand, renames, includes/excludes, extra sources); before such a launch the IDE runs only those processResources tasks and copies their output over the plain copies. Builds that copy resources as they are run no extra Gradle step; if Gradle fails, the launch continues with a hint.

Closes #2675

Co-authored-by: MuhammadNFadhil <[email protected]>
2026-11-11 17:33:43 +08:00
411 changed files with 28086 additions and 1527 deletions

No files matched your search

+83
View File
@@ -0,0 +1,83 @@
---
name: eval
description: Run the code-intelligence evals (`pnpm eval`), pick the worst failing scenario, diagnose it from the attached timeline, fix it, prove the fix with before / after numbers measured on this machine without regressing other scenarios, and open one PR. Use for "跑 eval", "优化 Java 代码智能", "eval 失败了", an issue labelled `eval`, or any change to Java navigation / usages / gutter markers that needs numbers.
---
# Eval loop: measure → fix → prove
`pnpm eval` drives the production workbench in Chromium against a release `openidea-server`
with real language servers (Java / jdtls for now) and judges every scenario on what the user
sees: time to feedback, to the first result, to the final one, and the results themselves vs
`evals/<language>/golden/`. One machine, before and after: numbers from different machines or
from the nightly host don't judge a fix.
## 0. Preconditions
- `jdtls` on PATH and `JAVA_HOME` pointing at a JDK of the pinned `JAVA` version (projects
compile against it, and the JDK scenarios' results are in its sources; jdtls itself runs on
a newer installed JDK when it needs one). Pinned versions: `.github/tool-versions.env`. A
mismatch prints a warning and the report records both: say so in the PR when it applies.
- First run builds `cargo build --release -p ide-host --bin openidea-server` and the perf
bundle; later iterations of a frontend-only change can use `--no-build` after a
`pnpm exec vite build --mode perf --outDir target/eval/dist`, a backend-only one after the
cargo build alone.
- Run evals one at a time on this machine, with nothing heavy (builds, other evals) beside them.
## 1. Baseline on the base commit
```sh
git switch --detach origin/main # or the fix branch's merge base
pnpm eval --save-baseline # target/eval/baseline.json, this machine only
```
Read `target/eval/report.json`. Each verdict has `status`, `failures` (`kind`: `error`,
`timeout`, `degraded`, `correctness`, `budget`, `golden`), `regressions`, `repro` (the command
that reruns it) and, for failures, `diagnostics`: every command sent to the server in the
slowest run's window (`cmd`, `start`, `ms`, `ok`, `bytes`, `items`), background tasks, server
status changes and the server's log tail.
## 2. Pick one problem
Worst first: wrong or missing results and silent degradation, then timeouts, then the
feedback budget (100 ms), then the slowest `completeMs`. One problem per PR. Say which
scenario and metric, with its value vs budget.
## 3. Diagnose from the timeline
- Commands in the window ordered by `start`: what ran, what waited on what, what's serial
that could overlap, which request returned many `items`/`bytes`, which failed.
- `feedbackMs` far above 16 ms with a fast `firstResultMs`: the UI shows nothing while it
waits (a hint or popup that only appears after a delay).
- A task still running across the window: background work competing with the request.
- For the frontend side rerun with `--trace` and open `target/eval/traces/<id>@<cond>.json`
in Chrome's Performance panel.
- Name the cause at class level (as in `issue-fix`): it usually affects sibling scenarios.
## 4. Fix, then rerun that scenario until it passes
```sh
pnpm eval --only <scenario> --condition <warm|cold> # the verdict's `repro`
```
Correctness is enforced, so a faster answer that drops, truncates or serves stale results
fails. Don't trade a budget miss for a degraded answer.
## 5. Full suite, then PR
```sh
pnpm eval # vs the baseline from step 1: no new failures or regressions
```
PR per `issue-fix` §6, plus in **How it was verified** the before / after table from
`target/eval/report.md` (both on this machine, commits named) and the tool versions line.
## Guard rails
- Goldens (`evals/*/golden/`) and budgets (`evals/budgets.json`) are human-owned. Don't edit
them, and don't run `--update-golden`, to make a scenario pass. When a golden looks wrong
(a tool bump, a deliberate behaviour change), stop and say so: a PR that touches them gets
`needs-decision`.
- Don't weaken a scenario's checks or timeouts in `evals/<language>/scenarios.ts` to pass.
Adding a scenario (for a case the sweep or a user found) is fine; record its golden with
`--update-golden --only <id>` and ask a person to review the file.
- A flaky timing (passes on rerun) is not a regression: rerun once before acting on one.
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/eval
+2
View File
@@ -0,0 +1,2 @@
custom:
- https://github.com/l0ng-ai/openidea#sponsor
Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 97 KiB

+14
View File
@@ -0,0 +1,14 @@
# Language server and tool versions for Language smoke (.github/workflows/language-smoke.yml)
# and the code-intelligence evals (pnpm eval). Pinned so a release can't fail an unrelated
# PR or move a golden result; bump them deliberately (and re-record goldens: pnpm eval --update-golden).
VTSLS=0.3.0
TYPESCRIPT=6.0.3
BASEDPYRIGHT=1.40.2
RUFF=0.17.3
GOPLS=v0.24.0
JDTLS=1.63.0-202610191322
KOTLIN_LSP=263.7084.0
GRADLE=8.14.3
INTELEPHENSE=1.18.5
# The JDK projects compile against (JAVA_HOME): library navigation opens its sources.
JAVA=21
+4 -11
View File
@@ -24,6 +24,7 @@ on:
- "package.json"
- "pnpm-lock.yaml"
- ".github/workflows/language-smoke.yml"
- ".github/tool-versions.env"
pull_request:
branches: [main]
types: [opened, synchronize, reopened]
@@ -38,16 +39,6 @@ concurrency:
env:
CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: 0
# Pinned so a server release can't fail an unrelated PR; bump them deliberately.
VTSLS: 0.3.0
TYPESCRIPT: 6.0.3
BASEDPYRIGHT: 1.40.2
RUFF: 0.17.3
GOPLS: v0.24.0
JDTLS: 1.63.0-202610191322
KOTLIN_LSP: 263.7084.0
GRADLE: 8.14.3
INTELEPHENSE: 1.18.5
jobs:
smoke:
@@ -61,6 +52,8 @@ jobs:
timeout-minutes: 40
steps:
- uses: actions/checkout@v4
# Tool versions, shared with the evals (pnpm eval).
- run: grep -E '^[A-Z0-9_]+=' .github/tool-versions.env >> "$GITHUB_ENV"
- uses: dtolnay/rust-toolchain@stable
with: { components: "rust-analyzer, rustfmt" }
# One build cache for every language; main saves it.
@@ -96,7 +89,7 @@ jobs:
- if: matrix.language == 'java' || matrix.language == 'kotlin'
uses: actions/setup-java@v4
with: { distribution: temurin, java-version: "21" }
with: { distribution: temurin, java-version: "${{ env.JAVA }}" }
- if: matrix.language == 'java'
run: |
mkdir -p "$RUNNER_TEMP/jdtls"
+1
View File
@@ -40,6 +40,7 @@ CLAUDE.local.md
.claude/skills/*
!.claude/skills/verify
!.claude/skills/issue-fix
!.claude/skills/eval
# Local reference screenshots
pycharm-screenshots/
+1
View File
@@ -9,6 +9,7 @@
"dist",
"target",
"src/bindings.ts",
"evals/*/golden",
"src/editor/core/cm/angular/parser*.ts",
"src/editor/core/cm/jsp/parser*.ts",
"src/editor/core/cm/astro/parser*.ts",
+11 -3
View File
@@ -24,8 +24,8 @@ Visual style: dark, low-contrast greys, one cool accent
Plugin docs: `plugins/README.md`, `plugins/themes.md`.
- `java/` — JDT LS extension bundles (`java-usages`, `kotlin-bridge`) and the patched
Lombok agent, embedded into the app by `ide-sdk` / `ide-tools`.
- `e2e/` — Playwright UI tests (`pnpm e2e`). `fixtures/` — sample projects per language
for manual and automated testing.
- `e2e/` — Playwright UI tests (`pnpm e2e`). `evals/` — code-intelligence evals (`pnpm eval`).
`fixtures/` — sample projects per language for manual and automated testing.
- `docs/` — the Mintlify documentation site (`docs.json` navigation, `en/` and `zh/` pages);
preview with `npx mint dev` in `docs/`.
@@ -59,9 +59,12 @@ Unclear product decisions → `needs-decision` and stop.
## Skills
Do not use Impeccable or its related skills in this repository, including globally installed copies.
Shared agent skills live in `.agents/skills/` (`.claude/skills/` links to them):
`verify` drives the running app through Tauri MCP and captures screenshot evidence;
`issue-fix` takes an issue from reproduction to one PR.
`issue-fix` takes an issue from reproduction to one PR;
`eval` takes a failing code-intelligence eval to a fix with before / after numbers.
## Commands
@@ -81,6 +84,11 @@ Shared agent skills live in `.agents/skills/` (`.claude/skills/` links to them):
`pnpm gen:bindings`.
- `pnpm perf` (UI interaction latency / frames / leaks), `pnpm perf:rust` (criterion
benches vs budgets). Budgets: `perf/budgets.json`.
- `pnpm eval [--only <id>] [--runs n] [--save-baseline]` — Java code intelligence (navigation,
usages, gutter markers) in the production workbench against a release `openidea-server` and
jdtls: time to feedback / first result / complete, results vs `evals/<language>/golden/`.
Report: `target/eval/report.json`. Goldens and `evals/budgets.json` are changed by people only.
Tool versions: `.github/tool-versions.env` (shared with Language smoke).
- `pnpm release:prepare <version>` — the release commit's changes: app version everywhere,
`Cargo.lock`, and the plugin API snapshot when `wit/` moved to a new version.
Generated
+67 -46
View File
@@ -4145,7 +4145,7 @@ checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
[[package]]
name = "ide-agent"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"encoding_rs",
@@ -4169,7 +4169,7 @@ dependencies = [
[[package]]
name = "ide-analysis"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ast-grep-core",
"ide-core",
@@ -4183,7 +4183,7 @@ dependencies = [
[[package]]
name = "ide-android"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"serde",
@@ -4194,7 +4194,7 @@ dependencies = [
[[package]]
name = "ide-build"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"chardetng",
@@ -4212,7 +4212,7 @@ dependencies = [
[[package]]
name = "ide-cli"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dirs 6.0.0",
"serde",
@@ -4226,7 +4226,7 @@ dependencies = [
[[package]]
name = "ide-clipboard"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"gtk",
"objc2 0.6.5",
@@ -4236,7 +4236,7 @@ dependencies = [
[[package]]
name = "ide-core"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"encoding_rs",
@@ -4253,7 +4253,7 @@ dependencies = [
[[package]]
name = "ide-coverage"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-run",
@@ -4267,10 +4267,11 @@ dependencies = [
[[package]]
name = "ide-dap"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"futures",
"ide-core",
"ide-heap",
"ide-jdwp",
"ide-jsonrpc",
"ide-jvm-attach",
@@ -4292,7 +4293,7 @@ dependencies = [
[[package]]
name = "ide-db"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"bb8",
"blake3",
@@ -4322,7 +4323,7 @@ dependencies = [
[[package]]
name = "ide-docker"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"serde",
@@ -4335,7 +4336,7 @@ dependencies = [
[[package]]
name = "ide-extension"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"base64 0.22.1",
"blake3",
@@ -4356,7 +4357,7 @@ dependencies = [
[[package]]
name = "ide-fonts"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"objc2-core-foundation",
"objc2-core-text",
@@ -4368,7 +4369,7 @@ dependencies = [
[[package]]
name = "ide-github"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-vcs",
@@ -4379,9 +4380,27 @@ dependencies = [
"which",
]
[[package]]
name = "ide-heap"
version = "0.1.42"
dependencies = [
"criterion",
"ide-core",
"libc",
"memmap2",
"proptest",
"serde",
"serde_json",
"specta",
"tempfile",
"thiserror 2.0.21",
"tracing",
"uuid",
]
[[package]]
name = "ide-host"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"base64 0.22.1",
"blake3",
@@ -4401,6 +4420,7 @@ dependencies = [
"ide-db",
"ide-docker",
"ide-extension",
"ide-heap",
"ide-http",
"ide-index",
"ide-jsonrpc",
@@ -4449,7 +4469,7 @@ dependencies = [
[[package]]
name = "ide-http"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-vfs",
@@ -4468,7 +4488,7 @@ dependencies = [
[[package]]
name = "ide-index"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"criterion",
@@ -4490,7 +4510,7 @@ dependencies = [
[[package]]
name = "ide-jdwp"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-jvm-inspect",
"proptest",
@@ -4502,7 +4522,7 @@ dependencies = [
[[package]]
name = "ide-jsonrpc"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"encoding_rs",
"futures",
@@ -4519,7 +4539,7 @@ dependencies = [
[[package]]
name = "ide-jvm-attach"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"libc",
"tempfile",
@@ -4528,7 +4548,7 @@ dependencies = [
[[package]]
name = "ide-jvm-inspect"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"futures",
"serde",
@@ -4539,7 +4559,7 @@ dependencies = [
[[package]]
name = "ide-language"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"criterion",
@@ -4595,7 +4615,7 @@ dependencies = [
[[package]]
name = "ide-lsp"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"futures",
@@ -4622,7 +4642,7 @@ dependencies = [
[[package]]
name = "ide-mcp"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"bytes",
"dunce",
@@ -4641,7 +4661,7 @@ dependencies = [
[[package]]
name = "ide-notebook"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"futures",
"ide-core",
@@ -4664,7 +4684,7 @@ dependencies = [
[[package]]
name = "ide-plugin"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"dunce",
@@ -4687,9 +4707,10 @@ dependencies = [
[[package]]
name = "ide-profiler"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-jvm-attach",
"ide-run",
"serde",
"serde_json",
@@ -4700,7 +4721,7 @@ dependencies = [
[[package]]
name = "ide-remote"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"globset",
@@ -4719,7 +4740,7 @@ dependencies = [
[[package]]
name = "ide-run"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"encoding_rs",
@@ -4743,7 +4764,7 @@ dependencies = [
[[package]]
name = "ide-scaffold"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-tools",
@@ -4759,7 +4780,7 @@ dependencies = [
[[package]]
name = "ide-sdk"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dunce",
"ide-core",
@@ -4779,7 +4800,7 @@ dependencies = [
[[package]]
name = "ide-search"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"criterion",
"encoding_rs",
@@ -4800,7 +4821,7 @@ dependencies = [
[[package]]
name = "ide-spell"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"parking_lot",
@@ -4813,7 +4834,7 @@ dependencies = [
[[package]]
name = "ide-ssh"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dirs 6.0.0",
"flate2",
@@ -4834,7 +4855,7 @@ dependencies = [
[[package]]
name = "ide-syntax"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"ide-language",
@@ -4849,7 +4870,7 @@ dependencies = [
[[package]]
name = "ide-terminal"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"ide-core",
"libc",
@@ -4862,7 +4883,7 @@ dependencies = [
[[package]]
name = "ide-text"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"criterion",
"ide-core",
@@ -4875,7 +4896,7 @@ dependencies = [
[[package]]
name = "ide-tools"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"flate2",
"hex",
@@ -4896,7 +4917,7 @@ dependencies = [
[[package]]
name = "ide-vcs"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"dunce",
@@ -4917,7 +4938,7 @@ dependencies = [
[[package]]
name = "ide-vfs"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"chardetng",
"dunce",
@@ -4941,11 +4962,11 @@ dependencies = [
[[package]]
name = "ide-web"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"dirs 6.0.0",
"ide-core",
"open",
"parking_lot",
"percent-encoding",
"serde",
"specta",
@@ -4957,7 +4978,7 @@ dependencies = [
[[package]]
name = "ide-workspace"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"blake3",
"dunce",
@@ -6535,7 +6556,7 @@ dependencies = [
[[package]]
name = "openidea"
version = "0.1.41"
version = "0.1.42"
dependencies = [
"base64 0.22.1",
"blake3",
@@ -7776,9 +7797,9 @@ dependencies = [
[[package]]
name = "rustix"
version = "1.1.5"
version = "1.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
checksum = "e5e76742e5294153d4dde1207dbec5bdd81eef6fabdc0d2118babadabfd858aa"
dependencies = [
"bitflags 2.13.2",
"errno",
+3 -1
View File
@@ -5,7 +5,7 @@ members = ["crates/*", "src-tauri"]
exclude = ["sdk", "plugins"]
[workspace.package]
version = "0.1.41"
version = "0.1.42"
edition = "2024"
rust-version = "1.96"
license = "Apache-2.0"
@@ -51,6 +51,7 @@ ide-cli = { path = "crates/ide-cli" }
ide-jvm-attach = { path = "crates/ide-jvm-attach" }
ide-jvm-inspect = { path = "crates/ide-jvm-inspect" }
ide-jdwp = { path = "crates/ide-jdwp" }
ide-heap = { path = "crates/ide-heap" }
# serialization & IPC typing
serde = { version = "1", features = ["derive"] }
@@ -86,6 +87,7 @@ trash = "5"
blake3 = "1"
dunce = "1"
dirs = "6"
memmap2 = "0.9"
zip = { version = "9.2.3", default-features = false, features = ["deflate"] }
# search, index, language
+13
View File
@@ -19,6 +19,8 @@
<a href="#plugins">Plugins</a>
&nbsp;·&nbsp;
<a href="#community">Community</a>
&nbsp;·&nbsp;
<a href="#sponsor">Sponsor</a>
</p>
https://github.com/user-attachments/assets/234dd54c-1255-4b3a-b35c-533d78d737c0
@@ -68,6 +70,17 @@ the [official plugins](plugins/official). The plugin API is still a preview.
- Contributing: see [CONTRIBUTING.md](CONTRIBUTING.md) and the [contributor docs](https://openidea.sh/en/contribute/setup).
## Sponsor
If OpenIDEA saves you time, you can support its development with Alipay or WeChat Pay.
<table>
<tr>
<td align="center"><img src=".github/assets/sponsor-alipay.png" alt="Alipay QR code" width="200"><br>Alipay</td>
<td align="center"><img src=".github/assets/sponsor-wechat.png" alt="WeChat Pay QR code" width="200"><br>WeChat Pay</td>
</tr>
</table>
## License
[Apache 2.0](LICENSE). Bundled third-party assets are listed in [`THIRD_PARTY_LICENSES.md`](THIRD_PARTY_LICENSES.md).
+70 -15
View File
@@ -8,6 +8,7 @@
use super::native::claude::history;
use serde::Serialize;
use specta::Type;
use std::io::BufRead;
use std::path::{Path, PathBuf};
/// A session on this machine that no task holds yet.
@@ -62,11 +63,28 @@ fn message_text(e: &serde_json::Value) -> Option<String> {
}
}
/// Claude Code transcripts: `<config>/projects/*/<id>.jsonl`.
pub fn claude_sessions(config_dir: &Path) -> Vec<ExternalSession> {
/// Whether a session that ran in `cwd` belongs to the project at `root` (in it or below it).
fn in_project(cwd: Option<&str>, root: &Path) -> bool {
cwd.is_some_and(|cwd| Path::new(cwd).starts_with(root))
}
/// Claude Code transcripts of the project at `root`: `<config>/projects/<escaped cwd>/<id>.jsonl`.
/// Only the folders named after `root` or a folder below it are read, so other projects'
/// transcripts (often gigabytes) are never opened.
pub fn claude_sessions(config_dir: &Path, root: &Path) -> Vec<ExternalSession> {
let mut out = Vec::new();
let projects = config_dir.join("projects");
let dirs: Vec<PathBuf> = std::fs::read_dir(&projects).into_iter().flatten().flatten().map(|d| d.path()).collect();
let escaped = history::escape(root);
let dirs: Vec<PathBuf> = std::fs::read_dir(&projects)
.into_iter()
.flatten()
.flatten()
.filter(|d| {
let name = d.file_name().to_string_lossy().into_owned();
name == escaped || name.starts_with(&format!("{escaped}-"))
})
.map(|d| d.path())
.collect();
for dir in dirs {
let files: Vec<PathBuf> = std::fs::read_dir(&dir)
.into_iter()
@@ -93,6 +111,10 @@ pub fn claude_sessions(config_dir: &Path) -> Vec<ExternalSession> {
.filter_map(|e| e.get("cwd").and_then(serde_json::Value::as_str))
.next()
.map(str::to_owned);
// Escaping is lossy (`a/b-c` and `a/b/c` share a folder): the transcript decides.
if !in_project(cwd.as_deref(), root) {
continue;
}
out.push(ExternalSession {
provider: "claude".to_owned(),
session_id: id,
@@ -138,12 +160,13 @@ fn codex_user_text(line: &serde_json::Value) -> Option<String> {
/// One rollout file's session, or `None` when it names no `session_meta`.
fn codex_file(path: &Path) -> Option<ExternalSession> {
let text = std::fs::read_to_string(path).ok()?;
let file = std::fs::File::open(path).ok()?;
let mut id: Option<(String, Option<String>)> = None;
let mut title: Option<String> = None;
// `session_meta` is first, but cap the scan: giant rollouts must not stall listing.
for line in text.lines().take(400) {
let Ok(v) = serde_json::from_str::<serde_json::Value>(line) else { continue };
for line in std::io::BufReader::new(file).lines().take(400) {
let Ok(line) = line else { break };
let Ok(v) = serde_json::from_str::<serde_json::Value>(&line) else { continue };
if id.is_none()
&& let Some(meta) = codex_meta(&v)
{
@@ -183,11 +206,11 @@ fn rollout_files(dir: &Path, depth: usize, out: &mut Vec<PathBuf>) {
}
}
/// Codex sessions: `<home>/.codex/sessions/**`.
pub fn codex_sessions(codex_home: &Path) -> Vec<ExternalSession> {
/// Codex sessions of the project at `root`: `<home>/.codex/sessions/**`.
pub fn codex_sessions(codex_home: &Path, root: &Path) -> Vec<ExternalSession> {
let mut files = Vec::new();
rollout_files(&codex_home.join("sessions"), 4, &mut files);
files.iter().filter_map(|p| codex_file(p)).collect()
files.iter().filter_map(|p| codex_file(p)).filter(|s| in_project(s.cwd.as_deref(), root)).collect()
}
#[cfg(test)]
@@ -222,7 +245,7 @@ mod tests {
],
);
write(&tmp.path().join("projects/-work-app/empty.jsonl"), &[]);
let sessions = claude_sessions(tmp.path());
let sessions = claude_sessions(tmp.path(), Path::new("/work/app"));
assert_eq!(sessions.len(), 1);
let s = &sessions[0];
assert_eq!((s.provider.as_str(), s.session_id.as_str()), ("claude", "abc123"));
@@ -234,8 +257,8 @@ mod tests {
#[test]
fn claude_title_falls_back_to_the_first_prompt() {
let tmp = tempfile::tempdir().unwrap();
write(&tmp.path().join("projects/-x/def456.jsonl"), &[claude_user("u1", "hello there")]);
let sessions = claude_sessions(tmp.path());
write(&tmp.path().join("projects/-work-app/def456.jsonl"), &[claude_user("u1", "hello there")]);
let sessions = claude_sessions(tmp.path(), Path::new("/work/app"));
assert_eq!(sessions.len(), 1);
assert_eq!(sessions[0].title, "hello there");
}
@@ -253,7 +276,7 @@ mod tests {
],
);
write(&tmp.path().join("sessions/notes.jsonl"), &[json!({"nope": true})]);
let sessions = codex_sessions(tmp.path());
let sessions = codex_sessions(tmp.path(), Path::new("/work"));
assert_eq!(sessions.len(), 1);
let s = &sessions[0];
assert_eq!((s.provider.as_str(), s.session_id.as_str()), ("codex", "thread-1"));
@@ -264,7 +287,39 @@ mod tests {
#[test]
fn missing_directories_list_nothing() {
let tmp = tempfile::tempdir().unwrap();
assert!(claude_sessions(&tmp.path().join("nope")).is_empty());
assert!(codex_sessions(&tmp.path().join("nope")).is_empty());
assert!(claude_sessions(&tmp.path().join("nope"), Path::new("/work")).is_empty());
assert!(codex_sessions(&tmp.path().join("nope"), Path::new("/work")).is_empty());
}
#[test]
fn only_sessions_of_the_project_are_listed() {
let tmp = tempfile::tempdir().unwrap();
let user = |cwd: &str| {
json!({"type": "user", "uuid": "u1", "parentUuid": null,
"message": {"role": "user", "content": "hi"}, "cwd": cwd})
};
write(&tmp.path().join("projects/-work-app/in-root.jsonl"), &[user("/work/app")]);
write(&tmp.path().join("projects/-work-app-web/below.jsonl"), &[user("/work/app/web")]);
// `/work/app-old` escapes like a folder below `/work/app`; its transcript says otherwise.
write(&tmp.path().join("projects/-work-app-old/sibling.jsonl"), &[user("/work/app-old")]);
write(&tmp.path().join("projects/-other/elsewhere.jsonl"), &[user("/other")]);
let mut ids: Vec<String> =
claude_sessions(tmp.path(), Path::new("/work/app")).into_iter().map(|s| s.session_id).collect();
ids.sort();
assert_eq!(ids, ["below", "in-root"]);
let rollout = |name: &str, cwd: &str| {
write(
&tmp.path().join(format!("codex/sessions/2026/01/02/{name}.jsonl")),
&[json!({"payload": {"type": "session_meta", "payload": {"id": name, "cwd": cwd}}})],
)
};
rollout("mine", "/work/app/api");
rollout("theirs", "/other");
let codex: Vec<String> = codex_sessions(&tmp.path().join("codex"), Path::new("/work/app"))
.into_iter()
.map(|s| s.session_id)
.collect();
assert_eq!(codex, ["mine"]);
}
}
+15 -12
View File
@@ -592,17 +592,19 @@ impl AgentManager {
self.tasks.read().values().filter_map(|t| t.state.lock().session_id.clone()).collect()
}
/// Sessions on this machine that no task holds yet, newest first (#312).
/// Sessions of this project on this machine that no task holds yet, newest first. Reads
/// transcripts from disk: call it off the UI and command threads.
pub fn external_sessions(&self) -> Vec<ExternalSession> {
let mut out = Vec::new();
let root = self.shared.paths.project_root.clone();
if let Some(spec) = self.shared.spec("claude")
&& let Some(dir) = native::claude::cli::config_dir(&spec.env)
{
out.extend(external::claude_sessions(&dir));
out.extend(external::claude_sessions(&dir, &root));
}
let home = self.shared.paths.home_dir.clone().or_else(home_dir);
if let Some(home) = home {
out.extend(external::codex_sessions(&home.join(".codex")));
out.extend(external::codex_sessions(&home.join(".codex"), &root));
}
let known = self.imported_session_ids();
out.retain(|s| !known.contains(&s.session_id));
@@ -1232,15 +1234,6 @@ impl AgentManager {
};
// A copy of the task (a fork) may work in the same worktree.
let shared = self.tasks.read().values().any(|t| t.id != id && t.state.lock().task.cwd.0 == cwd);
if remove_worktree && !shared && isolation == Isolation::Worktree && cwd.exists() {
// On Windows the agent's processes lock the folder (their working directory,
// open files) until they have exited.
if let Some(peer) = live {
let _ = tokio::time::timeout(EXIT_WAIT, peer.exited()).await;
}
let worktrees = self.shared.paths.worktrees_dir.clone();
blocking(move || worktree::remove(&root, &worktrees, &cwd)).await?;
}
self.tasks.write().remove(id);
*task.deleted.lock() = true;
let _ = std::fs::remove_file(self.shared.paths.store_dir.join(format!("{id}.json")));
@@ -1250,7 +1243,17 @@ impl AgentManager {
{
let _ = std::fs::remove_dir_all(self.shared.paths.store_dir.join(saved));
}
// The list drops the task now; removing a large worktree's folder can take a while.
self.shared.emit(AgentEvent::Removed { id: id.to_owned() });
if remove_worktree && !shared && isolation == Isolation::Worktree && cwd.exists() {
// On Windows the agent's processes lock the folder (their working directory,
// open files) until they have exited.
if let Some(peer) = live {
let _ = tokio::time::timeout(EXIT_WAIT, peer.exited()).await;
}
let worktrees = self.shared.paths.worktrees_dir.clone();
blocking(move || worktree::remove(&root, &worktrees, &cwd)).await?;
}
Ok(())
}
@@ -31,7 +31,7 @@ pub(super) fn transcript(config_dir: &Path, cwd: &Path, id: &str) -> PathBuf {
}
/// The project directory name Claude Code uses for `cwd`.
pub(super) fn escape(cwd: &Path) -> String {
pub(crate) fn escape(cwd: &Path) -> String {
cwd.to_string_lossy().chars().map(|c| if c.is_ascii_alphanumeric() { c } else { '-' }).collect()
}
+22 -7
View File
@@ -14,8 +14,18 @@ pub(crate) struct Created {
pub base_commit: String,
}
/// The repository of `path`, free to execute: the host runs every agent command that
/// reaches a repository only in a trusted project.
fn discover(path: &Path) -> IdeResult<Option<Repo>> {
let repo = Repo::discover(path)?;
if let Some(repo) = &repo {
repo.set_execution_allowed(true);
}
Ok(repo)
}
fn repo_at(path: &Path) -> IdeResult<Repo> {
Repo::discover(path)?.ok_or_else(|| IdeError::unavailable(format!("{} is not in a git repository", path.display())))
discover(path)?.ok_or_else(|| IdeError::unavailable(format!("{} is not in a git repository", path.display())))
}
/// The current branch (or commit when detached) and its commit.
@@ -91,10 +101,11 @@ fn retrying(attempts: u32, pause: Duration, mut op: impl FnMut() -> std::io::Res
/// Files under `cwd` that differ from `base` (untracked files included), with line counts.
/// Outside a repository there is nothing to compare against: no changes.
pub(crate) fn changes(cwd: &Path, base: &str) -> IdeResult<Vec<AgentChange>> {
let Some(repo) = Repo::discover(cwd)? else { return Ok(Vec::new()) };
let Some(repo) = discover(cwd)? else { return Ok(Vec::new()) };
let root = repo.root().to_path_buf();
let stats = repo.numstat_with_worktree(base)?;
let stat = |rel: &str| stats.iter().find(|(p, ..)| p == rel).map_or((0, 0), |&(_, a, r)| (a, r));
let stats: std::collections::HashMap<String, (u32, u32)> =
repo.numstat_with_worktree(base)?.into_iter().map(|(p, a, r)| (p, (a, r))).collect();
let stat = |rel: &str| stats.get(rel).copied().unwrap_or((0, 0));
let mut out: Vec<AgentChange> = repo
.diff_with_worktree(base)?
.into_iter()
@@ -103,8 +114,9 @@ pub(crate) fn changes(cwd: &Path, base: &str) -> IdeResult<Vec<AgentChange>> {
AgentChange { path: IpcPath(root.join(&f.path)), rel_path: f.path, kind: f.kind, added, removed }
})
.collect();
let tracked: std::collections::HashSet<String> = out.iter().map(|c| c.rel_path.clone()).collect();
for s in repo.status()? {
if s.unstaged == Some(ChangeKind::Untracked) && !out.iter().any(|c| c.rel_path == s.rel_path) {
if s.unstaged == Some(ChangeKind::Untracked) && !tracked.contains(&s.rel_path) {
let added = std::fs::read_to_string(&*s.path).map_or(0, |t| t.lines().count() as u32);
out.push(AgentChange { path: s.path, rel_path: s.rel_path, kind: ChangeKind::Added, added, removed: 0 });
}
@@ -124,7 +136,10 @@ pub(crate) fn stamp(path: &Path) -> Stamp {
/// The files already changed in `cwd` against `HEAD`, stamped so later edits to them show.
pub(crate) fn snapshot(cwd: &Path) -> IdeResult<BTreeMap<String, Stamp>> {
Ok(changes(cwd, "HEAD")?.into_iter().map(|c| (c.rel_path, stamp(&c.path))).collect())
// Paths and stamps only: no diff and no line counts, which read every untracked file
// and held up starting a task in a checkout with many of them.
let Some(repo) = Repo::discover(cwd)? else { return Ok(BTreeMap::new()) };
Ok(repo.status()?.into_iter().map(|s| (s.rel_path, stamp(&s.path))).collect())
}
/// Drops the changes that were there before the task and haven't been touched since.
@@ -204,7 +219,7 @@ pub(crate) fn restore(
}
pub(crate) fn file_at(cwd: &Path, path: &Path, rev: &str) -> IdeResult<Option<String>> {
let Some(repo) = Repo::discover(cwd)? else { return Ok(None) };
let Some(repo) = discover(cwd)? else { return Ok(None) };
let encoding = match ide_vfs::load_text(path) {
Ok(loaded) => encoding_rs::Encoding::for_label(loaded.encoding.as_bytes()).unwrap_or(encoding_rs::UTF_8),
Err(_) => ide_vfs::detect_encoding(&repo.file_bytes_at(path, Some(rev))?.unwrap_or_default(), None),
@@ -265,6 +265,14 @@ pub fn menu_fingerprint(
}
impl LaunchFingerprint {
/// Folders (relative to the project) of the Maven projects whose POM is an input.
pub(super) fn project_dirs(&self) -> impl Iterator<Item = &Path> {
self.files
.keys()
.filter(|p| p.is_relative() && p.file_name().is_some_and(|n| n == "pom.xml"))
.map(|pom| pom.parent().unwrap_or(Path::new("")))
}
/// Relative paths whose content changed while `(length, mtime)` stayed the same
/// compared to `previous` (timestamp-preserving external edits, coarse-granularity
/// filesystems). Timestamp-incremental builders (Maven, the Kotlin plugin) would
@@ -0,0 +1,253 @@
//! Resources a Gradle build changes while copying them (filters, `expand`, renames, extra
//! sources). The Java language server copies them as they are into Buildship's
//! `bin/<source set>` folders, which launches run from; IntelliJ IDEA runs Gradle's copy.
//! Before a launch, Gradle processes them and its output replaces those plain copies, so
//! each resource is on the classpath once, as the build produces it.
use super::{BuildPlan, BuildStep, BuildSystem, Format, run, subdir};
use crate::BuildModel;
use crate::gradle::{self, GradleModel, ReportedProject};
use ide_core::{CancellationToken, IdeError, IdeResult, Severity};
use std::fs;
use std::path::{Path, PathBuf};
/// A source set whose resources Gradle processes, with the folder the launch reads them from.
#[derive(Debug, PartialEq)]
struct Target {
/// Task path from the root build (`:app:processResources`).
task: String,
/// Where the task writes the resources.
output: PathBuf,
/// The Java language server's output folder of the source set, on the launch classpath.
jdtls: PathBuf,
}
fn targets(reported: &[ReportedProject], class_paths: &[String]) -> Vec<Target> {
// buildSrc is built before the build, never addressable as `:buildSrc:<task>`.
let addressable = |p: &&ReportedProject| !p.build.as_deref().is_some_and(|b| b.ends_with(":buildSrc"));
let sets: Vec<_> = reported
.iter()
.filter(addressable)
.flat_map(|p| p.source_sets.iter().filter_map(move |s| Some((p, s, s.processed.as_ref()?))))
.collect();
if sets.is_empty() {
return Vec::new();
}
// Canonical, as Gradle reports its folders: jdtls may spell them differently.
let entries: Vec<PathBuf> = class_paths.iter().filter_map(|p| dunce::canonicalize(p).ok()).collect();
sets.into_iter()
.filter_map(|(p, set, resources)| {
// Buildship's output folder of the source set.
let jdtls = dunce::canonicalize(Path::new(&p.dir).join("bin").join(&set.name)).ok()?;
let project = if p.path == ":" { "" } else { p.path.as_str() };
entries.contains(&jdtls).then(|| Target {
task: format!("{project}:{}", resources.task),
output: PathBuf::from(&resources.dir),
jdtls,
})
})
.collect()
}
/// Before a JVM launch from the Java language server's output folders on `class_paths`:
/// runs the Gradle tasks that process resources of those folders' source sets (see the
/// module docs) and copies their output over the plain copies. `cache_dir` holds the
/// linked builds' last syncs. Returns the tasks run, none for builds that copy their
/// resources as they are.
pub fn process_launch_resources(
root: &Path,
model: &BuildModel,
cache_dir: &Path,
class_paths: &[String],
cancel: &CancellationToken,
) -> IdeResult<Vec<String>> {
// A composite also reports the builds it includes: the deepest build owning a folder runs its task.
let mut builds: Vec<&GradleModel> = model.gradle.iter().collect();
builds.sort_by_key(|g| std::cmp::Reverse(subdir(root, &g.dir).components().count()));
let mut done: Vec<PathBuf> = Vec::new();
let mut tasks = Vec::new();
for g in builds {
let reported = crate::sync::reported_gradle_projects(&gradle::cache_dir(cache_dir, &g.dir));
let targets: Vec<Target> =
targets(&reported, class_paths).into_iter().filter(|t| !done.contains(&t.jdtls)).collect();
if targets.is_empty() {
continue;
}
let mut step = BuildStep::new(&g.command, &["--console=plain"], &subdir(root, &g.dir), Format::Gradle);
step.args.extend(targets.iter().map(|t| t.task.clone()));
step.env = g.env.clone();
let plan = BuildPlan { system: BuildSystem::Gradle, title: "Process resources".into(), steps: vec![step] };
let result = run(root, &plan, cancel, &|_| {})?;
if !result.success {
let errors: Vec<&str> = result
.messages
.iter()
.filter(|m| m.severity == Severity::Error)
.take(5)
.map(|m| m.text.as_str())
.collect();
return Err(IdeError::internal(errors.join("\n")));
}
for t in targets {
if cancel.is_cancelled() {
return Err(IdeError::cancelled());
}
overlay(&t.output, &t.output, &t.jdtls)?;
done.push(t.jdtls);
tasks.push(t.task);
}
}
Ok(tasks)
}
/// Copies the files under `dir` to the same place under `to` (relative to `base`) where
/// they differ, so unchanged resources are not rewritten on every launch.
fn overlay(base: &Path, dir: &Path, to: &Path) -> IdeResult<()> {
let entries = match fs::read_dir(dir) {
// No resources: the task had nothing to copy.
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()),
entries => entries?,
};
for entry in entries {
let entry = entry?;
let path = entry.path();
let kind = entry.file_type()?;
if kind.is_dir() {
overlay(base, &path, to)?;
} else if kind.is_file() {
let dest = to.join(path.strip_prefix(base).map_err(|e| IdeError::internal(e.to_string()))?);
if !same_content(&path, &dest) {
if let Some(parent) = dest.parent() {
fs::create_dir_all(parent)?;
}
fs::copy(&path, &dest)?;
}
}
}
Ok(())
}
fn same_content(a: &Path, b: &Path) -> bool {
match (fs::metadata(a), fs::metadata(b)) {
(Ok(m), Ok(n)) if m.len() == n.len() => matches!((fs::read(a), fs::read(b)), (Ok(x), Ok(y)) if x == y),
_ => false,
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::gradle::{ProcessedResources, ReportedSourceSet};
fn write(path: &Path, text: &str) {
fs::create_dir_all(path.parent().unwrap()).unwrap();
fs::write(path, text).unwrap();
}
fn set(name: &str, processed: Option<&Path>) -> ReportedSourceSet {
ReportedSourceSet {
name: name.into(),
test: name == "test",
processed: processed.map(|dir| ProcessedResources {
task: if name == "main" { "processResources".into() } else { "processTestResources".into() },
dir: dir.to_string_lossy().into_owned(),
}),
..Default::default()
}
}
fn project(path: &str, dir: &Path, sets: Vec<ReportedSourceSet>) -> ReportedProject {
ReportedProject {
path: path.into(),
dir: dir.to_string_lossy().into_owned(),
source_sets: sets,
..Default::default()
}
}
/// Only processed source sets whose jdtls output is on the classpath: the launched
/// module, upstream modules of the build, tests only for a test classpath; never buildSrc.
#[test]
fn targets_are_processed_sets_on_the_classpath() {
let tmp = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(tmp.path()).unwrap();
let (app, lib, plain, src) = (root.join("app"), root.join("lib"), root.join("plain"), root.join("buildSrc"));
for dir in [&root, &app, &lib, &plain, &src] {
for set in ["main", "test"] {
fs::create_dir_all(dir.join("bin").join(set)).unwrap();
}
}
let out = |dir: &Path, set: &str| dir.join("build/resources").join(set);
let mut build_src = project(":buildSrc", &src, vec![set("main", Some(&out(&src, "main")))]);
build_src.build = Some(":buildSrc".into());
let reported = [
project(":", &root, vec![set("main", Some(&out(&root, "main")))]),
project(":app", &app, vec![set("main", Some(&out(&app, "main"))), set("test", Some(&out(&app, "test")))]),
project(":lib", &lib, vec![set("main", Some(&out(&lib, "main")))]),
project(":plain", &plain, vec![set("main", None)]),
build_src,
];
let entry = |dir: &Path, set: &str| dir.join("bin").join(set).to_string_lossy().into_owned();
let runtime = [
entry(&app, "main"),
// Spelled differently from Gradle's canonical folders.
format!("{}/", entry(&lib, "main")),
entry(&plain, "main"),
entry(&src, "main"),
"/m2/spring-core.jar".into(),
];
let tasks = |cp: &[String]| targets(&reported, cp).into_iter().map(|t| t.task).collect::<Vec<_>>();
assert_eq!(tasks(&runtime), [":app:processResources", ":lib:processResources"]);
let mut test = runtime.to_vec();
test.insert(0, entry(&app, "test"));
assert_eq!(tasks(&test), [":app:processResources", ":app:processTestResources", ":lib:processResources"]);
assert_eq!(tasks(&[entry(&root, "main")]), [":processResources"], "the root project");
let app_targets = targets(&reported, &runtime);
let t = &app_targets[0];
assert_eq!(
(t.output.as_path(), t.jdtls.as_path()),
(out(&app, "main").as_path(), app.join("bin/main").as_path())
);
let unprocessed = [project(":app", &app, vec![set("main", None)])];
assert!(targets(&unprocessed, &runtime).is_empty());
}
/// The processed copy replaces jdtls's plain one (the reported `@version@` case), new
/// files are added, identical ones are left alone and files Gradle doesn't produce stay.
#[test]
fn overlay_replaces_plain_copies_with_gradle_output() {
let tmp = tempfile::tempdir().unwrap();
let (output, jdtls) = (tmp.path().join("build/resources/main"), tmp.path().join("bin/main"));
write(&output.join("application.yaml"), "spring:\n application:\n version: 0.0.1\n");
write(&output.join("static/extra.txt"), "from an extra source");
write(&output.join("migrations/V1__init.sql"), "create table t (id int);");
write(&jdtls.join("application.yaml"), "spring:\n application:\n version: @version@\n");
write(&jdtls.join("migrations/V1__init.sql"), "create table t (id int);");
write(&jdtls.join("com/example/App.class"), "bytecode");
let unchanged = fs::metadata(jdtls.join("migrations/V1__init.sql")).unwrap().modified().unwrap();
std::thread::sleep(std::time::Duration::from_millis(20));
overlay(&output, &output, &jdtls).unwrap();
let read = |rel: &str| fs::read_to_string(jdtls.join(rel)).unwrap();
assert_eq!(read("application.yaml"), "spring:\n application:\n version: 0.0.1\n");
assert_eq!(read("static/extra.txt"), "from an extra source");
assert_eq!(read("com/example/App.class"), "bytecode");
let modified = fs::metadata(jdtls.join("migrations/V1__init.sql")).unwrap().modified().unwrap();
assert_eq!(modified, unchanged, "identical files are not rewritten");
overlay(&tmp.path().join("missing"), &tmp.path().join("missing"), &jdtls).unwrap();
}
#[test]
fn builds_without_processed_resources_run_nothing() {
let tmp = tempfile::tempdir().unwrap();
// Running this command would fail the call.
let missing = tmp.path().join("missing-gradle").to_string_lossy().into_owned();
let model =
BuildModel { gradle: vec![GradleModel { command: missing, ..Default::default() }], ..Default::default() };
let class_paths = [tmp.path().join("bin/main").to_string_lossy().into_owned()];
let tasks =
process_launch_resources(tmp.path(), &model, tmp.path(), &class_paths, &CancellationToken::new()).unwrap();
assert!(tasks.is_empty());
}
}
+2
View File
@@ -4,11 +4,13 @@
//! for the Build tool window, the Problems view and the editor.
mod fingerprint;
mod launch_resources;
mod parse;
mod single;
mod up_to_date;
pub use fingerprint::{LaunchFingerprint, launch_fingerprint, menu_fingerprint};
pub use launch_resources::process_launch_resources;
pub use single::{
SingleFileTarget, compile_file_plan, find_javac, javac_step, single_file_classpath, single_file_target,
};
+49
View File
@@ -79,6 +79,14 @@ pub fn find_javac(env: &BTreeMap<String, String>) -> String {
/// so `-am` stays cheap). Entries are platform-separated (`;` on Windows).
pub fn single_file_classpath(root: &Path, model: &BuildModel, target: &SingleFileTarget) -> IdeResult<String> {
let mut entries: Vec<String> = Vec::new();
// The module's own classes first: the file almost always uses classes next to it.
let own: &[&str] = if target.test { &["target/test-classes", "target/classes"] } else { &["target/classes"] };
for out in own {
let path = target.module.join(out);
if path.is_dir() {
entries.push(path.to_string_lossy().into_owned());
}
}
if let Some(m) = &model.maven {
// Upstream reactor modules first: their freshly built classes shadow
// installed snapshots of the same artifacts.
@@ -139,6 +147,16 @@ pub fn javac_step(target: &SingleFileTarget, file: &Path, classpath: &str, javac
step.args.push(target.out_dir.to_string_lossy().into_owned());
step.args.push("-cp".into());
step.args.push(classpath.to_owned());
// Classes of the module that were never built (or are older than their source) are read
// from its sources; only `file` is written, so a test compile never puts main classes
// into test-classes.
let sources = source_roots(target);
if !sources.is_empty() {
let separator = if cfg!(windows) { ";" } else { ":" };
step.args.push("-sourcepath".into());
step.args.push(sources.iter().map(|p| p.to_string_lossy()).collect::<Vec<_>>().join(separator));
step.args.push("-implicit:none".into());
}
if let Some(version) = target.java_version {
step.args.push("--release".into());
step.args.push(version.to_string());
@@ -148,6 +166,27 @@ pub fn javac_step(target: &SingleFileTarget, file: &Path, classpath: &str, javac
Ok(step)
}
/// The module's Java source folders that exist, generated ones included; test sources see
/// the main ones too.
fn source_roots(target: &SingleFileTarget) -> Vec<PathBuf> {
let scopes: &[(&str, &str)] = if target.test {
&[("src/test/java", "target/generated-test-sources"), ("src/main/java", "target/generated-sources")]
} else {
&[("src/main/java", "target/generated-sources")]
};
let mut roots = Vec::new();
for (src, generated) in scopes {
roots.push(target.module.join(src));
let generated = target.module.join(generated);
let mut dirs: Vec<PathBuf> =
fs::read_dir(&generated).into_iter().flatten().flatten().map(|d| d.path()).filter(|p| p.is_dir()).collect();
dirs.sort();
roots.extend(dirs);
}
roots.retain(|p| p.is_dir());
roots
}
/// Resolves the classpath and plans the `javac` invocation for one Java source.
pub fn compile_file_plan(root: &Path, model: &BuildModel, file: &Path) -> IdeResult<BuildPlan> {
let target = single_file_target(root, model, file)?;
@@ -243,6 +282,11 @@ mod tests {
assert!(args.contains("-d ") && args.contains("test-classes"), "{args}");
assert!(args.contains("-cp /repo/a.jar"), "{args}");
assert!(args.contains("MainTest.java"), "{args}");
// Test sources resolve their own and the main sources, without writing them.
let sourcepath = step.args.iter().position(|a| a == "-sourcepath").map(|i| step.args[i + 1].clone());
let roots = sourcepath.expect("a -sourcepath");
assert!(roots.contains("src/test/java") && roots.contains("src/main/java"), "{roots}");
assert!(step.args.iter().any(|a| a == "-implicit:none"), "{args}");
// No `JAVA_HOME`: plain `javac` from `PATH`.
assert_eq!(find_javac(&BTreeMap::new()), exe);
}
@@ -275,6 +319,11 @@ mod tests {
// Reactor class directories first, so fresh outputs shadow installed snapshots.
let lib = root.join("lib/target/classes").to_string_lossy().into_owned();
assert!(cp.starts_with(&lib), "{cp}");
// The module's own classes come first once they exist.
fs::create_dir_all(root.join("app/target/classes")).unwrap();
let cp = single_file_classpath(root, &model, &target).unwrap();
let own = root.join("app/target/classes").to_string_lossy().into_owned();
assert!(cp.starts_with(&format!("{own}:{lib}")), "{cp}");
assert!(cp.contains("/repo/a.jar"), "{cp}");
assert!(cp.contains("/repo/b.jar"), "{cp}");
assert!(cp.split(separator).count() >= 3, "{cp}");
+36 -2
View File
@@ -14,6 +14,20 @@ use std::fs;
#[derive(serde::Serialize, serde::Deserialize)]
struct Record {
fingerprint: LaunchFingerprint,
/// Output folders of the built projects that existed after the build, relative to the
/// project. Deleting one outside the IDE (`mvn clean` in a terminal) leaves every input
/// unchanged, so a build is skipped only while they are all still there.
#[serde(default)]
outputs: Vec<PathBuf>,
}
/// The build's output folders that exist now, relative to `root`.
fn existing_outputs(root: &Path, fingerprint: &LaunchFingerprint) -> Vec<PathBuf> {
fingerprint
.project_dirs()
.flat_map(|dir| ["target/classes", "target/test-classes"].map(|out| dir.join(out)))
.filter(|rel| root.join(rel).is_dir())
.collect()
}
fn record_path(cache: &Path, root: &Path, file: Option<&Path>) -> PathBuf {
@@ -69,7 +83,7 @@ pub fn menu_up_to_date(
restamp(root, &record.fingerprint.mtime_preserved_edits(&current));
return false;
}
true
record.outputs.iter().all(|out| root.join(out).is_dir())
}
/// Records a successful menu build as the baseline for future up-to-date
@@ -91,7 +105,8 @@ pub fn record_menu_build(
if path.parent().is_some_and(|p| fs::create_dir_all(p).is_err()) {
return;
}
let bytes = match serde_json::to_vec(&Record { fingerprint: current }) {
let outputs = existing_outputs(root, &current);
let bytes = match serde_json::to_vec(&Record { fingerprint: current, outputs }) {
Ok(bytes) => bytes,
_ => return,
};
@@ -151,6 +166,25 @@ mod tests {
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
}
#[test]
fn deleted_outputs_invalidate_the_record() {
let dir = tempfile::tempdir().unwrap();
let root = dir.path();
let model = fixture(root);
let cache = root.join("cache");
let cancel = CancellationToken::new();
for out in ["app/target/classes/t", "lib/target/classes/t", "app/target/test-classes"] {
fs::create_dir_all(root.join(out)).unwrap();
}
record_menu_build(&cache, root, &model, None, &cancel);
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
// `mvn clean` outside the IDE: no input changed, but the classes are gone.
fs::remove_dir_all(root.join("lib/target")).unwrap();
assert!(!menu_up_to_date(&cache, root, &model, None, &cancel));
fs::create_dir_all(root.join("lib/target/classes")).unwrap();
assert!(menu_up_to_date(&cache, root, &model, None, &cancel));
}
#[test]
fn edit_invalidates_and_rerecord_restores_skip() {
let dir = tempfile::tempdir().unwrap();
+22 -1
View File
@@ -8,6 +8,24 @@ def canon = { File f -> f.canonicalPath }
// the identity path (`:lib:core`, `:buildSrc`) tells them apart and addresses their tasks
// from the root build. Internal API, hence the fallback.
def identity = { Project p -> try { p.identityPath.toString() } catch (ignored) { p.path } }
// The task copying a source set's resources and its output folder, when it changes them on the
// way (filters, `expand`, renames, includes / excludes, extra sources): jdtls copies resources as
// they are, so launches take them from Gradle. Internal API, hence the fallback.
def processedResources = { Project p, s ->
try {
def task = p.tasks.findByName(s.processResourcesTaskName)
if (task == null) return null
def specs = 0
def changes = task.mainSpec.sourcePaths.size() > 1
task.rootSpec.buildRootResolver().walk { r ->
specs++
changes |= !r.allCopyActions.isEmpty() || !r.allIncludes.isEmpty() || !r.allExcludes.isEmpty() ||
!r.allIncludeSpecs.isEmpty() || !r.allExcludeSpecs.isEmpty() || r.destPath.segments.length > 0
}
// The root and main specs exist for every task; more come from `from(…) { … }`.
(changes || specs > 2) ? [task: task.name, dir: canon(task.destinationDir)] : null
} catch (ignored) { null }
}
def describe = { Project p ->
def m = [path: identity(p), dir: canon(p.projectDir), buildDir: canon(p.layout.buildDirectory.get().asFile)]
@@ -26,7 +44,10 @@ def describe = { Project p ->
def test = s.name == 'test' || s.name == 'testFixtures' || suites.contains(s.name) ||
s.name.endsWith('Test') || s.name.endsWith('Tests') ||
(!src.isEmpty() && src.every { ideaTests.contains(it) })
[name: s.name, test: test, src: src, res: res]
def set = [name: s.name, test: test, src: src, res: res]
def processed = processedResources(p, s)
if (processed != null) set.processed = processed
set
}
}
try {
+13
View File
@@ -335,6 +335,19 @@ pub struct ReportedSourceSet {
/// Absolute resource folders.
#[serde(default)]
pub res: Vec<String>,
/// Set when Gradle changes the resources while copying them (filters, `expand`, renames,
/// includes / excludes, extra sources), which the Java language server's copy does not.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub processed: Option<ProcessedResources>,
}
/// The task that copies a source set's resources, and where it writes them.
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
pub struct ProcessedResources {
/// Task name in its project (`processResources`).
pub task: String,
/// Absolute output folder.
pub dir: String,
}
const MODEL_LINE: &str = "@openidea-model\t";
+25 -3
View File
@@ -73,6 +73,9 @@ struct GradleCache {
/// The projects as Gradle evaluated them; empty in caches from before they were read.
#[serde(default)]
projects: Vec<ReportedProject>,
/// The projects report how their resources are processed; older caches are stale.
#[serde(default)]
resources: bool,
}
/// Init script printing every project, its source sets and Java version once configured.
@@ -244,11 +247,12 @@ pub fn apply_gradle_cache(root: &Path, cache_dir: &Path, model: &mut GradleModel
}
}
}
// A successful sync from before projects were cached has not seen computed includes.
let without_projects = cache.error.is_none() && cache.projects.is_empty();
// A successful sync from before projects were cached has not seen computed includes,
// nor, before resources were read, which launches need Gradle to process them.
let outdated = cache.error.is_none() && (cache.projects.is_empty() || !cache.resources);
model.synced_at = Some(cache.at_ms);
model.sync_error = cache.error;
model.stale = without_projects || cache.hash != hash_gradle_files(&gradle::build_files(root, model));
model.stale = outdated || cache.hash != hash_gradle_files(&gradle::build_files(root, model));
}
// ---------------------------------------------------------------- processes
@@ -462,6 +466,7 @@ pub fn sync_gradle_model(
error: (!out.success).then(|| error_summary(&out.text)),
tasks: if out.success { gradle::parse_tasks(&out.text) } else { Vec::new() },
projects,
resources: true,
};
write_cache(&cache_dir, GRADLE_CACHE, &cache)
}
@@ -713,6 +718,7 @@ mod tests {
test,
src: src.iter().map(|p| core_src(p)).collect(),
res: res.iter().map(|p| core_src(p)).collect(),
processed: None,
};
// A settings script's own println comes first; task listing lines follow.
let out = format!(
@@ -737,6 +743,15 @@ mod tests {
);
let projects = parse_reported_projects(&out);
assert_eq!(projects.len(), 3, "the malformed line is skipped");
// As the init script prints a set whose resources Gradle filters or renames.
let processed = parse_reported_projects(&format!(
"@openidea-model\t{}",
r#"{"path":":","dir":"/p","sourceSets":[{"name":"main","test":false,"src":[],"res":[],"processed":{"task":"processResources","dir":"/p/build/resources/main"}}]}"#
));
assert_eq!(
processed[0].source_sets[0].processed,
Some(gradle::ProcessedResources { task: "processResources".into(), dir: "/p/build/resources/main".into() })
);
let src = |p: &str| PathBuf::from(core_src(p));
assert_eq!(
gradle::reported_source_roots(&projects),
@@ -802,12 +817,18 @@ mod tests {
..Default::default()
})
.collect(),
resources: true,
};
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
apply_gradle_cache(&root, &cache_dir, &mut model);
assert_eq!(model.projects[1].tasks.len(), 1);
assert!(model.projects[0].tasks.iter().any(|t| t.name == "wrapper"), "root keeps its defaults");
assert!(!model.stale);
write_cache(&cache_dir, GRADLE_CACHE, &GradleCache { resources: false, ..cache.clone() }).unwrap();
let mut m = gradle::load(&root).unwrap();
apply_gradle_cache(&root, &cache_dir, &mut m);
assert!(m.stale, "synced before resource processing was read");
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
let stale_after = |script: &str| {
std::fs::write(root.join("app/build.gradle"), script).unwrap();
@@ -841,6 +862,7 @@ mod tests {
dir: root.to_string_lossy().into_owned(),
..Default::default()
}],
resources: true,
..Default::default()
};
write_cache(&cache_dir, GRADLE_CACHE, &cache).unwrap();
+20
View File
@@ -333,6 +333,23 @@ pub struct InlayHintsSettings {
pub code_vision_usages: bool,
/// Code vision: last author of a declaration (VCS blame) beside its usages.
pub code_vision_author: bool,
/// Code vision: "no usages" on declarations nothing uses (IDEA shows it; clicking searches everywhere).
pub code_vision_no_usages: bool,
/// Code vision: where the usages go, above the declaration or at the end of its line.
pub code_vision_usages_position: CodeVisionPosition,
/// Code vision: where the author goes, above the declaration or at the end of its line.
pub code_vision_author_position: CodeVisionPosition,
}
/// Editor › Inlay Hints › Code Vision position (IDEA's "Default position for metrics").
#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub enum CodeVisionPosition {
/// A row of its own above the declaration.
#[default]
Top,
/// After the end of the declaration's first line.
Right,
}
impl Default for InlayHintsSettings {
@@ -344,6 +361,9 @@ impl Default for InlayHintsSettings {
disabled_languages: Vec::new(),
code_vision_usages: true,
code_vision_author: true,
code_vision_no_usages: true,
code_vision_usages_position: CodeVisionPosition::Top,
code_vision_author_position: CodeVisionPosition::Top,
}
}
}
+4 -9
View File
@@ -261,6 +261,9 @@ pub enum WhileRunning {
pub struct McpServerSettings {
/// Serve the tools on 127.0.0.1 while the IDE runs.
pub enabled: bool,
/// Run the tools that change the project or start processes without asking first
/// (IDEA's brave mode). Read from the IDE's own settings only, never a project's.
pub run_without_confirmation: bool,
}
/// Absent profiles retain automatic discovery for existing projects.
@@ -580,12 +583,10 @@ pub struct AppearanceSettings {
pub project_colors_in_toolbar: bool,
pub dnd_with_alt_only: bool,
pub menu_icons: bool,
/// `hamburger` | `merged`: main menu presentation in the frameless title bar.
/// `hamburger` | `merged` | `toolbar` (a separate row): main menu presentation in the frameless title bar.
pub main_menu_style: String,
/// A lone Alt press focuses or opens the main menu (IDEA `ide.windows.main.menu.focus.on.alt`).
pub main_menu_alt: bool,
/// Always show the main menu titles instead of the hamburger button (#1521).
pub always_show_main_menu: bool,
/// `subpixel` | `greyscale` | `none`
pub ide_antialiasing: String,
pub editor_antialiasing: String,
@@ -663,7 +664,6 @@ impl Default for AppearanceSettings {
menu_icons: true,
main_menu_style: "hamburger".into(),
main_menu_alt: true,
always_show_main_menu: false,
ide_antialiasing: "greyscale".into(),
editor_antialiasing: "greyscale".into(),
show_tool_window_bars: true,
@@ -1579,11 +1579,6 @@ mod tests {
let off: Settings = serde_json::from_str(r#"{"appearance":{"mainMenuAlt":false}}"#).unwrap();
assert!(!off.appearance.main_menu_alt);
assert_eq!(serde_json::to_value(off).unwrap()["appearance"]["mainMenuAlt"], false);
let old: Settings = serde_json::from_str(r#"{"appearance":{"compactMode":true}}"#).unwrap();
assert!(!old.appearance.always_show_main_menu);
let shown: Settings = serde_json::from_str(r#"{"appearance":{"alwaysShowMainMenu":true}}"#).unwrap();
assert!(shown.appearance.always_show_main_menu);
assert_eq!(serde_json::to_value(shown).unwrap()["appearance"]["alwaysShowMainMenu"], true);
}
#[test]
+1
View File
@@ -14,6 +14,7 @@ ide-jsonrpc.workspace = true
ide-jvm-attach.workspace = true
ide-jvm-inspect.workspace = true
ide-jdwp.workspace = true
ide-heap.workspace = true
serde.workspace = true
serde_json.workspace = true
specta.workspace = true
+12
View File
@@ -342,4 +342,16 @@ mod tests {
assert_eq!(error_message(&msg), "cannot find a.py");
assert_eq!(error_message(&json!({"message": "nope"})), "nope");
}
#[test]
fn a_failed_launch_response_keeps_delves_too_old_error_verbatim() {
// The UI's repair offer keys off this exact wording (`missingAdapter`), so the
// conversion to the IPC error must not fold it into a generic message.
let message = "Failed to launch /p/api: Version of Delve is too old for Go version go1.27.0 \
(maximum supported version 1.26, suppress this error with --check-go-version=false)";
let e = ide_core::IdeError::from(DapError::Failed { command: "launch".into(), message: message.into() });
assert_eq!(e.kind, ide_core::error::ErrorKind::InvalidInput);
assert_eq!(e.message, format!("launch failed: {message}"));
assert!(e.message.contains("Version of Delve is too old for Go version go1.27.0"));
}
}
+293
View File
@@ -0,0 +1,293 @@
//! A heap dump of a debugged JVM, written over its attach listener into a snapshot directory.
use super::snapshot::{AttachProbe, JdkProbe, attach_failure, gone, pick_jvm_pid};
use super::{
JvmMemoryState, JvmMemoryUnavailable, JvmTarget, histogram_pid, keep_found, lookup, resolve_jvm,
safe_while_suspended, session_target,
};
use crate::session::Session;
use ide_core::{CancellationToken, IdeError, IdeResult};
use ide_jvm_attach::{AttachError, HeapDumpReply};
use serde::{Deserialize, Serialize};
use specta::Type;
use std::future::Future;
use std::path::{Path, PathBuf};
use std::sync::Arc;
const DUMP_FILE: &str = "dump.hprof";
/// For a refusal whose reply holds no reason: empty, or only the line announcing the dump.
const NO_REASON: &str = "the JVM wrote no heap dump and gave no reason";
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum JvmHeapUnavailable {
/// The reason a class table could not be read applies to the dump too.
Memory {
reason: JvmMemoryUnavailable,
},
/// Bytes.
NotEnoughDiskSpace {
needed: u64,
available: u64,
},
DumpFailed {
message: String,
},
DumpTimedOut {
seconds: u32,
},
}
#[derive(Clone, Debug, PartialEq)]
pub struct JvmHeapDump {
pub path: PathBuf,
/// The class histogram taken just before the dump.
pub histogram: Vec<ide_heap::HistogramEntry>,
/// Raw `VM.flags`; empty when the JVM did not answer.
pub vm_flags: String,
/// 0 when unknown.
pub java_major: u32,
/// Empty when unknown.
pub java_version: String,
pub pid: u32,
pub taken_while_suspended: bool,
pub dump_ms: u32,
}
#[derive(Clone, Debug, PartialEq)]
pub enum JvmHeapOutcome {
Ready(JvmHeapDump),
/// The snapshot directory has been deleted.
Unavailable(JvmHeapUnavailable),
/// The snapshot directory has been deleted.
Cancelled,
}
pub(crate) async fn session_heap_dump(
session: Arc<Session>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let launched = histogram_pid(&session);
let capture = {
let (dir, cancel) = (dir.clone(), cancel.clone());
async move {
let target = session_target(&session, launched);
let probe = |suspended| -> Arc<dyn AttachProbe> { Arc::new(JdkProbe { suspended }) };
let stopped = session.clone();
let suspended = move || capture_suspended(&stopped);
heap_dump_with(&session.jvm_memory, suspended, probe, target, dir, cancel).await
}
};
detached(capture, dir, cancel).await
}
/// The JVM can be suspended before the adapter reports the stop, so a session with a JDWP link
/// also counts the link's view.
pub(super) fn capture_suspended(session: &Session) -> bool {
session.suspended.get()
|| session
.jdwp_link
.lock()
.as_ref()
.is_some_and(|link| matches!(link.state(), ide_jdwp::LinkState::Suspended { .. }))
}
/// Runs `capture` in a task of its own, so a caller that stops waiting cannot release the
/// session's lock while the JVM is still writing the dump and let a class-table request run
/// next to it. On `cancel` returns at once and deletes `dir`.
pub(super) async fn detached(
capture: impl Future<Output = IdeResult<JvmHeapOutcome>> + Send + 'static,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let task = tokio::spawn(capture);
tokio::select! {
joined = task => joined.map_err(|e| IdeError::internal(format!("JVM heap dump failed: {e}")))?,
() = cancel.cancelled() => {
discard_later(dir).await;
Ok(JvmHeapOutcome::Cancelled)
}
}
}
/// Holds `state`'s lock throughout, as a class-table request does; `suspended` and then `probe`
/// are read once it is held, and `suspended` again by [`take_heap_dump`]. Anything but `Ready`
/// deletes `dir`.
pub(super) async fn heap_dump_with(
state: &tokio::sync::Mutex<JvmMemoryState>,
suspended: impl Fn() -> bool + Send + Sync + 'static,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
// The capture's histogram never becomes the Memory view's baseline, so the state is unused.
let _turn = state.lock().await;
let outcome = capture(suspended, probe, target, dir.clone(), cancel).await;
if !matches!(outcome, Ok(JvmHeapOutcome::Ready(_))) {
discard_later(dir).await;
}
outcome
}
async fn capture(
suspended_now: impl Fn() -> bool + Send + Sync + 'static,
probe: impl FnOnce(bool) -> Arc<dyn AttachProbe>,
target: JvmTarget<'_>,
dir: PathBuf,
cancel: CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
let suspended = suspended_now();
let lookup = match lookup(target) {
Ok(lookup) => lookup,
Err(reason) => return Ok(memory(reason)),
};
let probe = probe(suspended);
let (found, outcome) = tokio::task::spawn_blocking(move || match resolve_jvm(lookup, &*probe, suspended) {
Ok((tree, found)) => (found, take_heap_dump(&*probe, &tree, suspended, &suspended_now, &dir, &cancel)),
Err(reason) => (None, Ok(memory(reason))),
})
.await
.map_err(|e| IdeError::internal(format!("JVM heap dump failed: {e}")))?;
keep_found(target, found);
outcome
}
/// `tree` starts with the session's reported pid, followed by its descendants. `suspended` is
/// the stop read when the capture began; `suspended_now` reads it again. Anything but `Ready`
/// deletes `dir`.
pub(crate) fn take_heap_dump(
probe: &dyn AttachProbe,
tree: &[i32],
suspended: bool,
suspended_now: &dyn Fn() -> bool,
dir: &Path,
cancel: &CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let outcome = dump(probe, tree, suspended, suspended_now, dir, cancel);
if !matches!(outcome, Ok(JvmHeapOutcome::Ready(_))) {
discard(dir);
}
outcome
}
fn dump(
probe: &dyn AttachProbe,
tree: &[i32],
suspended: bool,
suspended_now: &dyn Fn() -> bool,
dir: &Path,
cancel: &CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
use JvmHeapUnavailable as H;
use JvmMemoryUnavailable as U;
let Some(root) = tree.first().and_then(|&p| u32::try_from(p).ok()) else {
return Ok(memory(U::NoJvmProcess));
};
let Some(jvm) = pick_jvm_pid(tree, |p| probe.has_listener(p)) else {
return Ok(memory(gone(probe, root)));
};
let version = probe.version(jvm);
let java_major = version.as_ref().map_or(0, |v| v.major);
let safe = safe_while_suspended(version.as_ref());
let unsafe_stop = || Ok(memory(U::UnsafeWhileSuspended { java_major }));
if suspended && !safe {
return unsafe_stop();
}
// The program may stop during the histogram, and a stopped JDK below 17 hangs on the next
// request until it resumes.
let stopped_since = || !safe && suspended_now();
let text = match probe.histogram(jvm) {
Ok(text) => text,
Err(e) => return attach_failure(probe, jvm, e, "class histogram").map(memory),
};
let histogram: Vec<ide_heap::HistogramEntry> = ide_jvm_attach::parse_class_histogram(&text)
.rows
.into_iter()
.map(|r| ide_heap::HistogramEntry { name: r.name, instances: r.instances, bytes: r.bytes })
.collect();
let (instances, bytes) =
histogram.iter().fold((0u64, 0u64), |(i, b), e| (i.saturating_add(e.instances), b.saturating_add(e.bytes)));
let needed = ide_heap::capture_need(bytes, instances);
if let Some(available) = probe.free_bytes(dir)
&& available < needed
{
return Ok(JvmHeapOutcome::Unavailable(H::NotEnoughDiskSpace { needed, available }));
}
if stopped_since() {
return unsafe_stop();
}
// Without flags the index assumes the default object layout.
let vm_flags = probe.vm_flags(jvm).unwrap_or_else(|e| {
tracing::debug!("VM.flags of JVM {jvm}: {e}");
String::new()
});
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
if stopped_since() {
return unsafe_stop();
}
let path = dir.join(DUMP_FILE);
let timeout = ide_jvm_attach::dump_timeout(bytes);
let started = std::time::Instant::now();
let reply = probe.heap_dump(jvm, &path, timeout);
let dump_ms = u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX);
if cancel.is_cancelled() {
return Ok(JvmHeapOutcome::Cancelled);
}
let failed = |message: String| Ok(JvmHeapOutcome::Unavailable(H::DumpFailed { message }));
match reply {
Ok(HeapDumpReply::Created { .. }) => {
if !std::fs::metadata(&path).is_ok_and(|m| m.is_file() && m.len() > 0) {
return failed(format!("the JVM reported a heap dump, but {} is missing or empty", path.display()));
}
Ok(JvmHeapOutcome::Ready(JvmHeapDump {
path,
histogram,
vm_flags,
java_major,
java_version: version.map(|v| v.text).unwrap_or_default(),
pid: jvm,
taken_while_suspended: suspended,
dump_ms,
}))
}
Ok(HeapDumpReply::Refused { message }) if message.is_empty() || message.starts_with("Dumping heap to") => {
failed(NO_REASON.into())
}
Ok(HeapDumpReply::Refused { message }) => failed(message),
Err(AttachError::Timeout) => {
let seconds = u32::try_from(timeout.as_secs()).unwrap_or(u32::MAX);
Ok(JvmHeapOutcome::Unavailable(H::DumpTimedOut { seconds }))
}
// A running JDK rejects an argument with a non-zero result code.
Err(AttachError::Command { message, .. }) => failed(message),
Err(e) => attach_failure(probe, jvm, e, "heap dump").map(memory),
}
}
fn memory(reason: JvmMemoryUnavailable) -> JvmHeapOutcome {
JvmHeapOutcome::Unavailable(JvmHeapUnavailable::Memory { reason })
}
fn discard(dir: &Path) {
if let Err(e) = std::fs::remove_dir_all(dir)
&& e.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not delete heap snapshot directory {}: {e}", dir.display());
}
}
/// A dump may be large, so it is deleted off the async threads.
async fn discard_later(dir: PathBuf) {
let _ = tokio::task::spawn_blocking(move || discard(&dir)).await;
}
+2
View File
@@ -2,6 +2,7 @@
//! that is unavailable, counted by the debugger; and the JVM's objects through the debugger.
mod fallback;
mod heap;
mod inspect;
mod scan;
mod snapshot;
@@ -20,6 +21,7 @@ use std::sync::Arc;
use std::sync::atomic::{AtomicI64, Ordering};
use std::time::Duration;
pub use heap::*;
pub use inspect::{JvmFieldsResult, JvmInspectUnavailable, JvmInstancesResult, JvmObjectRef, JvmReferrersResult};
pub(crate) use inspect::{fields, instances, referrers, set_inspector, watch_link};
#[cfg(all(test, unix))]
+20 -2
View File
@@ -6,14 +6,16 @@ use super::{
safe_while_suspended,
};
use ide_core::IdeResult;
use ide_jvm_attach::{AttachError, HistogramRow, JavaVersion};
use ide_jvm_attach::{AttachError, HeapDumpReply, HistogramRow, JavaVersion};
use std::collections::HashMap;
use std::path::Path;
use std::time::Duration;
/// Version probe for a JVM whose `release` file could not be read.
const VERSION_TIMEOUT: Duration = Duration::from_secs(2);
const FLAGS_TIMEOUT: Duration = Duration::from_secs(2);
/// The attach calls a snapshot makes, so tests can stand in for a JVM.
/// The attach calls a snapshot or a heap dump makes, so tests can stand in for a JVM.
pub(crate) trait AttachProbe: Send + Sync {
fn has_listener(&self, pid: u32) -> bool;
/// Asks only the JVMs whose pid `may_query` accepts.
@@ -21,6 +23,10 @@ pub(crate) trait AttachProbe: Send + Sync {
fn version(&self, pid: u32) -> Option<JavaVersion>;
fn histogram(&self, pid: u32) -> Result<String, AttachError>;
fn alive(&self, pid: u32) -> bool;
fn vm_flags(&self, pid: u32) -> Result<String, AttachError>;
fn heap_dump(&self, pid: u32, path: &Path, timeout: Duration) -> Result<HeapDumpReply, AttachError>;
/// `None` where the free space is not known.
fn free_bytes(&self, dir: &Path) -> Option<u64>;
}
pub(super) struct JdkProbe {
@@ -55,6 +61,18 @@ impl AttachProbe for JdkProbe {
fn alive(&self, pid: u32) -> bool {
alive(pid)
}
fn vm_flags(&self, pid: u32) -> Result<String, AttachError> {
ide_jvm_attach::vm_flags(pid, FLAGS_TIMEOUT)
}
fn heap_dump(&self, pid: u32, path: &Path, timeout: Duration) -> Result<HeapDumpReply, AttachError> {
ide_jvm_attach::heap_dump(pid, path, timeout)
}
fn free_bytes(&self, dir: &Path) -> Option<u64> {
ide_heap::free_bytes(dir)
}
}
fn alive(pid: u32) -> bool {
+493 -4
View File
@@ -1,10 +1,13 @@
use super::fallback::{debugger_snapshot, falls_back};
use super::heap::*;
use super::scan::*;
use super::snapshot::*;
use super::test_support::{BareSession, FakeInspector, bare_session, bare_session_of};
use super::*;
use ide_jvm_attach::{AttachError, HistogramRow, JavaVersion};
use ide_core::CancellationToken;
use ide_jvm_attach::{AttachError, HeapDumpReply, HistogramRow, JavaVersion};
use std::collections::VecDeque;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
fn row(name: &str, instances: u64, bytes: u64) -> HistogramRow {
@@ -25,8 +28,52 @@ struct FakeProbe {
asked: parking_lot::Mutex<Vec<u32>>,
/// Pids the scan sent a request to.
queried: parking_lot::Mutex<Vec<u32>>,
/// Attach commands sent, in order.
/// Attach commands sent, in order: `histogram`, `vm_flags`, `heap_dump`.
commands: parking_lot::Mutex<Vec<&'static str>>,
dumps: parking_lot::Mutex<VecDeque<Result<HeapDumpReply, AttachError>>>,
dump_paths: parking_lot::Mutex<Vec<PathBuf>>,
/// Each heap dump waits for this gate to open.
dump_gate: Option<Arc<Gate>>,
/// A `Created` reply writes this many bytes to the dump path first.
dump_bytes: usize,
free: Option<u64>,
/// Answering this command stops the program: runs the hook.
stops_on: Option<(&'static str, StopHook)>,
}
type StopHook = Box<dyn Fn() + Send + Sync>;
/// Holds a blocking call until the test opens it.
#[derive(Default)]
struct Gate {
entered: tokio::sync::Notify,
open: parking_lot::Mutex<bool>,
opened: parking_lot::Condvar,
}
impl Gate {
fn pass(&self) {
self.entered.notify_one();
let mut open = self.open.lock();
while !*open {
self.opened.wait(&mut open);
}
}
fn release(&self) {
*self.open.lock() = true;
self.opened.notify_all();
}
}
/// Opens the gate when dropped: a failed assertion must not leave a blocking thread waiting,
/// since the runtime's shutdown waits for it.
struct OpenOnDrop(Arc<Gate>);
impl Drop for OpenOnDrop {
fn drop(&mut self) {
self.0.release();
}
}
impl FakeProbe {
@@ -44,11 +91,26 @@ impl FakeProbe {
asked: parking_lot::Mutex::new(Vec::new()),
queried: parking_lot::Mutex::new(Vec::new()),
commands: parking_lot::Mutex::new(Vec::new()),
dumps: parking_lot::Mutex::new(VecDeque::new()),
dump_paths: parking_lot::Mutex::new(Vec::new()),
dump_gate: None,
dump_bytes: 64,
free: None,
stops_on: None,
}
}
fn record(&self, command: &'static str) {
self.commands.lock().push(command);
if let Some((on, stop)) = &self.stops_on
&& *on == command
{
stop();
}
}
fn enter(&self, command: &'static str) {
self.commands.lock().push(command);
self.record(command);
let now = self.in_flight.fetch_add(1, Ordering::SeqCst) + 1;
self.max_in_flight.fetch_max(now, Ordering::SeqCst);
}
@@ -84,8 +146,31 @@ impl AttachProbe for FakeProbe {
fn alive(&self, _pid: u32) -> bool {
self.alive
}
fn vm_flags(&self, _pid: u32) -> Result<String, AttachError> {
self.record("vm_flags");
Ok(FLAGS.into())
}
fn heap_dump(&self, _pid: u32, path: &Path, _timeout: Duration) -> Result<HeapDumpReply, AttachError> {
self.enter("heap_dump");
self.dump_paths.lock().push(path.to_owned());
if let Some(gate) = &self.dump_gate {
gate.pass();
}
let reply = self.dumps.lock().pop_front().expect("unexpected heap dump request");
if matches!(reply, Ok(HeapDumpReply::Created { .. })) && self.dump_bytes > 0 {
// Fails like the JVM's write once the directory is gone.
let _ = std::fs::write(path, vec![0; self.dump_bytes]);
}
self.leave();
reply
}
fn free_bytes(&self, _dir: &Path) -> Option<u64> {
self.free
}
}
const FLAGS: &str = "-XX:+UseCompressedOops -XX:+StartAttachListener";
const HISTOGRAM: &str = " num #instances #bytes class name (module)\n-------------------------------------------------------\n 1: 5 80 A\n 2: 1 16 B\nTotal 6 96\n";
fn unavailable(r: IdeResult<Result<(JvmMemorySnapshot, ClassCounts), JvmMemoryUnavailable>>) -> JvmMemoryUnavailable {
@@ -691,12 +776,416 @@ fn debugger_rows_are_ordered_by_instances() {
assert_eq!(names, ["A", "C", "B"]);
}
/// A snapshot directory inside a temporary one, so the capture may delete it.
fn snapshot_dir(name: &str) -> (tempfile::TempDir, PathBuf) {
let root = tempfile::tempdir().unwrap();
let dir = root.path().join(name);
std::fs::create_dir(&dir).unwrap();
(root, dir)
}
/// JVM 12 at Java `major`, answering [`HISTOGRAM`] once and `dumps` in turn.
fn dumping(major: u32, dumps: Vec<Result<HeapDumpReply, AttachError>>) -> FakeProbe {
let mut probe = FakeProbe::new(12, vec![Ok(HISTOGRAM.into())]);
probe.version = Some(JavaVersion { major, text: format!("{major}.0.1") });
probe.dumps = parking_lot::Mutex::new(dumps.into());
probe
}
fn created() -> Result<HeapDumpReply, AttachError> {
Ok(HeapDumpReply::Created { bytes: Some(64) })
}
fn memory(reason: JvmMemoryUnavailable) -> JvmHeapOutcome {
JvmHeapOutcome::Unavailable(JvmHeapUnavailable::Memory { reason })
}
fn dump_failed(message: &str) -> JvmHeapOutcome {
JvmHeapOutcome::Unavailable(JvmHeapUnavailable::DumpFailed { message: message.into() })
}
fn ready(outcome: IdeResult<JvmHeapOutcome>) -> JvmHeapDump {
match outcome.unwrap() {
JvmHeapOutcome::Ready(dump) => dump,
other => panic!("{other:?}"),
}
}
fn entry(name: &str, instances: u64, bytes: u64) -> ide_heap::HistogramEntry {
ide_heap::HistogramEntry { name: name.into(), instances, bytes }
}
#[test]
fn suspended_jdk11_refuses_without_sending() {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(11, vec![created()]);
let outcome = take_heap_dump(&probe, &[12], true, &|| true, &dir, &CancellationToken::new()).unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::UnsafeWhileSuspended { java_major: 11 }));
assert!(probe.commands.lock().is_empty());
assert!(!dir.exists());
}
#[test]
fn suspended_jdk17_dumps() {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(17, vec![created()]);
let dump = ready(take_heap_dump(&probe, &[10, 12], true, &|| true, &dir, &CancellationToken::new()));
assert_eq!(*probe.commands.lock(), ["histogram", "vm_flags", "heap_dump"]);
assert_eq!(*probe.dump_paths.lock(), [dir.join("dump.hprof")]);
let expected = JvmHeapDump {
path: dir.join("dump.hprof"),
histogram: vec![entry("A", 5, 80), entry("B", 1, 16)],
vm_flags: FLAGS.into(),
java_major: 17,
java_version: "17.0.1".into(),
pid: 12,
taken_while_suspended: true,
dump_ms: dump.dump_ms,
};
assert_eq!(dump, expected);
assert!(dump.path.is_file());
}
#[test]
fn running_jdk8_dumps() {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(8, vec![created()]);
let dump = ready(take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()));
assert_eq!((dump.java_major, dump.taken_while_suspended), (8, false));
assert_eq!(*probe.commands.lock(), ["histogram", "vm_flags", "heap_dump"]);
}
#[test]
fn not_enough_disk_space_refuses_before_dump() {
let (_root, dir) = snapshot_dir("snap");
let mut probe = dumping(21, vec![created()]);
let gib = 1u64 << 30;
let histogram = format!(" 1: 10000000 {gib} [B\nTotal 10000000 {gib}\n");
probe.histograms = parking_lot::Mutex::new(vec![Ok(histogram)].into());
probe.free = Some(512 << 20);
let outcome = take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap();
let needed = gib + 1_060_000_000 + gib;
assert_eq!(
outcome,
JvmHeapOutcome::Unavailable(JvmHeapUnavailable::NotEnoughDiskSpace { needed, available: 512 << 20 })
);
assert_eq!(*probe.commands.lock(), ["histogram"]);
assert!(!dir.exists());
}
#[test]
fn refused_dump_reports_message() {
let (_root, dir) = snapshot_dir("snap");
let message = "Unable to create /tmp/snap/dump.hprof: File exists";
let probe = dumping(21, vec![Ok(HeapDumpReply::Refused { message: message.into() })]);
let outcome = take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap();
assert_eq!(outcome, dump_failed(message));
assert!(!dir.exists());
}
#[test]
fn refused_dump_without_a_reason_says_so() {
for body in ["", "Dumping heap to /tmp/snap/dump.hprof ..."] {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(21, vec![Ok(HeapDumpReply::Refused { message: body.into() })]);
let outcome = take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap();
let JvmHeapOutcome::Unavailable(JvmHeapUnavailable::DumpFailed { message }) = outcome else {
panic!("{body:?}: {outcome:?}")
};
assert!(!message.is_empty() && !message.starts_with("Dumping heap to"), "{body:?}: {message:?}");
assert!(!dir.exists());
}
}
#[test]
fn timed_out_dump_removes_snapshot_dir() {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(21, vec![Err(AttachError::Timeout)]);
let outcome = take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap();
assert_eq!(outcome, JvmHeapOutcome::Unavailable(JvmHeapUnavailable::DumpTimedOut { seconds: 30 }));
assert!(!dir.exists());
}
#[test]
fn capture_dir_with_space_is_quoted() {
let (_root, dir) = snapshot_dir("a b");
let probe = dumping(21, vec![created()]);
ready(take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()));
let path = probe.dump_paths.lock()[0].clone();
assert!(path.to_str().unwrap().contains("a b"), "{path:?}");
let command = ide_jvm_attach::heap_dump_command(&path).unwrap();
assert_eq!(command, format!("GC.heap_dump -all \"{}\"", path.display()));
}
#[test]
fn created_without_a_file_fails() {
let (_root, dir) = snapshot_dir("snap");
let mut probe = dumping(21, vec![created()]);
probe.dump_bytes = 0;
let outcome = take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap();
assert!(matches!(outcome, JvmHeapOutcome::Unavailable(JvmHeapUnavailable::DumpFailed { .. })), "{outcome:?}");
assert!(!dir.exists());
}
#[test]
fn dump_errors_map_like_the_class_table() {
let message = "java.lang.IllegalArgumentException: Unknown argument";
let cases = [
(Err(AttachError::Refused), false, memory(JvmMemoryUnavailable::ProgramExited)),
(Err(AttachError::NoListener), true, memory(JvmMemoryUnavailable::ListenerNotRunning)),
(Err(AttachError::Command { code: -1, message: message.into() }), true, dump_failed(message)),
];
for (reply, alive, expected) in cases {
let (_root, dir) = snapshot_dir("snap");
let mut probe = dumping(21, vec![reply]);
probe.alive = alive;
assert_eq!(take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).unwrap(), expected);
assert!(!dir.exists());
}
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(21, vec![Err(AttachError::Protocol("bad".into()))]);
assert!(take_heap_dump(&probe, &[12], false, &|| false, &dir, &CancellationToken::new()).is_err());
assert!(!dir.exists());
}
/// JDK `major`, running when the capture starts, stopped by the time it answers `command`.
fn stopping_after(major: u32, command: &'static str) -> (FakeProbe, Arc<AtomicBool>) {
let stopped = Arc::new(AtomicBool::new(false));
let mut probe = dumping(major, vec![created()]);
let flag = stopped.clone();
probe.stops_on = Some((command, Box::new(move || flag.store(true, Ordering::SeqCst))));
(probe, stopped)
}
#[test]
fn jdk11_stopped_during_the_histogram_gets_no_further_request() {
for (command, sent) in [("histogram", &["histogram"][..]), ("vm_flags", &["histogram", "vm_flags"][..])] {
let (_root, dir) = snapshot_dir("snap");
let (probe, stopped) = stopping_after(11, command);
let suspended_now = || stopped.load(Ordering::SeqCst);
let outcome = take_heap_dump(&probe, &[12], false, &suspended_now, &dir, &CancellationToken::new()).unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::UnsafeWhileSuspended { java_major: 11 }), "{command}");
assert_eq!(*probe.commands.lock(), sent, "{command}");
assert!(!dir.exists(), "{command}");
}
}
#[tokio::test]
async fn session_stopped_during_the_histogram_is_read_again_before_the_dump() {
let bare = bare_session("attach");
let mut probe = dumping(11, vec![created()]);
let stopping = bare.session.clone();
probe.stops_on = Some(("histogram", Box::new(move || stopping.suspended.stopped(Some(1), true))));
let fake = Arc::new(probe);
let probe: Arc<dyn AttachProbe> = fake.clone();
let pid = AtomicI64::new(12);
let target = JvmTarget::Attached { pid: &pid, jdwp_port: None };
let (_root, dir) = snapshot_dir("snap");
let session = bare.session.clone();
let suspended = move || capture_suspended(&session);
let s = &bare.session;
let outcome = heap_dump_with(&s.jvm_memory, suspended, |_| probe, target, dir.clone(), CancellationToken::new())
.await
.unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::UnsafeWhileSuspended { java_major: 11 }));
assert_eq!(*fake.commands.lock(), ["histogram"]);
assert!(!dir.exists());
assert!(s.jvm_memory.try_lock().is_ok(), "the lock is released");
}
#[test]
fn jdk17_stopped_during_the_histogram_still_dumps() {
let (_root, dir) = snapshot_dir("snap");
let (probe, stopped) = stopping_after(17, "histogram");
let suspended_now = || stopped.load(Ordering::SeqCst);
let dump = ready(take_heap_dump(&probe, &[12], false, &suspended_now, &dir, &CancellationToken::new()));
assert!(!dump.taken_while_suspended, "the stop read when the capture began");
assert_eq!(*probe.commands.lock(), ["histogram", "vm_flags", "heap_dump"]);
}
#[test]
fn cancelled_capture_sends_no_dump() {
let (_root, dir) = snapshot_dir("snap");
let probe = dumping(21, vec![created()]);
let cancel = CancellationToken::new();
cancel.cancel();
assert_eq!(take_heap_dump(&probe, &[12], false, &|| false, &dir, &cancel).unwrap(), JvmHeapOutcome::Cancelled);
assert!(!probe.commands.lock().contains(&"heap_dump"));
assert!(!dir.exists());
}
#[tokio::test]
async fn cancelled_capture_releases_lock() {
let state = Arc::new(tokio::sync::Mutex::new(JvmMemoryState::default()));
let gate = Arc::new(Gate::default());
let opener = OpenOnDrop(gate.clone());
let mut fake = dumping(21, vec![created()]);
fake.histograms.lock().push_back(Ok(HISTOGRAM.into()));
fake.dump_gate = Some(gate.clone());
let fake = Arc::new(fake);
let jvm_pid = Arc::new(AtomicI64::new(12));
let (_root, dir) = snapshot_dir("snap");
let cancel = CancellationToken::new();
let capture = {
let (state, fake, jvm_pid, dir, cancel) =
(state.clone(), fake.clone(), jvm_pid.clone(), dir.clone(), cancel.clone());
async move {
let target = JvmTarget::Attached { pid: &jvm_pid, jdwp_port: None };
heap_dump_with(&state, || false, |_| -> Arc<dyn AttachProbe> { fake }, target, dir, cancel).await
}
};
let caller = tokio::spawn(detached(capture, dir.clone(), cancel.clone()));
gate.entered.notified().await;
cancel.cancel();
assert_eq!(caller.await.unwrap().unwrap(), JvmHeapOutcome::Cancelled);
assert_eq!(fake.in_flight.load(Ordering::SeqCst), 1, "the JVM is still writing the dump");
assert!(!dir.exists());
let mut table = tokio::spawn({
let (state, fake, jvm_pid) = (state.clone(), fake.clone(), jvm_pid.clone());
async move {
let target = JvmTarget::Attached { pid: &jvm_pid, jdwp_port: None };
let probe: Arc<dyn AttachProbe> = fake;
snapshot(&state, &Suspension::default(), |_| probe, target, None).await
}
});
// While the gate is shut the dump holds the session's turn, so the class table cannot finish.
let early = tokio::time::timeout(Duration::from_millis(100), &mut table).await;
assert!(early.is_err(), "the class table ran beside the dump: {early:?}");
drop(opener);
let table = table.await.unwrap().unwrap();
assert!(matches!(table, JvmMemoryResult::Ready { .. }), "{table:?}");
assert_eq!(fake.max_in_flight.load(Ordering::SeqCst), 1);
assert_eq!(*fake.commands.lock(), ["histogram", "vm_flags", "heap_dump", "histogram"]);
assert!(!dir.exists(), "the dump that finished after the cancel is deleted");
}
#[tokio::test]
async fn capture_keeps_the_memory_baseline() {
let state = tokio::sync::Mutex::new(JvmMemoryState::default());
let running = Suspension::default();
let pid = AtomicI64::new(12);
let target = || JvmTarget::Attached { pid: &pid, jdwp_port: None };
let table = |fake: FakeProbe| {
let probe: Arc<dyn AttachProbe> = Arc::new(fake);
snapshot(&state, &running, |_| probe, target(), None)
};
table(FakeProbe::new(12, vec![Ok(HISTOGRAM.into())])).await.unwrap();
let mut capture = dumping(21, vec![created()]);
let grown = " 1: 50 800 A\nTotal 50 800\n";
capture.histograms = parking_lot::Mutex::new(vec![Ok(grown.into())].into());
let probe: Arc<dyn AttachProbe> = Arc::new(capture);
let (_root, dir) = snapshot_dir("snap");
ready(heap_dump_with(&state, || false, |_| probe, target(), dir, CancellationToken::new()).await);
let JvmMemoryResult::Ready { snapshot } = table(FakeProbe::new(12, vec![Ok(HISTOGRAM.into())])).await.unwrap()
else {
panic!("no class table")
};
let a = snapshot.rows.iter().find(|r| r.name == "A").unwrap();
assert_eq!((a.delta_instances, a.delta_bytes), (Some(0), Some(0)));
}
#[tokio::test]
async fn attached_session_dumps_the_scanned_jvm() {
let mut fake = dumping(21, vec![created()]);
fake.listener = 4242;
fake.jdwp = Some((5005, 4242));
let fake = Arc::new(fake);
let probe: Arc<dyn AttachProbe> = fake.clone();
let state = tokio::sync::Mutex::new(JvmMemoryState::default());
let pid = AtomicI64::new(0);
let target = JvmTarget::Attached { pid: &pid, jdwp_port: Some(5005) };
let (_root, dir) = snapshot_dir("snap");
let dump = ready(heap_dump_with(&state, || false, |_| probe, target, dir, CancellationToken::new()).await);
assert_eq!(dump.pid, 4242);
assert_eq!(pid.load(Ordering::SeqCst), 4242);
assert_eq!(fake.scans.load(Ordering::SeqCst), 1);
}
/// A JDWP link whose fake JVM sent a suspending event, which the adapter would report next:
/// (link, adapter end, JVM end).
async fn suspended_link() -> (ide_jdwp::JdwpLink, tokio::net::TcpStream, tokio::net::TcpStream) {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
const HANDSHAKE: &[u8; 14] = b"JDWP-Handshake";
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let options = ide_jdwp::LinkOptions { hold_resumes: false };
let (link, link_port) = ide_jdwp::JdwpLink::start("127.0.0.1".into(), port, options).await.unwrap();
let mut adapter = tokio::net::TcpStream::connect(("127.0.0.1", link_port)).await.unwrap();
adapter.write_all(HANDSHAKE).await.unwrap();
let (mut jvm, _) = tokio::time::timeout(Duration::from_secs(2), listener.accept()).await.unwrap().unwrap();
let mut buf = [0; 14];
jvm.read_exact(&mut buf).await.unwrap();
jvm.write_all(HANDSHAKE).await.unwrap();
adapter.read_exact(&mut buf).await.unwrap();
// Event.Composite (64/100): SUSPEND_ALL, one VM_START event, request 0, thread 1.
let body = [2, 0, 0, 0, 1, 90, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1];
let mut packet = u32::try_from(11 + body.len()).unwrap().to_be_bytes().to_vec();
packet.extend(1u32.to_be_bytes());
packet.extend([0, 64, 100]);
packet.extend(body);
jvm.write_all(&packet).await.unwrap();
let mut states = link.subscribe();
let suspended = states.wait_for(|s| matches!(s, ide_jdwp::LinkState::Suspended { .. }));
tokio::time::timeout(Duration::from_secs(2), suspended).await.unwrap().unwrap();
(link, adapter, jvm)
}
#[tokio::test]
async fn jdk11_is_refused_while_the_link_is_suspended_before_dap_stops() {
let bare = bare_session_of(crate::kotlin::ADAPTER, "attach");
let s = &bare.session;
let (link, _adapter, _jvm) = suspended_link().await;
*s.jdwp_link.lock() = Some(link);
assert!(!s.suspended.get());
let fake = Arc::new(jdk11(12));
let probe: Arc<dyn AttachProbe> = fake.clone();
let probe = |suspended: bool| {
assert!(suspended, "the probe must not ask a suspended JVM's version over attach");
probe
};
let pid = AtomicI64::new(12);
let target = JvmTarget::Attached { pid: &pid, jdwp_port: None };
let (_root, dir) = snapshot_dir("snap");
let session = bare.session.clone();
let suspended = move || capture_suspended(&session);
let outcome =
heap_dump_with(&s.jvm_memory, suspended, probe, target, dir.clone(), CancellationToken::new()).await.unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::UnsafeWhileSuspended { java_major: 11 }));
assert!(fake.commands.lock().is_empty());
assert!(!dir.exists());
}
#[tokio::test]
async fn capability_matches_memory() {
for adapter in JVM_MEMORY_ADAPTERS.iter().copied().chain(["debugpy"]) {
let bare = bare_session_of(adapter, "launch");
let caps = crate::session_capabilities(&bare.session);
let jvm = adapter != "debugpy";
assert_eq!(caps.jvm_memory, jvm, "{adapter}");
assert_eq!((caps.jvm_heap, caps.jvm_memory), (jvm, jvm), "{adapter}");
}
}
#[cfg(unix)]
#[tokio::test]
async fn session_without_a_jvm_reports_no_process() {
let bare = bare_session("launch");
let (_root, dir) = snapshot_dir("snap");
let outcome = session_heap_dump(bare.session.clone(), dir.clone(), CancellationToken::new()).await.unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::NoJvmProcess));
assert!(!dir.exists());
}
#[cfg(not(unix))]
#[tokio::test]
async fn capture_is_unsupported_off_unix() {
let bare = bare_session("launch");
let (_root, dir) = snapshot_dir("snap");
let outcome = session_heap_dump(bare.session.clone(), dir.clone(), CancellationToken::new()).await.unwrap();
assert_eq!(outcome, memory(JvmMemoryUnavailable::UnsupportedPlatform));
assert!(!dir.exists());
}
+18
View File
@@ -133,6 +133,8 @@ pub struct DebugCapabilities {
/// The session has a JVM inspector: instances, fields and referrers while suspended. Can
/// change during the session ([`ide_core::event::DebugEvent::Capabilities`]).
pub jvm_instances: bool,
/// The session can be asked for a heap dump ([`DebugManager::jvm_heap_dump`]).
pub jvm_heap: bool,
}
/// Result of [`DebugManager::redefine_classes`].
@@ -576,6 +578,7 @@ pub(crate) fn capabilities_from(caps: &Value) -> DebugCapabilities {
hot_code_replace: false,
jvm_memory: false,
jvm_instances: false,
jvm_heap: false,
}
}
@@ -584,6 +587,7 @@ pub(crate) fn session_capabilities(session: &Session) -> DebugCapabilities {
caps.hot_code_replace = HOT_CODE_REPLACE_ADAPTERS.contains(&session.adapter.as_str());
caps.jvm_memory = JVM_MEMORY_ADAPTERS.contains(&session.adapter.as_str());
caps.jvm_instances = session.jvm_inspector.lock().is_some();
caps.jvm_heap = JVM_MEMORY_ADAPTERS.contains(&session.adapter.as_str());
caps
}
@@ -1714,6 +1718,20 @@ impl DebugManager {
jvm_memory::session_classes(session).await
}
/// Writes a heap dump of the session's JVM to `dir/dump.hprof`, one class-table or dump
/// request per session at a time. Anything but `Ready` deletes `dir`, unless `s` is not a
/// session. On `cancel` the result is `Cancelled` at once, while a dump the JVM is writing
/// still holds the session's turn until the JVM answers.
pub async fn jvm_heap_dump(
&self,
s: DebugSessionId,
dir: PathBuf,
cancel: ide_core::CancellationToken,
) -> IdeResult<JvmHeapOutcome> {
let session = self.session(s)?;
jvm_memory::session_heap_dump(session, dir, cancel).await
}
/// Gives the session the backend for [`Self::jvm_memory_instances`] and the debugger's class
/// counts; the UI is told to reload the session's capabilities.
pub fn set_jvm_inspector(
+2
View File
@@ -2155,11 +2155,13 @@ async fn attach_session_reports_capability() {
let id = h.start(remote_attach(adapter, project.path())).await.unwrap();
let caps = h.manager.capabilities(id).unwrap();
assert!(caps.jvm_memory, "{adapter}");
assert_eq!(caps.jvm_heap, caps.jvm_memory, "{adapter}");
}
let id = h.start(config()).await.unwrap();
let caps = h.manager.capabilities(id).unwrap();
assert!(!caps.jvm_memory);
assert_eq!(caps.jvm_heap, caps.jvm_memory);
assert_eq!(
h.manager.jvm_memory_classes(id).await.unwrap(),
JvmMemoryResult::Unavailable { reason: JvmMemoryUnavailable::NoJvmProcess, can_count_when_paused: false }
+44 -17
View File
@@ -136,24 +136,19 @@ pub struct DockerRegistry {
pub username: String,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(rename_all = "camelCase")]
pub struct DockerSettings {
#[serde(default = "default_connections")]
#[serde(default)]
pub connections: Vec<DockerConnection>,
#[serde(default)]
pub registries: Vec<DockerRegistry>,
}
impl Default for DockerSettings {
fn default() -> Self {
Self { connections: default_connections(), registries: Vec::new() }
}
}
/// Before anything is configured the CLI's own daemon is shown, as before connections existed.
fn default_connections() -> Vec<DockerConnection> {
vec![DockerConnection { id: "default".into(), name: "Docker".into(), endpoint: DockerEndpoint::Default }]
/// Whether `c` is the connection older versions injected when nothing was configured:
/// Services lists only connections the user added, so this one is dropped on load.
fn is_injected_default(c: &DockerConnection) -> bool {
c.id == "default" && c.name == "Docker" && c.endpoint == DockerEndpoint::Default
}
const SETTINGS_FILE: &str = "docker.json";
@@ -161,10 +156,17 @@ const PASSWORDS_FILE: &str = "docker-passwords.json";
pub fn load_settings(data_dir: &Path) -> DockerSettings {
match std::fs::read(data_dir.join(SETTINGS_FILE)) {
Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_else(|e| {
tracing::warn!("{SETTINGS_FILE}: {e}");
DockerSettings::default()
}),
// A file written before the opt-in Services rework carries the injected default
// (saved along with a registry or another connection); drop it unless edited.
Ok(bytes) => serde_json::from_slice(&bytes)
.map(|mut s: DockerSettings| {
s.connections.retain(|c| !is_injected_default(c));
s
})
.unwrap_or_else(|e| {
tracing::warn!("{SETTINGS_FILE}: {e}");
DockerSettings::default()
}),
Err(_) => DockerSettings::default(),
}
}
@@ -373,7 +375,8 @@ mod tests {
fn settings_round_trip_and_passwords() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path();
assert_eq!(load_settings(dir).connections[0].endpoint, DockerEndpoint::Default);
// Nothing configured: no connection is invented.
assert!(load_settings(dir).connections.is_empty());
let settings = DockerSettings {
connections: vec![DockerConnection {
id: "c1".into(),
@@ -395,8 +398,32 @@ mod tests {
assert!(auth_for(dir, "postgres").is_none());
save_settings(dir, &DockerSettings { registries: Vec::new(), ..settings }).unwrap();
assert_eq!(registry_password(dir, "r1"), None);
// An empty list stays empty (the default connection is only for a missing file).
// An empty list stays empty.
save_settings(dir, &DockerSettings { connections: Vec::new(), registries: Vec::new() }).unwrap();
assert!(load_settings(dir).connections.is_empty());
}
#[test]
fn load_drops_the_injected_default_connection() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path();
let injected =
|| DockerConnection { id: "default".into(), name: "Docker".into(), endpoint: DockerEndpoint::Default };
let registry =
DockerRegistry { id: "r1".into(), name: "GHCR".into(), address: "ghcr.io".into(), username: "me".into() };
// Saved untouched by an older version (e.g. along with a registry): dropped on load.
save_settings(dir, &DockerSettings { connections: vec![injected()], registries: vec![registry.clone()] })
.unwrap();
let loaded = load_settings(dir);
assert!(loaded.connections.is_empty());
assert_eq!(loaded.registries, vec![registry]);
// A connection the user renamed or re-pointed stays theirs.
let renamed = DockerConnection { name: "Local engine".into(), ..injected() };
save_settings(dir, &DockerSettings { connections: vec![renamed.clone()], registries: Vec::new() }).unwrap();
assert_eq!(load_settings(dir).connections, vec![renamed]);
let repointed =
DockerConnection { endpoint: DockerEndpoint::Socket { path: "/var/run/docker.sock".into() }, ..injected() };
save_settings(dir, &DockerSettings { connections: vec![repointed.clone()], registries: Vec::new() }).unwrap();
assert_eq!(load_settings(dir).connections, vec![repointed]);
}
}
+17 -2
View File
@@ -71,7 +71,6 @@ impl Capability {
match self {
Self::ProjectRead | Self::EditorRead | Self::Storage | Self::Events | Self::VcsRead => Risk::Low,
Self::EditorWrite
| Self::ProjectWrite
| Self::DatabaseSources
| Self::VcsWrite
| Self::Net(_)
@@ -80,7 +79,14 @@ impl Capability {
| Self::Secrets
| Self::InputEvents
| Self::Clipboard => Risk::Medium,
Self::IdeCommands | Self::Ipc(_) | Self::Process(_) | Self::FsRead(_) | Self::FsWrite(_) => Risk::High,
// A trusted project's own files name the programs the IDE starts (its settings, its
// build scripts), so writing them is running code.
Self::ProjectWrite
| Self::IdeCommands
| Self::Ipc(_)
| Self::Process(_)
| Self::FsRead(_)
| Self::FsWrite(_) => Risk::High,
}
}
@@ -402,6 +408,15 @@ mod tests {
assert!(Capability::Storage.auto_migrates() && !Capability::EditorWrite.auto_migrates());
}
/// A trusted project's files name programs the IDE starts by itself: the git and language
/// server executables in `.openidea/settings.toml`, its build scripts. Writing them is
/// running code, and the install dialog says so.
#[test]
fn writing_project_files_is_a_high_risk_grant() {
assert_eq!(Capability::ProjectWrite.risk(), Risk::High);
assert!(!Capability::ProjectWrite.auto_migrates());
}
#[test]
fn host_patterns() {
assert!(host_matches("crates.io", "crates.io"));
+110 -3
View File
@@ -352,6 +352,22 @@ pub struct IndexEntry {
/// the IDE before showing it.
#[serde(default)]
pub icon: Option<PluginIcon>,
/// Where to read more: the manifest's homepage, else the registry's source repository.
#[serde(default)]
pub homepage: Option<String>,
/// This version's release notes (the manifest's `change-notes`).
#[serde(default)]
pub notes: Option<String>,
/// Downloads of the plugin's packages, as counted by whoever serves them.
#[serde(default)]
pub downloads: Option<u64>,
/// Picked by the marketplace's maintainers.
#[serde(default)]
pub featured: bool,
/// Project-relative paths whose presence makes the plugin worth suggesting
/// (`pom.xml`, `config/routes.rb`).
#[serde(default)]
pub suggest: Vec<String>,
}
impl IndexEntry {
@@ -389,10 +405,31 @@ impl Index {
/// revocations, publisher keys (revoked ones) and `moved` — and lists no
/// plugins ([`Index::is_newer`]): the IDE needs updating to install from it.
pub fn parse(text: &str) -> Result<Self, String> {
let (index, dropped) = Self::parse_dropping(text)?;
if let Some(first) = dropped.first() {
// One line per index, and a bounded one: whoever wrote the index chose the text.
let id: String = first.id.chars().take(80).collect();
tracing::warn!(
"index: {} entries are not offered, their id or version is not one a manifest may carry (first: {id:?})",
dropped.len()
);
}
Ok(index)
}
/// [`Index::parse`], with the entries it does not offer: those whose id or
/// version no manifest may carry, so no IDE could install them.
pub fn parse_dropping(text: &str) -> Result<(Self, Vec<IndexEntry>), String> {
let value: serde_json::Value = serde_json::from_str(text).map_err(|e| format!("index: {e}"))?;
let format = value.get("format").and_then(serde_json::Value::as_u64).ok_or("index: `format` is missing")?;
if format == u64::from(INDEX_FORMAT) {
return serde_json::from_value(value).map_err(|e| format!("index: {e}"));
let mut index: Index = serde_json::from_value(value).map_err(|e| format!("index: {e}"))?;
// What no manifest may carry is not offered: the id and version also name files on disk.
let (offered, dropped) = std::mem::take(&mut index.plugins)
.into_iter()
.partition(|e| crate::validate_plugin_id(&e.id).is_ok() && e.semver().is_some());
index.plugins = offered;
return Ok((index, dropped));
}
if format < u64::from(INDEX_FORMAT) {
return Err(format!("index format {format} is not supported (this IDE reads {INDEX_FORMAT})"));
@@ -401,13 +438,14 @@ impl Index {
let items = value.get(field).and_then(serde_json::Value::as_array);
items.into_iter().flatten().filter_map(|v| serde_json::from_value(v.clone()).ok()).collect()
}
Ok(Index {
let index = Index {
format: u32::try_from(format).unwrap_or(u32::MAX),
publishers: each(&value, "publishers"),
plugins: Vec::new(),
revoked: each(&value, "revoked"),
moved: value.get("moved").and_then(serde_json::Value::as_str).map(str::to_owned),
})
};
Ok((index, Vec::new()))
}
/// Written in a format newer than this IDE reads: it lists no plugins.
@@ -685,9 +723,33 @@ mod tests {
engine: "^1.0".into(),
categories: Vec::new(),
icon: None,
homepage: None,
notes: None,
downloads: None,
featured: false,
suggest: Vec::new(),
}
}
#[test]
fn listing_details_are_optional() {
let mut json = serde_json::to_value(entry("acme.a", "acme")).unwrap();
for key in ["homepage", "notes", "downloads", "featured", "suggest"] {
json.as_object_mut().unwrap().remove(key);
}
assert_eq!(serde_json::from_value::<IndexEntry>(json).unwrap(), entry("acme.a", "acme"));
let full = IndexEntry {
homepage: Some("https://example.com".into()),
notes: Some("Fixes".into()),
downloads: Some(42),
featured: true,
suggest: vec!["pom.xml".into()],
..entry("acme.a", "acme")
};
let back: IndexEntry = serde_json::from_str(&serde_json::to_string(&full).unwrap()).unwrap();
assert_eq!(back, full);
}
fn version(id: &str, version: &str, engine: &str) -> IndexEntry {
IndexEntry { version: version.into(), engine: engine.into(), ..entry(id, "acme") }
}
@@ -868,6 +930,51 @@ mod tests {
}
}
#[test]
fn an_index_offers_only_the_ids_and_versions_a_manifest_may_carry() {
let invalid_ids = [
"../x",
"../../escape",
"acme/../x",
r"acme.\..\x",
"acme./../../../escape",
"/abs",
r"C:\x",
"ACME.x",
"acme.",
"acme..x",
"acme.x y",
"",
];
let invalid_versions =
["../1.0.0", "1.0.0/../../../escape", r"1.0.0\..\x", "1.0.0-..", " 1.0.0", "1.0", "latest", ""];
let valid = [("acme.x", "1.0.0"), ("acme.tools-2_b", "1.0.0-rc.1+b.2")];
let mut listed: Vec<(&str, &str)> = valid.to_vec();
listed.extend(invalid_ids.iter().map(|id| (*id, "1.0.0")));
listed.extend(invalid_versions.iter().map(|version| ("acme.x", *version)));
let plugins: Vec<serde_json::Value> = listed
.iter()
.map(|(id, version)| {
serde_json::json!({
"id": id, "name": "X", "version": version, "publisher": "acme",
"download": "x.oiplugin", "digest": "d", "engine": "^1.0",
})
})
.collect();
let text = serde_json::json!({ "format": INDEX_FORMAT, "plugins": plugins }).to_string();
let pairs = |entries: &[IndexEntry]| -> Vec<(String, String)> {
entries.iter().map(|e| (e.id.clone(), e.version.clone())).collect()
};
let owned = |pairs: &[(&str, &str)]| -> Vec<(String, String)> {
pairs.iter().map(|(id, version)| ((*id).to_owned(), (*version).to_owned())).collect()
};
assert_eq!(pairs(&Index::parse(&text).unwrap().plugins), owned(&valid));
let (index, dropped) = Index::parse_dropping(&text).unwrap();
assert_eq!(pairs(&index.plugins), owned(&valid));
assert_eq!(pairs(&dropped), owned(&listed[valid.len()..]));
}
#[test]
fn a_newer_index_still_revokes_and_moves() {
let newer = format!(
+10 -2
View File
@@ -102,6 +102,14 @@ fn parse<T>(r: serde_json::Result<T>) -> IdeResult<T> {
r.map_err(|e| IdeError::new(ide_core::error::ErrorKind::Protocol, format!("unexpected gh output: {e}")))
}
/// The repository of `root`, free to execute: the host runs the `gh_*` commands only in a
/// trusted project.
fn trusted_repo(root: &Path) -> IdeResult<Repo> {
let repo = Repo::discover(root)?.ok_or_else(|| IdeError::invalid("not a git repository"))?;
repo.set_execution_allowed(true);
Ok(repo)
}
/// Whether gh is installed and signed in, and which GitHub repository `root` maps to.
pub fn availability(root: &Path) -> GhAvailability {
if gh_executable().is_none() {
@@ -233,7 +241,7 @@ pub fn create(
return Err(IdeError::invalid("choose a base branch"));
}
if req.push {
let repo = Repo::discover(root)?.ok_or_else(|| IdeError::invalid("not a git repository"))?;
let repo = trusted_repo(root)?;
repo.push(false, true, cancel, on_output)?;
}
let mut args = vec!["pr", "create", "--title", req.title.trim(), "--body", &req.body, "--base", req.base.trim()];
@@ -354,7 +362,7 @@ pub fn prepare_diff(
cancel: &CancellationToken,
on_output: &dyn Fn(String),
) -> IdeResult<PrDiffRevisions> {
let repo = Repo::discover(root)?.ok_or_else(|| IdeError::invalid("not a git repository"))?;
let repo = trusted_repo(root)?;
if !repo.has_commit(head_oid) || !repo.has_commit(base_oid) {
let remotes = repo.remotes()?;
let remote = remote_for(&remotes, name_with_owner)
+28
View File
@@ -0,0 +1,28 @@
[package]
name = "ide-heap"
description = "HotSpot .hprof parser and heap-snapshot analysis for the debugger's JVM Memory view"
version.workspace = true
edition.workspace = true
license.workspace = true
[dependencies]
ide-core.workspace = true
memmap2.workspace = true
serde.workspace = true
serde_json.workspace = true
specta.workspace = true
thiserror.workspace = true
tracing.workspace = true
uuid.workspace = true
[target.'cfg(unix)'.dependencies]
libc = "0.2"
[dev-dependencies]
criterion = { version = "0.5", default-features = false }
proptest.workspace = true
tempfile.workspace = true
[[bench]]
name = "heap"
harness = false
+100
View File
@@ -0,0 +1,100 @@
//! `cargo bench -p ide-heap`: whole-dump analysis and the dominator tree at a million objects.
use criterion::{Criterion, black_box, criterion_group, criterion_main};
use ide_core::CancellationToken;
use ide_heap::testing::{MiniHeap, XorShift};
use ide_heap::{AnalyzeOptions, BasicType, HeapError, HeapPhase, HeapRootKind, Store, analyze, dominators};
const CHAINS: u64 = 100_000;
const CHAIN_LEN: u64 = 5;
const NODES: u64 = CHAINS * CHAIN_LEN;
/// Every object takes one slot: node `k` at slot `2k`, its payload at `2k + 1`.
const BASE: u64 = 0x1_0000_0000;
const SLOT: u64 = 0x20;
fn node_addr(k: u64) -> u64 {
BASE + 2 * k * SLOT
}
fn payload_addr(k: u64) -> u64 {
node_addr(k) + SLOT
}
/// 100k rooted chains of five `Node { next, payload }`, each node with its own `byte[16]`. Every
/// seventh node points at the payload of the same position in the next chain instead, so that
/// payload has two owners and its own one is garbage: 1M objects in all.
fn million_objects() -> MiniHeap {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("payload", BasicType::Object)]);
for k in 0..NODES {
let next = if (k + 1) % CHAIN_LEN == 0 { 0 } else { node_addr(k + 1) };
let payload = if k % 7 == 0 { payload_addr((k + CHAIN_LEN) % NODES) } else { payload_addr(k) };
heap.object(node_addr(k), "Node", &[("next", next), ("payload", payload)]);
heap.prim(payload_addr(k), BasicType::Byte, 16);
if k % CHAIN_LEN == 0 {
heap.root(HeapRootKind::JniGlobal, node_addr(k));
}
}
heap
}
/// CSR of a random tree on `nodes` nodes rooted at 0 plus `extra` random edges, targets of each
/// node in scrambled order.
fn random_graph(nodes: u32, extra: u32, seed: u64) -> (Vec<u32>, Vec<u32>) {
let mut rng = XorShift::new(seed);
let mut edges: Vec<(u32, u32)> = (1..nodes).map(|v| (rng.below(v), v)).collect();
edges.extend((0..extra).map(|_| (rng.below(nodes), rng.below(nodes))));
for i in (1..edges.len()).rev() {
edges.swap(i, rng.below(i as u32 + 1) as usize);
}
let mut start = vec![0u32; nodes as usize + 1];
for &(from, _) in &edges {
start[from as usize + 1] += 1;
}
for v in 0..nodes as usize {
start[v + 1] += start[v];
}
let mut cursor = start.clone();
let mut out = vec![0u32; edges.len()];
for &(from, to) in &edges {
out[cursor[from as usize] as usize] = to;
cursor[from as usize] += 1;
}
(start, out)
}
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn bench(c: &mut Criterion) {
let dir = tempfile::tempdir().expect("temp dir");
let hprof = dir.path().join("dump.hprof");
million_objects().write_to(&hprof);
let (start, out) = random_graph(1_000_000, 500_000, 0x5EED_D0D0);
let cancel = CancellationToken::new();
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
let summary = analyze(&hprof, dir.path(), opts, &no_progress, &cancel).expect("analyze").summary;
assert_eq!(summary.objects, 2 * NODES, "{summary:?}");
assert_eq!(summary.unreachable_objects, NODES.div_ceil(7), "{summary:?}");
let mut g = c.benchmark_group("heap");
g.sample_size(10);
g.bench_function("index_1m_objects", |b| {
b.iter(|| {
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
let index = analyze(&hprof, dir.path(), opts, &no_progress, &cancel).expect("analyze");
black_box(index.summary.objects)
})
});
g.bench_function("dominators_1m_random", |b| {
b.iter(|| {
let d = dominators(&start, &out, 0, &Store::Ram, &no_progress, &cancel).expect("dominators");
black_box(d.reachable)
})
});
g.finish();
}
criterion_group!(benches, bench);
criterion_main!(benches);
+202
View File
@@ -0,0 +1,202 @@
//! Ties the passes together: graph, dominators, shallow and retained sizes, the class table and
//! paths to GC roots, into a [`HeapIndex`].
mod classes;
mod paths;
mod retained;
use std::collections::HashMap;
use std::fs::File;
use std::path::Path;
use std::time::Instant;
use ide_core::CancellationToken;
use memmap2::Mmap;
use crate::column::Store;
use crate::dominators::{Dominators, dominators};
use crate::error::HeapError;
use crate::files::INDEX_FILES;
use crate::graph::{HeapGraph, build_graph};
use crate::index::{HeapIndex, HeapSummary, write_summary};
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
use crate::shallow::{HistogramEntry, ShallowModel, VmLayout};
#[derive(Clone, Debug, Default)]
pub struct AnalyzeOptions {
/// The JVM's class histogram from the same stop as the dump.
pub histogram: Option<Vec<HistogramEntry>>,
/// `VM.flags` output of the JVM that wrote the dump.
pub vm_flags: Option<String>,
pub java_major: Option<u32>,
/// Keep every array in RAM and write nothing to `index_dir`.
pub in_memory: bool,
}
/// Analyzes the dump at `hprof`, writing the index columns and `summary.json` into `index_dir`
/// (which must exist and may hold the dump itself). `progress` gets every [`HeapPhase`] in order;
/// an `Err` from it, a cancel or any other error removes every index file from `index_dir` and
/// leaves the rest of the directory alone.
pub fn analyze(
hprof: &Path,
index_dir: &Path,
opts: AnalyzeOptions,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapIndex, HeapError> {
let store = if opts.in_memory { Store::Ram } else { Store::Dir(index_dir.to_path_buf()) };
let result = run(hprof, &store, opts, progress, cancel);
// `run` has dropped every column by now: Windows cannot delete a mapped file.
if result.is_err() && !matches!(store, Store::Ram) {
remove_index_files(index_dir);
}
result
}
fn run(
hprof: &Path,
store: &Store,
opts: AnalyzeOptions,
report: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapIndex, HeapError> {
let started = Instant::now();
let progress = Progress::new(report, cancel);
let file = File::open(hprof)?;
// SAFETY: nothing writes the dump while it is analyzed or viewed: a capture finishes writing
// it before analysis and deletes it only with the snapshot that holds this index.
let dump = unsafe { Mmap::map(&file)? };
let graph = build_graph(&dump, store, report, cancel)?;
let Dominators { idom, preorder, .. } =
dominators(&graph.out_start, &graph.out, graph.root, store, report, cancel)?;
let nodes = u64::from(graph.root) + 1;
// Seven walks over the nodes, roughly: sizes, retained, child counts, child sorting, class
// counts and the tree walk, which visits every node twice.
let mut meter = Meter::start(&progress, HeapPhase::Retained, 7 * nodes)?;
let layout = VmLayout::from_vm_flags(opts.vm_flags.as_deref().unwrap_or(""), opts.java_major);
let model = ShallowModel::new(&graph, layout, opts.histogram.as_deref(), cancel)?;
let (shallow8, retained) = retained::sizes(&graph, &model, &idom, &preorder, store, &mut meter)?;
drop(preorder);
let (kids_start, kids) = retained::children(&idom, &retained, graph.root, store, &mut meter)?;
let tree = retained::Tree { idom, shallow8, retained, kids_start, kids };
let (classes, totals) = classes::table(&graph, &tree, store, &mut meter)?;
meter.finish()?;
let mut meter = Meter::start(&progress, HeapPhase::Paths, nodes + graph.out.len() as u64)?;
let gc_parent = paths::gc_parents(&graph, store, &mut meter)?;
meter.finish()?;
progress.report(HeapPhase::Saving, 0, 1)?;
let HeapGraph {
ids,
types,
offsets,
roots,
classes: class_info,
class_slot,
thread_objects,
weak_edges_skipped,
id_size,
..
} = graph;
let summary = HeapSummary {
objects: totals.reachable_objects + totals.unreachable_objects,
reachable_objects: totals.reachable_objects,
reachable_bytes: totals.reachable_bytes,
unreachable_objects: totals.unreachable_objects,
unreachable_bytes: totals.unreachable_bytes,
classes: class_info.iter().filter(|c| c.id != 0).count() as u32,
weak_edges_skipped,
size_source: model.source,
analysis_ms: u32::try_from(started.elapsed().as_millis()).unwrap_or(u32::MAX),
};
if let Store::Dir(dir) = store {
write_summary(dir, &summary)?;
}
let mut root_kinds = HashMap::with_capacity(roots.len());
for (object, kind, serial) in roots {
root_kinds.entry(object).or_insert((kind, serial));
}
progress.report(HeapPhase::Saving, 1, 1)?;
let retained::Tree { idom, shallow8, retained, kids_start, kids } = tree;
Ok(HeapIndex {
hprof: dump,
ids,
types,
offsets,
shallow8,
retained,
idom,
kids_start,
kids,
gc_parent,
classes,
class_info,
class_slot,
roots: root_kinds,
thread_objects,
id_size,
summary,
})
}
/// Removes what an analysis may have written; a file that is not there is fine.
fn remove_index_files(dir: &Path) {
for name in INDEX_FILES {
let path = dir.join(name);
if let Err(err) = std::fs::remove_file(&path)
&& err.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not remove {}: {err}", path.display());
}
}
}
/// Steps of one phase: reports progress every [`CHECK_EVERY`] steps, and each loop looks at the
/// token before its first step.
pub(super) struct Meter<'a> {
progress: &'a Progress<'a>,
phase: HeapPhase,
/// An estimate; `done` is clamped to it.
total: u64,
done: u64,
next_report: u64,
}
impl<'a> Meter<'a> {
fn start(progress: &'a Progress<'a>, phase: HeapPhase, total: u64) -> Result<Self, HeapError> {
progress.report(phase, 0, total)?;
Ok(Self { progress, phase, total, done: 0, next_report: CHECK_EVERY })
}
pub(super) fn begin_loop(&self) -> Result<(), HeapError> {
self.progress.check()
}
pub(super) fn token(&self) -> &'a CancellationToken {
self.progress.token()
}
#[inline]
pub(super) fn tick(&mut self) -> Result<(), HeapError> {
self.advance(1)
}
pub(super) fn advance(&mut self, steps: u64) -> Result<(), HeapError> {
self.done += steps;
if self.done < self.next_report {
return Ok(());
}
self.next_report = self.done + CHECK_EVERY;
self.progress.report(self.phase, self.done.min(self.total), self.total)
}
fn finish(self) -> Result<(), HeapError> {
self.progress.report(self.phase, self.total, self.total)
}
}
#[cfg(test)]
mod tests;
+117
View File
@@ -0,0 +1,117 @@
//! The class table: counts and shallow sizes per type slot, retained sizes per type from a walk of
//! the dominator tree, and the reachable / unreachable totals of the summary.
use super::Meter;
use super::retained::Tree;
use crate::column::{NONE, Store};
use crate::error::HeapError;
use crate::graph::{HeapGraph, PRIM_SLOTS};
use crate::hprof::BasicType;
use crate::index::ClassStat;
use crate::names::display_name;
/// Instances and arrays only: class objects are left out.
#[derive(Default)]
pub(super) struct Totals {
pub(super) reachable_objects: u64,
pub(super) reachable_bytes: u64,
pub(super) unreachable_objects: u64,
pub(super) unreachable_bytes: u64,
}
pub(super) fn table(
graph: &HeapGraph,
tree: &Tree,
store: &Store,
meter: &mut Meter,
) -> Result<(Vec<ClassStat>, Totals), HeapError> {
let slots = graph.classes.len() + PRIM_SLOTS as usize;
let mut classes: Vec<ClassStat> = (0..slots as u32)
.map(|slot| ClassStat { name: slot_name(graph, slot), count: 0, shallow: 0, retained: 0, unreachable_count: 0 })
.collect();
let mut totals = Totals::default();
meter.begin_loop()?;
for v in 0..graph.root as usize {
meter.tick()?;
let slot = graph.types[v];
let bytes = u64::from(tree.shallow8[v]) * 8;
let stat = &mut classes[slot as usize];
let instance = slot != graph.class_slot;
if tree.idom[v] == NONE {
stat.unreachable_count += 1;
if instance {
totals.unreachable_objects += 1;
totals.unreachable_bytes += bytes;
}
} else {
stat.count += 1;
stat.shallow += bytes;
if instance {
totals.reachable_objects += 1;
totals.reachable_bytes += bytes;
}
}
}
retained_by_type(graph, tree, &mut classes, store, meter)?;
Ok((classes, totals))
}
/// Adds to each type the retained size of its instances that have no instance of the same type
/// above them in the dominator tree. The walk keeps no stack: it goes up through `idom` and
/// resumes a node's children from `cursor`, while `active[t]` counts the instances of type `t` on
/// the path from the root.
fn retained_by_type(
graph: &HeapGraph,
tree: &Tree,
classes: &mut [ClassStat],
store: &Store,
meter: &mut Meter,
) -> Result<(), HeapError> {
let root = graph.root;
let mut cursor = store.scratch::<u32>(root as usize + 1, 0)?;
let mut active = vec![0u32; classes.len()];
cursor[root as usize] = tree.kids_start[root as usize];
let mut v = root;
meter.begin_loop()?;
loop {
meter.tick()?;
let next = cursor[v as usize];
if next < tree.kids_start[v as usize + 1] {
cursor[v as usize] = next + 1;
let child = tree.kids[next as usize];
let slot = graph.types[child as usize] as usize;
if active[slot] == 0 {
classes[slot].retained += tree.retained[child as usize];
}
active[slot] += 1;
cursor[child as usize] = tree.kids_start[child as usize];
v = child;
} else if v == root {
return Ok(());
} else {
active[graph.types[v as usize] as usize] -= 1;
v = tree.idom[v as usize];
}
}
}
fn slot_name(graph: &HeapGraph, slot: u32) -> String {
match graph.classes.get(slot as usize) {
Some(class) => display_name(&class.name),
None => graph.prim_type(slot).map_or_else(String::new, |ty| prim_array_name(ty).to_owned()),
}
}
fn prim_array_name(ty: BasicType) -> &'static str {
match ty {
BasicType::Object => "java.lang.Object[]",
BasicType::Boolean => "boolean[]",
BasicType::Char => "char[]",
BasicType::Float => "float[]",
BasicType::Double => "double[]",
BasicType::Byte => "byte[]",
BasicType::Short => "short[]",
BasicType::Int => "int[]",
BasicType::Long => "long[]",
}
}
+32
View File
@@ -0,0 +1,32 @@
//! Parents on shortest paths from the GC roots, by breadth-first search over strong edges.
use super::Meter;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::GC_PARENT_FILE;
use crate::graph::HeapGraph;
pub(super) fn gc_parents(graph: &HeapGraph, store: &Store, meter: &mut Meter) -> Result<Column<u32>, HeapError> {
let root = graph.root;
let nodes = root as usize + 1;
let mut parent = store.column::<u32>(GC_PARENT_FILE, nodes, NONE)?;
let mut queue = store.scratch::<u32>(nodes, 0)?;
parent[root as usize] = root;
queue[0] = root;
let (mut head, mut tail) = (0, 1);
meter.begin_loop()?;
while head < tail {
meter.tick()?;
let v = queue[head];
head += 1;
for &w in graph.out_edges(v) {
meter.tick()?;
if parent[w as usize] == NONE {
parent[w as usize] = v;
queue[tail] = w;
tail += 1;
}
}
}
Ok(parent)
}
+209
View File
@@ -0,0 +1,209 @@
//! Shallow and retained sizes, and the dominator tree as child lists sorted by retained size.
use std::cmp::Reverse;
use super::Meter;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::{KIDS_FILE, KIDS_START_FILE, RETAINED_FILE, SHALLOW8_FILE};
use crate::graph::{HeapGraph, accumulate};
use crate::progress::CHECK_EVERY;
use crate::shallow::ShallowModel;
/// The dominator tree with sizes; see [`crate::HeapIndex`] for each column.
pub(super) struct Tree {
pub(super) idom: Column<u32>,
pub(super) shallow8: Column<u32>,
pub(super) retained: Column<u64>,
pub(super) kids_start: Column<u32>,
pub(super) kids: Column<u32>,
}
/// `(shallow8, retained)` of every node, the root included. `preorder` is the DFS preorder of the
/// reachable nodes, from the root.
pub(super) fn sizes(
graph: &HeapGraph,
model: &ShallowModel,
idom: &[u32],
preorder: &[u32],
store: &Store,
meter: &mut Meter,
) -> Result<(Column<u32>, Column<u64>), HeapError> {
let nodes = idom.len();
let mut shallow8 = store.column::<u32>(SHALLOW8_FILE, nodes, 0)?;
let mut retained = store.column::<u64>(RETAINED_FILE, nodes, 0)?;
meter.begin_loop()?;
for v in 0..graph.root {
meter.tick()?;
let units = u32::try_from(model.size_of(graph, v).div_ceil(8)).unwrap_or(u32::MAX);
shallow8[v as usize] = units;
if idom[v as usize] != NONE {
retained[v as usize] = u64::from(units) * 8;
}
}
// A dominator is a DFS ancestor, so in reverse preorder every node is complete before it is
// added to its dominator.
meter.begin_loop()?;
for &v in preorder.iter().skip(1).rev() {
meter.tick()?;
let own = retained[v as usize];
retained[idom[v as usize] as usize] += own;
}
Ok((shallow8, retained))
}
/// `(kids_start, kids)`: the children of every reachable node in the dominator tree, as CSR.
pub(super) fn children(
idom: &[u32],
retained: &[u64],
root: u32,
store: &Store,
meter: &mut Meter,
) -> Result<(Column<u32>, Column<u32>), HeapError> {
let nodes = idom.len();
// Counted into `starts[d + 1]`; filled through `starts[d]` as a cursor, which leaves it at d's
// end until the column is shifted one slot.
let mut starts = store.column::<u32>(KIDS_START_FILE, nodes + 1, 0)?;
let mut longest = 0;
meter.begin_loop()?;
for (v, &d) in idom.iter().enumerate() {
meter.tick()?;
if d != NONE && v != root as usize {
let count = &mut starts[d as usize + 1];
*count += 1;
longest = longest.max(*count);
}
}
let total = accumulate(&mut starts, meter.token())?;
let mut kids = store.column::<u32>(KIDS_FILE, total as usize, 0)?;
meter.begin_loop()?;
for (v, &d) in idom.iter().enumerate() {
meter.tick()?;
if d != NONE && v != root as usize {
let cursor = &mut starts[d as usize];
kids[*cursor as usize] = v as u32;
*cursor += 1;
}
}
starts.copy_within(0..nodes, 1);
starts[0] = 0;
sort_children(&mut kids, &starts, retained, longest as usize, CHECK_EVERY as usize, store, meter)?;
Ok((starts, kids))
}
/// Sorts each child list by retained size descending, then by node. A list longer than `run` is
/// sorted in runs of `run` that are then merged, so no single step takes longer than one run.
fn sort_children(
kids: &mut [u32],
starts: &[u32],
retained: &[u64],
longest: usize,
run: usize,
store: &Store,
meter: &mut Meter,
) -> Result<(), HeapError> {
let key = |&k: &u32| (Reverse(retained[k as usize]), k);
let mut buffer = if longest > run { store.scratch::<u32>(longest, 0)? } else { Column::ram(0, 0) };
meter.begin_loop()?;
for v in 0..starts.len() - 1 {
meter.tick()?;
let list = &mut kids[starts[v] as usize..starts[v + 1] as usize];
if list.len() <= run {
list.sort_unstable_by_key(key);
meter.advance(list.len() as u64)?;
continue;
}
for chunk in list.chunks_mut(run) {
meter.begin_loop()?;
chunk.sort_unstable_by_key(key);
meter.advance(chunk.len() as u64)?;
}
let len = list.len();
let (mut from, mut to) = (list, &mut buffer[..len]);
let mut in_buffer = false;
let mut width = run;
while width < len {
meter.begin_loop()?;
for start in (0..len).step_by(2 * width) {
let mid = (start + width).min(len);
let end = (start + 2 * width).min(len);
merge(&from[start..mid], &from[mid..end], &mut to[start..end], &key, meter)?;
}
std::mem::swap(&mut from, &mut to);
in_buffer = !in_buffer;
width *= 2;
}
if in_buffer {
to.copy_from_slice(from);
}
}
Ok(())
}
fn merge<K: Ord>(
left: &[u32],
right: &[u32],
out: &mut [u32],
key: &impl Fn(&u32) -> K,
meter: &mut Meter,
) -> Result<(), HeapError> {
let (mut i, mut j) = (0, 0);
for slot in out.iter_mut() {
meter.tick()?;
let take_left = j == right.len() || (i < left.len() && key(&left[i]) <= key(&right[j]));
if take_left {
*slot = left[i];
i += 1;
} else {
*slot = right[j];
j += 1;
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use ide_core::CancellationToken;
use super::*;
use crate::error::HeapError;
use crate::progress::{HeapPhase, Progress};
use crate::testing::XorShift;
fn sorted_with_run(lists: &[Vec<u32>], retained: &[u64], run: usize) -> Vec<u32> {
let report = |_, _, _| Ok::<(), HeapError>(());
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
let mut meter = Meter::start(&progress, HeapPhase::Retained, 1).unwrap();
let mut starts = vec![0u32];
for list in lists {
starts.push(starts.last().unwrap() + list.len() as u32);
}
let mut kids: Vec<u32> = lists.concat();
let longest = lists.iter().map(Vec::len).max().unwrap_or(0);
sort_children(&mut kids, &starts, retained, longest, run, &Store::Ram, &mut meter).unwrap();
kids
}
#[test]
fn long_child_lists_are_merged_from_sorted_runs() {
let mut rng = XorShift::new(7);
let retained: Vec<u64> = (0..500).map(|_| u64::from(rng.below(40))).collect();
let mut nodes: Vec<u32> = (0..500).collect();
for i in (1..nodes.len()).rev() {
nodes.swap(i, rng.below(i as u32 + 1) as usize);
}
// Lists below, at and above the run length, and one that ends on a partial run.
let lists = vec![nodes[..3].to_vec(), nodes[3..7].to_vec(), nodes[7..400].to_vec(), nodes[400..].to_vec()];
for run in [1, 4, 7, 64, 1000] {
let mut expected = Vec::new();
for list in &lists {
let mut list = list.clone();
list.sort_by_key(|&k| (Reverse(retained[k as usize]), k));
expected.extend(list);
}
assert_eq!(sorted_with_run(&lists, &retained, run), expected, "run {run}");
}
}
}
+265
View File
@@ -0,0 +1,265 @@
use std::cell::Cell;
use std::collections::BTreeSet;
use std::path::PathBuf;
use super::*;
use crate::column::{Column, NONE};
use crate::files::INDEX_FILES;
use crate::hprof::{BasicType, HeapRootKind};
use crate::index::{ClassStat, HeapSummary};
use crate::shallow::SizeSource;
use crate::testing::{MiniHeap, sample};
const DUMP: &str = "dump.hprof";
const ROOT: Option<&str> = Some("java.lang.Object");
const NODE_FIELDS: [(&str, BasicType); 2] = [("next", BasicType::Object), ("v", BasicType::Long)];
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn in_memory() -> AnalyzeOptions {
AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() }
}
/// A temp dir holding `heap` as `dump.hprof`, the way a capture lays out its snapshot dir.
fn dump_dir(heap: &MiniHeap) -> (tempfile::TempDir, PathBuf) {
let dir = tempfile::tempdir().unwrap();
let dump = dir.path().join(DUMP);
heap.write_to(&dump);
(dir, dump)
}
fn analyze_with(heap: &MiniHeap, opts: AnalyzeOptions) -> HeapIndex {
let (dir, dump) = dump_dir(heap);
analyze(&dump, dir.path(), opts, &no_progress, &CancellationToken::new()).unwrap()
}
fn file_names(dir: &Path) -> BTreeSet<String> {
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect()
}
fn node(index: &HeapIndex, addr: u64) -> u32 {
index.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the dump"))
}
fn shallow(index: &HeapIndex, addr: u64) -> u64 {
u64::from(index.shallow8[node(index, addr) as usize]) * 8
}
fn retained(index: &HeapIndex, addr: u64) -> u64 {
index.retained[node(index, addr) as usize]
}
fn idom(index: &HeapIndex, addr: u64) -> u32 {
index.idom[node(index, addr) as usize]
}
fn class<'a>(index: &'a HeapIndex, name: &str) -> &'a ClassStat {
let mut rows = index.classes.iter().filter(|c| c.name == name);
let row = rows.next().unwrap_or_else(|| panic!("no class {name}"));
assert!(rows.next().is_none(), "two classes named {name}");
row
}
fn stat(c: &ClassStat) -> (u64, u64, u64, u64) {
(c.count, c.shallow, c.retained, c.unreachable_count)
}
fn kids_of(index: &HeapIndex, v: u32) -> &[u32] {
&index.kids[index.kids_start[v as usize] as usize..index.kids_start[v as usize + 1] as usize]
}
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N2: u64 = 0x1200;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const R: u64 = 0x1500;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
#[test]
fn sample_retained_sizes() {
let index = analyze_with(&sample(), in_memory());
let shallows: Vec<u64> = [H, N1, L, W, R].iter().map(|&a| shallow(&index, a)).collect();
assert_eq!(shallows, [24, 24, 32, 24, 32]);
let retained_sizes: Vec<u64> = [L, N3, N1, N2, H, W].iter().map(|&a| retained(&index, a)).collect();
assert_eq!(retained_sizes, [32, 56, 24, 24, 72, 24]);
let root = index.root();
assert_eq!(
[idom(&index, N1), idom(&index, N2), idom(&index, N3), idom(&index, L)],
[node(&index, H), node(&index, H), root, node(&index, N3)]
);
assert_eq!((idom(&index, R), idom(&index, N4)), (NONE, NONE));
assert_eq!((retained(&index, R), retained(&index, N4)), (0, 0));
// Children by retained size descending, ties by node.
assert_eq!(kids_of(&index, node(&index, H)), [node(&index, N1), node(&index, N2)]);
let top = kids_of(&index, root);
assert_eq!(top[..2], [node(&index, H), node(&index, N3)]);
assert!(top.windows(2).all(|p| (index.retained[p[0] as usize], p[1]) > (index.retained[p[1] as usize], p[0])));
assert_eq!(index.retained[root as usize], top.iter().map(|&k| index.retained[k as usize]).sum::<u64>());
}
#[test]
fn sample_class_table() {
let index = analyze_with(&sample(), in_memory());
assert_eq!(stat(class(&index, "Node")), (3, 72, 104, 1));
assert_eq!(stat(class(&index, "Holder")), (1, 24, 72, 0));
assert_eq!(stat(class(&index, "long[]")), (1, 32, 32, 0));
assert_eq!(stat(class(&index, "java.lang.ref.WeakReference")), (1, 24, 24, 0));
let objects = class(&index, "java.lang.Object[]");
assert_eq!((objects.count, objects.unreachable_count), (0, 1));
// Indexed by type slot, primitive array slots included.
assert_eq!(index.classes.len(), index.class_info.len() + crate::graph::PRIM_SLOTS as usize);
let slot = index.types[node(&index, H) as usize] as usize;
assert_eq!(index.classes[slot].name, "Holder");
assert_eq!(index.class_info[slot].name, "Holder");
}
#[test]
fn sample_summary() {
let index = analyze_with(&sample(), in_memory());
let s = &index.summary;
assert_eq!(
(s.reachable_objects, s.reachable_bytes, s.unreachable_objects, s.unreachable_bytes, s.weak_edges_skipped),
(6, 152, 2, 56, 1)
);
assert_eq!(s.objects, 8);
assert_eq!(s.classes, 7);
assert_eq!(s.size_source, SizeSource::Layout);
}
#[test]
fn sample_gc_parents() {
let index = analyze_with(&sample(), in_memory());
let parent = |addr| index.gc_parent[node(&index, addr) as usize];
let root = index.root();
assert_eq!(index.gc_parent[root as usize], root);
assert_eq!([parent(H), parent(N3), parent(W)], [root; 3]);
assert_eq!([parent(N1), parent(L)], [node(&index, H), node(&index, N3)]);
assert_eq!((parent(R), parent(N4)), (NONE, NONE));
assert_eq!(index.roots[&node(&index, N3)], (HeapRootKind::JavaFrame, Some(1)));
}
#[test]
fn cycle_is_retained_once() {
let mut heap = MiniHeap::new();
heap.class("Node", ROOT, &NODE_FIELDS)
.object(0xA000, "Node", &[("next", 0xB000)])
.object(0xB000, "Node", &[("next", 0xA000)])
.root(HeapRootKind::JniGlobal, 0xA000);
let index = analyze_with(&heap, in_memory());
assert_eq!((retained(&index, 0xA000), retained(&index, 0xB000)), (48, 24));
assert_eq!(class(&index, "Node").retained, 48);
}
#[test]
fn deep_chain_retained_by_class() {
// A small stack proves no pass recurses along the chain.
std::thread::Builder::new()
.stack_size(256 << 10)
.spawn(|| {
const NODES: u64 = 1_000_000;
let addr = |i: u64| 0x10_0000 + i * 0x100;
let mut heap = MiniHeap::new();
heap.class("Node", ROOT, &NODE_FIELDS);
for i in 0..NODES {
let next = if i + 1 < NODES { addr(i + 1) } else { 0 };
heap.object(addr(i), "Node", &[("next", next)]);
}
heap.root(HeapRootKind::JniGlobal, addr(0));
let index = analyze_with(&heap, in_memory());
assert_eq!(retained(&index, addr(0)), 24_000_000);
assert_eq!(retained(&index, addr(NODES - 1)), 24);
assert_eq!(class(&index, "Node").retained, 24_000_000);
assert_eq!(index.summary.reachable_objects, NODES);
})
.unwrap()
.join()
.unwrap();
}
#[test]
fn histogram_sizes_flow_into_retained() {
let histogram = vec![HistogramEntry { name: "Node".to_owned(), instances: 3, bytes: 96 }];
let index = analyze_with(&sample(), AnalyzeOptions { histogram: Some(histogram), ..in_memory() });
assert_eq!(shallow(&index, N3), 32);
assert_eq!(retained(&index, N3), 64);
assert_eq!(index.summary.size_source, SizeSource::Jvm);
}
#[test]
fn vm_flags_of_an_unknown_jdk_set_the_layout() {
// Without compressed oops a Node is 12 + 8 + 8 = 28 → 32.
let opts = AnalyzeOptions { vm_flags: Some("-XX:MaxHeapSize=42949672960".to_owned()), ..in_memory() };
let index = analyze_with(&sample(), opts);
assert_eq!(shallow(&index, N4), 32);
}
#[test]
fn cancel_stops_analysis() {
let (dir, dump) = dump_dir(&sample());
let cancel = CancellationToken::new();
let phases = Cell::new(Vec::new());
let progress = |phase, _, _| {
let mut seen = phases.take();
seen.push(phase);
phases.set(seen);
if phase == HeapPhase::Dominators {
cancel.cancel();
}
Ok(())
};
let result = analyze(&dump, dir.path(), AnalyzeOptions::default(), &progress, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)), "{:?}", result.err());
assert_eq!(phases.take().last(), Some(&HeapPhase::Dominators), "stops at the first check");
// The dump the capture analyzes in place stays; every file the analysis wrote is gone.
assert_eq!(file_names(dir.path()), BTreeSet::from([DUMP.to_owned()]));
}
#[test]
fn progress_error_stops_analysis_and_removes_its_files() {
let (dir, dump) = dump_dir(&sample());
// The last report, once every index file is written.
let progress = |phase, done, _| match (phase, done) {
(HeapPhase::Saving, 1) => Err(HeapError::NotEnoughDiskSpace { needed: 2, available: 1 }),
_ => Ok(()),
};
let result = analyze(&dump, dir.path(), AnalyzeOptions::default(), &progress, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::NotEnoughDiskSpace { .. })), "{:?}", result.err());
assert_eq!(file_names(dir.path()), BTreeSet::from([DUMP.to_owned()]));
}
#[test]
fn writes_index_files() {
let (dir, dump) = dump_dir(&sample());
let index = analyze(&dump, dir.path(), AnalyzeOptions::default(), &no_progress, &CancellationToken::new()).unwrap();
let names = file_names(dir.path());
for name in [
"types.u32",
"shallow8.u32",
"retained.u64",
"idom.u32",
"kids_start.u32",
"kids.u32",
"gc_parent.u32",
"offsets.u64",
"summary.json",
] {
assert!(names.contains(name), "{name} missing from {names:?}");
}
assert!(names.iter().any(|n| n.starts_with("ids.")), "{names:?}");
// Nothing but the index files and the dump: scratch arrays are gone.
let unexpected: Vec<_> = names.iter().filter(|n| *n != DUMP && !INDEX_FILES.contains(&n.as_str())).collect();
assert!(unexpected.is_empty(), "{unexpected:?}");
let summary: HeapSummary =
serde_json::from_slice(&std::fs::read(dir.path().join("summary.json")).unwrap()).unwrap();
assert_eq!(summary, index.summary);
drop(index);
let retained = Column::<u64>::open(&dir.path().join("retained.u64")).unwrap();
let in_ram = analyze_with(&sample(), in_memory());
assert_eq!(&*retained, &*in_ram.retained);
}
+306
View File
@@ -0,0 +1,306 @@
//! Fixed-width arrays that live in RAM or in memory-mapped files, so the analysis arrays of a
//! large heap do not count against the process's anonymous memory.
use std::fs::{File, OpenOptions};
use std::io;
use std::marker::PhantomData;
use std::mem::size_of;
use std::ops::{Deref, DerefMut};
use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
/// Index value meaning "no object" in every `u32` column.
pub const NONE: u32 = u32::MAX;
/// # Safety
/// Every bit pattern of `Self` is a valid value and `Self` has no padding, so a mapped file's
/// bytes can be viewed as `[Self]` and a file can be written from one.
pub unsafe trait Pod: Copy + 'static {}
unsafe impl Pod for u8 {}
unsafe impl Pod for u32 {}
unsafe impl Pod for u64 {}
enum Backing<T> {
Ram(Vec<T>),
Map(MmapMut),
ReadOnly(Mmap),
}
/// A fixed-length array of `T`. File contents use the machine's byte order: the arrays are a
/// cache for this machine and are never moved to another one.
pub struct Column<T: Pod> {
backing: Backing<T>,
/// Scratch file to delete on drop where it could not be unlinked while mapped.
delete_on_drop: Option<PathBuf>,
_marker: PhantomData<T>,
}
fn byte_len<T: Pod>(len: usize) -> io::Result<usize> {
len.checked_mul(size_of::<T>())
.filter(|&bytes| bytes <= isize::MAX as usize)
.ok_or_else(|| io::Error::new(io::ErrorKind::InvalidInput, "column is too long"))
}
fn is_zero<T: Pod>(value: &T) -> bool {
// SAFETY: `Pod` types have no padding, so all `size_of::<T>()` bytes are initialized.
let bytes = unsafe { std::slice::from_raw_parts((value as *const T).cast::<u8>(), size_of::<T>()) };
bytes.iter().all(|&b| b == 0)
}
impl<T: Pod> Column<T> {
fn new(backing: Backing<T>, delete_on_drop: Option<PathBuf>) -> Self {
Self { backing, delete_on_drop, _marker: PhantomData }
}
pub fn ram(len: usize, fill: T) -> Self {
Self::new(Backing::Ram(vec![fill; len]), None)
}
/// Creates (or truncates) `path` and maps it; the file stays after the column is dropped.
pub fn file(path: &Path, len: usize, fill: T) -> io::Result<Self> {
let bytes = byte_len::<T>(len)?;
let file = OpenOptions::new().read(true).write(true).create(true).truncate(true).open(path)?;
match Self::map_new(&file, len, bytes, fill) {
Ok(column) => Ok(column),
Err(err) => {
drop(file);
let _ = std::fs::remove_file(path);
Err(err)
}
}
}
/// A temporary column in `dir` that never outlives the process. On unix the file is unlinked
/// right after mapping, so even a crash leaves nothing behind; elsewhere (Windows cannot delete
/// a mapped file) it is deleted on drop.
pub fn scratch(dir: &Path, len: usize, fill: T) -> io::Result<Self> {
let bytes = byte_len::<T>(len)?;
if bytes == 0 {
return Ok(Self::ram(0, fill));
}
let path = dir.join(format!("scratch-{}.tmp", uuid::Uuid::new_v4().simple()));
let file = OpenOptions::new().read(true).write(true).create_new(true).open(&path)?;
let mut column = match Self::map_new(&file, len, bytes, fill) {
Ok(column) => column,
Err(err) => {
drop(file);
let _ = std::fs::remove_file(&path);
return Err(err);
}
};
// If the unlink fails the drop-time delete is the fallback, so the file is never left behind.
if !cfg!(unix) || std::fs::remove_file(&path).is_err() {
column.delete_on_drop = Some(path);
}
Ok(column)
}
/// Maps an existing file read-only; writing through the column panics.
pub fn open(path: &Path) -> io::Result<Self> {
let file = File::open(path)?;
let bytes = file.metadata()?.len();
if bytes % size_of::<T>() as u64 != 0 {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
format!("{} is {bytes} bytes, not a multiple of {}", path.display(), size_of::<T>()),
));
}
if bytes == 0 {
return Ok(Self::ram(0, zero_value()));
}
// SAFETY: the file is private to the snapshot directory; nothing truncates it while mapped.
let map = unsafe { Mmap::map(&file)? };
Ok(Self::new(Backing::ReadOnly(map), None))
}
fn map_new(file: &File, len: usize, bytes: usize, fill: T) -> io::Result<Self> {
if bytes == 0 {
return Ok(Self::ram(0, fill));
}
file.set_len(bytes as u64)?;
// SAFETY: as in `open`; the file was just created by this process.
let map = unsafe { MmapMut::map_mut(file)? };
let mut column = Self::new(Backing::Map(map), None);
// A fresh file reads as zeros, and writing a zero fill would fault in every page.
if !is_zero(&fill) {
debug_assert_eq!(column.len(), len);
column.fill(fill);
}
Ok(column)
}
}
fn zero_value<T: Pod>() -> T {
// SAFETY: every bit pattern of a `Pod` type is a valid value.
unsafe { std::mem::zeroed() }
}
impl<T: Pod> Deref for Column<T> {
type Target = [T];
fn deref(&self) -> &[T] {
match &self.backing {
Backing::Ram(v) => v,
// SAFETY: maps are page-aligned, `T` is `Pod` and the length is a multiple of its width.
Backing::Map(m) => unsafe { std::slice::from_raw_parts(m.as_ptr().cast(), m.len() / size_of::<T>()) },
Backing::ReadOnly(m) => unsafe { std::slice::from_raw_parts(m.as_ptr().cast(), m.len() / size_of::<T>()) },
}
}
}
impl<T: Pod> DerefMut for Column<T> {
fn deref_mut(&mut self) -> &mut [T] {
match &mut self.backing {
Backing::Ram(v) => v,
// SAFETY: as in `deref`; `&mut self` makes the access exclusive.
Backing::Map(m) => unsafe {
std::slice::from_raw_parts_mut(m.as_mut_ptr().cast(), m.len() / size_of::<T>())
},
Backing::ReadOnly(_) => panic!("column opened from a file is read-only"),
}
}
}
impl<T: Pod> Drop for Column<T> {
fn drop(&mut self) {
if let Some(path) = self.delete_on_drop.take() {
// Unmap before deleting: Windows refuses to delete a mapped file.
self.backing = Backing::Ram(Vec::new());
let _ = std::fs::remove_file(path);
}
}
}
/// Where a computation puts its arrays: result columns are named files in `Dir`, temporary arrays
/// are scratch columns in the same directory.
pub enum Store {
Ram,
Dir(PathBuf),
}
impl Store {
/// `name` is the whole file name (`types.u32`); it is used as given.
pub fn column<T: Pod>(&self, name: &str, len: usize, fill: T) -> io::Result<Column<T>> {
match self {
Store::Ram => Ok(Column::ram(len, fill)),
Store::Dir(dir) => Column::file(&dir.join(name), len, fill),
}
}
pub fn scratch<T: Pod>(&self, len: usize, fill: T) -> io::Result<Column<T>> {
match self {
Store::Ram => Ok(Column::ram(len, fill)),
Store::Dir(dir) => Column::scratch(dir, len, fill),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn dir_is_empty(dir: &Path) -> bool {
std::fs::read_dir(dir).unwrap().next().is_none()
}
#[test]
fn file_column_round_trips() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("x.u32");
{
let mut col = Column::<u32>::file(&path, 4, 7).unwrap();
assert_eq!(&*col, &[7, 7, 7, 7]);
col.copy_from_slice(&[1, 2, 3, 4]);
}
let col = Column::<u32>::open(&path).unwrap();
assert_eq!(&*col, &[1, 2, 3, 4]);
}
#[test]
fn scratch_column_leaves_no_file() {
let dir = tempfile::tempdir().unwrap();
{
let mut col = Column::<u64>::scratch(dir.path(), 1024, 9).unwrap();
col[1023] = 5;
assert_eq!((col[0], col[1023]), (9, 5));
// Windows cannot delete a mapped file, so only unix unlinks while the map is alive.
#[cfg(unix)]
assert!(dir_is_empty(dir.path()));
}
assert!(dir_is_empty(dir.path()));
}
#[test]
fn zero_fill_and_non_zero_fill() {
let dir = tempfile::tempdir().unwrap();
let zeros = Column::<u64>::file(&dir.path().join("z.u64"), 3000, 0).unwrap();
assert!(zeros.iter().all(|&v| v == 0));
let ones = Column::<u8>::file(&dir.path().join("o.u8"), 10_000, 0xAB).unwrap();
assert!(ones.iter().all(|&v| v == 0xAB));
let ram = Column::<u32>::ram(5, NONE);
assert_eq!(&*ram, &[NONE; 5]);
}
#[test]
fn empty_columns_work_everywhere() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("e.u32");
assert_eq!(Column::<u32>::ram(0, 1).len(), 0);
assert_eq!(Column::<u32>::file(&path, 0, 1).unwrap().len(), 0);
assert_eq!(Column::<u32>::open(&path).unwrap().len(), 0);
assert_eq!(Column::<u32>::scratch(dir.path(), 0, 1).unwrap().len(), 0);
}
#[test]
fn open_rejects_a_length_that_is_not_a_multiple_of_the_width() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("odd.u32");
std::fs::write(&path, [0u8; 6]).unwrap();
let err = Column::<u32>::open(&path).err().unwrap();
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
}
#[test]
fn file_rejects_a_byte_length_that_overflows() {
let dir = tempfile::tempdir().unwrap();
let err = Column::<u64>::file(&dir.path().join("big.u64"), usize::MAX, 0).err().unwrap();
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
assert!(dir_is_empty(dir.path()));
}
#[test]
#[should_panic(expected = "read-only")]
fn opened_columns_are_read_only() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("r.u32");
drop(Column::<u32>::file(&path, 2, 0).unwrap());
let mut col = Column::<u32>::open(&path).unwrap();
col[0] = 1;
}
#[test]
fn store_names_files_exactly_as_given() {
let dir = tempfile::tempdir().unwrap();
let store = Store::Dir(dir.path().to_path_buf());
let mut col = store.column::<u32>("types.u32", 3, 0).unwrap();
col[2] = 8;
drop(col);
let names: Vec<_> =
std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
assert_eq!(names, ["types.u32"]);
assert_eq!(&*Column::<u32>::open(&dir.path().join("types.u32")).unwrap(), &[0, 0, 8]);
}
#[test]
fn store_scratch_is_anonymous_and_ram_store_touches_no_disk() {
let dir = tempfile::tempdir().unwrap();
let store = Store::Dir(dir.path().to_path_buf());
drop(store.scratch::<u32>(16, 1).unwrap());
assert!(dir_is_empty(dir.path()));
let ram = Store::Ram.column::<u8>("ignored.u8", 4, 2).unwrap();
assert_eq!(&*ram, &[2; 4]);
assert_eq!(&*Store::Ram.scratch::<u8>(2, 3).unwrap(), &[3, 3]);
}
}
+59
View File
@@ -0,0 +1,59 @@
//! Class-by-class differences between two snapshots.
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
use crate::index::HeapIndex;
/// One class name in two snapshots; a side without the name has zeros.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapClassDiff {
pub name: String,
pub base_count: u64,
pub count: u64,
pub base_shallow: u64,
pub shallow: u64,
pub base_retained: u64,
pub retained: u64,
/// Several classes have this name on one side or both, so the sides cannot be matched: each
/// side shows the sum over its classes, and there is no difference to show.
pub ambiguous: bool,
}
/// Joined by class name, the only key two dumps share. By the change in retained size
/// descending (none for an ambiguous name), then by name.
pub fn compare(base: &HeapIndex, current: &HeapIndex) -> Vec<HeapClassDiff> {
let mut by_name: HashMap<String, HeapClassDiff> = HashMap::new();
for (index, is_base) in [(base, true), (current, false)] {
for row in index.class_rows() {
let diff = by_name.entry(row.name).or_insert_with_key(|name| HeapClassDiff {
name: name.clone(),
base_count: 0,
count: 0,
base_shallow: 0,
shallow: 0,
base_retained: 0,
retained: 0,
ambiguous: false,
});
let (count, shallow, retained) = if is_base {
(&mut diff.base_count, &mut diff.base_shallow, &mut diff.base_retained)
} else {
(&mut diff.count, &mut diff.shallow, &mut diff.retained)
};
*count = count.saturating_add(row.count);
*shallow = shallow.saturating_add(row.shallow);
*retained = retained.saturating_add(row.retained);
diff.ambiguous |= row.ambiguous;
}
}
let change = |d: &HeapClassDiff| if d.ambiguous { 0 } else { d.retained.abs_diff(d.base_retained) };
let mut diffs: Vec<HeapClassDiff> = by_name.into_values().collect();
diffs.sort_by(|a, b| change(b).cmp(&change(a)).then_with(|| a.name.cmp(&b.name)));
diffs
}
#[cfg(test)]
mod tests;
+78
View File
@@ -0,0 +1,78 @@
use super::*;
use crate::hprof::HeapRootKind;
use crate::test_index::analyzed;
use crate::testing::{MiniHeap, sample, sample_with_two_more_nodes};
fn diff<'a>(diffs: &'a [HeapClassDiff], name: &str) -> &'a HeapClassDiff {
let mut rows = diffs.iter().filter(|d| d.name == name);
let row = rows.next().unwrap_or_else(|| panic!("no diff for {name}"));
assert!(rows.next().is_none(), "two diffs for {name}");
row
}
fn sort_key(d: &HeapClassDiff) -> u64 {
if d.ambiguous { 0 } else { d.retained.abs_diff(d.base_retained) }
}
fn assert_sorted(diffs: &[HeapClassDiff]) {
assert!(diffs.windows(2).all(|p| (sort_key(&p[1]), &p[0].name) < (sort_key(&p[0]), &p[1].name)), "{diffs:#?}");
}
#[test]
fn compare_counts_by_class() {
let base = analyzed(&sample());
let mut current = sample_with_two_more_nodes();
current
.class("Extra", Some("java.lang.Object"), &[])
.object(0x4000, "Extra", &[])
.root(HeapRootKind::JniGlobal, 0x4000);
let diffs = compare(&base, &analyzed(&current));
let node = diff(&diffs, "Node");
assert_eq!((node.base_count, node.count), (3, 5));
assert_eq!(node.shallow - node.base_shallow, 48);
assert_eq!((node.base_retained, node.retained), (104, 152));
assert!(!node.ambiguous);
let extra = diff(&diffs, "Extra");
assert_eq!((extra.base_count, extra.base_shallow, extra.base_retained, extra.count), (0, 0, 0, 1));
assert_eq!(diffs[0].name, "Node");
let holder = diff(&diffs, "Holder");
assert_eq!((holder.base_count, holder.count, holder.base_retained, holder.retained), (1, 1, 72, 72));
assert_sorted(&diffs);
// A class only in the base snapshot.
let gone = compare(&analyzed(&current), &base);
let extra = diff(&gone, "Extra");
assert_eq!((extra.base_count, extra.count, extra.shallow, extra.retained), (1, 0, 0, 0));
assert_sorted(&gone);
}
fn two_foos(extra: bool) -> MiniHeap {
let mut heap = sample();
heap.class("Foo", Some("java.lang.Object"), &[])
.object(0x3000, "Foo", &[])
.class("Foo", Some("java.lang.Object"), &[])
.object(0x3100, "Foo", &[]);
if extra {
heap.object(0x3200, "Foo", &[]);
}
for addr in [0x3000, 0x3100, 0x3200] {
heap.root(HeapRootKind::JniGlobal, addr);
}
heap
}
#[test]
fn compare_gives_no_diff_for_a_name_of_several_classes() {
let diffs = compare(&analyzed(&two_foos(false)), &analyzed(&two_foos(true)));
let foo = diff(&diffs, "Foo");
assert!(foo.ambiguous);
assert_eq!((foo.base_count, foo.count), (2, 3));
assert!(diffs.iter().filter(|d| d.name != "Foo").all(|d| !d.ambiguous));
assert_sorted(&diffs);
// Ambiguous on one side only is still ambiguous.
let diffs = compare(&analyzed(&sample()), &analyzed(&two_foos(false)));
let foo = diff(&diffs, "Foo");
assert!(foo.ambiguous);
assert_eq!((foo.base_count, foo.count), (0, 2));
}
+137
View File
@@ -0,0 +1,137 @@
//! Disk-space estimates and the per-snapshot directories under the app's cache.
//!
//! A snapshot directory holds the index columns and, for a capture, the dump itself. Its `owner`
//! file names the host process that created it, so a host that starts later can remove the
//! directories of hosts that died without cleaning up.
use std::fs;
use std::io::{self, Read};
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime};
/// Space to leave free on top of an estimate.
pub const DISK_MARGIN: u64 = 1 << 30;
/// hprof ≈ `bytes + 16 × instances`; index columns and scratch arrays ≈ `90 × instances`.
const CAPTURE_BYTES_PER_INSTANCE: u64 = 106;
const INDEX_BYTES_PER_OBJECT: u64 = 90;
const OWNER_FILE: &str = "owner";
const TMP_SUFFIX: &str = ".tmp";
/// A directory still being created by a live host has no `owner` for a moment; one that stays
/// unfinished for longer than this was abandoned.
const TMP_GRACE: Duration = Duration::from_secs(60 * 60);
/// Bytes of free space that a dump of `heap_bytes` bytes holding `instances` instances needs,
/// including its index.
pub fn capture_need(heap_bytes: u64, instances: u64) -> u64 {
heap_bytes.saturating_add(CAPTURE_BYTES_PER_INSTANCE.saturating_mul(instances)).saturating_add(DISK_MARGIN)
}
/// Bytes of free space that indexing a dump of `objects` objects needs.
pub fn index_need(objects: u64) -> u64 {
INDEX_BYTES_PER_OBJECT.saturating_mul(objects).saturating_add(DISK_MARGIN)
}
/// Bytes available to this process on the volume holding `dir`; `None` where it is not checked
/// (not unix) or cannot be read, so callers go ahead rather than refuse.
pub fn free_bytes(dir: &Path) -> Option<u64> {
#[cfg(unix)]
{
use std::os::unix::ffi::OsStrExt;
let path = std::ffi::CString::new(dir.as_os_str().as_bytes()).ok()?;
// SAFETY: `path` is NUL-terminated and `stat` is only read after `statvfs` filled it in.
let stat = unsafe {
let mut stat = std::mem::MaybeUninit::<libc::statvfs>::zeroed();
if libc::statvfs(path.as_ptr(), stat.as_mut_ptr()) != 0 {
return None;
}
stat.assume_init()
};
// The field widths differ between unix targets.
#[allow(clippy::unnecessary_cast)]
Some((stat.f_bavail as u64).saturating_mul(stat.f_frsize as u64))
}
#[cfg(not(unix))]
{
let _ = dir;
None
}
}
/// `<local_data_dir>/cache/heap-snapshots`.
pub fn snapshots_root(local_data_dir: &Path) -> PathBuf {
local_data_dir.join("cache").join("heap-snapshots")
}
/// Creates `<root>/<uuid>/` holding an `owner` file with this process's pid. The directory is
/// built as `<uuid>.tmp` and renamed, so another host scanning `root` never sees it without its
/// owner and removes it as abandoned.
pub fn create_snapshot_dir(root: &Path) -> io::Result<PathBuf> {
fs::create_dir_all(root)?;
let name = uuid::Uuid::new_v4().to_string();
let building = root.join(format!("{name}{TMP_SUFFIX}"));
let dir = root.join(&name);
create_private_dir(&building)?;
let published =
fs::write(building.join(OWNER_FILE), std::process::id().to_string()).and_then(|()| fs::rename(&building, &dir));
if let Err(err) = published {
let _ = fs::remove_dir_all(&building);
return Err(err);
}
Ok(dir)
}
/// A dump holds whatever was in the JVM's memory, passwords and tokens included, so on unix only
/// the owner may enter the directory. Windows keeps the ACL inherited from the user's data dir.
fn create_private_dir(path: &Path) -> io::Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::DirBuilderExt;
fs::DirBuilder::new().mode(0o700).create(path)
}
#[cfg(not(unix))]
{
fs::DirBuilder::new().create(path)
}
}
/// Removes the directories under `root` whose owner file is missing, unreadable or names a pid
/// for which `alive` is false, and unfinished `*.tmp` ones older than an hour. Only directories
/// directly under `root` are touched; a symlink there is left alone.
pub fn remove_stale(root: &Path, alive: impl Fn(u32) -> bool) {
remove_stale_at(root, alive, SystemTime::now());
}
fn remove_stale_at(root: &Path, alive: impl Fn(u32) -> bool, now: SystemTime) {
let Ok(entries) = fs::read_dir(root) else { return };
for entry in entries.flatten() {
let path = entry.path();
// `symlink_metadata` does not follow: a link out of `root` is not a directory here.
let Ok(meta) = fs::symlink_metadata(&path) else { continue };
if !meta.file_type().is_dir() {
continue;
}
let stale = if path.extension().is_some_and(|ext| ext == "tmp") {
meta.modified()
.ok()
.and_then(|modified| now.duration_since(modified).ok())
.is_some_and(|age| age >= TMP_GRACE)
} else {
read_owner(&path).is_none_or(|pid| !alive(pid))
};
if stale && let Err(err) = fs::remove_dir_all(&path) {
tracing::warn!(dir = %path.display(), %err, "could not remove a stale heap snapshot directory");
}
}
}
fn read_owner(dir: &Path) -> Option<u32> {
let mut text = String::new();
fs::File::open(dir.join(OWNER_FILE)).ok()?.take(32).read_to_string(&mut text).ok()?;
text.trim().parse().ok()
}
#[cfg(test)]
mod tests;
+137
View File
@@ -0,0 +1,137 @@
use std::time::{Duration, SystemTime};
use super::*;
fn names(dir: &Path) -> Vec<String> {
let mut names: Vec<_> =
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
names
}
fn dir_with_owner(root: &Path, name: &str, owner: Option<&str>) {
let dir = root.join(name);
std::fs::create_dir(&dir).unwrap();
std::fs::write(dir.join("dump.hprof"), b"x").unwrap();
if let Some(owner) = owner {
std::fs::write(dir.join(OWNER_FILE), owner).unwrap();
}
}
#[test]
fn estimates() {
assert_eq!(capture_need(664_018_520, 20_087_506), 664_018_520 + 106 * 20_087_506 + (1 << 30));
assert_eq!(index_need(10), 900 + (1 << 30));
}
#[test]
fn estimates_saturate_instead_of_wrapping() {
assert_eq!(capture_need(u64::MAX, u64::MAX), u64::MAX);
assert_eq!(index_need(u64::MAX), u64::MAX);
}
#[cfg(unix)]
#[test]
fn not_enough_space_for_estimate() {
let dir = tempfile::tempdir().unwrap();
let free = free_bytes(dir.path()).expect("statvfs works on a temp dir");
assert!(free > 0);
assert!(capture_need(u64::MAX / 4, 0) > free);
}
#[cfg(unix)]
#[test]
fn free_bytes_of_a_missing_dir_is_unknown() {
let dir = tempfile::tempdir().unwrap();
assert_eq!(free_bytes(&dir.path().join("missing")), None);
}
#[cfg(not(unix))]
#[test]
fn free_space_is_not_checked_off_unix() {
assert_eq!(free_bytes(Path::new(".")), None);
}
#[test]
fn snapshots_root_is_under_the_cache() {
let local = Path::new("local");
assert_eq!(snapshots_root(local), local.join("cache").join("heap-snapshots"));
}
#[test]
fn snapshot_dir_records_its_owner_and_leaves_no_tmp() {
let root = tempfile::tempdir().unwrap();
let missing_parent = root.path().join("cache").join("heap-snapshots");
let dir = create_snapshot_dir(&missing_parent).unwrap();
assert_eq!(dir.parent(), Some(missing_parent.as_path()));
assert!(uuid::Uuid::parse_str(dir.file_name().unwrap().to_str().unwrap()).is_ok());
assert_eq!(std::fs::read_to_string(dir.join(OWNER_FILE)).unwrap(), std::process::id().to_string());
assert_eq!(names(&missing_parent), [dir.file_name().unwrap().to_str().unwrap()]);
assert_ne!(create_snapshot_dir(&missing_parent).unwrap(), dir);
}
#[cfg(unix)]
#[test]
fn snapshot_dir_is_private_to_the_owner() {
use std::os::unix::fs::PermissionsExt;
let root = tempfile::tempdir().unwrap();
let dir = create_snapshot_dir(root.path()).unwrap();
assert_eq!(std::fs::metadata(&dir).unwrap().permissions().mode() & 0o777, 0o700);
}
#[test]
fn stale_dirs_removed_on_start() {
let root = tempfile::tempdir().unwrap();
dir_with_owner(root.path(), "a", Some("999999"));
dir_with_owner(root.path(), "b", Some(&std::process::id().to_string()));
dir_with_owner(root.path(), "c", None);
remove_stale(root.path(), |pid| pid == std::process::id());
assert_eq!(names(root.path()), ["b"]);
}
#[test]
fn owner_that_is_not_a_pid_is_stale() {
let root = tempfile::tempdir().unwrap();
dir_with_owner(root.path(), "empty", Some(""));
dir_with_owner(root.path(), "word", Some("not-a-pid"));
dir_with_owner(root.path(), "negative", Some("-1"));
dir_with_owner(root.path(), "padded", Some(&format!(" {}\n", std::process::id())));
remove_stale(root.path(), |pid| pid == std::process::id());
assert_eq!(names(root.path()), ["padded"]);
}
#[test]
fn stale_scan_tolerates_a_missing_root_and_leaves_loose_files_alone() {
let root = tempfile::tempdir().unwrap();
remove_stale(&root.path().join("missing"), |_| false);
std::fs::write(root.path().join("notes.txt"), b"keep").unwrap();
remove_stale(root.path(), |_| false);
assert_eq!(names(root.path()), ["notes.txt"]);
}
#[test]
fn unfinished_dir_is_left_alone_for_an_hour() {
let root = tempfile::tempdir().unwrap();
// Another host may be between `create_dir` and the rename of its own snapshot dir.
dir_with_owner(root.path(), "young.tmp", None);
let now = SystemTime::now();
remove_stale_at(root.path(), |_| false, now);
assert_eq!(names(root.path()), ["young.tmp"]);
remove_stale_at(root.path(), |_| false, now + TMP_GRACE - Duration::from_secs(60));
assert_eq!(names(root.path()), ["young.tmp"]);
remove_stale_at(root.path(), |_| false, now + TMP_GRACE + Duration::from_secs(60));
assert!(names(root.path()).is_empty());
}
#[cfg(unix)]
#[test]
fn stale_scan_never_follows_a_symlink_out_of_the_root() {
let outside = tempfile::tempdir().unwrap();
std::fs::write(outside.path().join("precious"), b"x").unwrap();
let root = tempfile::tempdir().unwrap();
std::os::unix::fs::symlink(outside.path(), root.path().join("link")).unwrap();
remove_stale(root.path(), |_| false);
remove_stale_at(root.path(), |_| false, SystemTime::now() + 2 * TMP_GRACE);
assert_eq!(names(outside.path()), ["precious"]);
}
+247
View File
@@ -0,0 +1,247 @@
//! Dominator tree of the object graph, by Lengauer–Tarjan with path compression and no recursion:
//! the DFS "stack" is the chain of DFS parents, and `eval` reverses the links of the path it
//! compresses instead of keeping a stack.
//!
//! Between the DFS and the final translation every array is indexed by DFS number, so the root is
//! 0 and a vertex's DFS ancestors all have smaller numbers.
use ide_core::CancellationToken;
use crate::column::{Column, NONE, Store};
use crate::error::HeapError;
use crate::files::IDOM_FILE;
use crate::graph::accumulate;
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
const PHASE: HeapPhase = HeapPhase::Dominators;
pub struct Dominators {
/// Immediate dominator of each node, saved as `idom.u32` in the store; [`NONE`] for
/// unreachable nodes, `root` for `root`.
pub idom: Column<u32>,
/// Nodes in DFS preorder from `root`, `reachable` entries.
pub preorder: Column<u32>,
pub reachable: u32,
}
/// The dominators of the graph whose node `v` has targets `out[out_start[v]..out_start[v + 1]]`,
/// from `root` (which must be a node). `progress` gets `Dominators` from `(0, total)` to
/// `(total, total)`; an `Err` from it stops the computation with that error.
pub fn dominators(
out_start: &[u32],
out: &[u32],
root: u32,
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<Dominators, HeapError> {
compute(out_start, out, root, store, progress, cancel).map(|(dominators, _)| dominators)
}
/// [`dominators`], plus the number of steps of work it took.
fn compute(
out_start: &[u32],
out: &[u32],
root: u32,
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<(Dominators, u64), HeapError> {
let nodes = out_start.len().saturating_sub(1);
if nodes >= NONE as usize {
return Err(HeapError::TooLarge);
}
assert!((root as usize) < nodes, "root {root} is not one of the {nodes} nodes");
// Three walks of at most `nodes` vertices each: DFS, semidominators, immediate dominators.
let span = nodes as u64;
let mut work = Work::new(Progress::new(progress, cancel), 3 * span);
work.report(0)?;
// DFS: `pre` maps nodes to DFS numbers and `vertex` back; `cursor[d]` is the next edge of
// DFS vertex d to follow.
let mut pre = store.scratch::<u32>(nodes, NONE)?;
let mut vertex = store.scratch::<u32>(nodes, 0)?;
let mut parent = store.scratch::<u32>(nodes, 0)?;
let mut cursor = store.scratch::<u32>(nodes, 0)?;
pre[root as usize] = 0;
vertex[0] = root;
cursor[0] = out_start[root as usize];
let (mut reachable, mut d) = (1u32, 0u32);
loop {
work.tick()?;
let edge = cursor[d as usize];
if edge < out_start[vertex[d as usize] as usize + 1] {
cursor[d as usize] = edge + 1;
let w = out[edge as usize] as usize;
if pre[w] == NONE {
let next = reachable as usize;
pre[w] = reachable;
vertex[next] = w as u32;
parent[next] = d;
cursor[next] = out_start[w];
d = reachable;
reachable += 1;
work.done = u64::from(reachable);
}
} else if d == 0 {
break;
} else {
d = parent[d as usize];
}
}
let r = reachable as usize;
work.report(span)?;
// Predecessors by DFS number, as CSR: counted into `starts[w + 1]`, then filled through
// `starts[w]` as a cursor, which leaves it at w's end until the column is shifted one slot.
let mut semi = cursor;
let mut label = store.scratch::<u32>(r, 0)?;
let mut starts = store.scratch::<u32>(r + 1, 0)?;
for d in 0..r {
work.tick()?;
semi[d] = d as u32;
label[d] = d as u32;
for &w in targets(out_start, out, vertex[d]) {
work.tick()?;
starts[pre[w as usize] as usize + 1] += 1;
}
}
let edges = accumulate(&mut starts, cancel)?;
let mut preds = store.scratch::<u32>(edges as usize, 0)?;
for d in 0..r {
work.tick()?;
for &w in targets(out_start, out, vertex[d]) {
work.tick()?;
let slot = &mut starts[pre[w as usize] as usize];
preds[*slot as usize] = d as u32;
*slot += 1;
}
}
starts.copy_within(0..r, 1);
starts[0] = 0;
drop(pre);
work.report(span)?;
// Semidominators in reverse preorder, linking each finished vertex to its parent. Vertex w
// waits in its semidominator's bucket until a child of that semidominator is linked; then
// `eval` gives w's idom, or a vertex whose idom is also w's, fixed up in preorder below.
// `idom[v]` is the next vertex in v's bucket until v leaves the bucket.
let mut ancestor = store.scratch::<u32>(r, NONE)?;
let mut bucket = store.scratch::<u32>(r, NONE)?;
let mut idom = store.scratch::<u32>(r, NONE)?;
for w in (1..r).rev() {
work.done = span + (r - w) as u64;
work.tick()?;
let mut s = semi[w];
for &v in &preds[starts[w] as usize..starts[w + 1] as usize] {
work.tick()?;
let u = eval(v, &mut ancestor, &mut label, &semi, &mut work)?;
s = s.min(semi[u as usize]);
}
semi[w] = s;
idom[w] = bucket[s as usize];
bucket[s as usize] = w as u32;
let p = parent[w];
ancestor[w] = p;
let mut v = std::mem::replace(&mut bucket[p as usize], NONE);
while v != NONE {
work.tick()?;
let next = idom[v as usize];
let u = eval(v, &mut ancestor, &mut label, &semi, &mut work)?;
idom[v as usize] = if semi[u as usize] < semi[v as usize] { u } else { p };
v = next;
}
}
drop((label, starts, preds, ancestor, bucket, parent));
work.report(2 * span)?;
// Immediate dominators, translated back to nodes; in preorder, so a deferred vertex's
// stand-in already has its final idom.
let mut node_idom = store.column::<u32>(IDOM_FILE, nodes, NONE)?;
let mut preorder = store.scratch::<u32>(r, 0)?;
idom[0] = 0;
node_idom[root as usize] = root;
preorder[0] = root;
for w in 1..r {
work.done = 2 * span + w as u64;
work.tick()?;
if idom[w] != semi[w] {
idom[w] = idom[idom[w] as usize];
}
node_idom[vertex[w] as usize] = vertex[idom[w] as usize];
preorder[w] = vertex[w];
}
work.report(3 * span)?;
Ok((Dominators { idom: node_idom, preorder, reachable }, work.steps))
}
fn targets<'a>(out_start: &[u32], out: &'a [u32], v: u32) -> &'a [u32] {
let v = v as usize;
&out[out_start[v] as usize..out_start[v + 1] as usize]
}
/// Counts steps of work and reports progress, which also looks at the cancellation token, at
/// every phase boundary and after every [`CHECK_EVERY`] steps, whichever loop they are in.
struct Work<'a> {
progress: Progress<'a>,
total: u64,
/// Progress to report at the next check; the loops keep it current.
done: u64,
steps: u64,
next_report: u64,
}
impl<'a> Work<'a> {
fn new(progress: Progress<'a>, total: u64) -> Self {
Self { progress, total, done: 0, steps: 0, next_report: CHECK_EVERY }
}
fn report(&mut self, done: u64) -> Result<(), HeapError> {
self.done = done;
self.next_report = self.steps + CHECK_EVERY;
self.progress.report(PHASE, done, self.total)
}
#[inline]
fn tick(&mut self) -> Result<(), HeapError> {
self.steps += 1;
if self.steps < self.next_report { Ok(()) } else { self.report(self.done) }
}
}
/// The vertex of smallest semidominator on the forest path from `v` up to, not including, its
/// tree's root (`v` itself when it is a root), compressing that path so the root becomes the
/// ancestor of every vertex on it. The walk up reverses the `ancestor` links so the walk down can
/// fold labels nearest-root first, as the recursive compress does, without a stack.
fn eval(v: u32, ancestor: &mut [u32], label: &mut [u32], semi: &[u32], work: &mut Work) -> Result<u32, HeapError> {
let at = |x: u32| x as usize;
if ancestor[at(v)] == NONE {
return Ok(v);
}
let (mut x, mut below) = (v, NONE);
while ancestor[at(ancestor[at(x)])] != NONE {
work.tick()?;
let up = ancestor[at(x)];
ancestor[at(x)] = below;
below = x;
x = up;
}
// `x` is the root's child, already compressed; `below` is the vertex under it.
let root = ancestor[at(x)];
let mut best = label[at(x)];
while below != NONE {
work.tick()?;
let y = at(below);
if semi[at(best)] < semi[at(label[y])] {
label[y] = best;
} else {
best = label[y];
}
below = ancestor[y];
ancestor[y] = root;
}
Ok(label[at(v)])
}
#[cfg(test)]
mod tests;
+348
View File
@@ -0,0 +1,348 @@
use std::cell::RefCell;
use std::time::Instant;
use proptest::collection::vec;
use proptest::prelude::*;
use super::*;
use crate::NONE;
use crate::graph::build_graph;
use crate::testing::{XorShift, sample};
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
/// CSR of `edges` over `n` nodes, targets in edge order.
fn csr(n: usize, edges: &[(u32, u32)]) -> (Vec<u32>, Vec<u32>) {
let mut start = vec![0u32; n + 1];
for &(from, _) in edges {
start[from as usize + 1] += 1;
}
for v in 0..n {
start[v + 1] += start[v];
}
let mut cursor = start.clone();
let mut out = vec![0u32; edges.len()];
for &(from, to) in edges {
out[cursor[from as usize] as usize] = to;
cursor[from as usize] += 1;
}
(start, out)
}
fn run(n: usize, edges: &[(u32, u32)], root: u32) -> Dominators {
let (start, out) = csr(n, edges);
dominators(&start, &out, root, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap()
}
fn preds(n: usize, edges: &[(u32, u32)]) -> Vec<Vec<u32>> {
let mut preds = vec![Vec::new(); n];
for &(from, to) in edges {
preds[to as usize].push(from);
}
preds
}
/// Immediate dominators from iterated `Dom` sets (one `u64` bitset per node, `n <= 64`).
fn dataflow_idom(n: usize, edges: &[(u32, u32)], root: u32) -> Vec<u32> {
let (start, out) = csr(n, edges);
let mut reachable = vec![false; n];
let mut todo = vec![root];
reachable[root as usize] = true;
while let Some(v) = todo.pop() {
for &w in &out[start[v as usize] as usize..start[v as usize + 1] as usize] {
if !std::mem::replace(&mut reachable[w as usize], true) {
todo.push(w);
}
}
}
let preds = preds(n, edges);
let mut dom = vec![u64::MAX; n];
dom[root as usize] = 1 << root;
let mut changed = true;
while changed {
changed = false;
for v in (0..n).filter(|&v| reachable[v] && v != root as usize) {
let meet =
preds[v].iter().filter(|&&p| reachable[p as usize]).fold(u64::MAX, |acc, &p| acc & dom[p as usize]);
let next = meet | 1 << v;
if next != dom[v] {
dom[v] = next;
changed = true;
}
}
}
(0..n)
.map(|v| {
if !reachable[v] {
NONE
} else if v == root as usize {
root
} else {
// The strict dominator dominated by all the others has the most dominators itself.
let strict = dom[v] & !(1 << v);
(0..n as u32).filter(|&d| strict & 1 << d != 0).max_by_key(|&d| dom[d as usize].count_ones()).unwrap()
}
})
.collect()
}
/// Cooper–Harvey–Kennedy iteration over reverse postorder from an explicit-stack DFS.
fn chk_idom(n: usize, edges: &[(u32, u32)], root: u32) -> Vec<u32> {
let (start, out) = csr(n, edges);
let mut post_number = vec![NONE; n];
let mut postorder = Vec::new();
let mut seen = vec![false; n];
let mut stack = vec![(root, start[root as usize])];
seen[root as usize] = true;
while let Some(&mut (v, ref mut next)) = stack.last_mut() {
if *next < start[v as usize + 1] {
let w = out[*next as usize];
*next += 1;
if !std::mem::replace(&mut seen[w as usize], true) {
stack.push((w, start[w as usize]));
}
} else {
post_number[v as usize] = postorder.len() as u32;
postorder.push(v);
stack.pop();
}
}
let preds = preds(n, edges);
let mut idom = vec![NONE; n];
idom[root as usize] = root;
let intersect = |idom: &[u32], mut a: u32, mut b: u32| {
while a != b {
while post_number[a as usize] < post_number[b as usize] {
a = idom[a as usize];
}
while post_number[b as usize] < post_number[a as usize] {
b = idom[b as usize];
}
}
a
};
let mut changed = true;
while changed {
changed = false;
for &v in postorder.iter().rev().filter(|&&v| v != root) {
let mut new = NONE;
for &p in &preds[v as usize] {
if idom[p as usize] != NONE {
new = if new == NONE { p } else { intersect(&idom, p, new) };
}
}
if new != idom[v as usize] {
idom[v as usize] = new;
changed = true;
}
}
}
idom
}
#[test]
fn diamond() {
let d = run(5, &[(0, 1), (0, 2), (1, 3), (2, 3), (3, 4)], 0);
assert_eq!(&*d.idom, &[0, 0, 0, 0, 3]);
assert_eq!(d.reachable, 5);
assert_eq!(&*d.preorder, &[0, 1, 3, 4, 2]);
}
#[test]
fn unreachable_is_none() {
let d = run(6, &[(0, 1), (0, 2), (1, 3), (2, 3), (3, 4), (5, 3)], 0);
assert_eq!(&*d.idom, &[0, 0, 0, 0, 3, NONE]);
assert_eq!(d.reachable, 5);
assert_eq!(d.preorder.len(), 5);
}
#[test]
fn cycle_back_to_root() {
let d = run(3, &[(0, 1), (1, 2), (2, 1), (2, 0)], 0);
assert_eq!(&*d.idom, &[0, 0, 1]);
}
#[test]
fn root_need_not_be_node_zero() {
let d = run(4, &[(3, 1), (1, 0), (3, 2), (2, 0), (0, 3)], 3);
assert_eq!(&*d.idom, &[3, 3, 3, 3]);
assert_eq!(d.preorder[0], 3);
}
#[test]
fn sample_heap_dominators() {
let bytes = sample().bytes();
let g = build_graph(&bytes, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let d = dominators(&g.out_start, &g.out, g.root, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let node = |id: u64| g.ids.rank(id).unwrap();
let idom_of = |id: u64| d.idom[node(id) as usize];
assert_eq!(idom_of(0x1100), node(0x1000), "N1 → H");
assert_eq!(idom_of(0x1200), node(0x1000), "N2 → H");
assert_eq!(idom_of(0x1300), g.root, "N3 → root");
assert_eq!(idom_of(0x1400), node(0x1300), "L → N3");
assert_eq!(idom_of(0x1500), NONE, "R is unreachable");
assert_eq!(idom_of(0x1700), NONE, "N4 only through the weak referent");
assert_eq!(d.idom[g.root as usize], g.root);
}
#[test]
fn idom_is_written_to_the_store() {
let dir = tempfile::tempdir().unwrap();
let (start, out) = csr(3, &[(0, 1), (1, 2)]);
let store = Store::Dir(dir.path().to_path_buf());
let d = dominators(&start, &out, 0, &store, &no_progress, &CancellationToken::new()).unwrap();
drop(d);
assert_eq!(&*Column::<u32>::open(&dir.path().join("idom.u32")).unwrap(), &[0, 0, 1]);
let names: Vec<_> = std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name()).collect();
assert_eq!(names, ["idom.u32"], "scratch columns leave no files");
}
#[test]
fn reports_dominators_progress_and_stops_on_cancel() {
let (start, out) = csr(3, &[(0, 1), (1, 2)]);
let calls = RefCell::new(Vec::new());
let record = |phase, done, total| {
calls.borrow_mut().push((phase, done, total));
Ok(())
};
dominators(&start, &out, 0, &Store::Ram, &record, &CancellationToken::new()).unwrap();
let calls = calls.into_inner();
let total = calls[0].2;
assert_eq!(calls.first(), Some(&(HeapPhase::Dominators, 0, total)));
assert_eq!(calls.last(), Some(&(HeapPhase::Dominators, total, total)));
assert!(calls.windows(2).all(|w| w[0].1 <= w[1].1), "progress never goes back: {calls:?}");
let cancel = CancellationToken::new();
let cancel_on_first = |_, _, _| {
cancel.cancel();
Ok(())
};
let result = dominators(&start, &out, 0, &Store::Ram, &cancel_on_first, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)));
let refuse = |_, _, _| Err(HeapError::TooLarge);
let result = dominators(&start, &out, 0, &Store::Ram, &refuse, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::TooLarge)));
}
#[test]
fn cancel_during_idom_phase() {
// Long enough that the last phase alone passes a check interval.
const NODES: u32 = (CHECK_EVERY + CHECK_EVERY / 2) as u32;
let edges: Vec<_> = (1..NODES).map(|v| (v - 1, v)).collect();
let (start, out) = csr(NODES as usize, &edges);
let span = u64::from(NODES);
let cancel = CancellationToken::new();
let calls = RefCell::new(Vec::new());
let cancel_past_semidominators = |_, done, _| {
calls.borrow_mut().push(done);
if done > 2 * span {
cancel.cancel();
}
Ok(())
};
let result = dominators(&start, &out, 0, &Store::Ram, &cancel_past_semidominators, &cancel);
assert!(matches!(result, Err(HeapError::Cancelled)));
let calls = calls.into_inner();
assert!(calls.contains(&(2 * span)), "the semidominator phase finished: {calls:?}");
let last = *calls.last().unwrap();
assert!(last > 2 * span && last < 3 * span, "stopped inside the idom phase: {calls:?}");
}
proptest! {
#[test]
fn matches_dataflow_on_random_graphs(n in 2usize..64, edges in vec((0u32..64, 0u32..64), 0..192)) {
let edges: Vec<_> = edges.into_iter().map(|(a, b)| (a % n as u32, b % n as u32)).collect();
let d = run(n, &edges, 0);
prop_assert_eq!(d.idom.to_vec(), dataflow_idom(n, &edges, 0));
prop_assert_eq!(d.preorder.len(), d.reachable as usize);
}
}
#[test]
fn matches_chk_on_a_large_random_graph() {
const NODES: u32 = 200_000;
const EXTRA_EDGES: u32 = 100_000;
let mut rng = XorShift::new(0x5EED_D0D0);
let mut edges: Vec<_> = (1..NODES).map(|v| (rng.below(v), v)).collect();
edges.extend((0..EXTRA_EDGES).map(|_| (rng.below(NODES), rng.below(NODES))));
// Visit edges in a scrambled order so the DFS tree differs from the generating tree.
for i in (1..edges.len()).rev() {
edges.swap(i, rng.below(i as u32 + 1) as usize);
}
let started = Instant::now();
let d = run(NODES as usize, &edges, 0);
let elapsed = started.elapsed();
let expected = chk_idom(NODES as usize, &edges, 0);
let mismatches = d.idom.iter().zip(&expected).filter(|(a, b)| a != b).count();
assert_eq!(mismatches, 0);
assert_eq!(d.reachable, NODES);
eprintln!("dominators of {NODES} random nodes: {elapsed:?}");
}
/// A doubly linked list whose items an array list also holds: super-root 0 → list 1 → node 0,
/// 0 → array list 2 → element array 3 → every item; node j → item j, next, prev.
fn shared_list_and_array(k: u32) -> (usize, Vec<(u32, u32)>) {
let node = |j: u32| 4 + 2 * j;
let item = |j: u32| 5 + 2 * j;
let mut edges = vec![(0, 1), (0, 2), (2, 3), (1, node(0))];
for j in 0..k {
edges.push((node(j), item(j)));
if j + 1 < k {
edges.push((node(j), node(j + 1)));
}
if j > 0 {
edges.push((node(j), node(j - 1)));
}
edges.push((3, item(j)));
}
((4 + 2 * k) as usize, edges)
}
#[test]
fn shared_list_and_array_is_near_linear() {
let (n, edges) = shared_list_and_array(29);
assert_eq!(run(n, &edges, 0).idom.to_vec(), dataflow_idom(n, &edges, 0));
let (n, edges) = shared_list_and_array(3_000);
assert_eq!(run(n, &edges, 0).idom.to_vec(), chk_idom(n, &edges, 0));
const K: u32 = 200_000;
let (n, edges) = shared_list_and_array(K);
let (start, out) = csr(n, &edges);
let started = Instant::now();
let (d, steps) = compute(&start, &out, 0, &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let elapsed = started.elapsed();
let item = |j: u32| (5 + 2 * j) as usize;
assert!((0..K).all(|j| d.idom[item(j)] == 0), "every item is held by both collections");
let m = edges.len() as u64;
let bound = (n as u64 + m) * u64::from(n.ilog2());
eprintln!("shared list k={K}: {steps} steps (n={n}, m={m}, bound {bound}), {elapsed:?}");
assert!(steps <= bound, "{steps} steps for n={n}, m={m}");
}
#[test]
fn deep_chain_has_linear_dominators() {
// A small stack proves the algorithm does not recurse along the chain.
std::thread::Builder::new()
.stack_size(256 << 10)
.spawn(|| {
const NODES: u32 = 1_000_000;
let mut edges: Vec<_> = (1..NODES).map(|v| (v - 1, v)).collect();
edges.push((0, NODES - 1));
// Makes `eval` compress the whole chain at once.
edges.push((NODES - 1, 1));
let started = Instant::now();
let d = run(NODES as usize, &edges, 0);
eprintln!("dominators of a {NODES}-node chain: {:?}", started.elapsed());
assert_eq!(d.reachable, NODES);
assert_eq!(d.idom[0], 0);
assert_eq!(d.idom[NODES as usize - 1], 0);
let wrong = (1..NODES - 1).filter(|&v| d.idom[v as usize] != v - 1).count();
assert_eq!(wrong, 0);
})
.unwrap()
.join()
.unwrap();
}
+26
View File
@@ -0,0 +1,26 @@
/// Offsets (`at`) are byte positions in the `.hprof` file.
#[derive(Debug, thiserror::Error)]
pub enum HeapError {
#[error("the dump ends early at byte {at}")]
Truncated { at: u64 },
#[error("not a heap dump (no JAVA PROFILE header)")]
NotHprof,
#[error("unsupported identifier size {0}")]
UnsupportedIdSize(u32),
#[error("unknown heap sub-record tag {tag:#04x} at byte {at}")]
UnknownSubRecord { tag: u8, at: u64 },
#[error("unknown value type {ty} at byte {at}")]
UnknownType { ty: u8, at: u64 },
#[error("class {id:#x} used at byte {at} is not in the dump")]
MissingClass { id: u64, at: u64 },
#[error("the heap has too many objects to analyze")]
TooLarge,
#[error("the dump's class and field names are too long; it is not a dump a JVM wrote")]
NamesTooLarge,
#[error("not enough disk space: {needed} bytes needed, {available} free")]
NotEnoughDiskSpace { needed: u64, available: u64 },
#[error("cancelled")]
Cancelled,
#[error("{0}")]
Io(#[from] std::io::Error),
}
+34
View File
@@ -0,0 +1,34 @@
//! Names of the files an analysis writes into its index directory. They live in one list so a
//! failed analysis removes exactly these and never the dump or anything else sharing the directory.
pub(crate) const IDS_BITS_FILE: &str = "ids.bits.u64";
pub(crate) const IDS_WRANK_FILE: &str = "ids.wrank.u32";
pub(crate) const IDS_TABLE_FILE: &str = "ids.table.u32";
pub(crate) const IDS_CHUNKS_FILE: &str = "ids.chunks.u32";
pub(crate) const IDS_SORTED_FILE: &str = "ids.sorted.u64";
pub(crate) const TYPES_FILE: &str = "types.u32";
pub(crate) const OFFSETS_FILE: &str = "offsets.u64";
pub(crate) const IDOM_FILE: &str = "idom.u32";
pub(crate) const SHALLOW8_FILE: &str = "shallow8.u32";
pub(crate) const RETAINED_FILE: &str = "retained.u64";
pub(crate) const KIDS_START_FILE: &str = "kids_start.u32";
pub(crate) const KIDS_FILE: &str = "kids.u32";
pub(crate) const GC_PARENT_FILE: &str = "gc_parent.u32";
pub(crate) const SUMMARY_FILE: &str = "summary.json";
pub(crate) const INDEX_FILES: &[&str] = &[
IDS_BITS_FILE,
IDS_WRANK_FILE,
IDS_TABLE_FILE,
IDS_CHUNKS_FILE,
IDS_SORTED_FILE,
TYPES_FILE,
OFFSETS_FILE,
IDOM_FILE,
SHALLOW8_FILE,
RETAINED_FILE,
KIDS_START_FILE,
KIDS_FILE,
GC_PARENT_FILE,
SUMMARY_FILE,
];
+274
View File
@@ -0,0 +1,274 @@
//! The object graph of a heap dump in CSR form: one node per object, plus a super-root.
mod classes;
mod link;
use std::collections::hash_map::Entry;
use std::collections::{HashMap, HashSet};
use ide_core::CancellationToken;
use self::classes::{ClassTable, IdHash};
use self::link::Linker;
use crate::column::{Column, Store};
use crate::error::HeapError;
use crate::files::{OFFSETS_FILE, TYPES_FILE};
use crate::hprof::{BasicType, HeapRecord, HeapRootKind, Record, parse_header, scan, scan_segment};
use crate::ids::{IdMap, IdMapBuilder};
use crate::progress::{CHECK_EVERY, HeapPhase, Progress};
/// Number of primitive array type slots, one per non-object [`BasicType`].
pub const PRIM_SLOTS: u32 = 8;
pub struct ClassInfo {
pub id: u64,
/// The hprof name with `/` turned into `.` (`java.lang.String`, `[Ljava.lang.Object;`).
pub name: String,
/// Index in [`HeapGraph::classes`]. Chains always end: a super cycle in the dump is cut where it closes.
pub super_index: Option<u32>,
/// This class's own instance fields, in dump order.
pub fields: Vec<(String, BasicType)>,
/// `(name, object id)` of reference statics; a null static has object id 0.
pub statics: Vec<(String, u64)>,
pub static_bytes: u64,
/// Index in `fields` of `java.lang.ref.Reference.referent`, on that class only.
pub weak_field: Option<usize>,
}
pub struct HeapGraph {
pub ids: IdMap,
/// Type slot of each object: class index for instances and object arrays,
/// [`Self::class_slot`] for class objects, [`Self::prim_slot`] for primitive arrays.
pub types: Column<u32>,
/// Array length, 0 for instances and classes.
pub lens: Column<u32>,
/// File offset of each object's record, for [`crate::record_at`].
pub offsets: Column<u64>,
/// `n + 2` entries: node `v`'s targets are `out[out_start[v]..out_start[v + 1]]`.
pub out_start: Column<u32>,
pub out: Column<u32>,
/// The super-root, node `n`, with an edge to every rooted object.
pub root: u32,
/// `(object, kind, thread serial)` of every root record whose object is in the dump.
pub roots: Vec<(u32, HeapRootKind, Option<u32>)>,
/// Class dumps in dump order, then a synthetic `java.lang.Class` when the dump has none.
pub classes: Vec<ClassInfo>,
/// Type slot of class objects (`java.lang.Class`).
pub class_slot: u32,
/// Thread serial → thread object, from `ROOT THREAD OBJECT` records.
pub thread_objects: HashMap<u32, u32>,
/// `Reference.referent` edges left out of the graph.
pub weak_edges_skipped: u64,
pub id_size: u8,
pub timestamp_ms: u64,
}
impl HeapGraph {
pub fn prim_slot(&self, ty: BasicType) -> u32 {
self.classes.len() as u32 + (ty as u32 - BasicType::Boolean as u32)
}
/// The element type of primitive array slot `slot`, or `None` for a class slot.
pub fn prim_type(&self, slot: u32) -> Option<BasicType> {
let offset = slot.checked_sub(self.classes.len() as u32).filter(|&o| o < PRIM_SLOTS)?;
BasicType::from_u8(offset as u8 + BasicType::Boolean as u8)
}
pub fn out_edges(&self, v: u32) -> &[u32] {
let v = v as usize;
&self.out[self.out_start[v] as usize..self.out_start[v + 1] as usize]
}
}
/// Builds the graph of the dump in `buf`. Strong edges only: `Reference.referent` is counted in
/// [`HeapGraph::weak_edges_skipped`] instead. `progress` gets `Reading` by bytes, then
/// `Linking` by objects, starting with `(Linking, 0, objects)` once the object count is known;
/// an `Err` from it stops the build with that error.
pub fn build_graph(
buf: &[u8],
store: &Store,
progress: &dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &CancellationToken,
) -> Result<HeapGraph, HeapError> {
let progress = Progress::new(progress, cancel);
let header = parse_header(buf)?;
let id_size = header.id_size;
let len = buf.len() as u64;
// Pass 1: names, classes, roots and every object id.
progress.report(HeapPhase::Reading, 0, len)?;
let mut names: IdHash<&[u8]> = IdHash::default();
let mut class_names: IdHash<u64> = IdHash::default();
let mut dumps = Vec::new();
let mut class_by_id: IdHash<u32> = IdHash::default();
let mut raw_roots = Vec::new();
let mut segments = Vec::new();
let mut builder = IdMapBuilder::new();
let (mut step, mut pos) = (0, header.body_start as u64);
scan(buf, &header, &mut |record| {
progress.step(step, HeapPhase::Reading, pos, len)?;
step += 1;
match record {
Record::Utf8 { id, text } => {
names.insert(id, text);
}
Record::LoadClass { class_id, name_id } => {
class_names.insert(class_id, name_id);
}
Record::Segment(range) => {
pos = range.start as u64;
segments.push(range);
}
Record::Heap { at, record } => {
pos = at as u64;
match record {
HeapRecord::Root { kind, id, thread_serial } => raw_roots.push((id, kind, thread_serial)),
HeapRecord::Class(dump) => {
builder.insert(dump.id);
if let Entry::Vacant(slot) = class_by_id.entry(dump.id) {
slot.insert(dumps.len() as u32);
dumps.push(dump);
}
}
HeapRecord::Instance { id, .. }
| HeapRecord::ObjectArray { id, .. }
| HeapRecord::PrimitiveArray { id, .. } => builder.insert(id),
}
}
}
Ok(())
})?;
progress.report(HeapPhase::Reading, len, len)?;
let ids = builder.build(store, cancel)?;
let table = classes::resolve(dumps, class_by_id, &names, &class_names, id_size, buf.len())?;
drop((names, class_names));
let n = ids.len();
let objects = u64::from(n);
progress.report(HeapPhase::Linking, 0, objects)?;
let mut roots = Vec::new();
let mut root_targets = Vec::new();
let mut thread_objects = HashMap::new();
let (mut seen_records, mut seen_targets) = (HashSet::new(), HashSet::new());
for (i, &(id, kind, serial)) in raw_roots.iter().enumerate() {
progress.step(i as u64, HeapPhase::Linking, 0, objects)?;
let Some(object) = ids.rank(id).filter(|_| id != 0) else { continue };
if seen_records.insert((object, kind as u8, serial)) {
roots.push((object, kind, serial));
}
if seen_targets.insert(object) {
root_targets.push(object);
}
if let (HeapRootKind::ThreadObject, Some(serial)) = (kind, serial) {
thread_objects.entry(serial).or_insert(object);
}
}
drop((raw_roots, seen_records, seen_targets));
let linker = Linker::new(&ids, &table, id_size);
let nodes = n as usize;
let mut types = store.column::<u32>(TYPES_FILE, nodes, 0)?;
let mut lens = store.scratch::<u32>(nodes, 0)?;
let mut offsets = store.column::<u64>(OFFSETS_FILE, nodes, 0)?;
let mut out_start = store.scratch::<u32>(nodes + 2, 0)?;
// Both passes visit every object record once; report them as one walk over the objects.
let mut visited = 0u64;
let tick = |visited: &mut u64| {
let result = progress.step(*visited, HeapPhase::Linking, (*visited / 2).min(objects), objects);
*visited += 1;
result
};
// Pass 2: node columns and out-degrees, counted into `out_start[v + 1]`.
let mut weak_edges_skipped = 0;
for segment in &segments {
scan_segment(buf, segment.clone(), id_size, &mut |at, record| {
let Some(node) = linker.node(at, &record)? else { return Ok(()) };
tick(&mut visited)?;
let v = node.index as usize;
types[v] = node.ty;
lens[v] = node.len;
offsets[v] = at as u64;
let mut degree = 0u64;
linker.edges(&record, &mut |_, weak| {
if weak {
weak_edges_skipped += 1;
} else {
degree += 1;
}
});
let count = &mut out_start[v + 1];
*count = u32::try_from(u64::from(*count) + degree).map_err(|_| HeapError::TooLarge)?;
Ok(())
})?;
}
out_start[nodes + 1] = u32::try_from(root_targets.len()).map_err(|_| HeapError::TooLarge)?;
let edges = accumulate(&mut out_start, cancel)?;
// Pass 3: `out_start[v]` is node v's write cursor; once filled it holds v's end, so the
// column is shifted one slot right to become the starts again.
let mut out = store.scratch::<u32>(edges as usize, 0)?;
for segment in &segments {
scan_segment(buf, segment.clone(), id_size, &mut |at, record| {
let Some(node) = linker.node(at, &record)? else { return Ok(()) };
tick(&mut visited)?;
let v = node.index as usize;
let mut cursor = out_start[v] as usize;
linker.edges(&record, &mut |target, weak| {
if !weak && let Some(slot) = out.get_mut(cursor) {
*slot = target;
cursor += 1;
}
});
out_start[v] = cursor as u32;
Ok(())
})?;
}
let first_root_edge = out_start[nodes] as usize;
for (slot, &target) in out[first_root_edge..].iter_mut().zip(&root_targets) {
*slot = target;
}
out_start[nodes] = edges;
out_start.copy_within(0..nodes + 1, 1);
out_start[0] = 0;
progress.report(HeapPhase::Linking, objects, objects)?;
drop(linker);
let ClassTable { classes, class_slot, .. } = table;
Ok(HeapGraph {
ids,
types,
lens,
offsets,
out_start,
out,
root: n,
roots,
classes,
class_slot,
thread_objects,
weak_edges_skipped,
id_size,
timestamp_ms: header.timestamp_ms,
})
}
/// Turns per-node counts into running totals in place (`counts[k]` becomes the sum of
/// `counts[..=k]`) and returns the last one; past `u32::MAX` the graph is `TooLarge`.
pub(crate) fn accumulate(counts: &mut [u32], cancel: &CancellationToken) -> Result<u32, HeapError> {
let mut sum = 0u32;
for chunk in counts.chunks_mut(CHECK_EVERY as usize) {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
for count in chunk {
sum = sum.checked_add(*count).ok_or(HeapError::TooLarge)?;
*count = sum;
}
}
Ok(sum)
}
#[cfg(test)]
mod tests;
+188
View File
@@ -0,0 +1,188 @@
//! Class dumps resolved into [`ClassInfo`]s, plus where the references sit in instance data.
use std::collections::HashMap;
use std::hash::BuildHasherDefault;
use super::{ClassInfo, PRIM_SLOTS};
use crate::column::NONE;
use crate::error::HeapError;
use crate::hprof::{BasicType, ClassDump};
use crate::ids::ChunkHasher;
/// Keyed by hprof id (an address).
pub(super) type IdHash<V> = HashMap<u64, V, BuildHasherDefault<ChunkHasher>>;
const REFERENCE: &str = "java.lang.ref.Reference";
const JAVA_LANG_CLASS: &str = "java.lang.Class";
/// Not computed yet, in [`ClassTable::fielded`].
const UNSET: u32 = NONE - 1;
pub(super) struct ClassTable {
pub(super) classes: Vec<ClassInfo>,
/// Class id → class index.
pub(super) by_id: IdHash<u32>,
pub(super) class_slot: u32,
/// Per class: `(offset, is the referent)` of each reference field in the class's own part of
/// the instance data.
pub(super) refs: Vec<Vec<(usize, bool)>>,
/// Per class: bytes of its own part of the instance data.
pub(super) own_bytes: Vec<usize>,
/// Per class: the nearest class up its chain (itself included) whose own part is not empty, or
/// `NONE`. Walking these instead of every super keeps a walk bounded by the data it reads.
pub(super) fielded: Vec<u32>,
}
/// The longest name a JVM gives a class or field (a constant pool UTF8 has a `u2` length).
const MAX_NAME_BYTES: usize = 65_535;
/// Name bytes a dump may decode beyond 4 per byte of the file.
const NAME_BUDGET_FLOOR: usize = 64 << 20;
/// Names decoded from the UTF8 records within a budget: every field of every class may name the
/// same record, so a small crafted file could otherwise ask for terabytes and abort the IDE.
struct Names<'m, 'b> {
records: &'m IdHash<&'b [u8]>,
left: usize,
}
impl Names<'_, '_> {
/// Empty for an id with no record.
fn text(&mut self, id: u64) -> Result<String, HeapError> {
let Some(bytes) = self.records.get(&id) else { return Ok(String::new()) };
if bytes.len() > MAX_NAME_BYTES || bytes.len() > self.left {
return Err(HeapError::NamesTooLarge);
}
self.left -= bytes.len();
Ok(String::from_utf8_lossy(bytes).into_owned())
}
}
/// `dumps` holds one dump per class id, and `by_id` its position; `names` are the UTF8 records
/// and `class_names` the LOAD CLASS class id → name id pairs. `file_len` bounds the name bytes.
pub(super) fn resolve(
dumps: Vec<ClassDump>,
by_id: IdHash<u32>,
names: &IdHash<&[u8]>,
class_names: &IdHash<u64>,
id_size: u8,
file_len: usize,
) -> Result<ClassTable, HeapError> {
// Room for the synthetic java.lang.Class and the primitive slots in a `u32` type id.
if dumps.len() > (u32::MAX - PRIM_SLOTS - 1) as usize {
return Err(HeapError::TooLarge);
}
let mut names = Names { records: names, left: file_len.saturating_mul(4).saturating_add(NAME_BUDGET_FLOOR) };
let mut supers: Vec<Option<u32>> =
dumps.iter().map(|d| if d.super_id == 0 { None } else { by_id.get(&d.super_id).copied() }).collect();
cut_super_cycles(&mut supers);
let mut classes = Vec::with_capacity(dumps.len() + 1);
let mut refs = Vec::with_capacity(dumps.len() + 1);
let mut own_bytes = Vec::with_capacity(dumps.len() + 1);
for (dump, super_index) in dumps.into_iter().zip(supers) {
let name = match class_names.get(&dump.id) {
Some(&name_id) if names.records.contains_key(&name_id) => names.text(name_id)?.replace('/', "."),
_ => format!("{:#x}", dump.id),
};
let fields =
dump.fields.iter().map(|&(n, ty)| Ok((names.text(n)?, ty))).collect::<Result<Vec<_>, HeapError>>()?;
let weak_field = if name == REFERENCE {
fields.iter().position(|(n, ty)| n == "referent" && *ty == BasicType::Object)
} else {
None
};
let mut offset = 0;
let mut class_refs = Vec::new();
for (k, &(_, ty)) in fields.iter().enumerate() {
if ty == BasicType::Object {
class_refs.push((offset, weak_field == Some(k)));
}
offset += ty.hprof_size(id_size);
}
refs.push(class_refs);
own_bytes.push(offset);
classes.push(ClassInfo {
id: dump.id,
name,
super_index,
fields,
statics: dump
.statics
.iter()
.map(|&(n, object)| Ok((names.text(n)?, object)))
.collect::<Result<_, HeapError>>()?,
static_bytes: dump.static_bytes,
weak_field,
});
}
let class_slot = match classes.iter().position(|c| c.name == JAVA_LANG_CLASS) {
Some(slot) => slot as u32,
None => {
classes.push(ClassInfo {
id: 0,
name: JAVA_LANG_CLASS.to_owned(),
super_index: None,
fields: Vec::new(),
statics: Vec::new(),
static_bytes: 0,
weak_field: None,
});
refs.push(Vec::new());
own_bytes.push(0);
classes.len() as u32 - 1
}
};
let fielded = nearest_fielded(&classes, &own_bytes);
Ok(ClassTable { classes, by_id, class_slot, refs, own_bytes, fielded })
}
/// A dump may claim `A extends B extends A`; the edge that closes a cycle is dropped so every
/// walk up the chain ends. Iterative, so a long chain cannot overflow the stack.
fn cut_super_cycles(supers: &mut [Option<u32>]) {
const NEW: u8 = 0;
const ON_PATH: u8 = 1;
const DONE: u8 = 2;
let mut state = vec![NEW; supers.len()];
let mut path = Vec::new();
for start in 0..supers.len() {
let mut c = start;
while state[c] == NEW {
state[c] = ON_PATH;
path.push(c);
match supers[c] {
Some(s) if state[s as usize] == ON_PATH => {
supers[c] = None;
break;
}
Some(s) => c = s as usize,
None => break,
}
}
for p in path.drain(..) {
state[p] = DONE;
}
}
}
fn nearest_fielded(classes: &[ClassInfo], own_bytes: &[usize]) -> Vec<u32> {
let mut fielded: Vec<u32> =
own_bytes.iter().enumerate().map(|(c, &b)| if b > 0 { c as u32 } else { UNSET }).collect();
let mut path = Vec::new();
for start in 0..classes.len() {
let mut c = start;
let found = loop {
if fielded[c] != UNSET {
break fielded[c];
}
path.push(c);
match classes[c].super_index {
Some(s) => c = s as usize,
None => break NONE,
}
};
for p in path.drain(..) {
fielded[p] = found;
}
}
fielded
}
+107
View File
@@ -0,0 +1,107 @@
//! Turns one heap sub-record into its node (index, type, length) and its outgoing edges. Both
//! linking passes go through here, so the degrees counted in the first match the edges written
//! in the second.
use super::classes::ClassTable;
use crate::column::NONE;
use crate::error::HeapError;
use crate::hprof::{BasicType, HeapRecord};
use crate::ids::IdMap;
pub(super) struct Node {
pub(super) index: u32,
pub(super) ty: u32,
pub(super) len: u32,
}
pub(super) struct Linker<'a> {
ids: &'a IdMap,
table: &'a ClassTable,
/// Per class: the node of its class object, `NONE` for the synthetic `java.lang.Class`.
class_objects: Vec<u32>,
id_size: u8,
}
impl<'a> Linker<'a> {
pub(super) fn new(ids: &'a IdMap, table: &'a ClassTable, id_size: u8) -> Self {
let class_objects =
table.classes.iter().map(|c| if c.id == 0 { NONE } else { ids.rank(c.id).unwrap_or(NONE) }).collect();
Self { ids, table, class_objects, id_size }
}
/// `None` for root records. `at` is the record's offset, for the error of an instance or array
/// whose class is not in the dump.
pub(super) fn node(&self, at: usize, record: &HeapRecord<'_>) -> Result<Option<Node>, HeapError> {
let class_index = |class_id: u64| {
self.table.by_id.get(&class_id).copied().ok_or(HeapError::MissingClass { id: class_id, at: at as u64 })
};
let (id, ty, len) = match record {
HeapRecord::Root { .. } => return Ok(None),
HeapRecord::Class(c) => (c.id, self.table.class_slot, 0),
HeapRecord::Instance { id, class_id, .. } => (*id, class_index(*class_id)?, 0),
HeapRecord::ObjectArray { id, class_id, len, .. } => (*id, class_index(*class_id)?, *len),
HeapRecord::PrimitiveArray { id, ty, len, .. } => {
let prim_base = self.table.classes.len() as u32;
(*id, prim_base + (*ty as u32 - BasicType::Boolean as u32), *len)
}
};
Ok(self.ids.rank(id).map(|index| Node { index, ty, len }))
}
/// Calls `edge(target, is_referent)` for every reference to an object in the dump, in field
/// order; `is_referent` marks `Reference.referent`, which callers leave out of the graph.
pub(super) fn edges(&self, record: &HeapRecord<'_>, edge: &mut impl FnMut(u32, bool)) {
let mut to = |id: u64, weak: bool| {
if let Some(target) = self.target(id) {
edge(target, weak);
}
};
match record {
HeapRecord::Root { .. } | HeapRecord::PrimitiveArray { .. } => {}
HeapRecord::Class(c) => {
to(c.super_id, false);
to(c.loader_id, false);
for &(_, object) in &c.statics {
to(object, false);
}
}
HeapRecord::ObjectArray { data, .. } => {
for item in data.chunks_exact(usize::from(self.id_size)) {
to(read_id(item, 0, self.id_size).unwrap_or(0), false);
}
}
HeapRecord::Instance { class_id, data, .. } => {
let Some(&class) = self.table.by_id.get(class_id) else { return };
let mut base = 0;
let mut next = self.table.fielded[class as usize];
'chain: while next != NONE && base < data.len() {
let c = next as usize;
for &(offset, weak) in &self.table.refs[c] {
let Some(id) = read_id(data, base + offset, self.id_size) else { break 'chain };
to(id, weak);
}
base += self.table.own_bytes[c];
next = self.table.classes[c].super_index.map_or(NONE, |s| self.table.fielded[s as usize]);
}
let object = self.class_objects[class as usize];
if object != NONE {
edge(object, false);
}
}
}
}
/// Id 0 is null; ids not in the dump have no node.
fn target(&self, id: u64) -> Option<u32> {
if id == 0 { None } else { self.ids.rank(id) }
}
}
/// The big-endian id at `at`, or `None` when `data` ends first.
fn read_id(data: &[u8], at: usize, id_size: u8) -> Option<u64> {
let bytes = data.get(at..at.checked_add(usize::from(id_size))?)?;
Some(match *bytes {
[a, b, c, d] => u64::from(u32::from_be_bytes([a, b, c, d])),
_ => u64::from_be_bytes(bytes.try_into().ok()?),
})
}
+479
View File
@@ -0,0 +1,479 @@
use std::cell::RefCell;
use proptest::prelude::*;
use super::*;
use crate::hprof::{HeapRecord, record_at};
use crate::testing::{HprofWriter, MiniHeap, sample};
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N2: u64 = 0x1200;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const R: u64 = 0x1500;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn build(bytes: &[u8]) -> Result<HeapGraph, HeapError> {
build_graph(bytes, &Store::Ram, &no_progress, &CancellationToken::new())
}
fn index(g: &HeapGraph, id: u64) -> u32 {
g.ids.rank(id).unwrap_or_else(|| panic!("{id:#x} is not in the graph"))
}
fn class(g: &HeapGraph, name: &str) -> (u32, u64) {
let slot = g.classes.iter().position(|c| c.name == name).unwrap_or_else(|| panic!("no class {name}"));
(slot as u32, g.classes[slot].id)
}
/// Targets of `id`'s edges, as ids, in edge order.
fn out_of(g: &HeapGraph, id: u64) -> Vec<u64> {
g.out_edges(index(g, id)).iter().map(|&t| g.ids.id_at(t)).collect()
}
fn root_targets(g: &HeapGraph) -> Vec<u64> {
g.out_edges(g.root).iter().map(|&t| g.ids.id_at(t)).collect()
}
fn assert_csr(g: &HeapGraph) {
let n = g.ids.len();
assert_eq!(g.root, n);
assert_eq!(g.out_start.len(), n as usize + 2);
assert_eq!(g.out_start[0], 0);
assert!(g.out_start.windows(2).all(|w| w[0] <= w[1]), "out_start is not monotone");
assert_eq!(g.out_start[n as usize + 1] as usize, g.out.len());
assert!(g.out.iter().all(|&t| t < n), "edge to a node outside the objects");
assert_eq!((g.types.len(), g.lens.len(), g.offsets.len()), (n as usize, n as usize, n as usize));
let slots = g.classes.len() as u32 + PRIM_SLOTS;
assert!(g.types.iter().all(|&t| t < slots), "type outside the slots");
for c in &g.classes {
let mut steps = 0;
let mut cursor = c.super_index;
while let Some(s) = cursor {
steps += 1;
assert!(steps <= g.classes.len(), "super chain of {} does not end", c.name);
cursor = g.classes[s as usize].super_index;
}
}
}
#[test]
fn edges_follow_fields_and_skip_weak_referents() {
let g = build(&sample().bytes()).unwrap();
assert_csr(&g);
let (_, holder) = class(&g, "Holder");
let (_, node) = class(&g, "Node");
let (_, weak) = class(&g, "java.lang.ref.WeakReference");
assert_eq!(out_of(&g, H), [N1, N2, holder]);
assert_eq!(out_of(&g, N1), [N3, node]);
assert_eq!(out_of(&g, N3), [L, node]);
assert_eq!(out_of(&g, W), [weak]);
assert_eq!(out_of(&g, N4), [node]);
assert_eq!(out_of(&g, L), Vec::<u64>::new());
assert_eq!(g.weak_edges_skipped, 1);
let (reference, _) = class(&g, "java.lang.ref.Reference");
assert_eq!(g.classes[reference as usize].weak_field, Some(0));
assert!(g.classes.iter().filter(|c| c.name != "java.lang.ref.Reference").all(|c| c.weak_field.is_none()));
}
#[test]
fn object_arrays_skip_null_and_unknown() {
let mut heap = sample();
heap.array_of(0x1800, "[Ljava.lang.Object;", &[N2, 0, 0x9990, N1, 0xDEAD_BEEF]);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, R), [N1]);
assert_eq!(out_of(&g, 0x1800), [N2, N1]);
}
#[test]
fn super_root_points_at_each_root_once() {
let mut heap = sample();
heap.root(HeapRootKind::JniGlobal, H).root(HeapRootKind::MonitorUsed, H).root(HeapRootKind::JniLocal, 0x9990);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
let targets = root_targets(&g);
assert_eq!(targets.iter().filter(|&&t| t == H).count(), 1);
assert!(targets.contains(&N3) && targets.contains(&W));
assert!(!targets.contains(&N1) && !targets.contains(&R) && !targets.contains(&N4));
let classes = g.classes.iter().filter(|c| c.id != 0).count();
assert_eq!(targets.len(), 3 + classes, "H, N3, W and every sticky class");
let mut sorted = targets.clone();
sorted.sort_unstable();
sorted.dedup();
assert_eq!(sorted.len(), targets.len());
let h = index(&g, H);
let of_h: Vec<_> = g.roots.iter().filter(|r| r.0 == h).map(|r| (r.1, r.2)).collect();
assert_eq!(of_h, [(HeapRootKind::JniGlobal, None), (HeapRootKind::MonitorUsed, None)]);
assert!(g.roots.contains(&(index(&g, N3), HeapRootKind::JavaFrame, Some(1))));
assert!(g.roots.iter().all(|r| r.0 < g.root));
}
#[test]
fn class_dump_edges_cover_statics() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object")
.load_class(0x100, 1)
.class(0x100, 0, &[], &[])
.string(2, "Config")
.string(3, "INSTANCE")
.string(4, "EMPTY")
.string(5, "LOST")
.load_class(0x200, 2)
.class_with_loader(0x200, 0x100, 0x1200, &[], &[(3, N1), (4, 0), (5, 0x9990)])
.instance(N1, 0x100, &[])
.instance(0x1200, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x200), [0x100, 0x1200, N1]);
assert_eq!(out_of(&g, 0x100), Vec::<u64>::new());
let (config, _) = class(&g, "Config");
let info = &g.classes[config as usize];
let statics: Vec<_> = info.statics.iter().map(|(n, id)| (n.as_str(), *id)).collect();
assert_eq!(statics, [("INSTANCE", N1), ("EMPTY", 0), ("LOST", 0x9990)]);
assert_eq!(info.super_index, Some(class(&g, "java.lang.Object").0));
}
#[test]
fn missing_class_is_an_error() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]).instance(N1, 0x999, &[]);
let bytes = w.finish();
let at = instance_offset(&bytes, N1);
let err = build(&bytes).err().unwrap();
assert!(matches!(err, HeapError::MissingClass { id: 0x999, at: a } if a == at as u64), "{err:?}");
let mut w = HprofWriter::new();
w.obj_array(0x10, 0x999, &[]);
assert!(matches!(build(&w.finish()), Err(HeapError::MissingClass { id: 0x999, .. })));
}
fn instance_offset(bytes: &[u8], id: u64) -> usize {
let header = crate::hprof::parse_header(bytes).unwrap();
let mut found = None;
crate::hprof::scan(bytes, &header, &mut |r| {
if let crate::hprof::Record::Heap { at, record: HeapRecord::Instance { id: i, .. } } = r
&& i == id
{
found = Some(at);
}
Ok(())
})
.unwrap();
found.unwrap()
}
#[test]
fn cancel_during_linking_returns_cancelled() {
let bytes = sample().bytes();
let cancelled = CancellationToken::new();
cancelled.cancel();
let result = build_graph(&bytes, &Store::Ram, &no_progress, &cancelled);
assert!(matches!(result, Err(HeapError::Cancelled)));
let cancel = CancellationToken::new();
let phases = RefCell::new(Vec::new());
let result = build_graph(
&bytes,
&Store::Ram,
&|phase, done, total| {
phases.borrow_mut().push((phase, done, total));
if phase == HeapPhase::Linking {
cancel.cancel();
}
Ok(())
},
&cancel,
);
assert!(matches!(result, Err(HeapError::Cancelled)));
let phases = phases.into_inner();
assert_eq!(phases.first(), Some(&(HeapPhase::Reading, 0, bytes.len() as u64)));
assert_eq!(phases.last(), Some(&(HeapPhase::Linking, 0, 15)), "stops at the first Linking report");
}
#[test]
fn progress_error_stops_the_build() {
let bytes = sample().bytes();
let fail = |phase, _, total| {
if phase == HeapPhase::Linking {
Err(HeapError::NotEnoughDiskSpace { needed: total * 90, available: 0 })
} else {
Ok(())
}
};
let result = build_graph(&bytes, &Store::Ram, &fail, &CancellationToken::new());
assert!(matches!(result, Err(HeapError::NotEnoughDiskSpace { needed: 1350, available: 0 })));
}
#[test]
fn progress_reads_bytes_then_links_objects() {
let bytes = sample().bytes();
let phases = RefCell::new(Vec::new());
let g = build_graph(
&bytes,
&Store::Ram,
&|phase, done, total| {
phases.borrow_mut().push((phase, done, total));
Ok(())
},
&CancellationToken::new(),
)
.unwrap();
let phases = phases.into_inner();
let len = bytes.len() as u64;
let objects = u64::from(g.root);
assert_eq!(objects, 15, "7 classes and 8 objects");
assert_eq!(phases.first(), Some(&(HeapPhase::Reading, 0, len)));
assert!(phases.contains(&(HeapPhase::Reading, len, len)));
let linking: Vec<_> = phases.iter().filter(|p| p.0 == HeapPhase::Linking).collect();
assert_eq!(linking.first(), Some(&&(HeapPhase::Linking, 0, objects)));
assert_eq!(linking.iter().filter(|p| ***p == (HeapPhase::Linking, 0, objects)).count(), 1, "{linking:?}");
assert_eq!(linking.last(), Some(&&(HeapPhase::Linking, objects, objects)));
assert!(phases.iter().all(|&(_, done, total)| done <= total));
}
#[test]
fn types_lengths_and_offsets() {
let bytes = sample().bytes();
let g = build(&bytes).unwrap();
let (holder, holder_id) = class(&g, "Holder");
let (node, _) = class(&g, "Node");
let (array, _) = class(&g, "[Ljava.lang.Object;");
let (java_lang_class, _) = class(&g, "java.lang.Class");
assert_eq!(g.class_slot, java_lang_class);
let ty = |id| g.types[index(&g, id) as usize];
let len = |id| g.lens[index(&g, id) as usize];
assert_eq!([ty(H), ty(N1), ty(R), ty(holder_id)], [holder, node, array, java_lang_class]);
assert_eq!(ty(L), g.prim_slot(BasicType::Long));
assert_eq!(g.prim_type(ty(L)), Some(BasicType::Long));
assert_eq!(g.prim_type(holder), None);
assert_eq!(g.prim_type(g.classes.len() as u32 + PRIM_SLOTS), None);
assert_eq!(g.prim_slot(BasicType::Boolean), g.classes.len() as u32);
assert_eq!(g.prim_slot(BasicType::Long), g.classes.len() as u32 + PRIM_SLOTS - 1);
assert_eq!([len(H), len(L), len(R), len(holder_id)], [0, 2, 3, 0]);
for i in 0..g.root {
let id = match record_at(&bytes, g.offsets[i as usize] as usize, g.id_size).unwrap() {
HeapRecord::Class(c) => c.id,
HeapRecord::Instance { id, .. } | HeapRecord::ObjectArray { id, .. } => id,
HeapRecord::PrimitiveArray { id, .. } => id,
other => panic!("offset of object {i} points at {other:?}"),
};
assert_eq!(id, g.ids.id_at(i));
}
assert_eq!((g.id_size, g.timestamp_ms), (8, 1_700_000_000_000));
let holder_info = &g.classes[holder as usize];
let fields: Vec<_> = holder_info.fields.iter().map(|(n, t)| (n.as_str(), *t)).collect();
assert_eq!(fields, [("a", BasicType::Object), ("b", BasicType::Object)]);
assert_eq!(holder_info.super_index, Some(class(&g, "java.lang.Object").0));
assert_eq!(g.classes[class(&g, "java.lang.Object").0 as usize].super_index, None);
}
#[test]
fn dump_without_java_lang_class_gets_a_synthetic_slot() {
let mut w = HprofWriter::new();
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]).instance(N1, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(g.classes.len(), 2);
let synthetic = &g.classes[g.class_slot as usize];
assert_eq!((synthetic.id, synthetic.name.as_str(), synthetic.super_index), (0, "java.lang.Class", None));
assert_eq!(g.types[index(&g, 0x100) as usize], g.class_slot);
assert_eq!(g.types[index(&g, N1) as usize], 0);
}
#[test]
fn classes_without_a_name_record_get_their_address() {
let mut w = HprofWriter::new();
w.class(0x100, 0, &[(9, BasicType::Object)], &[])
.instance(N1, 0x100, &N2.to_be_bytes())
.instance(N2, 0x100, &[0; 8]);
let g = build(&w.finish()).unwrap();
assert_eq!(g.classes[0].name, "0x100");
assert_eq!(g.classes[0].fields, [(String::new(), BasicType::Object)]);
assert_eq!(out_of(&g, N1), [N2, 0x100]);
}
#[test]
fn thread_objects_map_serials_to_threads() {
let mut heap = sample();
heap.class("java.lang.Thread", Some("java.lang.Object"), &[])
.object(0x2000, "java.lang.Thread", &[])
.object(0x2100, "java.lang.Thread", &[])
.root_in_thread(HeapRootKind::ThreadObject, 0x2000, 1)
.root_in_thread(HeapRootKind::ThreadObject, 0x2100, 7)
.root_in_thread(HeapRootKind::ThreadObject, 0x9990, 9);
let g = build(&heap.bytes()).unwrap();
let expected: HashMap<u32, u32> = [(1, index(&g, 0x2000)), (7, index(&g, 0x2100))].into();
assert_eq!(g.thread_objects, expected);
assert!(g.roots.contains(&(index(&g, 0x2100), HeapRootKind::ThreadObject, Some(7))));
}
#[test]
fn inherited_fields_follow_the_class_first_layout() {
let mut heap = MiniHeap::new();
heap.class("A", Some("java.lang.Object"), &[("a", BasicType::Object), ("n", BasicType::Int)])
.class("Empty", Some("A"), &[])
.class("B", Some("Empty"), &[("flag", BasicType::Byte), ("b", BasicType::Object)])
.object(0x10, "B", &[("a", 0x20), ("b", 0x30), ("flag", 1)])
.object(0x20, "A", &[])
.object(0x30, "A", &[]);
let g = build(&heap.bytes()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x30, 0x20, class(&g, "B").1]);
}
#[test]
fn super_class_cycles_are_cut() {
let mut w = HprofWriter::new();
w.string(1, "A")
.string(2, "B")
.string(3, "S")
.string(4, "f")
.load_class(0x100, 1)
.load_class(0x200, 2)
.load_class(0x300, 3)
.class(0x100, 0x200, &[(4, BasicType::Object)], &[])
.class(0x200, 0x100, &[(4, BasicType::Object)], &[])
.class(0x300, 0x300, &[], &[])
.instance(0x10, 0x100, &[[0; 7].as_slice(), &[0x20], &[0; 7], &[0x30]].concat())
.instance(0x20, 0x300, &[])
.instance(0x30, 0x300, &[0; 64]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x20, 0x30, 0x100]);
assert_eq!(out_of(&g, 0x30), [0x300]);
}
#[test]
fn short_instance_data_stops_at_the_last_whole_field() {
let mut w = HprofWriter::new();
w.string(1, "P")
.string(2, "x")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (2, BasicType::Object)], &[])
.instance(0x10, 0x100, &[[0; 7].as_slice(), &[0x20], &[0; 5]].concat())
.instance(0x20, 0x100, &[]);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(out_of(&g, 0x10), [0x20, 0x100]);
}
#[test]
fn id_size_4_dump_links() {
let mut w = HprofWriter::with_id_size(4);
w.string(1, "Node")
.string(2, "next")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object)], &[(2, 0x20)])
.instance(0x10, 0x100, &0x20u32.to_be_bytes())
.instance(0x20, 0x100, &[0; 4])
.obj_array(0x30, 0x100, &[0x10, 0, 0x20])
.root(HeapRootKind::JniGlobal, 0x30);
let g = build(&w.finish()).unwrap();
assert_csr(&g);
assert_eq!(g.id_size, 4);
assert_eq!(out_of(&g, 0x10), [0x20, 0x100]);
assert_eq!(out_of(&g, 0x30), [0x10, 0x20]);
assert_eq!(out_of(&g, 0x100), [0x20]);
assert_eq!(root_targets(&g), [0x30]);
}
#[test]
fn dir_store_keeps_types_and_offsets_only() {
let dir = tempfile::tempdir().unwrap();
// The id map's own files (`ids.*`) depend on its layout; MiniHeap's class ids are unaligned.
let names = || {
let mut names: Vec<_> = std::fs::read_dir(dir.path())
.unwrap()
.map(|e| e.unwrap().file_name().into_string().unwrap())
.filter(|name| !name.starts_with("ids."))
.collect();
names.sort();
names
};
let g =
build_graph(&sample().bytes(), &Store::Dir(dir.path().to_path_buf()), &no_progress, &CancellationToken::new())
.unwrap();
assert_eq!(out_of(&g, H).len(), 3);
let kept = ["offsets.u64", "types.u32"];
assert!(dir.path().join("ids.sorted.u64").exists());
#[cfg(unix)]
assert_eq!(names(), kept);
drop(g);
assert_eq!(names(), kept);
}
#[test]
fn empty_dump_has_only_the_super_root() {
let g = build(&HprofWriter::new().finish()).unwrap();
assert_csr(&g);
assert_eq!((g.root, g.out.len(), g.classes.len()), (0, 0, 1));
assert_eq!(g.class_slot, 0);
}
#[test]
fn edge_count_past_u32_is_too_large() {
let cancel = CancellationToken::new();
let mut counts = [0, 2, 0, 3, 1];
assert_eq!(accumulate(&mut counts, &cancel).unwrap(), 6);
assert_eq!(counts, [0, 2, 2, 5, 6]);
let mut full = [0, u32::MAX - 1, 1];
assert_eq!(accumulate(&mut full, &cancel).unwrap(), u32::MAX);
let mut over = [0, u32::MAX, 1];
assert!(matches!(accumulate(&mut over, &cancel), Err(HeapError::TooLarge)));
cancel.cancel();
assert!(matches!(accumulate(&mut [0, 1], &cancel), Err(HeapError::Cancelled)));
}
/// A class with `fields` instance fields, every one named by the same `name_len`-byte UTF8 record.
fn shared_long_field_names(name_len: usize, fields: usize) -> Vec<u8> {
let mut w = HprofWriter::new();
let long = "f".repeat(name_len);
w.string(1, "java/lang/Object").load_class(0x100, 1).class(0x100, 0, &[], &[]);
w.string(2, "Wide").string(3, &long).load_class(0x200, 2);
w.class(0x200, 0x100, &vec![(3, BasicType::Int); fields], &[]);
w.finish()
}
#[test]
fn names_that_would_decode_far_larger_than_the_file_are_refused() {
// 128 MiB of names from a file of about 70 KB.
let bytes = shared_long_field_names(32 << 10, 4000);
assert!(bytes.len() < 100 << 10, "{}", bytes.len());
let err = build(&bytes).err().expect("refused");
assert!(matches!(err, HeapError::NamesTooLarge), "{err:?}");
}
#[test]
fn a_name_longer_than_the_jvm_allows_is_refused() {
let mut w = HprofWriter::new();
w.string(1, &"C".repeat(70_000)).load_class(0x100, 1).class(0x100, 0, &[], &[]);
assert!(matches!(build(&w.finish()), Err(HeapError::NamesTooLarge)));
}
#[test]
fn real_sized_names_shared_by_many_fields_are_fine() {
let g = build(&shared_long_field_names(40, 4000)).unwrap();
assert_eq!(g.classes[class(&g, "Wide").0 as usize].fields.len(), 4000);
}
proptest! {
#[test]
fn build_graph_never_panics_on_mutated_dumps(flips in proptest::collection::vec((0usize..4096, any::<u8>()), 1..12)) {
let mut bytes = sample().bytes();
for (pos, value) in flips {
let at = pos % bytes.len();
bytes[at] = value;
}
if let Ok(g) = build(&bytes) {
assert_csr(&g);
}
}
}
+945
View File
@@ -0,0 +1,945 @@
//! Reader for HotSpot `.hprof` heap dumps (format 1.0.1 / 1.0.2, 4- or 8-byte ids).
use std::ops::Range;
use serde::{Deserialize, Serialize};
use crate::error::HeapError;
use crate::reader::Reader;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum BasicType {
Object = 2,
Boolean = 4,
Char = 5,
Float = 6,
Double = 7,
Byte = 8,
Short = 9,
Int = 10,
Long = 11,
}
impl BasicType {
pub fn from_u8(v: u8) -> Option<Self> {
Some(match v {
2 => Self::Object,
4 => Self::Boolean,
5 => Self::Char,
6 => Self::Float,
7 => Self::Double,
8 => Self::Byte,
9 => Self::Short,
10 => Self::Int,
11 => Self::Long,
_ => return None,
})
}
/// Bytes one value takes in the dump; `id_size` is the dump's identifier size.
pub fn hprof_size(self, id_size: u8) -> usize {
match self {
Self::Object => usize::from(id_size),
Self::Boolean | Self::Byte => 1,
Self::Char | Self::Short => 2,
Self::Float | Self::Int => 4,
Self::Double | Self::Long => 8,
}
}
/// Bytes one value takes in the JVM heap; `ref_size` is 4 with compressed oops, else 8.
pub fn heap_size(self, ref_size: u8) -> u64 {
match self {
Self::Object => u64::from(ref_size),
other => other.hprof_size(ref_size) as u64,
}
}
/// JVM class name of an array of this type (`[I`); `Object` arrays have no single name, so
/// they report the most general one.
pub fn array_name(self) -> &'static str {
match self {
Self::Object => "[Ljava.lang.Object;",
Self::Boolean => "[Z",
Self::Char => "[C",
Self::Float => "[F",
Self::Double => "[D",
Self::Byte => "[B",
Self::Short => "[S",
Self::Int => "[I",
Self::Long => "[J",
}
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum HeapRootKind {
Unknown,
JniGlobal,
JniLocal,
JavaFrame,
NativeStack,
StickyClass,
ThreadBlock,
MonitorUsed,
ThreadObject,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct HprofHeader {
pub version: String,
pub id_size: u8,
pub timestamp_ms: u64,
/// Offset of the first record.
pub body_start: usize,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClassDump {
pub id: u64,
pub super_id: u64,
pub loader_id: u64,
/// `(name id, object id)` of reference statics; a null static has object id 0.
pub statics: Vec<(u64, u64)>,
/// Heap bytes taken by the primitive statics.
pub static_bytes: u64,
/// Instance fields in dump order, `(name id, type)`.
pub fields: Vec<(u64, BasicType)>,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum HeapRecord<'a> {
/// `thread_serial` is present for the kinds that carry one.
Root {
kind: HeapRootKind,
id: u64,
thread_serial: Option<u32>,
},
Class(ClassDump),
/// `data` is the field values, class first, then each superclass.
Instance {
id: u64,
class_id: u64,
data: &'a [u8],
},
/// `data` is `len` ids.
ObjectArray {
id: u64,
class_id: u64,
len: u32,
data: &'a [u8],
},
/// `data` is `len` big-endian elements.
PrimitiveArray {
id: u64,
ty: BasicType,
len: u32,
data: &'a [u8],
},
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum Record<'a> {
Utf8 {
id: u64,
text: &'a [u8],
},
LoadClass {
class_id: u64,
name_id: u64,
},
/// Byte range of a heap-dump segment's sub-records; its contents follow as `Heap` records.
Segment(Range<usize>),
/// `at` is the offset of the sub-record's tag, for [`record_at`].
Heap {
at: usize,
record: HeapRecord<'a>,
},
}
const MAGIC_1_0_2: &[u8] = b"JAVA PROFILE 1.0.2\0";
const MAGIC_1_0_1: &[u8] = b"JAVA PROFILE 1.0.1\0";
const TAG_UTF8: u8 = 0x01;
const TAG_LOAD_CLASS: u8 = 0x02;
const TAG_HEAP_DUMP: u8 = 0x0C;
const TAG_HEAP_DUMP_SEGMENT: u8 = 0x1C;
const TAG_HEAP_DUMP_END: u8 = 0x2C;
/// Tag (u1), timestamp (u4) and body length (u4).
const RECORD_HEADER_LEN: usize = 9;
fn check_id_size(id_size: u8) -> Result<(), HeapError> {
match id_size {
4 | 8 => Ok(()),
other => Err(HeapError::UnsupportedIdSize(u32::from(other))),
}
}
fn truncated(buf: &[u8]) -> HeapError {
HeapError::Truncated { at: buf.len() as u64 }
}
pub fn parse_header(buf: &[u8]) -> Result<HprofHeader, HeapError> {
let magic =
[MAGIC_1_0_2, MAGIC_1_0_1].into_iter().find(|magic| buf.starts_with(magic)).ok_or(HeapError::NotHprof)?;
let mut r = Reader::new(buf, magic.len(), 8);
let id_size = r.u4()?;
let id_size = u8::try_from(id_size).map_err(|_| HeapError::UnsupportedIdSize(id_size))?;
check_id_size(id_size)?;
let timestamp_ms = r.u8()?;
let version = String::from_utf8_lossy(&magic[..magic.len() - 1]).into_owned();
Ok(HprofHeader { version, id_size, timestamp_ms, body_start: r.pos() })
}
/// Walks every top-level record and, for heap-dump segments, the sub-records inside them.
/// A record whose length runs past the end of `buf` is `Truncated`, and so are segments that no
/// HEAP DUMP END follows: HotSpot writes whole segments, so a dump it did not finish ends after one.
pub fn scan<'a>(
buf: &'a [u8],
header: &HprofHeader,
visit: &mut dyn FnMut(Record<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
let id_size = header.id_size;
check_id_size(id_size)?;
if header.body_start > buf.len() {
return Err(truncated(buf));
}
let mut pos = header.body_start;
let mut unended_segments = false;
while pos < buf.len() {
let mut head = Reader::new(buf, pos, id_size);
let tag = head.u1()?;
head.skip(4)?; // timestamp
let len = head.u4()? as usize;
let start = pos + RECORD_HEADER_LEN;
let end = start.checked_add(len).filter(|&end| end <= buf.len()).ok_or_else(|| truncated(buf))?;
let body = &buf[..end];
let mut r = Reader::new(body, start, id_size);
match tag {
TAG_UTF8 => {
let id = r.id()?;
let text = r.bytes(end - r.pos())?;
visit(Record::Utf8 { id, text })?;
}
TAG_LOAD_CLASS => {
r.skip(4)?; // class serial
let class_id = r.id()?;
r.skip(4)?; // stack trace serial
let name_id = r.id()?;
visit(Record::LoadClass { class_id, name_id })?;
}
TAG_HEAP_DUMP | TAG_HEAP_DUMP_SEGMENT => {
// A single HEAP DUMP (JDK 8, under 2 GB) has no end record.
unended_segments |= tag == TAG_HEAP_DUMP_SEGMENT;
visit(Record::Segment(start..end))?;
walk_segment(buf, start..end, id_size, &mut |at, record| visit(Record::Heap { at, record }))?;
}
TAG_HEAP_DUMP_END => unended_segments = false,
_ => {}
}
pos = end;
}
if unended_segments {
return Err(truncated(buf));
}
Ok(())
}
/// Visits the sub-records of one segment (the range from [`Record::Segment`]) with their offsets.
/// The last sub-record must end exactly at the segment's end.
pub fn scan_segment<'a>(
buf: &'a [u8],
segment: Range<usize>,
id_size: u8,
visit: &mut dyn FnMut(usize, HeapRecord<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
check_id_size(id_size)?;
walk_segment(buf, segment, id_size, visit)
}
fn walk_segment<'a>(
buf: &'a [u8],
segment: Range<usize>,
id_size: u8,
visit: &mut dyn FnMut(usize, HeapRecord<'a>) -> Result<(), HeapError>,
) -> Result<(), HeapError> {
if segment.start > segment.end || segment.end > buf.len() {
return Err(truncated(buf));
}
let mut r = Reader::new(&buf[..segment.end], segment.start, id_size);
while r.pos() < segment.end {
let at = r.pos();
let record = read_sub_record(&mut r)?;
visit(at, record)?;
}
Ok(())
}
/// Reads the sub-record whose tag is at `at`, as given by [`Record::Heap`].
pub fn record_at(buf: &[u8], at: usize, id_size: u8) -> Result<HeapRecord<'_>, HeapError> {
check_id_size(id_size)?;
read_sub_record(&mut Reader::new(buf, at, id_size))
}
fn read_sub_record<'a>(r: &mut Reader<'a>) -> Result<HeapRecord<'a>, HeapError> {
let at = r.pos();
let tag = r.u1()?;
// (kind, carries a thread serial, bytes after it)
let root = match tag {
0xFF => Some((HeapRootKind::Unknown, false, 0)),
0x01 => Some((HeapRootKind::JniGlobal, false, usize::from(r.id_size()))),
0x02 => Some((HeapRootKind::JniLocal, true, 4)),
0x03 => Some((HeapRootKind::JavaFrame, true, 4)),
0x04 => Some((HeapRootKind::NativeStack, true, 0)),
0x05 => Some((HeapRootKind::StickyClass, false, 0)),
0x06 => Some((HeapRootKind::ThreadBlock, true, 0)),
0x07 => Some((HeapRootKind::MonitorUsed, false, 0)),
0x08 => Some((HeapRootKind::ThreadObject, true, 4)),
_ => None,
};
if let Some((kind, has_serial, trailing)) = root {
let id = r.id()?;
let thread_serial = if has_serial { Some(r.u4()?) } else { None };
r.skip(trailing)?;
return Ok(HeapRecord::Root { kind, id, thread_serial });
}
match tag {
0x20 => read_class_dump(r).map(HeapRecord::Class),
0x21 => {
let id = r.id()?;
r.skip(4)?; // stack trace serial
let class_id = r.id()?;
let len = r.u4()? as usize;
Ok(HeapRecord::Instance { id, class_id, data: r.bytes(len)? })
}
0x22 => {
let id = r.id()?;
r.skip(4)?;
let len = r.u4()?;
let class_id = r.id()?;
let size = (len as usize).checked_mul(usize::from(r.id_size())).ok_or_else(|| truncated(r.buf()))?;
Ok(HeapRecord::ObjectArray { id, class_id, len, data: r.bytes(size)? })
}
0x23 => {
let id = r.id()?;
r.skip(4)?;
let len = r.u4()?;
let ty = read_type(r)?;
if ty == BasicType::Object {
return Err(HeapError::UnknownType { ty: ty as u8, at: (r.pos() - 1) as u64 });
}
let size = (len as usize).checked_mul(ty.hprof_size(r.id_size())).ok_or_else(|| truncated(r.buf()))?;
Ok(HeapRecord::PrimitiveArray { id, ty, len, data: r.bytes(size)? })
}
tag => Err(HeapError::UnknownSubRecord { tag, at: at as u64 }),
}
}
fn read_type(r: &mut Reader<'_>) -> Result<BasicType, HeapError> {
let at = r.pos();
let ty = r.u1()?;
BasicType::from_u8(ty).ok_or(HeapError::UnknownType { ty, at: at as u64 })
}
fn read_class_dump(r: &mut Reader<'_>) -> Result<ClassDump, HeapError> {
let id_size = r.id_size();
let id = r.id()?;
r.skip(4)?; // stack trace serial
let super_id = r.id()?;
let loader_id = r.id()?;
r.skip(4 * usize::from(id_size))?; // signers, protection domain, 2 reserved
r.skip(4)?; // instance size
for _ in 0..r.u2()? {
r.skip(2)?; // constant pool index
let ty = read_type(r)?;
r.skip(ty.hprof_size(id_size))?;
}
let mut statics = Vec::new();
let mut static_bytes = 0;
for _ in 0..r.u2()? {
let name = r.id()?;
let ty = read_type(r)?;
if ty == BasicType::Object {
statics.push((name, r.id()?));
} else {
r.skip(ty.hprof_size(id_size))?;
static_bytes += ty.heap_size(id_size);
}
}
let mut fields = Vec::new();
for _ in 0..r.u2()? {
let name = r.id()?;
fields.push((name, read_type(r)?));
}
Ok(ClassDump { id, super_id, loader_id, statics, static_bytes, fields })
}
#[cfg(test)]
mod tests {
use super::*;
use crate::testing::{HprofWriter, MiniHeap};
use proptest::prelude::*;
fn collect(bytes: &[u8]) -> Result<Vec<Record<'_>>, HeapError> {
let header = parse_header(bytes)?;
let mut out = Vec::new();
scan(bytes, &header, &mut |r| {
out.push(r);
Ok(())
})?;
Ok(out)
}
fn segment_of(bytes: &[u8]) -> Range<usize> {
collect(bytes)
.unwrap()
.into_iter()
.find_map(|r| if let Record::Segment(s) = r { Some(s) } else { None })
.expect("a segment")
}
fn all_kinds(w: &mut HprofWriter) {
w.string(1, "Node")
.string(2, "next")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (3, BasicType::Long)], &[(2, 0x10)])
.instance(0x10, 0x100, &[0u8; 16])
.obj_array(0x20, 0x100, &[0x10, 0])
.prim_array(0x30, BasicType::Int, 3)
.root(HeapRootKind::JavaFrame, 0x10)
.root(HeapRootKind::StickyClass, 0x100);
}
#[test]
fn parses_header() {
let header = parse_header(&MiniHeap::new().bytes()).unwrap();
assert_eq!((header.version.as_str(), header.id_size, header.body_start), ("JAVA PROFILE 1.0.2", 8, 31));
let mut old = b"JAVA PROFILE 1.0.1\0".to_vec();
old.extend_from_slice(&[0, 0, 0, 4, 0, 0, 0, 1, 0, 0, 0, 2]);
let header = parse_header(&old).unwrap();
assert_eq!(
(header.version.as_str(), header.id_size, header.timestamp_ms),
("JAVA PROFILE 1.0.1", 4, (1 << 32) + 2)
);
let bad_size = b"JAVA PROFILE 1.0.2\0\0\0\0\x03\0\0\0\0\0\0\0\0";
assert!(matches!(parse_header(bad_size), Err(HeapError::UnsupportedIdSize(3))));
assert!(matches!(parse_header(b"GIF89a"), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b""), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b"JAVA PROFILE 1.0.3\0\0\0\0\x08"), Err(HeapError::NotHprof)));
assert!(matches!(parse_header(b"JAVA PROFILE 1.0.2\0\0\0"), Err(HeapError::Truncated { at: 21 })));
}
#[test]
fn scans_every_record_kind() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let records = collect(&bytes).unwrap();
let (mut utf8, mut loads, mut segments) = (0, 0, 0);
let (mut classes, mut instances, mut obj_arrays, mut prim_arrays, mut roots) = (0, 0, 0, 0, 0);
for record in &records {
match record {
Record::Utf8 { .. } => utf8 += 1,
Record::LoadClass { class_id, name_id } => {
loads += 1;
assert_eq!((*class_id, *name_id), (0x100, 1));
}
Record::Segment(_) => segments += 1,
Record::Heap { record, .. } => match record {
HeapRecord::Class(c) => {
classes += 1;
assert_eq!((c.id, c.super_id, c.loader_id), (0x100, 0, 0));
assert_eq!(c.statics, [(2, 0x10)]);
assert_eq!(c.fields, [(2, BasicType::Object), (3, BasicType::Long)]);
}
HeapRecord::Instance { id, class_id, data } => {
instances += 1;
assert_eq!((*id, *class_id, data.len()), (0x10, 0x100, 16));
}
HeapRecord::ObjectArray { id, class_id, len, data } => {
obj_arrays += 1;
assert_eq!((*id, *class_id, *len, data.len()), (0x20, 0x100, 2, 16));
}
HeapRecord::PrimitiveArray { id, ty, len, data } => {
prim_arrays += 1;
assert_eq!((*id, *ty, *len, data.len()), (0x30, BasicType::Int, 3, 12));
}
HeapRecord::Root { kind, id, thread_serial } => {
roots += 1;
match kind {
HeapRootKind::JavaFrame => assert_eq!((*id, *thread_serial), (0x10, Some(1))),
HeapRootKind::StickyClass => assert_eq!((*id, *thread_serial), (0x100, None)),
other => panic!("unexpected root {other:?}"),
}
}
},
}
}
assert_eq!((utf8, loads, segments), (2, 1, 1));
assert_eq!((classes, instances, obj_arrays, prim_arrays, roots), (1, 1, 1, 1, 2));
assert!(matches!(&records[3], Record::Segment(s) if s.end <= bytes.len()), "segment precedes its contents");
}
#[test]
fn root_kinds_carry_the_thread_serial_they_have() {
let kinds = [
(HeapRootKind::Unknown, false),
(HeapRootKind::JniGlobal, false),
(HeapRootKind::JniLocal, true),
(HeapRootKind::JavaFrame, true),
(HeapRootKind::NativeStack, true),
(HeapRootKind::StickyClass, false),
(HeapRootKind::ThreadBlock, true),
(HeapRootKind::MonitorUsed, false),
(HeapRootKind::ThreadObject, true),
];
let mut w = HprofWriter::new();
for (i, (kind, _)) in kinds.iter().enumerate() {
w.root_in_thread(*kind, 0x40 + i as u64, 7);
w.root(*kind, 0x80 + i as u64);
}
let bytes = w.finish();
let mut seen = Vec::new();
for r in collect(&bytes).unwrap() {
if let Record::Heap { record: HeapRecord::Root { kind, id, thread_serial }, .. } = r {
seen.push((kind, id, thread_serial));
}
}
assert_eq!(seen.len(), kinds.len() * 2);
for (i, (kind, carries)) in kinds.iter().enumerate() {
let explicit = seen[2 * i];
let default = seen[2 * i + 1];
assert_eq!(explicit, (*kind, 0x40 + i as u64, carries.then_some(7)));
assert_eq!(default, (*kind, 0x80 + i as u64, carries.then_some(1)));
}
}
#[test]
fn primitive_arrays_expose_their_contents() {
let mut w = HprofWriter::new();
w.prim_bytes(0x1, BasicType::Char, &[0, b'h', 0, 0xE9]).prim_bytes(0x2, BasicType::Byte, b"v42");
let bytes = w.finish();
let arrays: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::PrimitiveArray { ty, len, data, .. }, .. } => {
Some((ty, len, data.to_vec()))
}
_ => None,
})
.collect();
assert_eq!(arrays, [(BasicType::Char, 2, vec![0, b'h', 0, 0xE9]), (BasicType::Byte, 3, b"v42".to_vec())]);
}
#[test]
fn mini_heap_builds_strings_and_defines_object_and_class() {
let mut heap = MiniHeap::new();
heap.string(0x10, "v42", true).string(0x20, "hé", false);
let bytes = heap.bytes();
let records = collect(&bytes).unwrap();
let names: Vec<_> = records
.iter()
.filter_map(|r| {
if let Record::Utf8 { text, .. } = r { Some(String::from_utf8_lossy(text).into_owned()) } else { None }
})
.collect();
assert!(names.iter().any(|n| n == "java/lang/Object") && names.iter().any(|n| n == "java/lang/Class"));
let arrays: Vec<_> = records
.iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::PrimitiveArray { data, .. }, .. } => Some(data.to_vec()),
_ => None,
})
.collect();
assert_eq!(arrays, [b"v42".to_vec(), vec![b'h', 0, 0xE9, 0]]);
let coders: Vec<_> = records
.iter()
.filter_map(|r| match r {
Record::Heap { record: HeapRecord::Instance { id, data, .. }, .. } => Some((*id, data[8])),
_ => None,
})
.collect();
assert_eq!(coders, [(0x10, 0), (0x20, 1)]);
}
#[test]
fn mini_heap_lays_out_fields_class_first_then_super() {
let mut heap = MiniHeap::new();
heap.class("A", Some("java.lang.Object"), &[("a", BasicType::Int)])
.class("B", Some("A"), &[("b", BasicType::Object), ("c", BasicType::Byte)])
.object(0x10, "B", &[("a", 7), ("b", 0x99), ("c", 0x1FF)]);
let bytes = heap.bytes();
let data = collect(&bytes)
.unwrap()
.into_iter()
.find_map(|r| match r {
Record::Heap { record: HeapRecord::Instance { data, .. }, .. } => Some(data.to_vec()),
_ => None,
})
.unwrap();
assert_eq!(data, [0, 0, 0, 0, 0, 0, 0, 0x99, 0xFF, 0, 0, 0, 7]);
}
#[test]
fn record_at_reads_back_an_instance() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let at = collect(&bytes)
.unwrap()
.into_iter()
.find_map(|r| match r {
Record::Heap { at, record: HeapRecord::Instance { .. } } => Some(at),
_ => None,
})
.unwrap();
match record_at(&bytes, at, 8).unwrap() {
HeapRecord::Instance { id, class_id, data } => assert_eq!((id, class_id, data.len()), (0x10, 0x100, 16)),
other => panic!("expected an instance, got {other:?}"),
}
assert!(matches!(record_at(&bytes, bytes.len(), 8), Err(HeapError::Truncated { .. })));
assert!(matches!(record_at(&bytes, at, 3), Err(HeapError::UnsupportedIdSize(3))));
}
#[test]
fn scan_segment_visits_the_same_records_scan_does() {
let mut w = HprofWriter::new();
all_kinds(&mut w);
let bytes = w.finish();
let from_scan: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| if let Record::Heap { at, record } = r { Some((at, record)) } else { None })
.collect();
let mut from_segment = Vec::new();
scan_segment(&bytes, segment_of(&bytes), 8, &mut |at, record| {
from_segment.push((at, record));
Ok(())
})
.unwrap();
assert_eq!(from_scan.len(), 6);
assert_eq!(from_scan, from_segment);
}
#[test]
fn visitor_errors_stop_the_scan() {
let bytes = MiniHeap::new().bytes();
let header = parse_header(&bytes).unwrap();
let mut calls = 0;
let result = scan(&bytes, &header, &mut |_| {
calls += 1;
Err(HeapError::Cancelled)
});
assert!(matches!(result, Err(HeapError::Cancelled)));
assert_eq!(calls, 1);
}
#[test]
fn both_heap_dump_tags_are_segments() {
let bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
let mut old_style = bytes.clone();
old_style[segment.start - 9] = 0x0C;
assert_eq!(segment_of(&old_style), segment);
}
#[test]
fn id_size_4_dump_parses() {
let mut w = HprofWriter::with_id_size(4);
w.string(1, "Node")
.load_class(0x100, 1)
.class(0x100, 0, &[(2, BasicType::Object), (3, BasicType::Long)], &[(2, 0x10)])
.instance(0x10, 0x100, &[0u8; 12])
.obj_array(0x20, 0x100, &[0x10, 0, 0xFFFF_FFFF])
.prim_array(0x30, BasicType::Short, 5)
.root(HeapRootKind::ThreadObject, 0x10);
let bytes = w.finish();
let header = parse_header(&bytes).unwrap();
assert_eq!((header.id_size, header.body_start), (4, 31));
let records: Vec<_> = collect(&bytes)
.unwrap()
.into_iter()
.filter_map(|r| if let Record::Heap { record, .. } = r { Some(record) } else { None })
.collect();
assert_eq!(records.len(), 5);
match &records[0] {
HeapRecord::Class(c) => {
assert_eq!((c.id, c.statics.as_slice(), c.fields.len()), (0x100, &[(2, 0x10)][..], 2));
}
other => panic!("expected the class, got {other:?}"),
}
assert!(matches!(&records[1], HeapRecord::Instance { id: 0x10, class_id: 0x100, data } if data.len() == 12));
assert!(
matches!(&records[2], HeapRecord::ObjectArray { len: 3, data, .. } if data.len() == 12 && data[8..] == [0xFF; 4])
);
assert!(
matches!(&records[3], HeapRecord::PrimitiveArray { ty: BasicType::Short, len: 5, data, .. } if data.len() == 10)
);
assert!(matches!(
&records[4],
HeapRecord::Root { kind: HeapRootKind::ThreadObject, id: 0x10, thread_serial: Some(1) }
));
}
fn push_id(out: &mut Vec<u8>, v: u64) {
out.extend_from_slice(&v.to_be_bytes());
}
/// A class dump written by hand: loader, a constant-pool entry, and statics of each shape.
fn class_dump_bytes(last_field_type: u8) -> Vec<u8> {
let mut b = vec![0x20];
push_id(&mut b, 0x500);
b.extend_from_slice(&0u32.to_be_bytes());
for id in [0x400, 0x77, 0, 0, 0, 0] {
push_id(&mut b, id); // super, loader, signers, domain, 2 reserved
}
b.extend_from_slice(&24u32.to_be_bytes());
b.extend_from_slice(&[0, 1, 0, 7, 10, 0, 0, 0, 99]); // constant pool: one int
b.extend_from_slice(&[0, 3]);
push_id(&mut b, 0xA);
b.push(2);
push_id(&mut b, 0xBEEF); // object static
push_id(&mut b, 0xB);
b.push(10);
b.extend_from_slice(&5u32.to_be_bytes()); // int static
push_id(&mut b, 0xC);
b.push(11);
b.extend_from_slice(&6u64.to_be_bytes()); // long static
b.extend_from_slice(&[0, 2]);
push_id(&mut b, 0xD);
b.push(2);
push_id(&mut b, 0xE);
b.push(last_field_type);
b
}
#[test]
fn class_dump_reads_loader_statics_and_skips_the_constant_pool() {
let bytes = class_dump_bytes(11);
let HeapRecord::Class(c) = record_at(&bytes, 0, 8).unwrap() else { panic!("expected a class") };
assert_eq!((c.id, c.super_id, c.loader_id), (0x500, 0x400, 0x77));
assert_eq!(c.statics, [(0xA, 0xBEEF)]);
assert_eq!(c.static_bytes, 12);
assert_eq!(c.fields, [(0xD, BasicType::Object), (0xE, BasicType::Long)]);
}
#[test]
fn unknown_value_type_reports_its_offset() {
let bytes = class_dump_bytes(0x63);
let err = record_at(&bytes, 0, 8).unwrap_err();
assert!(matches!(err, HeapError::UnknownType { ty: 0x63, at } if at == bytes.len() as u64 - 1), "{err:?}");
}
#[test]
fn truncated_dump_reports_offset() {
let mut bytes = MiniHeap::new().bytes();
let body_start = parse_header(&bytes).unwrap().body_start as u64;
bytes.truncate(bytes.len() - 5);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at > body_start && at == bytes.len() as u64), "{err:?}");
assert!(err.to_string().contains(&bytes.len().to_string()));
}
/// HotSpot flushes whole segments, so a JVM killed while dumping leaves a file that ends
/// right after one, without HEAP DUMP END.
#[test]
fn a_dump_the_jvm_did_not_finish_ends_early() {
let mut bytes = MiniHeap::new().bytes();
bytes.truncate(bytes.len() - RECORD_HEADER_LEN);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at == bytes.len() as u64), "{err:?}");
assert!(err.to_string().starts_with("the dump ends early"), "{err}");
}
#[test]
fn every_prefix_is_ok_or_truncated_at_its_length() {
let mut heap = MiniHeap::new();
heap.class("N", Some("java.lang.Object"), &[("next", BasicType::Object)])
.object(0x10, "N", &[("next", 0x20)])
.string(0x20, "x", true)
.root(HeapRootKind::ThreadObject, 0x10);
let bytes = heap.bytes();
let header = parse_header(&bytes).unwrap();
let segment = segment_of(&bytes);
for cut in header.body_start..bytes.len() {
let prefix = &bytes[..cut];
match scan(prefix, &header, &mut |_| Ok(())) {
// Only a prefix without heap data is a complete (empty) dump.
Ok(()) => assert!(cut < segment.start, "cut at {cut} has heap data but no end"),
Err(HeapError::Truncated { at }) => assert_eq!(at, cut as u64, "cut at {cut}"),
Err(other) => panic!("cut at {cut}: {other:?}"),
}
}
let past_end = HprofHeader { body_start: bytes.len() + 1, ..header };
assert!(matches!(scan(&bytes, &past_end, &mut |_| Ok(())), Err(HeapError::Truncated { .. })));
}
#[test]
fn sub_record_overrunning_its_segment_is_truncated() {
let mut bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
let len_at = segment.start - 4;
let shrunk = (segment.len() as u32 - 3).to_be_bytes();
bytes[len_at..segment.start].copy_from_slice(&shrunk);
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::Truncated { at } if at == segment.end as u64 - 3), "{err:?}");
}
#[test]
fn unknown_sub_record_is_an_error() {
let mut bytes = MiniHeap::new().bytes();
let segment = segment_of(&bytes);
bytes[segment.start] = 0x99;
let err = collect(&bytes).unwrap_err();
assert!(matches!(err, HeapError::UnknownSubRecord { tag: 0x99, at } if at == segment.start as u64), "{err:?}");
assert!(err.to_string().contains("0x99"));
}
#[test]
fn array_lengths_larger_than_the_dump_do_not_allocate_or_panic() {
let mut b = vec![0x23];
push_id(&mut b, 1);
b.extend_from_slice(&0u32.to_be_bytes());
b.extend_from_slice(&u32::MAX.to_be_bytes());
b.push(11);
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::Truncated { .. })));
b[0] = 0x22;
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::Truncated { .. })));
}
#[test]
fn primitive_array_of_objects_is_an_unknown_type() {
let mut b = vec![0x23];
push_id(&mut b, 1);
b.extend_from_slice(&[0, 0, 0, 0, 0, 0, 0, 0]);
b.push(2);
assert!(matches!(record_at(&b, 0, 8), Err(HeapError::UnknownType { ty: 2, .. })));
}
#[test]
fn basic_type_sizes_and_names() {
assert_eq!(BasicType::from_u8(10), Some(BasicType::Int));
assert_eq!([0, 1, 3, 12, 255].map(BasicType::from_u8), [None; 5]);
assert_eq!((BasicType::Object.hprof_size(4), BasicType::Object.hprof_size(8)), (4, 8));
assert_eq!((BasicType::Object.heap_size(4), BasicType::Long.heap_size(4)), (4, 8));
assert_eq!((BasicType::Char.hprof_size(8), BasicType::Boolean.heap_size(8)), (2, 1));
assert_eq!(
(BasicType::Int.array_name(), BasicType::Boolean.array_name(), BasicType::Long.array_name()),
("[I", "[Z", "[J")
);
}
fn sample() -> Vec<u8> {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("v", BasicType::Long)])
.class("[LNode;", Some("java.lang.Object"), &[])
.object(0x10, "Node", &[("next", 0x20), ("v", 5)])
.object(0x20, "Node", &[])
.array_of(0x30, "[LNode;", &[0x10, 0x20, 0])
.prim(0x40, BasicType::Int, 3)
.string(0x50, "hello", true)
.root(HeapRootKind::ThreadObject, 0x10)
.root(HeapRootKind::JniGlobal, 0x30);
heap.bytes()
}
fn with_header(id_size: u8, body: &[u8]) -> Vec<u8> {
let mut bytes = MAGIC_1_0_2.to_vec();
bytes.extend_from_slice(&u32::from(id_size).to_be_bytes());
bytes.extend_from_slice(&1_700_000_000_000u64.to_be_bytes());
bytes.extend_from_slice(body);
bytes
}
proptest! {
#[test]
fn parser_never_panics(bytes in proptest::collection::vec(any::<u8>(), 0..4096)) {
let _ = parse_header(&bytes).and_then(|h| scan(&bytes, &h, &mut |_| Ok(())));
}
/// Random bytes alone fail the magic check, so this puts a valid header in front to make
/// `scan` and `scan_segment` run on arbitrary record streams.
#[test]
fn random_bytes_after_a_valid_header_never_panic(
id_size in prop_oneof![Just(4u8), Just(8u8)],
body in proptest::collection::vec(any::<u8>(), 0..4096),
) {
let bytes = with_header(id_size, &body);
let header = parse_header(&bytes).unwrap();
prop_assert_eq!(header.id_size, id_size);
let _ = scan(&bytes, &header, &mut |_| Ok(()));
let _ = scan_segment(&bytes, header.body_start..bytes.len(), id_size, &mut |_, _| Ok(()));
for at in header.body_start..bytes.len() {
let _ = record_at(&bytes, at, id_size);
}
}
/// Same, but every record has a correct length so the random bytes reach the heap-dump
/// segments and the sub-record parsers instead of stopping at the first bad length.
#[test]
fn random_records_after_a_valid_header_never_panic(
id_size in prop_oneof![Just(4u8), Just(8u8)],
records in proptest::collection::vec(
(
prop_oneof![Just(0x01u8), Just(0x02), Just(0x0C), Just(0x1C), Just(0x2C), any::<u8>()],
prop_oneof![
Just(0x20u8), Just(0x21), Just(0x22), Just(0x23), Just(0x01), Just(0x02), Just(0x03),
Just(0x04), Just(0x05), Just(0x06), Just(0x07), Just(0x08), Just(0xFF), any::<u8>()
],
proptest::collection::vec(any::<u8>(), 0..96),
),
0..12,
),
) {
let mut body = Vec::new();
for (tag, first, rest) in records {
let mut record_body = vec![first];
record_body.extend_from_slice(&rest);
body.push(tag);
body.extend_from_slice(&0u32.to_be_bytes());
body.extend_from_slice(&(record_body.len() as u32).to_be_bytes());
body.extend_from_slice(&record_body);
}
let bytes = with_header(id_size, &body);
let header = parse_header(&bytes).unwrap();
let mut segments = Vec::new();
let _ = scan(&bytes, &header, &mut |r| {
if let Record::Segment(s) = r {
segments.push(s);
}
Ok(())
});
for s in segments {
let _ = scan_segment(&bytes, s, id_size, &mut |_, _| Ok(()));
}
}
#[test]
fn mutated_dump_never_panics(flips in proptest::collection::vec((0usize..2048, any::<u8>()), 1..16)) {
let pristine = sample();
let header = parse_header(&pristine).unwrap();
let segment = segment_of(&pristine);
let mut bytes = pristine.clone();
for (pos, value) in flips {
let at = pos % bytes.len();
bytes[at] = value;
}
let _ = scan(&bytes, &header, &mut |_| Ok(()));
for id_size in [4, 8] {
let _ = scan_segment(&bytes, segment.clone(), id_size, &mut |_, _| Ok(()));
}
for at in 0..bytes.len() {
let _ = record_at(&bytes, at, 8);
}
}
}
}
+327
View File
@@ -0,0 +1,327 @@
//! Maps hprof object ids (addresses) to dense `u32` indices in ascending address order.
//!
//! Object ids are addresses and objects are at least 8 bytes apart, so one bit per 8-byte slot
//! ranks ids at about a byte per object. Addresses are grouped in 1 MiB chunks of
//! [`WORDS_PER_CHUNK`] words; only chunks holding an id get words, and a dense table maps a chunk
//! to its first word. The rank of an id is the number of set bits before its word plus the set
//! bits below it in the word. Ids that are not multiples of 8, or that span too wide an address
//! range for the table, use a sorted array and binary search instead.
mod builder;
pub(crate) use builder::ChunkHasher;
pub use builder::IdMapBuilder;
use crate::column::{Column, NONE};
const SLOT_SHIFT: u32 = 3;
const CHUNK_SLOT_BITS: u32 = 17;
const WORD_BITS: u64 = 64;
const WORDS_PER_CHUNK: usize = (1 << CHUNK_SLOT_BITS) / WORD_BITS as usize;
// The bit table is matched in hot paths; boxing the big variant would only add
// an indirection to save stack space `IdMap` never holds by value in bulk.
#[allow(clippy::large_enum_variant)]
enum Repr {
Bits {
/// Chunk number of the first table entry.
min_chunk: u64,
words: Column<u64>,
/// Set bits before each word.
wrank: Column<u32>,
/// Per chunk from `min_chunk` up: index of its first word, or `NONE` when it has no id.
table: Column<u32>,
/// Per chunk that has words, in word order: its offset from `min_chunk`.
chunk_offsets: Column<u32>,
len: u32,
},
Sorted(Column<u64>),
}
/// Ids ranked in ascending address order: the index of an object is its position in the order.
pub struct IdMap {
repr: Repr,
}
impl IdMap {
pub fn len(&self) -> u32 {
match &self.repr {
Repr::Bits { len, .. } => *len,
Repr::Sorted(ids) => ids.len() as u32,
}
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
/// The index of `id`, or `None` when it is not in the map.
pub fn rank(&self, id: u64) -> Option<u32> {
match &self.repr {
Repr::Sorted(ids) => ids.binary_search(&id).ok().map(|r| r as u32),
Repr::Bits { min_chunk, words, wrank, table, .. } => {
if id & 7 != 0 {
return None;
}
let slot = id >> SLOT_SHIFT;
let offset = (slot >> CHUNK_SLOT_BITS).checked_sub(*min_chunk)?;
let first = *table.get(usize::try_from(offset).ok()?)?;
if first == NONE {
return None;
}
let w = first as usize + (slot >> 6) as usize % WORDS_PER_CHUNK;
let bit = slot & 63;
let word = words[w];
if word >> bit & 1 == 0 {
return None;
}
Some(wrank[w] + (word & ((1u64 << bit) - 1)).count_ones())
}
}
}
/// The id with index `rank`. Panics when `rank >= len()`.
pub fn id_at(&self, rank: u32) -> u64 {
match &self.repr {
Repr::Sorted(ids) => ids[rank as usize],
Repr::Bits { words, wrank, len, .. } => {
assert!(rank < *len, "rank {rank} out of range for {len} ids");
let w = wrank.partition_point(|&before| before <= rank) - 1;
let mut word = words[w];
for _ in 0..rank - wrank[w] {
word &= word - 1;
}
(self.word_first_slot(w) + u64::from(word.trailing_zeros())) << SLOT_SHIFT
}
}
}
/// All ids in ascending order.
pub fn ids(&self) -> impl Iterator<Item = u64> + '_ {
Ids { map: self, next: 0, slot_base: 0, rest: 0 }
}
/// Bytes of all the columns, in RAM or on disk.
pub fn column_bytes(&self) -> u64 {
match &self.repr {
Repr::Sorted(ids) => std::mem::size_of_val(&**ids) as u64,
Repr::Bits { words, wrank, table, chunk_offsets, .. } => {
(std::mem::size_of_val(&**words)
+ std::mem::size_of_val(&**wrank)
+ std::mem::size_of_val(&**table)
+ std::mem::size_of_val(&**chunk_offsets)) as u64
}
}
}
/// Slot number of bit 0 of word `w` (bitmap layout only).
fn word_first_slot(&self, w: usize) -> u64 {
let Repr::Bits { min_chunk, chunk_offsets, .. } = &self.repr else {
unreachable!("sorted layout has no words")
};
let chunk = min_chunk + u64::from(chunk_offsets[w / WORDS_PER_CHUNK]);
(chunk << CHUNK_SLOT_BITS) + (w % WORDS_PER_CHUNK) as u64 * WORD_BITS
}
}
struct Ids<'a> {
map: &'a IdMap,
/// Next word (bitmap layout) or next id (sorted layout).
next: usize,
slot_base: u64,
/// Bits of the current word not yielded yet.
rest: u64,
}
impl Iterator for Ids<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
match &self.map.repr {
Repr::Sorted(ids) => {
let id = *ids.get(self.next)?;
self.next += 1;
Some(id)
}
Repr::Bits { words, .. } => loop {
if self.rest != 0 {
let bit = u64::from(self.rest.trailing_zeros());
self.rest &= self.rest - 1;
return Some((self.slot_base + bit) << SLOT_SHIFT);
}
self.rest = *words.get(self.next)?;
self.slot_base = self.map.word_first_slot(self.next);
self.next += 1;
},
}
}
}
#[cfg(test)]
mod tests {
use ide_core::CancellationToken;
use super::*;
use crate::column::Store;
/// Deterministic generator so failures replay; the crate has no `rand`.
struct XorShift(u64);
impl XorShift {
fn next(&mut self) -> u64 {
self.0 ^= self.0 << 13;
self.0 ^= self.0 >> 7;
self.0 ^= self.0 << 17;
self.0
}
}
fn build(ids: &[u64], store: &Store) -> IdMap {
let mut builder = IdMapBuilder::new();
for &id in ids {
builder.insert(id);
}
builder.build(store, &CancellationToken::new()).unwrap()
}
fn assert_matches_model(map: &IdMap, model: &[u64]) {
assert_eq!(map.len() as usize, model.len());
assert_eq!(map.ids().collect::<Vec<_>>(), model);
for (rank, &id) in model.iter().enumerate() {
assert_eq!(map.rank(id), Some(rank as u32), "rank of {id:#x}");
assert_eq!(map.id_at(rank as u32), id, "id at {rank}");
for near in [id.wrapping_sub(8), id.wrapping_add(8), id.wrapping_add(1), id.wrapping_add(4)] {
if model.binary_search(&near).is_err() {
assert_eq!(map.rank(near), None, "{near:#x} was never inserted");
}
}
}
}
#[test]
fn rank_follows_address_order() {
let map = build(&[0x2000, 0x1000, 0x1010, 0x7_0000_0000], &Store::Ram);
assert_eq!(map.len(), 4);
assert_eq!(
[0x1000, 0x1010, 0x2000, 0x7_0000_0000].map(|id| map.rank(id)),
[Some(0), Some(1), Some(2), Some(3)]
);
assert_eq!(map.rank(0x1008), None);
assert_eq!(map.rank(0), None);
assert_eq!(map.id_at(2), 0x2000);
assert_eq!(map.ids().collect::<Vec<_>>(), [0x1000, 0x1010, 0x2000, 0x7_0000_0000]);
}
#[test]
fn unaligned_ids_use_sorted_fallback() {
let map = build(&[0x2000, 0x1004, 0x1000, 0x1010, 0x7_0000_0000], &Store::Ram);
assert_eq!(
[0x1000, 0x1004, 0x1010, 0x2000, 0x7_0000_0000].map(|id| map.rank(id)),
[Some(0), Some(1), Some(2), Some(3), Some(4)]
);
assert_eq!(map.rank(0x1008), None);
assert_eq!(map.id_at(1), 0x1004);
}
#[test]
fn sparse_wide_span_uses_fallback() {
let map = build(&[0x1000 + (1 << 50), 0x1000], &Store::Ram);
assert_eq!(map.rank(0x1000), Some(0));
assert_eq!(map.rank(0x1000 + (1 << 50)), Some(1));
assert_eq!(map.rank(0x1008), None);
assert!(map.column_bytes() < 64, "a two-id map must not allocate a chunk table");
}
#[test]
fn ten_million_ids_cost_about_a_byte_each() {
let mut builder = IdMapBuilder::new();
for i in 0..10_000_000u64 {
builder.insert(0x7_0000_0000 + i * 32);
}
let map = builder.build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(map.len(), 10_000_000);
assert!(map.column_bytes() < 16_000_000, "{} bytes", map.column_bytes());
assert_eq!(map.rank(0x7_0000_0000 + 9_999_999 * 32), Some(9_999_999));
assert_eq!(map.id_at(5_000_000), 0x7_0000_0000 + 5_000_000 * 32);
}
#[test]
fn empty_map_and_duplicates() {
let map = build(&[], &Store::Ram);
assert!(map.is_empty());
assert_eq!(map.rank(0x1000), None);
assert_eq!(map.ids().count(), 0);
let map = build(&[0x1000, 0x1000, 0x1008, 0x1000], &Store::Ram);
assert_eq!(map.len(), 2);
assert_eq!(map.rank(0x1008), Some(1));
}
#[test]
fn chunk_boundaries_and_gaps_between_chunks() {
let ids = [0, 0xF_FFF8, 0x10_0000, 0x10_0008, 0x50_0000, 0x1_0000_0000];
let mut model = ids.to_vec();
model.sort_unstable();
let map = build(&ids, &Store::Ram);
assert_matches_model(&map, &model);
}
#[test]
fn ids_outside_the_populated_chunks_are_absent() {
let ids = [0x30_0000, 0x30_0008, 0x70_0000];
let map = build(&ids, &Store::Ram);
assert_matches_model(&map, &ids);
// Chunk 5 lies between the lowest and highest chunk but holds nothing.
assert_eq!(map.rank(0x50_0000), None);
// Chunk 1 is below the lowest populated chunk.
assert_eq!(map.rank(0x10_0000), None);
// Chunk 9 is past the last table entry.
assert_eq!(map.rank(0x900_0000), None);
assert_eq!(map.rank(u64::MAX - 7), None);
}
#[test]
fn highest_address_round_trips() {
let map = build(&[0xFFFF_FFFF_FFFF_FFF8], &Store::Ram);
assert_eq!(map.rank(0xFFFF_FFFF_FFFF_FFF8), Some(0));
assert_eq!(map.id_at(0), 0xFFFF_FFFF_FFFF_FFF8);
let map = build(&[0, 0xFFFF_FFFF_FFFF_FFF8], &Store::Ram);
assert_matches_model(&map, &[0, 0xFFFF_FFFF_FFFF_FFF8]);
}
#[test]
fn random_ids_match_a_sorted_model_in_ram_and_on_disk() {
for seed in 1..=8u64 {
let mut rng = XorShift(seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1);
let aligned = seed % 4 != 0;
let wide = seed % 3 == 0;
let ids: Vec<u64> = (0..3000)
.map(|_| {
let r = rng.next();
let base = if wide { (r >> 20) << 12 } else { 0x6_0000_0000 + (r % (6 << 20)) };
if aligned { base & !7 } else { base }
})
.collect();
let mut model = ids.clone();
model.sort_unstable();
model.dedup();
let dir = tempfile::tempdir().unwrap();
for store in [Store::Ram, Store::Dir(dir.path().to_path_buf())] {
assert_matches_model(&build(&ids, &store), &model);
}
}
}
#[test]
fn dir_store_keeps_the_index_columns_under_stable_names() {
let dir = tempfile::tempdir().unwrap();
let map = build(&[0x1000, 0x2000], &Store::Dir(dir.path().to_path_buf()));
let mut names: Vec<_> =
std::fs::read_dir(dir.path()).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
assert_eq!(names, ["ids.bits.u64", "ids.chunks.u32", "ids.table.u32", "ids.wrank.u32"]);
assert_eq!(map.rank(0x2000), Some(1));
let dir = tempfile::tempdir().unwrap();
build(&[0x1004], &Store::Dir(dir.path().to_path_buf()));
assert!(dir.path().join("ids.sorted.u64").exists());
}
}
+409
View File
@@ -0,0 +1,409 @@
//! Collects object ids in any order and lays them out as an [`IdMap`].
use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};
use std::hash::{BuildHasherDefault, Hasher};
use ide_core::CancellationToken;
use super::{CHUNK_SLOT_BITS, IdMap, Repr, SLOT_SHIFT, WORD_BITS, WORDS_PER_CHUNK};
use crate::column::{NONE, Store};
use crate::error::HeapError;
use crate::files::{IDS_BITS_FILE, IDS_CHUNKS_FILE, IDS_SORTED_FILE, IDS_TABLE_FILE, IDS_WRANK_FILE};
/// Widest chunk range the dense table covers (256 MiB of table at most).
const MAX_CHUNK_SPAN: u64 = 1 << 26;
/// Chunks whose first word index still fits in the table's `u32`.
const MAX_CHUNKS: usize = (NONE as usize - 1) / WORDS_PER_CHUNK;
/// Largest object count: `NONE` and the one after it stay free as sentinels for the columns above.
const MAX_IDS: u64 = NONE as u64 - 2;
const CHUNK_BYTES: u64 = (WORDS_PER_CHUNK * size_of::<u64>()) as u64;
/// The bitmaps may take this much before any id is inserted, then this many bytes per distinct id.
/// A heap that fits in 4 GiB of address space never reaches the base allowance whatever the
/// insertion order; sparse input (one id per chunk) is moved to plain `u64` ids, 8 bytes each,
/// after at most `BITMAP_BASE_BYTES + BITMAP_BYTES_PER_ID * ids` of bitmaps, so builder memory is
/// bounded by the number of ids rather than by the address range they spread over.
const BITMAP_BASE_BYTES: u64 = 64 << 20;
const BITMAP_BYTES_PER_ID: u64 = 32;
/// Longest stretch of work between two looks at the cancellation token.
const CHECK_EVERY: usize = 1 << 20;
/// Chunk numbers are already well spread by a multiplicative hash; SipHash would dominate the
/// per-insert cost on a 100M-object heap. The product keeps the low zero bits of its input, so
/// the high half is folded down for the bucket index. The graph builder keys class and string
/// ids (addresses) with it for the same reason.
#[derive(Default)]
pub(crate) struct ChunkHasher(u64);
impl Hasher for ChunkHasher {
fn finish(&self) -> u64 {
self.0 ^ (self.0 >> 32)
}
fn write(&mut self, bytes: &[u8]) {
for &b in bytes {
self.write_u64(self.0.rotate_left(8) ^ u64::from(b));
}
}
fn write_u64(&mut self, n: u64) {
self.0 = n.wrapping_mul(0x9E37_79B9_7F4A_7C15);
}
}
struct BitChunks {
/// Chunk number → position in `words`.
index: HashMap<u64, usize, BuildHasherDefault<ChunkHasher>>,
words: Vec<Box<[u64]>>,
min: u64,
max: u64,
/// The chunk of the previous insert; ids usually arrive in runs.
last: Option<(u64, usize)>,
distinct: u64,
}
impl BitChunks {
fn new() -> Self {
Self { index: HashMap::default(), words: Vec::new(), min: u64::MAX, max: 0, last: None, distinct: 0 }
}
/// `false` when the id's chunk does not fit the dense table or the bitmap budget; the id is
/// then not recorded and the caller moves to plain ids.
fn insert(&mut self, id: u64) -> bool {
let slot = id >> SLOT_SHIFT;
let chunk = slot >> CHUNK_SLOT_BITS;
let at = match self.last {
Some((c, at)) if c == chunk => at,
_ => match self.index.get(&chunk) {
Some(&at) => at,
None => {
let (min, max) = (self.min.min(chunk), self.max.max(chunk));
let bitmap_bytes = (self.words.len() as u64 + 1) * CHUNK_BYTES;
if self.words.len() >= MAX_CHUNKS
|| max - min >= MAX_CHUNK_SPAN
|| bitmap_bytes > BITMAP_BASE_BYTES + BITMAP_BYTES_PER_ID * self.distinct
{
return false;
}
(self.min, self.max) = (min, max);
self.words.push(vec![0u64; WORDS_PER_CHUNK].into_boxed_slice());
self.index.insert(chunk, self.words.len() - 1);
self.words.len() - 1
}
},
};
self.last = Some((chunk, at));
let word = &mut self.words[at][(slot >> 6) as usize % WORDS_PER_CHUNK];
let bit = 1u64 << (slot & 63);
if *word & bit == 0 {
*word |= bit;
self.distinct += 1;
}
true
}
/// `(chunk number, position in words)` in ascending address order.
fn sorted_chunks(&self) -> Vec<(u64, usize)> {
let mut chunks: Vec<_> = self.index.iter().map(|(&chunk, &at)| (chunk, at)).collect();
chunks.sort_unstable();
chunks
}
/// The recorded ids, ascending.
fn ids(&self) -> BitIds<'_> {
BitIds { chunks: self.sorted_chunks(), words: &self.words, chunk: 0, word: 0, base: 0, rest: 0 }
}
}
struct BitIds<'a> {
chunks: Vec<(u64, usize)>,
words: &'a [Box<[u64]>],
chunk: usize,
word: usize,
/// Slot number of bit 0 of the word `rest` came from.
base: u64,
rest: u64,
}
impl Iterator for BitIds<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
loop {
if self.rest != 0 {
let bit = u64::from(self.rest.trailing_zeros());
self.rest &= self.rest - 1;
return Some((self.base + bit) << SLOT_SHIFT);
}
let &(chunk, at) = self.chunks.get(self.chunk)?;
self.rest = self.words[at][self.word];
self.base = (chunk << CHUNK_SLOT_BITS) + self.word as u64 * WORD_BITS;
self.word += 1;
if self.word == WORDS_PER_CHUNK {
self.word = 0;
self.chunk += 1;
}
}
}
}
pub struct IdMapBuilder {
bits: BitChunks,
/// Set once an id cannot go into `bits`; every later id goes here, and `bits` keeps what it
/// had. Unsorted, with duplicates.
plain: Option<Vec<u64>>,
max_ids: u64,
}
impl Default for IdMapBuilder {
fn default() -> Self {
Self::new()
}
}
impl IdMapBuilder {
pub fn new() -> Self {
Self::with_limit(MAX_IDS)
}
/// `max_ids` is a parameter so tests can reach the too-large path without billions of inserts.
fn with_limit(max_ids: u64) -> Self {
Self { bits: BitChunks::new(), plain: None, max_ids }
}
/// Inserting an id twice is the same as once.
pub fn insert(&mut self, id: u64) {
match &mut self.plain {
Some(plain) => plain.push(id),
None => {
if id & 7 != 0 || !self.bits.insert(id) {
self.plain = Some(vec![id]);
}
}
}
}
/// Fails with [`HeapError::TooLarge`] when there are more than `u32::MAX - 2` distinct ids,
/// and with [`HeapError::Cancelled`] when `cancel` fires (checked at least every `1 << 20`
/// steps).
pub fn build(self, store: &Store, cancel: &CancellationToken) -> Result<IdMap, HeapError> {
match self.plain {
None if self.bits.words.is_empty() => sorted_map(store, &[]),
None => bit_map(&self.bits, store, self.max_ids, cancel),
Some(plain) => merged_map(&self.bits, plain, store, self.max_ids, cancel),
}
}
#[cfg(test)]
fn allocated_bytes(&self) -> u64 {
self.bits.words.len() as u64 * CHUNK_BYTES
+ self.plain.as_ref().map_or(0, |p| (p.capacity() * size_of::<u64>()) as u64)
}
}
fn check(cancel: &CancellationToken) -> Result<(), HeapError> {
if cancel.is_cancelled() { Err(HeapError::Cancelled) } else { Ok(()) }
}
fn sorted_map(store: &Store, ids: &[u64]) -> Result<IdMap, HeapError> {
let mut column = store.column(IDS_SORTED_FILE, ids.len(), 0u64)?;
column.copy_from_slice(ids);
Ok(IdMap { repr: Repr::Sorted(column) })
}
fn bit_map(bits: &BitChunks, store: &Store, max_ids: u64, cancel: &CancellationToken) -> Result<IdMap, HeapError> {
if bits.distinct > max_ids {
return Err(HeapError::TooLarge);
}
let chunks = bits.sorted_chunks();
let span = (bits.max - bits.min + 1) as usize;
let mut words = store.column(IDS_BITS_FILE, chunks.len() * WORDS_PER_CHUNK, 0u64)?;
let mut wrank = store.column(IDS_WRANK_FILE, chunks.len() * WORDS_PER_CHUNK, 0u32)?;
let mut table = store.column(IDS_TABLE_FILE, span, NONE)?;
let mut chunk_offsets = store.column(IDS_CHUNKS_FILE, chunks.len(), 0u32)?;
let mut before = 0u32;
for (ordinal, &(chunk, at)) in chunks.iter().enumerate() {
check(cancel)?;
let first = ordinal * WORDS_PER_CHUNK;
let offset = (chunk - bits.min) as usize;
table[offset] = first as u32;
chunk_offsets[ordinal] = offset as u32;
for (w, &word) in bits.words[at].iter().enumerate() {
words[first + w] = word;
wrank[first + w] = before;
before += word.count_ones();
}
}
Ok(IdMap { repr: Repr::Bits { min_chunk: bits.min, words, wrank, table, chunk_offsets, len: before } })
}
enum Source<'a> {
Bits(BitIds<'a>),
Run(std::iter::Copied<std::slice::Iter<'a, u64>>),
}
impl Iterator for Source<'_> {
type Item = u64;
fn next(&mut self) -> Option<u64> {
match self {
Source::Bits(it) => it.next(),
Source::Run(it) => it.next(),
}
}
}
/// Sorts `plain` in runs and merges the runs with the ids already in `bits` (itself a sorted
/// source), dropping duplicates. A single `sort_unstable` of a very large vector could not be
/// interrupted.
fn merged_map(
bits: &BitChunks,
mut plain: Vec<u64>,
store: &Store,
max_ids: u64,
cancel: &CancellationToken,
) -> Result<IdMap, HeapError> {
for run in plain.chunks_mut(CHECK_EVERY) {
check(cancel)?;
run.sort_unstable();
}
let mut sources = vec![Source::Bits(bits.ids())];
sources.extend(plain.chunks(CHECK_EVERY).map(|run| Source::Run(run.iter().copied())));
let mut heap = BinaryHeap::with_capacity(sources.len());
for (i, source) in sources.iter_mut().enumerate() {
if let Some(id) = source.next() {
heap.push(Reverse((id, i)));
}
}
let upper_bound = bits.distinct as usize + plain.len();
let mut merged = store.scratch::<u64>(upper_bound, 0)?;
let mut count = 0usize;
let mut steps = 0usize;
while let Some(Reverse((id, i))) = heap.pop() {
steps += 1;
if steps.is_multiple_of(CHECK_EVERY) {
check(cancel)?;
}
if let Some(next) = sources[i].next() {
heap.push(Reverse((next, i)));
}
if count > 0 && merged[count - 1] == id {
continue;
}
if count as u64 >= max_ids {
return Err(HeapError::TooLarge);
}
merged[count] = id;
count += 1;
}
let mut column = store.column(IDS_SORTED_FILE, count, 0u64)?;
for (to, from) in column.chunks_mut(CHECK_EVERY).zip(merged[..count].chunks(CHECK_EVERY)) {
check(cancel)?;
to.copy_from_slice(from);
}
Ok(IdMap { repr: Repr::Sorted(column) })
}
#[cfg(test)]
mod tests {
use super::*;
fn insert_all(builder: &mut IdMapBuilder, ids: &[u64]) {
for &id in ids {
builder.insert(id);
}
}
#[test]
fn too_many_ids_is_too_large() {
// Dense layout, then the merged layout (one unaligned id); the duplicate does not count.
for unaligned in [false, true] {
let filled = |limit| {
let mut builder = IdMapBuilder::with_limit(limit);
insert_all(&mut builder, &[0x1000, 0x1008, 0x1010, 0x1010]);
if unaligned {
builder.insert(0x1_0004);
}
builder
};
let distinct = if unaligned { 4 } else { 3 };
let map = filled(distinct).build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(u64::from(map.len()), distinct);
let err = filled(distinct - 1).build(&Store::Ram, &CancellationToken::new()).err().unwrap();
assert!(matches!(err, HeapError::TooLarge), "{err}");
}
}
#[test]
fn cancelled_build_returns_cancelled() {
let cancelled = CancellationToken::new();
cancelled.cancel();
let mut dense = IdMapBuilder::new();
insert_all(&mut dense, &[0x1000, 0x1008, 0x20_0000]);
assert!(matches!(dense.build(&Store::Ram, &cancelled), Err(HeapError::Cancelled)));
let mut merged = IdMapBuilder::new();
insert_all(&mut merged, &[0x1000, 0x1004, 0x20_0000]);
assert!(matches!(merged.build(&Store::Ram, &cancelled), Err(HeapError::Cancelled)));
let mut ok = IdMapBuilder::new();
insert_all(&mut ok, &[0x1000, 0x1004]);
assert_eq!(ok.build(&Store::Ram, &CancellationToken::new()).unwrap().len(), 2);
}
#[test]
fn sparse_ids_do_not_amplify_builder_memory() {
const IDS: u64 = 2_000_000;
let mut builder = IdMapBuilder::new();
let mut peak = 0;
for i in 0..IDS {
builder.insert(0x1_0000_0000 + (i << 20));
if i % 1024 == 0 {
peak = peak.max(builder.allocated_bytes());
}
}
peak = peak.max(builder.allocated_bytes());
// Bitmaps up to the budget, plus the plain ids' vector, which may have doubled its capacity.
let budget = BITMAP_BASE_BYTES + (BITMAP_BYTES_PER_ID + 2 * size_of::<u64>() as u64) * IDS;
assert!(peak <= budget, "peak {peak} bytes, budget {budget}");
// One bitmap per id would have taken 16 KiB each.
assert!(peak < IDS * CHUNK_BYTES / 100, "peak {peak} bytes");
assert!(builder.plain.is_some(), "sparse input must have left the bitmaps");
let map = builder.build(&Store::Ram, &CancellationToken::new()).unwrap();
assert_eq!(map.len() as u64, IDS);
assert_eq!(map.rank(0x1_0000_0000 + (1_234_567 << 20)), Some(1_234_567));
assert_eq!(map.rank(0x1_0000_0000 + (1_234_567 << 20) + 8), None);
}
#[test]
fn dense_input_in_any_order_stays_in_bitmaps() {
// 4 GiB of address space is the base allowance; random order must not trip the budget.
let mut state = 0x2545_F491_4F6C_DD1Du64;
let mut builder = IdMapBuilder::new();
for _ in 0..300_000 {
state ^= state << 13;
state ^= state >> 7;
state ^= state << 17;
builder.insert((0x7_0000_0000 + (state % (4 << 30))) & !7);
}
assert!(builder.plain.is_none());
assert!(builder.allocated_bytes() <= BITMAP_BASE_BYTES);
}
#[test]
fn chunk_hasher_spreads_chunks_that_share_low_zero_bits() {
let buckets: std::collections::HashSet<u64> = (0..1024u64)
.map(|i| {
let mut hasher = ChunkHasher::default();
hasher.write_u64(i << 12);
hasher.finish() & 1023
})
.collect();
assert!(buckets.len() > 500, "{} buckets for 1024 keys", buckets.len());
}
}
+109
View File
@@ -0,0 +1,109 @@
//! The result of analyzing a heap dump: per-object columns, the class table and the summary.
mod fields;
mod path;
mod queries;
mod rows;
use std::collections::HashMap;
use std::path::Path;
use memmap2::Mmap;
use serde::{Deserialize, Serialize};
use crate::column::Column;
use crate::error::HeapError;
use crate::files::SUMMARY_FILE;
use crate::graph::ClassInfo;
use crate::hprof::HeapRootKind;
use crate::ids::IdMap;
use crate::shallow::SizeSource;
pub(crate) use self::fields::{id_value, instance_fields};
pub use self::rows::{
HeapClassRow, HeapEdge, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapPathStep, HeapRootPath, MAX_PAGE,
};
/// Counts over instances and arrays; class objects (class dumps) are left out.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapSummary {
/// `reachable_objects + unreachable_objects`.
pub objects: u64,
pub reachable_objects: u64,
pub reachable_bytes: u64,
pub unreachable_objects: u64,
pub unreachable_bytes: u64,
/// Class dumps in the dump.
pub classes: u32,
pub weak_edges_skipped: u64,
pub size_source: SizeSource,
pub analysis_ms: u32,
}
/// One row of the class table, for one type slot.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ClassStat {
/// [`crate::display_name`] of the class (`long[]` for a primitive array slot).
pub name: String,
/// Reachable instances.
pub count: u64,
/// Shallow bytes of the reachable instances.
pub shallow: u64,
/// Bytes the reachable instances retain together: the sum of the retained sizes of those
/// with no instance of the same type above them in the dominator tree.
pub retained: u64,
pub unreachable_count: u64,
}
/// Columns are indexed by node: objects `0..root`, then the super-root `root`. Columns of the
/// objects only (`types`, `offsets`) have no entry for the root.
pub struct HeapIndex {
/// The dump, for [`crate::record_at`] at `offsets[v]`.
pub hprof: Mmap,
pub ids: IdMap,
/// Type slot of each object, as in [`crate::HeapGraph::types`].
pub types: Column<u32>,
pub offsets: Column<u64>,
/// Shallow size in units of 8 bytes, rounded up.
pub shallow8: Column<u32>,
/// Retained bytes; 0 for unreachable objects, every reachable byte for the root.
pub retained: Column<u64>,
/// Immediate dominator; [`crate::NONE`] for unreachable objects, `root` for the root.
pub idom: Column<u32>,
/// Children of node `v` in the dominator tree are `kids[kids_start[v]..kids_start[v + 1]]`,
/// by retained size descending, then by node.
pub kids_start: Column<u32>,
pub kids: Column<u32>,
/// Parent on a shortest path from the root (BFS over strong edges); [`crate::NONE`] for
/// unreachable objects, `root` for the root.
pub gc_parent: Column<u32>,
/// By type slot: class slots, then the primitive array slots.
pub classes: Vec<ClassStat>,
/// By class slot, as in [`crate::HeapGraph::classes`].
pub class_info: Vec<ClassInfo>,
/// Type slot of class objects.
pub class_slot: u32,
/// Object → kind and thread serial of its first root record.
pub roots: HashMap<u32, (HeapRootKind, Option<u32>)>,
/// Thread serial → thread object.
pub thread_objects: HashMap<u32, u32>,
pub id_size: u8,
pub summary: HeapSummary,
}
impl HeapIndex {
/// The super-root, whose children in both trees are the rooted objects.
pub fn root(&self) -> u32 {
self.ids.len()
}
}
pub(crate) fn write_summary(dir: &Path, summary: &HeapSummary) -> Result<(), HeapError> {
let json = serde_json::to_vec_pretty(summary).map_err(std::io::Error::from)?;
std::fs::write(dir.join(SUMMARY_FILE), json)?;
Ok(())
}
#[cfg(test)]
mod tests;
+41
View File
@@ -0,0 +1,41 @@
//! Named field values of an instance record.
use super::HeapIndex;
use crate::hprof::BasicType;
pub(crate) struct FieldValue<'a> {
pub(crate) name: &'a str,
pub(crate) ty: BasicType,
/// Big-endian, as in the dump.
pub(crate) bytes: &'a [u8],
/// `java.lang.ref.Reference.referent`, which the graph leaves out.
pub(crate) weak: bool,
}
/// The fields of an instance of class slot `class` whose record holds `data`: the class's own,
/// then each superclass's, as HotSpot lays them out. Stops where `data` ends.
pub(crate) fn instance_fields<'a>(index: &'a HeapIndex, class: u32, data: &'a [u8]) -> Vec<FieldValue<'a>> {
let mut out = Vec::new();
let mut base = 0usize;
let mut cursor = Some(class);
// Super chains end (their cycles are cut), so this is only a bound on corrupt input.
for _ in 0..index.class_info.len() {
let Some(info) = cursor.and_then(|c| index.class_info.get(c as usize)) else { break };
for (k, (name, ty)) in info.fields.iter().enumerate() {
let size = ty.hprof_size(index.id_size);
let Some(bytes) = base.checked_add(size).and_then(|end| data.get(base..end)) else { return out };
out.push(FieldValue { name, ty: *ty, bytes, weak: info.weak_field == Some(k) });
base += size;
}
cursor = info.super_index;
}
out
}
/// A 4- or 8-byte big-endian id.
pub(crate) fn id_value(bytes: &[u8]) -> Option<u64> {
match *bytes {
[a, b, c, d] => Some(u64::from(u32::from_be_bytes([a, b, c, d]))),
_ => Some(u64::from_be_bytes(bytes.try_into().ok()?)),
}
}
+105
View File
@@ -0,0 +1,105 @@
//! Paths to GC roots, along the BFS tree of `gc_parent`.
use super::{HeapEdge, HeapIndex, HeapObjectRef, HeapPathStep, HeapRootPath, id_value, instance_fields};
use crate::column::NONE;
use crate::hprof::{BasicType, HeapRecord, HeapRootKind};
use crate::strings::{char_array_text, string_preview};
const THREAD_NAME_CHARS: usize = 100;
/// Steps a long path keeps at each end: one deep in a long linked list would otherwise send
/// every node over IPC.
const PATH_END_STEPS: u32 = 500;
impl HeapIndex {
/// From `object` up to its GC root, each step with how the next one holds it; past 1000
/// steps, only the 500 at each end. `None` for an unreachable object or a ref that names no
/// object.
pub fn path_to_root(&self, object: HeapObjectRef) -> Option<HeapRootPath> {
let root = self.root();
let mut v = object.0;
if v >= root || self.gc_parent[v as usize] == NONE {
return None;
}
let len = self.path_len(v)?;
let end = PATH_END_STEPS;
let cut = len > 2 * end;
let mut steps = Vec::with_capacity(len.min(2 * end) as usize);
for i in 0..len {
let parent = self.gc_parent[v as usize];
if !cut || i < end || i >= len - end {
let via = if parent == root { self.root_edge(v) } else { self.edge(parent, v) };
steps.push(HeapPathStep { object: self.object_row(HeapObjectRef(v))?, via });
}
v = parent;
}
let omitted = if cut { len - 2 * end } else { 0 };
Some(HeapRootPath { steps, omitted, omitted_after: if cut { end } else { 0 } })
}
/// Steps from `v` to the root; `None` when `gc_parent` leads nowhere.
fn path_len(&self, mut v: u32) -> Option<u32> {
let root = self.root();
// A BFS tree has no cycles; the bound only stops a corrupt column from looping.
for len in 1..=root {
let parent = self.gc_parent[v as usize];
if parent == root {
return Some(len);
}
if parent > root {
return None;
}
v = parent;
}
None
}
fn root_edge(&self, v: u32) -> HeapEdge {
let (kind, serial) = self.roots.get(&v).copied().unwrap_or((HeapRootKind::Unknown, None));
let thread = serial.and_then(|s| self.thread_objects.get(&s)).and_then(|&t| self.thread_name(t));
HeapEdge::Root { root: kind, thread }
}
/// `Thread.name`: a String, or a `char[]` before JDK 9.
fn thread_name(&self, thread: u32) -> Option<String> {
let Some(HeapRecord::Instance { data, .. }) = self.record(thread) else { return None };
let fields = instance_fields(self, *self.types.get(thread as usize)?, data);
let name = fields.iter().find(|f| f.name == "name" && f.ty == BasicType::Object)?;
let target = self.ids.rank(id_value(name.bytes)?)?;
string_preview(self, target, THREAD_NAME_CHARS).or_else(|| char_array_text(self, target, THREAD_NAME_CHARS))
}
/// How `parent` refers to `child`, checked in the order the graph's edges were made. A strong
/// edge from `parent` always exists; the empty field name only covers a dump that changed.
fn edge(&self, parent: u32, child: u32) -> HeapEdge {
self.find_edge(parent, self.ids.id_at(child)).unwrap_or(HeapEdge::Field { name: String::new() })
}
fn find_edge(&self, parent: u32, child_id: u64) -> Option<HeapEdge> {
match self.record(parent)? {
HeapRecord::Instance { data, .. } => {
let class = self.types[parent as usize];
let field = instance_fields(self, class, data)
.into_iter()
.find(|f| !f.weak && f.ty == BasicType::Object && id_value(f.bytes) == Some(child_id));
match field {
Some(f) => Some(HeapEdge::Field { name: f.name.to_owned() }),
None => (self.class_info.get(class as usize)?.id == child_id).then_some(HeapEdge::ClassOf),
}
}
HeapRecord::ObjectArray { data, .. } => {
let index =
data.chunks_exact(usize::from(self.id_size)).position(|item| id_value(item) == Some(child_id))?;
Some(HeapEdge::Element { index: u32::try_from(index).ok()? })
}
HeapRecord::Class(dump) if dump.super_id == child_id => Some(HeapEdge::SuperClass),
HeapRecord::Class(dump) if dump.loader_id == child_id => Some(HeapEdge::ClassLoader),
HeapRecord::Class(_) => {
let info = self.class_info.get(self.class_of_object(parent)?)?;
let (name, _) = info.statics.iter().find(|(_, object)| *object == child_id)?;
Some(HeapEdge::Static { name: name.clone() })
}
HeapRecord::Root { .. } | HeapRecord::PrimitiveArray { .. } => None,
}
}
}
+164
View File
@@ -0,0 +1,164 @@
//! Class table rows, pages of the dominator tree and single object rows.
use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};
use super::{HeapClassRow, HeapIndex, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapSummary, MAX_PAGE};
use crate::column::NONE;
use crate::hprof::HeapRecord;
use crate::strings::string_preview;
const PREVIEW_CHARS: usize = 100;
impl HeapIndex {
pub fn summary(&self) -> &HeapSummary {
&self.summary
}
/// Types that have objects, by retained size descending, then by name.
pub fn class_rows(&self) -> Vec<HeapClassRow> {
let class_slots = &self.classes[..self.class_info.len().min(self.classes.len())];
let mut names: HashMap<&str, u32> = HashMap::with_capacity(class_slots.len());
for class in class_slots {
*names.entry(class.name.as_str()).or_default() += 1;
}
let mut rows: Vec<HeapClassRow> = self
.classes
.iter()
.enumerate()
.filter(|(_, c)| c.count > 0 || c.unreachable_count > 0)
.map(|(slot, c)| HeapClassRow {
class_index: slot as u32,
name: c.name.clone(),
count: c.count,
shallow: c.shallow,
retained: c.retained,
unreachable_count: c.unreachable_count,
// Primitive array slots have no class dump and so no second loader.
ambiguous: slot < class_slots.len() && names[c.name.as_str()] > 1,
})
.collect();
rows.sort_by(|a, b| b.retained.cmp(&a.retained).then_with(|| a.name.cmp(&b.name)));
rows
}
/// A page of objects by retained size descending, then by index:
///
/// - no `parent`, no `class_index`: the children of the super-root, class objects left out;
/// - `parent`: its children in the dominator tree, of type `class_index` if given;
/// - `class_index` only: every reachable object of that type.
///
/// `limit` is clamped to [`MAX_PAGE`]. A `parent` or `class_index` that names nothing gives an
/// empty page.
pub fn dominated(
&self,
parent: Option<HeapObjectRef>,
class_index: Option<u32>,
offset: u32,
limit: u32,
) -> HeapObjectPage {
let limit = limit.min(MAX_PAGE);
let root = self.root();
if class_index.is_some_and(|c| c as usize >= self.classes.len()) {
return HeapObjectPage { rows: Vec::new(), total: 0 };
}
match (parent, class_index) {
(Some(HeapObjectRef(p)), _) if p >= root => HeapObjectPage { rows: Vec::new(), total: 0 },
(Some(HeapObjectRef(p)), class) => self.page(
self.children(p).filter(|&v| class.is_none_or(|c| self.types[v as usize] == c)),
offset,
limit,
),
(None, None) => {
self.page(self.children(root).filter(|&v| self.types[v as usize] != self.class_slot), offset, limit)
}
(None, Some(class)) => self.largest_of(class, offset, limit),
}
}
/// `None` for a ref that names no object, the super-root included.
pub fn object_row(&self, object: HeapObjectRef) -> Option<HeapObjectRow> {
let v = object.0;
if v >= self.root() {
return None;
}
let i = v as usize;
let slot = self.types[i];
let preview = if slot == self.class_slot {
self.class_of_object(v).and_then(|c| self.classes.get(c)).map(|c| c.name.clone())
} else {
string_preview(self, v, PREVIEW_CHARS)
};
Some(HeapObjectRow {
object,
class_name: self.classes.get(slot as usize).map(|c| c.name.clone()).unwrap_or_default(),
address: format!("{:#x}", self.ids.id_at(v)),
shallow: u64::from(self.shallow8[i]) * 8,
retained: self.retained[i],
has_children: self.kids_start[i + 1] > self.kids_start[i],
preview,
})
}
/// The record of object `v`, or `None` when it cannot be read.
pub(crate) fn record(&self, v: u32) -> Option<HeapRecord<'_>> {
let at = usize::try_from(*self.offsets.get(v as usize)?).ok()?;
crate::hprof::record_at(&self.hprof, at, self.id_size).ok()
}
/// The class slot a class object stands for.
pub(crate) fn class_of_object(&self, v: u32) -> Option<usize> {
if v >= self.root() {
return None;
}
let id = self.ids.id_at(v);
self.class_info.iter().position(|c| c.id == id && c.id != 0)
}
/// Children of node `v` in the dominator tree, already in page order.
fn children(&self, v: u32) -> impl Iterator<Item = u32> + '_ {
let v = v as usize;
self.kids[self.kids_start[v] as usize..self.kids_start[v + 1] as usize].iter().copied()
}
fn page(&self, objects: impl Iterator<Item = u32>, offset: u32, limit: u32) -> HeapObjectPage {
let mut rows = Vec::new();
let mut total = 0u32;
for v in objects {
if total >= offset && total - offset < limit {
rows.extend(self.object_row(HeapObjectRef(v)));
}
total = total.saturating_add(1);
}
HeapObjectPage { rows, total }
}
/// Scans every object: the top `offset + limit` stay in a heap whose top is the smallest.
fn largest_of(&self, class: u32, offset: u32, limit: u32) -> HeapObjectPage {
let count = self.classes[class as usize].count;
let total = u32::try_from(count).unwrap_or(u32::MAX);
let keep = match offset.checked_add(limit) {
Some(keep) if offset < total && limit > 0 => keep.min(total) as usize,
_ => return HeapObjectPage { rows: Vec::new(), total },
};
let mut top: BinaryHeap<Reverse<(u64, Reverse<u32>)>> = BinaryHeap::with_capacity(keep + 1);
for v in 0..self.root() {
let i = v as usize;
if self.types[i] != class || self.idom[i] == NONE {
continue;
}
top.push(Reverse((self.retained[i], Reverse(v))));
if top.len() > keep {
top.pop();
}
}
// Ascending `Reverse` order is descending retained, then ascending index.
let rows = top
.into_sorted_vec()
.into_iter()
.skip(offset as usize)
.filter_map(|Reverse((_, Reverse(v)))| self.object_row(HeapObjectRef(v)))
.collect();
HeapObjectPage { rows, total }
}
}
+95
View File
@@ -0,0 +1,95 @@
//! What the index queries return over IPC.
use serde::{Deserialize, Serialize};
use crate::hprof::HeapRootKind;
/// Most rows one page of [`crate::HeapIndex::dominated`] returns.
pub const MAX_PAGE: u32 = 500;
/// An object of one snapshot: its index in the [`crate::HeapIndex`] columns. Any value may
/// arrive over IPC; queries answer an index that names no object with nothing.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, specta::Type)]
#[serde(transparent)]
pub struct HeapObjectRef(pub u32);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapClassRow {
/// Type slot, as `classIndex` in [`crate::HeapIndex::dominated`].
pub class_index: u32,
pub name: String,
pub count: u64,
pub shallow: u64,
pub retained: u64,
pub unreachable_count: u64,
/// Another class has the same name (several class loaders): nothing joined by name, such as a
/// second snapshot, can tell them apart.
pub ambiguous: bool,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapObjectRow {
pub object: HeapObjectRef,
pub class_name: String,
/// The object's hprof id in hex, `0x1300`.
pub address: String,
pub shallow: u64,
pub retained: u64,
/// Has children in the dominator tree.
pub has_children: bool,
/// The text of a `java.lang.String`, or the name of the class a class object stands for.
pub preview: Option<String>,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapObjectPage {
pub rows: Vec<HeapObjectRow>,
/// Rows of every page together.
pub total: u32,
}
/// How an object is held by the next step of a path to a GC root.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapEdge {
/// An instance field; the name is empty when the dump does not show which one.
Field {
name: String,
},
Element {
index: u32,
},
Static {
name: String,
},
/// From an instance to its class.
ClassOf,
SuperClass,
ClassLoader,
/// `thread` is the name of the thread whose root it is, when the dump has it.
Root {
root: HeapRootKind,
thread: Option<String>,
},
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapPathStep {
pub object: HeapObjectRow,
/// How the next step holds this object; on the last step, the root.
pub via: HeapEdge,
}
/// A path from an object up to its GC root. A long one keeps only the steps nearest each end.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapRootPath {
pub steps: Vec<HeapPathStep>,
/// Steps left out between `steps[omitted_after - 1]` and `steps[omitted_after]`; 0 for none.
pub omitted: u32,
pub omitted_after: u32,
}
+336
View File
@@ -0,0 +1,336 @@
use super::*;
use crate::hprof::BasicType;
use crate::test_index::{analyzed, analyzed_bytes, obj, slot};
use crate::testing::{HprofWriter, MiniHeap, sample, sample_with_string};
const H: u64 = 0x1000;
const N1: u64 = 0x1100;
const N3: u64 = 0x1300;
const L: u64 = 0x1400;
const W: u64 = 0x1600;
const N4: u64 = 0x1700;
fn addresses(page: &HeapObjectPage) -> Vec<&str> {
page.rows.iter().map(|r| r.address.as_str()).collect()
}
fn retained(page: &HeapObjectPage) -> Vec<u64> {
page.rows.iter().map(|r| r.retained).collect()
}
fn path(index: &HeapIndex, addr: u64) -> Option<Vec<(String, HeapEdge)>> {
let path = index.path_to_root(obj(index, addr))?;
assert_eq!((path.omitted, path.omitted_after), (0, 0));
Some(path.steps.into_iter().map(|s| (s.object.address, s.via)).collect())
}
fn field(name: &str) -> HeapEdge {
HeapEdge::Field { name: name.to_owned() }
}
fn root(root: HeapRootKind, thread: Option<&str>) -> HeapEdge {
HeapEdge::Root { root, thread: thread.map(str::to_owned) }
}
#[test]
fn top_level_dominators_sorted_by_retained() {
let index = analyzed(&sample());
let page = index.dominated(None, None, 0, 10);
assert_eq!(addresses(&page), ["0x1000", "0x1300", "0x1600"]);
assert_eq!(retained(&page), [72, 56, 24]);
// The class objects, rooted as sticky classes, are left out.
assert_eq!(page.total, 3);
assert_eq!(addresses(&index.dominated(None, None, 1, 1)), ["0x1300"]);
}
#[test]
fn children_of_holder() {
let index = analyzed(&sample());
let page = index.dominated(Some(obj(&index, H)), None, 0, 10);
assert_eq!(addresses(&page), ["0x1100", "0x1200"]);
assert_eq!(retained(&page), [24, 24]);
assert_eq!(page.total, 2);
assert!(page.rows.iter().all(|r| !r.has_children && r.class_name == "Node"));
let n3 = index.object_row(obj(&index, N3)).unwrap();
assert!(n3.has_children);
assert_eq!(index.dominated(Some(n3.object), None, 0, 10).rows[0].class_name, "long[]");
}
#[test]
fn largest_of_class() {
let index = analyzed(&sample());
let node = slot(&index, "Node");
let page = index.dominated(None, Some(node), 0, 2);
assert_eq!(addresses(&page), ["0x1300", "0x1100"]);
assert_eq!(retained(&page), [56, 24]);
assert_eq!(page.total, 3);
assert_eq!(addresses(&index.dominated(None, Some(node), 2, 2)), ["0x1200"]);
assert!(index.dominated(None, Some(node), 3, 2).rows.is_empty());
// Within a parent's children.
assert_eq!(addresses(&index.dominated(Some(obj(&index, H)), Some(node), 1, 5)), ["0x1200"]);
assert_eq!(index.dominated(Some(obj(&index, H)), Some(slot(&index, "Holder")), 0, 5).total, 0);
}
#[test]
fn object_row_describes_object() {
let index = analyzed(&sample());
let row = index.object_row(obj(&index, N3)).unwrap();
assert_eq!(
row,
HeapObjectRow {
object: obj(&index, N3),
class_name: "Node".to_owned(),
address: "0x1300".to_owned(),
shallow: 24,
retained: 56,
has_children: true,
preview: None,
}
);
let unreachable = index.object_row(obj(&index, N4)).unwrap();
assert_eq!((unreachable.retained, unreachable.has_children), (0, false));
let holder = &index.class_info[slot(&index, "Holder") as usize];
let class_object = obj(&index, holder.id);
let row = index.object_row(class_object).unwrap();
assert_eq!((row.class_name.as_str(), row.preview.as_deref()), ("java.lang.Class", Some("Holder")));
}
#[test]
fn refs_that_name_no_object_answer_nothing() {
let index = analyzed(&sample());
let node = slot(&index, "Node");
for bad in [index.root(), index.root() + 1, u32::MAX] {
let bad = HeapObjectRef(bad);
assert_eq!(index.object_row(bad), None);
assert_eq!(index.path_to_root(bad), None);
assert_eq!(index.dominated(Some(bad), None, 0, 10), HeapObjectPage { rows: Vec::new(), total: 0 });
assert_eq!(index.dominated(Some(bad), Some(node), 0, 10).total, 0);
}
for bad in [index.classes.len() as u32, u32::MAX] {
assert_eq!(index.dominated(None, Some(bad), 0, 10).total, 0);
}
}
#[test]
fn path_to_root_names_fields_and_root() {
let index = analyzed(&sample());
assert_eq!(
path(&index, L),
Some(vec![("0x1400".to_owned(), field("next")), ("0x1300".to_owned(), root(HeapRootKind::JavaFrame, None))])
);
assert_eq!(
path(&index, N1),
Some(vec![("0x1100".to_owned(), field("a")), ("0x1000".to_owned(), root(HeapRootKind::JniGlobal, None))])
);
assert_eq!(path(&index, W), Some(vec![("0x1600".to_owned(), root(HeapRootKind::JniGlobal, None))]));
assert_eq!(path(&index, N4), None);
}
/// A rooted linked list of `len` Nodes from `0x100000`, each held by the next one's `next`.
fn chain(len: u64) -> HeapIndex {
let mut heap = sample();
for i in 0..len {
let next = if i == 0 { 0 } else { 0x10_0000 + (i - 1) * 0x20 };
heap.object(0x10_0000 + i * 0x20, "Node", &[("next", next)]);
}
heap.root(HeapRootKind::JniGlobal, 0x10_0000 + (len - 1) * 0x20);
analyzed(&heap)
}
#[test]
fn path_of_an_object_deep_in_a_linked_list_keeps_both_ends() {
let index = chain(3000);
let path = index.path_to_root(obj(&index, 0x10_0000)).unwrap();
assert_eq!((path.steps.len(), path.omitted, path.omitted_after), (1000, 2000, 500));
let address = |i: usize| path.steps[i].object.address.clone();
assert_eq!([address(0), address(499)], ["0x100000", "0x103e60"]);
// The step after the gap is the one 2000 nodes further up.
assert_eq!([address(500), address(999)], ["0x113880", "0x1176e0"]);
assert_eq!(path.steps[999].via, root(HeapRootKind::JniGlobal, None));
assert!(path.steps[..999].iter().all(|s| s.via == field("next")));
}
#[test]
fn path_of_exactly_a_thousand_steps_is_whole() {
let index = chain(1000);
let path = index.path_to_root(obj(&index, 0x10_0000)).unwrap();
assert_eq!((path.steps.len(), path.omitted, path.omitted_after), (1000, 0, 0));
}
#[test]
fn path_skips_the_weak_referent() {
let mut heap = sample();
heap.object(0x1c00, "java.lang.ref.WeakReference", &[("referent", 0x1d00), ("queue", 0x1d00)])
.object(0x1d00, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1c00);
let index = analyzed(&heap);
assert_eq!(path(&index, 0x1d00).unwrap()[0].1, field("queue"));
}
/// Kid extends Base, loaded by loader `0x300`; Base's static `cache` holds X; K is a rooted Kid;
/// `0x700` is a rooted `Object[] { null, Y }`. No class is rooted on its own.
fn class_edges_heap() -> Vec<u8> {
let mut w = HprofWriter::new();
let names = [(1, "Base"), (2, "Kid"), (3, "Ldr"), (4, "[Ljava/lang/Object;"), (5, "cache")];
for (id, name) in names {
w.string(id, name);
}
for (class, name) in [(0x100, 1), (0x200, 2), (0x400, 3), (0x800, 4)] {
w.load_class(class, name);
}
w.class(0x100, 0, &[], &[(5, 0x500)])
.class_with_loader(0x200, 0x100, 0x300, &[], &[])
.class(0x400, 0, &[], &[])
.class(0x800, 0, &[], &[])
.instance(0x300, 0x400, &[])
.instance(0x500, 0x400, &[])
.instance(0x600, 0x200, &[])
.obj_array(0x700, 0x800, &[0, 0x900])
.instance(0x900, 0x400, &[])
.root(HeapRootKind::JniGlobal, 0x600)
.root(HeapRootKind::JniGlobal, 0x700);
w.finish()
}
#[test]
fn path_names_statics_supers_loaders_classes_and_elements() {
let index = analyzed_bytes(&class_edges_heap());
let k = ("0x600".to_owned(), root(HeapRootKind::JniGlobal, None));
assert_eq!(
path(&index, 0x500),
Some(vec![
("0x500".to_owned(), HeapEdge::Static { name: "cache".to_owned() }),
("0x100".to_owned(), HeapEdge::SuperClass),
("0x200".to_owned(), HeapEdge::ClassOf),
k.clone(),
])
);
assert_eq!(
path(&index, 0x300),
Some(vec![("0x300".to_owned(), HeapEdge::ClassLoader), ("0x200".to_owned(), HeapEdge::ClassOf), k])
);
assert_eq!(
path(&index, 0x900),
Some(vec![
("0x900".to_owned(), HeapEdge::Element { index: 1 }),
("0x700".to_owned(), root(HeapRootKind::JniGlobal, None))
])
);
}
fn with_main_thread(mut heap: MiniHeap, name: impl FnOnce(&mut MiniHeap, u64)) -> MiniHeap {
heap.class("java.lang.Thread", Some("java.lang.Object"), &[("name", BasicType::Object)]);
name(&mut heap, 0x2000);
heap.object(0x2100, "java.lang.Thread", &[("name", 0x2000)]).root_in_thread(HeapRootKind::ThreadObject, 0x2100, 1);
heap
}
#[test]
fn thread_name_on_java_frame_root() {
let heap = with_main_thread(sample_with_string(), |h, at| {
h.string(at, "main", true);
});
let index = analyzed(&heap);
let steps = path(&index, L).unwrap();
assert_eq!(steps.last().unwrap().1, root(HeapRootKind::JavaFrame, Some("main")));
// A JNI global carries no thread.
assert_eq!(path(&index, H).unwrap()[0].1, root(HeapRootKind::JniGlobal, None));
}
#[test]
fn jdk8_thread_name_is_a_char_array() {
let heap = with_main_thread(sample(), |h, at| {
let utf16: Vec<u8> = "main".encode_utf16().flat_map(u16::to_be_bytes).collect();
h.prim_bytes(at, BasicType::Char, &utf16);
});
let index = analyzed(&heap);
assert_eq!(path(&index, N3).unwrap()[0].1, root(HeapRootKind::JavaFrame, Some("main")));
}
#[test]
fn page_limit_is_clamped() {
let mut heap = MiniHeap::new();
heap.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object)]);
for k in 0..600u64 {
let addr = 0x10_0000 + k * 0x100;
heap.object(addr, "Node", &[]).root(HeapRootKind::JniGlobal, addr);
}
let index = analyzed(&heap);
let node = slot(&index, "Node");
let page = index.dominated(None, Some(node), 0, 10_000);
assert_eq!((page.rows.len(), page.total), (MAX_PAGE as usize, 600));
let top = index.dominated(None, None, 0, 10_000);
assert_eq!((top.rows.len(), top.total), (MAX_PAGE as usize, 600));
assert_eq!(index.dominated(None, None, 550, 10_000).rows.len(), 50);
for (offset, limit) in [(u32::MAX, u32::MAX), (u32::MAX - 10, 100), (599, u32::MAX)] {
let page = index.dominated(None, Some(node), offset, limit);
assert_eq!((page.rows.len(), page.total), (usize::from(offset == 599), 600), "{offset} {limit}");
assert!(index.dominated(None, None, offset, limit).rows.len() <= 1);
}
}
#[test]
fn class_rows_skip_empty_slots() {
let index = analyzed(&sample());
let rows = index.class_rows();
let node = rows.iter().find(|r| r.name == "Node").unwrap();
assert_eq!(
(node.class_index, node.count, node.shallow, node.retained, node.unreachable_count, node.ambiguous),
(slot(&index, "Node"), 3, 72, 104, 1, false)
);
assert!(rows.iter().all(|r| r.count + r.unreachable_count > 0));
assert!(!rows.iter().any(|r| r.name == "boolean[]"));
assert!(rows.windows(2).all(|p| p[0].retained >= p[1].retained));
assert_eq!(index.summary().objects, 8);
}
#[test]
fn class_rows_mark_names_of_several_classes() {
let mut heap = sample();
// `object` takes the class defined last under a name, so each Foo gets one instance.
heap.class("Foo", Some("java.lang.Object"), &[])
.object(0x3000, "Foo", &[])
.class("Foo", Some("java.lang.Object"), &[])
.object(0x3100, "Foo", &[])
.object(0x3200, "Foo", &[]);
let index = analyzed(&heap);
let rows = index.class_rows();
let foos: Vec<_> = rows.iter().filter(|r| r.name == "Foo").collect();
assert_eq!(foos.len(), 2);
assert!(foos.iter().all(|r| r.ambiguous));
let mut counts: Vec<u64> = foos.iter().map(|r| r.unreachable_count).collect();
counts.sort_unstable();
assert_eq!(counts, [1, 2]);
assert!(rows.iter().filter(|r| r.name != "Foo").all(|r| !r.ambiguous));
}
#[test]
fn ipc_shapes() {
assert_eq!(json(&HeapObjectRef(7)), "7");
assert_eq!(json(&HeapEdge::ClassOf), r#"{"kind":"classOf"}"#);
assert_eq!(json(&HeapEdge::Element { index: 2 }), r#"{"kind":"element","index":2}"#);
assert_eq!(
json(&root(HeapRootKind::JavaFrame, Some("main"))),
r#"{"kind":"root","root":"javaFrame","thread":"main"}"#
);
let row = HeapClassRow {
class_index: 1,
name: "A".to_owned(),
count: 2,
shallow: 3,
retained: 4,
unreachable_count: 5,
ambiguous: false,
};
assert_eq!(
json(&row),
r#"{"classIndex":1,"name":"A","count":2,"shallow":3,"retained":4,"unreachableCount":5,"ambiguous":false}"#
);
let back: HeapObjectRef = serde_json::from_str("4294967295").unwrap();
assert_eq!(back, HeapObjectRef(u32::MAX));
}
fn json(value: &impl serde::Serialize) -> String {
serde_json::to_string(value).unwrap()
}
+44
View File
@@ -0,0 +1,44 @@
//! HotSpot `.hprof` heap-dump reading and analysis for the debugger's JVM Memory view.
mod analysis;
mod column;
mod compare;
mod disk;
mod dominators;
mod error;
mod files;
mod graph;
mod hprof;
mod ids;
mod index;
mod names;
mod progress;
mod reader;
mod registry;
mod shallow;
mod strings;
#[cfg(test)]
mod test_index;
/// Fixture builders shared by this crate's tests and by dependants' tests and benches; not API.
#[doc(hidden)]
pub mod testing;
pub use analysis::{AnalyzeOptions, analyze};
pub use column::{Column, NONE, Pod, Store};
pub use compare::{HeapClassDiff, compare};
pub use disk::{DISK_MARGIN, capture_need, create_snapshot_dir, free_bytes, index_need, remove_stale, snapshots_root};
pub use dominators::{Dominators, dominators};
pub use error::HeapError;
pub use graph::{ClassInfo, HeapGraph, PRIM_SLOTS, build_graph};
pub use hprof::{
BasicType, ClassDump, HeapRecord, HeapRootKind, HprofHeader, Record, parse_header, record_at, scan, scan_segment,
};
pub use ids::{IdMap, IdMapBuilder};
pub use index::{
ClassStat, HeapClassRow, HeapEdge, HeapIndex, HeapObjectPage, HeapObjectRef, HeapObjectRow, HeapPathStep,
HeapRootPath, HeapSummary, MAX_PAGE,
};
pub use names::display_name;
pub use progress::HeapPhase;
pub use registry::{HeapOrigin, HeapRegistry, HeapSnapshotId, HeapSnapshotInfo, OpenSnapshot};
pub use shallow::{HistogramEntry, ShallowModel, SizeSource, VmLayout};
+66
View File
@@ -0,0 +1,66 @@
//! Class names in the form the JVM's class histogram and the UI use.
/// The Java source form of an hprof class name (`java/util/HashMap$Node`, `[Ljava/lang/String;`),
/// as `ide_jvm_attach::display_class_name` writes it, plus `+0x` → `/0x` for hidden classes,
/// which the histogram spells with a slash. Names that already use dots pass through unchanged.
pub fn display_name(hprof_name: &str) -> String {
let dotted = hprof_name.replace('/', ".");
let dims = dotted.bytes().take_while(|&b| b == b'[').count();
let shown = if dims == 0 { dotted } else { array_name(&dotted, dims) };
// After the slash replacement, or the new slash would become a dot.
shown.replace("+0x", "/0x")
}
/// `[[I` → `int[][]`; a descriptor that is not one stays as it is.
fn array_name(dotted: &str, dims: usize) -> String {
let element = &dotted[dims..];
let base = match element {
"Z" => "boolean",
"B" => "byte",
"C" => "char",
"S" => "short",
"I" => "int",
"J" => "long",
"F" => "float",
"D" => "double",
_ => match element.strip_prefix('L').and_then(|e| e.strip_suffix(';')) {
Some(class) => class,
None => return dotted.to_owned(),
},
};
format!("{base}{}", "[]".repeat(dims))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn display_names() {
for (hprof, shown) in [
("[Z", "boolean[]"),
("[J", "long[]"),
("[[D", "double[][]"),
("[Ljava/lang/String;", "java.lang.String[]"),
("java/util/HashMap$Node", "java.util.HashMap$Node"),
("Gen$$Lambda+0x0000007800000a00", "Gen$$Lambda/0x0000007800000a00"),
] {
assert_eq!(display_name(hprof), shown, "{hprof}");
}
}
#[test]
fn dotted_names_and_hidden_class_arrays() {
for (hprof, shown) in [
("java.lang.String", "java.lang.String"),
("[Ljava.lang.Object;", "java.lang.Object[]"),
("[[Lpkg.Gen$$Lambda+0x0000007800000a00;", "pkg.Gen$$Lambda/0x0000007800000a00[][]"),
("[Ljava.lang.String", "[Ljava.lang.String"),
("[Q", "[Q"),
("0x7f00", "0x7f00"),
("", ""),
] {
assert_eq!(display_name(hprof), shown, "{hprof}");
}
}
}
+103
View File
@@ -0,0 +1,103 @@
//! Phases of a heap analysis and the progress/cancellation checks every long pass makes.
use ide_core::CancellationToken;
use serde::{Deserialize, Serialize};
use crate::error::HeapError;
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum HeapPhase {
Reading,
Linking,
Dominators,
Retained,
Paths,
Saving,
}
/// Longest stretch of work between two progress reports (and looks at the cancellation token).
pub(crate) const CHECK_EVERY: u64 = 1 << 20;
/// The caller's progress callback plus the token, looked at right after every report so a callback
/// that cancels stops the work before the next step.
pub(crate) struct Progress<'a> {
report: &'a dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &'a CancellationToken,
}
impl<'a> Progress<'a> {
pub(crate) fn new(
report: &'a dyn Fn(HeapPhase, u64, u64) -> Result<(), HeapError>,
cancel: &'a CancellationToken,
) -> Self {
Self { report, cancel }
}
pub(crate) fn report(&self, phase: HeapPhase, done: u64, total: u64) -> Result<(), HeapError> {
(self.report)(phase, done, total)?;
self.check()
}
pub(crate) fn token(&self) -> &'a CancellationToken {
self.cancel
}
/// Looks at the token without reporting.
pub(crate) fn check(&self) -> Result<(), HeapError> {
if self.cancel.is_cancelled() { Err(HeapError::Cancelled) } else { Ok(()) }
}
/// Looks at the token at step 0 and reports every [`CHECK_EVERY`] steps after it. Step 0 does
/// not report: a loop starts right after its phase's own report.
pub(crate) fn step(&self, step: u64, phase: HeapPhase, done: u64, total: u64) -> Result<(), HeapError> {
if step == 0 {
self.check()
} else if step.is_multiple_of(CHECK_EVERY) {
self.report(phase, done, total)
} else {
Ok(())
}
}
}
#[cfg(test)]
mod tests {
use std::cell::RefCell;
use super::*;
#[test]
fn steps_check_at_zero_and_report_every_check_interval() {
let calls = RefCell::new(Vec::new());
let report = |phase, done, total| {
calls.borrow_mut().push((phase, done, total));
Ok(())
};
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
for step in [0, 1, CHECK_EVERY - 1, CHECK_EVERY, CHECK_EVERY + 1, 3 * CHECK_EVERY] {
progress.step(step, HeapPhase::Linking, step, 9).unwrap();
}
assert_eq!(*calls.borrow(), [(HeapPhase::Linking, CHECK_EVERY, 9), (HeapPhase::Linking, 3 * CHECK_EVERY, 9)]);
cancel.cancel();
assert!(matches!(progress.step(0, HeapPhase::Linking, 0, 9), Err(HeapError::Cancelled)));
assert!(matches!(progress.report(HeapPhase::Saving, 1, 1), Err(HeapError::Cancelled)));
assert_eq!(calls.borrow().last(), Some(&(HeapPhase::Saving, 1, 1)), "reports before it looks at the token");
}
#[test]
fn report_errors_pass_through() {
let report = |_, _, _| Err(HeapError::NotEnoughDiskSpace { needed: 2, available: 1 });
let cancel = CancellationToken::new();
let progress = Progress::new(&report, &cancel);
assert!(matches!(progress.report(HeapPhase::Linking, 0, 1), Err(HeapError::NotEnoughDiskSpace { .. })));
}
#[test]
fn phases_serialize_in_camel_case() {
assert_eq!(serde_json::to_string(&HeapPhase::Dominators).unwrap(), "\"dominators\"");
assert_eq!(serde_json::from_str::<HeapPhase>("\"saving\"").unwrap(), HeapPhase::Saving);
}
}
+92
View File
@@ -0,0 +1,92 @@
use crate::error::HeapError;
/// Big-endian cursor over a dump. Every read is bounds-checked; running off `buf` yields
/// `Truncated` at `buf.len()`, so callers bound a record by slicing `buf` to its end.
pub(crate) struct Reader<'a> {
buf: &'a [u8],
pos: usize,
id_size: u8,
}
impl<'a> Reader<'a> {
pub(crate) fn new(buf: &'a [u8], pos: usize, id_size: u8) -> Self {
Self { buf, pos, id_size }
}
pub(crate) fn pos(&self) -> usize {
self.pos
}
pub(crate) fn id_size(&self) -> u8 {
self.id_size
}
pub(crate) fn buf(&self) -> &'a [u8] {
self.buf
}
pub(crate) fn bytes(&mut self, n: usize) -> Result<&'a [u8], HeapError> {
let end = self.pos.checked_add(n).filter(|&end| end <= self.buf.len());
let Some(end) = end else {
return Err(HeapError::Truncated { at: self.buf.len() as u64 });
};
let slice = &self.buf[self.pos..end];
self.pos = end;
Ok(slice)
}
pub(crate) fn skip(&mut self, n: usize) -> Result<(), HeapError> {
self.bytes(n).map(|_| ())
}
fn array<const N: usize>(&mut self) -> Result<[u8; N], HeapError> {
let mut out = [0u8; N];
out.copy_from_slice(self.bytes(N)?);
Ok(out)
}
pub(crate) fn u1(&mut self) -> Result<u8, HeapError> {
Ok(self.array::<1>()?[0])
}
pub(crate) fn u2(&mut self) -> Result<u16, HeapError> {
Ok(u16::from_be_bytes(self.array()?))
}
pub(crate) fn u4(&mut self) -> Result<u32, HeapError> {
Ok(u32::from_be_bytes(self.array()?))
}
pub(crate) fn u8(&mut self) -> Result<u64, HeapError> {
Ok(u64::from_be_bytes(self.array()?))
}
/// `id_size` is 4 or 8; the public entry points reject anything else before building a reader.
pub(crate) fn id(&mut self) -> Result<u64, HeapError> {
if self.id_size == 4 { self.u4().map(u64::from) } else { self.u8() }
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reads_big_endian_and_ids_of_both_sizes() {
let buf = [1, 0, 2, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 5];
let mut r = Reader::new(&buf, 0, 8);
assert_eq!((r.u1().unwrap(), r.u2().unwrap(), r.u4().unwrap(), r.u8().unwrap()), (1, 2, 3, 4));
assert_eq!(r.pos(), 15);
assert_eq!(Reader::new(&buf[15..], 0, 4).id().unwrap(), 5);
}
#[test]
fn running_off_the_end_reports_the_buffer_end() {
let buf = [0u8; 6];
let mut r = Reader::new(&buf, 4, 8);
assert!(matches!(r.u4(), Err(HeapError::Truncated { at: 6 })));
assert_eq!(r.pos(), 4, "a failed read does not advance");
assert!(matches!(r.skip(usize::MAX), Err(HeapError::Truncated { at: 6 })));
assert!(matches!(Reader::new(&buf, 99, 8).u1(), Err(HeapError::Truncated { at: 6 })));
}
}
+163
View File
@@ -0,0 +1,163 @@
//! The heap snapshots that are open in this process.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
use ide_core::process::process_alive;
use ide_core::{DebugSessionId, IpcPath};
use serde::{Deserialize, Serialize};
use crate::disk::remove_stale;
use crate::index::{HeapIndex, HeapSummary};
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, specta::Type)]
#[serde(transparent)]
pub struct HeapSnapshotId(pub u32);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(tag = "kind", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapOrigin {
/// Dumped from a live debug session.
Captured { session: DebugSessionId, pid: u32, java_version: String, taken_while_suspended: bool, dump_ms: u32 },
/// An `.hprof` the user opened; it is never deleted.
File { path: IpcPath },
}
impl HeapOrigin {
/// Text identifying the snapshot in the UI: the process id of a capture, the file name of a
/// file. The capture time is `created_ms`, which the UI formats in local time.
pub fn label(&self) -> String {
match self {
HeapOrigin::Captured { pid, .. } => pid.to_string(),
HeapOrigin::File { path } => path.file_name().unwrap_or(path.as_os_str()).to_string_lossy().into_owned(),
}
}
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub struct HeapSnapshotInfo {
pub id: HeapSnapshotId,
pub label: String,
pub origin: HeapOrigin,
pub summary: HeapSummary,
/// Unix time in milliseconds when the snapshot was opened.
pub created_ms: u64,
}
/// An analyzed snapshot together with the directory that holds its files. Dropping it unmaps the
/// index and then removes the directory, dump included for a capture; a dump opened from a file
/// is outside the directory and stays.
pub struct OpenSnapshot {
pub info: HeapSnapshotInfo,
/// `None` only while dropping, so the mappings are gone before the directory is deleted
/// (Windows cannot delete a mapped file).
index: Option<HeapIndex>,
/// Holds the index files and the dump of a capture.
pub dir: PathBuf,
pub dump: PathBuf,
/// Set by [`HeapRegistry::insert`] once `dir` is known to be one of the registry's own.
remove_dir: bool,
}
impl OpenSnapshot {
/// The directory is not removed on drop until the snapshot is in a [`HeapRegistry`] whose
/// root holds it.
pub fn new(info: HeapSnapshotInfo, index: HeapIndex, dir: PathBuf, dump: PathBuf) -> Self {
Self { info, index: Some(index), dir, dump, remove_dir: false }
}
pub fn index(&self) -> &HeapIndex {
self.index.as_ref().expect("the index is only taken while dropping")
}
}
impl Drop for OpenSnapshot {
fn drop(&mut self) {
drop(self.index.take());
if !self.remove_dir {
return;
}
// The user's own dump that happens to sit inside `dir` (an opened file is analyzed from a
// private copy, which goes with the directory).
if matches!(&self.info.origin, HeapOrigin::File { path } if **path == self.dump)
&& self.dump.starts_with(&self.dir)
{
return;
}
// A link is not followed: only a real directory is deleted.
if std::fs::symlink_metadata(&self.dir).is_ok_and(|meta| meta.file_type().is_dir())
&& let Err(err) = std::fs::remove_dir_all(&self.dir)
{
tracing::warn!(dir = %self.dir.display(), %err, "could not remove a heap snapshot directory");
}
}
}
struct State {
next_id: u32,
open: BTreeMap<u32, Arc<OpenSnapshot>>,
}
/// The open snapshots, shared between threads. Ids are never reused within a process.
pub struct HeapRegistry {
root: PathBuf,
state: Mutex<State>,
}
impl HeapRegistry {
/// Registry for snapshot directories under `root`; first removes the directories there that
/// belong to processes that no longer exist.
pub fn new(root: PathBuf) -> Self {
remove_stale(&root, process_alive);
Self { root, state: Mutex::new(State { next_id: 1, open: BTreeMap::new() }) }
}
pub fn root(&self) -> &Path {
&self.root
}
fn state(&self) -> MutexGuard<'_, State> {
self.state.lock().unwrap_or_else(PoisonError::into_inner)
}
/// Registers the snapshot that `make` builds for a fresh id. `make` runs outside the lock.
pub fn insert(&self, make: impl FnOnce(HeapSnapshotId) -> OpenSnapshot) -> HeapSnapshotInfo {
let id = {
let mut state = self.state();
let id = state.next_id;
state.next_id = id.saturating_add(1);
HeapSnapshotId(id)
};
let mut snapshot = make(id);
snapshot.info.id = id;
snapshot.remove_dir = snapshot.dir.file_name().is_some() && snapshot.dir.parent() == Some(self.root.as_path());
if !snapshot.remove_dir {
tracing::warn!(dir = %snapshot.dir.display(), "heap snapshot directory is outside the registry root; keeping it");
}
let info = snapshot.info.clone();
self.state().open.insert(id.0, Arc::new(snapshot));
info
}
pub fn get(&self, id: HeapSnapshotId) -> Option<Arc<OpenSnapshot>> {
self.state().open.get(&id.0).cloned()
}
/// Forgets the snapshot; its directory goes once the last query using it lets go. `false` for
/// an id that is not open.
pub fn close(&self, id: HeapSnapshotId) -> bool {
// Bound first so the lock is released before the snapshot may be dropped: that deletes files.
let removed = self.state().open.remove(&id.0);
removed.is_some()
}
/// In the order the snapshots were opened.
pub fn list(&self) -> Vec<HeapSnapshotInfo> {
self.state().open.values().map(|snapshot| snapshot.info.clone()).collect()
}
}
#[cfg(test)]
mod tests;
+265
View File
@@ -0,0 +1,265 @@
use std::sync::Arc;
use ide_core::{CancellationToken, DebugSessionId};
use super::*;
use crate::analysis::{AnalyzeOptions, analyze};
use crate::disk::create_snapshot_dir;
use crate::error::HeapError;
use crate::progress::HeapPhase;
use crate::testing::sample;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn captured() -> HeapOrigin {
HeapOrigin::Captured {
session: DebugSessionId(7),
pid: 4242,
java_version: "21.0.2".into(),
taken_while_suspended: true,
dump_ms: 35,
}
}
fn external(path: &Path) -> HeapOrigin {
HeapOrigin::File { path: IpcPath(path.to_path_buf()) }
}
/// A snapshot whose index files are mapped from `dir`, like a real one: `dump` is analyzed in
/// place and `dir` receives the columns.
fn open_in(dir: &Path, dump: &Path, origin: HeapOrigin) -> impl FnOnce(HeapSnapshotId) -> OpenSnapshot {
let dir = dir.to_path_buf();
let dump = dump.to_path_buf();
move |id| {
std::fs::write(&dump, sample().bytes()).unwrap();
let index = analyze(&dump, &dir, AnalyzeOptions::default(), &no_progress, &CancellationToken::new()).unwrap();
let info = HeapSnapshotInfo {
id,
label: origin.label(),
origin,
summary: index.summary.clone(),
created_ms: 1_700_000_000_000,
};
OpenSnapshot::new(info, index, dir, dump)
}
}
/// Inserts a captured snapshot with its own directory under the registry's root.
fn insert_captured(registry: &HeapRegistry) -> (HeapSnapshotInfo, PathBuf) {
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("dump.hprof");
let info = registry.insert(open_in(&dir, &dump, captured()));
(info, dir)
}
fn names(dir: &Path) -> Vec<String> {
let mut names: Vec<_> =
std::fs::read_dir(dir).unwrap().map(|e| e.unwrap().file_name().into_string().unwrap()).collect();
names.sort();
names
}
#[test]
fn closing_snapshot_removes_dir() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (info, dir) = insert_captured(&registry);
assert!(dir.join("dump.hprof").exists() && dir.join("summary.json").exists());
assert!(registry.close(info.id));
assert!(!dir.exists());
assert!(!registry.close(info.id));
assert!(registry.get(info.id).is_none());
}
#[test]
fn external_dump_is_never_deleted() {
let root = tempfile::tempdir().unwrap();
let outside = tempfile::tempdir().unwrap();
let dump = outside.path().join("app.hprof");
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let info = registry.insert(open_in(&dir, &dump, external(&dump)));
assert_eq!(info.label, "app.hprof");
assert!(dir.join("summary.json").exists());
assert!(registry.close(info.id));
assert!(!dir.exists());
assert!(dump.exists());
}
#[test]
fn a_dump_inside_the_dir_of_a_file_snapshot_keeps_the_dir() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("opened.hprof");
let info = registry.insert(open_in(&dir, &dump, external(&dump)));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[test]
fn a_dir_outside_the_root_is_never_deleted() {
let root = tempfile::tempdir().unwrap();
let elsewhere = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dump = elsewhere.path().join("dump.hprof");
let info = registry.insert(open_in(elsewhere.path(), &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
// Neither the root itself nor a path that climbs out of it counts as a snapshot dir.
let sneaky = root.path().join("..").join(elsewhere.path().file_name().unwrap());
let info = registry.insert(open_in(&sneaky, &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[cfg(unix)]
#[test]
fn a_symlinked_dir_is_not_followed() {
let root = tempfile::tempdir().unwrap();
let outside = tempfile::tempdir().unwrap();
let dump = outside.path().join("dump.hprof");
let link = root.path().join("link");
std::os::unix::fs::symlink(outside.path(), &link).unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let info = registry.insert(open_in(&link, &dump, captured()));
assert!(registry.close(info.id));
assert!(dump.exists());
}
#[test]
fn ids_are_never_reused() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (first, _) = insert_captured(&registry);
assert!(registry.close(first.id));
let (second, _) = insert_captured(&registry);
let (third, _) = insert_captured(&registry);
assert!(first.id.0 < second.id.0 && second.id.0 < third.id.0);
}
#[test]
fn the_registry_decides_the_id_in_the_info() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let dir = create_snapshot_dir(registry.root()).unwrap();
let dump = dir.join("dump.hprof");
let make = open_in(&dir, &dump, captured());
let info = registry.insert(|id| {
let mut snapshot = make(id);
snapshot.info.id = HeapSnapshotId(u32::MAX);
snapshot
});
assert_ne!(info.id, HeapSnapshotId(u32::MAX));
assert_eq!(registry.get(info.id).unwrap().info.id, info.id);
}
#[test]
fn list_is_in_opening_order_and_get_shares_the_snapshot() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (a, _) = insert_captured(&registry);
let (b, _) = insert_captured(&registry);
let (c, _) = insert_captured(&registry);
assert!(registry.close(b.id));
assert_eq!(registry.list(), [a.clone(), c.clone()]);
let one = registry.get(a.id).unwrap();
let two = registry.get(a.id).unwrap();
assert!(Arc::ptr_eq(&one, &two));
assert_eq!(one.info, a);
assert_eq!(one.index().summary, a.summary);
}
#[test]
fn a_snapshot_in_use_keeps_its_dir_until_the_last_holder_lets_go() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (info, dir) = insert_captured(&registry);
let held = registry.get(info.id).unwrap();
assert!(registry.close(info.id));
assert!(registry.get(info.id).is_none() && registry.list().is_empty());
assert!(dir.exists());
drop(held);
assert!(!dir.exists());
}
#[test]
fn dropping_the_registry_removes_the_open_snapshots() {
let root = tempfile::tempdir().unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
let (_, dir) = insert_captured(&registry);
drop(registry);
assert!(!dir.exists());
assert!(names(root.path()).is_empty());
}
#[cfg(unix)]
#[test]
fn new_clears_dirs_whose_owner_died() {
let root = tempfile::tempdir().unwrap();
let mut child = std::process::Command::new("true").spawn().unwrap();
let dead = child.id();
child.wait().unwrap();
let dead_dir = root.path().join("dead");
std::fs::create_dir(&dead_dir).unwrap();
std::fs::write(dead_dir.join("owner"), dead.to_string()).unwrap();
let live_dir = create_snapshot_dir(root.path()).unwrap();
let registry = HeapRegistry::new(root.path().to_path_buf());
assert!(!dead_dir.exists());
assert!(live_dir.exists());
assert_eq!(registry.root(), root.path());
}
#[test]
fn concurrent_inserts_get_distinct_ids() {
let root = tempfile::tempdir().unwrap();
let registry = Arc::new(HeapRegistry::new(root.path().to_path_buf()));
let handles: Vec<_> = (0..6)
.map(|_| {
let registry = registry.clone();
std::thread::spawn(move || (0..3).map(|_| insert_captured(&registry).0.id).collect::<Vec<_>>())
})
.collect();
let mut ids: Vec<_> = handles.into_iter().flat_map(|h| h.join().unwrap()).collect();
ids.sort_by_key(|id| id.0);
ids.dedup();
assert_eq!(ids.len(), 18);
assert_eq!(registry.list().len(), 18);
for id in ids {
assert!(registry.close(id));
}
assert!(names(root.path()).is_empty());
}
#[test]
fn labels_name_the_pid_or_the_file() {
assert_eq!(captured().label(), "4242");
assert_eq!(external(Path::new("/tmp/dumps/app.hprof")).label(), "app.hprof");
assert_eq!(external(Path::new("/")).label(), "/");
}
#[test]
fn origin_and_info_serialize_as_the_ui_expects() {
let value = serde_json::to_value(captured()).unwrap();
assert_eq!(
value,
serde_json::json!({
"kind": "captured", "session": 7, "pid": 4242, "javaVersion": "21.0.2",
"takenWhileSuspended": true, "dumpMs": 35
})
);
assert_eq!(serde_json::to_value(external(Path::new("/a/b.hprof"))).unwrap()["path"], "/a/b.hprof");
assert_eq!(serde_json::to_value(HeapSnapshotId(3)).unwrap(), serde_json::json!(3));
let back: HeapOrigin = serde_json::from_value(value).unwrap();
assert_eq!(back, captured());
}
// `HeapRegistry::get` hands out `Arc<OpenSnapshot>` to query threads.
const _: fn() = || {
fn check<T: Send + Sync>() {}
check::<HeapIndex>();
check::<OpenSnapshot>();
check::<HeapRegistry>();
};
+265
View File
@@ -0,0 +1,265 @@
//! Shallow sizes: the bytes each object takes in the JVM heap, from the object layout rules, the
//! JVM's own class histogram, or the distances between object addresses in the dump.
mod layout;
mod mirrors;
use std::collections::HashMap;
use ide_core::CancellationToken;
use serde::{Deserialize, Serialize};
pub use self::layout::VmLayout;
use self::mirrors::{Mirrors, injected_by_table};
use crate::error::HeapError;
use crate::graph::{ClassInfo, HeapGraph};
use crate::hprof::BasicType;
use crate::names::display_name;
use crate::progress::CHECK_EVERY;
/// How far past the layout size an address gap may be and still count as an object size.
const GAP_SLACK: u64 = 64;
/// Share of the instance types that can have a gap (two or more instances), in tenths, that must
/// take their size from address gaps for the sizes to be reported as coming from them.
const GAP_TRUST_TENTHS: usize = 9;
/// Where the sizes of instances came from.
///
/// Class objects (mirrors) are approximate in every case: HotSpot leaves the fields it injects
/// into `java.lang.Class` out of the dump, so their layout size is short by a fixed amount per
/// mirror. With a histogram that amount is fitted to its `java.lang.Class` row; otherwise (or when
/// the fit is rejected) it comes from a table measured per JDK and reference size, within a few
/// bytes per mirror, or is 32 when the JDK is unknown.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize, specta::Type)]
#[serde(rename_all = "camelCase")]
pub enum SizeSource {
/// The JVM's class histogram from the capture.
Jvm,
/// The smallest distance between neighbouring objects of a type, in a dump the IDE did not
/// capture.
AddressGaps,
Layout,
}
/// A row of the JVM's class histogram.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct HistogramEntry {
/// Display form: `java.lang.String`, `int[]`, `Gen$$Lambda/0x…`.
pub name: String,
pub instances: u64,
pub bytes: u64,
}
enum TypeSize {
Fixed(u64),
ObjectArray,
/// Class objects are sized one by one; this is for an instance of `java.lang.Class` that is
/// not a class object.
ClassObject(u64),
}
/// Shallow size of every object of a [`HeapGraph`]. Instances of a type share one size; arrays
/// follow the layout; class objects add their statics to the size of `java.lang.Class`.
pub struct ShallowModel {
pub source: SizeSource,
layout: VmLayout,
/// Per type slot of a class; primitive array slots follow and have no entry.
types: Vec<TypeSize>,
/// `(node, size)` of each class object, by node.
class_objects: Vec<(u32, u64)>,
/// Per class slot: another class has the same display name.
ambiguous: Vec<bool>,
}
impl ShallowModel {
/// `histogram` is the JVM's class histogram of the same stop as the dump, when the IDE captured
/// it. Without it, sizes come from address gaps where they agree with the layout.
///
/// A name shared by several classes (several class loaders) never takes a histogram size,
/// since the name is the only link between the dump and the histogram; see
/// [`Self::is_ambiguous`]. `java.lang.Class` never does either, because mirror sizes vary with
/// their statics; the histogram only tells how many bytes HotSpot adds to each mirror.
pub fn new(
graph: &HeapGraph,
layout: VmLayout,
histogram: Option<&[HistogramEntry]>,
cancel: &CancellationToken,
) -> Result<Self, HeapError> {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
let classes = &graph.classes;
let field_bytes = field_bytes(classes, layout);
let by_layout: Vec<u64> = field_bytes.iter().map(|&bytes| layout.instance_size(bytes)).collect();
let names: Vec<String> = classes.iter().map(|c| display_name(&c.name)).collect();
let ambiguous = ambiguous_names(&names);
let class_slot = graph.class_slot as usize;
let mirrors = Mirrors::collect(graph, by_layout[class_slot], layout);
let mut fitted = None;
if let Some(rows) = histogram
&& !ambiguous[class_slot]
{
let mut class_rows = rows.iter().filter(|row| row.name == names[class_slot]);
if let (Some(row), None) = (class_rows.next(), class_rows.next()) {
fitted = mirrors.fit_injected(graph, layout, by_layout[class_slot], row, cancel)?;
}
}
let injected = fitted.unwrap_or_else(|| injected_by_table(layout));
let mut types: Vec<TypeSize> = classes
.iter()
.enumerate()
.map(|(slot, class)| {
if class.name.starts_with('[') {
TypeSize::ObjectArray
} else if slot == class_slot {
TypeSize::ClassObject(layout.align(by_layout[slot] + injected))
} else {
TypeSize::Fixed(by_layout[slot])
}
})
.collect();
let source = match histogram {
Some(rows) => {
apply_histogram(&mut types, &names, &ambiguous, rows);
SizeSource::Jvm
}
None => apply_address_gaps(&mut types, graph, &by_layout, cancel)?,
};
Ok(Self { source, layout, types, class_objects: mirrors.sizes(layout, injected), ambiguous })
}
/// Shallow size of `object` (a node of the graph); 0 for the super-root.
pub fn size_of(&self, graph: &HeapGraph, object: u32) -> u64 {
let Some(&slot) = graph.types.get(object as usize) else { return 0 };
match self.types.get(slot as usize) {
Some(&TypeSize::Fixed(size)) => size,
Some(TypeSize::ObjectArray) => self.array_size(BasicType::Object, graph.lens[object as usize]),
Some(&TypeSize::ClassObject(fallback)) => self
.class_objects
.binary_search_by_key(&object, |&(node, _)| node)
.map_or(fallback, |at| self.class_objects[at].1),
None => match graph.prim_type(slot) {
Some(ty) => self.array_size(ty, graph.lens[object as usize]),
None => 0,
},
}
}
/// Whether another class has the same display name as the class in `slot`. Sizes of such
/// classes come from the layout, and anything joined to them by name (histogram rows, a
/// second snapshot) cannot tell them apart.
pub fn is_ambiguous(&self, slot: u32) -> bool {
self.ambiguous.get(slot as usize).copied().unwrap_or(false)
}
fn array_size(&self, ty: BasicType, len: u32) -> u64 {
self.layout.array_size(ty, u64::from(len))
}
}
/// Bytes of instance fields per class, its superclasses included. Super chains always end
/// (`ClassInfo::super_index` has its cycles cut).
fn field_bytes(classes: &[ClassInfo], layout: VmLayout) -> Vec<u64> {
let mut total: Vec<Option<u64>> = vec![None; classes.len()];
let mut path = Vec::new();
for start in 0..classes.len() {
let mut cursor = Some(start);
let mut below = 0;
while let Some(c) = cursor {
if let Some(known) = total[c] {
below = known;
break;
}
path.push(c);
cursor = classes[c].super_index.map(|s| s as usize);
}
while let Some(c) = path.pop() {
let own: u64 = classes[c].fields.iter().map(|(_, ty)| ty.heap_size(layout.ref_size)).sum();
below += own;
total[c] = Some(below);
}
}
total.into_iter().map(Option::unwrap_or_default).collect()
}
fn ambiguous_names(names: &[String]) -> Vec<bool> {
let mut counts: HashMap<&str, u32> = HashMap::with_capacity(names.len());
for name in names {
*counts.entry(name).or_default() += 1;
}
names.iter().map(|name| counts[name.as_str()] > 1).collect()
}
/// `bytes / instances` for each plain class with a histogram row of its name. A name that is
/// ambiguous in the dump or repeated in the histogram, or a row that is not a whole number of
/// objects, keeps its layout size.
fn apply_histogram(types: &mut [TypeSize], names: &[String], ambiguous: &[bool], rows: &[HistogramEntry]) {
let mut per_object: HashMap<&str, Option<u64>> = HashMap::with_capacity(rows.len());
for row in rows {
let size = row.bytes.checked_div(row.instances).filter(|&s| s > 0 && row.bytes % row.instances == 0);
per_object.entry(&row.name).and_modify(|seen| *seen = None).or_insert(size);
}
for (slot, ty) in types.iter_mut().enumerate() {
if let TypeSize::Fixed(size) = ty
&& !ambiguous[slot]
&& let Some(&Some(from_histogram)) = per_object.get(names[slot].as_str())
{
*size = from_histogram;
}
}
}
/// Takes the smallest distance from an instance to the next object in address order as the size
/// of its type, for types with two or more instances where that distance is no smaller than the
/// layout size and at most [`GAP_SLACK`] bigger. The result says [`SizeSource::AddressGaps`] when
/// nine in ten of the types with two or more instances did; types with one instance have no gap
/// to measure, so they neither count for nor against.
fn apply_address_gaps(
types: &mut [TypeSize],
graph: &HeapGraph,
by_layout: &[u64],
cancel: &CancellationToken,
) -> Result<SizeSource, HeapError> {
let slots = types.len();
let mut instances = vec![0u32; slots];
let mut smallest_gap = vec![u64::MAX; slots];
let mut ids = graph.ids.ids();
let mut current = ids.next();
let mut object = 0usize;
while let Some(id) = current {
if (object as u64).is_multiple_of(CHECK_EVERY) && cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
let next = ids.next();
let slot = graph.types[object] as usize;
if matches!(types.get(slot), Some(TypeSize::Fixed(_))) {
instances[slot] = instances[slot].saturating_add(1);
if let Some(next) = next.filter(|&n| n > id) {
smallest_gap[slot] = smallest_gap[slot].min(next - id);
}
}
current = next;
object += 1;
}
let (mut measurable, mut by_gap) = (0usize, 0usize);
for (slot, ty) in types.iter_mut().enumerate() {
let TypeSize::Fixed(size) = ty else { continue };
if instances[slot] < 2 {
continue;
}
measurable += 1;
let gap = smallest_gap[slot];
if (by_layout[slot]..=by_layout[slot] + GAP_SLACK).contains(&gap) {
*size = gap;
by_gap += 1;
}
}
let trusted = measurable > 0 && by_gap * 10 >= measurable * GAP_TRUST_TENTHS;
Ok(if trusted { SizeSource::AddressGaps } else { SizeSource::Layout })
}
#[cfg(test)]
mod tests;
+113
View File
@@ -0,0 +1,113 @@
//! The object layout rules of the JVM that wrote a dump.
use std::collections::HashMap;
use crate::hprof::BasicType;
/// Object layout of the JVM that wrote the dump.
#[derive(Clone, Copy, Debug, PartialEq)]
pub struct VmLayout {
pub header: u8,
/// Bytes of a reference: 4 with compressed oops.
pub ref_size: u8,
/// Offset of the length field of an array.
pub length_offset: u8,
/// JDK 21 and older put array elements at the next 8-byte boundary after the length, later
/// JDKs at the next boundary of the element size.
pub legacy_array_base: bool,
pub alignment: u16,
/// The JDK the flags came from; `None` when nothing is known about the JVM.
pub java_major: Option<u32>,
}
impl VmLayout {
pub const DEFAULT: VmLayout = VmLayout {
header: 12,
ref_size: 4,
length_offset: 12,
legacy_array_base: false,
alignment: 8,
java_major: None,
};
/// `flags` is the output of `VM.flags`, which lists the flags set on this run. Compressed oops
/// are off by default and only switched on by ergonomics, so `-XX:+UseCompressedOops` is
/// printed exactly when references take 4 bytes; a heap too big for them prints no such flag.
/// Before JDK 15 compressed class pointers follow the same rule; with `java_major: None` (JDK
/// unknown) they follow the JDK 15+ rule and the array base that of the newest JDKs. Flags
/// without any `-XX:` option (an external dump) give [`Self::DEFAULT`] with the array base of
/// `java_major` and an unknown JDK, since the reference size is a guess.
/// An `ObjectAlignmentInBytes` that is not a power of two in `8..=256` is ignored.
pub fn from_vm_flags(flags: &str, java_major: Option<u32>) -> Self {
let legacy_array_base = java_major.map_or(Self::DEFAULT.legacy_array_base, |major| major <= 21);
let flags = parse_flags(flags);
if flags.is_empty() {
return VmLayout { legacy_array_base, ..Self::DEFAULT };
}
let on = |name: &str| flags.get(name) == Some(&"true");
let compressed_classes = if java_major.is_none_or(|major| major >= 15) {
flags.get("UseCompressedClassPointers") != Some(&"false")
} else {
on("UseCompressedClassPointers")
};
let (header, length_offset) = if on("UseCompactObjectHeaders") {
(8, 8)
} else if compressed_classes {
(12, 12)
} else {
(16, 16)
};
let alignment = flags
.get("ObjectAlignmentInBytes")
.and_then(|v| v.parse::<u16>().ok())
.filter(|a| (8..=256).contains(a) && a.is_power_of_two())
.unwrap_or(Self::DEFAULT.alignment);
VmLayout {
header,
ref_size: if on("UseCompressedOops") { 4 } else { 8 },
length_offset,
legacy_array_base,
alignment,
java_major,
}
}
/// Size of an instance whose fields (its class and every superclass) take `field_bytes`.
pub fn instance_size(&self, field_bytes: u64) -> u64 {
let smallest = if self.header == 8 { 8 } else { 16 };
self.align(u64::from(self.header) + field_bytes).max(smallest)
}
pub fn array_size(&self, ty: BasicType, len: u64) -> u64 {
let element = ty.heap_size(self.ref_size);
let base_alignment = if self.legacy_array_base { 8 } else { element };
let base = align_to(u64::from(self.length_offset) + 4, base_alignment);
self.align(base + element * len)
}
/// `value` rounded up to the object alignment.
pub(super) fn align(&self, value: u64) -> u64 {
align_to(value, u64::from(self.alignment))
}
}
fn align_to(value: u64, to: u64) -> u64 {
value.div_ceil(to.max(1)) * to.max(1)
}
/// `-XX:+Name` / `-XX:-Name` / `-XX:Name=value` tokens by name; booleans become `"true"` / `"false"`.
/// A flag given twice keeps its last value, as in the JVM.
fn parse_flags(flags: &str) -> HashMap<&str, &str> {
let mut parsed = HashMap::new();
for token in flags.split_whitespace() {
let Some(body) = token.strip_prefix("-XX:") else { continue };
if let Some(name) = body.strip_prefix('+') {
parsed.insert(name, "true");
} else if let Some(name) = body.strip_prefix('-') {
parsed.insert(name, "false");
} else if let Some((name, value)) = body.split_once('=') {
parsed.insert(name, value);
}
}
parsed
}
+119
View File
@@ -0,0 +1,119 @@
//! Sizes of class objects (mirrors): the `java.lang.Class` instance plus the class's statics, plus
//! the fields HotSpot injects into every mirror and leaves out of the class dump.
use ide_core::CancellationToken;
use super::{HistogramEntry, VmLayout};
use crate::error::HeapError;
use crate::graph::HeapGraph;
use crate::progress::CHECK_EVERY;
/// Largest injected byte count (per mirror, a multiple of 4) the histogram fit tries.
const MAX_INJECTED: u64 = 128;
/// How far, in percent of the histogram's total, the best fit may be off before the row is taken
/// for something else than the mirrors of this dump.
const MAX_FIT_ERROR_PERCENT: u128 = 5;
/// Bytes HotSpot injects into every mirror (`klass`, `array_klass`, `oop_size`,
/// `static_oop_field_count`, and oops such as `protection_domain`) without listing them in the class
/// dump, as measured by comparing the dump with `GC.class_histogram` on JDK 8, 11, 21, 23 and 25,
/// with and without compressed oops. Each X is the best fit in steps of 4; the real value is
/// within a few bytes (statics padding). `None` for a reference size the table does not cover.
pub(super) fn injected_mirror_bytes(java_major: u32, ref_size: u8) -> Option<u64> {
match (ref_size, java_major) {
// 8, 11, 21 and 23 measure 36; 25 is the first JDK that shows 28.
(4, ..=24) => Some(36),
(4, 25..) => Some(28),
// 8, 11 and 21 measure 44; 23 is the first to show 40 and 25 the first to show 32.
(8, ..=22) => Some(44),
(8, 23..=24) => Some(40),
(8, 25..) => Some(32),
_ => None,
}
}
/// Used when the JVM is unknown (a dump without `VM.flags`): the midpoint of the measured values
/// with 4-byte references, 36 and 28.
const UNKNOWN_JVM_INJECTED: u64 = 32;
/// The injected bytes per mirror the layout implies, without looking at a histogram.
pub(super) fn injected_by_table(layout: VmLayout) -> u64 {
layout.java_major.and_then(|major| injected_mirror_bytes(major, layout.ref_size)).unwrap_or(UNKNOWN_JVM_INJECTED)
}
pub(super) struct Mirrors {
/// `(node, size before alignment)` of every class dump that is an object of the graph, by node.
dumped: Vec<(u32, u64)>,
}
impl Mirrors {
/// `instance` is the layout size of `java.lang.Class`, the base of every mirror.
pub(super) fn collect(graph: &HeapGraph, instance: u64, layout: VmLayout) -> Self {
let mut dumped: Vec<(u32, u64)> = graph
.classes
.iter()
.filter(|class| class.id != 0)
.filter_map(|class| {
let node = graph.ids.rank(class.id)?;
let references = class.statics.len() as u64 * u64::from(layout.ref_size);
Some((node, instance + class.static_bytes + references))
})
.collect();
dumped.sort_unstable_by_key(|&(node, _)| node);
Self { dumped }
}
/// `(node, size)` of each class object when HotSpot adds `injected` bytes to every mirror.
pub(super) fn sizes(&self, layout: VmLayout, injected: u64) -> Vec<(u32, u64)> {
self.dumped.iter().map(|&(node, bytes)| (node, layout.align(bytes + injected))).collect()
}
/// The injected bytes per mirror (a multiple of 4, at most 128) that make the mirrors of this
/// dump add up to `class_row`'s share of the histogram, or `None` when no value gets within 5%:
/// a histogram of other mirrors than these (a different JVM stop) must not skew the sizes.
///
/// The fit compares averages, since the histogram also counts mirrors the dump has no class
/// dump for (primitive types, classes only archived by CDS). Every mirror still gets the same
/// extra bytes, never the average size, because statics make mirror sizes differ.
pub(super) fn fit_injected(
&self,
graph: &HeapGraph,
layout: VmLayout,
instance: u64,
class_row: &HistogramEntry,
cancel: &CancellationToken,
) -> Result<Option<u64>, HeapError> {
if class_row.instances == 0 {
return Ok(None);
}
// Mirrors without a class dump (primitive types) have no statics.
let class_typed = count_slot(graph, graph.class_slot, cancel)?;
let bare = class_typed.saturating_sub(self.dumped.len() as u64);
let mirrors = self.dumped.len() as u64 + bare;
if mirrors == 0 {
return Ok(None);
}
let target = u128::from(class_row.bytes) * u128::from(mirrors) / u128::from(class_row.instances);
let total = |injected: u64| -> u128 {
let dumped: u128 = self.dumped.iter().map(|&(_, bytes)| u128::from(layout.align(bytes + injected))).sum();
dumped + u128::from(bare) * u128::from(layout.align(instance + injected))
};
let (best, error) = (0..=MAX_INJECTED)
.step_by(4)
.map(|injected| (injected, total(injected).abs_diff(target)))
.min_by_key(|&(injected, error)| (error, injected))
.unwrap_or((0, u128::MAX));
Ok((error * 100 <= target * MAX_FIT_ERROR_PERCENT).then_some(best))
}
}
fn count_slot(graph: &HeapGraph, slot: u32, cancel: &CancellationToken) -> Result<u64, HeapError> {
let mut count = 0;
for chunk in graph.types.chunks(CHECK_EVERY as usize) {
if cancel.is_cancelled() {
return Err(HeapError::Cancelled);
}
count += chunk.iter().filter(|&&ty| ty == slot).count() as u64;
}
Ok(count)
}
+593
View File
@@ -0,0 +1,593 @@
use super::*;
use crate::column::Store;
use crate::graph::build_graph;
use crate::progress::HeapPhase;
use crate::testing::{HprofWriter, MiniHeap, sample};
const ROOT: Option<&str> = Some("java.lang.Object");
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
fn graph_of(heap: &MiniHeap) -> HeapGraph {
build_graph(&heap.bytes(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap()
}
fn model(g: &HeapGraph, layout: VmLayout, histogram: Option<&[HistogramEntry]>) -> ShallowModel {
ShallowModel::new(g, layout, histogram, &CancellationToken::new()).unwrap()
}
fn size_at(g: &HeapGraph, m: &ShallowModel, addr: u64) -> u64 {
m.size_of(g, g.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the graph")))
}
fn slot_at(g: &HeapGraph, addr: u64) -> u32 {
g.types[g.ids.rank(addr).unwrap() as usize]
}
fn entry(name: &str, instances: u64, bytes: u64) -> HistogramEntry {
HistogramEntry { name: name.to_owned(), instances, bytes }
}
fn flags(flags: &str, java_major: u32) -> VmLayout {
VmLayout::from_vm_flags(flags, Some(java_major))
}
const SHAPES: [&str; 9] = ["A", "B", "C", "D", "E", "F", "G", "H", "Node"];
/// One instance of each shape, 0x100 apart so no address gap qualifies.
fn shapes() -> MiniHeap {
use BasicType::{Byte, Int, Long, Object, Short};
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", Byte)])
.class("B", ROOT, &[("f", Long)])
.class("C", ROOT, &[("i", Int), ("l", Long), ("b", Byte)])
.class("D", Some("C"), &[("s", Short), ("o", Object)])
.class("E", ROOT, &[("a", Object), ("b", Object), ("c", Object)])
.class("F", Some("A"), &[("g", Byte)])
.class("G", ROOT, &[])
.class("H", Some("B"), &[("i", Int)])
.class("Node", ROOT, &[("l", Long), ("a", Object), ("b", Object)]);
for (i, name) in SHAPES.iter().enumerate() {
heap.object(0x1000 + i as u64 * 0x100, name, &[]);
}
heap
}
fn shape_sizes(layout: VmLayout) -> Vec<u64> {
let g = graph_of(&shapes());
let m = model(&g, layout, None);
(0..SHAPES.len() as u64).map(|i| size_at(&g, &m, 0x1000 + i * 0x100)).collect()
}
// `jcmd <pid> VM.flags` of `-Xmx1g` and `-Xmx40g` runs (the flags that matter here).
const J8_11_1G: &str = "-XX:MaxHeapSize=1073741824 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops";
const J8_11_40G: &str = "-XX:MaxHeapSize=42949672960";
const J21_25_1G: &str = "-XX:MaxHeapSize=1073741824 -XX:SoftMaxHeapSize=1073741824 -XX:+UseCompressedOops";
const J21_25_40G: &str = "-XX:MaxHeapSize=42949672960 -XX:SoftMaxHeapSize=42949672960";
/// `-Xmx40g -XX:+UseCompressedOops`: the JVM warns and turns them off.
const J21_40G_OFF: &str = "-XX:MaxHeapSize=42949672960 -XX:SoftMaxHeapSize=42949672960 -XX:-UseCompressedOops";
const J25_COMPACT: &str = "-XX:MaxHeapSize=1073741824 -XX:+UseCompressedOops -XX:+UseCompactObjectHeaders";
#[test]
fn layout_from_flags() {
// The layout of a JVM of `major` with default flags otherwise.
let jvm = |major: u32, layout: VmLayout| VmLayout { java_major: Some(major), ..layout };
let legacy = VmLayout { legacy_array_base: true, ..VmLayout::DEFAULT };
assert_eq!(flags(J8_11_1G, 8), jvm(8, legacy));
assert_eq!(flags(J8_11_1G, 11), jvm(11, legacy));
assert_eq!(flags(J21_25_1G, 21), jvm(21, legacy));
assert_eq!(flags(J21_25_1G, 25), jvm(25, VmLayout::DEFAULT));
assert_eq!(flags(J25_COMPACT, 25), jvm(25, VmLayout { header: 8, length_offset: 8, ..VmLayout::DEFAULT }));
assert_eq!(flags(&format!("{J21_25_1G} -XX:+UseG1GC"), 21), jvm(21, legacy));
assert_eq!(
flags(&format!("{J21_25_1G} -XX:ObjectAlignmentInBytes=16"), 21),
jvm(21, VmLayout { alignment: 16, ..legacy })
);
}
#[test]
fn heaps_too_big_for_compressed_oops_print_no_flag() {
let jvm = |major: u32, layout: VmLayout| VmLayout { java_major: Some(major), ..layout };
let wide_legacy =
VmLayout { header: 16, ref_size: 8, length_offset: 16, legacy_array_base: true, ..VmLayout::DEFAULT };
assert_eq!(flags(J8_11_40G, 8), jvm(8, wide_legacy));
assert_eq!(flags(J8_11_40G, 11), jvm(11, wide_legacy));
// From JDK 15 compressed class pointers stay on without compressed oops.
let wide = VmLayout { ref_size: 8, ..VmLayout::DEFAULT };
assert_eq!(flags(J21_25_40G, 21), jvm(21, VmLayout { legacy_array_base: true, ..wide }));
assert_eq!(flags(J21_25_40G, 25), jvm(25, wide));
assert_eq!(flags(J21_40G_OFF, 21), flags(J21_25_40G, 21));
}
#[test]
fn flags_of_an_unknown_jdk_follow_the_jdk_15_class_pointer_rule() {
let unknown = |f: &str| VmLayout::from_vm_flags(f, None);
assert_eq!(unknown(J21_25_1G), VmLayout::DEFAULT);
assert_eq!(unknown(J8_11_40G), VmLayout { ref_size: 8, ..VmLayout::DEFAULT });
let no_ccp = unknown("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers");
assert_eq!((no_ccp.header, no_ccp.length_offset, no_ccp.java_major), (16, 16, None));
assert_eq!(unknown(J25_COMPACT).header, 8);
// The mirror table has no row for an unknown JDK.
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
assert_eq!(mirror_sizes(&g, &addrs, unknown(J21_25_1G), None), expected_mirrors(&MIRROR_STATICS, 32));
}
#[test]
fn layout_flag_edge_cases() {
let legacy = VmLayout { legacy_array_base: true, ..VmLayout::DEFAULT };
// No VM.flags (an external dump): the default layout, of an unknown JVM.
assert_eq!(flags("", 25), VmLayout::DEFAULT);
assert_eq!(flags("", 21), legacy);
assert_eq!(flags("-Xmx1g --add-opens=x -XX:Foo", 25), VmLayout::DEFAULT);
// Before JDK 15 compressed class pointers are on only when the flags say so.
assert_eq!(flags("-XX:+UseCompressedOops", 11).header, 16);
assert_eq!(flags("-XX:+UseCompressedOops -XX:+UseCompressedClassPointers", 11).header, 12);
assert_eq!(flags("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers", 11).header, 16);
assert_eq!(flags("-XX:+UseCompressedOops", 15).header, 12);
let no_ccp = flags("-XX:+UseCompressedOops -XX:-UseCompressedClassPointers", 21);
assert_eq!((no_ccp.header, no_ccp.length_offset, no_ccp.ref_size), (16, 16, 4));
// Compact headers win over the class pointer flag; the last mention of a flag wins.
assert_eq!(flags("-XX:-UseCompressedClassPointers -XX:+UseCompactObjectHeaders", 25).header, 8);
assert_eq!(flags("-XX:-UseCompressedOops -XX:+UseCompressedOops", 21).ref_size, 4);
assert_eq!(flags("-XX:+UseCompressedOops -XX:-UseCompressedOops", 21).ref_size, 8);
}
#[test]
fn object_alignment_accepts_powers_of_two_from_8_to_256() {
for good in [8u16, 16, 32, 64, 128, 256] {
assert_eq!(flags(&format!("-XX:ObjectAlignmentInBytes={good}"), 25).alignment, good);
}
for bad in ["0", "4", "7", "12", "255", "512", "abc", "", "999999999999"] {
assert_eq!(flags(&format!("-XX:ObjectAlignmentInBytes={bad}"), 25).alignment, 8, "{bad:?}");
}
let wide = flags("-XX:+UseCompressedOops -XX:ObjectAlignmentInBytes=256", 21);
assert_eq!(wide.instance_size(1), 256);
assert_eq!(wide.instance_size(300), 512);
assert_eq!(wide.array_size(BasicType::Byte, 1), 256);
}
#[test]
fn instance_sizes_match_the_jvm() {
let at = |sizes: &[u64], name: &str| sizes[SHAPES.iter().position(|n| *n == name).unwrap()];
let coops = shape_sizes(flags(J21_25_1G, 21));
for (name, size) in
[("A", 16), ("B", 24), ("C", 32), ("D", 32), ("E", 24), ("F", 16), ("G", 16), ("H", 24), ("Node", 32)]
{
assert_eq!(at(&coops, name), size, "{name} with compressed oops");
}
let wide = shape_sizes(flags(J21_25_40G, 21));
for (name, size) in [("D", 40), ("E", 40), ("Node", 40)] {
assert_eq!(at(&wide, name), size, "{name} without compressed oops");
}
let compact = shape_sizes(flags(J25_COMPACT, 25));
for (name, size) in [("G", 8), ("B", 16), ("C", 24), ("D", 32), ("Node", 24)] {
assert_eq!(at(&compact, name), size, "{name} with compact headers");
}
}
#[test]
fn instance_size_bounds() {
assert_eq!(VmLayout::DEFAULT.instance_size(0), 16);
assert_eq!(flags("-XX:+UseCompactObjectHeaders", 25).instance_size(0), 8);
assert_eq!(flags("-XX:ObjectAlignmentInBytes=16", 21).instance_size(1), 16);
assert_eq!(flags("-XX:ObjectAlignmentInBytes=16", 21).instance_size(8), 32);
}
#[test]
fn array_sizes() {
let d = VmLayout::DEFAULT;
assert_eq!(d.array_size(BasicType::Long, 2), 32);
assert_eq!(d.array_size(BasicType::Byte, 5), 24);
assert_eq!(d.array_size(BasicType::Object, 3), 32);
let compact = flags("-XX:+UseCompactObjectHeaders", 25);
assert_eq!(compact.array_size(BasicType::Byte, 5), 24);
assert_eq!(compact.array_size(BasicType::Int, 1), 16);
assert_eq!(compact.array_size(BasicType::Long, 1), 24);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 21).array_size(BasicType::Byte, 1), 32);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 25).array_size(BasicType::Byte, 1), 24);
assert_eq!(flags("-XX:-UseCompressedClassPointers", 25).array_size(BasicType::Long, 1), 32);
// Empty arrays and the largest length fit.
assert_eq!(d.array_size(BasicType::Int, 0), 16);
assert_eq!(d.array_size(BasicType::Long, u64::from(u32::MAX)), 16 + 8 * u64::from(u32::MAX));
}
#[test]
fn arrays_in_the_graph_follow_the_layout() {
let g = graph_of(&sample());
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x1400), 32, "long[2]");
assert_eq!(size_at(&g, &m, 0x1500), 32, "Object[3]");
assert_eq!(size_at(&g, &m, 0x1100), 24, "Node");
let wide = model(&g, flags("-XX:-UseCompressedOops", 21), None);
assert_eq!(size_at(&g, &wide, 0x1500), 40, "Object[3] with 8-byte references");
assert_eq!(m.size_of(&g, g.root), 0, "the super-root has no size");
assert_eq!(m.source, SizeSource::Layout);
}
#[test]
fn histogram_wins_for_instances() {
use BasicType::Int;
let fields: Vec<_> = (0..23).map(|i| (format!("f{i}"), Int)).collect();
let fields: Vec<(&str, BasicType)> = fields.iter().map(|(n, t)| (n.as_str(), *t)).collect();
let mut heap = MiniHeap::new();
heap.class("java.lang.Thread", ROOT, &fields)
.class("Plain", ROOT, &[("f", Int)])
.object(0x1000, "java.lang.Thread", &[])
.object(0x1100, "Plain", &[]);
let g = graph_of(&heap);
let without = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &without, 0x1000), 104);
let histogram = [entry("java.lang.Thread", 13, 1560), entry("Unrelated", 2, 64)];
let with = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(with.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &with, 0x1000), 120);
assert_eq!(size_at(&g, &with, 0x1100), 16, "no row: layout");
}
#[test]
fn histogram_rows_that_do_not_divide_or_are_empty_fall_back_to_layout() {
let mut heap = MiniHeap::new();
heap.class("Plain", ROOT, &[("f", BasicType::Int)]).object(0x1000, "Plain", &[]);
let g = graph_of(&heap);
for row in [entry("Plain", 3, 100), entry("Plain", 0, 0), entry("Plain", 0, 48), entry("Plain", 2, 0)] {
let m = model(&g, VmLayout::DEFAULT, Some(std::slice::from_ref(&row)));
assert_eq!(m.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &m, 0x1000), 16, "{row:?}");
}
// The same name twice in the histogram cannot be told apart either.
let twice = [entry("Plain", 1, 48), entry("Plain", 1, 64)];
assert_eq!(size_at(&g, &model(&g, VmLayout::DEFAULT, Some(&twice)), 0x1000), 16);
}
#[test]
fn histogram_matches_arrays_by_display_name_but_never_sizes_them() {
let g = graph_of(&sample());
let histogram = [entry("java.lang.Object[]", 1, 4096), entry("long[]", 1, 4096), entry("Node", 4, 160)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(size_at(&g, &m, 0x1500), 32);
assert_eq!(size_at(&g, &m, 0x1400), 32);
assert_eq!(size_at(&g, &m, 0x1100), 40, "Node from the histogram");
}
#[test]
fn a_name_shared_by_several_classes_never_takes_a_histogram_size() {
use BasicType::{Int, Long};
let mut heap = MiniHeap::new();
heap.class("dup.Foo", ROOT, &[("a", Int)])
.object(0x1000, "dup.Foo", &[])
.class("dup.Foo", ROOT, &[("a", Long), ("b", Long)])
.object(0x1100, "dup.Foo", &[])
.class("Solo", ROOT, &[("a", Int)])
.object(0x1200, "Solo", &[]);
let g = graph_of(&heap);
let histogram = [entry("dup.Foo", 2, 80), entry("Solo", 1, 48)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(size_at(&g, &m, 0x1000), 16, "first dup.Foo uses its layout");
assert_eq!(size_at(&g, &m, 0x1100), 32, "second dup.Foo uses its layout");
assert_eq!(size_at(&g, &m, 0x1200), 48, "Solo still takes its histogram size");
let (first, second, solo) = (slot_at(&g, 0x1000), slot_at(&g, 0x1100), slot_at(&g, 0x1200));
assert_ne!(first, second);
assert!(m.is_ambiguous(first) && m.is_ambiguous(second));
assert!(!m.is_ambiguous(solo));
let ambiguous: Vec<_> = (0..g.classes.len() as u32).filter(|&s| m.is_ambiguous(s)).collect();
assert_eq!(ambiguous.len(), 2, "no other class is flagged");
assert!(!m.is_ambiguous(u32::MAX), "slots past the classes are never ambiguous");
}
#[test]
fn ambiguity_covers_array_classes_and_ignores_the_source() {
let mut heap = MiniHeap::new();
heap.class("[Lx.Y;", ROOT, &[]).class("[Lx/Y;", ROOT, &[]).class("Solo", ROOT, &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
let flagged: Vec<_> =
g.classes.iter().enumerate().filter(|(i, _)| m.is_ambiguous(*i as u32)).map(|(_, c)| c.name.as_str()).collect();
assert_eq!(flagged, ["[Lx.Y;", "[Lx.Y;"]);
}
/// A dump with three classes: `java.lang.Class` (three reference fields), `Statics` (two reference
/// statics, one null) and `Plain`; `at` is where each class object lives.
fn class_dump() -> Vec<u8> {
use BasicType::Object;
let mut w = HprofWriter::new();
w.string(1, "java/lang/Class").load_class(0x100, 1).class(
0x100,
0,
&[(10, Object), (11, Object), (12, Object)],
&[],
);
w.string(2, "Statics").load_class(0x200, 2).class(0x200, 0, &[], &[(13, 0x300), (13, 0)]);
w.string(3, "Plain").load_class(0x300, 3).class(0x300, 0, &[], &[]);
w.finish()
}
#[test]
fn class_objects_use_the_layout_even_when_the_histogram_names_them() {
let g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let histogram = [entry("java.lang.Class", 3, 3000)];
let by_histogram = model(&g, VmLayout::DEFAULT, Some(&histogram));
let by_layout = model(&g, VmLayout::DEFAULT, None);
assert_eq!(by_histogram.source, SizeSource::Jvm);
for m in [&by_histogram, &by_layout] {
// java.lang.Class has 12 bytes of fields: 24; Statics adds two references; every mirror
// gets the 32 bytes assumed for an unknown JVM (the histogram row does not fit).
assert_eq!(size_at(&g, m, 0x100), 56, "java.lang.Class itself");
assert_eq!(size_at(&g, m, 0x200), 64, "two reference statics");
assert_eq!(size_at(&g, m, 0x300), 56, "no statics");
}
// 8-byte references: 40 for java.lang.Class, 16 for the statics, 44 injected (JDK 21), aligned.
let wide = model(&g, flags("-XX:-UseCompressedOops", 21), None);
assert_eq!(size_at(&g, &wide, 0x200), 104);
}
#[test]
fn primitive_statics_add_to_the_class_object() {
let mut g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let statics = g.classes.iter().position(|c| c.name == "Statics").unwrap();
g.classes[statics].static_bytes = 12;
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x200), 80, "24 + 12 + 2 × 4 + 32 injected = 76, aligned");
}
#[test]
fn a_dump_without_java_lang_class_sizes_class_objects_as_empty_mirrors() {
let mut w = HprofWriter::new();
w.string(1, "Statics").load_class(0x200, 1).class(0x200, 0, &[], &[(2, 0), (2, 0), (2, 0)]);
let g = build_graph(&w.finish(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x200), 64, "16 + 3 × 4 + 32 injected = 60, aligned");
}
fn one_class_two_instances(stride: u64) -> (HeapGraph, ShallowModel) {
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", BasicType::Byte)]).object(0x1000, "A", &[]).object(0x1000 + stride, "A", &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
(g, m)
}
#[test]
fn address_gaps_for_external_dumps() {
let mut heap = MiniHeap::new();
heap.class("A", ROOT, &[("f", BasicType::Byte)])
.class("Wide", ROOT, &[("f", BasicType::Long)])
.class("Module", ROOT, &[("f", BasicType::Long)])
.object(0x1000, "A", &[])
.object(0x1018, "A", &[])
.object(0x1030, "A", &[])
.object(0x1048, "Module", &[])
.object(0x2000, "Wide", &[])
.object(0x3000, "Wide", &[]);
let g = graph_of(&heap);
let m = model(&g, VmLayout::DEFAULT, None);
assert_eq!(size_at(&g, &m, 0x1000), 24, "layout says 16, the smallest gap between As is 24");
assert_eq!(size_at(&g, &m, 0x1030), 24, "every A takes the type's size");
assert_eq!(size_at(&g, &m, 0x1048), 24, "a single instance has no gap to trust: layout");
assert_eq!(size_at(&g, &m, 0x2000), 24, "a gap beyond layout + 64 is not an object size: layout");
assert_eq!(m.source, SizeSource::Layout, "one of three types agrees");
}
#[test]
fn address_gaps_are_taken_within_layout_and_layout_plus_64() {
for (stride, expected) in [(8, 16), (16, 16), (24, 24), (80, 80), (88, 16)] {
let (g, m) = one_class_two_instances(stride);
assert_eq!(size_at(&g, &m, 0x1000), expected, "stride {stride}");
assert_eq!(
m.source,
if expected == stride && stride >= 16 { SizeSource::AddressGaps } else { SizeSource::Layout }
);
}
}
/// `used` types with two instances 16 bytes apart (the layout size), `far` types with two
/// instances 0x1000 apart (a gap that is no object size) and `single` types with one instance.
fn types_by_gap_use(used: usize, far: usize, single: usize) -> SizeSource {
let mut heap = MiniHeap::new();
for i in 0..used + far + single {
let name = format!("T{i}");
heap.class(&name, ROOT, &[("f", BasicType::Byte)]);
let base = 0x10_0000 * (i as u64 + 1);
heap.object(base, &name, &[]);
if i < used {
heap.object(base + 16, &name, &[]);
} else if i < used + far {
heap.object(base + 0x1000, &name, &[]);
}
}
let g = graph_of(&heap);
model(&g, VmLayout::DEFAULT, None).source
}
#[test]
fn address_gap_source_needs_ninety_percent_of_the_types_that_have_a_gap() {
assert_eq!(types_by_gap_use(10, 0, 0), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(9, 1, 0), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(8, 2, 0), SizeSource::Layout);
// Types with one instance have no gap: they count neither for nor against.
assert_eq!(types_by_gap_use(9, 1, 5), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(8, 2, 5), SizeSource::Layout);
assert_eq!(types_by_gap_use(10, 0, 400), SizeSource::AddressGaps);
assert_eq!(types_by_gap_use(0, 0, 3), SizeSource::Layout, "no type has a gap");
assert_eq!(types_by_gap_use(0, 2, 0), SizeSource::Layout);
assert_eq!(types_by_gap_use(0, 0, 0), SizeSource::Layout, "nothing to size");
}
/// A dump of `java.lang.Class` (three reference fields, so 24 bytes of layout) and one class per
/// entry of `statics`: `(reference statics, primitive static bytes)`. Class `i` is at `0x1000 + 0x100 × i`.
fn mirror_heap(statics: &[(usize, u64)]) -> (HeapGraph, Vec<u64>) {
use BasicType::Object;
let mut w = HprofWriter::new();
w.string(1, "java/lang/Class").load_class(0x100, 1).class(
0x100,
0,
&[(10, Object), (11, Object), (12, Object)],
&[],
);
let mut addrs = Vec::new();
for (i, &(refs, _)) in statics.iter().enumerate() {
let (addr, name_id) = (0x1000 + 0x100 * i as u64, 100 + i as u64);
w.string(name_id, &format!("C{i}")).load_class(addr, name_id).class(addr, 0, &[], &vec![(13, 0); refs]);
addrs.push(addr);
}
let mut g = build_graph(&w.finish(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
for (i, &(_, bytes)) in statics.iter().enumerate() {
let class = g.classes.iter().position(|c| c.name == format!("C{i}")).unwrap();
g.classes[class].static_bytes = bytes;
}
(g, addrs)
}
/// What HotSpot would make of those classes if it added `injected` bytes to every mirror.
fn expected_mirrors(statics: &[(usize, u64)], injected: u64) -> Vec<u64> {
statics.iter().map(|&(refs, bytes)| (24 + bytes + 4 * refs as u64 + injected).next_multiple_of(8)).collect()
}
const MIRROR_STATICS: [(usize, u64); 8] = [(0, 0), (3, 0), (1, 12), (6, 8), (2, 0), (9, 24), (0, 4), (4, 1)];
#[test]
fn class_objects_get_the_injected_bytes_the_histogram_implies() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// java.lang.Class itself and the eight classes.
let mut sizes = expected_mirrors(&MIRROR_STATICS, 36);
let java_lang_class = 24 + 36;
let total = sizes.iter().sum::<u64>() + (java_lang_class as u64).next_multiple_of(8);
sizes.insert(0, (java_lang_class as u64).next_multiple_of(8));
let histogram = [entry("java.lang.Class", 9, total)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
assert_eq!(m.source, SizeSource::Jvm);
assert_eq!(size_at(&g, &m, 0x100), sizes[0]);
for (i, &addr) in addrs.iter().enumerate() {
assert_eq!(size_at(&g, &m, addr), sizes[i + 1], "class {i}");
}
let spread: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert!(spread.iter().any(|&s| s != spread[0]), "statics still make the mirrors differ: {spread:?}");
}
#[test]
fn injected_bytes_are_found_for_every_multiple_of_four() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
for injected in (0..=128).step_by(4) {
let total =
expected_mirrors(&MIRROR_STATICS, injected).iter().sum::<u64>() + (24 + injected).next_multiple_of(8);
let histogram = [entry("java.lang.Class", 9, total)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
let got: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert_eq!(got, expected_mirrors(&MIRROR_STATICS, injected), "injected {injected}");
}
}
#[test]
fn the_fit_tolerates_mirrors_the_dump_has_no_class_for() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// The histogram counts the same average mirror 90 times more often (CDS-archived classes).
let total = expected_mirrors(&MIRROR_STATICS, 28).iter().sum::<u64>() + (24u64 + 28).next_multiple_of(8);
let histogram = [entry("java.lang.Class", 9 * 91, total * 91)];
let m = model(&g, VmLayout::DEFAULT, Some(&histogram));
let got: Vec<_> = addrs.iter().map(|&a| size_at(&g, &m, a)).collect();
assert_eq!(got, expected_mirrors(&MIRROR_STATICS, 28));
}
fn mirror_sizes(g: &HeapGraph, addrs: &[u64], layout: VmLayout, histogram: Option<&[HistogramEntry]>) -> Vec<u64> {
let m = model(g, layout, histogram);
addrs.iter().map(|&a| size_at(g, &m, a)).collect()
}
#[test]
fn a_histogram_row_that_cannot_describe_these_mirrors_falls_back_to_the_table() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
let rows = [
entry("java.lang.Class", 9, 9 * 5000),
entry("java.lang.Class", 9, 9),
entry("java.lang.Class", 0, 0),
entry("Something.Else", 9, 9 * 100),
];
for row in &rows {
let one = std::slice::from_ref(row);
// Unknown JVM: the documented 32; JDK 11 with 4-byte references: 36.
assert_eq!(
mirror_sizes(&g, &addrs, VmLayout::DEFAULT, Some(one)),
expected_mirrors(&MIRROR_STATICS, 32),
"{row:?}"
);
assert_eq!(
mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(one)),
expected_mirrors(&MIRROR_STATICS, 36),
"{row:?}"
);
}
let twice = [entry("java.lang.Class", 9, 9 * 60), entry("java.lang.Class", 9, 9 * 60)];
assert_eq!(mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(&twice)), expected_mirrors(&MIRROR_STATICS, 36));
}
#[test]
fn an_accepted_fit_beats_the_table() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
let total = expected_mirrors(&MIRROR_STATICS, 20).iter().sum::<u64>() + 48;
let histogram = [entry("java.lang.Class", 9, total)];
// JDK 11's table says 36; the histogram says 20.
assert_eq!(mirror_sizes(&g, &addrs, flags(J8_11_1G, 11), Some(&histogram)), expected_mirrors(&MIRROR_STATICS, 20));
}
#[test]
fn injected_mirror_bytes_table() {
// (JDK, 4-byte references, 8-byte references): the measured cells, then the JDKs between.
for (major, narrow, wide) in
[(8, 36, 44), (11, 36, 44), (21, 36, 44), (22, 36, 44), (23, 36, 40), (24, 36, 40), (25, 28, 32), (26, 28, 32)]
{
assert_eq!(mirrors::injected_mirror_bytes(major, 4), Some(narrow), "JDK {major}, compressed oops");
assert_eq!(mirrors::injected_mirror_bytes(major, 8), Some(wide), "JDK {major}, 8-byte references");
}
assert_eq!(mirrors::injected_mirror_bytes(21, 2), None);
}
#[test]
fn without_a_histogram_class_objects_get_the_injected_bytes_of_their_jvm() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
// The class dump of the fixture has 4-byte references; the layout under test decides the table
// cell, so use layouts whose reference size matches it.
for (layout, injected) in [
(flags(J8_11_1G, 8), 36),
(flags(J8_11_1G, 11), 36),
(flags(J21_25_1G, 21), 36),
(flags(J21_25_1G, 23), 36),
(flags(J21_25_1G, 25), 28),
(flags(J25_COMPACT, 25), 28),
] {
assert_eq!(mirror_sizes(&g, &addrs, layout, None), expected_mirrors(&MIRROR_STATICS, injected), "{layout:?}");
}
}
#[test]
fn eight_byte_references_use_their_own_cells() {
let g = build_graph(&class_dump(), &Store::Ram, &no_progress, &CancellationToken::new()).unwrap();
// java.lang.Class = 12 + 3 × 8 = 36 → 40; Plain has no statics: 40 + injected, aligned.
for (major, injected) in [(8, 44), (21, 44), (23, 40), (25, 32)] {
let m = model(&g, flags(J21_25_40G, major), None);
assert_eq!(size_at(&g, &m, 0x300), (40u64 + injected).next_multiple_of(8), "JDK {major}");
}
}
#[test]
fn an_unknown_jvm_uses_the_midpoint_of_the_compressed_oop_cells() {
let (g, addrs) = mirror_heap(&MIRROR_STATICS);
assert_eq!(mirror_sizes(&g, &addrs, VmLayout::DEFAULT, None), expected_mirrors(&MIRROR_STATICS, 32));
// Flags without any -XX: option say nothing about the JVM either.
assert_eq!(mirror_sizes(&g, &addrs, flags("", 21), None), expected_mirrors(&MIRROR_STATICS, 32));
assert_eq!(VmLayout::DEFAULT.java_major, None);
}
#[test]
fn cancelled_token_stops_the_model() {
let g = graph_of(&sample());
let cancel = CancellationToken::new();
cancel.cancel();
assert!(matches!(ShallowModel::new(&g, VmLayout::DEFAULT, None, &cancel), Err(HeapError::Cancelled)));
let histogram = [entry("Node", 1, 24)];
assert!(matches!(ShallowModel::new(&g, VmLayout::DEFAULT, Some(&histogram), &cancel), Err(HeapError::Cancelled)));
}
+66
View File
@@ -0,0 +1,66 @@
//! Text of `java.lang.String` objects.
use crate::hprof::{BasicType, HeapRecord};
use crate::index::{HeapIndex, id_value, instance_fields};
const STRING: &str = "java.lang.String";
const LATIN1: u8 = 0;
const UTF16: u8 = 1;
/// The text of String `object`, cut to `max_chars` chars plus `…`. `None` for anything that is
/// not a String, or one whose `value` / `coder` the dump does not hold as expected.
///
/// JDK 9+ keeps a `byte[]` and a `coder`; UTF-16 bytes are in the JVM's native order, which
/// hprof copies as is, read here as little-endian (x86-64, AArch64; a big-endian JVM's text would
/// come out garbled). JDK 8 keeps a `char[]`, whose elements hprof writes big-endian.
pub(crate) fn string_preview(index: &HeapIndex, object: u32, max_chars: usize) -> Option<String> {
let slot = *index.types.get(object as usize)?;
if index.class_info.get(slot as usize)?.name != STRING {
return None;
}
let Some(HeapRecord::Instance { data, .. }) = index.record(object) else { return None };
let fields = instance_fields(index, slot, data);
let value = fields.iter().find(|f| f.name == "value" && f.ty == BasicType::Object)?;
let array = index.ids.rank(id_value(value.bytes).filter(|&id| id != 0)?)?;
let coder = fields.iter().find(|f| f.name == "coder" && f.ty == BasicType::Byte).and_then(|f| f.bytes.first());
let Some(HeapRecord::PrimitiveArray { ty, data, .. }) = index.record(array) else { return None };
match (ty, coder) {
(BasicType::Byte, Some(&LATIN1)) => Some(cut(data.iter().map(|&b| char::from(b)), max_chars)),
(BasicType::Byte, Some(&UTF16)) => utf16(data, u16::from_le_bytes, max_chars),
(BasicType::Char, _) => utf16(data, u16::from_be_bytes, max_chars),
_ => None,
}
}
/// The text of `char[]` `object`, as [`string_preview`] cuts it.
pub(crate) fn char_array_text(index: &HeapIndex, object: u32, max_chars: usize) -> Option<String> {
match index.record(object)? {
HeapRecord::PrimitiveArray { ty: BasicType::Char, data, .. } => utf16(data, u16::from_be_bytes, max_chars),
_ => None,
}
}
/// An odd byte count is `None`; a lone surrogate becomes U+FFFD.
fn utf16(bytes: &[u8], unit: fn([u8; 2]) -> u16, max_chars: usize) -> Option<String> {
if !bytes.len().is_multiple_of(2) {
return None;
}
let units = bytes.as_chunks::<2>().0.iter().map(|&pair| unit(pair));
Some(cut(char::decode_utf16(units).map(|c| c.unwrap_or(char::REPLACEMENT_CHARACTER)), max_chars))
}
/// Chars are whole code points, so a cut never splits a surrogate pair.
fn cut(chars: impl Iterator<Item = char>, max_chars: usize) -> String {
let mut out = String::new();
for (i, c) in chars.enumerate() {
if i == max_chars {
out.push('…');
break;
}
out.push(c);
}
out
}
#[cfg(test)]
mod tests;
+113
View File
@@ -0,0 +1,113 @@
use super::*;
use crate::hprof::{BasicType, HeapRootKind};
use crate::test_index::{analyzed, obj};
use crate::testing::{MiniHeap, sample_with_string};
const AT: u64 = 0x1000;
const ARRAY: u64 = 0x2000;
const JDK9: [(&str, BasicType); 2] = [("value", BasicType::Object), ("coder", BasicType::Byte)];
fn preview(heap: &MiniHeap, addr: u64) -> Option<String> {
let index = analyzed(heap);
string_preview(&index, obj(&index, addr).0, 100)
}
fn text(text: &str, latin1: bool) -> Option<String> {
let mut heap = MiniHeap::new();
heap.string(AT, text, latin1).root(HeapRootKind::JniGlobal, AT);
preview(&heap, AT)
}
/// A `java.lang.String` class with `fields`, and one String at [`AT`] with `values`.
fn custom(fields: &[(&str, BasicType)], values: &[(&str, u64)], array: impl FnOnce(&mut MiniHeap)) -> Option<String> {
let mut heap = MiniHeap::new();
heap.class("java.lang.String", Some("java.lang.Object"), fields);
array(&mut heap);
heap.object(AT, "java.lang.String", values);
preview(&heap, AT)
}
#[test]
fn string_preview_latin1_and_utf16() {
assert_eq!(preview(&sample_with_string(), 0x1800).as_deref(), Some("v42"));
assert_eq!(text("hé", false).as_deref(), Some("hé"));
assert_eq!(text("ÿ é", true).as_deref(), Some("ÿ é"));
assert_eq!(text("", true).as_deref(), Some(""));
let long = "a".repeat(150);
assert_eq!(text(&long, true), Some(format!("{}…", "a".repeat(100))));
assert_eq!(text(&long, false), Some(format!("{}…", "a".repeat(100))));
let exactly = "b".repeat(100);
assert_eq!(text(&exactly, true), Some(exactly));
}
#[test]
fn truncation_keeps_surrogate_pairs_whole() {
let s = format!("{}😀{}", "a".repeat(99), "b".repeat(50));
assert_eq!(text(&s, false), Some(format!("{}😀…", "a".repeat(99))));
// A lone surrogate becomes the replacement character.
let lone: Vec<u8> = [0x68u16, 0xD800, 0x69].iter().flat_map(|u| u.to_le_bytes()).collect();
let got = custom(&JDK9, &[("value", ARRAY), ("coder", 1)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, &lone);
});
assert_eq!(got.as_deref(), Some("h\u{FFFD}i"));
}
#[test]
fn jdk8_strings_hold_big_endian_chars() {
let chars: Vec<u8> = "hé".encode_utf16().flat_map(u16::to_be_bytes).collect();
let fields = [("value", BasicType::Object), ("hash", BasicType::Int)];
let got = custom(&fields, &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Char, &chars);
});
assert_eq!(got.as_deref(), Some("hé"));
}
#[test]
fn malformed_strings_have_no_preview() {
let odd = custom(&JDK9, &[("value", ARRAY), ("coder", 1)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, &[0x68, 0, 0x69]);
});
assert_eq!(odd, None);
let bad_coder = custom(&JDK9, &[("value", ARRAY), ("coder", 2)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(bad_coder, None);
let no_value = custom(&[("coder", BasicType::Byte)], &[], |_| {});
assert_eq!(no_value, None);
let no_coder = custom(&[("value", BasicType::Object)], &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(no_coder, None);
let null_value = custom(&JDK9, &[], |_| {});
assert_eq!(null_value, None);
let int_array = custom(&JDK9, &[("value", ARRAY)], |h| {
h.prim(ARRAY, BasicType::Int, 2);
});
assert_eq!(int_array, None);
let not_an_array = custom(&JDK9, &[("value", ARRAY)], |h| {
h.object(ARRAY, "java.lang.Object", &[]);
});
assert_eq!(not_an_array, None);
// `value` stored as a long instead of a reference.
let wrong_type = custom(&[("value", BasicType::Long), ("coder", BasicType::Byte)], &[("value", ARRAY)], |h| {
h.prim_bytes(ARRAY, BasicType::Byte, b"hi");
});
assert_eq!(wrong_type, None);
}
#[test]
fn only_strings_have_a_preview() {
let mut heap = MiniHeap::new();
heap.class("Text", Some("java.lang.Object"), &JDK9).prim_bytes(ARRAY, BasicType::Byte, b"hi").object(
AT,
"Text",
&[("value", ARRAY)],
);
assert_eq!(preview(&heap, AT), None);
assert_eq!(preview(&heap, ARRAY), None);
let index = analyzed(&heap);
for bad in [index.root(), u32::MAX] {
assert_eq!(string_preview(&index, bad, 100), None);
}
}
+35
View File
@@ -0,0 +1,35 @@
//! Analyzed fixtures for the query tests.
use ide_core::CancellationToken;
use crate::analysis::{AnalyzeOptions, analyze};
use crate::error::HeapError;
use crate::index::{HeapIndex, HeapObjectRef};
use crate::progress::HeapPhase;
use crate::testing::MiniHeap;
fn no_progress(_: HeapPhase, _: u64, _: u64) -> Result<(), HeapError> {
Ok(())
}
/// `heap` analyzed in memory, with layout sizes.
pub(crate) fn analyzed(heap: &MiniHeap) -> HeapIndex {
analyzed_bytes(&heap.bytes())
}
pub(crate) fn analyzed_bytes(hprof: &[u8]) -> HeapIndex {
let dir = tempfile::tempdir().unwrap();
let dump = dir.path().join("dump.hprof");
std::fs::write(&dump, hprof).unwrap();
let opts = AnalyzeOptions { in_memory: true, ..AnalyzeOptions::default() };
analyze(&dump, dir.path(), opts, &no_progress, &CancellationToken::new()).unwrap()
}
pub(crate) fn obj(index: &HeapIndex, addr: u64) -> HeapObjectRef {
HeapObjectRef(index.ids.rank(addr).unwrap_or_else(|| panic!("{addr:#x} is not in the dump")))
}
/// The slot of the class named `name`.
pub(crate) fn slot(index: &HeapIndex, name: &str) -> u32 {
index.classes.iter().position(|c| c.name == name).unwrap_or_else(|| panic!("no class {name}")) as u32
}
+430
View File
@@ -0,0 +1,430 @@
//! Builders for `.hprof` fixtures. They panic on misuse (unknown class name, text that does not
//! fit the requested encoding): they are test helpers, not input handling.
use std::collections::HashMap;
use std::path::Path;
use crate::hprof::{BasicType, HeapRootKind};
const TAG_UTF8: u8 = 0x01;
const TAG_LOAD_CLASS: u8 = 0x02;
const TAG_HEAP_SEGMENT: u8 = 0x1C;
const TAG_HEAP_DUMP_END: u8 = 0x2C;
const TIMESTAMP_MS: u64 = 1_700_000_000_000;
fn put_id(out: &mut Vec<u8>, id_size: u8, id: u64) {
if id_size == 4 {
out.extend_from_slice(&(id as u32).to_be_bytes());
} else {
out.extend_from_slice(&id.to_be_bytes());
}
}
/// Writes records one by one; heap sub-records are collected into a single heap-dump segment that
/// `finish` places after the UTF8 / LOAD CLASS records, followed by HEAP DUMP END.
pub struct HprofWriter {
id_size: u8,
top: Vec<u8>,
heap: Vec<u8>,
class_serial: u32,
}
impl Default for HprofWriter {
fn default() -> Self {
Self::new()
}
}
impl HprofWriter {
pub fn new() -> Self {
Self::with_id_size(8)
}
/// `id_size` is 4 or 8; ids are truncated to the width.
pub fn with_id_size(id_size: u8) -> Self {
assert!(id_size == 4 || id_size == 8, "hprof ids are 4 or 8 bytes");
let mut top = b"JAVA PROFILE 1.0.2\0".to_vec();
top.extend_from_slice(&u32::from(id_size).to_be_bytes());
top.extend_from_slice(&TIMESTAMP_MS.to_be_bytes());
Self { id_size, top, heap: Vec::new(), class_serial: 0 }
}
fn top_record(&mut self, tag: u8, body: &[u8]) {
self.top.push(tag);
self.top.extend_from_slice(&0u32.to_be_bytes());
self.top.extend_from_slice(&(body.len() as u32).to_be_bytes());
self.top.extend_from_slice(body);
}
fn heap_id(&mut self, id: u64) {
put_id(&mut self.heap, self.id_size, id);
}
fn heap_u4(&mut self, v: u32) {
self.heap.extend_from_slice(&v.to_be_bytes());
}
pub fn string(&mut self, id: u64, s: &str) -> &mut Self {
let mut body = Vec::new();
put_id(&mut body, self.id_size, id);
body.extend_from_slice(s.as_bytes());
self.top_record(TAG_UTF8, &body);
self
}
pub fn load_class(&mut self, class_id: u64, name_id: u64) -> &mut Self {
self.class_serial += 1;
let mut body = self.class_serial.to_be_bytes().to_vec();
put_id(&mut body, self.id_size, class_id);
body.extend_from_slice(&0u32.to_be_bytes());
put_id(&mut body, self.id_size, name_id);
self.top_record(TAG_LOAD_CLASS, &body);
self
}
/// `statics` are `(name id, object id)` reference statics.
pub fn class(&mut self, id: u64, super_id: u64, fields: &[(u64, BasicType)], statics: &[(u64, u64)]) -> &mut Self {
self.class_with_loader(id, super_id, 0, fields, statics)
}
pub fn class_with_loader(
&mut self,
id: u64,
super_id: u64,
loader_id: u64,
fields: &[(u64, BasicType)],
statics: &[(u64, u64)],
) -> &mut Self {
self.heap.push(0x20);
self.heap_id(id);
self.heap_u4(0);
self.heap_id(super_id);
self.heap_id(loader_id);
for _ in 0..4 {
self.heap_id(0); // signers, protection domain, 2 reserved
}
let instance_size: usize = fields.iter().map(|(_, ty)| ty.hprof_size(self.id_size)).sum();
self.heap_u4(instance_size as u32);
self.heap.extend_from_slice(&0u16.to_be_bytes()); // constant pool
self.heap.extend_from_slice(&(statics.len() as u16).to_be_bytes());
for &(name, object) in statics {
self.heap_id(name);
self.heap.push(BasicType::Object as u8);
self.heap_id(object);
}
self.heap.extend_from_slice(&(fields.len() as u16).to_be_bytes());
for &(name, ty) in fields {
self.heap_id(name);
self.heap.push(ty as u8);
}
self
}
pub fn instance(&mut self, id: u64, class_id: u64, data: &[u8]) -> &mut Self {
self.heap.push(0x21);
self.heap_id(id);
self.heap_u4(0);
self.heap_id(class_id);
self.heap_u4(data.len() as u32);
self.heap.extend_from_slice(data);
self
}
pub fn obj_array(&mut self, id: u64, class_id: u64, items: &[u64]) -> &mut Self {
self.heap.push(0x22);
self.heap_id(id);
self.heap_u4(0);
self.heap_u4(items.len() as u32);
self.heap_id(class_id);
for &item in items {
self.heap_id(item);
}
self
}
/// An array of `len` zeroed elements.
pub fn prim_array(&mut self, id: u64, ty: BasicType, len: u32) -> &mut Self {
let bytes = vec![0u8; len as usize * ty.hprof_size(self.id_size)];
self.prim_bytes(id, ty, &bytes)
}
/// An array whose elements are `bytes` (big-endian, as stored in the dump).
pub fn prim_bytes(&mut self, id: u64, ty: BasicType, bytes: &[u8]) -> &mut Self {
let size = ty.hprof_size(self.id_size);
assert!(ty != BasicType::Object && bytes.len().is_multiple_of(size), "{ty:?} array needs whole elements");
self.heap.push(0x23);
self.heap_id(id);
self.heap_u4(0);
self.heap_u4((bytes.len() / size) as u32);
self.heap.push(ty as u8);
self.heap.extend_from_slice(bytes);
self
}
/// Kinds that carry a thread serial get serial 1; see [`Self::root_in_thread`].
pub fn root(&mut self, kind: HeapRootKind, id: u64) -> &mut Self {
self.root_in_thread(kind, id, 1)
}
/// `serial` is ignored for kinds that do not carry one.
pub fn root_in_thread(&mut self, kind: HeapRootKind, id: u64, serial: u32) -> &mut Self {
let (tag, has_serial, trailing) = match kind {
HeapRootKind::Unknown => (0xFF, false, Trailing::None),
HeapRootKind::JniGlobal => (0x01, false, Trailing::Id),
HeapRootKind::JniLocal => (0x02, true, Trailing::U4),
HeapRootKind::JavaFrame => (0x03, true, Trailing::U4),
HeapRootKind::NativeStack => (0x04, true, Trailing::None),
HeapRootKind::StickyClass => (0x05, false, Trailing::None),
HeapRootKind::ThreadBlock => (0x06, true, Trailing::None),
HeapRootKind::MonitorUsed => (0x07, false, Trailing::None),
HeapRootKind::ThreadObject => (0x08, true, Trailing::U4),
};
self.heap.push(tag);
self.heap_id(id);
if has_serial {
self.heap_u4(serial);
}
match trailing {
Trailing::None => {}
Trailing::U4 => self.heap_u4(0),
Trailing::Id => self.heap_id(0),
}
self
}
pub fn finish(&self) -> Vec<u8> {
let mut out = self.top.clone();
out.push(TAG_HEAP_SEGMENT);
out.extend_from_slice(&0u32.to_be_bytes());
let heap_len = u32::try_from(self.heap.len()).expect("fixture heap segment exceeds 4 GiB");
out.extend_from_slice(&heap_len.to_be_bytes());
out.extend_from_slice(&self.heap);
out.push(TAG_HEAP_DUMP_END);
out.extend_from_slice(&0u32.to_be_bytes());
out.extend_from_slice(&0u32.to_be_bytes());
out
}
}
enum Trailing {
None,
U4,
Id,
}
struct MiniClass {
id: u64,
super_name: Option<String>,
fields: Vec<(String, BasicType)>,
}
/// Ids for classes, names and string backing arrays come from here, above any address a test
/// would pick by hand.
const GENERATED_ID_BASE: u64 = 1 << 32;
/// Heap fixtures by class and field name on top of [`HprofWriter`] (id size 8).
///
/// Class names are written in the HotSpot form (`.` becomes `/`); look-ups use the name as given.
pub struct MiniHeap {
w: HprofWriter,
next_id: u64,
classes: HashMap<String, MiniClass>,
}
impl Default for MiniHeap {
fn default() -> Self {
Self::new()
}
}
impl MiniHeap {
/// Starts with `java.lang.Object` and `java.lang.Class` already defined.
pub fn new() -> Self {
let mut heap = Self { w: HprofWriter::new(), next_id: GENERATED_ID_BASE, classes: HashMap::new() };
heap.class("java.lang.Object", None, &[]);
heap.class("java.lang.Class", Some("java.lang.Object"), &[]);
heap
}
fn fresh_id(&mut self) -> u64 {
self.next_id += 1;
self.next_id
}
fn class_id(&self, name: &str) -> u64 {
self.classes.get(name).unwrap_or_else(|| panic!("MiniHeap: class {name} is not defined")).id
}
/// `super_name: None` writes super id 0.
pub fn class(&mut self, name: &str, super_name: Option<&str>, fields: &[(&str, BasicType)]) -> &mut Self {
let super_id = super_name.map_or(0, |sup| self.class_id(sup));
let id = self.fresh_id();
let name_id = self.fresh_id();
let mut field_ids = Vec::with_capacity(fields.len());
for &(field, ty) in fields {
let field_id = self.fresh_id();
self.w.string(field_id, field);
field_ids.push((field_id, ty));
}
self.w.string(name_id, &name.replace('.', "/"));
self.w.load_class(id, name_id);
self.w.class(id, super_id, &field_ids, &[]);
self.w.root(HeapRootKind::StickyClass, id);
let fields = fields.iter().map(|&(field, ty)| (field.to_owned(), ty)).collect();
self.classes.insert(name.to_owned(), MiniClass { id, super_name: super_name.map(str::to_owned), fields });
self
}
/// Fields are laid out class first, then superclass, as HotSpot does. A field missing from
/// `values` is 0; narrower types keep the low bytes of the value.
pub fn object(&mut self, addr: u64, class: &str, values: &[(&str, u64)]) -> &mut Self {
let class_id = self.class_id(class);
let mut data = Vec::new();
let mut cursor = Some(class);
while let Some(name) = cursor {
let c = &self.classes[name];
for (field, ty) in &c.fields {
let value = values.iter().find(|(n, _)| *n == field.as_str()).map_or(0, |&(_, v)| v);
let width = ty.hprof_size(8);
data.extend_from_slice(&value.to_be_bytes()[8 - width..]);
}
cursor = c.super_name.as_deref();
}
self.w.instance(addr, class_id, &data);
self
}
pub fn array_of(&mut self, addr: u64, class: &str, items: &[u64]) -> &mut Self {
let class_id = self.class_id(class);
self.w.obj_array(addr, class_id, items);
self
}
/// A primitive array of `len` zeroed elements.
pub fn prim(&mut self, addr: u64, ty: BasicType, len: u32) -> &mut Self {
self.w.prim_array(addr, ty, len);
self
}
/// A primitive array holding `bytes` (big-endian elements).
pub fn prim_bytes(&mut self, addr: u64, ty: BasicType, bytes: &[u8]) -> &mut Self {
self.w.prim_bytes(addr, ty, bytes);
self
}
/// A `java.lang.String` (JDK 9+ layout: `value` byte[] + `coder`) at `addr`. `latin1` stores
/// one byte per char (coder 0, every char must be ≤ U+00FF); otherwise UTF-16LE (coder 1).
pub fn string(&mut self, addr: u64, text: &str, latin1: bool) -> &mut Self {
if !self.classes.contains_key("java.lang.String") {
let fields = [("value", BasicType::Object), ("coder", BasicType::Byte)];
self.class("java.lang.String", Some("java.lang.Object"), &fields);
}
let bytes: Vec<u8> = if latin1 {
text.chars()
.map(|c| u8::try_from(u32::from(c)).unwrap_or_else(|_| panic!("{c:?} is not Latin-1")))
.collect()
} else {
text.encode_utf16().flat_map(u16::to_le_bytes).collect()
};
let array = self.fresh_id();
self.w.prim_bytes(array, BasicType::Byte, &bytes);
self.object(addr, "java.lang.String", &[("value", array), ("coder", u64::from(!latin1))])
}
/// Kinds that carry a thread serial get serial 1.
pub fn root(&mut self, kind: HeapRootKind, addr: u64) -> &mut Self {
self.w.root(kind, addr);
self
}
pub fn root_in_thread(&mut self, kind: HeapRootKind, addr: u64, serial: u32) -> &mut Self {
self.w.root_in_thread(kind, addr, serial);
self
}
pub fn bytes(&self) -> Vec<u8> {
self.w.finish()
}
pub fn write_to(&self, path: &Path) {
std::fs::write(path, self.bytes()).unwrap_or_else(|e| panic!("MiniHeap: writing {}: {e}", path.display()));
}
}
/// The heap most analysis tests share:
///
/// - `Holder { a: Object, b: Object }`, `Node { next: Object, v: long }`,
/// `java.lang.ref.Reference { referent: Object, queue: Object }`,
/// `java.lang.ref.WeakReference extends Reference`, `[Ljava.lang.Object;`
/// - H `0x1000` Holder { a: N1, b: N2 }; N1 `0x1100` Node { next: N3 }; N2 `0x1200` Node { next: N3 };
/// N3 `0x1300` Node { next: L }; L `0x1400` long[2]; R `0x1500` Object[] { N1, null, null };
/// W `0x1600` WeakReference { referent: N4 }; N4 `0x1700` Node {}
/// - roots: JNI global H, Java frame N3 (thread serial 1), JNI global W
///
/// R and N4 are unreachable (N4 only through the weak referent).
pub fn sample() -> MiniHeap {
let mut heap = MiniHeap::new();
heap.class("Holder", Some("java.lang.Object"), &[("a", BasicType::Object), ("b", BasicType::Object)])
.class("Node", Some("java.lang.Object"), &[("next", BasicType::Object), ("v", BasicType::Long)])
.class(
"java.lang.ref.Reference",
Some("java.lang.Object"),
&[("referent", BasicType::Object), ("queue", BasicType::Object)],
)
.class("java.lang.ref.WeakReference", Some("java.lang.ref.Reference"), &[])
.class("[Ljava.lang.Object;", Some("java.lang.Object"), &[])
.object(0x1000, "Holder", &[("a", 0x1100), ("b", 0x1200)])
.object(0x1100, "Node", &[("next", 0x1300)])
.object(0x1200, "Node", &[("next", 0x1300)])
.object(0x1300, "Node", &[("next", 0x1400)])
.prim(0x1400, BasicType::Long, 2)
.array_of(0x1500, "[Ljava.lang.Object;", &[0x1100, 0, 0])
.object(0x1600, "java.lang.ref.WeakReference", &[("referent", 0x1700)])
.object(0x1700, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1000)
.root(HeapRootKind::JavaFrame, 0x1300)
.root(HeapRootKind::JniGlobal, 0x1600);
heap
}
/// [`sample`] plus S `0x1800` String `"v42"` (Latin-1) held as `a` of a second Holder `0x1900`,
/// rooted as a JNI global. That Holder retains 72 bytes, as much as H.
pub fn sample_with_string() -> MiniHeap {
let mut heap = sample();
heap.string(0x1800, "v42", true).object(0x1900, "Holder", &[("a", 0x1800)]).root(HeapRootKind::JniGlobal, 0x1900);
heap
}
/// [`sample`] plus two Nodes `0x1a00` and `0x1b00`, each rooted as a JNI global.
pub fn sample_with_two_more_nodes() -> MiniHeap {
let mut heap = sample();
heap.object(0x1a00, "Node", &[])
.object(0x1b00, "Node", &[])
.root(HeapRootKind::JniGlobal, 0x1a00)
.root(HeapRootKind::JniGlobal, 0x1b00);
heap
}
/// Seeded xorshift64 numbers for reproducible random test data (the workspace has no `rand`).
pub struct XorShift(u64);
impl XorShift {
pub fn new(seed: u64) -> Self {
// Zero is xorshift's one fixed point.
Self(seed.max(1))
}
pub fn next_u64(&mut self) -> u64 {
let mut x = self.0;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
self.0 = x;
x
}
/// A number in `0..bound`; `bound` must not be 0.
pub fn below(&mut self, bound: u32) -> u32 {
(self.next_u64() % u64::from(bound)) as u32
}
}
+1
View File
@@ -27,6 +27,7 @@ ide-search.workspace = true
ide-lsp.workspace = true
ide-vcs.workspace = true
ide-dap.workspace = true
ide-heap.workspace = true
ide-jvm-attach.workspace = true
ide-build.workspace = true
ide-analysis.workspace = true
@@ -22,7 +22,7 @@
mod templates;
use ide_extension::{Manifest, validate_plugin_id};
use ide_host::plugins::{PluginHost, plugin_icon, read_market_url, resolve_market_href};
use ide_host::plugins::{PluginHost, market_href_reachable_from, plugin_icon, read_market_url, resolve_market_href};
use ide_plugin::{Limits, PluginEngine};
use std::path::{Path, PathBuf};
use std::process::{Command, ExitCode};
@@ -591,7 +591,7 @@ fn cmd_index(args: &[String]) -> Result<(), String> {
let base = a.value("download-base").map(|b| b.trim_end_matches('/'));
// Revocations and other publishers of an existing index are kept.
let mut index = match std::fs::read_to_string(&out) {
Ok(text) => current_format(Index::parse(&text)?, &out.to_string_lossy())?,
Ok(text) => parse_index(&text, &out.to_string_lossy())?,
Err(_) => Index { format: package::INDEX_FORMAT, ..Index::default() },
};
let publisher = match (a.value("publisher"), a.value("key")) {
@@ -653,6 +653,11 @@ fn cmd_index(args: &[String]) -> Result<(), String> {
engine: package::engine_requirement(&m.engine, built_for.as_ref()),
categories: m.categories(),
icon: plugin_icon(&ide_extension::Package::new(m.clone(), scratch.clone()))?,
homepage: m.homepage.clone().filter(|h| !h.trim().is_empty()),
notes: m.change_notes.clone().filter(|n| !n.trim().is_empty()),
downloads: None,
featured: false,
suggest: Vec::new(),
})
})();
let entry = entry.map_err(|e| format!("{}: {e}", path.display()))?;
@@ -717,14 +722,22 @@ fn register_key(
Ok(publishers)
}
/// `index`, unless a newer `openidea-plugin` wrote it (rewriting it would drop what this one can't read).
fn current_format(index: Index, file: &str) -> Result<Index, String> {
/// The index in `text` (read from `file`), unless a newer `openidea-plugin` wrote it (rewriting it
/// would drop what this one can't read). Says which entries it does not keep: no IDE offers them.
fn parse_index(text: &str, file: &str) -> Result<Index, String> {
let (index, dropped) = Index::parse_dropping(text)?;
if index.is_newer() {
return Err(format!(
"{file}: index format {} is newer than this tool writes; update openidea-plugin",
index.format
));
}
for e in &dropped {
eprintln!(
"warning: {file}: dropped {:?} {:?}: its id or version is not one a manifest may carry",
e.id, e.version
);
}
Ok(index)
}
@@ -735,14 +748,15 @@ fn cmd_merge(args: &[String]) -> Result<(), String> {
};
let keep = a.value("keep").ok_or("--keep is required")?;
let out = a.value("out").unwrap_or(base);
let mut index =
current_format(Index::parse(&std::fs::read_to_string(base).map_err(|e| format!("{base}: {e}"))?)?, base)?;
let mut index = parse_index(&std::fs::read_to_string(base).map_err(|e| format!("{base}: {e}"))?, base)?;
let bytes = read_market_url(other, 16 << 20)?;
let theirs = Index::parse(&String::from_utf8(bytes).map_err(|_| format!("{other}: not UTF-8"))?)?;
let mut theirs = current_format(theirs, other)?;
let mut theirs = parse_index(&String::from_utf8(bytes).map_err(|_| format!("{other}: not UTF-8"))?, other)?;
// Their downloads are relative to their index, not to ours.
for e in &mut theirs.plugins {
e.download = resolve_market_href(other, &e.download);
if !market_href_reachable_from(other, &e.download) {
return Err(format!("{other}: the download of {} {} is not on the network", e.id, e.version));
}
}
let count = theirs.plugins.len();
index.include(theirs, &[keep]).map_err(|e| format!("{other}: {e}"))?;
@@ -942,6 +956,47 @@ mod tests {
assert!(err.contains("unknown publisher"), "{err}");
}
/// An index read over the network is not merged when it names a download on this machine:
/// the merged index is a local one, which may.
#[test]
fn an_index_read_over_the_network_is_not_merged_with_local_downloads() {
use std::io::{BufRead, BufReader, Write};
let tmp = tempfile::tempdir().unwrap();
let base = tmp.path().join("index.json");
write_index(&Index { format: package::INDEX_FORMAT, ..Index::default() }, &base).unwrap();
let local = tmp.path().join("alice.dawn-1.0.0.oiplugin").to_string_lossy().into_owned();
let theirs = serde_json::json!({
"format": package::INDEX_FORMAT,
"publishers": [{ "id": "alice", "name": "Alice", "keys": [] }],
"plugins": [{
"id": "alice.dawn", "name": "Dawn", "version": "1.0.0", "publisher": "alice",
"download": local, "digest": "d", "engine": "^1.0",
}],
})
.to_string();
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let other = format!("http://127.0.0.1:{}/index.json", listener.local_addr().unwrap().port());
std::thread::spawn(move || {
for mut stream in listener.incoming().flatten() {
let mut reader = BufReader::new(stream.try_clone().unwrap());
let mut line = String::new();
while reader.read_line(&mut line).unwrap_or(0) > 0 && line != "\r\n" {
line.clear();
}
let _ = write!(
stream,
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{theirs}",
theirs.len()
);
}
});
let err = cmd_merge(&strings(&[base.to_str().unwrap(), &other, "--keep", "openidea"])).unwrap_err();
assert!(err.contains("is not on the network"), "{err}");
let merged = Index::parse(&std::fs::read_to_string(&base).unwrap()).unwrap();
assert!(merged.plugins.is_empty(), "{:?}", merged.plugins);
}
#[test]
fn the_index_carries_plugin_icons() {
let tmp = tempfile::tempdir().unwrap();
+14 -1
View File
@@ -73,6 +73,11 @@ pub struct Host {
pub processes: ProcessManager,
/// Debug sessions (adapters run on this machine, next to the project).
pub debugger: Arc<ide_dap::DebugManager>,
/// Heap snapshots open in any window, captured from a debug session or opened from a file.
pub heap: Arc<ide_heap::HeapRegistry>,
/// The window that owns each open heap snapshot; its project's close closes them.
pub(crate) heap_windows: Mutex<HashMap<ide_heap::HeapSnapshotId, String>>,
pub(crate) heap_requests: ops::heap::HeapRequests,
/// SSH connections of Settings › SSH Configurations.
pub ssh: Arc<ide_ssh::SshManager>,
/// SFTP deployment, remote file editing and runs on run targets.
@@ -143,6 +148,9 @@ impl Host {
terminals: TerminalManager::new(),
processes: ProcessManager::new(),
debugger: ide_dap::DebugManager::new(languages.clone()),
heap: Arc::new(ide_heap::HeapRegistry::new(ide_heap::snapshots_root(&local_data_dir))),
heap_windows: Mutex::default(),
heap_requests: Default::default(),
languages,
syntax,
config_dir,
@@ -255,6 +263,8 @@ impl Host {
for request in self.project_requests.write().remove(self.owner_window(window)).unwrap_or_default() {
self.cancel_request(request);
}
// After the cancels: a capture or open still running then discards what it registers.
ops::heap::close_window_snapshots(self, window);
let project = self.projects.write().remove(window)?;
if self.windows_for_root(&project.root).is_empty()
&& let Some(plugins) = self.plugin_host()
@@ -405,9 +415,12 @@ impl Host {
}
}
/// Stops what this host started: notebook kernels, language servers, agents, runs and terminals.
/// Stops what this host started: heap snapshots, notebook kernels, language servers, agents,
/// runs and terminals.
/// For a process that exits without dropping the host (the remote server).
pub async fn shutdown(&self) {
// A capture's dump may hold secrets from the debuggee's memory: delete it before exiting.
ops::heap::shutdown(self).await;
self.environment.shutdown();
self.processes.kill_all();
self.terminals.kill_all();
+5
View File
@@ -51,7 +51,12 @@ impl Call {
pub async fn ask(&self, request: ide_core::event::McpUiRequest) -> IdeResult<()> {
self.host.mcp_requests.ask(self, request).await
}
/// Asks the user before a tool changes the project or starts a process, unless brave mode
/// is on — read from the IDE's own settings only, which a project's settings can't set.
pub async fn confirm(&self, subject: &str, detail: Option<String>) -> IdeResult<()> {
if self.host.global_settings.effective().mcp_server.run_without_confirmation {
return self.project.require_trust();
}
self.ask(ide_core::event::McpUiRequest::Confirm {
client: self.client.clone(),
tool: self.tool.clone(),
+311 -28
View File
@@ -45,26 +45,12 @@ fn innermost_repo<T>(repos: Vec<(PathBuf, T)>, path: &Path) -> Option<(PathBuf,
// Read tools must never invoke repository-configured executables, including in Restricted Mode.
fn read_command(git: &str, root: &Path) -> std::process::Command {
let mut command = std::process::Command::new(git);
ide_vcs::restrict(&mut command);
command
.current_dir(root)
.args([
"-c",
"core.fsmonitor=false",
"-c",
"core.hooksPath=",
"-c",
"diff.external=",
"-c",
"core.attributesFile=",
"-c",
"submodule.recurse=false",
"-c",
"color.ui=false",
])
.env("GIT_NO_LAZY_FETCH", "1")
.args(["-c", "color.ui=false"])
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_OPTIONAL_LOCKS", "0")
.env_remove("GIT_EXTERNAL_DIFF")
.env_remove("GIT_DIR")
.env_remove("GIT_WORK_TREE")
.env_remove("GIT_INDEX_FILE");
@@ -72,6 +58,32 @@ fn read_command(git: &str, root: &Path) -> std::process::Command {
command
}
/// `-c` arguments that empty the clean and process filters the repository's own config names:
/// git runs them to compare a changed file with the index. Filters of the user's config stay
/// (Git LFS), as they are not the repository's choice.
fn filter_overrides(git: &str, root: &Path) -> IdeResult<Vec<String>> {
let mut cmd = read_command(git, root);
cmd.args(["config", "--null", "--list", "--show-scope"]);
let out = cmd.output().map_err(|e| IdeError::unavailable(format!("cannot run git: {e}")))?;
if !out.status.success() {
return Err(IdeError::internal(String::from_utf8_lossy(&out.stderr).trim().to_owned()));
}
repository_filter_overrides(&out.stdout)
}
/// [`filter_overrides`] out of `listing`, the output of `git config --null --list --show-scope`.
fn repository_filter_overrides(listing: &[u8]) -> IdeResult<Vec<String>> {
let mut fields = listing.split(|b| *b == 0);
let mut own = Vec::new();
while let (Some(scope), Some(record)) = (fields.next(), fields.next()) {
if matches!(scope, b"local" | b"worktree") {
own.extend_from_slice(record);
own.push(0);
}
}
ide_vcs::filter_overrides_from(&own)
}
/// The branch HEAD is on (unborn too); `None` when HEAD is detached or git fails.
fn read_branch(git: &str, root: &Path) -> Option<String> {
let mut cmd = read_command(git, root);
@@ -167,7 +179,10 @@ fn git_executable(call: &Call) -> String {
/// `git status` of the repository at `root`, paths relative to it.
fn read_status(git: &str, root: &Path, untracked: bool, ignored: bool) -> IdeResult<Vec<StatusEntry>> {
let mut cmd = read_command(git, root);
cmd.args(filter_overrides(git, root)?);
cmd.args(["-c", "core.quotepath=false", "status", "--porcelain", "-z", "--find-renames"]);
// git asks a submodule with changes for its own status, which runs under the submodule's config.
cmd.arg("--ignore-submodules=dirty");
// git refuses `--ignored=matching` with `--untracked-files=no`: untracked files are dropped below.
cmd.arg(if untracked || ignored { "--untracked-files=all" } else { "--untracked-files=no" });
if ignored {
@@ -183,6 +198,39 @@ fn read_status(git: &str, root: &Path, untracked: bool, ignored: bool) -> IdeRes
Ok(entries)
}
/// `git diff` of the repository at `root` (of `file` alone, when given): the index against HEAD
/// when `staged`, else disk against the index.
fn read_diff(
git: &str,
root: &Path,
file: Option<&Path>,
staged: bool,
cancel: &ide_core::CancellationToken,
) -> IdeResult<String> {
let mut cmd = read_command(git, root);
cmd.args(filter_overrides(git, root)?);
cmd.args(["--no-pager", "diff", "--no-ext-diff", "--no-textconv"]);
// git asks a submodule with changes for its own status, and for its own diff when the
// repository's `diff.submodule` says so: both run under the submodule's config.
cmd.args(["--ignore-submodules=dirty", "--submodule=short"]);
if staged {
cmd.arg("--cached");
}
cmd.arg("--");
if let Some(file) = file {
cmd.arg(file.strip_prefix(root).map_err(|e| IdeError::invalid(e.to_string()))?);
}
ide_core::process::prepare(&mut cmd);
if cancel.is_cancelled() {
return Err(IdeError::cancelled());
}
let out = cmd.output()?;
if !out.status.success() {
return Err(IdeError::unavailable(String::from_utf8_lossy(&out.stderr).trim()));
}
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}
/// A git repository in the project, for paths outside the repositories IDEA knows: the
/// closest folder with `.git` from `start` up to the project root.
fn find_git_root(root: &Path, start: &Path) -> Option<PathBuf> {
@@ -307,18 +355,7 @@ fn git_diff() -> Tool {
let staged = call.flag("staged", false)?;
let git = git_executable(&call);
let cancel = call.cancel.clone();
let text = call.host.blocking(move || {
let mut cmd = read_command(&git, &root);
cmd.args(["--no-pager", "diff", "--no-ext-diff", "--no-textconv"]);
if staged { cmd.arg("--cached"); }
cmd.arg("--");
if let Some(file) = file { cmd.arg(file.strip_prefix(&root).map_err(|e| IdeError::invalid(e.to_string()))?); }
ide_core::process::prepare(&mut cmd);
if cancel.is_cancelled() { return Err(IdeError::cancelled()); }
let out = cmd.output()?;
if !out.status.success() { return Err(IdeError::unavailable(String::from_utf8_lossy(&out.stderr).trim())); }
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
}).await?;
let text = call.host.blocking(move || read_diff(&git, &root, file.as_deref(), staged, &cancel)).await?;
Ok(Output::text(text))
})
.param(Param::optional("repositoryPathRelativeToProject", ParamType::String, "Select the containing repository"))
@@ -342,6 +379,252 @@ mod tests {
assert!(e[3].conflicted() && !e[3].staged() && !e[3].unstaged());
}
/// Security audit lead 2 (variant): reading status or a diff must not run the repository's filters.
#[test]
fn status_and_diff_do_not_run_repository_filters() {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap();
let git = |args: &[&str]| {
let out = std::process::Command::new("git").current_dir(&root).args(args).output().unwrap();
assert!(out.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&out.stderr));
};
git(&["init", "-q"]);
git(&["config", "user.email", "[email protected]"]);
git(&["config", "user.name", "t"]);
git(&["config", "commit.gpgsign", "false"]);
std::fs::write(root.join(".gitattributes"), "*.txt filter=marker\n").unwrap();
std::fs::write(root.join("a.txt"), "one\n").unwrap();
git(&["add", "."]);
git(&["commit", "-qm", "init"]);
git(&["config", "filter.marker.clean", "echo x > executed-marker; cat"]);
git(&["config", "filter.marker.process", "echo x > executed-marker; cat"]);
git(&["config", "filter.marker.required", "true"]);
let marker = root.join("executed-marker");
let mut ran = Vec::new();
// The size it has in the index, so git has to read the file to tell whether it changed.
std::fs::write(root.join("a.txt"), "two\n").unwrap();
let status = read_status("git", &root, false, false);
if marker.exists() {
ran.push("status");
}
let _ = std::fs::remove_file(&marker);
std::fs::write(root.join("a.txt"), "one\ntwo\nthree\n").unwrap();
let diff = read_diff("git", &root, None, false, &ide_core::CancellationToken::new());
if marker.exists() {
ran.push("diff");
}
// Control: the fixture's filter really runs when git is used directly.
let _ = std::fs::remove_file(&marker);
std::fs::write(root.join("a.txt"), "six\n").unwrap();
let _ = std::process::Command::new("git").current_dir(&root).args(["status", "--porcelain"]).output();
assert!(marker.exists(), "control: plain `git status` did not run the filter, fixture is not live");
assert!(ran.is_empty(), "a read tool ran the repository's filter via: {ran:?}");
assert_eq!(status.unwrap().len(), 1);
let diff = diff.unwrap();
assert!(diff.contains("+three"), "{diff}");
}
#[test]
fn only_filters_of_the_repository_s_own_config_are_emptied() {
let listing = b"system\0filter.lfs.clean\ngit-lfs clean -- %f\0\
global\0filter.lfs.process\ngit-lfs filter-process\0\
global\0filter.lfs.required\ntrue\0\
local\0core.bare\nfalse\0\
local\0filter.a.clean\nx\0\
worktree\0filter.b.process\ny\0\
local\0filter.a.required\ntrue\0\
command\0filter.c.clean\nz\0";
assert_eq!(
repository_filter_overrides(listing).unwrap(),
["-c", "filter.a.clean=", "-c", "filter.b.process=", "-c", "filter.a.required=false"]
);
}
fn git(dir: &Path, args: &[&str]) {
let out = std::process::Command::new("git").current_dir(dir).args(args).output().unwrap();
assert!(out.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&out.stderr));
}
/// A repository at `dir` with `a.txt` committed.
fn repository(dir: &Path, attributes: &[u8]) {
std::fs::create_dir_all(dir).unwrap();
git(dir, &["init", "-q"]);
git(dir, &["config", "user.email", "[email protected]"]);
git(dir, &["config", "user.name", "t"]);
git(dir, &["config", "commit.gpgsign", "false"]);
std::fs::write(dir.join(".gitattributes"), attributes).unwrap();
std::fs::write(dir.join("a.txt"), "one\n").unwrap();
git(dir, &["add", "."]);
git(dir, &["commit", "-qm", "init"]);
}
/// A shell command that leaves `marker` behind and passes its input on.
fn leaves(marker: &Path) -> String {
format!("echo x > '{}'; cat", marker.to_string_lossy().replace('\\', "/"))
}
/// Whether a program of the fixture left `marker` behind; removes it.
fn left(marker: &Path) -> bool {
let was = marker.exists();
let _ = std::fs::remove_file(marker);
was
}
/// Reads the status and then the diff of `root` after rewriting `file`, then has plain git
/// read the status as a control. Each write keeps the size the file has in the index, so git
/// has to read it to tell whether it changed.
fn read_both(root: &Path, file: &Path, marker: &Path) -> (IdeResult<Vec<StatusEntry>>, IdeResult<String>) {
let mut ran = Vec::new();
std::fs::write(file, "two\n").unwrap();
let status = read_status("git", root, false, false);
if left(marker) {
ran.push("status");
}
std::fs::write(file, "six\n").unwrap();
let diff = read_diff("git", root, None, false, &ide_core::CancellationToken::new());
if left(marker) {
ran.push("diff");
}
std::fs::write(file, "ten\n").unwrap();
let _ = std::process::Command::new("git").current_dir(root).args(["status", "--porcelain"]).output();
assert!(left(marker), "control: plain `git status` ran no program of the fixture, it is not live");
assert!(ran.is_empty(), "a read tool ran a program of the repository via: {ran:?}");
(status, diff)
}
#[test]
fn status_and_diff_do_not_run_filters_of_any_repository_config() {
type Configure = fn(&Path, &str);
let cases: [(&str, Configure); 4] = [
("clean", |root, command| git(root, &["config", "filter.marker.clean", command])),
("process", |root, command| git(root, &["config", "filter.marker.process", command])),
("included file", |root, command| {
let included = format!("[filter \"marker\"]\n\tclean = \"{command}\"\n");
std::fs::write(root.join("extra.gitconfig"), included).unwrap();
git(root, &["config", "include.path", "../extra.gitconfig"]);
}),
("worktree config", |root, command| {
git(root, &["config", "extensions.worktreeConfig", "true"]);
git(root, &["config", "--worktree", "filter.marker.clean", command]);
}),
];
for (case, configure) in cases {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap().join("repo");
let marker = root.with_file_name("marker");
repository(&root, b"*.txt filter=marker\n");
configure(&root, &leaves(&marker));
let (status, diff) = read_both(&root, &root.join("a.txt"), &marker);
assert_eq!(status.unwrap().len(), 1, "{case}");
let diff = diff.unwrap();
assert!(diff.contains("+six"), "{case}: {diff}");
}
}
/// git splits `-c name=value` at the first `=` and takes no name that is not UTF-8 on every
/// platform, so such a filter cannot be emptied: the tools read nothing instead.
#[test]
fn status_and_diff_refuse_a_repository_filter_git_cannot_be_told_to_skip() {
for name in [&b"x=y"[..], &[0xE9]] {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap().join("repo");
let marker = root.with_file_name("marker");
repository(&root, &[b"*.txt filter=", name, b"\n"].concat());
let section =
[b"[filter \"", name, format!("\"]\n\tclean = \"{}\"\n", leaves(&marker)).as_bytes()].concat();
let config = root.join(".git").join("config");
std::fs::write(&config, [std::fs::read(&config).unwrap(), section].concat()).unwrap();
let (status, diff) = read_both(&root, &root.join("a.txt"), &marker);
assert!(status.is_err() && diff.is_err(), "{name:?}");
}
}
/// git asks a submodule for its own status to tell whether it is dirty, under the
/// submodule's config.
#[test]
fn status_and_diff_do_not_run_filters_of_a_submodule() {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap().join("repo");
let (sub, marker) = (root.join("sub"), root.with_file_name("marker"));
repository(&root, b"");
repository(&sub, b"*.txt filter=marker\n");
git(&root, &["add", "sub"]);
git(&root, &["commit", "-qm", "submodule"]);
git(&sub, &["config", "filter.marker.clean", &leaves(&marker)]);
git(&root, &["config", "diff.ignoreSubmodules", "none"]);
git(&root, &["config", "submodule.sub.ignore", "none"]);
let (status, diff) = read_both(&root, &sub.join("a.txt"), &marker);
status.unwrap();
diff.unwrap();
}
/// `git diff` writes the index back when it finds a file whose stat changed and whose content
/// did not, and git runs the hooks a config names for `post-index-change` on that write.
#[test]
fn status_and_diff_do_not_run_hooks_of_the_repository_s_config() {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap().join("repo");
let marker = root.with_file_name("marker");
repository(&root, b"");
git(&root, &["config", "hook.marker.event", "post-index-change"]);
let command = format!("echo x > '{}'", marker.to_string_lossy().replace('\\', "/"));
git(&root, &["config", "hook.marker.command", &command]);
let listed = std::process::Command::new("git")
.current_dir(&root)
.args(["hook", "list", "post-index-change"])
.output()
.unwrap();
if !String::from_utf8_lossy(&listed.stdout).contains("marker") {
return; // This git has no hooks named by a config.
}
// The content it has in the index, with a time of its own that is not the index's.
let touch = |seconds: u64| {
let file = std::fs::File::options().write(true).open(root.join("a.txt")).unwrap();
file.set_modified(std::time::SystemTime::now() - std::time::Duration::from_secs(seconds)).unwrap();
};
let mut ran = Vec::new();
touch(60);
let status = read_status("git", &root, false, false);
if left(&marker) {
ran.push("status");
}
touch(120);
let diff = read_diff("git", &root, None, false, &ide_core::CancellationToken::new());
if left(&marker) {
ran.push("diff");
}
touch(180);
let _ = std::process::Command::new("git").current_dir(&root).args(["--no-pager", "diff"]).output();
assert!(left(&marker), "control: plain `git diff` did not run the hook, fixture is not live");
assert!(ran.is_empty(), "a read tool ran a hook of the repository's config via: {ran:?}");
assert!(status.unwrap().is_empty());
assert_eq!(diff.unwrap(), "");
}
/// `diff.submodule=diff` has git run `git diff` in the submodule, under the submodule's config.
#[test]
fn diff_does_not_run_a_diff_driver_of_a_submodule() {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap().join("repo");
let (sub, marker) = (root.join("sub"), root.with_file_name("marker"));
repository(&root, b"");
repository(&sub, b"*.txt diff=marker\n");
git(&root, &["add", "sub"]);
git(&root, &["commit", "-qm", "submodule"]);
std::fs::write(sub.join("a.txt"), "one\ntwo\n").unwrap();
git(&sub, &["commit", "-qam", "second"]);
git(&sub, &["config", "diff.marker.command", &leaves(&marker)]);
git(&root, &["config", "diff.submodule", "diff"]);
let diff = read_diff("git", &root, None, false, &ide_core::CancellationToken::new());
let ran = left(&marker);
let _ = std::process::Command::new("git").current_dir(&root).args(["--no-pager", "diff"]).output();
assert!(left(&marker), "control: plain `git diff` did not run the diff driver, fixture is not live");
assert!(!ran, "git_diff ran the submodule's diff driver");
let diff = diff.unwrap();
assert!(diff.contains("Subproject commit"), "{diff}");
assert_eq!(read_status("git", &root, false, false).unwrap().len(), 1);
}
#[cfg(unix)]
#[test]
fn paths_through_a_symlinked_root_compare_canonically() {
+11 -8
View File
@@ -181,15 +181,18 @@ pub fn entries(host: &Host, window: &str, task: String, subagent: Option<String>
host.project(window)?.agents.entries(&task, subagent)
}
/// Sessions on this machine that no task holds yet, newest first (#312).
pub fn external_sessions(host: &Host, window: &str) -> IdeResult<Vec<ExternalSession>> {
Ok(host.project(window)?.agents.external_sessions())
/// Sessions of the project on this machine that no task holds yet, newest first. Reads
/// transcripts from disk on the blocking pool.
pub async fn external_sessions(host: &Host, window: &str) -> IdeResult<Vec<ExternalSession>> {
let agents = host.project(window)?.agents.clone();
tokio::task::spawn_blocking(move || agents.external_sessions()).await.map_err(join_err)
}
/// Imports an external session as a task: its first turn loads and replays it (#312).
pub fn import_session(host: &Host, window: &str, provider: String, session_id: String) -> IdeResult<AgentTask> {
/// Imports an external session as a task: its first turn loads and replays it.
pub async fn import_session(host: &Host, window: &str, provider: String, session_id: String) -> IdeResult<AgentTask> {
host.require_trust(window)?;
host.project(window)?.agents.import_session(&provider, &session_id)
let agents = host.project(window)?.agents.clone();
tokio::task::spawn_blocking(move || agents.import_session(&provider, &session_id)).await.map_err(join_err)?
}
pub async fn create(host: &Host, window: &str, mut request: NewAgentTask) -> IdeResult<AgentTask> {
@@ -409,9 +412,9 @@ pub(crate) fn register(d: &mut Dispatcher) {
d.add("agent_commands", |c| async move { commands(&c.host, &c.window, c.arg("provider")?, c.arg("task")?).await });
d.add("agent_entries", |c| async move { entries(&c.host, &c.window, c.arg("task")?, c.arg("subagent")?) });
d.add("agent_create", |c| async move { create(&c.host, &c.window, c.arg("request")?).await });
d.add("agent_external_sessions", |c| async move { external_sessions(&c.host, &c.window) });
d.add("agent_external_sessions", |c| async move { external_sessions(&c.host, &c.window).await });
d.add("agent_import_session", |c| async move {
import_session(&c.host, &c.window, c.arg("provider")?, c.arg("sessionId")?)
import_session(&c.host, &c.window, c.arg("provider")?, c.arg("sessionId")?).await
});
d.add("agent_prompt", |c| async move {
prompt(&c.host, &c.window, c.arg("task")?, c.arg("prompt")?, c.arg("now")?).await
+5 -3
View File
@@ -17,9 +17,11 @@ pub async fn fs_read_range(_host: &Host, _window: &str, path: PathBuf, offset: u
blocking(move || ide_vfs::read_range(&path, offset, len)).await
}
/// An image file as a `data:` URL.
pub async fn fs_read_image(_host: &Host, _window: &str, path: PathBuf) -> IdeResult<String> {
blocking(move || ide_vfs::read_data_url(&path)).await
/// An image file as a `data:` URL. A network path is read only from a share an open project is
/// on: the path may come from Markdown, and opening a share signs in to its server.
pub async fn fs_read_image(host: &Host, _window: &str, path: PathBuf) -> IdeResult<String> {
let opened: Vec<PathBuf> = host.projects().iter().map(|project| project.root.clone()).collect();
blocking(move || ide_vfs::read_data_url(&path, &opened)).await
}
/// Commands of this module the remote server answers.
+10 -3
View File
@@ -5,7 +5,7 @@ use crate::rpc::Dispatcher;
use ide_core::{DocId, IdeError, IdeResult, OutlineItem, Range};
use ide_language::{FoldRange, SyntaxHighlights};
use ide_text::LineEnding;
use ide_workspace::{DocumentInfo, HistoryEntry};
use ide_workspace::{DocumentEncoding, DocumentInfo, HistoryEntry};
use std::path::PathBuf;
/// Runs blocking work (disk I/O, parsing, large copies) on the blocking pool,
@@ -29,18 +29,24 @@ pub async fn doc_ui_mode(host: &Host, window: &str, path: PathBuf) -> IdeResult<
blocking(move || Ok(project.documents.ui_mode_of(&path))).await
}
/// "Reload with Encoding" (`reload`) or "Convert to Encoding"; returns the new encoding label.
/// "Reload with Encoding" (`reload`) or "Convert to Encoding"; returns the new encoding and BOM state.
pub async fn doc_set_encoding(
host: &Host,
window: &str,
doc: DocId,
encoding: String,
reload: bool,
) -> IdeResult<String> {
) -> IdeResult<DocumentEncoding> {
let project = host.project(window)?;
host.blocking(move || project.documents.set_encoding(doc, &encoding, reload)).await
}
/// Adds or removes the BOM on a UTF-8 document; the change is written on save.
pub async fn doc_set_bom(host: &Host, window: &str, doc: DocId, has_bom: bool) -> IdeResult<DocumentEncoding> {
let project = host.project(window)?;
host.blocking(move || project.documents.set_bom(doc, has_bom)).await
}
pub fn doc_set_line_ending(host: &Host, window: &str, doc: DocId, line_ending: LineEnding) -> IdeResult<()> {
host.project(window)?.documents.set_line_ending(doc, line_ending)
}
@@ -175,6 +181,7 @@ pub(crate) fn register(d: &mut Dispatcher) {
d.add("doc_set_encoding", |c| async move {
doc_set_encoding(&c.host, &c.window, c.arg("doc")?, c.arg("encoding")?, c.arg("reload")?).await
});
d.add("doc_set_bom", |c| async move { doc_set_bom(&c.host, &c.window, c.arg("doc")?, c.arg("hasBom")?).await });
d.add("doc_set_line_ending", |c| async move {
blocking(move || doc_set_line_ending(&c.host, &c.window, c.arg("doc")?, c.arg("lineEnding")?)).await
});
+520
View File
@@ -0,0 +1,520 @@
//! Heap snapshots for the debugger's JVM Memory view: captured from a debug session or opened from
//! an `.hprof`, analyzed into an index, then queried, compared, saved and closed. A snapshot
//! belongs to the window that made it and closes with that window's project.
use crate::Host;
use crate::ops::Channel;
use crate::rpc::Dispatcher;
use ide_core::{CancellationToken, DebugSessionId, IdeError, IdeResult, IpcPath, RequestId};
use ide_dap::{JvmHeapOutcome, JvmHeapUnavailable};
use ide_heap::{AnalyzeOptions, HeapError, HeapOrigin, OpenSnapshot};
pub use ide_heap::{
HeapClassDiff, HeapClassRow, HeapObjectPage, HeapObjectRef, HeapPhase, HeapRootPath, HeapSnapshotId,
HeapSnapshotInfo,
};
use serde::{Deserialize, Serialize};
use specta::Type;
use std::cell::Cell;
use std::collections::HashMap;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// Shortest gap between two progress messages of one phase.
const PROGRESS_INTERVAL: Duration = Duration::from_millis(100);
/// Longest each of [`shutdown`]'s two waits lasts; `openidea-server` gives the whole host
/// shutdown 10 s.
const SHUTDOWN_WAIT: Duration = Duration::from_secs(3);
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "type", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapProgress {
Dumping,
Analyzing { phase: HeapPhase, done: u64, total: u64 },
}
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapCaptureResult {
Ready { snapshot: HeapSnapshotInfo },
Unavailable { reason: JvmHeapUnavailable },
Cancelled,
}
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize, Type)]
#[serde(tag = "status", rename_all = "camelCase", rename_all_fields = "camelCase")]
pub enum HeapOpenResult {
Ready {
snapshot: HeapSnapshotInfo,
},
/// Bytes, for the index next to the dump.
NotEnoughDiskSpace {
needed: u64,
available: u64,
},
Cancelled,
}
/// Dumps the session's JVM into a new snapshot directory and analyzes the dump there; cancel with
/// `request`. Anything but `Ready` leaves no directory behind.
pub async fn debug_heap_capture(
host: &Host,
window: &str,
session: DebugSessionId,
request: RequestId,
on_progress: Channel<HeapProgress>,
) -> IdeResult<HeapCaptureResult> {
host.require_trust(window)?;
let cancel = host.begin_project_request(window, request);
let running = Running::begin(host, request, &cancel);
let result = capture(host, window, session, &cancel, &on_progress).await;
drop(running);
host.end_request(request);
result
}
async fn capture(
host: &Host,
window: &str,
session: DebugSessionId,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<HeapCaptureResult> {
if cancel.is_cancelled() {
return Ok(HeapCaptureResult::Cancelled);
}
let dir = new_snapshot_dir(host).await?;
let _ = on_progress.send(HeapProgress::Dumping);
// `jvm_heap_dump` deletes the directory itself on every outcome but `Ready`.
let dump = match host.debugger.jvm_heap_dump(session, dir.clone(), cancel.clone()).await {
Ok(JvmHeapOutcome::Ready(dump)) => dump,
Ok(JvmHeapOutcome::Unavailable(reason)) => return Ok(HeapCaptureResult::Unavailable { reason }),
Ok(JvmHeapOutcome::Cancelled) => return Ok(HeapCaptureResult::Cancelled),
Err(e) => {
remove_dir(host, dir).await;
return Err(e);
}
};
let origin = HeapOrigin::Captured {
session,
pid: dump.pid,
java_version: dump.java_version,
taken_while_suspended: dump.taken_while_suspended,
dump_ms: dump.dump_ms,
};
let options = AnalyzeOptions {
histogram: Some(dump.histogram).filter(|h| !h.is_empty()),
vm_flags: Some(dump.vm_flags).filter(|f| !f.is_empty()),
java_major: Some(dump.java_major).filter(|&v| v > 0),
in_memory: false,
};
// The capture already checked the space for the dump and its index before dumping.
let analyzed = analyze(host, window, origin, dump.path, dir, options, false, cancel, on_progress).await?;
Ok(match analyzed {
Analyzed::Ready(snapshot) => HeapCaptureResult::Ready { snapshot },
Analyzed::NotEnoughDiskSpace { needed, available } => {
HeapCaptureResult::Unavailable { reason: JvmHeapUnavailable::NotEnoughDiskSpace { needed, available } }
}
Analyzed::Cancelled => HeapCaptureResult::Cancelled,
})
}
/// Analyzes the `.hprof` at `path` into a new snapshot directory; the file itself is never
/// changed or deleted. Cancel with `request`.
pub async fn debug_heap_open(
host: &Host,
window: &str,
path: IpcPath,
request: RequestId,
on_progress: Channel<HeapProgress>,
) -> IdeResult<HeapOpenResult> {
host.require_trust(window)?;
let cancel = host.begin_project_request(window, request);
let running = Running::begin(host, request, &cancel);
let result = open(host, window, path.into_path_buf(), &cancel, &on_progress).await;
drop(running);
host.end_request(request);
result
}
/// The captures and opens in progress, so [`shutdown`] can cancel them and wait for them to delete
/// their directories.
#[derive(Default)]
pub(crate) struct HeapRequests(parking_lot::Mutex<RequestsState>);
#[derive(Default)]
struct RequestsState {
/// Set by [`shutdown`]: what begins afterwards starts out cancelled.
closed: bool,
running: HashMap<RequestId, CancellationToken>,
}
/// One capture or open, from its start until it has cleaned up after itself.
struct Running<'a> {
host: &'a Host,
request: RequestId,
}
impl<'a> Running<'a> {
fn begin(host: &'a Host, request: RequestId, cancel: &CancellationToken) -> Self {
let mut state = host.heap_requests.0.lock();
if state.closed {
cancel.cancel();
} else {
state.running.insert(request, cancel.clone());
}
Self { host, request }
}
}
impl Drop for Running<'_> {
fn drop(&mut self) {
self.host.heap_requests.0.lock().running.remove(&self.request);
}
}
async fn open(
host: &Host,
window: &str,
path: PathBuf,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<HeapOpenResult> {
if cancel.is_cancelled() {
return Ok(HeapOpenResult::Cancelled);
}
let dir = new_snapshot_dir(host).await?;
let origin = HeapOrigin::File { path: path.clone().into() };
// The snapshot maps its dump for as long as it is open; a file that something else truncates or
// rewrites meanwhile would crash the app, so it works on a private copy (a clone on APFS / btrfs).
let copy = {
let (path, dir) = (path.clone(), dir.clone());
host.blocking(move || private_copy(&path, &dir)).await
};
let dump = match copy {
Ok(Ok(dump)) => dump,
Ok(Err((needed, available))) => {
remove_dir(host, dir).await;
return Ok(HeapOpenResult::NotEnoughDiskSpace { needed, available });
}
Err(e) => {
remove_dir(host, dir).await;
return Err(e);
}
};
let analyzed =
analyze(host, window, origin, dump, dir, AnalyzeOptions::default(), true, cancel, on_progress).await?;
Ok(match analyzed {
Analyzed::Ready(snapshot) => HeapOpenResult::Ready { snapshot },
Analyzed::NotEnoughDiskSpace { needed, available } => HeapOpenResult::NotEnoughDiskSpace { needed, available },
Analyzed::Cancelled => HeapOpenResult::Cancelled,
})
}
enum Analyzed {
Ready(HeapSnapshotInfo),
NotEnoughDiskSpace { needed: u64, available: u64 },
Cancelled,
}
/// Indexes `dump` into `dir` and registers the snapshot for `window`. Anything but `Ready` deletes
/// `dir`, a captured dump inside it included.
#[allow(clippy::too_many_arguments)]
async fn analyze(
host: &Host,
window: &str,
origin: HeapOrigin,
dump: PathBuf,
dir: PathBuf,
options: AnalyzeOptions,
check_disk: bool,
cancel: &CancellationToken,
on_progress: &Channel<HeapProgress>,
) -> IdeResult<Analyzed> {
let indexed = {
let (dump, dir, cancel, progress) = (dump.clone(), dir.clone(), cancel.clone(), on_progress.clone());
tokio::task::spawn_blocking(move || {
let free = || if check_disk { ide_heap::free_bytes(&dir) } else { None };
ide_heap::analyze(&dump, &dir, options, &reporter(&progress, free), &cancel)
})
.await
};
let failed = match indexed {
Ok(Ok(index)) => {
let info = HeapSnapshotInfo {
id: HeapSnapshotId(0),
label: origin.label(),
summary: index.summary().clone(),
origin,
created_ms: now_ms(),
};
let info = host.heap.insert(|_| OpenSnapshot::new(info, index, dir, dump));
host.heap_windows.lock().insert(info.id, host.owner_window(window).to_owned());
// A project close cancels the request before it closes the window's snapshots, so one
// registered after that close is caught here.
if cancel.is_cancelled() {
close(host, info.id).await?;
return Ok(Analyzed::Cancelled);
}
return Ok(Analyzed::Ready(info));
}
Ok(Err(HeapError::Cancelled)) => Ok(Analyzed::Cancelled),
Ok(Err(HeapError::NotEnoughDiskSpace { needed, available })) => {
Ok(Analyzed::NotEnoughDiskSpace { needed, available })
}
Ok(Err(e)) => Err(IdeError::invalid(format!("{e}"))),
Err(e) => Err(IdeError::internal(format!("heap analysis failed: {e}"))),
};
remove_dir(host, dir).await;
failed
}
/// The progress callback of an analysis: sends a phase's first report and then at most one every
/// [`PROGRESS_INTERVAL`]. Once the dump's objects are counted (the first `Linking` report), it
/// fails with `NotEnoughDiskSpace` when `free` reports less than their index needs.
fn reporter<'a>(
progress: &'a Channel<HeapProgress>,
free: impl Fn() -> Option<u64> + 'a,
) -> impl Fn(HeapPhase, u64, u64) -> Result<(), HeapError> + 'a {
let last: Cell<Option<(HeapPhase, Instant)>> = Cell::new(None);
move |phase, done, total| {
let now = Instant::now();
let previous = last.get();
if phase == HeapPhase::Linking
&& previous.is_none_or(|(p, _)| p != HeapPhase::Linking)
&& let Some(available) = free()
{
let needed = ide_heap::index_need(total);
if available < needed {
return Err(HeapError::NotEnoughDiskSpace { needed, available });
}
}
if previous.is_none_or(|(p, at)| p != phase || now.duration_since(at) >= PROGRESS_INTERVAL) {
last.set(Some((phase, now)));
let _ = progress.send(HeapProgress::Analyzing { phase, done, total });
}
Ok(())
}
}
/// The class table of a snapshot.
pub async fn debug_heap_classes(host: &Host, window: &str, snapshot: HeapSnapshotId) -> IdeResult<Vec<HeapClassRow>> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().class_rows())).await
}
/// A page of the dominator tree; see [`ide_heap::HeapIndex::dominated`] for what `parent` and
/// `class_index` select.
pub async fn debug_heap_dominators(
host: &Host,
window: &str,
snapshot: HeapSnapshotId,
parent: Option<HeapObjectRef>,
class_index: Option<u32>,
offset: u32,
limit: u32,
) -> IdeResult<HeapObjectPage> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().dominated(parent, class_index, offset, limit))).await
}
/// The shortest path from `object` to a GC root; `None` when it is unreachable.
pub async fn debug_heap_path(
host: &Host,
window: &str,
snapshot: HeapSnapshotId,
object: HeapObjectRef,
) -> IdeResult<Option<HeapRootPath>> {
let snapshot = open_snapshot(host, window, snapshot)?;
host.blocking(move || Ok(snapshot.index().path_to_root(object))).await
}
/// Class-by-class differences from `base` to `current`.
pub async fn debug_heap_compare(
host: &Host,
window: &str,
base: HeapSnapshotId,
current: HeapSnapshotId,
) -> IdeResult<Vec<HeapClassDiff>> {
let base = open_snapshot(host, window, base)?;
let current = open_snapshot(host, window, current)?;
host.blocking(move || Ok(ide_heap::compare(base.index(), current.index()))).await
}
/// Copies the snapshot's dump to `path`, replacing a file there (the UI has asked).
pub async fn debug_heap_save(host: &Host, window: &str, snapshot: HeapSnapshotId, path: IpcPath) -> IdeResult<()> {
let snapshot = open_snapshot(host, window, snapshot)?;
let target = path.into_path_buf();
host.blocking(move || copy_replacing(&snapshot.dump, &target)).await
}
/// Forgets the snapshot and deletes its directory once no query is using it.
pub async fn debug_heap_close(host: &Host, window: &str, snapshot: HeapSnapshotId) -> IdeResult<()> {
open_snapshot(host, window, snapshot)?;
close(host, snapshot).await
}
async fn close(host: &Host, id: HeapSnapshotId) -> IdeResult<()> {
let closed = take(host, id);
// Usually the last reference: dropping it deletes files.
host.blocking(move || {
drop(closed);
Ok(())
})
.await
}
/// Forgets the snapshot and returns it, so the caller chooses the thread that deletes its files.
fn take(host: &Host, id: HeapSnapshotId) -> Option<Arc<OpenSnapshot>> {
host.heap_windows.lock().remove(&id);
let snapshot = host.heap.get(id);
host.heap.close(id);
snapshot
}
/// Closes the snapshots `window` owns; their directories are deleted on the blocking pool.
pub(crate) fn close_window_snapshots(host: &Host, window: &str) {
let owner = host.owner_window(window);
let ids: Vec<HeapSnapshotId> =
host.heap_windows.lock().iter().filter(|(_, w)| *w == owner).map(|(id, _)| *id).collect();
let closed: Vec<Arc<OpenSnapshot>> = ids.into_iter().filter_map(|id| take(host, id)).collect();
if !closed.is_empty() {
host.runtime.spawn_blocking(move || drop(closed));
}
}
/// For a process that exits without dropping the host (the remote server): cancels the captures
/// and opens in progress and waits for them, closes every snapshot, then removes what is left of
/// this process's snapshot directories. Each wait gives up after [`SHUTDOWN_WAIT`] (a stuck
/// analysis or delete).
pub(crate) async fn shutdown(host: &Host) {
shutdown_within(host, SHUTDOWN_WAIT).await;
}
async fn shutdown_within(host: &Host, wait: Duration) {
let deadline = tokio::time::Instant::now() + wait;
{
let mut state = host.heap_requests.0.lock();
state.closed = true;
for token in state.running.values() {
token.cancel();
}
}
let ids: Vec<HeapSnapshotId> = host.heap.list().into_iter().map(|info| info.id).collect();
let closed: Vec<Arc<OpenSnapshot>> = ids.into_iter().filter_map(|id| take(host, id)).collect();
while !host.heap_requests.0.lock().running.is_empty() {
if tokio::time::Instant::now() >= deadline {
tracing::warn!("heap captures or opens still running at shutdown");
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
let root = host.heap.root().to_path_buf();
let me = std::process::id();
let cleanup = host.runtime.spawn_blocking(move || {
drop(closed);
// Directories of work that did not finish in time count as abandoned, like a dead host's.
ide_heap::remove_stale(&root, |pid| pid != me && ide_core::process::process_alive(pid));
});
if tokio::time::timeout(wait, cleanup).await.is_err() {
tracing::warn!("heap snapshot directories were not deleted within {wait:?}");
}
}
/// Every op that takes a snapshot id goes through here, so none skips the trust check.
fn open_snapshot(host: &Host, window: &str, id: HeapSnapshotId) -> IdeResult<Arc<OpenSnapshot>> {
host.require_trust(window)?;
let owned = host.heap_windows.lock().get(&id).is_some_and(|w| w == host.owner_window(window));
owned
.then(|| host.heap.get(id))
.flatten()
.ok_or_else(|| IdeError::not_found(format!("heap snapshot {} is not open", id.0)))
}
async fn new_snapshot_dir(host: &Host) -> IdeResult<PathBuf> {
let root = host.heap.root().to_path_buf();
host.blocking(move || Ok(ide_heap::create_snapshot_dir(&root)?)).await
}
async fn remove_dir(host: &Host, dir: PathBuf) {
let _ = host
.blocking(move || {
if let Err(e) = std::fs::remove_dir_all(&dir)
&& e.kind() != std::io::ErrorKind::NotFound
{
tracing::warn!("could not delete heap snapshot directory {}: {e}", dir.display());
}
Ok(())
})
.await;
}
/// Copies an opened dump into its snapshot directory; `Err((needed, available))` when the disk
/// can't hold the copy.
fn private_copy(source: &Path, dir: &Path) -> IdeResult<Result<PathBuf, (u64, u64)>> {
let needed = std::fs::metadata(source)?.len();
if let Some(available) = ide_heap::free_bytes(dir)
&& available < needed
{
return Ok(Err((needed, available)));
}
let target = dir.join("opened.hprof");
std::fs::copy(source, &target)?;
Ok(Ok(target))
}
/// Written beside `target` and renamed over it: `target` may be the dump of an open snapshot,
/// whose index maps that file and must keep reading the old bytes.
fn copy_replacing(source: &Path, target: &Path) -> IdeResult<()> {
let name =
target.file_name().ok_or_else(|| IdeError::invalid(format!("{} is not a file path", target.display())))?;
let partial = target.with_file_name(format!(".{}.{}.tmp", name.to_string_lossy(), uuid::Uuid::new_v4()));
let copied = std::fs::copy(source, &partial).and_then(|_| std::fs::rename(&partial, target));
if let Err(e) = copied {
let _ = std::fs::remove_file(&partial);
return Err(e.into());
}
Ok(())
}
fn now_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |d| u64::try_from(d.as_millis()).unwrap_or(u64::MAX))
}
/// Commands of this module the remote server answers.
pub(crate) fn register(d: &mut Dispatcher) {
d.add("debug_heap_capture", |c| async move {
debug_heap_capture(&c.host, &c.window, c.arg("session")?, c.arg("request")?, c.sender("onProgress")?).await
});
d.add("debug_heap_open", |c| async move {
debug_heap_open(&c.host, &c.window, c.arg("path")?, c.arg("request")?, c.sender("onProgress")?).await
});
d.add("debug_heap_classes", |c| async move { debug_heap_classes(&c.host, &c.window, c.arg("snapshot")?).await });
d.add("debug_heap_dominators", |c| async move {
debug_heap_dominators(
&c.host,
&c.window,
c.arg("snapshot")?,
c.arg("parent")?,
c.arg("classIndex")?,
c.arg("offset")?,
c.arg("limit")?,
)
.await
});
d.add("debug_heap_path", |c| async move {
debug_heap_path(&c.host, &c.window, c.arg("snapshot")?, c.arg("object")?).await
});
d.add("debug_heap_compare", |c| async move {
debug_heap_compare(&c.host, &c.window, c.arg("base")?, c.arg("current")?).await
});
d.add("debug_heap_save", |c| async move {
debug_heap_save(&c.host, &c.window, c.arg("snapshot")?, c.arg("path")?).await
});
d.add("debug_heap_close", |c| async move { debug_heap_close(&c.host, &c.window, c.arg("snapshot")?).await });
}
#[cfg(test)]
mod tests;
+479
View File
@@ -0,0 +1,479 @@
//! The heap commands through a real `Host` with a trusted project, on `MiniHeap` dumps.
use super::*;
use crate::HostConfig;
use ide_core::error::ErrorKind;
use ide_dap::{JvmMemoryUnavailable, LaunchConfig};
use ide_heap::testing::{MiniHeap, sample, sample_with_two_more_nodes};
use parking_lot::Mutex;
use serde_json::{Value, json};
use std::path::{Path, PathBuf};
use std::sync::Arc;
use tokio::io::{AsyncBufReadExt, AsyncReadExt, AsyncWriteExt, BufReader};
const WINDOW: &str = "main";
struct Fixture {
host: Host,
base: PathBuf,
root: PathBuf,
_temp: tempfile::TempDir,
}
impl Fixture {
async fn new() -> Self {
let temp = tempfile::tempdir().unwrap();
let base = dunce::canonicalize(temp.path()).unwrap();
let root = base.join("project");
std::fs::create_dir_all(&root).unwrap();
let host = Host::new(HostConfig {
config_dir: base.join("config"),
data_dir: base.join("data"),
local_data_dir: base.join("local"),
languages: Arc::new(ide_language::LanguageRegistry::load(None).unwrap()),
syntax: Arc::default(),
sink_for: Box::new(|_| Arc::new(|_| {})),
owner: |w| w,
runtime: tokio::runtime::Handle::current(),
project_environment: false,
});
host.open_project(WINDOW, &root).unwrap();
host.set_project_trust(&root, true, false).await.unwrap();
Self { host, base, root, _temp: temp }
}
fn dump(&self, name: &str, heap: &MiniHeap) -> PathBuf {
let path = self.base.join(name);
heap.write_to(&path);
path
}
async fn open_with(&self, path: &Path, request: u32, progress: Channel<HeapProgress>) -> IdeResult<HeapOpenResult> {
debug_heap_open(&self.host, WINDOW, path.into(), RequestId(request), progress).await
}
async fn open(&self, path: &Path) -> HeapSnapshotInfo {
match self.open_with(path, 1, Channel::new(|_| {})).await.unwrap() {
HeapOpenResult::Ready { snapshot } => snapshot,
other => panic!("expected a snapshot, got {other:?}"),
}
}
/// Opens a second trusted project in `window`.
async fn open_window(&self, window: &str) {
let root = self.base.join(window);
std::fs::create_dir_all(&root).unwrap();
self.host.open_project(window, &root).unwrap();
self.host.set_project_trust(&root, true, false).await.unwrap();
}
/// The snapshot directories under the registry's root (none when it does not exist yet).
fn snapshot_dirs(&self) -> Vec<PathBuf> {
let Ok(entries) = std::fs::read_dir(self.host.heap.root()) else { return Vec::new() };
entries.map(|e| e.unwrap().path()).collect()
}
}
fn recorder() -> (Channel<HeapProgress>, Arc<Mutex<Vec<HeapProgress>>>) {
let seen = Arc::new(Mutex::new(Vec::new()));
let sink = seen.clone();
(Channel::new(move |p| sink.lock().push(p)), seen)
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn open_and_query_a_dump() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
assert_eq!(info.label, "sample.hprof");
assert_eq!(info.origin, ide_heap::HeapOrigin::File { path: path.clone().into() });
assert_eq!(f.snapshot_dirs().len(), 1);
// The snapshot reads a private copy: the user's file may change while it is open.
std::fs::write(&path, b"").unwrap();
let classes = debug_heap_classes(&f.host, WINDOW, info.id).await.unwrap();
let node = classes.iter().find(|c| c.name == "Node").unwrap();
assert_eq!((node.count, node.retained), (3, 104));
let top = debug_heap_dominators(&f.host, WINDOW, info.id, None, None, 0, 10).await.unwrap();
assert_eq!(top.rows[0].retained, 72);
let n3 = top.rows.iter().find(|r| r.class_name == "Node").unwrap();
let below = debug_heap_dominators(&f.host, WINDOW, info.id, Some(n3.object), None, 0, 10).await.unwrap();
let long_array = below.rows.first().unwrap();
assert_eq!(long_array.address, "0x1400");
let path_to_root = debug_heap_path(&f.host, WINDOW, info.id, long_array.object).await.unwrap().unwrap();
assert_eq!(path_to_root.steps.len(), 2);
debug_heap_close(&f.host, WINDOW, info.id).await.unwrap();
assert!(f.snapshot_dirs().is_empty(), "the index directory goes with the snapshot");
assert!(path.is_file(), "an opened dump is the user's file");
let gone = debug_heap_classes(&f.host, WINDOW, info.id).await.unwrap_err();
assert_eq!(gone.kind, ErrorKind::NotFound);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn open_reports_progress_in_order() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let (progress, seen) = recorder();
assert!(matches!(f.open_with(&path, 1, progress).await.unwrap(), HeapOpenResult::Ready { .. }));
let phases: Vec<HeapPhase> = seen
.lock()
.iter()
.map(|p| match p {
HeapProgress::Analyzing { phase, .. } => *phase,
HeapProgress::Dumping => panic!("an open dumps nothing"),
})
.collect();
assert_eq!(phases.first(), Some(&HeapPhase::Reading));
assert!(phases.is_sorted_by_key(|&p| p as u8), "phases never go back: {phases:?}");
assert_eq!(phases.last(), Some(&HeapPhase::Saving));
let mut distinct = phases.clone();
distinct.dedup();
assert_eq!(
distinct,
[
HeapPhase::Reading,
HeapPhase::Linking,
HeapPhase::Dominators,
HeapPhase::Retained,
HeapPhase::Paths,
HeapPhase::Saving
],
"every phase is reported at least once however fast it runs"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn cancelled_open_leaves_nothing() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
f.host.cancel_request(RequestId(7));
assert_eq!(f.open_with(&path, 7, Channel::new(|_| {})).await.unwrap(), HeapOpenResult::Cancelled);
assert!(f.snapshot_dirs().is_empty());
assert!(f.host.heap.list().is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn not_a_dump_is_an_error() {
let f = Fixture::new().await;
let path = f.base.join("hello.hprof");
std::fs::write(&path, "hello").unwrap();
let err = f.open_with(&path, 1, Channel::new(|_| {})).await.unwrap_err();
assert!(err.message.contains("not a heap dump"), "{}", err.message);
assert!(f.snapshot_dirs().is_empty());
assert!(path.is_file());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn a_dump_without_its_end_record_is_an_error() {
let f = Fixture::new().await;
let path = f.base.join("killed.hprof");
let bytes = sample().bytes();
// HEAP DUMP END: tag, timestamp and length.
std::fs::write(&path, &bytes[..bytes.len() - 9]).unwrap();
let err = f.open_with(&path, 1, Channel::new(|_| {})).await.unwrap_err();
assert!(err.message.contains("the dump ends early"), "{}", err.message);
assert!(f.snapshot_dirs().is_empty());
assert!(f.host.heap.list().is_empty());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn compare_two_snapshots() {
let f = Fixture::new().await;
let base = f.open(&f.dump("base.hprof", &sample())).await;
let current = f.open(&f.dump("current.hprof", &sample_with_two_more_nodes())).await;
assert_ne!(base.id, current.id);
let diff = debug_heap_compare(&f.host, WINDOW, base.id, current.id).await.unwrap();
let node = diff.iter().find(|d| d.name == "Node").unwrap();
assert_eq!((node.base_count, node.count), (3, 5));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn unknown_snapshot_is_not_found() {
let f = Fixture::new().await;
let missing = HeapSnapshotId(99);
let not_found = |r: IdeError| assert_eq!(r.kind, ErrorKind::NotFound, "{}", r.message);
not_found(debug_heap_classes(&f.host, WINDOW, missing).await.unwrap_err());
not_found(debug_heap_dominators(&f.host, WINDOW, missing, None, None, 0, 10).await.unwrap_err());
not_found(debug_heap_path(&f.host, WINDOW, missing, HeapObjectRef(0)).await.unwrap_err());
not_found(debug_heap_save(&f.host, WINDOW, missing, f.base.join("x.hprof").into()).await.unwrap_err());
not_found(debug_heap_close(&f.host, WINDOW, missing).await.unwrap_err());
let open = f.open(&f.dump("sample.hprof", &sample())).await;
not_found(debug_heap_compare(&f.host, WINDOW, open.id, missing).await.unwrap_err());
// A snapshot belongs to the window that opened it.
f.open_window("other").await;
not_found(debug_heap_classes(&f.host, "other", open.id).await.unwrap_err());
not_found(debug_heap_close(&f.host, "other", open.id).await.unwrap_err());
assert!(debug_heap_classes(&f.host, WINDOW, open.id).await.is_ok());
}
/// The error of every op that takes a snapshot id, in `window`.
async fn query_errors(f: &Fixture, window: &str, id: HeapSnapshotId) -> Vec<IdeError> {
let host = &f.host;
vec![
debug_heap_classes(host, window, id).await.unwrap_err(),
debug_heap_dominators(host, window, id, None, None, 0, 10).await.unwrap_err(),
debug_heap_path(host, window, id, HeapObjectRef(0)).await.unwrap_err(),
debug_heap_compare(host, window, id, id).await.unwrap_err(),
debug_heap_save(host, window, id, f.base.join("saved.hprof").into()).await.unwrap_err(),
debug_heap_close(host, window, id).await.unwrap_err(),
]
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn queries_check_trust_before_the_snapshot() {
let f = Fixture::new().await;
let open = f.open(&f.dump("sample.hprof", &sample())).await;
let no_project = f.host.require_trust("nowhere").unwrap_err();
for err in query_errors(&f, "nowhere", open.id).await {
assert_eq!((err.kind, &err.message), (no_project.kind, &no_project.message));
}
f.host.set_project_trust(&f.root, false, false).await.unwrap();
let restricted = f.host.require_trust(WINDOW).unwrap_err();
assert_ne!(restricted.kind, ErrorKind::NotFound);
// An id that is not open is refused for trust too: the check comes before the lookup.
for id in [open.id, HeapSnapshotId(99)] {
for err in query_errors(&f, WINDOW, id).await {
assert_eq!((err.kind, &err.message), (restricted.kind, &restricted.message));
}
}
assert!(!f.base.join("saved.hprof").exists());
assert!(f.host.heap.get(open.id).is_some(), "a refused close keeps the snapshot");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn shutdown_closes_every_snapshot() {
let f = Fixture::new().await;
let external = f.dump("sample.hprof", &sample());
f.open(&external).await;
// A capture: the dump sits in the snapshot directory.
let captured = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let dump = captured.join("dump.hprof");
sample_with_two_more_nodes().write_to(&dump);
let origin = ide_heap::HeapOrigin::Captured {
session: DebugSessionId(1),
pid: 42,
java_version: "21".into(),
taken_while_suspended: false,
dump_ms: 1,
};
let analyzed = analyze(
&f.host,
WINDOW,
origin,
dump.clone(),
captured.clone(),
AnalyzeOptions::default(),
false,
&CancellationToken::new(),
&Channel::new(|_| {}),
)
.await
.unwrap();
assert!(matches!(analyzed, Analyzed::Ready(_)));
// An open still running: shutdown cancels it and waits for it to clean up.
let running_dir = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let cancel = CancellationToken::new();
let running = Running::begin(&f.host, RequestId(50), &cancel);
assert_eq!(f.snapshot_dirs().len(), 3);
let in_flight = async {
let cancelled = tokio::time::timeout(std::time::Duration::from_secs(5), cancel.cancelled()).await;
assert!(cancelled.is_ok(), "shutdown cancels a running capture or open");
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
std::fs::remove_dir_all(&running_dir).unwrap();
drop(running);
};
tokio::join!(f.host.shutdown(), in_flight);
assert!(f.snapshot_dirs().is_empty(), "{:?}", f.snapshot_dirs());
assert!(f.host.heap.list().is_empty());
assert!(external.is_file(), "an opened dump is the user's file");
let late = CancellationToken::new();
let _late = Running::begin(&f.host, RequestId(51), &late);
assert!(late.is_cancelled(), "nothing starts after shutdown");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn shutdown_does_not_wait_forever_for_a_stuck_capture() {
let f = Fixture::new().await;
let stuck_dir = ide_heap::create_snapshot_dir(f.host.heap.root()).unwrap();
let cancel = CancellationToken::new();
let _stuck = Running::begin(&f.host, RequestId(60), &cancel);
let started = std::time::Instant::now();
shutdown_within(&f.host, std::time::Duration::from_millis(200)).await;
assert!(started.elapsed() < std::time::Duration::from_secs(5));
assert!(cancel.is_cancelled());
assert!(!stuck_dir.exists(), "its directory is removed once the wait is over");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn save_copies_the_dump_and_replaces_the_target() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
let target = f.base.join("saved.hprof");
std::fs::write(&target, "older").unwrap();
debug_heap_save(&f.host, WINDOW, info.id, target.clone().into()).await.unwrap();
assert_eq!(std::fs::read(&target).unwrap(), sample().bytes());
// Over the dump the snapshot itself maps: the index keeps reading the old file.
debug_heap_save(&f.host, WINDOW, info.id, path.clone().into()).await.unwrap();
assert_eq!(std::fs::read(&path).unwrap(), sample().bytes());
assert!(debug_heap_classes(&f.host, WINDOW, info.id).await.is_ok());
let leftovers: Vec<_> = std::fs::read_dir(&f.base)
.unwrap()
.map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
.filter(|name| name.ends_with(".tmp"))
.collect();
assert!(leftovers.is_empty(), "{leftovers:?}");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn closing_the_project_closes_its_snapshots() {
let f = Fixture::new().await;
let path = f.dump("sample.hprof", &sample());
let info = f.open(&path).await;
f.host.close_project(WINDOW);
assert!(f.host.heap.get(info.id).is_none());
// The directory is deleted on the blocking pool.
for _ in 0..200 {
if f.snapshot_dirs().is_empty() {
break;
}
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
assert!(f.snapshot_dirs().is_empty());
assert!(path.is_file());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn capture_without_jvm_session_is_unavailable() {
let f = Fixture::new().await;
let listener = tokio::net::TcpListener::bind(("127.0.0.1", 0)).await.unwrap();
let port = listener.local_addr().unwrap().port();
tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
fake_adapter(stream).await;
});
let config = LaunchConfig {
name: "Remote".into(),
adapter: "delve".into(),
request: "attach".into(),
program: None,
args: vec![],
cwd: Some(f.root.clone().into()),
env: Default::default(),
extra_json: Some(json!({"debugServer": format!("127.0.0.1:{port}"), "mode": "remote"}).to_string()),
};
let session = crate::ops::debug::debug_start(&f.host, WINDOW, config).await.unwrap();
let (progress, seen) = recorder();
let result = debug_heap_capture(&f.host, WINDOW, session, RequestId(3), progress).await.unwrap();
assert_eq!(
result,
HeapCaptureResult::Unavailable {
reason: JvmHeapUnavailable::Memory { reason: JvmMemoryUnavailable::NoJvmProcess }
}
);
assert_eq!(*seen.lock(), [HeapProgress::Dumping]);
assert!(f.snapshot_dirs().is_empty());
f.host.debugger.stop(session).await.unwrap();
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn capture_of_a_missing_session_deletes_its_directory() {
let f = Fixture::new().await;
let missing = DebugSessionId(404);
assert!(debug_heap_capture(&f.host, WINDOW, missing, RequestId(4), Channel::new(|_| {})).await.is_err());
assert!(f.snapshot_dirs().is_empty());
}
#[test]
fn progress_is_thinned_but_keeps_every_phase() {
let (progress, seen) = recorder();
let report = reporter(&progress, || None);
for done in 0..50 {
report(HeapPhase::Reading, done, 50).unwrap();
}
report(HeapPhase::Linking, 0, 10).unwrap();
assert_eq!(
*seen.lock(),
[
HeapProgress::Analyzing { phase: HeapPhase::Reading, done: 0, total: 50 },
HeapProgress::Analyzing { phase: HeapPhase::Linking, done: 0, total: 10 },
]
);
}
#[test]
fn too_little_disk_for_the_index_stops_at_linking() {
let (progress, _) = recorder();
let report = reporter(&progress, || Some(1));
report(HeapPhase::Reading, 0, 50).unwrap();
let needed = ide_heap::index_need(1000);
match report(HeapPhase::Linking, 0, 1000) {
Err(ide_heap::HeapError::NotEnoughDiskSpace { needed: n, available: 1 }) => assert_eq!(n, needed),
other => panic!("expected NotEnoughDiskSpace, got {other:?}"),
}
let (progress, _) = recorder();
let report = reporter(&progress, || Some(u64::MAX));
assert!(report(HeapPhase::Linking, 0, 1000).is_ok());
}
/// Answers every request of a session that never runs a JVM.
async fn fake_adapter(stream: tokio::net::TcpStream) {
let (r, mut w) = stream.into_split();
let mut r = BufReader::new(r);
let mut seq = 0;
loop {
let mut length = None;
loop {
let mut line = String::new();
if r.read_line(&mut line).await.unwrap_or(0) == 0 {
return;
}
let line = line.trim();
if line.is_empty() {
break;
}
if let Some(n) = line.strip_prefix("Content-Length:") {
length = n.trim().parse::<usize>().ok();
}
}
let mut body = vec![0; length.unwrap()];
r.read_exact(&mut body).await.unwrap();
let msg: Value = serde_json::from_slice(&body).unwrap();
if msg["type"] != "request" {
continue;
}
let command = msg["command"].as_str().unwrap_or_default().to_owned();
let reply = match command.as_str() {
"initialize" => json!({"supportsConfigurationDoneRequest": true}),
"threads" => json!({"threads": []}),
_ => json!({}),
};
let mut out = vec![
json!({"type": "response", "request_seq": msg["seq"], "command": command, "success": true, "body": reply}),
];
if command == "launch" || command == "attach" {
out.push(json!({"type": "event", "event": "initialized"}));
}
for mut message in out {
seq += 1;
message["seq"] = seq.into();
let bytes = serde_json::to_vec(&message).unwrap();
w.write_all(format!("Content-Length: {}\r\n\r\n", bytes.len()).as_bytes()).await.unwrap();
w.write_all(&bytes).await.unwrap();
}
if command == "disconnect" {
return;
}
}
}
+14
View File
@@ -6,6 +6,7 @@ use crate::rpc::Dispatcher;
use super::build::load;
use ide_build::compile;
use ide_core::error::ErrorKind;
use ide_core::{CancellationToken, IdeError, IdeResult, RequestId};
use ide_lsp::jdtls::{JavaBuildStatus, JavaLaunchInfo};
use ide_run::BuildPolicy;
@@ -15,6 +16,7 @@ use std::path::PathBuf;
/// Builds the workspace (when `build`), then resolves the runtime (or test)
/// classpath and the JDK of the module at `module` (absolute, or relative to the project root).
/// A module with Kotlin / Groovy / Scala sources is built by its build system, whose class directories come first.
/// Resources a Gradle build filters or renames are processed by Gradle, replacing the language server's plain copies.
pub async fn java_resolve_launch(
host: &Host,
window: &str,
@@ -109,6 +111,18 @@ async fn resolve_launch(
}
info.class_paths = class_paths;
apply_library_classpath(&mut info, &project.root, &settings.project_structure.libraries, test);
if build && !delegated {
let cache_dir = project.build_cache_dir();
match compile::process_launch_resources(&project.root, &model, &cache_dir, &info.class_paths, &cancel) {
Ok(tasks) if !tasks.is_empty() => tracing::info!(?tasks, "Processed launch resources with Gradle"),
Ok(_) => {}
Err(e) if e.kind == ErrorKind::Cancelled => return Err(e),
Err(e) => info.hints.push(format!(
"Gradle could not process the resources, so they are used as they are in the source folders:\n{}",
e.message
)),
}
}
let compile_ms = started.elapsed().as_millis() - lsp_ms;
// The module/project JDK, else the JDK jdtls compiles the module against, else one matching its Java version.
let project_jdk = ide_sdk::apply::module_jdk(&project.root, &module, &settings);
+200 -16
View File
@@ -830,7 +830,7 @@ pub async fn lsp_code_actions(
};
let plugin_actions: Vec<CodeAction> = plugin_actions
.into_iter()
.map(|(plugin, a)| plugin_code_action(&plugin, &project.root, a))
.map(|(plugin, a)| plugin_code_action_in(&plugin, &plugin_edit_roots(&project), a))
.filter(|a| {
only.is_empty()
|| a.kind.as_ref().is_some_and(|k| only.iter().any(|o| k == o || k.starts_with(&format!("{o}."))))
@@ -846,10 +846,21 @@ pub async fn lsp_code_actions(
prepare_actions(project, doc, version, actions).await
}
/// Where a plugin's code action may change or open files: the project root and its content
/// roots (a group's members, extra Project Structure folders), never git's own files.
fn plugin_edit_roots(project: &crate::Project) -> Vec<std::path::PathBuf> {
std::iter::once(project.root.clone()).chain(project.content_roots()).collect()
}
#[cfg(test)]
fn plugin_code_action(plugin: &str, root: &Path, a: ide_plugin::CodeAction) -> CodeAction {
plugin_code_action_in(plugin, &[root.to_path_buf()], a)
}
fn plugin_code_action_in(plugin: &str, roots: &[std::path::PathBuf], a: ide_plugin::CodeAction) -> CodeAction {
// With `data` and nothing to apply, the plugin computes the edits when the action is picked.
let lazy = a.edits.is_empty() && a.operations.is_empty() && a.data.is_some();
let (files, open, disabled_reason) = match plugin_file_edits(root, &a) {
let (files, open, disabled_reason) = match plugin_file_edits(roots, &a) {
Ok((files, open)) => (files, open, None),
Err(reason) => (Vec::new(), None, Some(reason)),
};
@@ -876,17 +887,16 @@ fn plugin_code_action(plugin: &str, root: &Path, a: ide_plugin::CodeAction) -> C
}
/// The action's file operations, then its text edits, as one workspace edit's files, and the
/// file to open. An action with file operations or an `open` target stays inside the project
/// root as a whole: a path outside it disables the action.
/// file to open. Every path stays inside `roots` (the project root first) and off git's own
/// files: one that does not disables the action.
fn plugin_file_edits(
root: &Path,
roots: &[std::path::PathBuf],
a: &ide_plugin::CodeAction,
) -> Result<(Vec<ide_core::FileEdit>, Option<std::path::PathBuf>), String> {
use ide_core::FileEdit;
use ide_plugin::FileOperation;
let confine = !a.operations.is_empty() || a.open.is_some();
let confined = |path: &Path| ide_plugin::resolve_new_in_root(root, &path.to_string_lossy());
let writable = |path: &Path| ide_plugin::resolve_writable_in_root(root, &path.to_string_lossy());
let confined = |path: &Path| ide_plugin::resolve_new_in_roots(roots, &path.to_string_lossy());
let writable = |path: &Path| ide_plugin::resolve_writable_in_roots(roots, &path.to_string_lossy());
let mut files = Vec::new();
let mut created_with_text = Vec::new();
for operation in &a.operations {
@@ -913,13 +923,14 @@ fn plugin_file_edits(
}
}
for (path, edits) in &a.edits {
let path = if confine { writable(path)? } else { path.clone() };
let path = writable(path)?;
if created_with_text.contains(&path) {
return Err("put the text of a created file in its `create`, not in an edit".into());
}
files.push(FileEdit::Edit { path: path.into(), edits: edits.clone(), version: None });
}
let open = a.open.as_deref().map(confined).transpose()?;
// Opening a file hands its text to the plugin, so a link is judged by the file it leads to.
let open = a.open.as_deref().map(writable).transpose()?;
Ok((files, open))
}
@@ -947,8 +958,9 @@ async fn resolve_action(host: &Host, window: &str, doc: DocId, action: CodeActio
let plugin = r.plugin.clone();
let invocation = ide_plugin::Invocation { client: Some(window.to_owned()), ..Default::default() };
let resolved = blocking(move || plugins.resolve_code_action(&plugin, &d, pending, invocation)).await?;
let root = host.project(window)?.root.clone();
Ok(plugin_code_action(&r.plugin, &root, ide_plugin::CodeAction { data: None, ..resolved }))
let project = host.project(window)?;
let roots = plugin_edit_roots(&project);
Ok(plugin_code_action_in(&r.plugin, &roots, ide_plugin::CodeAction { data: None, ..resolved }))
}
/// Cancellable code actions with the kinds the servers advertise (Refactor This, refactoring availability).
@@ -2246,18 +2258,18 @@ mod tests {
};
for name in ["outside", "dangling"] {
let create = FileOperation::Create { path: root.join(name), text: "text".into(), overwrite: true };
assert!(plugin_file_edits(&root, &action(vec![create], Vec::new())).is_err());
assert!(plugin_file_edits(std::slice::from_ref(&root), &action(vec![create], Vec::new())).is_err());
let edits = vec![(root.join(name), vec![TextEdit { range: Range::default(), new_text: "text".into() }])];
assert!(plugin_file_edits(&root, &action(Vec::new(), edits)).is_err());
assert!(plugin_file_edits(std::slice::from_ref(&root), &action(Vec::new(), edits)).is_err());
}
let create = FileOperation::Create { path: root.join("inside"), text: "text".into(), overwrite: true };
let (files, _) = plugin_file_edits(&root, &action(vec![create], Vec::new())).unwrap();
let (files, _) = plugin_file_edits(std::slice::from_ref(&root), &action(vec![create], Vec::new())).unwrap();
assert!(matches!(&files[0], FileEdit::Create { path, .. } if **path == root.join("target.txt")));
let operations = vec![
FileOperation::Rename { from: root.join("outside"), to: root.join("dangling") },
FileOperation::Delete { path: root.join("outside") },
];
let (files, _) = plugin_file_edits(&root, &action(operations, Vec::new())).unwrap();
let (files, _) = plugin_file_edits(std::slice::from_ref(&root), &action(operations, Vec::new())).unwrap();
assert!(matches!(&files[0], FileEdit::Rename { from, to }
if **from == root.join("outside") && **to == root.join("dangling")));
assert!(matches!(&files[1], FileEdit::Delete { path, .. } if **path == root.join("outside")));
@@ -2341,6 +2353,178 @@ mod tests {
assert_eq!((r.data.as_deref(), r.file), (Some("ref"), true));
}
/// Advisory #4b: an action carrying only text edits (no file operation, no `open`) must still
/// be confined to the project root and kept off git's own files. A plugin needs no grant to
/// offer a quick fix, so this is decided here.
#[test]
fn a_plugin_edits_only_action_stays_in_the_project_and_off_git() {
let tmp = tempfile::tempdir().unwrap();
let root = tmp.path().join("proj");
std::fs::create_dir_all(root.join(".git")).unwrap();
std::fs::write(root.join(".git/config"), "[core]\n").unwrap();
std::fs::write(root.join("a.txt"), "a").unwrap();
let root = dunce::canonicalize(&root).unwrap();
let edits_only = |path: std::path::PathBuf| ide_plugin::CodeAction {
title: "Add import".into(),
kind: Some("quickfix".into()),
preferred: false,
edits: vec![(path, vec![TextEdit { range: Range::default(), new_text: "x".into() }])],
operations: Vec::new(),
open: None,
data: None,
file_action: false,
};
// In the project root: allowed.
assert_eq!(plugin_code_action("acme.x", &root, edits_only(root.join("a.txt"))).disabled_reason, None);
// Outside the root: disabled (previously an edits-only action was not confined).
let outside = plugin_code_action("acme.x", &root, edits_only(tmp.path().join("evil.txt")));
assert!(
outside.disabled_reason.as_deref().is_some_and(|r| r.contains("outside the project root")),
"{outside:?}"
);
assert!(outside.edit.unwrap().files.is_empty());
// Into .git: disabled.
let git = plugin_code_action("acme.x", &root, edits_only(root.join(".git/config")));
assert!(git.disabled_reason.as_deref().is_some_and(|r| r.contains("git's own files")), "{git:?}");
}
/// A project window group's files live in its members, outside the container root: a quick
/// fix reaches them, and their `.git` stays out of reach.
#[test]
fn a_plugin_action_reaches_the_content_roots_but_not_their_git() {
use ide_plugin::FileOperation::Create;
let tmp = tempfile::tempdir().unwrap();
for dir in ["group/.openidea", "member/.git", "member/src", "elsewhere"] {
std::fs::create_dir_all(tmp.path().join(dir)).unwrap();
}
let base = dunce::canonicalize(tmp.path()).unwrap();
std::fs::write(base.join("member/src/a.txt"), "a").unwrap();
std::fs::write(base.join("member/.git/config"), "[core]\n").unwrap();
let roots = [base.join("group"), base.join("member")];
let action =
|edits: Vec<std::path::PathBuf>, operations: Vec<ide_plugin::FileOperation>| ide_plugin::CodeAction {
title: "Fix".into(),
kind: Some("quickfix".into()),
preferred: false,
edits: edits
.into_iter()
.map(|path| (path, vec![TextEdit { range: Range::default(), new_text: "x".into() }]))
.collect(),
operations,
open: None,
data: None,
file_action: false,
};
let offer = |a| plugin_code_action_in("acme.x", &roots, a);
assert_eq!(offer(action(vec![base.join("member/src/a.txt")], Vec::new())).disabled_reason, None);
let created = Create { path: base.join("member/src/b.txt"), text: String::new(), overwrite: false };
assert_eq!(offer(action(Vec::new(), vec![created])).disabled_reason, None);
for refused in [base.join("member/.git/config"), base.join("elsewhere/c.txt")] {
let offered = offer(action(vec![refused.clone()], Vec::new()));
assert!(offered.disabled_reason.is_some(), "{refused:?}: {offered:?}");
assert!(offered.edit.unwrap().files.is_empty());
}
let hook = Create { path: base.join("member/.git/hooks/pre-commit"), text: "x".into(), overwrite: false };
assert!(offer(action(Vec::new(), vec![hook])).disabled_reason.is_some());
}
/// Advisory #4b: a quick fix's file operations and its `open` are held to the same rule as its
/// edits, under every name that reaches git's own files.
#[test]
fn a_plugin_action_cannot_change_or_open_git_files() {
use ide_plugin::FileOperation::{Create, Delete, Rename};
let tmp = tempfile::tempdir().unwrap();
for dir in ["proj/.git/hooks", "worktree"] {
std::fs::create_dir_all(tmp.path().join(dir)).unwrap();
}
let root = dunce::canonicalize(tmp.path().join("proj")).unwrap();
let worktree = dunce::canonicalize(tmp.path().join("worktree")).unwrap();
std::fs::write(
root.join(".git/config"),
"[core]
",
)
.unwrap();
std::fs::write(root.join("a.txt"), "a").unwrap();
std::fs::write(
worktree.join(".git"),
"gitdir: elsewhere
",
)
.unwrap();
std::fs::write(tmp.path().join("profile.txt"), "outside").unwrap();
let action =
|edits: Vec<std::path::PathBuf>, operations: Vec<ide_plugin::FileOperation>| ide_plugin::CodeAction {
title: "Add import".into(),
kind: Some("quickfix".into()),
preferred: false,
edits: edits
.into_iter()
.map(|path| (path, vec![TextEdit { range: Range::default(), new_text: "x".into() }]))
.collect(),
operations,
open: None,
data: None,
file_action: false,
};
let refused = |root: &Path, action: ide_plugin::CodeAction, why: &str| {
let offered = plugin_code_action("acme.x", root, action);
assert!(offered.disabled_reason.as_deref().is_some_and(|r| r.contains(why)), "{offered:?}");
assert!(offered.edit.unwrap().files.is_empty());
};
for git in [root.join(".git/config"), root.join(".GIT/config"), ".git/hooks/pre-commit".into()] {
refused(&root, action(vec![git], Vec::new()), "git's own files");
}
for operation in [
Create {
path: root.join(".git/hooks/pre-commit"),
text: "#!/bin/sh
"
.into(),
overwrite: false,
},
Rename { from: root.join("a.txt"), to: root.join(".git/config") },
Rename { from: root.join(".git/config"), to: root.join("b.txt") },
Delete { path: root.join(".git/config") },
] {
refused(&root, action(Vec::new(), vec![operation]), "git's own files");
}
// Opening a file hands its text to the plugin's providers.
let opens =
|path: std::path::PathBuf| ide_plugin::CodeAction { open: Some(path), ..action(Vec::new(), Vec::new()) };
refused(&root, opens(root.join(".git/config")), "git's own files");
// An edit lands in the file a link leads to, and `open` shows that file: both are judged by it.
#[cfg(unix)]
{
std::os::unix::fs::symlink(tmp.path().join("profile.txt"), root.join("link.txt")).unwrap();
std::os::unix::fs::symlink(root.join(".git/config"), root.join("alias")).unwrap();
refused(&root, action(vec![root.join("link.txt")], Vec::new()), "outside the project root");
refused(&root, action(vec![root.join("alias")], Vec::new()), "git's own files");
refused(&root, opens(root.join("link.txt")), "outside the project root");
refused(&root, opens(root.join("alias")), "git's own files");
}
// A linked worktree's `.git` file, under the short name Windows opens it by as well.
if cfg!(windows) && worktree.join("GIT~1").exists() {
for operation in [
Delete { path: worktree.join("GIT~1") },
Create {
path: worktree.join("GIT~1"),
text: "gitdir: other
"
.into(),
overwrite: true,
},
] {
refused(&worktree, action(Vec::new(), vec![operation]), "git's own files");
}
refused(&worktree, action(vec![worktree.join("GIT~1")], Vec::new()), "git's own files");
refused(&worktree, opens(worktree.join("GIT~1")), "git's own files");
} else {
eprintln!("short name not tried: this volume keeps none");
}
}
#[test]
fn every_completion_kind_the_plugin_api_documents_is_understood() {
let wit = include_str!("../../../../wit/providers.wit");
+2
View File
@@ -15,6 +15,7 @@ pub mod doc;
pub mod docker;
pub mod env;
pub mod fs;
pub mod heap;
pub mod history;
pub mod java;
pub mod jvm_icons;
@@ -83,6 +84,7 @@ pub fn register_all(d: &mut crate::rpc::Dispatcher) {
sdk::register(d);
java::register(d);
debug::register(d);
heap::register(d);
bytes::register(d);
vcs::register(d);
doc::register(d);
+183 -36
View File
@@ -71,6 +71,12 @@ pub trait PluginWorkspace: Send + Sync {
/// Roots of the open projects.
fn projects(&self) -> Vec<PathBuf>;
/// Roots of the workspace members attached to the project at `root` (a
/// project window group's folders, as the editor and LSP resolve them).
/// Empty for a plain project.
fn member_roots(&self, _root: &Path) -> Vec<PathBuf> {
Vec::new()
}
/// Unsaved text of the open document at `path`.
fn buffer_text(&self, path: &Path) -> Option<String>;
/// A window's unsaved text, with the normal project overlay as fallback.
@@ -1257,6 +1263,8 @@ pub(super) fn commit_draft(
let Some(r) = ide_vcs::Repo::discover(&repo.root.0).map_err(failed)? else {
return Err(HostError::NotFound(format!("{} is not in a git repository", repo.root.0.display())));
};
// Both callers have checked that the window's project is trusted.
r.set_execution_allowed(true);
let pending = r.pending_changes(&repo.options, budget, &|path| Ok(workspace.vcs_encoding(path))).map_err(
|e| match e.kind {
ide_core::error::ErrorKind::InvalidInput => HostError::Invalid(e.message),
@@ -2019,6 +2027,10 @@ impl HostApi for PluginApi {
self.shared.workspace().map(|w| w.projects()).unwrap_or_default()
}
fn member_roots(&self, root: &Path) -> Vec<PathBuf> {
self.shared.workspace().map(|w| w.member_roots(root)).unwrap_or_default()
}
fn buffer_text(&self, path: &Path) -> Option<String> {
self.shared.workspace()?.buffer_text(path)
}
@@ -2083,7 +2095,7 @@ impl HostApi for PluginApi {
}
fn vcs_status(&self, root: &Path) -> Result<Option<RepositoryStatus>, HostError> {
let Some(repo) = ide_vcs::Repo::discover(root).map_err(failed)? else { return Ok(None) };
let Some(repo) = self.repo(root)? else { return Ok(None) };
let info = repo.branch_info().map_err(failed)?;
let changed = repo.status().map_err(failed)?.len();
Ok(Some(RepositoryStatus {
@@ -2109,7 +2121,7 @@ impl HostApi for PluginApi {
if !workspace.is_some_and(|w| w.trusted(root)) {
return Err(HostError::Failed("the project is in Restricted Mode: git does not run there".into()));
}
let Some(repo) = ide_vcs::Repo::discover(root).map_err(failed)? else {
let Some(repo) = self.repo(root)? else {
return Err(HostError::NotFound("the project is not in a git repository".into()));
};
match repo.fetch(cancel, &|_| {}) {
@@ -2159,23 +2171,29 @@ impl HostApi for PluginApi {
out.sort();
return Ok(out);
}
// A project window group's members are project scope too (their files
// are in the group's index, and a group root holds none of them).
let mut roots = vec![root.to_path_buf()];
roots.extend(self.member_roots(root));
let mut out = Vec::new();
for (n, entry) in ignore::WalkBuilder::new(root)
.hidden(false)
.filter_entry(|e| e.file_name() != ".git")
.build()
.flatten()
.enumerate()
{
if n >= MAX_WALK || out.len() >= limit {
break;
}
if !entry.file_type().is_some_and(|t| t.is_file()) {
continue;
}
let Ok(rel) = entry.path().strip_prefix(root) else { continue };
if matcher.is_match(rel) {
out.push(to_ipc(entry.path()));
'walk: for root in &roots {
for (n, entry) in ignore::WalkBuilder::new(root)
.hidden(false)
.filter_entry(|e| e.file_name() != ".git")
.build()
.flatten()
.enumerate()
{
if n >= MAX_WALK || out.len() >= limit {
break 'walk;
}
if !entry.file_type().is_some_and(|t| t.is_file()) {
continue;
}
let Ok(rel) = entry.path().strip_prefix(root) else { continue };
if matcher.is_match(rel) {
out.push(to_ipc(entry.path()));
}
}
}
out.sort();
@@ -2892,11 +2910,26 @@ impl HostWorkspace {
.and_then(|w| host.project(w).ok())
.filter(|p| p.root == root)
.or_else(|| host.windows_for_root(root).first().and_then(|w| host.project(w).ok()))
// A project window group's member root: its files belong to the
// group's project, like everywhere else in the editor.
.or_else(|| {
host.projects()
.into_iter()
.find(|p| crate::project::workspace_member_roots(&p.root).iter().any(|m| m.as_path() == root))
})
}
fn projects(&self) -> Vec<Arc<crate::project::Project>> {
self.0.upgrade().map(|h| h.projects()).unwrap_or_default()
}
/// The open project whose workspace group contains `root` (the group root
/// itself matches `p.root == root` and needs no member lookup).
fn project_covering(&self, root: &Path) -> Option<Arc<crate::project::Project>> {
self.projects().into_iter().find(|p| {
p.root == root || crate::project::workspace_member_roots(&p.root).iter().any(|m| m.as_path() == root)
})
}
/// The project holding an open document at `path`, and its id.
fn open_document(&self, path: &Path) -> Option<(Arc<crate::project::Project>, ide_core::DocId)> {
self.projects().into_iter().find_map(|p| p.documents.by_path(path).map(|doc| (p, doc)))
@@ -2986,6 +3019,10 @@ impl PluginWorkspace for HostWorkspace {
) -> Result<ide_plugin::TextSearch, HostError> {
let host = self.0.upgrade().ok_or(HostError::Cancelled)?;
let project = self.project_for(client, root).ok_or_else(|| HostError::NotFound("project closed".into()))?;
// Matches must land in the project root or, for a project window group,
// in any of its members.
let mut roots = vec![root.to_path_buf()];
roots.extend(crate::project::workspace_member_roots(root));
if limit == 0 {
return Ok(ide_plugin::TextSearch::default());
}
@@ -3015,8 +3052,9 @@ impl PluginWorkspace for HostWorkspace {
if cancel.is_cancelled() {
return Err(HostError::Cancelled);
}
// Restrict project.read to the project root, including symlink targets.
let Ok(path) = ide_plugin::resolve_in_root(root, &to_ipc(&file.path.0)) else {
// Restrict project.read to the project root and its workspace
// members, including symlink targets.
let Ok(path) = ide_plugin::resolve_in_roots(&roots, &to_ipc(&file.path.0)) else {
continue;
};
let text = project.documents.text_of(&path).or_else(|| {
@@ -3082,12 +3120,19 @@ impl PluginWorkspace for HostWorkspace {
}
fn projects(&self) -> Vec<PathBuf> {
let mut roots: Vec<PathBuf> = HostWorkspace::projects(self).iter().map(|p| p.root.clone()).collect();
let mut roots: Vec<PathBuf> = HostWorkspace::projects(self)
.iter()
.flat_map(|p| std::iter::once(p.root.clone()).chain(crate::project::workspace_member_roots(&p.root)))
.collect();
roots.sort();
roots.dedup();
roots
}
fn member_roots(&self, root: &Path) -> Vec<PathBuf> {
crate::project::workspace_member_roots(root)
}
fn buffer_text(&self, path: &Path) -> Option<String> {
HostWorkspace::projects(self).iter().find_map(|p| p.documents.text_of(path))
}
@@ -3098,9 +3143,9 @@ impl PluginWorkspace for HostWorkspace {
}
fn indexed_files(&self, root: &Path, matches: &dyn Fn(&Path) -> bool, limit: usize) -> Option<Vec<PathBuf>> {
// The index of the project rooted at `root` (a workspace member's files are in its own).
let project = HostWorkspace::projects(self).into_iter().find(|p| p.root == root)?;
project.index.files_matching(root, matches, limit)
// The index covering `root`: the project rooted there, or the workspace
// group project whose member it is (a member's files are in the group's).
HostWorkspace::project_covering(self, root)?.index.files_matching(root, matches, limit)
}
fn symbols(
@@ -3111,9 +3156,7 @@ impl PluginWorkspace for HostWorkspace {
cancel: &CancellationToken,
deadline: Instant,
) -> Result<Vec<ide_plugin::SymbolInfo>, HostError> {
let project = HostWorkspace::projects(self)
.into_iter()
.find(|p| p.root == root)
let project = HostWorkspace::project_covering(self, root)
.ok_or_else(|| HostError::NotFound("the project is no longer open".into()))?;
let stopped = || cancel.is_cancelled() || Instant::now() >= deadline;
let interrupted = || {
@@ -3123,15 +3166,13 @@ impl PluginWorkspace for HostWorkspace {
HostError::Failed("symbol search timed out".into())
}
};
// The symbols of the requested root and, for a project window group, of
// its members (a member's symbols are in the group's index).
let members = crate::project::workspace_member_roots(root);
let in_scope = |path: &Path| path.starts_with(root) || members.iter().any(|m| path.starts_with(m));
let symbols = project
.index
.find_symbols_until(
query,
ide_index::SymbolFilter::All,
limit.min(1000),
&|path| path.starts_with(root),
&stopped,
)
.find_symbols_until(query, ide_index::SymbolFilter::All, limit.min(1000), &in_scope, &stopped)
.ok_or_else(interrupted)?;
let mut out = Vec::new();
for s in symbols {
@@ -3173,7 +3214,8 @@ impl PluginWorkspace for HostWorkspace {
}
fn trusted(&self, root: &Path) -> bool {
HostWorkspace::projects(self).iter().any(|p| p.root == root && p.is_trusted())
// A workspace member is as trusted as the group's project it was opened with.
HostWorkspace::project_covering(self, root).is_some_and(|p| p.is_trusted())
}
fn project_of(&self, window: &str) -> Option<PathBuf> {
@@ -3198,6 +3240,12 @@ impl PluginWorkspace for HostWorkspace {
.iter()
.filter(|p| path.starts_with(&p.root))
.max_by_key(|p| p.root.as_os_str().len())
// A project window group's member files report through the group's project.
.or_else(|| {
projects
.iter()
.find(|p| crate::project::workspace_member_roots(&p.root).iter().any(|m| path.starts_with(m)))
})
.ok_or_else(|| HostError::Invalid(format!("{} is not in an open project", path.display())))?;
project.sink.emit(ide_core::IdeEvent::Diagnostics {
path: path.to_path_buf().into(),
@@ -3212,15 +3260,25 @@ impl PluginWorkspace for HostWorkspace {
.into_iter()
.filter(|p| path.starts_with(&p.root))
.max_by_key(|p| p.root.as_os_str().len())
.or_else(|| {
HostWorkspace::projects(self)
.into_iter()
.find(|p| crate::project::workspace_member_roots(&p.root).iter().any(|m| path.starts_with(m)))
})
.and_then(|p| p.vcs_encoding(path, None).ok())
.unwrap_or_else(|| encoding_rs::UTF_8.into())
}
fn plugin_settings(&self, plugin: &str, root: Option<&Path>) -> serde_json::Map<String, serde_json::Value> {
let Some(host) = self.0.upgrade() else { return serde_json::Map::new() };
// A member root reads the settings of the group's project, like the
// group's own root does.
let settings = match root.and_then(|r| host.projects().into_iter().find(|p| p.root == r)) {
Some(p) => p.settings.effective(),
None => host.global_settings.effective(),
None => match root.map(|r| HostWorkspace::project_covering(self, r)) {
Some(Some(p)) => p.settings.effective(),
_ => host.global_settings.effective(),
},
};
settings.plugins.get(plugin).cloned().unwrap_or_default()
}
@@ -3526,6 +3584,55 @@ mod tests {
);
}
/// A workspace whose container root has no files: the member holds them.
struct Members(PathBuf, Vec<PathBuf>);
impl PluginWorkspace for Members {
fn projects(&self) -> Vec<PathBuf> {
vec![self.0.clone()]
}
fn member_roots(&self, root: &Path) -> Vec<PathBuf> {
if root == self.0 { self.1.clone() } else { Vec::new() }
}
fn buffer_text(&self, _: &Path) -> Option<String> {
None
}
fn document(&self, _: &Path) -> Option<(u64, String)> {
None
}
fn apply_edits(&self, _: &Path, _: u64, _: &[TextEdit]) -> Result<u64, HostError> {
Err(HostError::Stale)
}
fn trusted(&self, _: &Path) -> bool {
true
}
fn project_of(&self, _: &str) -> Option<PathBuf> {
None
}
fn publish_diagnostics(&self, _: &str, _: &Path, _: Vec<ide_core::Diagnostic>) -> Result<(), HostError> {
Ok(())
}
}
#[test]
fn find_files_cover_workspace_members() {
let tmp = tempfile::tempdir().unwrap();
let base = dunce::canonicalize(tmp.path()).unwrap();
let (root, member) = (base.join("workspace"), base.join("member"));
std::fs::create_dir_all(root.join(".openidea")).unwrap();
std::fs::create_dir_all(member.join("src")).unwrap();
std::fs::write(root.join(".openidea/workspace.json"), "{}").unwrap();
std::fs::write(member.join("src/UserMapper.xml"), "<mapper/>").unwrap();
let api = api(&root, Arc::default());
*api.shared.workspace.write() = Some(Arc::new(Members(root.clone(), vec![member.clone()])));
// The container holds no sources: the member's files are project scope.
assert_eq!(api.find_files(&root, "**/*Mapper.xml", 10).unwrap(), [to_ipc(&member.join("src/UserMapper.xml"))]);
// A member root lists its own files, like any project root.
assert_eq!(api.find_files(&member, "src/*.xml", 10).unwrap(), [to_ipc(&member.join("src/UserMapper.xml"))]);
// And a path outside the group is never listed.
std::fs::write(base.join("OutsideMapper.xml"), "<mapper/>").unwrap();
assert!(api.find_files(&root, "**/*Mapper.xml", 10).unwrap().len() == 1);
}
#[test]
fn process_environment_only_adds_variables() {
let tmp = tempfile::tempdir().unwrap();
@@ -3831,6 +3938,16 @@ fn valid_range(document: &ide_text::Document, range: ide_core::Range) -> bool {
}
impl PluginApi {
/// The repository of `root`, which may execute only when `root` is a trusted project.
/// Status needs it there: a restricted repository is read without its config, so it knows
/// no upstream. The other reads use [`Self::read_repo`] and never execute.
fn repo(&self, root: &Path) -> Result<Option<ide_vcs::Repo>, HostError> {
let repo = ide_vcs::Repo::discover(root).map_err(failed)?;
if let Some(repo) = &repo {
repo.set_execution_allowed(self.shared.workspace().is_some_and(|w| w.trusted(root)));
}
Ok(repo)
}
fn read_repo(&self, root: &Path) -> Result<Option<ide_vcs::Repo>, HostError> {
if !self.shared.workspace().is_some_and(|w| w.trusted(root)) {
return Err(HostError::Failed("the project is in Restricted Mode: git does not run there".into()));
@@ -4095,6 +4212,36 @@ mod workbench_tests {
std::fs::write(&path, "x".repeat(5 << 20)).unwrap();
assert!(api.vcs_diff(&root, None, None, &cancel).is_err());
}
/// Security audit lead 2: status has no trust check to stop it, so in Restricted Mode the
/// repository it opens must not run what its config names.
#[test]
fn vcs_status_runs_no_repository_program_in_restricted_mode() {
let dir = tempfile::tempdir().unwrap();
let root = dunce::canonicalize(dir.path()).unwrap();
git(&root, &["init", "-q"]);
git(&root, &["config", "user.name", "Test"]);
git(&root, &["config", "user.email", "[email protected]"]);
git(&root, &["config", "commit.gpgsign", "false"]);
std::fs::write(root.join("a.txt"), "base\n").unwrap();
git(&root, &["add", "."]);
git(&root, &["commit", "-qm", "base"]);
let marker = dir.path().join("FILTER-RAN");
let clean = format!("echo x > '{}'; cat", marker.to_str().unwrap().replace('\\', "/"));
git(&root, &["config", "filter.marker.clean", &clean]);
std::fs::write(root.join(".gitattributes"), "*.txt filter=marker\n").unwrap();
// Same length as the committed text: only reading the file tells them apart.
std::fs::write(root.join("a.txt"), "BASE\n").unwrap();
let restricted = api(&root, false, None);
let status = restricted.vcs_status(&root);
assert!(!marker.exists(), "status ran the repository's clean filter in Restricted Mode");
assert_eq!(status.unwrap().unwrap().changed_files, 2);
// Control: the same call runs the fixture's filter in a trusted project.
api(&root, true, None).vcs_status(&root).unwrap();
assert!(marker.exists(), "control: a trusted status did not run the clean filter, fixture is not live");
}
#[cfg(unix)]
#[test]
fn vcs_reads_never_run_repository_configured_helpers() {
+9 -2
View File
@@ -27,8 +27,12 @@ pub async fn plugin_navigation_files(
if files.len() > 20_000 {
return Err(IdeError::unavailable("plugin navigation supports up to 20,000 matching files"));
}
// A project window group's members are project scope: their files are
// in the group's index (the group root holds none of them).
let members = crate::project::workspace_member_roots(&project.root);
let in_scope = |p: &PathBuf| p.starts_with(&project.root) || members.iter().any(|m| p.starts_with(m));
Ok(NavigationFiles {
paths: files.into_iter().filter(|p| p.starts_with(&project.root)).map(Into::into).collect(),
paths: files.into_iter().filter(in_scope).map(Into::into).collect(),
indexing,
extensions,
})
@@ -64,11 +68,14 @@ pub async fn plugin_navigation_index(
if !project.is_trusted() {
return Err(IdeError::unavailable("project is not trusted"));
}
// A project window group's members are project scope.
let mut roots = vec![project.root.clone()];
roots.extend(crate::project::workspace_member_roots(&project.root));
for path in paths {
if cancel.is_cancelled() {
return Err(IdeError::cancelled());
}
let Ok(path) = ide_plugin::resolve_in_root(&project.root, &path.to_string_lossy()) else {
let Ok(path) = ide_plugin::resolve_in_roots(&roots, &path.to_string_lossy()) else {
continue;
};
let extensions = plugins.navigation_extensions(&project.root);
+142 -17
View File
@@ -14,6 +14,7 @@ use ide_extension::{MANIFEST_FILE, Manifest};
use semver::Version;
use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf};
use std::time::Duration;
/// The official marketplace: the index the release workflow publishes with
/// the app, next to the signed packages of the official plugins.
@@ -31,6 +32,7 @@ const MAX_STAGED_FILES: usize = 256;
const MAX_STAGED_BYTES: usize = 16 << 20;
/// `moved` links followed before giving up (a loop, or a misconfiguration).
const MAX_MOVES: usize = 3;
const MAX_SUGGESTIONS: usize = 8;
/// A marketplace entry with what is installed.
#[derive(Clone, Debug, Serialize, Deserialize, specta::Type)]
@@ -42,6 +44,9 @@ pub struct MarketPlugin {
pub installed: Option<String>,
/// Whether this IDE can load `entry` (the newest version it can, else the newest).
pub compatible: bool,
/// The publisher's display name, as the index lists it.
#[serde(default)]
pub publisher_name: Option<String>,
}
impl MarketPlugin {
@@ -76,22 +81,42 @@ fn safe_icon(icon: PluginIcon) -> Option<PluginIcon> {
Some(PluginIcon { light: clean(icon.light)?, dark: icon.dark.and_then(clean) })
}
/// Paths an index suggests a plugin for, as the UI may look them up: relative,
/// inside the project, and few.
fn safe_suggestions(paths: Vec<String>) -> Vec<String> {
let ok = |p: &String| {
!p.is_empty()
&& p.len() <= 128
&& !p.starts_with('/')
&& !p.contains(['\\', ':', '\0', '*', '?'])
&& p.split('/').all(|part| !matches!(part, "" | "." | ".."))
};
paths.into_iter().filter(ok).take(MAX_SUGGESTIONS).collect()
}
/// Whether `url` is read over the network: `http(s)://…`, not `file://…` or a local path.
fn remote(url: &str) -> bool {
url.starts_with("https://") || url.starts_with("http://")
}
/// Whether an index read from `base` may send the IDE to `href` (resolved). One read over
/// the network names nothing on this machine: the IDE would read the file for it, or sign
/// in to the share it names.
pub fn reachable_from(base: &str, href: &str) -> bool {
!remote(base) || remote(href)
}
/// The slowest transfer a read over the network waits for, in bytes per second: with
/// `READ_GRACE`, the largest index is given up on after about five minutes, the largest
/// package after a little over an hour.
const SLOWEST_READ: usize = 64 << 10;
/// What a read over the network may take before its first byte.
const READ_GRACE: Duration = Duration::from_secs(30);
/// Reads `url`: `http(s)://…`, `file://…` or a local path.
pub fn read_url(url: &str, max: usize) -> Result<Vec<u8>, String> {
if url.starts_with("https://") || url.starts_with("http://") {
let client = ide_http::client::shared().map_err(|e| e.message)?;
let url = url.to_owned();
return super::api::net_runtime().block_on(async move {
let response = client.get(&url).send().await.map_err(|e| format!("{url}: {e}"))?;
if !response.status().is_success() {
return Err(format!("{url}: HTTP {}", response.status()));
}
let bytes = response.bytes().await.map_err(|e| format!("{url}: {e}"))?;
if bytes.len() > max {
return Err(format!("{url}: larger than {} MiB", max >> 20));
}
Ok(bytes.to_vec())
});
if remote(url) {
return read_remote(url, max, READ_GRACE + Duration::from_secs((max / SLOWEST_READ) as u64));
}
let path = url.strip_prefix("file://").unwrap_or(url);
let bytes = std::fs::read(path).map_err(|e| format!("{path}: {e}"))?;
@@ -101,6 +126,35 @@ pub fn read_url(url: &str, max: usize) -> Result<Vec<u8>, String> {
Ok(bytes)
}
/// Reads `url` over http(s): no more than `max` bytes, whatever the server announces or
/// sends, and within `deadline` from the request to the last byte. The marketplace is read
/// with nobody asking, so its server decides neither how much is kept nor for how long.
fn read_remote(url: &str, max: usize, deadline: Duration) -> Result<Vec<u8>, String> {
let client = ide_http::client::shared().map_err(|e| e.message)?;
super::api::net_runtime().block_on(async move {
let failed = |e: reqwest::Error| match e.is_timeout() {
true => format!("{url}: no answer in time"),
false => format!("{url}: {e}"),
};
let too_large = || format!("{url}: larger than {} MiB", max >> 20);
let mut response = client.get(url).timeout(deadline).send().await.map_err(failed)?;
if !response.status().is_success() {
return Err(format!("{url}: HTTP {}", response.status()));
}
if response.content_length().is_some_and(|announced| announced > max as u64) {
return Err(too_large());
}
let mut bytes = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(failed)? {
if bytes.len() + chunk.len() > max {
return Err(too_large());
}
bytes.extend_from_slice(&chunk);
}
Ok(bytes)
})
}
/// The index at `url`, following `moved`, with downloads made absolute against
/// where it was read from.
fn read_index(url: &str) -> IdeResult<Index> {
@@ -115,12 +169,25 @@ fn read_index(url: &str) -> IdeResult<Index> {
if url.starts_with("https://") && !next.starts_with("https://") {
return Err(IdeError::invalid(format!("the marketplace moved to {next}, which isn't https")));
}
if !reachable_from(&url, &next) {
return Err(IdeError::invalid("the marketplace moved to an address that is not on the network"));
}
url = next;
}
None => {
for e in &mut index.plugins {
e.download = resolve(&url, &e.download);
}
// Judged here, against where this index was read: the address in use may be
// an index on this machine that moved to it.
let listed = index.plugins.len();
index.plugins.retain(|e| reachable_from(&url, &e.download));
if index.plugins.len() < listed {
tracing::warn!(
"plugin marketplace: {} downloads are not on the network; they are not offered",
listed - index.plugins.len()
);
}
return Ok(index);
}
}
@@ -253,7 +320,9 @@ impl PluginHost {
.map(|p| p.manifest().version.clone());
let mut entry = entry.clone();
entry.icon = entry.icon.and_then(safe_icon);
MarketPlugin { entry, installed, compatible }
entry.suggest = safe_suggestions(std::mem::take(&mut entry.suggest));
let publisher_name = index.publishers.iter().find(|p| p.id == entry.publisher).map(|p| p.name.clone());
MarketPlugin { entry, installed, compatible, publisher_name }
})
.collect()
}
@@ -276,13 +345,23 @@ impl PluginHost {
entry.name, entry.version, entry.engine
)));
}
let bytes = read_url(&resolve(&url, &entry.download), MAX_PACKAGE).map_err(IdeError::unavailable)?;
let download = resolve(&url, &entry.download);
// An index read by this version holds only downloads it may reach. The one kept by an
// earlier version does not say where it was read: it is judged against the marketplace in use.
if !reachable_from(&url, &download) {
return Err(IdeError::invalid(format!(
"the marketplace's download of {} {} is not on the network",
entry.id, entry.version
)));
}
let bytes = read_url(&download, MAX_PACKAGE).map_err(IdeError::unavailable)?;
let downloads = self.plugins_dir.join(DOWNLOADS);
std::fs::create_dir_all(&downloads)?;
let archive = downloads.join(format!("{}-{}.{}", entry.id, entry.version, package::PACKAGE_EXTENSION));
std::fs::write(&archive, bytes)?;
let dir = self.unpack_package(&archive)?;
let dir = self.unpack_package(&archive);
let _ = std::fs::remove_file(&archive);
let dir = dir?;
let manifest = Manifest::load(&dir).map_err(IdeError::invalid)?;
let files = self.package_files_in(&dir)?;
let digest = package::package_digest(&dir, &files)?;
@@ -482,4 +561,50 @@ mod tests {
assert_eq!(resolve(r"C:\market\index.json", "a.oiplugin"), r"C:\market\a.oiplugin");
assert_eq!(resolve("https://x.dev/index.json", "https://cdn.dev/a"), "https://cdn.dev/a");
}
#[test]
fn suggestions_stay_inside_the_project() {
let paths = ["pom.xml", "config/routes.rb", "/etc/passwd", "../x", "a//b", "C:x", "*.php", ""];
assert_eq!(safe_suggestions(paths.iter().map(|p| p.to_string()).collect()), ["pom.xml", "config/routes.rb"]);
assert_eq!(safe_suggestions(vec!["a".into(); 20]).len(), MAX_SUGGESTIONS);
}
#[test]
fn an_index_read_over_the_network_reaches_only_the_network() {
let base = "https://x.dev/market/index.json";
let reaches = |href: &str| reachable_from(base, &resolve(base, href));
for href in ["p/a.oiplugin", "../a.oiplugin", "https://cdn.dev/a"] {
assert!(reaches(href), "{href}");
}
for href in ["/etc/passwd", "//attacker/share/a", "file:///etc/passwd", "file://attacker/share/a"] {
assert!(!reaches(href), "{href}");
}
#[cfg(windows)]
for href in [r"C:\Users\me\.ssh\id_ed25519", r"\\attacker\share\a", r"\\?\UNC\attacker\share\a"] {
assert!(!reaches(href), "{href}");
}
// A local index names what whoever put it there chose.
assert!(reachable_from("/srv/market/index.json", "/srv/market/a.oiplugin"));
}
#[test]
fn a_server_that_stops_answering_does_not_hold_the_read() {
use std::io::{BufRead, BufReader, Write};
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let url = format!("http://127.0.0.1:{}/index.json", listener.local_addr().unwrap().port());
std::thread::spawn(move || {
for mut stream in listener.incoming().flatten() {
let mut reader = BufReader::new(stream.try_clone().unwrap());
let mut line = String::new();
while reader.read_line(&mut line).unwrap_or(0) > 0 && line != "\r\n" {
line.clear();
}
// The headers, then nothing for longer than the read waits.
let _ = stream.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\n");
std::thread::sleep(Duration::from_secs(5));
}
});
let err = read_remote(&url, MAX_INDEX, Duration::from_millis(300)).unwrap_err();
assert!(err.contains("no answer in time"), "{err}");
}
}
+1 -1
View File
@@ -63,7 +63,7 @@ pub use icons::{
pub(crate) use inlay_hints::merge as merge_inlay_hints;
pub use market::{
INDEX_FILE as MARKET_INDEX_FILE, MarketPlugin, OFFICIAL_INDEX as OFFICIAL_MARKET_INDEX, OFFICIAL_PUBLISHER,
read_url as read_market_url, resolve as resolve_market_href,
reachable_from as market_href_reachable_from, read_url as read_market_url, resolve as resolve_market_href,
};
pub use points::{COMMANDS, CommandDecl, register_builtin_points};
pub use providers::{ProviderKind, set_editor_providers};
+5 -1
View File
@@ -204,7 +204,11 @@ impl PluginHost {
if text.len() > MAX_DOCUMENT_BYTES {
return Err(IdeError::invalid("navigation document is too large"));
}
let real = ide_plugin::resolve_in_root(root, &path.to_string_lossy()).map_err(IdeError::invalid)?;
// A project window group's members are project scope: a member's file
// is indexed into the group's graph like any file of the group root.
let mut roots = vec![root.to_path_buf()];
roots.extend(crate::project::workspace_member_roots(root));
let real = ide_plugin::resolve_in_roots(&roots, &path.to_string_lossy()).map_err(IdeError::invalid)?;
if real != path {
return Err(IdeError::invalid("navigation requires a canonical project path"));
}
Loaded 100 of 411 files, more files were not shown because too many files have changed in this diff. Show more