sharkdp/hyperfine
1
0
Fork 0
mirror of https://github.com/sharkdp/hyperfine.git synced 2026-10-12 02:22:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Alternative performance metrics in the terminal (#986)

Browse Source
This commit is contained in:
David Peter
2026-10-06 15:50:19 +02:00
committed by GitHub
parent 3b64ea1bf5
commit bdbfae9907
43 changed files with 1872 additions and 2056 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+10
View File
@@ -1,5 +1,11 @@
# Unreleased
## Features
- Add support for alternative performance metrics (peak memory usage, CPU cycles, instructions, cache misses, branch misses, ...). By default, hyperfine will now display wall-clock time and peak memory usage,
but users can use the new `--metrics` option to select different performance metrics.
- The terminal output now shows an overview of all measured performance metrics, including relative changes.
## Breaking changes
- Benchmarked commands are now always executed directly, without an intermediate shell (`--shell=none`). Use `-S` (an alias for `--shell=default`) to restore the previous behavior (`sh` on Unix, `cmd.exe` on Windows), or select a shell with `--shell <SHELL>`.
@@ -46,6 +52,10 @@
```
Times are always exported in seconds and memory in bytes.
- `--time-unit`/`-u` has been removed. Units can now be selected using the `--metrics METRIC[:UNIT],…` option, e.g. `--metrics time_wall_clock:ms,memory_peak_resident:MiB`.
- `--sort` has been removed. It complicated hyperfine significantly and doesn't make too much sense with the new output format.
- `--reference` and `--reference-name` have been removed (for now). The first command is always considered as the reference command.
- The format of Markdown, AsciiDoc, org-mode and CSV exports has also been changed. CSV now includes columns with the name and the unit of the primary metric.
# v1.21.0
README.md
+61 -24
View File
@@ -16,7 +16,7 @@ A command-line benchmarking tool.
* Support for arbitrary shell commands.
* Constant feedback about the benchmark progress and current estimates.
* Warmup runs can be executed before the actual benchmark.
* Cache-clearing commands can be set up before each timing run.
* Cache-clearing commands can be set up before each benchmark run.
* Statistical outlier detection to detect interference from other programs and caching effects.
* Export results to various formats: CSV, JSON, Markdown, AsciiDoc.
* Parameterized benchmarks (e.g. vary the number of threads).
@@ -41,11 +41,14 @@ number of runs, you can use the `-r`/`--runs` option:
hyperfine --runs 5 'sleep 0.3'
```
If you want to compare the runtimes of different programs, you can pass multiple commands:
If you want to compare different programs, you can pass multiple commands:
```sh
hyperfine 'hexdump file' 'xxd file'
```
Commands run and appear in input order. The first command is the reference for all comparisons;
each subsequent result shows its change from that reference.
### Warmup runs and preparation commands
For programs that perform a lot of disk I/O, the benchmarking results can be heavily influenced
@@ -58,7 +61,7 @@ hyperfine -S --warmup 3 'grep -R TODO *'
```
Conversely, if you want to run the benchmark for a cold cache, you can use the `-p`/`--prepare`
option to run a special command before *each* timing run. For example, to clear Linux filesystem caches,
option to run a special command before *each* benchmark run. For example, to clear Linux filesystem caches,
you can run
```sh
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
@@ -90,7 +93,7 @@ hyperfine -L compiler g++,clang++ '{compiler} -O2 main.cpp'
```
A common use case is comparing the same command across multiple Git branches. Use `--setup`
to switch branches once before each set of timing runs, so the branch switch is not part of
to switch branches once before each set of benchmark runs, so the branch switch is not part of
the measured command:
```sh
hyperfine \
@@ -129,7 +132,7 @@ If any of these commands need shell syntax, enable a shell explicitly.
When a shell is enabled, hyperfine *corrects for the shell spawning time*. It runs the shell with an
empty command multiple times to measure its startup time, then subtracts this time from each
measurement.
time measurement.
### Shell functions
@@ -148,38 +151,40 @@ Otherwise, inline the function into the benchmarked command:
hyperfine -S 'my_function() { sleep 1; }; my_function'
```
### Exporting results
### Choosing metrics and units
Hyperfine has multiple options for exporting benchmark results to CSV, JSON, Markdown and other
formats (see `--help` text for details).
By default, hyperfine displays wall-clock time and memory usage
(`time_wall_clock,memory_peak_resident`). Use `--metrics` with a
comma-separated list to select a different set of metrics. Each
metric can have an optional unit after it:
#### Markdown
```sh
hyperfine \
--metrics memory_peak_resident:MiB,time_wall_clock:ms,instructions \
'./baseline' './candidate'
```
You can use the `--export-markdown <file>` option to create tables like the following:
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `find . -iregex '.*[0-9]\.jpg$'` | 2.275 ± 0.046 | 2.243 | 2.397 | 9.79 ± 0.22 |
| `find . -iname '*[0-9].jpg'` | 1.427 ± 0.026 | 1.405 | 1.468 | 6.14 ± 0.13 |
| `fd -HI '.*[0-9]\.jpg$'` | 0.232 ± 0.002 | 0.230 | 0.236 | 1.00 |
#### JSON
The JSON export includes the following metrics for each measured run (excluding warmup runs):
The following metrics are available:
- **`time_wall_clock`**: Time from start to finish, including time spent waiting, in seconds.
- **`time_user`**: CPU time spent running the program's code, summed across threads, in seconds.
- **`time_user`**: CPU time spent executing application and library code in user mode,
summed across threads, in seconds.
- Linux/macOS: Includes child-process time when parents wait for their children to finish.
- Windows: Includes the command and its child processes.
- **`time_system`**: CPU time spent running operating-system code for the program, for example
- **`time_system`**: CPU time spent executing kernel code on the program's behalf, for example
to read files, summed across threads, in seconds.
- Linux/macOS: Includes child-process time when parents wait for their children to finish.
- Windows: Includes the command and its child processes.
- **`time_cpu`**: Total CPU time, calculated as `time_user + time_system` for each run, in seconds.
It excludes time spent sleeping or waiting without executing. CPU time is summed across
threads, so it can exceed wall-clock time: four threads running on four cores for one second
can consume roughly four CPU-seconds.
- **`memory_peak_resident`**: Peak memory held in physical RAM, in bytes.
- Linux/macOS: Peak resident set size (RSS). This is the largest per-process peak among the
@@ -215,7 +220,39 @@ The JSON export includes the following metrics for each measured run (excluding
- Linux: Same scope as `cpu_cycles`.
- macOS/Windows: Currently not supported.
Hardware counters are not available if a `--shell` is used.
Note that hardware counters are not available if a `--shell` is used.
Time measurements support units `ns`, `us`, `ms`, `s`, `min`, and `h`. Memory measurements
support `B`, `kB`, `MB`, `GB`, `TB`, `KiB`, `MiB`, `GiB`, and `TiB`. Hardware counters
support `count` (1), `k` (thousand), `M` (million), and `B` (billion).
You can also use a preset to select a group of metrics:
| Preset | Metrics |
|:---|:---|
| `--metrics=default` | `time_wall_clock,memory_peak_resident` |
| `--metrics=time` | `time_wall_clock,time_cpu,time_user,time_system` |
| `--metrics=all` | All available metrics |
### Exporting results
Hyperfine can export results to CSV, JSON, Markdown, AsciiDoc, and org-mode. Non-JSON formats
contain only the primary metric.
#### Markdown
You can use the `--export-markdown <file>` option to create tables like the following:
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `find . -iregex '.*[0-9]\.jpg$'` | 2.275 ± 0.046 | | |
| `find . -iname '*[0-9].jpg'` | 1.427 ± 0.026 | -37.3% | (1.6x faster) |
| `fd -HI '.*[0-9]\.jpg$'` | 0.232 ± 0.002 | -89.8% | (9.8x faster) |
#### JSON
The JSON export includes all available metrics listed in [Choosing metrics and units](#choosing-metrics-and-units)
for each measured run (excluding warmup runs).
The JSON output is useful if you want to analyze the benchmark results in more detail. The
[`scripts/`](https://github.com/sharkdp/hyperfine/tree/master/scripts) folder includes a lot
@@ -229,7 +266,7 @@ multiple benchmarks:
### Detailed benchmark flowchart
The following chart explains the execution order of various timing runs when using options
The following chart explains the execution order of various benchmark runs when using options
like `--warmup`, `--prepare <cmd>`, `--setup <cmd>` or `--cleanup <cmd>`:
![](doc/execution-order.png)
doc/hyperfine.1
+43 -66
View File
@@ -15,10 +15,6 @@ hyperfine \- command\-line benchmarking tool
.IR NUM ]
.RB [ \-\-setup
.IR CMD ]
.RB [ \-\-reference
.IR CMD ]
.RB [ \-\-reference\-name
.IR NAME ]
.RB [ \-\-prepare
.IR CMD ]
.RB [ \-\-conclude
@@ -38,10 +34,8 @@ hyperfine \- command\-line benchmarking tool
.IR SHELL ]
.RB [ \-\-style
.IR TYPE ]
.RB [ \-\-sort
.IR METHOD ]
.RB [ \-\-time-unit
.IR UNIT ]
.RB [ \-\-metrics
.IR METRIC[:UNIT],...|default|time|all ]
.RB [ \-\-export\-asciidoc
.IR FILE ]
.RB [ \-\-export\-csv
@@ -75,7 +69,7 @@ A command\-line benchmarking tool which includes:
* Warmup runs can be executed before the actual benchmark
.RE
.RS
* Cache-clearing commands can be set up before each timing run
* Cache-clearing commands can be set up before each benchmark run
.RE
.RS
* Statistical outlier detection to detect interference from other programs and caching effects
@@ -109,28 +103,14 @@ Perform exactly \fINUM\fP runs for each command. If this option is not specified
.HP
\fB\-s\fR, \fB\-\-setup\fR \fICMD...\fP
.IP
Execute \fICMD\fP once before each set of timing runs. This is useful
Execute \fICMD\fP once before each set of benchmark runs. This is useful
for compiling your software with the provided parameters, or to do any
other work that should happen once before a series of benchmark runs,
not every time as would happen with the \fB\-\-prepare\fR option.
.HP
\fB\-\-reference\fR \fICMD\fP
.IP
Use \fICMD\fP as the reference for the relative comparison of results. Without
parameters, benchmark \fICMD\fP as a separate reference command. If this option
is not specified, the fastest command is used as the reference.
With \fB\-\-parameter\-scan\fR or \fB\-\-parameter\-list\fR, select an existing
benchmark by its exact printed name. The name must match exactly one benchmark.
.HP
\fB\-\-reference\-name\fR \fINAME\fP
.IP
Give a meaningful \fINAME\fP to the reference command.
Requires \fB\-\-reference\fR.
Cannot be used with parameterized benchmarks; use \fB\-\-command\-name\fR instead.
.HP
\fB\-p\fR, \fB\-\-prepare\fR \fICMD...\fP
.IP
Execute \fICMD\fP before each timing run. This is useful for clearing disk caches,
Execute \fICMD\fP before each benchmark run. This is useful for clearing disk caches,
for example.
The \fB\-\-prepare\fR option can be specified once for all commands or multiple times,
once for each command. In the latter case, each preparation command will be
@@ -138,7 +118,7 @@ run prior to the corresponding benchmark command.
.HP
\fB\-C\fR, \fB\-\-conclude\fR \fICMD...\fP
.IP
Execute \fICMD\fP after each timing run. This is useful for clearing disk caches,
Execute \fICMD\fP after each benchmark run. This is useful for clearing disk caches,
for example.
The \fB\-\-conclude\fR option can be specified once for all commands or multiple times,
once for each command. In the latter case, each conclusion command will be
@@ -248,65 +228,62 @@ interactive output without any colors. Set this to 'color' to keep the colors
without any interactive output. Set this to 'none' to disable all the output
of the tool.
.HP
\fB\-\-sort\fR \fIMETHOD\fP
\fB\-\-metrics\fR \fIMETRIC[:UNIT],...|default|time|all\fP
.IP
Specify the sort order of the speed comparison summary and the
exported tables for markup formats (Markdown, AsciiDoc, org\-mode):
Select the metrics to display, in order, with an optional unit for each.
Default: \fBtime_wall_clock,memory_peak_resident\fR, or \fBtime_wall_clock\fR
on Windows, where peak resident memory is not yet supported.
.IP
Use \fBdefault\fR for the platform defaults, \fBtime\fR for
\fBtime_wall_clock,time_cpu,time_user,time_system\fR, or \fBall\fR for all available
metrics in the order listed below.
.IP
Time metrics \fBtime_wall_clock\fR, \fBtime_cpu\fR, \fBtime_user\fR, and \fBtime_system\fR
accept units \fBns\fR, \fBus\fR, \fBms\fR, \fBs\fR, \fBmin\fR, and \fBh\fR.
Peak memory, \fBmemory_peak_resident\fR, accepts \fBB\fR, \fBkB\fR,
\fBMB\fR, \fBGB\fR, \fBTB\fR, \fBKiB\fR, \fBMiB\fR, \fBGiB\fR,
and \fBTiB\fR.
.IP
Counter metrics \fBcpu_cycles\fR, \fBinstructions\fR, \fBcache_references\fR,
\fBcache_misses\fR, and \fBbranch_misses\fR accept \fBcount\fR (unscaled),
\fBk\fR (thousand), \fBM\fR (million), and \fBB\fR (billion).
.IP
If a unit is omitted, terminal and markup output automatically select a suitable
unit.
.RS
.IP "auto (default)"
the speed comparison will be ordered by time and
the markup tables will be ordered by command (input order).
.IP "command"
order benchmarks in the way they were specified
.IP "mean\-time"
order benchmarks by mean runtime
.RE
.HP
\fB\-u\fR, \fB\-\-time\-unit\fR \fIUNIT\fP
.IP
Set the time unit to be used. If the option is not given, the time unit is
determined automatically. Possible values:
Example:
.RS
.IP "microsecond"
Aliases: µs, us, microseconds.
.IP "millisecond"
Aliases: ms, milliseconds.
.IP "second"
Aliases: s, seconds.
.IP "minute"
Aliases: min, minutes.
.IP "hour"
Aliases: h, hours.
\fBhyperfine\fR \fB\-\-metrics\fR memory_peak_resident:MiB,time_wall_clock:ms,instructions:B './baseline' './candidate'
.RE
.RE
.IP
This option affects the standard output as well as all export formats
except for CSV and JSON.
.HP
\fB\-\-export\-asciidoc\fR \fIFILE\fP
.IP
Export the timing summary statistics as an AsciiDoc table to the given \fIFILE\fP.
The output time unit can be changed using the \fB\-\-time\-unit\fR option.
Export the primary metric's summary statistics as an AsciiDoc table to the given
\fIFILE\fP. Use its selected unit, or choose a unit automatically when omitted.
.HP
\fB\-\-export\-csv\fR \fIFILE\fP
.IP
Export the timing summary statistics as CSV to the given \fIFILE\fP. If you need the
timing results for each individual run, use the JSON export format.
The output time unit is always seconds.
Export the primary metric's summary statistics as CSV to the given \fIFILE\fP,
identifying the metric and unit explicitly. Use the explicitly selected unit,
or the base unit (seconds, bytes, or unscaled counts) when omitted.
.HP
\fB\-\-export\-json\fR \fIFILE\fP
.IP
Export the timing summary statistics and timings of individual runs as JSON to
the given \fIFILE\fP. The output time unit is always seconds.
Export summary statistics and individual samples for all collected metrics as
JSON to the given \fIFILE\fP. Include the primary metric, and always use base
units (seconds, bytes, or unscaled counts), regardless of display settings.
.HP
\fB\-\-export\-markdown\fR \fIFILE\fP
.IP
Export the timing summary statistics as a Markdown table to the given \fIFILE\fP.
The output time unit can be changed using the \fB\-\-time\-unit\fR option.
Export the primary metric's summary statistics as a Markdown table to the given
\fIFILE\fP. Use its selected unit, or choose a unit automatically when omitted.
.HP
\fB\-\-export\-orgmode\fR \fIFILE\fP
.IP
Export the timing summary statistics as an Emacs org\-mode table to the
given \fIFILE\fP. The output time unit can be changed using the \fB\-\-time\-unit\fR option.
Export the primary metric's summary statistics as an Emacs org\-mode table to
the given \fIFILE\fP. Use its selected unit, or choose a unit automatically
when omitted.
.HP
\fB\-\-show\-output\fR
.IP
scripts/README.md
+1 -3
View File
@@ -7,9 +7,7 @@ hyperfine 'sleep 0.020' 'sleep 0.021' 'sleep 0.022' --export-json sleep.json
uv run plot_whisker.py sleep.json
```
All plotting scripts accept `--metric` to select a metric. Hardware counters
(`cpu_cycles`, `instructions`, `cache_references`, `cache_misses`, and `branch_misses`)
are plotted as unscaled counts when available in the JSON export. For example:
All plotting and analysis scripts use the JSON export's `primary_metric` by default.
```bash
uv run plot_histogram.py benchmark.json --metric instructions
scripts/advanced_statistics.py
+27 -45
View File
@@ -7,74 +7,56 @@
# ///
import argparse
import json
from enum import Enum
import numpy as np
class Unit(Enum):
SECOND = 1
MILLISECOND = 2
def factor(self):
match self:
case Unit.SECOND:
return 1
case Unit.MILLISECOND:
return 1e3
def __str__(self):
match self:
case Unit.SECOND:
return "s"
case Unit.MILLISECOND:
return "ms"
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser()
parser.add_argument("file", help="JSON file with benchmark results")
add_metric_argument(parser)
parser.add_argument(
"--time-unit",
help="The unit of time.",
default="second",
action="store",
help="Display unit for time metrics (default: second).",
choices=["second", "millisecond"],
dest="unit",
)
args = parser.parse_args()
unit = Unit.MILLISECOND if args.unit == "millisecond" else Unit.SECOND
unit_str = str(unit)
metric, [results] = load_results(parser, [args.file], args.metric)
description = METRICS[metric]
unit, scale = description.unit, description.scale
if args.time_unit is not None:
if not metric.startswith("time_"):
parser.error("--time-unit can only be used with a time metric")
if args.time_unit == "millisecond":
unit, scale = "ms", 0.001
with open(args.file) as f:
results = json.load(f)["results"]
def format_value(value):
suffix = f" {unit}" if unit else ""
return f"{value:.3f}{suffix}"
print(f"Metric: {description.label}\n")
for result in results:
summary = result["summary"]["time_wall_clock"]
ts = [m["time_wall_clock"]["value"] * unit.factor() for m in result["measurements"]]
summary = result["summary"][metric]
values = [m[metric]["value"] / scale for m in result["measurements"]]
p05 = np.percentile(ts, 5)
p25 = np.percentile(ts, 25)
p75 = np.percentile(ts, 75)
p95 = np.percentile(ts, 95)
p05, p25, p75, p95 = np.percentile(values, [5, 25, 75, 95])
iqr = p75 - p25
print(f"Command '{result.get('name', result['command'])}'")
print(f" runs: {summary['count']:8d}")
print(f" mean: {summary['mean'] * unit.factor():8.3f} {unit_str}")
if summary["stddev"] is None:
print(f" stddev: {'N/A':>8}")
else:
print(f" stddev: {summary['stddev'] * unit.factor():8.3f} {unit_str}")
print(f" median: {summary['median'] * unit.factor():8.3f} {unit_str}")
print(f" min: {summary['min'] * unit.factor():8.3f} {unit_str}")
print(f" max: {summary['max'] * unit.factor():8.3f} {unit_str}")
for statistic in ["mean", "stddev", "median", "min", "max"]:
value = summary[statistic]
formatted = "N/A" if value is None else format_value(value / scale)
print(f" {statistic + ':':<8}{formatted:>8}")
print()
print(" percentiles:")
print(f" P_05 .. P_95: {p05:.3f} {unit_str} .. {p95:.3f} {unit_str}")
print(f" P_05 .. P_95: {format_value(p05)} .. {format_value(p95)}")
print(
f" P_25 .. P_75: {p25:.3f} {unit_str} .. {p75:.3f} {unit_str} (IQR = {iqr:.3f} {unit_str})"
f" P_25 .. P_75: {format_value(p25)} .. {format_value(p75)}"
f" (IQR = {format_value(iqr)})"
)
print()
scripts/plot_benchmark_comparison.py
+6 -9
View File
@@ -14,13 +14,12 @@ Note all the input files must contain results for all commands.
"""
import argparse
import json
import pathlib
import matplotlib.pyplot as plt
import numpy as np
from plot_utils import METRICS, add_metric_argument, validate_metric
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser(description=__doc__)
add_metric_argument(parser)
@@ -34,7 +33,8 @@ parser.add_argument(
parser.add_argument("-o", "--output", help="Save image to the given filename")
args = parser.parse_args()
metric_label, metric_scale = METRICS[args.metric]
metric, datasets = load_results(parser, args.files, args.metric)
metric_info = METRICS[metric]
commands = None
data = []
@@ -45,10 +45,7 @@ if args.benchmark_names:
"Number of benchmark names must match the number of input files."
)
for i, filename in enumerate(args.files):
with open(filename) as f:
results = json.load(f)["results"]
validate_metric(parser, results, args.metric)
for i, (filename, results) in enumerate(zip(args.files, datasets)):
benchmark_commands = [b.get("name", b["command"]) for b in results]
if commands is None:
commands = benchmark_commands
@@ -56,7 +53,7 @@ for i, filename in enumerate(args.files):
assert commands == benchmark_commands, (
f"Unexpected commands in {filename}: {benchmark_commands}, expected: {commands}"
)
data.append([b["summary"][args.metric]["mean"] / metric_scale for b in results])
data.append([b["summary"][metric]["mean"] / metric_info.scale for b in results])
if args.benchmark_names:
inputs.append(args.benchmark_names[i])
else:
@@ -79,7 +76,7 @@ ax.grid(visible=True, axis="y")
if args.title:
plt.title(args.title)
plt.xlabel("Benchmark")
plt.ylabel(metric_label)
plt.ylabel(metric_info.axis_label)
plt.legend(title="Command")
if args.output:
scripts/plot_histogram.py
+5 -9
View File
@@ -11,12 +11,11 @@
"""This program shows `hyperfine` benchmark results as a histogram."""
import argparse
import json
import matplotlib.pyplot as plt
import numpy as np
from plot_utils import METRICS, add_metric_argument, validate_metric
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser(description=__doc__)
add_metric_argument(parser)
@@ -70,18 +69,15 @@ parser.add_argument(
)
args = parser.parse_args()
metric_label, metric_scale = METRICS[args.metric]
with open(args.file) as f:
results = json.load(f)["results"]
validate_metric(parser, results, args.metric)
metric, [results] = load_results(parser, [args.file], args.metric)
metric_info = METRICS[metric]
if args.labels:
labels = args.labels.split(",")
else:
labels = [b.get("name", b["command"]) for b in results]
all_values = [
[m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results
[m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results
]
value_min = (
@@ -113,7 +109,7 @@ plt.legend(
prop={"size": 10, "family": ["Source Code Pro", "Fira Mono", "Courier New"]},
)
plt.xlabel(metric_label)
plt.xlabel(metric_info.axis_label)
if args.title:
plt.title(args.title)
scripts/plot_parametrized.py
+8 -12
View File
@@ -11,12 +11,11 @@
errorbar plot."""
import argparse
import json
import sys
import matplotlib.pyplot as plt
from plot_utils import METRICS, add_metric_argument, validate_metric
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser(description=__doc__)
add_metric_argument(parser)
@@ -43,7 +42,8 @@ parser.add_argument(
parser.add_argument("-o", "--output", help="Save image to the given filename.")
args = parser.parse_args()
metric_label, metric_scale = METRICS[args.metric]
metric, datasets = load_results(parser, args.file, args.metric)
metric_info = METRICS[metric]
if args.parameter_name is not None:
sys.stderr.write(
"warning: --parameter-name is deprecated; names are inferred from "
@@ -84,11 +84,7 @@ def unique_parameter(benchmark):
parameter_name = None
for filename in args.file:
with open(filename) as f:
results = json.load(f)["results"]
validate_metric(parser, results, args.metric)
for results in datasets:
(this_parameter_name, parameter_values) = extract_parameters(results)
if parameter_name is not None and this_parameter_name != parameter_name:
die(
@@ -96,17 +92,17 @@ for filename in args.file:
)
parameter_name = this_parameter_name
summaries = [b["summary"][args.metric] for b in results]
means = [s["mean"] / metric_scale for s in summaries]
summaries = [b["summary"][metric] for b in results]
means = [s["mean"] / metric_info.scale for s in summaries]
stddevs = [
s["stddev"] / metric_scale if s["stddev"] is not None else float("nan")
s["stddev"] / metric_info.scale if s["stddev"] is not None else float("nan")
for s in summaries
]
plt.errorbar(x=parameter_values, y=means, yerr=stddevs, capsize=2)
plt.xlabel(parameter_name)
plt.ylabel(metric_label)
plt.ylabel(metric_info.axis_label)
if args.log_y:
plt.yscale("log")
scripts/plot_progression.py
+5 -9
View File
@@ -14,12 +14,11 @@ thermal throttling and similar effects.
"""
import argparse
import json
import matplotlib.pyplot as plt
import numpy as np
from plot_utils import METRICS, add_metric_argument, validate_metric
from plot_utils import METRICS, add_metric_argument, load_results
def moving_average(values, num_runs):
@@ -50,15 +49,12 @@ parser.add_argument(
args = parser.parse_args()
metric_label, metric_scale = METRICS[args.metric]
with open(args.file) as f:
results = json.load(f)["results"]
validate_metric(parser, results, args.metric)
metric, [results] = load_results(parser, [args.file], args.metric)
metric_info = METRICS[metric]
for result in results:
label = result.get("name", result["command"])
values = [m[args.metric]["value"] / metric_scale for m in result["measurements"]]
values = [m[metric]["value"] / metric_info.scale for m in result["measurements"]]
num = len(values)
nums = range(num)
@@ -86,7 +82,7 @@ for result in results:
legend.append("moving average")
plt.legend(legend)
plt.ylabel(metric_label)
plt.ylabel(metric_info.axis_label)
if args.output:
plt.savefig(args.output)
scripts/plot_utils.py
+54 -12
View File
@@ -1,15 +1,30 @@
"""Shared metric options for the plotting scripts."""
"""Shared metric selection and units for analysis and plotting scripts."""
import json
from typing import NamedTuple
class Metric(NamedTuple):
label: str
unit: str = ""
scale: float = 1
@property
def axis_label(self):
return f"{self.label} [{self.unit}]" if self.unit else self.label
METRICS = {
"time_wall_clock": ("Time [s]", 1),
"time_user": ("User CPU time [s]", 1),
"time_system": ("System CPU time [s]", 1),
"memory_peak_resident": ("Peak resident memory [MiB]", 1024**2),
"cpu_cycles": ("CPU cycles", 1),
"instructions": ("Completed CPU instructions", 1),
"cache_references": ("Cache references", 1),
"cache_misses": ("Cache misses", 1),
"branch_misses": ("Mispredicted branches", 1),
"time_wall_clock": Metric("Wall-clock time", "s"),
"time_cpu": Metric("Total CPU time", "s"),
"time_user": Metric("User CPU time", "s"),
"time_system": Metric("System CPU time", "s"),
"memory_peak_resident": Metric("Peak resident memory", "MiB", 1024**2),
"cpu_cycles": Metric("CPU cycles"),
"instructions": Metric("Completed CPU instructions"),
"cache_references": Metric("Cache references"),
"cache_misses": Metric("Cache misses"),
"branch_misses": Metric("Mispredicted branches"),
}
@@ -17,8 +32,8 @@ def add_metric_argument(parser):
parser.add_argument(
"--metric",
choices=METRICS,
default="time_wall_clock",
help="Metric to plot (default: time_wall_clock). "
help="Metric to analyze (default: primary_metric from the JSON export, "
"or time_wall_clock if unspecified). "
"Times are in seconds; memory is in MiB; hardware counters are unscaled counts.",
)
@@ -30,3 +45,30 @@ def validate_metric(parser, results, metric):
f"metric {metric!r} is unavailable for "
f"{result.get('name', result['command'])!r}"
)
def load_results(parser, filenames, metric=None):
exports = []
for filename in filenames:
with open(filename, encoding="utf-8") as f:
export = json.load(f)
if not export["results"]:
parser.error(f"no benchmark results in {filename}")
exports.append(export)
if metric is None:
primary_metrics = {
export.get("primary_metric", "time_wall_clock") for export in exports
}
if len(primary_metrics) != 1:
parser.error(
"input files use different primary metrics; select one with --metric"
)
metric = primary_metrics.pop()
if metric not in METRICS:
parser.error(f"unknown metric {metric!r}; select one with --metric")
results = [export["results"] for export in exports]
for benchmarks in results:
validate_metric(parser, benchmarks, metric)
return metric, results
scripts/plot_whisker.py
+6 -10
View File
@@ -16,11 +16,10 @@ Quoting from the matplotlib documentation:
"""
import argparse
import json
import matplotlib.pyplot as plt
from plot_utils import METRICS, add_metric_argument, validate_metric
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser(description=__doc__)
add_metric_argument(parser)
@@ -33,22 +32,19 @@ parser.add_argument(
parser.add_argument("-o", "--output", help="Save image to the given filename.")
args = parser.parse_args()
metric_label, metric_scale = METRICS[args.metric]
with open(args.file, encoding="utf-8") as f:
results = json.load(f)["results"]
validate_metric(parser, results, args.metric)
metric, [results] = load_results(parser, [args.file], args.metric)
metric_info = METRICS[metric]
if args.labels:
labels = args.labels.split(",")
else:
labels = [b.get("name", b["command"]) for b in results]
values = [
[m[args.metric]["value"] / metric_scale for m in b["measurements"]] for b in results
[m[metric]["value"] / metric_info.scale for m in b["measurements"]] for b in results
]
if args.sort_by == "median":
medians = [b["summary"][args.metric]["median"] for b in results]
medians = [b["summary"][metric]["median"] for b in results]
indices = sorted(range(len(labels)), key=lambda k: medians[k])
labels = [labels[i] for i in indices]
values = [values[i] for i in indices]
@@ -64,7 +60,7 @@ for patch, color in zip(boxplot["boxes"], colors):
if args.title:
plt.title(args.title)
plt.legend(handles=boxplot["boxes"], labels=labels, loc="best", fontsize="medium")
plt.ylabel(metric_label)
plt.ylabel(metric_info.axis_label)
plt.ylim(0, None)
plt.xticks(list(range(1, len(labels) + 1)), labels, rotation=45)
if args.output:
scripts/welch_ttest.py
+15 -10
View File
@@ -7,32 +7,37 @@
# ///
"""This script performs Welch's t-test on a JSON export file with two
benchmark results to test for a difference in their population mean runtimes."""
benchmark results to test for a difference in their population means."""
import argparse
import json
import sys
import math
from scipy import stats
from plot_utils import METRICS, add_metric_argument, load_results
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("file", help="JSON file with two benchmark results")
add_metric_argument(parser)
args = parser.parse_args()
with open(args.file) as f:
results = json.load(f)["results"]
metric, [results] = load_results(parser, [args.file], args.metric)
if len(results) != 2:
print("The input file has to contain exactly two benchmarks")
sys.exit(1)
parser.error("the input file must contain exactly two benchmarks")
a, b = (x.get("name", x["command"]) for x in results[:2])
X, Y = ([m["time_wall_clock"]["value"] for m in x["measurements"]] for x in results[:2])
X, Y = ([m[metric]["value"] for m in x["measurements"]] for x in results[:2])
if min(len(X), len(Y)) < 2:
parser.error("Welch's t-test requires at least two measurements per benchmark")
print(f"Metric: {METRICS[metric].label}")
print(f"Command 1: {a}")
print(f"Command 2: {b}\n")
t, p = stats.ttest_ind(X, Y, equal_var=False)
if not math.isfinite(p):
parser.error("Welch's t-test is undefined for these measurements")
th = 0.05
dispose = p < th
print(f"t = {t:.3}, p = {p:.3}")
@@ -40,9 +45,9 @@ print()
if dispose:
print(
f"A statistically significant difference in mean runtime was detected (p < {th})."
f"A statistically significant difference in mean value was detected (p < {th})."
)
else:
print(
f"No statistically significant difference in mean runtime was detected (p >= {th})."
f"No statistically significant difference in mean value was detected (p >= {th})."
)
src/benchmark/benchmark_result.rs
-6
View File
@@ -3,7 +3,6 @@ use std::collections::BTreeMap;
use serde::Serialize;
use crate::benchmark::measurement::Measurements;
use crate::quantity::Time;
/// Parameter value
#[derive(Debug, Default, Clone, Serialize, PartialEq)]
@@ -47,9 +46,4 @@ impl BenchmarkResult {
pub fn get_name(&self) -> &str {
self.name.as_deref().unwrap_or(&self.command)
}
/// The average wall clock time
pub fn mean_wall_clock_time(&self) -> Time {
self.measurements.time_wall_clock_mean()
}
}
src/benchmark/executor.rs
+3
View File
@@ -220,6 +220,7 @@ impl Executor for ShellExecutor<'_> {
ensure_non_negative(measurement.time_user - spawning_time.time_user);
measurement.time_system =
ensure_non_negative(measurement.time_system - spawning_time.time_system);
measurement.time_cpu = measurement.time_user + measurement.time_system;
}
Ok(measurement)
@@ -278,6 +279,7 @@ impl Executor for ShellExecutor<'_> {
self.shell_spawning_time = Some(Measurement {
time_wall_clock: measurements.time_wall_clock_mean(),
time_cpu: measurements.time_user_mean() + measurements.time_system_mean(),
time_user: measurements.time_user_mean(),
time_system: measurements.time_system_mean(),
memory_peak_resident: None,
@@ -337,6 +339,7 @@ impl Executor for MockExecutor {
Ok(Measurement {
time_wall_clock: Self::extract_time(command.get_command_line()),
time_cpu: Time::zero(),
time_user: Time::zero(),
time_system: Time::zero(),
memory_peak_resident: Some(Information::zero()),
src/benchmark/measurement.rs
+39 -30
View File
@@ -2,7 +2,8 @@ use std::process::ExitStatus;
use serde::Serialize;
use crate::quantity::statistics::{max, mean, median, min, modified_zscores, standard_deviation};
use crate::metric::Metric;
use crate::quantity::statistics::{mean, modified_zscores_f64};
use crate::quantity::{serialize_information, serialize_time, Information, Time};
use crate::util::exit_code::extract_exit_code;
@@ -77,6 +78,10 @@ pub struct Measurement {
#[serde(serialize_with = "serialize_time")]
pub time_wall_clock: Time,
/// Total CPU time (user and kernel mode)
#[serde(serialize_with = "serialize_time")]
pub time_cpu: Time,
/// Time spent in user mode
#[serde(serialize_with = "serialize_time")]
pub time_user: Time,
@@ -131,35 +136,14 @@ impl Measurements {
mean(self.wall_clock_times())
}
/// The standard deviation of all wall clock times. Not available if only one run has been performed
pub fn stddev(&self) -> Option<Time> {
let times: Vec<_> = self.wall_clock_times().collect(); // TODO: Avoid collecting
if times.len() < 2 {
None
} else {
Some(standard_deviation(times))
}
}
/// The median wall clock time
pub fn median(&self) -> Time {
median(self.wall_clock_times())
}
/// The minimum wall clock time
pub fn min(&self) -> Time {
min(self.wall_clock_times())
}
/// The maximum wall clock time
pub fn max(&self) -> Time {
max(self.wall_clock_times())
}
/// Compute modified Z-scores for the wall clock times
pub fn modified_zscores(&self) -> Vec<f64> {
modified_zscores(&self.wall_clock_times().collect::<Vec<_>>())
/// Compute modified Z-scores for a metric validated to be available in every run.
pub fn modified_zscores(&self, metric: Metric) -> Vec<f64> {
let values: Vec<_> = self
.measurements
.iter()
.map(|m| metric.value(m).expect("Validated metric is available"))
.collect();
modified_zscores_f64(&values)
}
/// The average user time
@@ -172,3 +156,28 @@ impl Measurements {
mean(self.measurements.iter().map(|m| m.time_system))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::outlier_detection::OUTLIER_THRESHOLD;
use crate::quantity::{byte, second};
#[test]
fn outliers_follow_the_selected_metric() {
let measurements = Measurements::new(
[(1.0, 100.0), (1.0, 1.0), (1.0, 1.0), (100.0, 1.0)]
.iter()
.copied()
.map(|(time, memory)| Measurement {
time_wall_clock: Time::new::<second>(time),
memory_peak_resident: Some(Information::new::<byte>(memory)),
..Measurement::default()
})
.collect(),
);
assert!(measurements.modified_zscores(Metric::TimeWallClock)[0] < OUTLIER_THRESHOLD);
assert!(measurements.modified_zscores(Metric::MemoryPeakResident)[0] > OUTLIER_THRESHOLD);
}
}
src/benchmark/mod.rs
+66 -93
View File
@@ -1,7 +1,6 @@
pub mod benchmark_result;
pub mod executor;
pub mod measurement;
pub mod relative_speed;
pub mod scheduler;
use std::cmp;
@@ -18,14 +17,15 @@ use crate::options::{
use crate::outlier_detection::OUTLIER_THRESHOLD;
use crate::output::console_writeln;
use crate::output::progress_bar::{
finish_initial_measurement, get_progress_bar, start_initial_measurement,
finish_initial_measurement, get_progress_bar, set_benchmark_header, start_initial_measurement,
};
use crate::output::report;
use crate::output::warnings::{OutlierWarningOptions, Warnings};
use crate::parameter::ParameterNameAndValue;
use crate::quantity::{self, const_time_from_seconds, FormatQuantity, Time, Zero};
use crate::quantity::{self, const_time_from_seconds, Time, Zero};
use benchmark_result::BenchmarkResult;
use anyhow::{anyhow, Result};
use anyhow::{anyhow, ensure, Result};
use colored::*;
use self::executor::Executor;
@@ -155,20 +155,21 @@ impl<'a> Benchmark<'a> {
self.run_intermediate_command(command, error_output, output_policy, iteration)
}
/// Run the benchmark for a single command
pub fn run(&self) -> Result<BenchmarkResult> {
if self.options.output_style != OutputStyleOption::Disabled {
console_writeln!(
io::stdout(),
"{}{}: {}",
"Benchmark ".bold(),
(self.number + 1).to_string().bold(),
self.command.get_name_with_unused_parameters(),
)?;
fn validate_measurement(&self, measurement: &Measurement) -> Result<()> {
if self.options.skip_unavailable_metrics {
return Ok(());
}
for selection in &self.options.metrics {
ensure!(selection.metric.value(measurement).is_some(),
"Metric '{}' is unavailable for '{}'. Check platform support and hardware-counter permissions.",
selection.metric.name(), self.command.get_command_line());
}
Ok(())
}
/// Run the benchmark for a single command
pub fn run(&self, reference: Option<&BenchmarkResult>) -> Result<BenchmarkResult> {
let mut measurements = Measurements::default();
let mut all_succeeded = true;
let output_policy = &self.options.command_output_policies[self.number];
@@ -213,6 +214,15 @@ impl<'a> Benchmark<'a> {
self.run_setup_command(self.command.get_parameters().iter().cloned(), output_policy)?;
let progress_header = format!(
"{}: {}",
format!("Benchmark {}", self.number + 1).bold(),
self.command
.get_name_with_unused_parameters()
.white()
.bold()
);
// Warmup phase
if self.options.warmup_count > 0 {
let progress_bar = if self.options.output_style != OutputStyleOption::Disabled {
@@ -225,6 +235,10 @@ impl<'a> Benchmark<'a> {
None
};
if let Some(bar) = &progress_bar {
set_benchmark_header(bar, progress_header.clone());
}
for i in 0..self.options.warmup_count {
let warmup_iteration = BenchmarkIteration::Warmup(i);
let _ = run_preparation_command(warmup_iteration)?;
@@ -259,6 +273,10 @@ impl<'a> Benchmark<'a> {
None
};
if let Some(bar) = &progress_bar {
set_benchmark_header(bar, progress_header);
}
let benchmark_iteration = BenchmarkIteration::Benchmark(0);
let preparation_result = run_preparation_command(benchmark_iteration)?;
let preparation_overhead = preparation_result.map_or(Time::zero(), |res| {
@@ -275,18 +293,14 @@ impl<'a> Benchmark<'a> {
None,
output_policy,
)?;
let success = measurement.exit_status.success();
self.validate_measurement(&measurement)?;
let mut all_succeeded = measurement.exit_status.success();
if let Some(bar) = progress_bar.as_ref() {
let time_unit = self
.options
.time_unit
.unwrap_or(measurement.time_wall_clock.suitable_unit());
let estimate = measurement.time_wall_clock.format(time_unit);
finish_initial_measurement(
bar,
format!("Current estimate: {}", estimate.to_string().green()),
);
let primary = self.options.metrics[0];
let value = primary.metric.value(&measurement).unwrap();
let estimate = primary.display_unit(value).format(value);
finish_initial_measurement(bar, format!("Current estimate: {}", estimate.green()));
}
let conclusion_result = run_conclusion_command(benchmark_iteration)?;
@@ -316,10 +330,10 @@ impl<'a> Benchmark<'a> {
let count_remaining = count - 1;
// Save the first result
let primary = self.options.metrics[0];
let mut primary_total = primary.metric.value(&measurement).unwrap();
measurements.push(measurement);
all_succeeded = all_succeeded && success;
// Re-configure the progress bar
if let Some(bar) = progress_bar.as_ref() {
bar.set_length(count);
@@ -330,18 +344,12 @@ impl<'a> Benchmark<'a> {
for i in 0..count_remaining {
let benchmark_iteration = BenchmarkIteration::Benchmark(i + 1);
let msg = {
let t_wall_clock_mean = measurements.time_wall_clock_mean();
let time_unit = self
.options
.time_unit
.unwrap_or(t_wall_clock_mean.suitable_unit());
let mean = t_wall_clock_mean.format(time_unit);
format!("Current estimate: {}", mean.to_string().green())
};
if let Some(bar) = progress_bar.as_ref() {
bar.set_message(msg.to_owned())
let mean = primary_total / measurements.len() as f64;
bar.set_message(format!(
"Current estimate: {}",
primary.display_unit(mean).format(mean).green()
));
}
run_preparation_command(benchmark_iteration)?;
@@ -352,7 +360,9 @@ impl<'a> Benchmark<'a> {
None,
output_policy,
)?;
self.validate_measurement(&measurement)?;
let success = measurement.exit_status.success();
primary_total += primary.metric.value(&measurement).unwrap();
measurements.push(measurement);
all_succeeded = all_succeeded && success;
@@ -368,63 +378,27 @@ impl<'a> Benchmark<'a> {
bar.finish_and_clear()
}
// Formatting and console output
let t_wall_clock_mean = measurements.time_wall_clock_mean();
let time_unit = self
.options
.time_unit
.unwrap_or(t_wall_clock_mean.suitable_unit());
let mean_str = t_wall_clock_mean.format(time_unit);
let min_str = measurements.min().format(time_unit);
let max_str = measurements.max().format(time_unit);
let num_str = format!("{num_runs} runs", num_runs = measurements.len());
let user_str = measurements.time_user_mean().format(time_unit);
let system_str = measurements.time_system_mean().format(time_unit);
if self.options.output_style != OutputStyleOption::Disabled {
let mut stdout = io::stdout().lock();
if measurements.len() == 1 {
console_writeln!(
stdout,
" Time ({} ≡): {:>8} {:>8} [User: {}, System: {}]",
"abs".green().bold(),
mean_str.green().bold(),
" ", // alignment
user_str.blue(),
system_str.blue()
)?;
} else {
let stddev_str = measurements.stddev().unwrap().format(time_unit);
console_writeln!(
stdout,
" Time ({} ± {}): {:>8} ± {:>8} [User: {}, System: {}]",
"mean".green().bold(),
"σ".green(),
mean_str.green().bold(),
stddev_str.green(),
user_str.blue(),
system_str.blue()
)?;
console_writeln!(
stdout,
" Range ({} … {}): {:>8} … {:>8} {}",
"min".cyan(),
"max".purple(),
min_str.cyan(),
max_str.purple(),
num_str.dimmed()
)?;
}
report::print(
self.number,
&self.command.get_name_with_unused_parameters(),
&measurements,
&self.options.metrics,
self.options.skip_unavailable_metrics,
reference,
)?;
}
// Warnings
let mut warnings = vec![];
// Check execution time
if matches!(self.options.executor_kind, ExecutorKind::Shell(_))
if self
.options
.metrics
.iter()
.any(|m| m.metric == crate::metric::Metric::TimeWallClock)
&& matches!(self.options.executor_kind, ExecutorKind::Shell(_))
&& measurements
.wall_clock_times()
.any(|t| t < MIN_EXECUTION_TIME)
@@ -438,7 +412,7 @@ impl<'a> Benchmark<'a> {
}
// Run outlier detection
let scores = measurements.modified_zscores();
let scores = measurements.modified_zscores(primary.metric);
let outlier_warning_options = OutlierWarningOptions {
warmup_in_use: self.options.warmup_count > 0,
@@ -452,12 +426,11 @@ impl<'a> Benchmark<'a> {
};
if scores[0] > OUTLIER_THRESHOLD {
warnings.push(Warnings::SlowInitialRun(
measurements.wall_clock_times().next().unwrap(),
warnings.push(Warnings::InitialRunOutlier(
primary,
primary.metric.value(&measurements.measurements[0]).unwrap(),
outlier_warning_options,
));
} else if scores.iter().any(|&s| s.abs() > OUTLIER_THRESHOLD) {
warnings.push(Warnings::OutliersDetected(outlier_warning_options));
}
if !warnings.is_empty() {
src/benchmark/relative_speed.rs
-237
View File
@@ -1,237 +0,0 @@
use std::cmp::Ordering;
use super::benchmark_result::BenchmarkResult;
use crate::{
options::SortOrder,
quantity::{self, Ratio, Time, Zero},
};
#[derive(Debug)]
pub struct BenchmarkResultWithRelativeSpeed<'a> {
pub result: &'a BenchmarkResult,
pub relative_speed: f64,
pub relative_speed_stddev: Option<f64>,
pub is_reference: bool,
// Less means faster
pub relative_ordering: Ordering,
}
impl BenchmarkResultWithRelativeSpeed<'_> {
pub fn reference_label(&self) -> &'static str {
if self.is_reference {
" (reference)"
} else {
match self.relative_ordering {
Ordering::Less => " (faster)",
Ordering::Equal => " (same speed)",
Ordering::Greater => " (slower)",
}
}
}
}
pub fn compare_mean_time(l: &BenchmarkResult, r: &BenchmarkResult) -> Ordering {
l.mean_wall_clock_time()
.partial_cmp(&r.mean_wall_clock_time())
.unwrap_or(Ordering::Equal)
}
pub fn fastest_of(results: &[BenchmarkResult]) -> &BenchmarkResult {
results
.iter()
.min_by(|&l, &r| compare_mean_time(l, r))
.expect("at least one benchmark result")
}
fn compute_relative_speeds<'a>(
results: &'a [BenchmarkResult],
reference: &'a BenchmarkResult,
sort_order: SortOrder,
) -> Vec<BenchmarkResultWithRelativeSpeed<'a>> {
let mut results: Vec<_> = results
.iter()
.map(|result| {
// Separate benchmarks can have identical names and measurements.
let is_reference = std::ptr::eq(result, reference);
let relative_ordering = compare_mean_time(result, reference);
if result.mean_wall_clock_time() == Time::zero()
|| reference.mean_wall_clock_time() == Time::zero()
{
return BenchmarkResultWithRelativeSpeed {
result,
relative_speed: if is_reference { 1.0 } else { f64::INFINITY },
relative_speed_stddev: None,
is_reference,
relative_ordering,
};
}
let ratio = match relative_ordering {
Ordering::Less => reference.mean_wall_clock_time() / result.mean_wall_clock_time(),
Ordering::Equal => Ratio::new::<quantity::ratio>(1.0),
Ordering::Greater => {
result.mean_wall_clock_time() / reference.mean_wall_clock_time()
}
};
// https://en.wikipedia.org/wiki/Propagation_of_uncertainty#Example_formulas
// Covariance assumed to be 0, i.e. variables are assumed to be independent
let ratio_stddev = match (
result.measurements.stddev(),
reference.measurements.stddev(),
) {
(Some(result_stddev), Some(reference_stddev)) => Some(
ratio
* ((result_stddev / result.mean_wall_clock_time())
.powi(uom::typenum::P2::new())
+ (reference_stddev / reference.mean_wall_clock_time())
.powi(uom::typenum::P2::new()))
.sqrt(),
),
_ => None,
};
BenchmarkResultWithRelativeSpeed {
result,
relative_speed: ratio.get::<quantity::ratio>(),
relative_speed_stddev: ratio_stddev.map(|r| r.get::<quantity::ratio>()),
is_reference,
relative_ordering,
}
})
.collect();
match sort_order {
SortOrder::Command => {}
SortOrder::MeanTime => {
results.sort_unstable_by(|r1, r2| compare_mean_time(r1.result, r2.result));
}
}
results
}
pub fn compute_with_check_from_reference<'a>(
results: &'a [BenchmarkResult],
reference: &'a BenchmarkResult,
sort_order: SortOrder,
) -> Option<Vec<BenchmarkResultWithRelativeSpeed<'a>>> {
if fastest_of(results).mean_wall_clock_time() == Time::zero()
|| reference.mean_wall_clock_time() == Time::zero()
{
return None;
}
Some(compute_relative_speeds(results, reference, sort_order))
}
pub fn compute_with_check(
results: &[BenchmarkResult],
sort_order: SortOrder,
) -> Option<Vec<BenchmarkResultWithRelativeSpeed<'_>>> {
let fastest = fastest_of(results);
if fastest.mean_wall_clock_time() == Time::zero() {
return None;
}
Some(compute_relative_speeds(results, fastest, sort_order))
}
/// Compute relative speeds against the given reference, or the fastest result.
pub fn compute<'a>(
results: &'a [BenchmarkResult],
sort_order: SortOrder,
reference: Option<&'a BenchmarkResult>,
) -> Vec<BenchmarkResultWithRelativeSpeed<'a>> {
let reference = reference.unwrap_or_else(|| fastest_of(results));
compute_relative_speeds(results, reference, sort_order)
}
#[cfg(test)]
fn create_result(name: &str, mean: f64) -> BenchmarkResult {
use std::collections::BTreeMap;
use crate::benchmark::measurement::{Measurement, Measurements};
use crate::quantity::second;
BenchmarkResult {
command: name.into(),
display_name: name.into(),
name: None,
measurements: Measurements {
measurements: vec![Measurement {
time_wall_clock: Time::new::<second>(mean),
time_user: Time::new::<second>(mean),
..Default::default()
}],
},
parameters: BTreeMap::new(),
}
}
#[test]
fn test_compute_relative_speed() {
use approx::assert_relative_eq;
let results = vec![
create_result("cmd1", 3.0),
create_result("cmd2", 2.0),
create_result("cmd3", 5.0),
];
let annotated_results = compute_with_check(&results, SortOrder::Command).unwrap();
assert_relative_eq!(1.5, annotated_results[0].relative_speed);
assert_relative_eq!(1.0, annotated_results[1].relative_speed);
assert_relative_eq!(2.5, annotated_results[2].relative_speed);
}
#[test]
fn test_compute_relative_speed_with_reference() {
use approx::assert_relative_eq;
let results = vec![create_result("cmd2", 2.0), create_result("cmd3", 5.0)];
let reference = create_result("cmd2", 4.0);
let annotated_results =
compute_with_check_from_reference(&results, &reference, SortOrder::Command).unwrap();
assert_relative_eq!(2.0, annotated_results[0].relative_speed);
assert_relative_eq!(1.25, annotated_results[1].relative_speed);
}
#[test]
fn test_compute_relative_speed_for_zero_times() {
let results = vec![create_result("cmd1", 1.0), create_result("cmd2", 0.0)];
let annotated_results = compute_with_check(&results, SortOrder::Command);
assert!(annotated_results.is_none());
}
#[test]
fn reference_identity_distinguishes_equal_results() {
let results = vec![create_result("same", 1.0), create_result("same", 1.0)];
let entries = compute_relative_speeds(&results, &results[0], SortOrder::Command);
assert!(entries[0].is_reference);
assert!(!entries[1].is_reference);
}
#[test]
fn reference_ratios_handle_zero_times() {
let mut results = vec![create_result("reference", 2.0), create_result("zero", 0.0)];
for result in &mut results {
let measurement = result.measurements.measurements[0].clone();
result.measurements.measurements.push(measurement);
}
let entries = compute_relative_speeds(&results, &results[0], SortOrder::Command);
assert_eq!(entries[1].relative_speed, f64::INFINITY);
let entries = compute_relative_speeds(&results, &results[1], SortOrder::Command);
assert_eq!(entries[0].relative_speed, f64::INFINITY);
assert_eq!(entries[0].relative_speed_stddev, None);
assert_eq!(entries[1].relative_speed, 1.0);
}
src/benchmark/scheduler.rs
+24 -130
View File
@@ -1,14 +1,10 @@
use super::benchmark_result::BenchmarkResult;
use super::executor::{Executor, MockExecutor, RawExecutor, ShellExecutor};
use super::{relative_speed, Benchmark};
use colored::*;
use std::cmp::Ordering;
use std::io::{self, Write};
use super::Benchmark;
use crate::command::{Command, Commands};
use crate::command::Commands;
use crate::export::ExportManager;
use crate::options::{ExecutorKind, Options, OutputStyleOption, SortOrder};
use crate::output::console_writeln;
use crate::options::{ExecutorKind, Options};
use anyhow::Result;
@@ -40,133 +36,23 @@ impl<'a> Scheduler<'a> {
ExecutorKind::Shell(ref shell) => Box::new(ShellExecutor::new(shell, self.options)),
};
let reference = self
.options
.reference_command
.as_ref()
.map(|cmd| Command::new(self.options.reference_name.as_deref(), cmd));
executor.calibrate()?;
for (number, cmd) in reference.iter().chain(self.commands.iter()).enumerate() {
self.results
.push(Benchmark::new(number, cmd, self.options, &*executor).run()?);
for (number, cmd) in self.commands.iter().enumerate() {
self.results.push(
Benchmark::new(number, cmd, self.options, &*executor).run(self.results.first())?,
);
// We export results after each individual benchmark, because
// we would risk losing them if a later benchmark fails.
self.export_manager
.write_results(&self.results, true, self.options.reference_index)?;
}
Ok(())
}
pub fn print_relative_speed_comparison(&self) -> Result<()> {
if self.options.output_style == OutputStyleOption::Disabled {
return Ok(());
}
if self.results.len() < 2 {
return Ok(());
}
let reference = self
.options
.reference_index
.map(|index| &self.results[index])
.unwrap_or_else(|| relative_speed::fastest_of(&self.results));
if let Some(annotated_results) = relative_speed::compute_with_check_from_reference(
&self.results,
reference,
self.options.sort_order_speed_comparison,
) {
let mut stdout = io::stdout().lock();
match self.options.sort_order_speed_comparison {
SortOrder::MeanTime => {
console_writeln!(stdout, "{}", "Summary".bold())?;
let reference = annotated_results.iter().find(|r| r.is_reference).unwrap();
let others = annotated_results.iter().filter(|r| !r.is_reference);
console_writeln!(stdout, " {} ran", reference.result.display_name.cyan())?;
for item in others {
let stddev = if let Some(stddev) = item.relative_speed_stddev {
format!(" ± {}", format!("{:.2}", stddev).green())
} else {
"".into()
};
let comparator = match item.relative_ordering {
Ordering::Less => format!(
"{}{} times slower than",
format!("{:8.2}", item.relative_speed).bold().green(),
stddev
),
Ordering::Greater => format!(
"{}{} times faster than",
format!("{:8.2}", item.relative_speed).bold().green(),
stddev
),
Ordering::Equal => format!(
" As fast ({}{}) as",
format!("{:.2}", item.relative_speed).bold().green(),
stddev
),
};
console_writeln!(
stdout,
"{} {}",
comparator,
&item.result.display_name.magenta()
)?;
}
}
SortOrder::Command => {
console_writeln!(stdout, "{}", "Relative speed comparison".bold())?;
for item in annotated_results {
let relationship = if self.options.reference_index.is_some() {
item.reference_label()
} else {
""
};
console_writeln!(
stdout,
" {}{} {}{}",
format!("{:10.2}", item.relative_speed).bold().green(),
if item.is_reference {
" ".into()
} else if let Some(stddev) = item.relative_speed_stddev {
format!(" ± {}", format!("{stddev:5.2}").green())
} else {
" ".into()
},
item.result.display_name,
relationship,
)?;
}
}
}
} else {
console_writeln!(
io::stderr(),
"{}: The benchmark comparison could not be computed as some benchmark times are zero. \
This could be caused by background interference during the initial calibration phase \
of hyperfine, in combination with very fast commands (faster than a few milliseconds). \
Try to re-run the benchmark on a quiet system. If you did not do so already, try the \
--shell=none/-N option. If it does not help either, you command is most likely too fast \
to be accurately benchmarked by hyperfine.",
"Note".bold().red()
)?;
self.export_manager.write_results(&self.results, true)?;
}
Ok(())
}
pub fn final_export(&self) -> Result<()> {
self.export_manager
.write_results(&self.results, false, self.options.reference_index)
self.export_manager.write_results(&self.results, false)
}
}
@@ -183,11 +69,7 @@ fn generate_results(args: &[&'static str]) -> Result<Vec<BenchmarkResult>> {
assert_eq!(options.executor_kind, ExecutorKind::Mock(None));
let commands = Commands::from_cli_arguments(&cli_arguments)?;
let export_manager = ExportManager::from_cli_arguments(
&cli_arguments,
options.time_unit,
options.sort_order_exports,
)?;
let export_manager = ExportManager::from_cli_arguments(&cli_arguments, options.metrics[0])?;
options.validate_against_command_list(&commands)?;
@@ -199,12 +81,15 @@ fn generate_results(args: &[&'static str]) -> Result<Vec<BenchmarkResult>> {
#[test]
fn scheduler_basic() -> Result<()> {
insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @r#"
insta::assert_yaml_snapshot!(generate_results(&["--runs=2", "sleep 0.123", "sleep 0.456"])?, @"
- command: sleep 0.123
measurements:
- time_wall_clock:
value: 0.123
unit: second
time_cpu:
value: 0
unit: second
time_user:
value: 0
unit: second
@@ -218,6 +103,9 @@ fn scheduler_basic() -> Result<()> {
- time_wall_clock:
value: 0.123
unit: second
time_cpu:
value: 0
unit: second
time_user:
value: 0
unit: second
@@ -233,6 +121,9 @@ fn scheduler_basic() -> Result<()> {
- time_wall_clock:
value: 0.456
unit: second
time_cpu:
value: 0
unit: second
time_user:
value: 0
unit: second
@@ -246,6 +137,9 @@ fn scheduler_basic() -> Result<()> {
- time_wall_clock:
value: 0.456
unit: second
time_cpu:
value: 0
unit: second
time_user:
value: 0
unit: second
@@ -256,7 +150,7 @@ fn scheduler_basic() -> Result<()> {
value: 0
unit: byte
exit_code: 0
"#);
");
Ok(())
}
src/cli.rs
+49 -70
View File
@@ -7,6 +7,12 @@ use clap::{
crate_version, Arg, ArgAction, ArgMatches, Command, ValueHint,
};
pub const DEFAULT_METRICS: &str = if cfg!(windows) {
"time_wall_clock"
} else {
"time_wall_clock,memory_peak_resident"
};
pub fn get_cli_arguments<'a, I, T>(args: I) -> ArgMatches
where
I: IntoIterator<Item = T>,
@@ -38,12 +44,40 @@ fn build_command() -> Command {
line like \"grep -i todo\" or a shell command like \"sleep 0.5 && echo test\". \
The latter requires a shell, which can be enabled via '-S' or '--shell=...'. \
If multiple commands are given, hyperfine will show a \
comparison of the respective runtimes.")
comparison with the first command as the reference.")
.required(true)
.action(ArgAction::Append)
.value_hint(ValueHint::CommandString)
.value_parser(NonEmptyStringValueParser::new()),
)
.arg(
Arg::new("metrics")
.long("metrics")
.value_name("PRESET|METRIC[:UNIT],…")
.default_value("default")
.hide_default_value(true)
.help("Performance metrics to measure, in order.\n\n\
This is either a preset name, or a comma-separated list of metrics with optional units. \
The first metric is used for non-JSON exports, \
JSON always includes all collected metrics.\n\n\
Presets (use one on its own):\n \
default Wall-clock time and peak RSS\n \
time Wall-clock, CPU, user, and system time\n \
all All available metrics\n\n\
Presets use automatic units.\n\n\
Metrics and optional units:\n \
Time: time_wall_clock, time_cpu, time_user, time_system\n \
Units: ns, us, ms, s, min, h\n \
Memory: memory_peak_resident\n \
Units: B, kB, MB, GB, TB, KiB, MiB, GiB, TiB\n \
Counters: cpu_cycles, instructions, cache_references,\n \
cache_misses, branch_misses\n \
Units: count, k (thousand), M (million), B (billion)\n\n\
Examples:\n \
--metrics time_cpu,instructions\n \
--metrics memory_peak_resident:MiB,time_wall_clock:ms\n \
--metrics time"),
)
.arg(
Arg::new("warmup")
.long("warmup")
@@ -93,34 +127,12 @@ fn build_command() -> Command {
.value_name("CMD")
.value_hint(ValueHint::CommandString)
.help(
"Execute CMD before each set of timing runs. This is useful for \
"Execute CMD before each set of benchmark runs. This is useful for \
compiling your software with the provided parameters, or to do any \
other work that should happen once before a series of benchmark runs, \
not every time as would happen with the --prepare option."
),
)
.arg(
Arg::new("reference")
.long("reference")
.action(ArgAction::Set)
.value_name("CMD")
.help(
"The reference for the relative comparison of results. Without parameters, \
CMD is run as a separate reference command. With --parameter-scan or \
--parameter-list, CMD must exactly match one unique benchmark name as \
shown in the output. If this is unset, results are compared with the \
fastest command as reference."
)
)
.arg(
Arg::new("reference-name")
.long("reference-name")
.action(ArgAction::Set)
.value_name("CMD")
.help("Give a meaningful name to the reference command. This cannot be used \
with parameterized benchmarks; use --command-name instead.")
.requires("reference")
)
.arg(
Arg::new("prepare")
.long("prepare")
@@ -130,7 +142,7 @@ fn build_command() -> Command {
.value_name("CMD")
.value_hint(ValueHint::CommandString)
.help(
"Execute CMD before each timing run. This is useful for \
"Execute CMD before each benchmark run. This is useful for \
clearing disk caches, for example.\nThe --prepare option can \
be specified once for all commands or multiple times, once for \
each command. In the latter case, each preparation command will \
@@ -146,7 +158,7 @@ fn build_command() -> Command {
.value_name("CMD")
.value_hint(ValueHint::CommandString)
.help(
"Execute CMD after each timing run. This is useful for killing \
"Execute CMD after each benchmark run. This is useful for killing \
long-running processes started (e.g. a web server started in --prepare), \
for example.\nThe --conclude option can be specified once for all \
commands or multiple times, once for each command. In the latter case, \
@@ -275,47 +287,14 @@ fn build_command() -> Command {
the output of the tool.",
),
)
.arg(
Arg::new("sort")
.long("sort")
.action(ArgAction::Set)
.value_name("METHOD")
.value_parser(["auto", "command", "mean-time"])
.default_value("auto")
.hide_default_value(true)
.help(
"Specify the sort order of the speed comparison summary and the exported tables for \
markup formats (Markdown, AsciiDoc, org-mode):\n \
* 'auto' (default): the speed comparison will be ordered by time and\n \
the markup tables will be ordered by command (input order).\n \
* 'command': order benchmarks in the way they were specified\n \
* 'mean-time': order benchmarks by mean runtime\n"
),
)
.arg(
Arg::new("time-unit")
.long("time-unit")
.short('u')
.action(ArgAction::Set)
.value_name("UNIT")
.value_parser(["µs", "us", "microsecond", "microseconds", "ms", "millisecond", "milliseconds", "s", "second", "seconds", "min", "minute", "minutes", "h", "hour", "hours"])
.help("Set the time unit to be used. If the option is not given, the time unit is determined automatically. \
This option affects the standard output as well as all export formats except for CSV and JSON.\n\
Possible values:\n \
* 'µs', 'us', 'microsecond', 'microseconds'\n \
* 'ms', 'millisecond', 'milliseconds'\n \
* 's', 'second', 'seconds'\n \
* 'min', 'minute', 'minutes'\n \
* 'h', 'hour', 'hours'"),
)
.arg(
Arg::new("export-asciidoc")
.long("export-asciidoc")
.action(ArgAction::Set)
.value_name("FILE")
.value_hint(ValueHint::FilePath)
.help("Export the timing summary statistics as an AsciiDoc table to the given FILE. \
The output time unit can be changed using the --time-unit option."),
.help("Export the primary metric summary statistics as an AsciiDoc table to the given FILE. \
The unit can be set with --metrics METRIC:UNIT."),
)
.arg(
Arg::new("export-csv")
@@ -323,9 +302,9 @@ fn build_command() -> Command {
.action(ArgAction::Set)
.value_name("FILE")
.value_hint(ValueHint::FilePath)
.help("Export the timing summary statistics as CSV to the given FILE. If you need \
the timing results for each individual run, use the JSON export format. \
The output time unit is always seconds."),
.help("Export the primary metric summary statistics as CSV to the given FILE. If you need \
all metrics for each individual run, use the JSON export format. \
Use the explicitly selected unit, or the base unit when unspecified."),
)
.arg(
Arg::new("export-json")
@@ -333,8 +312,8 @@ fn build_command() -> Command {
.action(ArgAction::Set)
.value_name("FILE")
.value_hint(ValueHint::FilePath)
.help("Export the timing summary statistics and timings of individual runs as JSON to the given FILE. \
The output time unit is always seconds"),
.help("Export all collected metrics and individual runs as JSON to the given FILE. \
Values always use base units (seconds, bytes, counts)."),
)
.arg(
Arg::new("export-markdown")
@@ -342,8 +321,8 @@ fn build_command() -> Command {
.action(ArgAction::Set)
.value_name("FILE")
.value_hint(ValueHint::FilePath)
.help("Export the timing summary statistics as a Markdown table to the given FILE. \
The output time unit can be changed using the --time-unit option."),
.help("Export the primary metric summary statistics as a Markdown table to the given FILE. \
The unit can be set with --metrics METRIC:UNIT."),
)
.arg(
Arg::new("export-orgmode")
@@ -351,8 +330,8 @@ fn build_command() -> Command {
.action(ArgAction::Set)
.value_name("FILE")
.value_hint(ValueHint::FilePath)
.help("Export the timing summary statistics as an Emacs org-mode table to the given FILE. \
The output time unit can be changed using the --time-unit option."),
.help("Export the primary metric summary statistics as an Emacs org-mode table to the given FILE. \
The unit can be set with --metrics METRIC:UNIT."),
)
.arg(
Arg::new("show-output")
src/command.rs
+2 -2
View File
@@ -257,8 +257,8 @@ impl<'a> Commands<'a> {
self.0.iter()
}
pub fn num_commands(&self, has_reference_command: bool) -> usize {
self.0.len() + if has_reference_command { 1 } else { 0 }
pub fn num_commands(&self) -> usize {
self.0.len()
}
/// Finds all the strings that appear multiple times in the input iterator, returning them in
src/error.rs
+2
View File
@@ -41,6 +41,8 @@ impl From<DecimalError> for ParameterScanError {
#[derive(Debug, Error)]
pub enum OptionsError<'a> {
#[error("{0}")]
InvalidMetrics(String),
#[error(
"Conflicting requirements for the number of runs (empty range, min is larger than max)"
)]
src/export/asciidoc.rs
+11 -2
View File
@@ -24,7 +24,16 @@ impl MarkupExporter for AsciidocExporter {
}
fn table_row(&self, cells: &[&str]) -> String {
format!("\n| {} \n", cells.join(" \n| "))
let mut row = String::from("\n");
for cell in cells {
row.push('|');
if !cell.is_empty() {
row.push(' ');
row.push_str(cell);
}
row.push('\n');
}
row
}
fn table_divider(&self, _cell_aligmnents: &[Alignment]) -> String {
@@ -43,7 +52,7 @@ fn test_asciidoc_exporter_table_data() {
let data = vec!["a", "b", "c"];
let actual = exporter.table_row(&data);
let expect = "\n| a \n| b \n| c \n";
let expect = "\n| a\n| b\n| c\n";
assert_eq!(expect, actual);
}
src/export/csv.rs
+81 -53
View File
@@ -1,65 +1,73 @@
use std::borrow::Cow;
use csv::WriterBuilder;
use super::Exporter;
use crate::benchmark::benchmark_result::BenchmarkResult;
use crate::options::SortOrder;
use crate::quantity::{FormatQuantity, TimeUnit};
use crate::metric::MetricSelection;
use anyhow::Result;
use anyhow::{Context, Result};
#[derive(Default)]
pub struct CsvExporter {}
impl Exporter for CsvExporter {
fn serialize(
&self,
results: &[BenchmarkResult],
_time_unit: Option<TimeUnit>,
_sort_order: SortOrder,
_reference_index: Option<usize>,
) -> Result<Vec<u8>> {
const CSV_UNIT: TimeUnit = TimeUnit::Second;
const CSV_PRECISION: usize = 6;
fn serialize(&self, results: &[BenchmarkResult], primary: MetricSelection) -> Result<Vec<u8>> {
// CSV never chooses units automatically: an omitted suffix means base units.
let unit = primary.csv_unit();
let mut writer = WriterBuilder::new().from_writer(vec![]);
{
let mut headers: Vec<Cow<[u8]>> = [
// The list of times and exit codes cannot be exported to the CSV file - omit them.
"command", "mean", "stddev", "median", "user", "system", "min", "max",
]
.iter()
.map(|x| Cow::Borrowed(x.as_bytes()))
.collect();
if let Some(res) = results.first() {
for param_name in res.parameters.keys() {
headers.push(Cow::Owned(format!("parameter_{param_name}").into_bytes()));
}
}
writer.write_record(headers)?;
let mut headers: Vec<String> = [
"command", "metric", "unit", "mean", "stddev", "median", "min", "max",
]
.iter()
.copied()
.map(str::to_owned)
.collect();
if let Some(result) = results.first() {
headers.extend(
result
.parameters
.keys()
.map(|name| format!("parameter_{name}")),
);
}
writer.write_record(headers)?;
for res in results {
let mut fields = vec![Cow::Borrowed(res.get_name().as_bytes())];
for f in &[
res.mean_wall_clock_time(),
res.measurements.stddev().unwrap_or_default(),
res.measurements.median(),
res.measurements.time_user_mean(),
res.measurements.time_system_mean(),
res.measurements.min(),
res.measurements.max(),
] {
fields.push(Cow::Owned(
f.format_with_precision(CSV_UNIT, CSV_PRECISION)
.into_bytes(),
))
}
for v in res.parameters.values() {
fields.push(Cow::Borrowed(v.value.as_bytes()))
}
for result in results {
let stats = primary
.metric
.summarize(&result.measurements)
.with_context(|| {
format!(
"Metric '{}' is unavailable for '{}'",
primary.metric.name(),
result.get_name()
)
})?;
let mut fields = vec![
result.get_name().to_owned(),
primary.metric.name().to_owned(),
unit.symbol.to_owned(),
];
// Preserve numeric precision rather than using rounded display values.
fields.extend(
[
Some(stats.mean),
stats.stddev,
Some(stats.median),
Some(stats.min),
Some(stats.max),
]
.map(|value| {
value
.map(|value| (value / unit.scale).to_string())
.unwrap_or_default()
}),
);
fields.extend(
result
.parameters
.values()
.map(|parameter| parameter.value.clone()),
);
writer.write_record(fields)?;
}
@@ -86,6 +94,7 @@ fn test_csv() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(7.0),
time_cpu: Time::new::<second>(7.0),
time_user: Time::new::<second>(7.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -94,6 +103,7 @@ fn test_csv() {
},
Measurement {
time_wall_clock: Time::new::<second>(8.0),
time_cpu: Time::new::<second>(8.0),
time_user: Time::new::<second>(8.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -102,6 +112,7 @@ fn test_csv() {
},
Measurement {
time_wall_clock: Time::new::<second>(12.0),
time_cpu: Time::new::<second>(12.0),
time_user: Time::new::<second>(12.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -133,6 +144,7 @@ fn test_csv() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(17.0),
time_cpu: Time::new::<second>(17.0),
time_user: Time::new::<second>(17.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -141,6 +153,7 @@ fn test_csv() {
},
Measurement {
time_wall_clock: Time::new::<second>(18.0),
time_cpu: Time::new::<second>(18.0),
time_user: Time::new::<second>(18.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -149,6 +162,7 @@ fn test_csv() {
},
Measurement {
time_wall_clock: Time::new::<second>(19.0),
time_cpu: Time::new::<second>(19.0),
time_user: Time::new::<second>(19.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -177,14 +191,28 @@ fn test_csv() {
let actual = String::from_utf8(
exporter
.serialize(&results, Some(TimeUnit::Second), SortOrder::Command, None)
.serialize(&results, MetricSelection::default())
.unwrap(),
)
.unwrap();
insta::assert_snapshot!(actual, @r#"
command,mean,stddev,median,user,system,min,max,parameter_bar,parameter_foo
command_a,9.000000,2.645751,8.000000,9.000000,0.000000,7.000000,12.000000,two,one
command_b,18.000000,1.000000,18.000000,18.000000,0.000000,17.000000,19.000000,seven,one
command,metric,unit,mean,stddev,median,min,max,parameter_bar,parameter_foo
command_a,time_wall_clock,s,9,2.6457513110645907,8,7,12,two,one
command_b,time_wall_clock,s,18,1,18,17,19,seven,one
"#);
for (selection, unit, mean) in [
("memory_peak_resident", "B", "1024"),
("memory_peak_resident:KiB", "KiB", "1"),
] {
let output = exporter
.serialize(&results, MetricSelection::parse_list(selection).unwrap()[0])
.unwrap();
let mut reader = csv::Reader::from_reader(output.as_slice());
let row = reader.records().next().unwrap().unwrap();
assert_eq!(&row[1], "memory_peak_resident");
assert_eq!(&row[2], unit);
assert_eq!(&row[3], mean);
}
}
src/export/json.rs
+33 -178
View File
@@ -1,14 +1,11 @@
use std::ops::{Add, AddAssign, Div};
use std::collections::BTreeMap;
use serde::*;
use serde::Serialize;
use serde_json::to_vec_pretty;
use super::Exporter;
use crate::benchmark::benchmark_result::BenchmarkResult;
use crate::benchmark::measurement::Measurements;
use crate::options::SortOrder;
use crate::quantity::statistics::{max, mean, median, min, standard_deviation, UnsafeRawValue};
use crate::quantity::{byte, ratio, second, QuantityInUnit, Ratio, TimeUnit, Zero};
use crate::metric::{Metric, MetricSelection, Stats};
use anyhow::Result;
@@ -21,6 +18,7 @@ const JSON_SCHEMA_VERSION: u32 = 2;
struct HyperfineSummary<'a> {
schema_version: u32,
metadata: &'a Metadata,
primary_metric: &'static str,
results: Vec<JsonBenchmarkResult<'a>>,
}
@@ -28,98 +26,7 @@ struct HyperfineSummary<'a> {
struct JsonBenchmarkResult<'a> {
#[serde(flatten)]
result: &'a BenchmarkResult,
#[serde(skip_serializing_if = "std::ops::Not::not")]
is_reference: bool,
summary: BenchmarkSummary,
}
#[derive(Serialize)]
struct BenchmarkSummary {
time_wall_clock: StatisticalSummary,
time_user: StatisticalSummary,
time_system: StatisticalSummary,
#[serde(skip_serializing_if = "Option::is_none")]
memory_peak_resident: Option<StatisticalSummary>,
#[serde(skip_serializing_if = "Option::is_none")]
cpu_cycles: Option<StatisticalSummary>,
#[serde(skip_serializing_if = "Option::is_none")]
instructions: Option<StatisticalSummary>,
#[serde(skip_serializing_if = "Option::is_none")]
cache_references: Option<StatisticalSummary>,
#[serde(skip_serializing_if = "Option::is_none")]
cache_misses: Option<StatisticalSummary>,
#[serde(skip_serializing_if = "Option::is_none")]
branch_misses: Option<StatisticalSummary>,
}
impl BenchmarkSummary {
fn from_measurements(measurements: &Measurements) -> Self {
let values = &measurements.measurements;
Self {
time_wall_clock: StatisticalSummary::from_values(
values.iter().map(|m| Some(m.time_wall_clock)),
Some(second),
)
.expect("Benchmarks have at least one measurement"),
time_user: StatisticalSummary::from_values(
values.iter().map(|m| Some(m.time_user)),
Some(second),
)
.expect("Benchmarks have at least one measurement"),
time_system: StatisticalSummary::from_values(
values.iter().map(|m| Some(m.time_system)),
Some(second),
)
.expect("Benchmarks have at least one measurement"),
memory_peak_resident: StatisticalSummary::from_values(
values
.iter()
.filter_map(|m| m.memory_peak_resident)
.map(Some),
Some(byte),
),
cpu_cycles: StatisticalSummary::from_values(
values.iter().map(|m| {
m.hardware_counters
.cpu_cycles
.map(|v| Ratio::new::<ratio>(v as f64))
}),
None::<ratio>,
),
instructions: StatisticalSummary::from_values(
values.iter().map(|m| {
m.hardware_counters
.instructions
.map(|v| Ratio::new::<ratio>(v as f64))
}),
None::<ratio>,
),
cache_references: StatisticalSummary::from_values(
values.iter().map(|m| {
m.hardware_counters
.cache_references
.map(|v| Ratio::new::<ratio>(v as f64))
}),
None::<ratio>,
),
cache_misses: StatisticalSummary::from_values(
values.iter().map(|m| {
m.hardware_counters
.cache_misses
.map(|v| Ratio::new::<ratio>(v as f64))
}),
None::<ratio>,
),
branch_misses: StatisticalSummary::from_values(
values.iter().map(|m| {
m.hardware_counters
.branch_misses
.map(|v| Ratio::new::<ratio>(v as f64))
}),
None::<ratio>,
),
}
}
summary: BTreeMap<&'static str, StatisticalSummary>,
}
#[derive(Serialize)]
@@ -135,37 +42,20 @@ struct StatisticalSummary {
}
impl StatisticalSummary {
fn from_values<Q, P, U>(
values: impl IntoIterator<Item = Option<Q>>,
unit: Option<U>,
) -> Option<Self>
where
Q: Copy
+ PartialOrd
+ Add<Output = Q>
+ AddAssign
+ Zero
+ Div<Ratio, Output = P>
+ UnsafeRawValue
+ QuantityInUnit<U>,
P: Into<Q>,
U: uom::si::Unit,
{
// Only summarize a metric when it is available for every supplied measurement.
let values: Vec<_> = values.into_iter().collect::<Option<_>>()?;
if values.is_empty() {
return None;
fn new(metric: Metric, stats: Stats) -> Self {
Self {
unit: match metric.base_unit().symbol {
"s" => Some("second"),
"B" => Some("byte"),
_ => None,
},
count: stats.count,
mean: stats.mean,
stddev: stats.stddev,
median: stats.median,
min: stats.min,
max: stats.max,
}
Some(Self {
unit: unit.map(|_| U::singular()),
count: values.len(),
mean: mean(values.iter().copied()).value_in_unit(),
stddev: (values.len() > 1)
.then(|| standard_deviation(values.iter().copied()).value_in_unit()),
median: median(values.iter().copied()).value_in_unit(),
min: min(values.iter().copied()).value_in_unit(),
max: max(values.iter().copied()).value_in_unit(),
})
}
}
@@ -175,63 +65,28 @@ pub struct JsonExporter {
}
impl Exporter for JsonExporter {
fn serialize(
&self,
results: &[BenchmarkResult],
_time_unit: Option<TimeUnit>,
_sort_order: SortOrder,
reference_index: Option<usize>,
) -> Result<Vec<u8>> {
fn serialize(&self, results: &[BenchmarkResult], primary: MetricSelection) -> Result<Vec<u8>> {
let mut output = to_vec_pretty(&HyperfineSummary {
schema_version: JSON_SCHEMA_VERSION,
metadata: &self.metadata,
primary_metric: primary.metric.name(),
results: results
.iter()
.enumerate()
.map(|(index, result)| JsonBenchmarkResult {
.map(|result| JsonBenchmarkResult {
result,
is_reference: reference_index == Some(index),
summary: BenchmarkSummary::from_measurements(&result.measurements),
summary: Metric::ALL
.iter()
.copied()
.filter_map(|metric| {
metric.summarize(&result.measurements).map(|stats| {
(metric.name(), StatisticalSummary::new(metric, stats))
})
})
.collect(),
})
.collect(),
});
if let Ok(ref mut content) = output {
content.push(b'\n');
}
Ok(output?)
}
}
#[test]
fn test_statistical_summary_unit_conversion() {
use crate::quantity::{kibibyte, millisecond, Information, Time};
let time = StatisticalSummary::from_values(
[Time::new::<second>(1.0), Time::new::<second>(3.0)].map(Some),
Some(millisecond),
)
.unwrap();
let information = StatisticalSummary::from_values(
[
Information::new::<byte>(1024.0),
Information::new::<byte>(3072.0),
]
.map(Some),
Some(kibibyte),
)
.unwrap();
for (summary, unit, scale) in [
(time, "millisecond", 1000.0),
(information, "kibibyte", 1.0),
] {
assert_eq!(summary.unit, Some(unit));
assert_eq!(summary.count, 2);
approx::assert_relative_eq!(summary.mean, 2.0 * scale);
approx::assert_relative_eq!(summary.median, 2.0 * scale);
approx::assert_relative_eq!(summary.min, scale);
approx::assert_relative_eq!(summary.max, 3.0 * scale);
approx::assert_relative_eq!(summary.stddev.unwrap(), 2.0_f64.sqrt() * scale);
})?;
output.push(b'\n');
Ok(output)
}
}
src/export/markup.rs
+61 -101
View File
@@ -1,11 +1,8 @@
use crate::benchmark::relative_speed::BenchmarkResultWithRelativeSpeed;
use crate::benchmark::{benchmark_result::BenchmarkResult, relative_speed};
use crate::options::SortOrder;
use crate::quantity::{FormatQuantity, IsUnit, Time, TimeUnit, Zero};
use crate::benchmark::benchmark_result::BenchmarkResult;
use crate::metric::{MetricSelection, Stats, Unit};
use super::Exporter;
use anyhow::Result;
use std::cmp::Ordering;
use anyhow::{Context, Result};
pub enum Alignment {
Left,
@@ -15,132 +12,95 @@ pub enum Alignment {
pub trait MarkupExporter {
fn table_results(
&self,
entries: &[BenchmarkResultWithRelativeSpeed],
time_unit: TimeUnit,
reference_pending: bool,
explicit_reference: bool,
results: &[BenchmarkResult],
summaries: &[Stats],
primary: MetricSelection,
unit: Unit,
) -> String {
// prepare table header strings
let notation = format!("[{}]", time_unit.short_name());
// prepare table cells alignment
let notation = format!("[{}]", unit.symbol);
let cells_alignment = [
Alignment::Left,
Alignment::Right,
Alignment::Right,
Alignment::Right,
Alignment::Right,
Alignment::Left,
];
// emit table header format
let mut table = self.table_header(&cells_alignment);
// emit table header data
table.push_str(&self.table_row(&[
"Command",
&format!("Mean {notation}"),
&format!("Min {notation}"),
&format!("Max {notation}"),
"Relative",
&format!("Mean {} {notation}", primary.metric.label()),
"Change",
"Factor",
]));
// emit horizontal line
table.push_str(&self.table_divider(&cells_alignment));
for entry in entries {
let result = &entry.result;
// prepare data row strings
let cmd_str = result.display_name.replace('|', "\\|");
let mean_str = result.mean_wall_clock_time().format_value(time_unit);
let stddev_str = if let Some(stddev) = result.measurements.stddev() {
format!(" ± {}", stddev.format_value(time_unit))
for (index, (result, stats)) in results.iter().zip(summaries).enumerate() {
let command = result.display_name.replace('|', "\\|");
let mean = unit.format_value(stats.mean);
let stddev = stats
.stddev
.map(|value| format!(" ± {}", unit.format_value(value)))
.unwrap_or_default();
let change = if index == 0 {
String::new()
} else {
"".into()
stats.format_change_from(&summaries[0])
};
let min_str = result.measurements.min().format_value(time_unit);
let max_str = result.measurements.max().format_value(time_unit);
// The ratio of two zero times is undefined, even if they compare equal.
let relative_unavailable = reference_pending
|| (explicit_reference
&& !entry.is_reference
&& entry.relative_ordering == Ordering::Equal
&& result.mean_wall_clock_time() == Time::zero());
let relative = if relative_unavailable {
"N/A".to_string()
let factor = if index == 0 {
String::new()
} else {
let stddev = match (entry.is_reference, entry.relative_speed_stddev) {
(false, Some(stddev)) => format!(" ± {stddev:.2}"),
_ => String::new(),
};
let label = if explicit_reference {
entry.reference_label()
} else {
""
};
format!("{:.2}{stddev}{label}", entry.relative_speed)
stats.format_factor_from(&summaries[0], primary.metric)
};
// prepare table row entries
table.push_str(&self.table_row(&[
&self.command(&cmd_str),
&format!("{mean_str}{stddev_str}"),
&min_str,
&max_str,
&relative,
]))
&self.command(&command),
&format!("{mean}{stddev}"),
&change,
&factor,
]));
}
// emit table footer format
table.push_str(&self.table_footer(&cells_alignment));
table
}
fn table_row(&self, cells: &[&str]) -> String;
fn table_divider(&self, cell_aligmnents: &[Alignment]) -> String;
fn table_divider(&self, cell_alignments: &[Alignment]) -> String;
fn table_header(&self, _cell_aligmnents: &[Alignment]) -> String {
"".to_string()
fn table_header(&self, _cell_alignments: &[Alignment]) -> String {
String::new()
}
fn table_footer(&self, _cell_aligmnents: &[Alignment]) -> String {
"".to_string()
fn table_footer(&self, _cell_alignments: &[Alignment]) -> String {
String::new()
}
fn command(&self, size: &str) -> String;
}
fn determine_unit_from_results(results: &[BenchmarkResult]) -> TimeUnit {
if let Some(first_result) = results.first() {
// Use the first BenchmarkResult entry to determine the unit for all entries.
first_result.mean_wall_clock_time().suitable_unit()
} else {
// Default to `Second`.
TimeUnit::Second
}
fn command(&self, command: &str) -> String;
}
impl<T: MarkupExporter> Exporter for T {
fn serialize(
&self,
results: &[BenchmarkResult],
time_unit: Option<TimeUnit>,
sort_order: SortOrder,
reference_index: Option<usize>,
) -> Result<Vec<u8>> {
let unit = time_unit.unwrap_or_else(|| determine_unit_from_results(results));
// Do not report ratios against another benchmark while the selected
// reference is still pending in an intermediate export.
let reference_pending = reference_index.is_some_and(|i| i >= results.len());
let entries = relative_speed::compute(
results,
sort_order,
reference_index.and_then(|i| results.get(i)),
);
let table =
self.table_results(&entries, unit, reference_pending, reference_index.is_some());
Ok(table.as_bytes().to_vec())
fn serialize(&self, results: &[BenchmarkResult], primary: MetricSelection) -> Result<Vec<u8>> {
let stats: Vec<_> = results
.iter()
.map(|result| {
primary
.metric
.summarize(&result.measurements)
.with_context(|| {
format!(
"Metric '{}' is unavailable for '{}'",
primary.metric.name(),
result.get_name()
)
})
})
.collect::<Result<_>>()?;
// Use one common unit, avoiding an unscaled column if the reference is zero.
let mean = stats
.iter()
.find(|stats| stats.mean != 0.0)
.map_or(0.0, |stats| stats.mean);
let unit = primary.display_unit(mean);
Ok(self
.table_results(results, &stats, primary, unit)
.into_bytes())
}
}
src/export/mod.rs
+7 -30
View File
@@ -17,9 +17,8 @@ use self::markdown::MarkdownExporter;
use self::orgmode::OrgmodeExporter;
use crate::benchmark::benchmark_result::BenchmarkResult;
use crate::options::SortOrder;
use crate::metric::MetricSelection;
use crate::output::console_writeln;
use crate::quantity::TimeUnit;
use anyhow::{Context, Result};
use clap::ArgMatches;
@@ -46,13 +45,7 @@ pub enum ExportType {
/// Interface for different exporters.
trait Exporter {
/// Export the given entries in the serialized form.
fn serialize(
&self,
results: &[BenchmarkResult],
time_unit: Option<TimeUnit>,
sort_order: SortOrder,
reference_index: Option<usize>,
) -> Result<Vec<u8>>;
fn serialize(&self, results: &[BenchmarkResult], primary: MetricSelection) -> Result<Vec<u8>>;
}
pub enum ExportTarget {
@@ -68,22 +61,16 @@ struct ExporterWithTarget {
/// Handles the management of multiple file exporters.
pub struct ExportManager {
exporters: Vec<ExporterWithTarget>,
time_unit: Option<TimeUnit>,
sort_order: SortOrder,
primary: MetricSelection,
}
impl ExportManager {
/// Build the ExportManager that will export the results specified
/// in the given ArgMatches
pub fn from_cli_arguments(
matches: &ArgMatches,
time_unit: Option<TimeUnit>,
sort_order: SortOrder,
) -> Result<Self> {
pub fn from_cli_arguments(matches: &ArgMatches, primary: MetricSelection) -> Result<Self> {
let mut export_manager = Self {
exporters: vec![],
time_unit,
sort_order,
primary,
};
{
let mut add_exporter = |flag, exporttype| -> Result<()> {
@@ -131,19 +118,9 @@ impl ExportManager {
/// results are written to all file targets (to always have them up to date, even
/// if a benchmark fails). In the latter case, we only print to stdout targets (in
/// order not to clutter the output of hyperfine with intermediate results).
/// `reference_index` refers to the complete benchmark sequence, so it can
/// point past `results` while that benchmark is still pending.
pub fn write_results(
&self,
results: &[BenchmarkResult],
intermediate: bool,
reference_index: Option<usize>,
) -> Result<()> {
pub fn write_results(&self, results: &[BenchmarkResult], intermediate: bool) -> Result<()> {
for e in &self.exporters {
let content = || {
e.exporter
.serialize(results, self.time_unit, self.sort_order, reference_index)
};
let content = || e.exporter.serialize(results, self.primary);
match e.target {
ExportTarget::File(ref filename) => {
src/export/tests.rs
+138 -125
View File
@@ -2,19 +2,27 @@ use super::Exporter;
use crate::benchmark::benchmark_result::BenchmarkResult;
use crate::benchmark::measurement::{Measurement, Measurements};
use crate::export::asciidoc::AsciidocExporter;
use crate::export::markdown::MarkdownExporter;
use crate::export::orgmode::OrgmodeExporter;
use crate::quantity::{byte, second, Information, Time, TimeUnit, Zero};
use crate::{export::markdown::MarkdownExporter, options::SortOrder};
use crate::metric::{MetricSelection, Unit};
use crate::quantity::{byte, second, Information, Time, Zero};
use std::collections::BTreeMap;
use std::process::ExitStatus;
fn get_output<E: Exporter + Default>(
results: &[BenchmarkResult],
unit: Option<TimeUnit>,
sort_order: SortOrder,
) -> String {
fn get_output<E: Exporter + Default>(results: &[BenchmarkResult], unit: Option<Unit>) -> String {
let exporter = E::default();
String::from_utf8(exporter.serialize(results, unit, sort_order, None).unwrap()).unwrap()
String::from_utf8(
exporter
.serialize(
results,
MetricSelection {
unit,
..MetricSelection::default()
},
)
.unwrap(),
)
.unwrap()
}
/// Ensure the makrup output includes the table header and the multiple
@@ -33,6 +41,7 @@ fn test_markup_export_auto_ms() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(0.09),
time_cpu: Time::new::<second>(0.09),
time_user: Time::new::<second>(0.09),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -41,6 +50,7 @@ fn test_markup_export_auto_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.10),
time_cpu: Time::new::<second>(0.10),
time_user: Time::new::<second>(0.10),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -49,6 +59,7 @@ fn test_markup_export_auto_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.14),
time_cpu: Time::new::<second>(0.14),
time_user: Time::new::<second>(0.14),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -65,6 +76,7 @@ fn test_markup_export_auto_ms() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(2.0),
time_cpu: Time::new::<second>(2.0),
time_user: Time::new::<second>(2.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -73,6 +85,7 @@ fn test_markup_export_auto_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(3.0),
time_cpu: Time::new::<second>(3.0),
time_user: Time::new::<second>(3.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -81,6 +94,7 @@ fn test_markup_export_auto_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(4.0),
time_cpu: Time::new::<second>(4.0),
time_user: Time::new::<second>(4.0),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -92,42 +106,39 @@ fn test_markup_export_auto_ms() {
},
];
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, None, SortOrder::Command), @r#"
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 0.1` | 110.0 ± 26.5 | 90.0 | 140.0 | 1.00 |
| `sleep 2` | 3000.0 ± 1000.0 | 2000.0 | 4000.0 | 27.27 ± 11.21 |
"#);
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, None), @"
| Command | Mean Wall Time [ms] | Change | Factor |
|:---|---:|---:|:---|
| `sleep 0.1` | 110.0 ± 26.5 | | |
| `sleep 2` | 3000.0 ± 1000.0 | +2627.3% | (27.3x slower) |
");
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, None, SortOrder::Command), @r#"
[cols="<,>,>,>,>"]
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, None), @r#"
[cols="<,>,>,<"]
|===
| Command
| Mean [ms]
| Min [ms]
| Max [ms]
| Relative
| Command
| Mean Wall Time [ms]
| Change
| Factor
| `sleep 0.1`
| 110.0 ± 26.5
| 90.0
| 140.0
| 1.00
| `sleep 0.1`
| 110.0 ± 26.5
|
|
| `sleep 2`
| 3000.0 ± 1000.0
| 2000.0
| 4000.0
| 27.27 ± 11.21
| `sleep 2`
| 3000.0 ± 1000.0
| +2627.3%
| (27.3x slower)
|===
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&results, None, SortOrder::Command), @r#"
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|--+--+--+--+--|
| =sleep 0.1= | 110.0 ± 26.5 | 90.0 | 140.0 | 1.00 |
| =sleep 2= | 3000.0 ± 1000.0 | 2000.0 | 4000.0 | 27.27 ± 11.21 |
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&results, None), @"
| Command | Mean Wall Time [ms] | Change | Factor |
|--+--+--+--|
| =sleep 0.1= | 110.0 ± 26.5 | | |
| =sleep 2= | 3000.0 ± 1000.0 | +2627.3% | (27.3x slower) |
");
}
/// This (again) demonstrates that the first entry's units (s) are used to set
@@ -142,6 +153,7 @@ fn test_markup_export_auto_s() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(2.1),
time_cpu: Time::new::<second>(2.1),
time_user: Time::new::<second>(2.1),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -150,6 +162,7 @@ fn test_markup_export_auto_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.2),
time_cpu: Time::new::<second>(2.2),
time_user: Time::new::<second>(2.2),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -158,6 +171,7 @@ fn test_markup_export_auto_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.3),
time_cpu: Time::new::<second>(2.3),
time_user: Time::new::<second>(2.3),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -174,6 +188,7 @@ fn test_markup_export_auto_s() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(0.1),
time_cpu: Time::new::<second>(0.1),
time_user: Time::new::<second>(0.1),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -182,6 +197,7 @@ fn test_markup_export_auto_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.2),
time_cpu: Time::new::<second>(0.2),
time_user: Time::new::<second>(0.2),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -190,6 +206,7 @@ fn test_markup_export_auto_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.3),
time_cpu: Time::new::<second>(0.3),
time_user: Time::new::<second>(0.3),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -201,42 +218,39 @@ fn test_markup_export_auto_s() {
},
];
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, None, SortOrder::Command), @r#"
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 2` | 2.200 ± 0.100 | 2.100 | 2.300 | 11.00 ± 5.52 |
| `sleep 0.1` | 0.200 ± 0.100 | 0.100 | 0.300 | 1.00 |
"#);
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, None), @"
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `sleep 2` | 2.200 ± 0.100 | | |
| `sleep 0.1` | 0.200 ± 0.100 | -90.9% | (11.0x faster) |
");
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, None, SortOrder::Command), @r#"
[cols="<,>,>,>,>"]
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, None), @r#"
[cols="<,>,>,<"]
|===
| Command
| Mean [s]
| Min [s]
| Max [s]
| Relative
| Command
| Mean Wall Time [s]
| Change
| Factor
| `sleep 2`
| 2.200 ± 0.100
| 2.100
| 2.300
| 11.00 ± 5.52
| `sleep 2`
| 2.200 ± 0.100
|
|
| `sleep 0.1`
| 0.200 ± 0.100
| 0.100
| 0.300
| 1.00
| `sleep 0.1`
| 0.200 ± 0.100
| -90.9%
| (11.0x faster)
|===
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&results, None, SortOrder::Command), @r#"
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|--+--+--+--+--|
| =sleep 2= | 2.200 ± 0.100 | 2.100 | 2.300 | 11.00 ± 5.52 |
| =sleep 0.1= | 0.200 ± 0.100 | 0.100 | 0.300 | 1.00 |
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&results, None), @"
| Command | Mean Wall Time [s] | Change | Factor |
|--+--+--+--|
| =sleep 2= | 2.200 ± 0.100 | | |
| =sleep 0.1= | 0.200 ± 0.100 | -90.9% | (11.0x faster) |
");
}
/// This (again) demonstrates that the given time unit (ms) is used to set
@@ -251,6 +265,7 @@ fn test_markup_export_manual_ms() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(2.1),
time_cpu: Time::new::<second>(2.1),
time_user: Time::new::<second>(2.1),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -259,6 +274,7 @@ fn test_markup_export_manual_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.2),
time_cpu: Time::new::<second>(2.2),
time_user: Time::new::<second>(2.2),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -267,6 +283,7 @@ fn test_markup_export_manual_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.3),
time_cpu: Time::new::<second>(2.3),
time_user: Time::new::<second>(2.3),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -283,6 +300,7 @@ fn test_markup_export_manual_ms() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(0.1),
time_cpu: Time::new::<second>(0.1),
time_user: Time::new::<second>(0.1),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -291,6 +309,7 @@ fn test_markup_export_manual_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.2),
time_cpu: Time::new::<second>(0.2),
time_user: Time::new::<second>(0.2),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -299,6 +318,7 @@ fn test_markup_export_manual_ms() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.3),
time_cpu: Time::new::<second>(0.3),
time_user: Time::new::<second>(0.3),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -310,42 +330,39 @@ fn test_markup_export_manual_ms() {
},
];
insta::assert_snapshot!(get_output::<MarkdownExporter>(&timing_results, Some(TimeUnit::MilliSecond), SortOrder::Command), @r#"
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 2` | 2200.0 ± 100.0 | 2100.0 | 2300.0 | 11.00 ± 5.52 |
| `sleep 0.1` | 200.0 ± 100.0 | 100.0 | 300.0 | 1.00 |
"#);
insta::assert_snapshot!(get_output::<MarkdownExporter>(&timing_results, Some(Unit { symbol: "ms", scale: 0.001 })), @"
| Command | Mean Wall Time [ms] | Change | Factor |
|:---|---:|---:|:---|
| `sleep 2` | 2200.0 ± 100.0 | | |
| `sleep 0.1` | 200.0 ± 100.0 | -90.9% | (11.0x faster) |
");
insta::assert_snapshot!(get_output::<AsciidocExporter>(&timing_results, Some(TimeUnit::MilliSecond), SortOrder::Command), @r#"
[cols="<,>,>,>,>"]
insta::assert_snapshot!(get_output::<AsciidocExporter>(&timing_results, Some(Unit { symbol: "ms", scale: 0.001 })), @r#"
[cols="<,>,>,<"]
|===
| Command
| Mean [ms]
| Min [ms]
| Max [ms]
| Relative
| Command
| Mean Wall Time [ms]
| Change
| Factor
| `sleep 2`
| 2200.0 ± 100.0
| 2100.0
| 2300.0
| 11.00 ± 5.52
| `sleep 2`
| 2200.0 ± 100.0
|
|
| `sleep 0.1`
| 200.0 ± 100.0
| 100.0
| 300.0
| 1.00
| `sleep 0.1`
| 200.0 ± 100.0
| -90.9%
| (11.0x faster)
|===
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&timing_results, Some(TimeUnit::MilliSecond), SortOrder::Command), @r#"
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|--+--+--+--+--|
| =sleep 2= | 2200.0 ± 100.0 | 2100.0 | 2300.0 | 11.00 ± 5.52 |
| =sleep 0.1= | 200.0 ± 100.0 | 100.0 | 300.0 | 1.00 |
"#);
insta::assert_snapshot!(get_output::<OrgmodeExporter>(&timing_results, Some(Unit { symbol: "ms", scale: 0.001 })), @"
| Command | Mean Wall Time [ms] | Change | Factor |
|--+--+--+--|
| =sleep 2= | 2200.0 ± 100.0 | | |
| =sleep 0.1= | 200.0 ± 100.0 | -90.9% | (11.0x faster) |
");
}
/// The given time unit (s) is used to set the units for all entries.
@@ -359,6 +376,7 @@ fn test_markup_export_manual_s() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(2.01),
time_cpu: Time::new::<second>(2.01),
time_user: Time::new::<second>(2.01),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -367,6 +385,7 @@ fn test_markup_export_manual_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.02),
time_cpu: Time::new::<second>(2.02),
time_user: Time::new::<second>(2.02),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -375,6 +394,7 @@ fn test_markup_export_manual_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(2.03),
time_cpu: Time::new::<second>(2.03),
time_user: Time::new::<second>(2.03),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -391,6 +411,7 @@ fn test_markup_export_manual_s() {
measurements: Measurements::new(vec![
Measurement {
time_wall_clock: Time::new::<second>(0.11),
time_cpu: Time::new::<second>(0.11),
time_user: Time::new::<second>(0.11),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -399,6 +420,7 @@ fn test_markup_export_manual_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.12),
time_cpu: Time::new::<second>(0.12),
time_user: Time::new::<second>(0.12),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -407,6 +429,7 @@ fn test_markup_export_manual_s() {
},
Measurement {
time_wall_clock: Time::new::<second>(0.13),
time_cpu: Time::new::<second>(0.13),
time_user: Time::new::<second>(0.13),
time_system: Time::zero(),
memory_peak_resident: Some(Information::new::<byte>(1024.)),
@@ -418,40 +441,30 @@ fn test_markup_export_manual_s() {
},
];
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, Some(TimeUnit::Second), SortOrder::Command), @r#"
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 2` | 2.020 ± 0.010 | 2.010 | 2.030 | 16.83 ± 1.41 |
| `sleep 0.1` | 0.120 ± 0.010 | 0.110 | 0.130 | 1.00 |
"#);
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, Some(Unit { symbol: "s", scale: 1.0 })), @"
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `sleep 2` | 2.020 ± 0.010 | | |
| `sleep 0.1` | 0.120 ± 0.010 | -94.1% | (16.8x faster) |
");
insta::assert_snapshot!(get_output::<MarkdownExporter>(&results, Some(TimeUnit::Second), SortOrder::MeanTime), @r#"
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 0.1` | 0.120 ± 0.010 | 0.110 | 0.130 | 1.00 |
| `sleep 2` | 2.020 ± 0.010 | 2.010 | 2.030 | 16.83 ± 1.41 |
"#);
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, Some(TimeUnit::Second), SortOrder::Command), @r#"
[cols="<,>,>,>,>"]
insta::assert_snapshot!(get_output::<AsciidocExporter>(&results, Some(Unit { symbol: "s", scale: 1.0 })), @r#"
[cols="<,>,>,<"]
|===
| Command
| Mean [s]
| Min [s]
| Max [s]
| Relative
| Command
| Mean Wall Time [s]
| Change
| Factor
| `sleep 2`
| 2.020 ± 0.010
| 2.010
| 2.030
| 16.83 ± 1.41
| `sleep 2`
| 2.020 ± 0.010
|
|
| `sleep 0.1`
| 0.120 ± 0.010
| 0.110
| 0.130
| 1.00
| `sleep 0.1`
| 0.120 ± 0.010
| -94.1%
| (16.8x faster)
|===
"#);
}
src/main.rs
+2 -6
View File
@@ -21,6 +21,7 @@ pub mod cli;
pub mod command;
pub mod error;
pub mod export;
pub mod metric;
pub mod options;
pub mod outlier_detection;
pub mod output;
@@ -39,15 +40,10 @@ fn run() -> Result<()> {
let commands = Commands::from_cli_arguments(&cli_arguments)?;
options.validate_against_command_list(&commands)?;
let export_manager = ExportManager::from_cli_arguments(
&cli_arguments,
options.time_unit,
options.sort_order_exports,
)?;
let export_manager = ExportManager::from_cli_arguments(&cli_arguments, options.metrics[0])?;
let mut scheduler = Scheduler::new(&commands, &options, &export_manager);
scheduler.run_benchmarks()?;
scheduler.print_relative_speed_comparison()?;
scheduler.final_export()?;
Ok(())
src/metric.rs
+505
View File
@@ -0,0 +1,505 @@
//! Metric selection, descriptive statistics, and display units.
use anyhow::{bail, ensure, Result};
use crate::benchmark::measurement::{Measurement, Measurements};
use crate::quantity::statistics::{max, mean, median, min, standard_deviation};
use crate::quantity::{byte, ratio, second, Ratio};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Metric {
TimeWallClock,
TimeCpu,
TimeUser,
TimeSystem,
MemoryPeakResident,
CpuCycles,
Instructions,
CacheReferences,
CacheMisses,
BranchMisses,
}
impl Metric {
pub const ALL: [Self; 10] = [
Self::TimeWallClock,
Self::TimeCpu,
Self::TimeUser,
Self::TimeSystem,
Self::MemoryPeakResident,
Self::CpuCycles,
Self::Instructions,
Self::CacheReferences,
Self::CacheMisses,
Self::BranchMisses,
];
pub fn name(self) -> &'static str {
match self {
Self::TimeWallClock => "time_wall_clock",
Self::TimeCpu => "time_cpu",
Self::TimeUser => "time_user",
Self::TimeSystem => "time_system",
Self::MemoryPeakResident => "memory_peak_resident",
Self::CpuCycles => "cpu_cycles",
Self::Instructions => "instructions",
Self::CacheReferences => "cache_references",
Self::CacheMisses => "cache_misses",
Self::BranchMisses => "branch_misses",
}
}
pub fn label(self) -> &'static str {
match self {
Self::TimeWallClock => "Wall Time",
Self::TimeCpu => "CPU time",
Self::TimeUser => "User time",
Self::TimeSystem => "System time",
Self::MemoryPeakResident => "Memory",
Self::CpuCycles => "CPU cycles",
Self::Instructions => "Instructions",
Self::CacheReferences => "Cache references",
Self::CacheMisses => "Cache misses",
Self::BranchMisses => "Branch misses",
}
}
/// Extract a value in canonical units: seconds, bytes, or counts.
pub fn value(self, measurement: &Measurement) -> Option<f64> {
let counters = &measurement.hardware_counters;
match self {
Self::TimeWallClock => Some(measurement.time_wall_clock.get::<second>()),
Self::TimeCpu => Some(measurement.time_cpu.get::<second>()),
Self::TimeUser => Some(measurement.time_user.get::<second>()),
Self::TimeSystem => Some(measurement.time_system.get::<second>()),
Self::MemoryPeakResident => measurement.memory_peak_resident.map(|v| v.get::<byte>()),
Self::CpuCycles => counters.cpu_cycles.map(|v| v as f64),
Self::Instructions => counters.instructions.map(|v| v as f64),
Self::CacheReferences => counters.cache_references.map(|v| v as f64),
Self::CacheMisses => counters.cache_misses.map(|v| v as f64),
Self::BranchMisses => counters.branch_misses.map(|v| v as f64),
}
}
pub fn is_time(self) -> bool {
matches!(
self,
Self::TimeWallClock | Self::TimeCpu | Self::TimeUser | Self::TimeSystem
)
}
pub fn is_counter(self) -> bool {
!self.is_time() && self != Self::MemoryPeakResident
}
/// Reject known unsupported configurations before any commands execute.
/// Availability that depends on permissions or hardware is checked after collection.
pub fn ensure_available(self, shell: bool) -> Result<()> {
if self.is_counter() {
ensure!(
!shell,
"Metric '{}' is unavailable when a shell is enabled",
self.name()
);
ensure!(
cfg!(any(
target_os = "macos",
all(
target_os = "linux",
target_pointer_width = "64",
any(
target_arch = "aarch64",
target_arch = "x86_64",
target_arch = "riscv64",
target_arch = "powerpc64",
target_arch = "loongarch64"
)
)
)),
"Metric '{}' is unavailable on this platform",
self.name()
);
ensure!(
!cfg!(target_os = "macos") || matches!(self, Self::CpuCycles | Self::Instructions),
"Metric '{}' is unavailable on macOS",
self.name()
);
}
ensure!(
!cfg!(windows) || self != Self::MemoryPeakResident,
"Metric '{}' is unavailable on Windows",
self.name()
);
Ok(())
}
pub fn base_unit(self) -> Unit {
if self.is_time() {
Unit::new("s", 1.0)
} else if self.is_counter() {
Unit::new("count", 1.0)
} else {
Unit::new("B", 1.0)
}
}
fn units(self) -> &'static [Unit] {
if self.is_time() {
TIME_UNITS
} else if self.is_counter() {
COUNT_UNITS
} else {
MEMORY_UNITS
}
}
/// Require complete coverage so different metrics describe the same runs.
pub fn summarize(self, measurements: &Measurements) -> Option<Stats> {
let values = measurements
.measurements
.iter()
.map(|measurement| self.value(measurement))
.collect::<Option<Vec<_>>>()?;
if values.is_empty() {
return None;
}
Some(Stats {
count: values.len(),
mean: mean(values.iter().copied()),
stddev: (values.len() > 1).then(|| {
standard_deviation(values.iter().map(|&v| Ratio::new::<ratio>(v))).get::<ratio>()
}),
median: median(values.iter().copied()),
min: min(values.iter().copied()),
max: max(values.iter().copied()),
})
}
}
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct MetricSelection {
pub metric: Metric,
pub unit: Option<Unit>,
}
impl Default for MetricSelection {
fn default() -> Self {
Self {
metric: Metric::TimeWallClock,
unit: None,
}
}
}
impl MetricSelection {
/// Parse an ordered, nonempty list of `METRIC[:UNIT]` selections.
pub fn parse_list(input: &str) -> Result<Vec<Self>> {
let mut selections = Vec::new();
for item in input.split(',') {
let (name, unit_name) = match item.split_once(':') {
Some((name, unit)) => (name, Some(unit)),
None => (item, None),
};
let metric = match Metric::ALL.iter().copied().find(|m| m.name() == name) {
Some(metric) => metric,
None => bail!(
"Unknown metric '{name}'. Expected one of: {}",
Metric::ALL.map(Metric::name).join(", ")
),
};
if selections
.iter()
.any(|selection: &Self| selection.metric == metric)
{
bail!("Metric '{name}' was selected more than once");
}
let unit = match unit_name {
None => None,
Some(unit_name) => {
// Accept an ASCII spelling for microseconds as well.
let unit_name = if unit_name == "us" { "µs" } else { unit_name };
match metric.units().iter().find(|unit| unit.symbol == unit_name) {
Some(unit) => Some(*unit),
None => bail!(
"Unit '{unit_name}' is not valid for metric '{name}'. Expected one of: {}",
metric.units().iter().map(|u| u.symbol).collect::<Vec<_>>().join(", ")
),
}
}
};
selections.push(Self { metric, unit });
}
Ok(selections)
}
pub fn display_unit(self, mean: f64) -> Unit {
if let Some(unit) = self.unit {
return unit;
}
if mean == 0.0 {
return self.metric.base_unit();
}
let units = if self.metric == Metric::MemoryPeakResident {
BINARY_MEMORY_UNITS
} else {
self.metric.units()
};
units
.iter()
.rev()
.find(|unit| mean.abs() >= unit.scale)
.copied()
.unwrap_or(units[0])
}
/// CSV never automatically scales values.
pub fn csv_unit(self) -> Unit {
self.unit.unwrap_or_else(|| self.metric.base_unit())
}
}
#[derive(Debug, Clone, Copy, PartialEq)]
pub struct Unit {
pub symbol: &'static str,
/// The number of canonical units in this unit (e.g. 1e9 for a billion).
pub scale: f64,
}
impl Unit {
const fn new(symbol: &'static str, scale: f64) -> Self {
Self { symbol, scale }
}
pub fn format(self, value: f64) -> String {
let value = self.format_value(value);
if self.symbol == "count" {
value
} else {
format!("{value} {}", self.symbol)
}
}
pub fn format_value(self, value: f64) -> String {
let precision = if self.symbol == "s" { 3 } else { 1 };
format!("{:.precision$}", value / self.scale)
}
}
const TIME_UNITS: &[Unit] = &[
Unit::new("ns", 1e-9),
Unit::new("µs", 1e-6),
Unit::new("ms", 1e-3),
Unit::new("s", 1.0),
Unit::new("min", 60.0),
Unit::new("h", 3600.0),
];
const BINARY_MEMORY_UNITS: &[Unit] = &[
Unit::new("B", 1.0),
Unit::new("KiB", 1024.0),
Unit::new("MiB", 1_048_576.0),
Unit::new("GiB", 1_073_741_824.0),
Unit::new("TiB", 1_099_511_627_776.0),
];
const MEMORY_UNITS: &[Unit] = &[
Unit::new("B", 1.0),
Unit::new("kB", 1e3),
Unit::new("MB", 1e6),
Unit::new("GB", 1e9),
Unit::new("TB", 1e12),
Unit::new("KiB", 1024.0),
Unit::new("MiB", 1_048_576.0),
Unit::new("GiB", 1_073_741_824.0),
Unit::new("TiB", 1_099_511_627_776.0),
];
const COUNT_UNITS: &[Unit] = &[
Unit::new("count", 1.0),
Unit::new("k", 1e3),
Unit::new("M", 1e6),
Unit::new("B", 1e9),
];
#[derive(Debug, Clone, Copy)]
pub struct Stats {
pub count: usize,
pub mean: f64,
pub stddev: Option<f64>,
pub median: f64,
pub min: f64,
pub max: f64,
}
impl Stats {
pub fn change_from(&self, reference: &Self) -> Option<f64> {
(reference.mean != 0.0).then(|| 100.0 * (self.mean / reference.mean - 1.0))
}
pub fn format_change_from(&self, reference: &Self) -> String {
let Some(change) = self.change_from(reference) else {
return "N/A".to_owned();
};
let mut formatted = format!("{change:+.1}%");
if formatted == "+0.0%" || formatted == "-0.0%" {
formatted = "0.0%".to_owned();
}
formatted
}
pub fn format_factor_from(&self, reference: &Self, metric: Metric) -> String {
if reference.mean > 0.0
&& (self.mean >= 1.3 * reference.mean || self.mean <= 0.7 * reference.mean)
{
let decrease = self.mean < reference.mean;
let factor = if decrease {
reference.mean / self.mean
} else {
self.mean / reference.mean
};
let direction = match (metric.is_time(), decrease) {
(true, true) => "faster",
(true, false) => "slower",
(false, true) => "less",
(false, false) => "more",
};
let factor = if factor.is_infinite() {
"∞".to_owned()
} else {
format!("{factor:.1}")
};
format!("({factor}x {direction})")
} else {
String::new()
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::quantity::{Information, Time};
#[test]
fn large_changes_include_factors() {
let reference = Stats {
count: 1,
mean: 100.0,
stddev: None,
median: 100.0,
min: 100.0,
max: 100.0,
};
for (mean, metric, expected) in [
(130.0, Metric::TimeWallClock, "+30.0% (1.3x slower)"),
(70.0, Metric::TimeCpu, "-30.0% (1.4x faster)"),
(10.0, Metric::Instructions, "-90.0% (10.0x less)"),
(1000.0, Metric::MemoryPeakResident, "+900.0% (10.0x more)"),
(129.99, Metric::TimeWallClock, "+30.0%"),
(70.01, Metric::Instructions, "-30.0%"),
(100.0, Metric::Instructions, "0.0%"),
(0.0, Metric::Instructions, "-100.0% (∞x less)"),
] {
let stats = Stats { mean, ..reference };
let (percentage, factor) = expected
.split_once(" (")
.map_or((expected, String::new()), |(p, f)| (p, format!("({f}")));
assert_eq!(stats.format_change_from(&reference), percentage);
assert_eq!(stats.format_factor_from(&reference, metric), factor);
}
let zero = Stats {
mean: 0.0,
..reference
};
assert_eq!(reference.format_change_from(&zero), "N/A");
}
#[test]
fn cpu_time_statistics_use_per_run_totals() {
let measurements = Measurements::new(
[(1.0, 3.0), (3.0, 1.0)]
.iter()
.map(|&(user, system)| Measurement {
time_cpu: Time::new::<second>(user + system),
time_user: Time::new::<second>(user),
time_system: Time::new::<second>(system),
..Measurement::default()
})
.collect(),
);
let selection = MetricSelection::parse_list("time_cpu:ms").unwrap()[0];
let stats = selection.metric.summarize(&measurements).unwrap();
assert_eq!(selection.csv_unit().format(stats.mean), "4000.0 ms");
// User and system time vary, but their per-run total is constant.
assert_eq!(stats.stddev, Some(0.0));
assert_eq!(stats.min, 4.0);
assert_eq!(stats.max, 4.0);
let json = serde_json::to_value(&measurements).unwrap();
for sample in json["measurements"].as_array().unwrap() {
assert_eq!(sample["time_cpu"]["value"], 4.0);
assert_eq!(sample["time_cpu"]["unit"], "second");
}
}
#[test]
fn selection_units_and_validation() {
let selections = MetricSelection::parse_list(
"instructions:B,memory_peak_resident:B,time_wall_clock:ms,time_user:us",
)
.unwrap();
assert_eq!(selections[0].csv_unit().scale, 1e9);
assert_eq!(selections[1].csv_unit().scale, 1.0);
assert_eq!(selections[2].csv_unit().format(0.125), "125.0 ms");
assert_eq!(selections[3].csv_unit().symbol, "µs");
let instructions = MetricSelection::parse_list("instructions").unwrap()[0];
assert_eq!(instructions.csv_unit().symbol, "count");
assert_eq!(
instructions.display_unit(143.2e9).format(143.2e9),
"143.2 B"
);
for invalid in [
"",
"unknown",
"time_wall_clock,",
"instructions,instructions:M",
"instructions:MiB",
"memory_peak_resident:ms",
"time_wall_clock:",
] {
assert!(MetricSelection::parse_list(invalid).is_err(), "{}", invalid);
}
}
#[test]
fn summaries_require_complete_samples() {
let mut measurements = Measurements::new(vec![Measurement {
memory_peak_resident: Some(Information::new::<byte>(10.0)),
..Measurement::default()
}]);
let single = Metric::MemoryPeakResident.summarize(&measurements).unwrap();
assert_eq!(single.stddev, None);
measurements.push(Measurement::default());
assert!(Metric::MemoryPeakResident
.summarize(&measurements)
.is_none());
measurements.measurements[1].memory_peak_resident = Some(Information::new::<byte>(30.0));
let stats = Metric::MemoryPeakResident.summarize(&measurements).unwrap();
assert_eq!(stats.mean, 20.0);
assert_eq!(stats.change_from(&single), Some(100.0));
assert_eq!(stats.format_change_from(&single), "+100.0%");
assert_eq!(single.format_change_from(&stats), "-50.0%");
assert_eq!(
Stats {
mean: 9.999,
..single
}
.format_change_from(&single),
"0.0%"
);
approx::assert_relative_eq!(stats.stddev.unwrap(), 200.0_f64.sqrt());
let zero = Metric::TimeWallClock.summarize(&measurements).unwrap();
assert_eq!(zero.change_from(&zero), None);
assert_eq!(zero.format_change_from(&zero), "N/A");
}
}
src/options.rs
+54 -90
View File
@@ -4,12 +4,13 @@ use std::path::PathBuf;
use std::process::{Command, Stdio};
use std::{cmp, env, fmt, io};
use anyhow::{bail, ensure};
use anyhow::ensure;
use clap::ArgMatches;
use crate::command::Commands;
use crate::error::OptionsError;
use crate::quantity::{second, Time, TimeUnit};
use crate::metric::{Metric, MetricSelection};
use crate::quantity::{second, Time};
use anyhow::Result;
@@ -97,12 +98,6 @@ pub enum OutputStyleOption {
Disabled,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SortOrder {
Command,
MeanTime,
}
/// Bounds for the number of benchmark runs
pub struct RunBounds {
/// Minimum number of benchmark runs
@@ -202,36 +197,21 @@ pub struct Options {
/// Whether or not to ignore non-zero exit codes
pub command_failure_action: CmdFailureAction,
// Standalone reference command, cleared when an existing benchmark is selected.
pub reference_command: Option<String>,
// Name of the reference command
pub reference_name: Option<String>,
// Index of the reference in the benchmark sequence, if any.
pub reference_index: Option<usize>,
/// Command(s) to run before each timing run
/// Command(s) to run before each benchmark run
pub preparation_command: Option<Vec<String>>,
/// Command(s) to run after each timing run
/// Command(s) to run after each benchmark run
pub conclusion_command: Option<Vec<String>>,
/// Command to run before each *batch* of timing runs, i.e. before each individual benchmark
/// Command to run before each *batch* of benchmark runs, i.e. before each individual benchmark
pub setup_command: Option<String>,
/// Command to run after each *batch* of timing runs, i.e. after each individual benchmark
/// Command to run after each *batch* of benchmark runs, i.e. after each individual benchmark
pub cleanup_command: Option<String>,
/// What color mode to use for the terminal output
pub output_style: OutputStyleOption,
/// How to order benchmarks in the relative speed comparison
pub sort_order_speed_comparison: SortOrder,
/// How to order benchmarks in the markup format exports
pub sort_order_exports: SortOrder,
/// Determines how we run commands
pub executor_kind: ExecutorKind,
@@ -241,8 +221,11 @@ pub struct Options {
/// What to do with the output of the benchmarked commands
pub command_output_policies: Vec<CommandOutputPolicy>,
/// Which time unit to use when displaying results
pub time_unit: Option<TimeUnit>,
/// Displayed metrics, with the primary metric first.
pub metrics: Vec<MetricSelection>,
/// The `all` preset displays only metrics with complete measurements.
pub skip_unavailable_metrics: bool,
}
impl Default for Options {
@@ -252,19 +235,16 @@ impl Default for Options {
warmup_count: 0,
min_benchmarking_time: Time::new::<second>(3.0),
command_failure_action: CmdFailureAction::RaiseError,
reference_command: None,
reference_name: None,
reference_index: None,
preparation_command: None,
conclusion_command: None,
setup_command: None,
cleanup_command: None,
output_style: OutputStyleOption::Full,
sort_order_speed_comparison: SortOrder::MeanTime,
sort_order_exports: SortOrder::Command,
executor_kind: ExecutorKind::default(),
command_output_policies: vec![CommandOutputPolicy::Null],
time_unit: None,
metrics: MetricSelection::parse_list(crate::cli::DEFAULT_METRICS)
.expect("valid default metrics"),
skip_unavailable_metrics: false,
command_input_policy: CommandInputPolicy::Null,
}
}
@@ -307,11 +287,6 @@ impl Options {
options.setup_command = matches.get_one::<String>("setup").map(String::from);
options.reference_command = matches.get_one::<String>("reference").map(String::from);
options.reference_name = matches
.get_one::<String>("reference-name")
.map(String::from);
options.preparation_command = matches
.get_many::<String>("prepare")
.map(|values| values.map(String::from).collect::<Vec<String>>());
@@ -383,16 +358,6 @@ impl Options {
OutputStyleOption::Disabled => {}
};
(
options.sort_order_speed_comparison,
options.sort_order_exports,
) = match matches.get_one::<String>("sort").map(|s| s.as_str()) {
None | Some("auto") => (SortOrder::MeanTime, SortOrder::Command),
Some("command") => (SortOrder::Command, SortOrder::Command),
Some("mean-time") => (SortOrder::MeanTime, SortOrder::MeanTime),
Some(_) => unreachable!("Unknown sort order"),
};
options.executor_kind = if matches.get_flag("no-shell") {
ExecutorKind::Raw
} else if matches.get_flag("default-shell") {
@@ -430,14 +395,36 @@ impl Options {
};
}
options.time_unit = match matches.get_one::<String>("time-unit").map(|s| s.as_str()) {
Some("µs" | "us" | "microsecond" | "microseconds") => Some(TimeUnit::MicroSecond),
Some("ms" | "millisecond" | "milliseconds") => Some(TimeUnit::MilliSecond),
Some("s" | "second" | "seconds") => Some(TimeUnit::Second),
Some("min" | "minute" | "minutes") => Some(TimeUnit::Minute),
Some("h" | "hour" | "hours") => Some(TimeUnit::Hour),
_ => None,
};
match matches.get_one::<String>("metrics").unwrap().as_str() {
"default" => {}
"time" => {
options.metrics = [
Metric::TimeWallClock,
Metric::TimeCpu,
Metric::TimeUser,
Metric::TimeSystem,
]
.map(|metric| MetricSelection { metric, unit: None })
.to_vec();
}
"all" => {
let shell = matches!(
options.executor_kind,
ExecutorKind::Shell(_) | ExecutorKind::Mock(Some(_))
);
options.metrics = Metric::ALL
.iter()
.copied()
.filter(|metric| metric.ensure_available(shell).is_ok())
.map(|metric| MetricSelection { metric, unit: None })
.collect();
options.skip_unavailable_metrics = true;
}
list => {
options.metrics = MetricSelection::parse_list(list)
.map_err(|error| OptionsError::InvalidMetrics(error.to_string()))?;
}
}
if let Some(time) = matches.get_one::<String>("min-benchmarking-time") {
options.min_benchmarking_time = Time::new::<second>(
@@ -466,43 +453,20 @@ impl Options {
}
pub fn validate_against_command_list(&mut self, commands: &Commands) -> Result<()> {
if let Some(reference) = &self.reference_command {
if commands
.iter()
.next()
.is_some_and(|command| !command.get_parameters().is_empty())
{
ensure!(
self.reference_name.is_none(),
"--reference-name cannot be used with a parameterized reference; use --command-name to name the benchmark"
);
let mut matches = commands
.iter()
.enumerate()
.filter(|(_, command)| command.get_name_with_unused_parameters() == *reference);
let Some((index, _)) = matches.next() else {
bail!(
"Reference '{reference}' does not match any parameterized benchmark. Use its full displayed benchmark name."
);
};
ensure!(
matches.next().is_none(),
"Reference '{reference}' matches multiple parameterized benchmarks. Use --command-name to give them unique names."
);
self.reference_index = Some(index);
self.reference_command = None;
} else {
self.reference_index = Some(0);
if !matches!(self.executor_kind, ExecutorKind::Mock(_)) {
for selection in &self.metrics {
selection
.metric
.ensure_available(matches!(self.executor_kind, ExecutorKind::Shell(_)))?;
}
}
let has_reference_command = self.reference_command.is_some();
let num_commands = commands.num_commands(has_reference_command);
let num_commands = commands.num_commands();
if let Some(preparation_command) = &self.preparation_command {
ensure!(
preparation_command.len() <= 1 || num_commands == preparation_command.len(),
"The '--prepare' option has to be provided just once or N times, where N={num_commands} is the \
number of benchmark commands (including a potential reference)."
number of benchmark commands."
);
}
@@ -510,7 +474,7 @@ impl Options {
ensure!(
conclusion_command.len() <= 1 || num_commands == conclusion_command.len(),
"The '--conclude' option has to be provided just once or N times, where N={num_commands} is the \
number of benchmark commands (including a potential reference)."
number of benchmark commands."
);
}
@@ -521,7 +485,7 @@ impl Options {
ensure!(
self.command_output_policies.len() == num_commands,
"The '--output' option has to be provided just once or N times, where N={num_commands} is the \
number of benchmark commands (including a potential reference)."
number of benchmark commands."
);
}
src/output/mod.rs
+1
View File
@@ -1,4 +1,5 @@
pub mod progress_bar;
pub mod report;
pub mod warnings;
/// Mark console write errors so broken pipes can be handled separately from file errors.
src/output/progress_bar.rs
+22 -15
View File
@@ -16,29 +16,36 @@ const DEFAULT_TEMPLATE: &str = " {spinner} {msg:<30} {wide_bar} ETA {eta_precise
const INITIAL_TEMPLATE: &str =
" {spinner} Initial run: {initial_elapsed:<17} {wide_bar} ETA {eta_precise} ";
fn style_with_template(bar: &ProgressBar, template: &str) -> ProgressStyle {
let template = if bar.prefix().is_empty() {
template.to_owned()
} else {
format!("{{prefix}}\n{template}")
};
bar.style().template(&template).expect("no template error")
}
/// Show the benchmark heading above the progress bar, clearing both on completion.
pub fn set_benchmark_header(bar: &ProgressBar, header: String) {
bar.set_prefix(header);
bar.set_style(style_with_template(bar, DEFAULT_TEMPLATE));
}
/// Show the time elapsed since the initial benchmark command started.
pub fn start_initial_measurement(bar: &ProgressBar, started: Instant) {
let style = bar
.style()
.template(INITIAL_TEMPLATE)
.expect("no template error")
.with_key(
"initial_elapsed",
move |_: &ProgressState, w: &mut dyn fmt::Write| {
write!(w, "{}", FormattedDuration(started.elapsed())).unwrap();
},
);
let style = style_with_template(bar, INITIAL_TEMPLATE).with_key(
"initial_elapsed",
move |_: &ProgressState, w: &mut dyn fmt::Write| {
write!(w, "{}", FormattedDuration(started.elapsed())).unwrap();
},
);
bar.set_style(style);
}
/// Stop showing the initial timer and display the next phase's message.
pub fn finish_initial_measurement(bar: &ProgressBar, message: String) {
bar.set_message(message);
bar.set_style(
bar.style()
.template(DEFAULT_TEMPLATE)
.expect("no template error"),
);
bar.set_style(style_with_template(bar, DEFAULT_TEMPLATE));
}
/// Return a pre-configured progress bar
src/output/report.rs
+197
View File
@@ -0,0 +1,197 @@
use std::io::{self, Write};
use anyhow::{ensure, Result};
use colored::Colorize;
use crate::benchmark::{benchmark_result::BenchmarkResult, measurement::Measurements};
use crate::metric::MetricSelection;
use crate::output::console_writeln;
// Reserve enough space for typical values and every supported unit suffix so
// independently printed benchmarks keep the same layout.
const MIN_VALUE_WIDTH: usize = 7;
const MIN_UNIT_WIDTH: usize = 3;
const MIN_CHANGE_WIDTH: usize = 14;
/// Print a completed benchmark immediately, comparing every metric with benchmark 1.
pub fn print(
number: usize,
name: &str,
measurements: &Measurements,
metrics: &[MetricSelection],
skip_unavailable_metrics: bool,
reference: Option<&BenchmarkResult>,
) -> Result<()> {
let comparison = reference.is_some();
let multiple_runs = measurements.len() > 1;
let value_columns = 1..if multiple_runs { 5 } else { 2 };
let mut header = vec![
String::new(),
"mean".to_owned(),
"σ".to_owned(),
"min".to_owned(),
"max".to_owned(),
];
if !multiple_runs {
header.truncate(2);
header[1] = "Value".to_owned();
}
if comparison {
header.extend([String::new(), String::new()]);
}
let mut rows = vec![(header, "")];
for selection in metrics {
let Some(summary) = selection.metric.summarize(measurements) else {
ensure!(
skip_unavailable_metrics,
"Metric '{}' is unavailable for '{name}'",
selection.metric.name()
);
continue;
};
let baseline = reference.and_then(|r| selection.metric.summarize(&r.measurements));
// Keep auto units consistent with the first command for easier comparison.
let scale_mean = baseline
.as_ref()
.filter(|stats| stats.mean != 0.0)
.unwrap_or(&summary)
.mean;
let unit = selection.display_unit(scale_mean);
let mut row = vec![
selection.metric.label().to_owned(),
unit.format_value(summary.mean),
];
if let Some(stddev) = summary.stddev {
row.push(unit.format_value(stddev));
row.push(unit.format_value(summary.min));
row.push(unit.format_value(summary.max));
}
if comparison {
row.push(baseline.as_ref().map_or_else(
|| "N/A".to_owned(),
|baseline| summary.format_change_from(baseline),
));
row.push(baseline.as_ref().map_or_else(String::new, |baseline| {
summary.format_factor_from(baseline, selection.metric)
}));
}
rows.push((
row,
if unit.symbol == "count" {
""
} else {
unit.symbol
},
));
}
// Align numbers and unit suffixes independently so mixed units do not shift
// the values or the ± and … separators.
let unit_width = rows
.iter()
.map(|(_, unit)| unit.chars().count())
.max()
.unwrap()
.max(MIN_UNIT_WIDTH);
// Keep labels aligned across benchmarks even when some metrics are skipped.
let metric_width = metrics
.iter()
.map(|selection| selection.metric.label().chars().count())
.max()
.unwrap_or(0);
let widths: Vec<_> = (0..rows[0].0.len())
.map(|column| {
let width = rows
.iter()
.map(|(row, _)| row[column].chars().count())
.max()
.unwrap();
if column == 0 {
width.max(metric_width)
} else if value_columns.contains(&column) {
width.max(MIN_VALUE_WIDTH)
} else {
width.max(MIN_CHANGE_WIDTH)
}
})
.collect();
let mut stdout = io::stdout().lock();
console_writeln!(
stdout,
"{}: {} {}",
format!("Benchmark {}", number + 1).bold(),
name.white().bold(),
format!(
"({} {})",
measurements.len(),
if measurements.len() == 1 {
"run"
} else {
"runs"
}
)
.dimmed()
)?;
for (index, (row, unit)) in rows.iter().enumerate() {
let mut line = String::new();
for (column, value) in row.iter().enumerate() {
let is_factor = comparison && column == row.len() - 1;
let is_change = comparison && column >= row.len() - 2;
if index == 0 && is_change {
break;
}
if column > 0 {
line.push_str(match (multiple_runs, column) {
(true, 2) => " ± ",
(true, 4) => " … ",
_ => " ",
});
}
let padding = " ".repeat(widths[column] - value.chars().count());
let has_unit = value_columns.contains(&column);
let text = if has_unit && !unit.is_empty() {
format!("{value} {unit}")
} else {
value.clone()
};
let styled = if is_change {
if row[row.len() - 2].starts_with('-') {
text.green().to_string()
} else if row[row.len() - 2].starts_with('+') {
text.red().to_string()
} else {
text.normal().to_string()
}
} else if column == 1 || column == 2 {
text.green().to_string()
} else if column == 3 {
text.cyan().to_string()
} else if column == 4 {
text.purple().to_string()
} else {
text.normal().to_string()
};
if column == 0 || is_factor {
line.push_str(&format!("{styled}{padding}"));
} else {
line.push_str(&format!("{padding}{styled}"));
}
if has_unit && unit_width > 0 {
// Empty suffixes (headers and unscaled counters) also need the
// space normally separating a number from its unit.
let unit_padding = unit_width - unit.chars().count() + usize::from(unit.is_empty());
line.push_str(&" ".repeat(unit_padding));
}
}
let line = line.trim_end();
console_writeln!(
stdout,
" {}",
if index == 1 {
line.bold()
} else {
line.normal()
}
)?;
}
Ok(())
}
src/output/warnings.rs
+14 -17
View File
@@ -1,7 +1,8 @@
use std::fmt;
use crate::benchmark::MIN_EXECUTION_TIME;
use crate::quantity::{FormatQuantity, Time, TimeUnit};
use crate::metric::{Metric, MetricSelection};
use crate::quantity::millisecond;
pub struct OutlierWarningOptions {
pub warmup_in_use: bool,
@@ -12,8 +13,7 @@ pub struct OutlierWarningOptions {
pub enum Warnings {
FastExecutionTime,
NonZeroExitCode,
SlowInitialRun(Time, OutlierWarningOptions),
OutliersDetected(OutlierWarningOptions),
InitialRunOutlier(MetricSelection, f64, OutlierWarningOptions),
}
impl fmt::Display for Warnings {
@@ -21,19 +21,25 @@ impl fmt::Display for Warnings {
match *self {
Warnings::FastExecutionTime => write!(
f,
"Command took less than {} ms to complete. Note that the results might be \
"Command took less than {} ms to complete. Note that the wall-clock measurements might be \
inaccurate because hyperfine can not calibrate the shell startup time much \
more precise than this limit. You can try to use the `-N`/`--shell=none` \
option to disable the shell completely.",
MIN_EXECUTION_TIME.format_with_precision(TimeUnit::MilliSecond, 0)
MIN_EXECUTION_TIME.get::<millisecond>()
),
Warnings::NonZeroExitCode => write!(f, "Ignoring non-zero exit code."),
Warnings::SlowInitialRun(time_first_run, ref options) => write!(
Warnings::InitialRunOutlier(selection, value, _) if selection.metric != Metric::TimeWallClock => write!(
f,
"The first benchmarking run for this command had an unusually high value for the primary '{}' metric ({}) compared with the rest.",
selection.metric.name(),
selection.display_unit(value).format(value),
),
Warnings::InitialRunOutlier(selection, value, ref options) => write!(
f,
"The first benchmarking run for this command was significantly slower than the \
rest ({time}). This could be caused by (filesystem) caches that were not filled until \
after the first run. {hints}",
time=time_first_run.format_auto(),
time=selection.display_unit(value).format(value),
hints=match (options.warmup_in_use, options.prepare_in_use) {
(true, true) => "You are already using both the '--warmup' option as well \
as the '--prepare' option. Consider re-running the benchmark on a quiet system. \
@@ -53,16 +59,7 @@ impl fmt::Display for Warnings {
option to clear the caches before each timing run."
}
),
Warnings::OutliersDetected(ref options) => write!(
f,
"Statistical outliers were detected. Consider re-running this benchmark on a quiet \
system without any interferences from other programs.{hint}",
hint=if options.warmup_in_use && options.prepare_in_use {
""
} else {
" It might help to use the '--warmup' or '--prepare' options."
}
),
}
}
}
src/quantity/mod.rs
-181
View File
@@ -11,80 +11,7 @@ pub use si::ratio::ratio;
pub use si::time::{hour, microsecond, millisecond, minute, nanosecond, second};
pub use uom::num_traits::Zero;
pub use units::{InformationUnit, IsUnit, TimeUnit};
pub mod statistics;
mod units;
pub trait QuantityInUnit<U> {
fn value_in_unit(self) -> f64;
}
impl<U: uom::si::time::Unit + uom::Conversion<f64, T = f64>> QuantityInUnit<U> for Time {
fn value_in_unit(self) -> f64 {
self.get::<U>()
}
}
impl<U: uom::si::information::Unit + uom::Conversion<f64, T = f64>> QuantityInUnit<U>
for Information
{
fn value_in_unit(self) -> f64 {
self.get::<U>()
}
}
impl<U: uom::si::ratio::Unit + uom::Conversion<f64, T = f64>> QuantityInUnit<U> for Ratio {
fn value_in_unit(self) -> f64 {
self.get::<U>()
}
}
pub trait FormatQuantity {
type Unit;
fn suitable_unit(&self) -> Self::Unit;
fn format_with_precision(&self, unit: Self::Unit, precision: usize) -> String;
fn format(&self, unit: Self::Unit) -> String;
fn format_auto(&self) -> String;
fn format_value(&self, unit: Self::Unit) -> String;
}
impl FormatQuantity for Time {
type Unit = TimeUnit;
fn suitable_unit(&self) -> TimeUnit {
if *self < Time::new::<millisecond>(1.0) {
TimeUnit::MicroSecond
} else if *self < Time::new::<second>(1.0) {
TimeUnit::MilliSecond
} else {
TimeUnit::Second
}
}
/// Format the time duration in the given unit with the given precision.
fn format_with_precision(&self, u: TimeUnit, precision: usize) -> String {
u.format(*self, precision)
}
/// Format the time duration in the given unit.
fn format(&self, unit: TimeUnit) -> String {
let value = self.format_with_precision(unit, unit.preferred_precision());
format!("{} {}", value, unit.short_name())
}
/// Format the given time duration. The unit will be determined automatically.
fn format_auto(&self) -> String {
self.format(self.suitable_unit())
}
/// Like `format`, but without displaying the unit.
fn format_value(&self, unit: TimeUnit) -> String {
self.format_with_precision(unit, unit.preferred_precision())
}
}
pub const fn const_time_from_seconds(value: f64) -> Time {
// Quantity::new in uom is not yet const: https://docs.rs/uom/0.36.0/uom/si/struct.Quantity.html
@@ -95,43 +22,6 @@ pub const fn const_time_from_seconds(value: f64) -> Time {
}
}
impl FormatQuantity for Information {
type Unit = InformationUnit;
fn suitable_unit(&self) -> InformationUnit {
if *self < Information::new::<kibibyte>(1.0) {
InformationUnit::Byte
} else if *self < Information::new::<mebibyte>(1.0) {
InformationUnit::KibiByte
} else if *self < Information::new::<gibibyte>(1.0) {
InformationUnit::MebiByte
} else {
InformationUnit::GibiByte
}
}
/// Format the information in the given unit with the given precision.
fn format_with_precision(&self, u: InformationUnit, precision: usize) -> String {
u.format(*self, precision)
}
/// Format the information in the given unit.
fn format(&self, unit: InformationUnit) -> String {
let value = self.format_with_precision(unit, unit.preferred_precision());
format!("{} {}", value, unit.short_name())
}
/// Format the given information. The unit will be determined automatically.
fn format_auto(&self) -> String {
self.format(self.suitable_unit())
}
/// Like `format`, but without displaying the unit.
fn format_value(&self, unit: InformationUnit) -> String {
self.format_with_precision(unit, unit.preferred_precision())
}
}
pub fn serialize_time<S>(t: &Time, s: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
@@ -177,74 +67,3 @@ fn test_information() {
let largest_exactly_representable = Information::new::<byte>(9_007_199_254_740_992.);
assert_eq!(largest_exactly_representable.get::<pebibyte>(), 8.);
}
#[test]
fn test_suiteable_unit_time() {
assert_eq!(Time::new::<second>(1.3).suitable_unit(), TimeUnit::Second);
assert_eq!(Time::new::<second>(1.0).suitable_unit(), TimeUnit::Second);
assert_eq!(
Time::new::<second>(0.999).suitable_unit(),
TimeUnit::MilliSecond
);
assert_eq!(
Time::new::<second>(0.0005).suitable_unit(),
TimeUnit::MicroSecond
);
assert_eq!(
Time::new::<second>(0.).suitable_unit(),
TimeUnit::MicroSecond
);
assert_eq!(
Time::new::<second>(1000.0).suitable_unit(),
TimeUnit::Second
);
}
#[test]
fn test_suitable_unit_information() {
assert_eq!(
Information::new::<byte>(512.0).suitable_unit(),
InformationUnit::Byte
);
assert_eq!(
Information::new::<byte>(2048.0).suitable_unit(),
InformationUnit::KibiByte
);
assert_eq!(
Information::new::<byte>(2_097_152.0).suitable_unit(),
InformationUnit::MebiByte
);
assert_eq!(
Information::new::<byte>(2_147_483_648.0).suitable_unit(),
InformationUnit::GibiByte
);
}
#[test]
fn test_format() {
let time = Time::new::<millisecond>(123.4);
assert_eq!(time.format(TimeUnit::Second), "0.123 s");
assert_eq!(time.format(TimeUnit::MilliSecond), "123.4 ms");
assert_eq!(time.format_auto(), "123.4 ms");
assert_eq!(time.format(TimeUnit::MicroSecond), "123400.0 µs");
let memory_peak_resident = Information::new::<kibibyte>(8.);
assert_eq!(memory_peak_resident.format_auto(), "8.0 KiB");
assert_eq!(memory_peak_resident.format(InformationUnit::Byte), "8192 B");
assert_eq!(
memory_peak_resident.format(InformationUnit::KibiByte),
"8.0 KiB"
);
}
#[test]
fn test_format_duration_unit_with_unit() {
let out = Time::new::<second>(1.3).format(TimeUnit::Second);
assert_eq!("1.300 s", out);
let out = Time::new::<second>(1.3).format(TimeUnit::MilliSecond);
assert_eq!("1300.0 ms", out);
let out = Time::new::<second>(1.3).format(TimeUnit::MicroSecond);
assert_eq!("1300000.0 µs", out);
}
src/quantity/statistics.rs
-5
View File
@@ -146,11 +146,6 @@ pub fn modified_zscores_f64(xs: &[f64]) -> Vec<f64> {
xs.iter().map(|&x| (x - x_median) / mad).collect()
}
pub fn modified_zscores<Q: UnsafeRawValue>(values: &[Q]) -> Vec<f64> {
let values: Vec<_> = values.iter().map(|q| q.unsafe_raw_value()).collect();
modified_zscores_f64(&values)
}
#[cfg(test)]
mod tests {
use super::*;
src/quantity/units.rs
-166
View File
@@ -1,166 +0,0 @@
//! This module contains common units for time and information quantities.
use std::marker::PhantomData;
use crate::quantity::{
byte, gibibyte, hour, kibibyte, mebibyte, microsecond, millisecond, minute, second, tebibyte,
Information, Time,
};
pub trait IsUnit {
type Quantity;
fn dispatch(&self) -> Box<dyn UnitImpl<Quantity = Self::Quantity>>;
fn preferred_precision(&self) -> usize;
fn short_name(&self) -> &'static str {
self.dispatch().short_name()
}
fn format(&self, value: Self::Quantity, precision: usize) -> String {
self.dispatch().format_value(value, precision)
}
}
/// Supported time units
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TimeUnit {
MicroSecond,
MilliSecond,
Second,
Minute,
Hour,
}
impl IsUnit for TimeUnit {
type Quantity = Time;
fn dispatch(&self) -> Box<dyn UnitImpl<Quantity = Time>> {
match self {
TimeUnit::MicroSecond => Box::new(TimeUnitDispatcher::<microsecond>::new()),
TimeUnit::MilliSecond => Box::new(TimeUnitDispatcher::<millisecond>::new()),
TimeUnit::Second => Box::new(TimeUnitDispatcher::<second>::new()),
TimeUnit::Minute => Box::new(TimeUnitDispatcher::<minute>::new()),
TimeUnit::Hour => Box::new(TimeUnitDispatcher::<hour>::new()),
}
}
fn preferred_precision(&self) -> usize {
match self {
TimeUnit::Second => 3,
_ => 1,
}
}
}
/// Supported information units
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum InformationUnit {
Byte,
KibiByte,
MebiByte,
GibiByte,
TebiByte,
}
impl IsUnit for InformationUnit {
type Quantity = Information;
fn dispatch(&self) -> Box<dyn UnitImpl<Quantity = Information>> {
match self {
InformationUnit::Byte => Box::new(InformationUnitDispatcher::<byte>::new()),
InformationUnit::KibiByte => Box::new(InformationUnitDispatcher::<kibibyte>::new()),
InformationUnit::MebiByte => Box::new(InformationUnitDispatcher::<mebibyte>::new()),
InformationUnit::GibiByte => Box::new(InformationUnitDispatcher::<gibibyte>::new()),
InformationUnit::TebiByte => Box::new(InformationUnitDispatcher::<tebibyte>::new()),
}
}
fn preferred_precision(&self) -> usize {
match self {
InformationUnit::Byte => 0,
_ => 1,
}
}
}
pub trait UnitImpl {
type Quantity;
fn short_name(&self) -> &'static str;
fn format_value(&self, value: Self::Quantity, precision: usize) -> String;
}
struct TimeUnitDispatcher<U: uom::si::time::Unit + uom::Conversion<f64, T = f64>> {
u: PhantomData<U>,
}
impl<U: uom::si::time::Unit + uom::Conversion<f64, T = f64>> TimeUnitDispatcher<U> {
fn new() -> Self {
TimeUnitDispatcher { u: PhantomData }
}
}
impl<U: uom::si::time::Unit + uom::Conversion<f64, T = f64>> UnitImpl for TimeUnitDispatcher<U> {
type Quantity = Time;
fn short_name(&self) -> &'static str {
U::abbreviation()
}
fn format_value(&self, value: Time, precision: usize) -> String {
format!("{value:.precision$}", value = value.get::<U>())
}
}
struct InformationUnitDispatcher<U: uom::si::information::Unit + uom::Conversion<f64, T = f64>> {
u: PhantomData<U>,
}
impl<U: uom::si::information::Unit + uom::Conversion<f64, T = f64>> InformationUnitDispatcher<U> {
fn new() -> Self {
InformationUnitDispatcher { u: PhantomData }
}
}
impl<U: uom::si::information::Unit + uom::Conversion<f64, T = f64>> UnitImpl
for InformationUnitDispatcher<U>
{
type Quantity = Information;
fn short_name(&self) -> &'static str {
U::abbreviation()
}
fn format_value(&self, value: Information, precision: usize) -> String {
format!("{value:.precision$}", value = value.get::<U>())
}
}
#[test]
fn test_time_unit_short_name() {
assert_eq!("s", TimeUnit::Second.short_name());
assert_eq!("ms", TimeUnit::MilliSecond.short_name());
assert_eq!("µs", TimeUnit::MicroSecond.short_name());
assert_eq!("min", TimeUnit::Minute.short_name());
assert_eq!("h", TimeUnit::Hour.short_name());
}
// Note - the values are rounded when formatted.
#[test]
fn test_unit_format() {
use crate::quantity::FormatQuantity;
let value = Time::new::<second>(123.456789);
assert_eq!("123.457", value.format_value(TimeUnit::Second));
assert_eq!("123456.8", value.format_value(TimeUnit::MilliSecond));
assert_eq!(
"1234.6",
Time::new::<second>(0.00123456).format_value(TimeUnit::MicroSecond)
);
assert_eq!("2.1", value.format_value(TimeUnit::Minute));
assert_eq!("0.0", value.format_value(TimeUnit::Hour));
}
src/timer/mod.rs
+1
View File
@@ -118,6 +118,7 @@ pub fn execute_and_measure(
Ok(Measurement {
time_wall_clock,
time_cpu: time_user + time_system,
time_user,
time_system,
memory_peak_resident,
tests/execution_order_tests.rs
+6 -11
View File
@@ -56,11 +56,6 @@ impl ExecutionOrderTest {
self.command(output)
}
fn reference(&mut self, output: &str) -> &mut Self {
self.arg("--reference");
self.command(output)
}
fn conclude(&mut self, output: &str) -> &mut Self {
self.arg("--conclude");
self.command(output)
@@ -373,10 +368,10 @@ fn multiple_parameter_values() {
}
#[test]
fn reference_is_executed_first() {
fn commands_execute_in_input_order() {
ExecutionOrderTest::new()
.arg("--runs=1")
.reference("reference")
.command("reference")
.command("command 1")
.command("command 2")
.expect_output("reference")
@@ -386,10 +381,10 @@ fn reference_is_executed_first() {
}
#[test]
fn reference_is_executed_separately_from_commands() {
fn repeated_commands_execute_separately() {
ExecutionOrderTest::new()
.arg("--runs=1")
.reference("command 1")
.command("command 1")
.command("command 1")
.command("command 2")
.expect_output("command 1")
@@ -405,7 +400,7 @@ fn setup_prepare_reference_conclude_cleanup_combined() {
.arg("--runs=2")
.setup("setup")
.prepare("prepare")
.reference("reference")
.command("reference")
.command("command1")
.command("command2")
.conclude("conclude")
@@ -497,7 +492,7 @@ fn setup_separate_prepare_reference_separate_conclude_cleanup_combined() {
.setup("setup")
.cleanup("cleanup")
.prepare("prepareref")
.reference("reference")
.command("reference")
.conclude("concluderef")
.prepare("prepare1")
.command("command1")
tests/integration_tests.rs
+179 -249
View File
@@ -111,13 +111,23 @@ fn fails_with_zero_runs() {
#[test]
fn min_runs_of_one_still_performs_one_run() {
hyperfine_debug()
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--style=basic")
.arg("--metrics=time_wall_clock")
.arg("--min-runs=1")
.arg("--warmup=0")
.arg("sleep 4")
.assert()
.success()
.stdout(predicate::str::contains("Time (abs ≡)"));
.arg("sleep 4"), @"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 4 (1 run)
Value
Wall Time 4.000 s
----- stderr -----
");
}
#[test]
@@ -159,7 +169,7 @@ fn fails_with_wrong_number_of_prepare_options() {
.arg("--prepare=echo ref")
.arg("--prepare=echo a")
.arg("--prepare=echo b")
.arg("--reference=echo ref")
.arg("echo ref")
.arg("echo a")
.arg("echo b")
.assert()
@@ -182,7 +192,7 @@ fn fails_with_wrong_number_of_prepare_options() {
.arg("--runs=1")
.arg("--prepare=echo a")
.arg("--prepare=echo b")
.arg("--reference=echo ref")
.arg("echo ref")
.arg("echo a")
.arg("echo b")
.assert()
@@ -208,7 +218,7 @@ fn fails_with_wrong_number_of_conclude_options() {
.arg("--conclude=echo ref")
.arg("--conclude=echo a")
.arg("--conclude=echo b")
.arg("--reference=echo ref")
.arg("echo ref")
.arg("echo a")
.arg("echo b")
.assert()
@@ -231,7 +241,7 @@ fn fails_with_wrong_number_of_conclude_options() {
.arg("--runs=1")
.arg("--conclude=echo a")
.arg("--conclude=echo b")
.arg("--reference=echo ref")
.arg("echo ref")
.arg("echo a")
.arg("echo b")
.assert()
@@ -509,29 +519,27 @@ fn returns_mean_time_in_correct_unit() {
.arg("sleep 1.234")
.assert()
.success()
.stdout(predicate::str::contains("Time (mean ± σ): 1.234 s ±"));
.stdout(predicate::str::is_match(r"Wall Time\s+1\.234 s\s+±").unwrap());
hyperfine_debug()
.arg("sleep 0.123")
.assert()
.success()
.stdout(predicate::str::contains("Time (mean ± σ): 123.0 ms ±"));
.stdout(predicate::str::is_match(r"Wall Time\s+123\.0 ms\s+±").unwrap());
hyperfine_debug()
.arg("--time-unit=millisecond")
.arg("--metrics=time_wall_clock:ms")
.arg("sleep 1.234")
.assert()
.success()
.stdout(predicate::str::contains("Time (mean ± σ): 1234.0 ms ±"));
.stdout(predicate::str::is_match(r"Wall Time\s+1234\.0 ms\s+±").unwrap());
hyperfine_debug()
.arg("--time-unit=microsecond")
.arg("--metrics=time_wall_clock:us")
.arg("sleep 1.234")
.assert()
.success()
.stdout(predicate::str::contains(
"Time (mean ± σ): 1234000.0 µs ±",
));
.stdout(predicate::str::is_match(r"Wall Time\s+1234000\.0 µs\s+±").unwrap());
}
#[test]
@@ -625,32 +633,57 @@ fn takes_both_preparation_and_conclusion_command_into_account_for_computing_numb
}
#[test]
fn shows_benchmark_comparison_with_relative_times() {
fn secondary_metrics_include_factors_for_large_changes() {
// Mock CPU time is zero; secondary wall time still gets ratio annotations.
let output = hyperfine_debug()
.args([
"--metrics=time_cpu,time_wall_clock",
"--style=basic",
"--runs=2",
"sleep 1",
"sleep 10",
"sleep 0.1",
])
.output()
.unwrap();
assert!(output.status.success(), "{:?}", output);
let stdout = String::from_utf8(output.stdout).unwrap();
assert!(stdout.contains("+900.0%"), "{}", stdout);
assert!(stdout.contains("-90.0%"), "{}", stdout);
assert!(stdout.contains("+900.0% (10.0x slower)"), "{}", stdout);
assert!(stdout.contains("-90.0% (10.0x faster)"), "{}", stdout);
}
#[test]
fn shows_multiple_metrics_relative_to_first_command() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--style=basic")
.arg("sleep 1.0")
.arg("sleep 2.0")
.arg("sleep 3.0"), @r"
.arg("--metrics=time_wall_clock:ms,time_user:us,memory_peak_resident:MiB")
.arg("sleep 0.0817")
.arg("sleep 0.6155")
.arg("sleep 0.1707"), @"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 1.0
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Benchmark 1: sleep 0.0817 (36 runs)
mean ± σ min … max
Wall Time 81.7 ms ± 0.0 ms 81.7 ms … 81.7 ms
User time 0.0 µs ± 0.0 µs 0.0 µs … 0.0 µs
Memory 0.0 MiB ± 0.0 MiB 0.0 MiB … 0.0 MiB
Benchmark 2: sleep 2.0
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 2: sleep 0.6155 (10 runs)
mean ± σ min … max
Wall Time 615.5 ms ± 0.0 ms 615.5 ms … 615.5 ms +653.4% (7.5x slower)
User time 0.0 µs ± 0.0 µs 0.0 µs … 0.0 µs N/A
Memory 0.0 MiB ± 0.0 MiB 0.0 MiB … 0.0 MiB N/A
Benchmark 3: sleep 3.0
Time (mean ± σ): 3.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 3.000 s … 3.000 s 10 runs
Benchmark 3: sleep 0.1707 (17 runs)
mean ± σ min … max
Wall Time 170.7 ms ± 0.0 ms 170.7 ms … 170.7 ms +108.9% (2.1x slower)
User time 0.0 µs ± 0.0 µs 0.0 µs … 0.0 µs N/A
Memory 0.0 MiB ± 0.0 MiB 0.0 MiB … 0.0 MiB N/A
Summary
sleep 1.0 ran
2.00 ± 0.00 times faster than sleep 2.0
3.00 ± 0.00 times faster than sleep 3.0
----- stderr -----
");
@@ -660,37 +693,37 @@ fn shows_benchmark_comparison_with_relative_times() {
fn shows_benchmark_comparison_with_same_time() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--metrics=time_wall_clock,memory_peak_resident")
.arg("--style=basic")
.arg("--command-name=A")
.arg("--command-name=B")
.arg("sleep 1.0")
.arg("sleep 1.0")
.arg("sleep 2.0")
.arg("sleep 1000.0"), @r"
.arg("sleep 1000.0"), @"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: A
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Benchmark 1: A (10 runs)
mean ± σ min … max
Wall Time 1.000 s ± 0.000 s 1.000 s … 1.000 s
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B
Benchmark 2: B
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Benchmark 2: B (10 runs)
mean ± σ min … max
Wall Time 1.000 s ± 0.000 s 1.000 s … 1.000 s 0.0%
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Benchmark 3: sleep 2.0
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 3: sleep 2.0 (10 runs)
mean ± σ min … max
Wall Time 2.000 s ± 0.000 s 2.000 s … 2.000 s +100.0% (2.0x slower)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Benchmark 4: sleep 1000.0
Time (mean ± σ): 1000.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1000.000 s … 1000.000 s 10 runs
Benchmark 4: sleep 1000.0 (10 runs)
mean ± σ min … max
Wall Time 1000.000 s ± 0.000 s 1000.000 s … 1000.000 s +99900.0% (1000.0x slower)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Summary
A ran
As fast (1.00 ± 0.00) as B
2.00 ± 0.00 times faster than sleep 2.0
1000.00 ± 0.00 times faster than sleep 1000.0
----- stderr -----
");
@@ -700,36 +733,36 @@ fn shows_benchmark_comparison_with_same_time() {
fn shows_benchmark_comparison_relative_to_reference() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--metrics=time_wall_clock,memory_peak_resident")
.arg("--style=basic")
.arg("--reference=sleep 2.0")
.arg("sleep 2.0")
.arg("sleep 1.0")
.arg("sleep 3.0"), @r"
.arg("sleep 3.0"), @"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 2.0
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 1: sleep 2.0 (10 runs)
mean ± σ min … max
Wall Time 2.000 s ± 0.000 s 2.000 s … 2.000 s
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B
Benchmark 2: sleep 1.0
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Benchmark 2: sleep 1.0 (10 runs)
mean ± σ min … max
Wall Time 1.000 s ± 0.000 s 1.000 s … 1.000 s -50.0% (2.0x faster)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Benchmark 3: sleep 3.0
Time (mean ± σ): 3.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 3.000 s … 3.000 s 10 runs
Benchmark 3: sleep 3.0 (10 runs)
mean ± σ min … max
Wall Time 3.000 s ± 0.000 s 3.000 s … 3.000 s +50.0% (1.5x slower)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Summary
sleep 2.0 ran
2.00 ± 0.00 times slower than sleep 1.0
1.50 ± 0.00 times faster than sleep 3.0
----- stderr -----
");
}
#[test]
fn command_sorted_comparison_and_markup_identify_reference() {
fn comparison_and_markup_identify_first_command_as_reference() {
let _settings = snapshot_settings().bind_to_scope();
let directory = tempfile::tempdir().unwrap();
let export_path = directory.path().join("results.md");
@@ -737,34 +770,31 @@ fn command_sorted_comparison_and_markup_identify_reference() {
.args([
"--style=basic",
"--runs=1",
"--sort=command",
"--reference",
"sleep 2",
"--reference-name",
"baseline",
"--command-name=baseline",
"--export-markdown",
])
.arg(&export_path)
.args(["sleep 1", "sleep 2", "sleep 3"])
.args(["sleep 2", "sleep 1", "sleep 2", "sleep 3"])
.output()
.unwrap();
assert!(output.status.success(), "{:?}", output);
let stdout = String::from_utf8(output.stdout).unwrap();
let comparison = stdout.split_once("Relative speed comparison\n").unwrap().1;
insta::assert_snapshot!(comparison, @r"
1.00 baseline (reference)
2.00 sleep 1 (faster)
1.00 sleep 2 (same speed)
1.50 sleep 3 (slower)
");
assert!(
stdout.contains("Benchmark 1: baseline (1 run)"),
"{}",
stdout
);
assert!(stdout.contains("-50.0%"), "{}", stdout);
assert!(stdout.contains("0.0%"), "{}", stdout);
assert!(stdout.contains("+50.0%"), "{}", stdout);
let markdown = std::fs::read_to_string(export_path).unwrap();
insta::assert_snapshot!(markdown, @r"
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `baseline` | 2.000 | 2.000 | 2.000 | 1.00 (reference) |
| `sleep 1` | 1.000 | 1.000 | 1.000 | 2.00 (faster) |
| `sleep 2` | 2.000 | 2.000 | 2.000 | 1.00 (same speed) |
| `sleep 3` | 3.000 | 3.000 | 3.000 | 1.50 (slower) |
insta::assert_snapshot!(markdown, @"
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `baseline` | 2.000 | | |
| `sleep 1` | 1.000 | -50.0% | (2.0x faster) |
| `sleep 2` | 2.000 | 0.0% | |
| `sleep 3` | 3.000 | +50.0% | (1.5x slower) |
");
// Equal zero times have no meaningful relative factor.
@@ -773,10 +803,8 @@ fn command_sorted_comparison_and_markup_identify_reference() {
"--style=none",
"--runs=1",
"--export-markdown=-",
"--reference",
"--command-name=baseline",
"sleep 0",
"--reference-name",
"baseline",
"sleep 0",
])
.output()
@@ -787,46 +815,14 @@ fn command_sorted_comparison_and_markup_identify_reference() {
.lines()
.find(|line| line.contains("sleep 0"))
.unwrap();
assert!(row.ends_with("| N/A |"), "{}", row);
}
#[test]
fn shows_reference_name() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--style=basic")
.arg("--reference=sleep 2.0")
.arg("--reference-name=refabc123")
.arg("sleep 1.0")
.arg("sleep 3.0"), @r"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: refabc123
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 2: sleep 1.0
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Benchmark 3: sleep 3.0
Time (mean ± σ): 3.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 3.000 s … 3.000 s 10 runs
Summary
refabc123 ran
2.00 ± 0.00 times slower than sleep 1.0
1.50 ± 0.00 times faster than sleep 3.0
----- stderr -----
");
assert!(row.ends_with("| N/A | |"), "{}", row);
}
#[test]
fn performs_all_benchmarks_in_parameter_scan() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.arg("--metrics=time_wall_clock,memory_peak_resident")
.arg("--style=basic")
.arg("--parameter-scan")
.arg("time")
@@ -834,31 +830,30 @@ fn performs_all_benchmarks_in_parameter_scan() {
.arg("45")
.arg("--parameter-step-size")
.arg("5")
.arg("sleep {time}"), @r"
.arg("sleep {time}"), @"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 30
Time (mean ± σ): 30.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 30.000 s … 30.000 s 10 runs
Benchmark 1: sleep 30 (10 runs)
mean ± σ min … max
Wall Time 30.000 s ± 0.000 s 30.000 s … 30.000 s
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B
Benchmark 2: sleep 35
Time (mean ± σ): 35.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 35.000 s … 35.000 s 10 runs
Benchmark 2: sleep 35 (10 runs)
mean ± σ min … max
Wall Time 35.000 s ± 0.000 s 35.000 s … 35.000 s +16.7%
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Benchmark 3: sleep 40
Time (mean ± σ): 40.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 40.000 s … 40.000 s 10 runs
Benchmark 3: sleep 40 (10 runs)
mean ± σ min … max
Wall Time 40.000 s ± 0.000 s 40.000 s … 40.000 s +33.3% (1.3x slower)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Benchmark 4: sleep 45
Time (mean ± σ): 45.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 45.000 s … 45.000 s 10 runs
Benchmark 4: sleep 45 (10 runs)
mean ± σ min … max
Wall Time 45.000 s ± 0.000 s 45.000 s … 45.000 s +50.0% (1.5x slower)
Memory 0.0 B ± 0.0 B 0.0 B … 0.0 B N/A
Summary
sleep 30 ran
1.17 ± 0.00 times faster than sleep 35
1.33 ± 0.00 times faster than sleep 40
1.50 ± 0.00 times faster than sleep 45
----- stderr -----
");
@@ -882,7 +877,7 @@ fn rejects_negative_parameter_steps() {
#[cfg(unix)]
#[test]
fn selects_reference_with_parameterized_prepare() {
fn preserves_input_order_with_parameterized_prepare() {
let directory = tempfile::tempdir().unwrap();
let csv_path = directory.path().join("results.csv");
let json_path = directory.path().join("results.json");
@@ -896,8 +891,6 @@ fn selects_reference_with_parameterized_prepare() {
"0.2,0.4,0.6",
"--prepare",
"sleep {delay}",
"--reference",
"echo a (delay = 0.4)",
"--export-csv",
])
.arg(&csv_path)
@@ -912,7 +905,7 @@ fn selects_reference_with_parameterized_prepare() {
for (number, delay) in ["0.2", "0.4", "0.6"].iter().enumerate() {
assert!(
stdout.contains(&format!(
"Benchmark {}: echo a (delay = {})",
"Benchmark {}: echo a (delay = {}) (1 run)",
number + 1,
delay
)),
@@ -920,7 +913,16 @@ fn selects_reference_with_parameterized_prepare() {
stdout
);
}
assert!(stdout.contains("echo a (delay = 0.4) ran"), "{}", stdout);
assert!(
!stdout
.split("Benchmark 2")
.next()
.unwrap()
.contains("Change"),
"{}",
stdout
);
assert!(!stdout.contains("Change vs #1"));
let mut csv = csv::Reader::from_path(csv_path).unwrap();
assert!(csv
@@ -946,14 +948,21 @@ fn selects_reference_with_parameterized_prepare() {
}
#[test]
fn rejects_unmatched_reference_before_export_or_setup() {
fn rejects_invalid_metric_before_export_or_setup() {
let directory = tempfile::tempdir().unwrap();
let export_path = directory.path().join("results.csv");
let marker_path = directory.path().join("setup-ran");
std::fs::write(&export_path, "previous contents").unwrap();
hyperfine()
.args(["--reference", "sleep 1", "-P", "secs", "2", "3", "--setup"])
.args([
"--metrics=memory_peak_resident:ms",
"-P",
"secs",
"2",
"3",
"--setup",
])
.arg(format!("echo touched > \"{}\"", marker_path.display()))
.arg("--export-csv")
.arg(&export_path)
@@ -962,7 +971,7 @@ fn rejects_unmatched_reference_before_export_or_setup() {
.failure()
.stdout(predicate::str::is_empty())
.stderr(predicate::str::contains(
"does not match any parameterized benchmark",
"Unit 'ms' is not valid for metric 'memory_peak_resident'",
));
assert_eq!(
std::fs::read_to_string(export_path).unwrap(),
@@ -971,29 +980,9 @@ fn rejects_unmatched_reference_before_export_or_setup() {
assert!(!marker_path.exists());
}
#[test]
fn rejects_ambiguous_parameterized_reference() {
hyperfine_debug()
.args([
"-L",
"x",
"1,2",
"--command-name",
"case",
"--reference",
"case",
"sleep {x}",
])
.assert()
.failure()
.stderr(predicate::str::contains(
"matches multiple parameterized benchmarks",
));
}
#[cfg(unix)]
#[test]
fn intermediate_markdown_waits_for_selected_reference() {
fn intermediate_markdown_retains_completed_reference() {
let directory = tempfile::tempdir().unwrap();
let export_path = directory.path().join("results.md");
hyperfine()
@@ -1004,8 +993,6 @@ fn intermediate_markdown_waits_for_selected_reference() {
"-L",
"delay",
"0.01,0.02,0.03",
"--reference",
"sleep 0.02",
"--prepare",
"true",
"--prepare",
@@ -1023,11 +1010,11 @@ fn intermediate_markdown_waits_for_selected_reference() {
.lines()
.find(|line| line.contains("sleep 0.01"))
.unwrap();
assert!(row.ends_with("| N/A |"), "{}", row);
assert!(row.ends_with("| | |"), "{}", row);
}
#[test]
fn markdown_export_uses_selected_parameterized_reference() {
fn markdown_export_uses_first_parameterized_benchmark_as_reference() {
let _settings = snapshot_settings().bind_to_scope();
assert_cmd_snapshot!(hyperfine_debug()
.args([
@@ -1040,19 +1027,17 @@ fn markdown_export_uses_selected_parameterized_reference() {
"3",
"--command-name",
"case-{x}",
"--reference",
"case-2",
"sleep {x}",
]), @r"
]), @"
success: true
exit_code: 0
----- stdout -----
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `case-1` | 1.000 | 1.000 | 1.000 | 2.00 (faster) |
| `case-2` | 2.000 | 2.000 | 2.000 | 1.00 (reference) |
| `case-3` | 3.000 | 3.000 | 3.000 | 1.50 (slower) |
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `case-1` | 1.000 | | |
| `case-2` | 2.000 | +100.0% | (2.0x slower) |
| `case-3` | 3.000 | +200.0% | (3.0x slower) |
----- stderr -----
@@ -1067,15 +1052,15 @@ fn intermediate_results_are_not_exported_to_stdout() {
.arg("--export-markdown")
.arg("-")
.arg("sleep 1")
.arg("sleep 2"), @r"
.arg("sleep 2"), @"
success: true
exit_code: 0
----- stdout -----
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `sleep 1` | 1.000 ± 0.000 | 1.000 | 1.000 | 1.00 |
| `sleep 2` | 2.000 ± 0.000 | 2.000 | 2.000 | 2.00 ± 0.00 |
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `sleep 1` | 1.000 ± 0.000 | | |
| `sleep 2` | 2.000 ± 0.000 | +100.0% | (2.0x slower) |
----- stderr -----
@@ -1158,9 +1143,9 @@ fn markdown_export_preserves_backticks_and_pipes_in_command_names() {
exit_code: 0
----- stdout -----
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| `` echo `uname` \| cat `` | 1.000 ± 0.000 | 1.000 | 1.000 | 1.00 |
| Command | Mean Wall Time [s] | Change | Factor |
|:---|---:|---:|:---|
| `` echo `uname` \| cat `` | 1.000 ± 0.000 | | |
----- stderr -----
@@ -1183,60 +1168,6 @@ fn unused_parameters_are_shown_in_benchmark_name() {
);
}
#[test]
fn speed_comparison_sort_order() {
let _settings = snapshot_settings().bind_to_scope();
insta::allow_duplicates! {
for sort_order in ["auto", "mean-time"] {
assert_cmd_snapshot!(hyperfine_debug()
.arg("--style=basic")
.arg("sleep 2")
.arg("sleep 1")
.arg(format!("--sort={sort_order}")), @r"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 2
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 2: sleep 1
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Summary
sleep 1 ran
2.00 ± 0.00 times faster than sleep 2
----- stderr -----
");
}
}
assert_cmd_snapshot!(hyperfine_debug()
.arg("--style=basic")
.arg("sleep 2")
.arg("sleep 1")
.arg("--sort=command"), @r"
success: true
exit_code: 0
----- stdout -----
Benchmark 1: sleep 2
Time (mean ± σ): 2.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 2.000 s … 2.000 s 10 runs
Benchmark 2: sleep 1
Time (mean ± σ): 1.000 s ± 0.000 s [User: 0.000 s, System: 0.000 s]
Range (min … max): 1.000 s … 1.000 s 10 runs
Relative speed comparison
2.00 ± 0.00 sleep 2
1.00 sleep 1
----- stderr -----
");
}
#[cfg(windows)]
#[test]
fn windows_quote_args() {
@@ -1308,8 +1239,8 @@ fn json_export_basic() {
"--export-json=-",
"--runs=2",
"--warmup=1",
"--reference-name=one second",
"--reference=sleep 1",
"--command-name=one second",
"sleep 1",
"--command-name=sleep 2",
"sleep 2",
]));
@@ -1338,7 +1269,6 @@ fn json_export_parameterized_with_reference() {
"duration",
"1,2",
"--command-name=sleep for {duration} seconds",
"--reference=sleep for 2 seconds",
"sleep {duration}",
]));
}
tests/snapshots/integration_tests__json_export_basic.snap
+56 -21
View File
@@ -1,5 +1,6 @@
---
source: tests/integration_tests.rs
assertion_line: 1278
info:
program: hyperfine
args:
@@ -8,8 +9,8 @@ info:
- "--export-json=-"
- "--runs=2"
- "--warmup=1"
- "--reference-name=one second"
- "--reference=sleep 1"
- "--command-name=one second"
- sleep 1
- "--command-name=sleep 2"
- sleep 2
---
@@ -27,6 +28,7 @@ exit_code: 0
"architecture": "[architecture]"
}
},
"primary_metric": "time_wall_clock",
"results": [
{
"command": "sleep 1",
@@ -37,6 +39,10 @@ exit_code: 0
"value": 1.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -56,6 +62,10 @@ exit_code: 0
"value": 1.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -71,18 +81,17 @@ exit_code: 0
"exit_code": 0
}
],
"is_reference": true,
"summary": {
"time_wall_clock": {
"unit": "second",
"memory_peak_resident": {
"unit": "byte",
"count": 2,
"mean": 1.0,
"mean": 0.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_user": {
"time_cpu": {
"unit": "second",
"count": 2,
"mean": 0.0,
@@ -100,14 +109,23 @@ exit_code: 0
"min": 0.0,
"max": 0.0
},
"memory_peak_resident": {
"unit": "byte",
"time_user": {
"unit": "second",
"count": 2,
"mean": 0.0,
"stddev": 0.0,
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 1.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
}
}
},
@@ -119,6 +137,10 @@ exit_code: 0
"value": 2.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -138,6 +160,10 @@ exit_code: 0
"value": 2.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -154,16 +180,16 @@ exit_code: 0
}
],
"summary": {
"time_wall_clock": {
"unit": "second",
"memory_peak_resident": {
"unit": "byte",
"count": 2,
"mean": 2.0,
"mean": 0.0,
"stddev": 0.0,
"median": 2.0,
"min": 2.0,
"max": 2.0
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_user": {
"time_cpu": {
"unit": "second",
"count": 2,
"mean": 0.0,
@@ -181,14 +207,23 @@ exit_code: 0
"min": 0.0,
"max": 0.0
},
"memory_peak_resident": {
"unit": "byte",
"time_user": {
"unit": "second",
"count": 2,
"mean": 0.0,
"stddev": 0.0,
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 2.0,
"stddev": 0.0,
"median": 2.0,
"min": 2.0,
"max": 2.0
}
}
}
tests/snapshots/integration_tests__json_export_parameterized_with_reference.snap
+54 -20
View File
@@ -1,5 +1,6 @@
---
source: tests/integration_tests.rs
assertion_line: 1305
info:
program: hyperfine
args:
@@ -11,7 +12,6 @@ info:
- duration
- "1,2"
- "--command-name=sleep for {duration} seconds"
- "--reference=sleep for 2 seconds"
- "sleep {duration}"
---
success: true
@@ -28,6 +28,7 @@ exit_code: 0
"architecture": "[architecture]"
}
},
"primary_metric": "time_wall_clock",
"results": [
{
"command": "sleep 1",
@@ -38,6 +39,10 @@ exit_code: 0
"value": 1.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -57,6 +62,10 @@ exit_code: 0
"value": 1.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -78,16 +87,16 @@ exit_code: 0
}
},
"summary": {
"time_wall_clock": {
"unit": "second",
"memory_peak_resident": {
"unit": "byte",
"count": 2,
"mean": 1.0,
"mean": 0.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_user": {
"time_cpu": {
"unit": "second",
"count": 2,
"mean": 0.0,
@@ -105,14 +114,23 @@ exit_code: 0
"min": 0.0,
"max": 0.0
},
"memory_peak_resident": {
"unit": "byte",
"time_user": {
"unit": "second",
"count": 2,
"mean": 0.0,
"stddev": 0.0,
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 1.0,
"stddev": 0.0,
"median": 1.0,
"min": 1.0,
"max": 1.0
}
}
},
@@ -125,6 +143,10 @@ exit_code: 0
"value": 2.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -144,6 +166,10 @@ exit_code: 0
"value": 2.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -164,18 +190,17 @@ exit_code: 0
"value": "2"
}
},
"is_reference": true,
"summary": {
"time_wall_clock": {
"unit": "second",
"memory_peak_resident": {
"unit": "byte",
"count": 2,
"mean": 2.0,
"mean": 0.0,
"stddev": 0.0,
"median": 2.0,
"min": 2.0,
"max": 2.0
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_user": {
"time_cpu": {
"unit": "second",
"count": 2,
"mean": 0.0,
@@ -193,14 +218,23 @@ exit_code: 0
"min": 0.0,
"max": 0.0
},
"memory_peak_resident": {
"unit": "byte",
"time_user": {
"unit": "second",
"count": 2,
"mean": 0.0,
"stddev": 0.0,
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_wall_clock": {
"unit": "second",
"count": 2,
"mean": 2.0,
"stddev": 0.0,
"median": 2.0,
"min": 2.0,
"max": 2.0
}
}
}
tests/snapshots/integration_tests__json_export_single_run.snap
+24 -9
View File
@@ -1,5 +1,6 @@
---
source: tests/integration_tests.rs
assertion_line: 1294
info:
program: hyperfine
args:
@@ -23,6 +24,7 @@ exit_code: 0
"architecture": "[architecture]"
}
},
"primary_metric": "time_wall_clock",
"results": [
{
"command": "sleep 1",
@@ -32,6 +34,10 @@ exit_code: 0
"value": 1.0,
"unit": "second"
},
"time_cpu": {
"value": 0.0,
"unit": "second"
},
"time_user": {
"value": 0.0,
"unit": "second"
@@ -48,16 +54,16 @@ exit_code: 0
}
],
"summary": {
"time_wall_clock": {
"unit": "second",
"memory_peak_resident": {
"unit": "byte",
"count": 1,
"mean": 1.0,
"mean": 0.0,
"stddev": null,
"median": 1.0,
"min": 1.0,
"max": 1.0
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_user": {
"time_cpu": {
"unit": "second",
"count": 1,
"mean": 0.0,
@@ -75,14 +81,23 @@ exit_code: 0
"min": 0.0,
"max": 0.0
},
"memory_peak_resident": {
"unit": "byte",
"time_user": {
"unit": "second",
"count": 1,
"mean": 0.0,
"stddev": null,
"median": 0.0,
"min": 0.0,
"max": 0.0
},
"time_wall_clock": {
"unit": "second",
"count": 1,
"mean": 1.0,
"stddev": null,
"median": 1.0,
"min": 1.0,
"max": 1.0
}
}
}