|
| 1 | +# Cargo Benchmarking |
| 2 | + |
| 3 | +This directory contains some benchmarks for cargo itself. This uses |
| 4 | +[Criterion] for running benchmarks. It is recommended to read the Criterion |
| 5 | +book to get familiar with how to use it. A basic usage would be: |
| 6 | + |
| 7 | +```sh |
| 8 | +cd benches/benchsuite |
| 9 | +cargo bench |
| 10 | +``` |
| 11 | + |
| 12 | +The tests involve downloading the index and benchmarking against some |
| 13 | +real-world and artificial workspaces located in the [`workspaces`](workspaces) |
| 14 | +directory. |
| 15 | + |
| 16 | +**Beware** that the initial download can take a fairly long amount of time (10 |
| 17 | +minutes minimum on an extremely fast network) and require significant disk |
| 18 | +space (around 4.5GB). The benchsuite will cache the index and downloaded |
| 19 | +crates in the `target/tmp/bench` directory, so subsequent runs should be |
| 20 | +faster. You can (and probably should) specify individual benchmarks to run to |
| 21 | +narrow it down to a more reasonable set, for example: |
| 22 | + |
| 23 | +```sh |
| 24 | +cargo bench -- resolve_ws/rust |
| 25 | +``` |
| 26 | + |
| 27 | +This will only download what's necessary for the rust-lang/rust workspace |
| 28 | +(which is about 330MB) and run the benchmarks against it (which should take |
| 29 | +about a minute). To get a list of all the benchmarks, run: |
| 30 | + |
| 31 | +```sh |
| 32 | +cargo bench -- --list |
| 33 | +``` |
| 34 | + |
| 35 | +## Viewing reports |
| 36 | + |
| 37 | +The benchmarks display some basic information on the command-line while they |
| 38 | +run. A more complete HTML report can be found at |
| 39 | +`target/criterion/report/index.html` which contains links to all the |
| 40 | +benchmarks and summaries. Check out the Criterion book for more information on |
| 41 | +the extensive reporting capabilities. |
| 42 | + |
| 43 | +## Comparing implementations |
| 44 | + |
| 45 | +Knowing the raw numbers can be useful, but what you're probably most |
| 46 | +interested in is checking if your changes help or hurt performance. To do |
| 47 | +that, you need to run the benchmarks multiple times. |
| 48 | + |
| 49 | +First, run the benchmarks from the master branch of cargo without any changes. |
| 50 | +To make it easier to compare, Criterion supports naming the baseline so that |
| 51 | +you can iterate on your code and compare against it multiple times. |
| 52 | + |
| 53 | +```sh |
| 54 | +cargo bench -- --save-baseline master |
| 55 | +``` |
| 56 | + |
| 57 | +Now you can switch to your branch with your changes. Re-run the benchmarks |
| 58 | +compared against the baseline: |
| 59 | + |
| 60 | +```sh |
| 61 | +cargo bench -- --baseline master |
| 62 | +``` |
| 63 | + |
| 64 | +You can repeat the last command as you make changes to re-compare against the |
| 65 | +master baseline. |
| 66 | + |
| 67 | +Without the baseline arguments, it will compare against the last run, which |
| 68 | +can be helpful for comparing incremental changes. |
| 69 | + |
| 70 | +## Capturing workspaces |
| 71 | + |
| 72 | +The [`workspaces`](workspaces) directory contains several workspaces that |
| 73 | +provide a variety of different workspaces intended to provide good exercises |
| 74 | +for benchmarks. Some of these are shadow copies of real-world workspaces. This |
| 75 | +is done with the tool in the [`capture`](capture) directory. The tool will |
| 76 | +copy `Cargo.lock` and all of the `Cargo.toml` files of the workspace members. |
| 77 | +It also adds an empty `lib.rs` so Cargo won't error, and sanitizes the |
| 78 | +`Cargo.toml` to some degree, removing unwanted elements. Finally, it |
| 79 | +compresses everything into a `tgz`. |
| 80 | + |
| 81 | +To run it, do: |
| 82 | + |
| 83 | +```sh |
| 84 | +cd benches/capture |
| 85 | +cargo run -- /path/to/workspace/foo |
| 86 | +``` |
| 87 | + |
| 88 | +The resolver benchmarks also support the `CARGO_BENCH_WORKSPACES` environment |
| 89 | +variable, which you can point to a Cargo workspace if you want to try |
| 90 | +different workspaces. For example: |
| 91 | + |
| 92 | +```sh |
| 93 | +CARGO_BENCH_WORKSPACES=/path/to/some/workspace cargo bench |
| 94 | +``` |
| 95 | + |
| 96 | +## TODO |
| 97 | + |
| 98 | +This is just a start for establishing a benchmarking suite for Cargo. There's |
| 99 | +a lot that can be added. Some ideas: |
| 100 | + |
| 101 | +* Fix the benchmarks so that the resolver setup doesn't run every iteration. |
| 102 | +* Benchmark [this section of |
| 103 | + code](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/ops/cargo_compile.rs#L470-L549) |
| 104 | + which builds the unit graph. The performance there isn't great, and it would |
| 105 | + be good to keep an eye on it. Unfortunately that would mean doing a bit of |
| 106 | + work to make `generate_targets` publicly visible, and there is a bunch of |
| 107 | + setup code that may need to be duplicated. |
| 108 | +* Benchmark the fingerprinting code. |
| 109 | +* Benchmark running the `cargo` executable. Running something like `cargo |
| 110 | + build` or `cargo check` with everything "Fresh" would be a good end-to-end |
| 111 | + exercise to measure the overall overhead of Cargo. |
| 112 | +* Benchmark pathological resolver scenarios. There might be some cases where |
| 113 | + the resolver can spend a significant amount of time. It would be good to |
| 114 | + identify if these exist, and create benchmarks for them. This may require |
| 115 | + creating an artificial index, similar to the `resolver-tests`. This should |
| 116 | + also consider scenarios where the resolver ultimately fails. |
| 117 | +* Benchmark without `Cargo.lock`. I'm not sure if this is particularly |
| 118 | + valuable, since we are mostly concerned with incremental builds which will |
| 119 | + always have a lock file. |
| 120 | +* Benchmark just |
| 121 | + [`resolve::resolve`](https://github.com/rust-lang/cargo/blob/a821e2cb24d7b6013433f069ab3bad53d160e100/src/cargo/core/resolver/mod.rs#L122) |
| 122 | + without anything else. This can help focus on just the resolver. |
| 123 | + |
| 124 | +[Criterion]: https://bheisler.github.io/criterion.rs/book/ |
0 commit comments