blob: acb6c8b5b25a272217e635f272375dda96c36632 [file] [view]
# Benchmark
This directory contains gRPC-Go's benchmarking utilities and libraries.
This README is the main guide to running the benchmarks and comparing their
results. It covers the directory layout, a workflow for measuring the effect of
a change, and the most common `benchmain` flags. For more advanced options, such
as weighted random payload size distributions, see
[Documentation/benchmark.md](../Documentation/benchmark.md).
All commands in this README are run from the repository root.
## Contents
- **benchmain/** - the main benchmark driver. Run with
`go run benchmark/benchmain/main.go -benchtime=60s -workloads=all`.
Supports configurable workload type, payload size, compression,
concurrency, network mode, and profiling output.
- **benchresult/** - formats and compares `benchmain` result files, e.g.
`go run benchmark/benchresult/main.go curPerf` to view a result, or pass
a base and current result file to compare performance across changes.
- **client/** - standalone gRPC benchmark client binary. Run against a
manually-started `server/` instance.
- **server/** - standalone gRPC benchmark server binary used by `client/`
and by `run_bench.sh`.
- **worker/** - a binary that can act as either a benchmark client or
server depending on runtime configuration, used for driver-controlled
benchmarking.
- **flags/** - flag-parsing helpers (e.g. `IntSlice`, `DurationSlice`) that
`benchmain` uses for flags that take comma-separated values.
- **latency/** - network latency/throughput simulation used to emulate
LAN/WAN network conditions during benchmarks.
- **primitives/** - low-level Go primitive benchmarks, not gRPC-specific.
- **stats/** - the `Features`/`Stats` types used to collect, serialize,
and report benchmark results.
- **benchmark.go** - shared helpers (`DoUnaryCall`, `StartServer`, etc.)
used to build custom benchmark clients and servers.
- **run_bench.sh** - builds `server/` and `client/` and runs a sweep of
benchmarks across combinations of RPC count, connection count, request/
response size, and RPC type (unary/streaming). Run
`benchmark/run_bench.sh -h` for options.
## Quick start
Run:
`go run benchmark/benchmain/main.go -benchtime=60s -workloads=all`
Here, `-benchtime=60s` runs each benchmark workload for 60 seconds, and
`-workloads=all` runs every available workload type (unary, streaming,
etc.) rather than a single one.
See [Common flags](#common-flags) below for the flags you are most likely to
use, or run `go run benchmark/benchmain/main.go -h` for the full list.
## Comparing performance before and after your change
If your PR is meant to improve performance, here's how to demonstrate that
in the PR description.
1. On a clean checkout of the base branch (e.g. `master`), run the benchmark
and save the result to a file:
`go run benchmark/benchmain/main.go -benchtime=60s -workloads=all -resultFile=basePerf`
2. Check out your branch with the change applied, and run the same
benchmark, saving to a different file:
`go run benchmark/benchmain/main.go -benchtime=60s -workloads=all -resultFile=curPerf`
3. To view a single result on its own:
`go run benchmark/benchresult/main.go curPerf`
4. To compare the two and see how performance changed:
`go run benchmark/benchresult/main.go basePerf curPerf`
This prints a diff-style report (latency, throughput, allocations, etc.)
between the two runs. Paste that output into your PR description as
evidence of the improvement (or to confirm no regression).
Both `basePerf` and `curPerf` are plain result files produced by
`-resultFile`; you can name them anything, as long as you're consistent
between steps. See [Common flags](#common-flags) for the flags you can vary
between runs.
For example, one configuration from a pair of short unary runs
(`-benchtime=3s -workloads=unary -maxConcurrentCalls=1 -reqSizeBytes=1 -respSizeBytes=1`)
shows up like this:
```
unary-networkMode_none-bufConn_false-keepalive_false-benchTime_3s-trace_false-latency_40ms-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false-clientReadBufferSize_-1-clientWriteBufferSize_-1-serverReadBufferSize_-1-serverWriteBufferSize_-1-sleepBetweenRPCs_0s-connections_1-recvBufferPool_simple-sharedWriteBuffer_true
Title Before After Percentage
TotalOps 37 37 0.00%
SendOps 0 0 NaN%
RecvOps 0 0 NaN%
Bytes/op 9705.08 9632.65 -0.75%
Allocs/op 167.68 163.03 -2.39%
ReqT/op 98.67 98.67 0.00%
RespT/op 98.67 98.67 0.00%
50th-Lat 81.4024ms 82.1738ms 0.95%
90th-Lat 81.8085ms 82.6636ms 1.05%
99th-Lat 82.0173ms 83.9512ms 2.36%
Avg-Lat 81.389024ms 82.293289ms 1.11%
GoVersion go1.27.0 go1.27.0
GrpcVersion 1.85.0-dev 1.85.0-dev
```
The report has one block per benchmark configuration, and the first line
of each block names that configuration. `Before` comes from the first file
(`basePerf`), `After` from the second (`curPerf`), and `Percentage` is the
change between them. Lower latency, `Bytes/op` and `Allocs/op` are better.
Higher `TotalOps` and throughput (`ReqT/op`, `RespT/op`) are better.
`SendOps` and `RecvOps` are only filled in for the `unconstrained`
workload, so they show `NaN%` otherwise.
Note: Latency and throughput measurements can vary by several percent
between identical runs, especially with short `-benchtime` durations or high
simulated latency (where total operations are low). For more reliable
comparisons, use a longer `-benchtime` (e.g. `60s` or more), run on an
otherwise idle machine, and treat small latency/throughput differences with
caution.
## Common flags
Flags marked "comma-separated list" accept several values, and the benchmark
runs every combination of them.
- `-benchtime` - how long to run each benchmark configuration (default `1s`).
- `-workloads` - which RPC types to run: `unary`, `streaming`,
`unconstrained`, or `all` (default `all`).
- `-maxConcurrentCalls` - number of concurrent RPCs per connection
(comma-separated list, default `1,8,64,512`). Total concurrency is
`-connections` multiplied by `-maxConcurrentCalls`.
- `-reqSizeBytes` / `-respSizeBytes` - request and response payload sizes in
bytes (comma-separated list). Both default to `1,1024,1048576`, so a run
covers 3 sizes each by default.
- `-latency` - simulated one-way network latency (comma-separated list,
default `0s,40ms`).
- `-kbps` - simulated network throughput in kbps, where `0` means unlimited
(comma-separated list, default `0,10240`).
- `-networkMode` - preset network conditions. Valid values are `none`
(default), `Local`, `LAN`, `WAN` and `Longhaul`. Values are case-sensitive,
so `-networkMode=lan` is rejected.
- `-compression` - compression mode: `off` (default), `gzip`, `nop` or `all`.
`nop` is a pass-through compressor that runs the compression code path
without compressing the data. `all` runs three configurations: `nop`,
`gzip` and `off`.
- `-connections` - number of connections (default `1`). Each one handles
`-maxConcurrentCalls` RPC streams.
- `-resultFile` - save results to a file for use with `benchresult`.
- `-cpuProfile` / `-memProfile` - write CPU or memory profiles to the given
file.
Because `-latency` and `-kbps` both default to two values, a run that doesn't
set them covers four network configurations. Pass `-latency=0s -kbps=0` to run
a single one.