| # Benchmark | |
| This directory contains gRPC-Go's benchmarking utilities and libraries. | |
| This README is the main guide to running the benchmarks and comparing their | |
| results. It covers the directory layout, a workflow for measuring the effect of | |
| a change, and the most common `benchmain` flags. For more advanced options, such | |
| as weighted random payload size distributions, see | |
| [Documentation/benchmark.md](../Documentation/benchmark.md). | |
| All commands in this README are run from the repository root. | |
| ## Contents | |
| - **benchmain/** - the main benchmark driver. Run with | |
| `go run benchmark/benchmain/main.go -benchtime=60s -workloads=all`. | |
| Supports configurable workload type, payload size, compression, | |
| concurrency, network mode, and profiling output. | |
| - **benchresult/** - formats and compares `benchmain` result files, e.g. | |
| `go run benchmark/benchresult/main.go curPerf` to view a result, or pass | |
| a base and current result file to compare performance across changes. | |
| - **client/** - standalone gRPC benchmark client binary. Run against a | |
| manually-started `server/` instance. | |
| - **server/** - standalone gRPC benchmark server binary used by `client/` | |
| and by `run_bench.sh`. | |
| - **worker/** - a binary that can act as either a benchmark client or | |
| server depending on runtime configuration, used for driver-controlled | |
| benchmarking. | |
| - **flags/** - flag-parsing helpers (e.g. `IntSlice`, `DurationSlice`) that | |
| `benchmain` uses for flags that take comma-separated values. | |
| - **latency/** - network latency/throughput simulation used to emulate | |
| LAN/WAN network conditions during benchmarks. | |
| - **primitives/** - low-level Go primitive benchmarks, not gRPC-specific. | |
| - **stats/** - the `Features`/`Stats` types used to collect, serialize, | |
| and report benchmark results. | |
| - **benchmark.go** - shared helpers (`DoUnaryCall`, `StartServer`, etc.) | |
| used to build custom benchmark clients and servers. | |
| - **run_bench.sh** - builds `server/` and `client/` and runs a sweep of | |
| benchmarks across combinations of RPC count, connection count, request/ | |
| response size, and RPC type (unary/streaming). Run | |
| `benchmark/run_bench.sh -h` for options. | |
| ## Quick start | |
| Run: | |
| `go run benchmark/benchmain/main.go -benchtime=60s -workloads=all` | |
| Here, `-benchtime=60s` runs each benchmark workload for 60 seconds, and | |
| `-workloads=all` runs every available workload type (unary, streaming, | |
| etc.) rather than a single one. | |
| See [Common flags](#common-flags) below for the flags you are most likely to | |
| use, or run `go run benchmark/benchmain/main.go -h` for the full list. | |
| ## Comparing performance before and after your change | |
| If your PR is meant to improve performance, here's how to demonstrate that | |
| in the PR description. | |
| 1. On a clean checkout of the base branch (e.g. `master`), run the benchmark | |
| and save the result to a file: | |
| `go run benchmark/benchmain/main.go -benchtime=60s -workloads=all -resultFile=basePerf` | |
| 2. Check out your branch with the change applied, and run the same | |
| benchmark, saving to a different file: | |
| `go run benchmark/benchmain/main.go -benchtime=60s -workloads=all -resultFile=curPerf` | |
| 3. To view a single result on its own: | |
| `go run benchmark/benchresult/main.go curPerf` | |
| 4. To compare the two and see how performance changed: | |
| `go run benchmark/benchresult/main.go basePerf curPerf` | |
| This prints a diff-style report (latency, throughput, allocations, etc.) | |
| between the two runs. Paste that output into your PR description as | |
| evidence of the improvement (or to confirm no regression). | |
| Both `basePerf` and `curPerf` are plain result files produced by | |
| `-resultFile`; you can name them anything, as long as you're consistent | |
| between steps. See [Common flags](#common-flags) for the flags you can vary | |
| between runs. | |
| For example, one configuration from a pair of short unary runs | |
| (`-benchtime=3s -workloads=unary -maxConcurrentCalls=1 -reqSizeBytes=1 -respSizeBytes=1`) | |
| shows up like this: | |
| ``` | |
| unary-networkMode_none-bufConn_false-keepalive_false-benchTime_3s-trace_false-latency_40ms-kbps_0-MTU_0-maxConcurrentCalls_1-reqSize_1B-respSize_1B-compressor_off-channelz_false-preloader_false-clientReadBufferSize_-1-clientWriteBufferSize_-1-serverReadBufferSize_-1-serverWriteBufferSize_-1-sleepBetweenRPCs_0s-connections_1-recvBufferPool_simple-sharedWriteBuffer_true | |
| Title Before After Percentage | |
| TotalOps 37 37 0.00% | |
| SendOps 0 0 NaN% | |
| RecvOps 0 0 NaN% | |
| Bytes/op 9705.08 9632.65 -0.75% | |
| Allocs/op 167.68 163.03 -2.39% | |
| ReqT/op 98.67 98.67 0.00% | |
| RespT/op 98.67 98.67 0.00% | |
| 50th-Lat 81.4024ms 82.1738ms 0.95% | |
| 90th-Lat 81.8085ms 82.6636ms 1.05% | |
| 99th-Lat 82.0173ms 83.9512ms 2.36% | |
| Avg-Lat 81.389024ms 82.293289ms 1.11% | |
| GoVersion go1.27.0 go1.27.0 | |
| GrpcVersion 1.85.0-dev 1.85.0-dev | |
| ``` | |
| The report has one block per benchmark configuration, and the first line | |
| of each block names that configuration. `Before` comes from the first file | |
| (`basePerf`), `After` from the second (`curPerf`), and `Percentage` is the | |
| change between them. Lower latency, `Bytes/op` and `Allocs/op` are better. | |
| Higher `TotalOps` and throughput (`ReqT/op`, `RespT/op`) are better. | |
| `SendOps` and `RecvOps` are only filled in for the `unconstrained` | |
| workload, so they show `NaN%` otherwise. | |
| Note: Latency and throughput measurements can vary by several percent | |
| between identical runs, especially with short `-benchtime` durations or high | |
| simulated latency (where total operations are low). For more reliable | |
| comparisons, use a longer `-benchtime` (e.g. `60s` or more), run on an | |
| otherwise idle machine, and treat small latency/throughput differences with | |
| caution. | |
| ## Common flags | |
| Flags marked "comma-separated list" accept several values, and the benchmark | |
| runs every combination of them. | |
| - `-benchtime` - how long to run each benchmark configuration (default `1s`). | |
| - `-workloads` - which RPC types to run: `unary`, `streaming`, | |
| `unconstrained`, or `all` (default `all`). | |
| - `-maxConcurrentCalls` - number of concurrent RPCs per connection | |
| (comma-separated list, default `1,8,64,512`). Total concurrency is | |
| `-connections` multiplied by `-maxConcurrentCalls`. | |
| - `-reqSizeBytes` / `-respSizeBytes` - request and response payload sizes in | |
| bytes (comma-separated list). Both default to `1,1024,1048576`, so a run | |
| covers 3 sizes each by default. | |
| - `-latency` - simulated one-way network latency (comma-separated list, | |
| default `0s,40ms`). | |
| - `-kbps` - simulated network throughput in kbps, where `0` means unlimited | |
| (comma-separated list, default `0,10240`). | |
| - `-networkMode` - preset network conditions. Valid values are `none` | |
| (default), `Local`, `LAN`, `WAN` and `Longhaul`. Values are case-sensitive, | |
| so `-networkMode=lan` is rejected. | |
| - `-compression` - compression mode: `off` (default), `gzip`, `nop` or `all`. | |
| `nop` is a pass-through compressor that runs the compression code path | |
| without compressing the data. `all` runs three configurations: `nop`, | |
| `gzip` and `off`. | |
| - `-connections` - number of connections (default `1`). Each one handles | |
| `-maxConcurrentCalls` RPC streams. | |
| - `-resultFile` - save results to a file for use with `benchresult`. | |
| - `-cpuProfile` / `-memProfile` - write CPU or memory profiles to the given | |
| file. | |
| Because `-latency` and `-kbps` both default to two values, a run that doesn't | |
| set them covers four network configurations. Pass `-latency=0s -kbps=0` to run | |
| a single one. |