This directory contains benchmarking schemas and data sets that you can use to test a variety of performance scenarios against your protobuf language runtime. If you are looking for performance numbers of officially support languages, see here
First, you need to follow the instruction in the root directory‘s README to build your language’s protobuf, then:
You need to install cmake before building the benchmark.
We are using google/benchmark as the benchmark tool for testing cpp. This will be automaticly made during build the cpp benchmark.
The cpp protobuf performance can be improved by linking with tcmalloc library. For using tcmalloc, you need to build gpertools to generate libtcmallc.so library.
We‘re using maven to build the java benchmarks, which is the same as to build the Java protobuf. There’re no other tools need to install. We're using google/caliper as benchmark tool, which can be automaticly included by maven.
We‘re using python C++ API for testing the generated CPP proto version of python protobuf, which is also a prerequisite for Python protobuf cpp implementation. You need to install the correct version of Python C++ extension package before run generated CPP proto version of Python protobuf’s benchmark. e.g. under Ubuntu, you need to
$ sudo apt-get install python-dev $ sudo apt-get install python3-dev
And you also need to make sure pkg-config
is installed.
Go protobufs are maintained at github.com/golang/protobuf. If not done already, you need to install the toolchain and the Go protoc-gen-go plugin for protoc.
To install protoc-gen-go, run:
$ go get -u github.com/golang/protobuf/protoc-gen-go $ export PATH=$PATH:$(go env GOPATH)/bin
The first command installs protoc-gen-go
into the bin
directory in your local GOPATH
. The second command adds the bin
directory to your PATH
so that protoc
can locate the plugin later.
There's some optional big testing data which is not included in the directory initially, you need to run the following command to download the testing data:
$ ./download_data.sh
After doing this the big data file will automaticly generated in the benchmark directory.
To run all the benchmark dataset:
$ make java
$ make cpp
For linking with tcmalloc:
$ env LD_PRELOAD={directory to libtcmalloc.so} make cpp
We have three versions of python protobuf implementation: pure python, cpp reflection and cpp generated code. To run these version benchmark, you need to:
$ make python-pure-python
$ make python-cpp-reflection
$ make python-cpp-generated-code
$ make go
To run a specific dataset or run with specific options:
$ make java-benchmark $ ./java-benchmark $(specific generated dataset file name) [$(caliper options)]
$ make cpp-benchmark $ ./cpp-benchmark $(specific generated dataset file name) [$(benchmark options)]
For Python benchmark we have --json
for outputing the json result
$ make python-pure-python-benchmark $ ./python-pure-python-benchmark [--json] $(specific generated dataset file name)
$ make python-cpp-reflection-benchmark $ ./python-cpp-reflection-benchmark [--json] $(specific generated dataset file name)
$ make python-cpp-generated-code-benchmark $ ./python-cpp-generated-code-benchmark [--json] $(specific generated dataset file name)
$ make go-benchmark $ ./go-benchmark $(specific generated dataset file name) [go testing options]
Each data set is in the format of benchmarks.proto:
The schema for the datasets is described in benchmarks.proto
.
Benchmark likely want to run several benchmarks against each data set (parse, serialize, possibly JSON, possibly using different APIs, etc).
We would like to add more data sets. In general we will favor data sets that make the overall suite diverse without being too large or having too many similar tests. Ideally everyone can run through the entire suite without the test run getting too long.