Main entry point to nanobench's benchmarking facility. More...

#include <nanobench.h>

Collaboration diagram for ankerl::nanobench::Bench:

Public Member Functions
	Bench ()
	Creates a new benchmark for configuration and running of benchmarks. More...

	Bench (Bench &&other) noexcept

Bench &	operator= (Bench &&other) noexcept(ANKERL_NANOBENCH(NOEXCEPT_STRING_MOVE))

	Bench (Bench const &other)

Bench &	operator= (Bench const &other)

	~Bench () noexcept

template<typename Op >
Bench &	run (char const *benchmarkName, Op &&op)
	Repeatedly calls `op()` based on the configuration, and performs measurements. More...

template<typename Op >
Bench &	run (std::string const &benchmarkName, Op &&op)

template<typename Op >
Bench &	run (Op &&op)
	Same as run(char const* benchmarkName, Op op), but instead uses the previously set name. More...

Bench &	title (char const *benchmarkTitle)
	Title of the benchmark, will be shown in the table header. More...

Bench &	title (std::string const &benchmarkTitle)

ANKERL_NANOBENCH(NODISCARD) std Bench &	name (char const *benchmarkName)
	Gets the title of the benchmark. More...

Bench &	name (std::string const &benchmarkName)

ANKERL_NANOBENCH(NODISCARD) std Bench &	context (char const variableName, char const variableValue)
	Set context information. More...

Bench &	context (std::string const &variableName, std::string const &variableValue)

Bench &	clearContext ()
	Reset context information. More...

template<typename T >
Bench &	batch (T b) noexcept
	Sets the batch size. More...

	ANKERL_NANOBENCH (NODISCARD) double batch() const noexcept

Bench &	unit (char const *unit)
	Sets the operation unit. More...

Bench &	unit (std::string const &unit)

ANKERL_NANOBENCH(NODISCARD) std Bench &	timeUnit (std::chrono::duration< double > const &tu, std::string const &tuName)
	Sets the time unit to be used for the default output. More...

ANKERL_NANOBENCH(NODISCARD) std ANKERL_NANOBENCH(NODISCARD) std Bench &	output (std::ostream *outstream) noexcept
	Set the output stream where the resulting markdown table will be printed to. More...

ANKERL_NANOBENCH(NODISCARD) std Bench &	clockResolutionMultiple (size_t multiple) noexcept
	Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds. More...

	ANKERL_NANOBENCH (NODISCARD) size_t clockResolutionMultiple() const noexcept

Bench &	epochs (size_t numEpochs) noexcept
	Controls number of epochs, the number of measurements to perform. More...

	ANKERL_NANOBENCH (NODISCARD) size_t epochs() const noexcept

Bench &	maxEpochTime (std::chrono::nanoseconds t) noexcept
	Upper limit for the runtime of each epoch. More...

ANKERL_NANOBENCH(NODISCARD) std Bench &	minEpochTime (std::chrono::nanoseconds t) noexcept
	Minimum time each epoch should take. More...

ANKERL_NANOBENCH(NODISCARD) std Bench &	minEpochIterations (uint64_t numIters) noexcept
	Sets the minimum number of iterations each epoch should take. More...

	ANKERL_NANOBENCH (NODISCARD) uint64_t minEpochIterations() const noexcept

Bench &	epochIterations (uint64_t numIters) noexcept
	Sets exactly the number of iterations for each epoch. More...

	ANKERL_NANOBENCH (NODISCARD) uint64_t epochIterations() const noexcept

Bench &	warmup (uint64_t numWarmupIters) noexcept
	Sets a number of iterations that are initially performed without any measurements. More...

	ANKERL_NANOBENCH (NODISCARD) uint64_t warmup() const noexcept

Bench &	relative (bool isRelativeEnabled) noexcept
	Marks the next run as the baseline. More...

	ANKERL_NANOBENCH (NODISCARD) bool relative() const noexcept

Bench &	performanceCounters (bool showPerformanceCounters) noexcept
	Enables/disables performance counters. More...

	ANKERL_NANOBENCH (NODISCARD) bool performanceCounters() const noexcept

template<typename Arg >
ANKERL_NANOBENCH(NODISCARD) std Bench &	doNotOptimizeAway (Arg &&arg)
	Retrieves all benchmark results collected by the bench object so far. More...

template<typename T >
Bench &	complexityN (T n) noexcept

	ANKERL_NANOBENCH (NODISCARD) double complexityN() const noexcept

std::vector< BigO >	complexityBigO () const

template<typename Op >
BigO	complexityBigO (char const *name, Op op) const
	Calculates bigO for a custom function. More...

template<typename Op >
BigO	complexityBigO (std::string const &name, Op op) const

Bench &	render (char const *templateContent, std::ostream &os)

Bench &	render (std::string const &templateContent, std::ostream &os)

Bench &	config (Config const &benchmarkConfig)

	ANKERL_NANOBENCH (NODISCARD) Config const &config() const noexcept

template<typename Arg >
Bench &	doNotOptimizeAway (Arg &&arg)

Private Attributes
Config	mConfig {}

std::vector< Result >	mResults {}

Detailed Description

Main entry point to nanobench's benchmarking facility.

It holds configuration and results from one or more benchmark runs. Usually it is used in a single line, where the object is constructed, configured, and then a benchmark is run. E.g. like this:

ankerl::nanobench::Bench().unit("byte").batch(1000).run("random fluctuations", [&] {
    // here be the benchmark code
});

In that example Bench() constructs the benchmark, it is then configured with unit() and batch(), and after configuration a benchmark is executed with run(). Once run() has finished, it prints the result to std::cout. It would also store the results in the Bench instance, but in this case the object is immediately destroyed so it's not available any more.

Definition at line 627 of file nanobench.h.

Constructor & Destructor Documentation

◆ Bench() [1/3]

ankerl::nanobench::Bench::Bench ( )

Creates a new benchmark for configuration and running of benchmarks.

◆ Bench() [2/3]

ankerl::nanobench::Bench::Bench ( Bench && other )

noexcept

◆ Bench() [3/3]

ankerl::nanobench::Bench::Bench ( Bench const & other )

◆ ~Bench()

ankerl::nanobench::Bench::~Bench ( )

noexcept

Member Function Documentation

◆ ANKERL_NANOBENCH() [1/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [2/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [3/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const &

noexcept

◆ ANKERL_NANOBENCH() [4/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [5/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [6/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [7/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [8/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [9/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ ANKERL_NANOBENCH() [10/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD ) const

noexcept

◆ batch()

template<typename T >

Bench & ankerl::nanobench::Bench::batch ( T b )

noexcept

Sets the batch size.

E.g. number of processed byte, or some other metric for the size of the processed data in each iteration. If you benchmark hashing of a 1000 byte long string and want byte/sec as a result, you can specify 1000 as the batch size.

Template Parameters

T	Any input type is internally cast to `double`.

Parameters

b	batch size

Definition at line 1258 of file nanobench.h.

Here is the caller graph for this function:

◆ clearContext()

Bench & ankerl::nanobench::Bench::clearContext ( )

Reset context information.

This may improve efficiency when using many context entries, or improve robustness by removing spurious context entries.

See also: context

◆ clockResolutionMultiple()

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::clockResolutionMultiple ( size_t multiple )

noexcept

Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds.

This is the main trick nanobech to be so fast: we find out how accurate the clock is, then run the benchmark only so often that the clock's accuracy is good enough for accurate measurements.

The default is to run one epoch for 1000 times the clock resolution. So for 20ns resolution and 11 epochs, this gives a total runtime of

$20ns * 1000 * 11 \approx 0.2ms$

To be precise, nanobench adds a 0-20% random noise to each evaluation. This is to prevent any aliasing effects, and further improves accuracy.

Total runtime will be higher though: Some initial time is needed to find out the target number of iterations for each epoch, and there is some overhead involved to start & stop timers and calculate resulting statistics and writing the output.

Parameters

multiple Target number of times of clock resolution. Usually 1000 is a good compromise between runtime and accuracy.

◆ complexityBigO() [1/3]

std::vector< BigO > ankerl::nanobench::Bench::complexityBigO ( ) const

Calculates Big O of the results with all preconfigured complexity functions. Currently these complexity functions are fitted into the benchmark results:

$\mathcal{O}(1)$ , $\mathcal{O}(n)$ , $\mathcal{O}(\log{}n)$ , $\mathcal{O}(n\log{}n)$ , $\mathcal{O}(n^2)$ , $\mathcal{O}(n^3)$ .

If we e.g. evaluate the complexity of std::sort, this is the result of std::cout << bench.complexityBigO():

|   coefficient |   err% | complexity
|--------------:|-------:|------------
|   5.08935e-09 |   2.6% | O(n log n)
|   6.10608e-08 |   8.0% | O(n)
|   1.29307e-11 |  47.2% | O(n^2)
|   2.48677e-15 |  69.6% | O(n^3)
|   9.88133e-06 | 132.3% | O(log n)
|   5.98793e-05 | 162.5% | O(1)

So in this case $\mathcal{O}(n\log{}n)$ provides the best approximation.

embed:rst
See the tutorial :ref:`asymptotic-complexity` for details.

Returns: Evaluation results, which can be printed or otherwise inspected.

Here is the caller graph for this function:

◆ complexityBigO() [2/3]

template<typename Op >

BigO ankerl::nanobench::Bench::complexityBigO	(	char const *	name,
		Op	op
	)		const

Calculates bigO for a custom function.

E.g. to calculate the mean squared error for $\mathcal{O}(\log{}\log{}n)$ , which is not part of the default set of complexityBigO(), you can do this:

auto logLogN = bench.complexityBigO("O(log log n)", [](double n) {
    return std::log2(std::log2(n));
});

The resulting mean squared error can be printed with std::cout << logLogN. E.g. it prints something like this:

2.46985e-05 * O(log log n), rms=1.48121

Template Parameters

Op	Type of mapping operation.

Parameters

name	Name for the function, e.g. "O(log log n)"
op	Op's operator() maps a `double` with the desired complexity function, e.g. `log2(log2(n))`.

Returns: BigO Error calculation, which is streamable to std::cout.

Definition at line 1246 of file nanobench.h.

Here is the call graph for this function:

◆ complexityBigO() [3/3]

template<typename Op >

BigO ankerl::nanobench::Bench::complexityBigO	(	std::string const &	name,
		Op	op
	)		const

Definition at line 1251 of file nanobench.h.

Here is the call graph for this function:

◆ complexityN()

template<typename T >

Bench & ankerl::nanobench::Bench::complexityN ( T n )

noexcept

embed:rst

Sets N for asymptotic complexity calculation, so it becomes possible to calculate `Big O
<https://en.wikipedia.org/wiki/Big_O_notation>`_ from multiple benchmark evaluations.

Use :cpp:func:`ankerl::nanobench::Bench::complexityBigO` when the evaluation has finished. See the tutorial
:ref:`asymptotic-complexity` for details.

Template Parameters

T	Any type is cast to `double`.

Parameters

n	Length of N for the next benchmark run, so it is possible to calculate `bigO`.

Definition at line 1265 of file nanobench.h.

Here is the caller graph for this function:

◆ config()

Bench & ankerl::nanobench::Bench::config ( Config const & benchmarkConfig )

◆ context() [1/2]

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::context	(	char const *	variableName,
		char const *	variableValue
	)

Set context information.

The information can be accessed using custom render templates via {{context(variableName)}}. Trying to render a variable that hasn't been set before raises an exception. Not included in (default) markdown table.

See also: clearContext, render

Parameters

variableName	The name of the context variable.
variableValue	The value of the context variable.

◆ context() [2/2]

Bench & ankerl::nanobench::Bench::context	(	std::string const &	variableName,
		std::string const &	variableValue
	)

◆ doNotOptimizeAway() [1/2]

template<typename Arg >

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::doNotOptimizeAway ( Arg && arg )

Retrieves all benchmark results collected by the bench object so far.

Each call to run() generates a Result that is stored within the Bench instance. This is mostly for advanced users who want to see all the nitty gritty details.

Returns: All results collected so far.

embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::doNotOptimizeAway`.

◆ doNotOptimizeAway() [2/2]

template<typename Arg >

Bench & ankerl::nanobench::Bench::doNotOptimizeAway ( Arg && arg )

Definition at line 1272 of file nanobench.h.

Here is the call graph for this function:

◆ epochIterations()

Bench & ankerl::nanobench::Bench::epochIterations ( uint64_t numIters )

noexcept

Sets exactly the number of iterations for each epoch.

Ignores all other epoch limits. This forces nanobench to use exactly the given number of iterations for each epoch, not more and not less. Default is 0 (disabled).

Parameters

numIters Exact number of iterations to use. Set to 0 to disable.

Here is the caller graph for this function:

◆ epochs()

Bench & ankerl::nanobench::Bench::epochs ( size_t numEpochs )

noexcept

Controls number of epochs, the number of measurements to perform.

The reported result will be the median of evaluation of each epoch. The higher you choose this, the more deterministic the result be and outliers will be more easily removed. Also the err% will be more accurate the higher this number is. Note that the err% will not necessarily decrease when number of epochs is increased. But it will be a more accurate representation of the benchmarked code's runtime stability.

Choose the value wisely. In practice, 11 has been shown to be a reasonable choice between runtime performance and accuracy. This setting goes hand in hand with minEpochIterations() (or minEpochTime()). If you are more interested in median runtime, you might want to increase epochs(). If you are more interested in mean runtime, you might want to increase minEpochIterations() instead.

Parameters

numEpochs Number of epochs.

Here is the caller graph for this function:

◆ maxEpochTime()

Bench & ankerl::nanobench::Bench::maxEpochTime ( std::chrono::nanoseconds t )

noexcept

Upper limit for the runtime of each epoch.

As a safety precaution if the clock is not very accurate, we can set an upper limit for the maximum evaluation time per epoch. Default is 100ms. At least a single evaluation of the benchmark is performed.

See also: minEpochTime, minEpochIterations

Parameters

t	Maximum target runtime for a single epoch.

◆ minEpochIterations()

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::minEpochIterations ( uint64_t numIters )

noexcept

Sets the minimum number of iterations each epoch should take.

Default is 1, and we rely on clockResolutionMultiple(). If the err% is high and you want a more smooth result, you might want to increase the minimum number of iterations, or increase the minEpochTime().

See also: minEpochTime, maxEpochTime, minEpochIterations

Parameters

numIters Minimum number of iterations per epoch.

Here is the caller graph for this function:

◆ minEpochTime()

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::minEpochTime ( std::chrono::nanoseconds t )

noexcept

Minimum time each epoch should take.

Default is zero, so we are fully relying on clockResolutionMultiple(). In most cases this is exactly what you want. If you see that the evaluation is unreliable with a high err%, you can increase either minEpochTime() or minEpochIterations().

See also: maxEpochTime, minEpochIterations

Parameters

t	Minimum time each epoch should take.

Here is the caller graph for this function:

◆ name() [1/2]

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::name ( char const * benchmarkName )

Gets the title of the benchmark.

Name of the benchmark, will be shown in the table row.

Here is the caller graph for this function:

◆ name() [2/2]

Bench & ankerl::nanobench::Bench::name ( std::string const & benchmarkName )

◆ operator=() [1/2]

Bench & ankerl::nanobench::Bench::operator= ( Bench && other )

noexcept

◆ operator=() [2/2]

Bench & ankerl::nanobench::Bench::operator= ( Bench const & other )

◆ output()

ANKERL_NANOBENCH(NODISCARD) std ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::output ( std::ostream * outstream )

noexcept

Set the output stream where the resulting markdown table will be printed to.

The default is &std::cout. You can disable all output by setting nullptr.

Parameters

outstream Pointer to output stream, can be nullptr.

Here is the caller graph for this function:

◆ performanceCounters()

Bench & ankerl::nanobench::Bench::performanceCounters ( bool showPerformanceCounters )

noexcept

Enables/disables performance counters.

On Linux nanobench has a powerful feature to use performance counters. This enables counting of retired instructions, count number of branches, missed branches, etc. On default this is enabled, but you can disable it if you don't need that feature.

Parameters

showPerformanceCounters True to enable, false to disable.

◆ relative()

Bench & ankerl::nanobench::Bench::relative ( bool isRelativeEnabled )

noexcept

Marks the next run as the baseline.

Call relative(true) to mark the run as the baseline. Successive runs will be compared to this run. It is calculated by

$100\% * \frac{baseline}{runtime}$

100% means it is exactly as fast as the baseline
>100% means it is faster than the baseline. E.g. 200% means the current run is twice as fast as the baseline.
<100% means it is slower than the baseline. E.g. 50% means it is twice as slow as the baseline.

See the tutorial section "Comparing Results" for example usage.

Parameters

isRelativeEnabled True to enable processing

◆ render() [1/2]

Bench & ankerl::nanobench::Bench::render	(	char const *	templateContent,
		std::ostream &	os
	)

embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::render`.

◆ render() [2/2]

Bench & ankerl::nanobench::Bench::render	(	std::string const &	templateContent,
		std::ostream &	os
	)

◆ run() [1/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run	(	char const *	benchmarkName,
		Op &&	op
	)

Repeatedly calls op() based on the configuration, and performs measurements.

This call is marked with noinline to prevent the compiler to optimize beyond different benchmarks. This can have quite a big effect on benchmark accuracy.

embed:rst
.. note::

  Each call to your lambda must have a side effect that the compiler can't possibly optimize it away. E.g. add a result to an
  externally defined number (like `x` in the above example), and finally call `doNotOptimizeAway` on the variables the compiler
  must not remove. You can also use :cpp:func:`ankerl::nanobench::doNotOptimizeAway` directly in the lambda, but be aware that
  this has a small overhead.

Template Parameters

Op	The code to benchmark.

Definition at line 1234 of file nanobench.h.

◆ run() [2/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run ( Op && op )

Same as run(char const* benchmarkName, Op op), but instead uses the previously set name.

Template Parameters

Op	The code to benchmark.

Definition at line 1212 of file nanobench.h.

Here is the call graph for this function:

◆ run() [3/3]

template<typename Op >

Bench & ankerl::nanobench::Bench::run	(	std::string const &	benchmarkName,
		Op &&	op
	)

Definition at line 1240 of file nanobench.h.

◆ timeUnit()

ANKERL_NANOBENCH(NODISCARD) std Bench & ankerl::nanobench::Bench::timeUnit	(	std::chrono::duration< double > const &	tu,
		std::string const &	tuName
	)

Sets the time unit to be used for the default output.

Nanobench defaults to using ns (nanoseconds) as output in the markdown. For some benchmarks this is too coarse, so it is possible to configure this. E.g. use timeUnit(1ms, "ms") to show ms/op instead of ns/op.

Parameters

tu	Time unit to display the results in, default is 1ns.
tuName	Name for the time unit, default is "ns"

◆ title() [1/2]

Bench & ankerl::nanobench::Bench::title ( char const * benchmarkTitle )

Title of the benchmark, will be shown in the table header.

Changing the title will start a new markdown table.

Parameters

benchmarkTitle The title of the benchmark.

◆ title() [2/2]

Bench & ankerl::nanobench::Bench::title ( std::string const & benchmarkTitle )

◆ unit() [1/2]

Bench & ankerl::nanobench::Bench::unit ( char const * unit )

Sets the operation unit.

Defaults to "op". Could be e.g. "byte" for string processing. This is used for the table header, e.g. to show ns/byte. Use singular (byte, not bytes). A change clears the currently collected results.

Parameters

unit	The unit name.

Here is the caller graph for this function:

◆ unit() [2/2]

Bench & ankerl::nanobench::Bench::unit ( std::string const & unit )

◆ warmup()

Bench & ankerl::nanobench::Bench::warmup ( uint64_t numWarmupIters )

noexcept

Sets a number of iterations that are initially performed without any measurements.

Some benchmarks need a few evaluations to warm up caches / database / whatever access. Normally this should not be needed, since we show the median result so initial outliers will be filtered away automatically. If the warmup effect is large though, you might want to set it. Default is 0.

Parameters

numWarmupIters Number of warmup iterations.

Member Data Documentation

◆ mConfig

Config ankerl::nanobench::Bench::mConfig {}

private

Definition at line 1011 of file nanobench.h.

◆ mResults

std::vector<Result> ankerl::nanobench::Bench::mResults {}

private

Definition at line 1012 of file nanobench.h.

The documentation for this class was generated from the following file:

src/bench/nanobench.h

Public Member Functions

Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Bench() [1/3]

◆ Bench() [2/3]

◆ Bench() [3/3]

◆ ~Bench()

Member Function Documentation

◆ ANKERL_NANOBENCH() [1/10]

◆ ANKERL_NANOBENCH() [2/10]

◆ ANKERL_NANOBENCH() [3/10]

◆ ANKERL_NANOBENCH() [4/10]

◆ ANKERL_NANOBENCH() [5/10]

◆ ANKERL_NANOBENCH() [6/10]

◆ ANKERL_NANOBENCH() [7/10]

◆ ANKERL_NANOBENCH() [8/10]

◆ ANKERL_NANOBENCH() [9/10]

◆ ANKERL_NANOBENCH() [10/10]

◆ batch()

◆ clearContext()

◆ clockResolutionMultiple()

◆ complexityBigO() [1/3]

◆ complexityBigO() [2/3]

◆ complexityBigO() [3/3]

◆ complexityN()

◆ config()

◆ context() [1/2]

◆ context() [2/2]

◆ doNotOptimizeAway() [1/2]

◆ doNotOptimizeAway() [2/2]

◆ epochIterations()

◆ epochs()

◆ maxEpochTime()

◆ minEpochIterations()

◆ minEpochTime()

◆ name() [1/2]

◆ name() [2/2]

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ output()

◆ performanceCounters()

◆ relative()

◆ render() [1/2]

◆ render() [2/2]

◆ run() [1/3]

◆ run() [2/3]

◆ run() [3/3]

◆ timeUnit()

◆ title() [1/2]

◆ title() [2/2]

◆ unit() [1/2]

◆ unit() [2/2]

◆ warmup()

Member Data Documentation

◆ mConfig

◆ mResults