Bitcoin Core  0.20.99
P2P Digital Currency
Public Member Functions | Private Attributes | List of all members
ankerl::nanobench::Bench Class Reference

Main entry point to nanobench's benchmarking facility. More...

#include <nanobench.h>

Collaboration diagram for ankerl::nanobench::Bench:
[legend]

Public Member Functions

 Bench ()
 Creates a new benchmark for configuration and running of benchmarks. More...
 
 Bench (Bench &&other)
 
Benchoperator= (Bench &&other)
 
 Bench (Bench const &other)
 
Benchoperator= (Bench const &other)
 
 ~Bench () noexcept
 
template<typename Op >
Benchrun (char const *benchmarkName, Op &&op)
 Repeatedly calls op() based on the configuration, and performs measurements. More...
 
template<typename Op >
Benchrun (std::string const &benchmarkName, Op &&op)
 
template<typename Op >
Benchrun (Op &&op)
 Same as run(char const* benchmarkName, Op op), but instead uses the previously set name. More...
 
Benchtitle (char const *benchmarkTitle)
 Title of the benchmark, will be shown in the table header. More...
 
Benchtitle (std::string const &benchmarkTitle)
 
ANKERL_NANOBENCH(NODISCARD) std Benchname (char const *benchmarkName)
 Name of the benchmark, will be shown in the table row. More...
 
Benchname (std::string const &benchmarkName)
 
template<typename T >
ANKERL_NANOBENCH(NODISCARD) std Benchbatch (T b) noexcept
 Sets the batch size. More...
 
 ANKERL_NANOBENCH (NODISCARD) double batch() const noexcept
 
Benchunit (char const *unit)
 Sets the operation unit. More...
 
Benchunit (std::string const &unit)
 
ANKERL_NANOBENCH(NODISCARD) std Benchoutput (std::ostream *outstream) noexcept
 Set the output stream where the resulting markdown table will be printed to. More...
 
ANKERL_NANOBENCH(NODISCARD) std BenchclockResolutionMultiple (size_t multiple) noexcept
 Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds. More...
 
 ANKERL_NANOBENCH (NODISCARD) size_t clockResolutionMultiple() const noexcept
 
Benchepochs (size_t numEpochs) noexcept
 Controls number of epochs, the number of measurements to perform. More...
 
 ANKERL_NANOBENCH (NODISCARD) size_t epochs() const noexcept
 
BenchmaxEpochTime (std::chrono::nanoseconds t) noexcept
 Upper limit for the runtime of each epoch. More...
 
ANKERL_NANOBENCH(NODISCARD) std BenchminEpochTime (std::chrono::nanoseconds t) noexcept
 Minimum time each epoch should take. More...
 
ANKERL_NANOBENCH(NODISCARD) std BenchminEpochIterations (uint64_t numIters) noexcept
 Sets the minimum number of iterations each epoch should take. More...
 
 ANKERL_NANOBENCH (NODISCARD) uint64_t minEpochIterations() const noexcept
 
BenchepochIterations (uint64_t numIters) noexcept
 Sets exactly the number of iterations for each epoch. More...
 
 ANKERL_NANOBENCH (NODISCARD) uint64_t epochIterations() const noexcept
 
Benchwarmup (uint64_t numWarmupIters) noexcept
 Sets a number of iterations that are initially performed without any measurements. More...
 
 ANKERL_NANOBENCH (NODISCARD) uint64_t warmup() const noexcept
 
Benchrelative (bool isRelativeEnabled) noexcept
 Marks the next run as the baseline. More...
 
 ANKERL_NANOBENCH (NODISCARD) bool relative() const noexcept
 
BenchperformanceCounters (bool showPerformanceCounters) noexcept
 Enables/disables performance counters. More...
 
 ANKERL_NANOBENCH (NODISCARD) bool performanceCounters() const noexcept
 
template<typename Arg >
ANKERL_NANOBENCH(NODISCARD) std BenchdoNotOptimizeAway (Arg &&arg)
 Retrieves all benchmark results collected by the bench object so far. More...
 
template<typename T >
BenchcomplexityN (T b) noexcept
 
 ANKERL_NANOBENCH (NODISCARD) double complexityN() const noexcept
 
std::vector< BigOcomplexityBigO () const
 
template<typename Op >
BigO complexityBigO (char const *name, Op op) const
 Calculates bigO for a custom function. More...
 
template<typename Op >
BigO complexityBigO (std::string const &name, Op op) const
 
Benchrender (char const *templateContent, std::ostream &os)
 
Benchconfig (Config const &benchmarkConfig)
 
 ANKERL_NANOBENCH (NODISCARD) Config const &config() const noexcept
 
template<typename T >
Benchbatch (T b) noexcept
 
template<typename Arg >
BenchdoNotOptimizeAway (Arg &&arg)
 

Private Attributes

Config mConfig {}
 
std::vector< ResultmResults {}
 

Detailed Description

Main entry point to nanobench's benchmarking facility.

It holds configuration and results from one or more benchmark runs. Usually it is used in a single line, where the object is constructed, configured, and then a benchmark is run. E.g. like this:

ankerl::nanobench::Bench().unit("byte").batch(1000).run("random fluctuations", [&] {
    // here be the benchmark code
});

In that example Bench() constructs the benchmark, it is then configured with unit() and batch(), and after configuration a benchmark is executed with run(). Once run() has finished, it prints the result to std::cout. It would also store the results in the Bench instance, but in this case the object is immediately destroyed so it's not available any more.

Definition at line 583 of file nanobench.h.

Constructor & Destructor Documentation

◆ Bench() [1/3]

ankerl::nanobench::Bench::Bench ( )

Creates a new benchmark for configuration and running of benchmarks.

Here is the caller graph for this function:

◆ Bench() [2/3]

ankerl::nanobench::Bench::Bench ( Bench &&  other)

◆ Bench() [3/3]

ankerl::nanobench::Bench::Bench ( Bench const &  other)

◆ ~Bench()

ankerl::nanobench::Bench::~Bench ( )
noexcept
Here is the caller graph for this function:

Member Function Documentation

◆ ANKERL_NANOBENCH() [1/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [2/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [3/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [4/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [5/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [6/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [7/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [8/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [9/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const
noexcept

◆ ANKERL_NANOBENCH() [10/10]

ankerl::nanobench::Bench::ANKERL_NANOBENCH ( NODISCARD  ) const &
noexcept

◆ batch() [1/2]

template<typename T >
ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::batch ( b)
noexcept

Sets the batch size.

E.g. number of processed byte, or some other metric for the size of the processed data in each iteration. If you benchmark hashing of a 1000 byte long string and want byte/sec as a result, you can specify 1000 as the batch size.

Template Parameters
TAny input type is internally cast to double.
Parameters
bbatch size
Here is the caller graph for this function:

◆ batch() [2/2]

template<typename T >
Bench& ankerl::nanobench::Bench::batch ( b)
noexcept

Definition at line 1158 of file nanobench.h.

◆ clockResolutionMultiple()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::clockResolutionMultiple ( size_t  multiple)
noexcept

Modern processors have a very accurate clock, being able to measure as low as 20 nanoseconds.

This is the main trick nanobech to be so fast: we find out how accurate the clock is, then run the benchmark only so often that the clock's accuracy is good enough for accurate measurements.

The default is to run one epoch for 1000 times the clock resolution. So for 20ns resolution and 11 epochs, this gives a total runtime of

\[ 20ns * 1000 * 11 \approx 0.2ms \]

To be precise, nanobench adds a 0-20% random noise to each evaluation. This is to prevent any aliasing effects, and further improves accuracy.

Total runtime will be higher though: Some initial time is needed to find out the target number of iterations for each epoch, and there is some overhead involved to start & stop timers and calculate resulting statistics and writing the output.

Parameters
multipleTarget number of times of clock resolution. Usually 1000 is a good compromise between runtime and accuracy.
Here is the caller graph for this function:

◆ complexityBigO() [1/3]

std::vector<BigO> ankerl::nanobench::Bench::complexityBigO ( ) const

Calculates Big O of the results with all preconfigured complexity functions. Currently these complexity functions are fitted into the benchmark results:

$ \mathcal{O}(1) $, $ \mathcal{O}(n) $, $ \mathcal{O}(\log{}n) $, $ \mathcal{O}(n\log{}n) $, $ \mathcal{O}(n^2) $, $ \mathcal{O}(n^3) $.

If we e.g. evaluate the complexity of std::sort, this is the result of std::cout << bench.complexityBigO():

| coefficient | err% | complexity
|--------------:|-------:|------------
| 5.08935e-09 | 2.6% | O(n log n)
| 6.10608e-08 | 8.0% | O(n)
| 1.29307e-11 | 47.2% | O(n^2)
| 2.48677e-15 | 69.6% | O(n^3)
| 9.88133e-06 | 132.3% | O(log n)
| 5.98793e-05 | 162.5% | O(1)

So in this case $ \mathcal{O}(n\log{}n) $ provides the best approximation.

embed:rst
See the tutorial :ref:`asymptotic-complexity` for details.
Returns
Evaluation results, which can be printed or otherwise inspected.
Here is the caller graph for this function:

◆ complexityBigO() [2/3]

template<typename Op >
BigO ankerl::nanobench::Bench::complexityBigO ( char const *  name,
Op  op 
) const

Calculates bigO for a custom function.

E.g. to calculate the mean squared error for $ \mathcal{O}(\log{}\log{}n) $, which is not part of the default set of complexityBigO(), you can do this:

auto logLogN = bench.complexityBigO("O(log log n)", [](double n) {
return std::log2(std::log2(n));
});

The resulting mean squared error can be printed with std::cout << logLogN. E.g. it prints something like this:

2.46985e-05 * O(log log n), rms=1.48121
Template Parameters
OpType of mapping operation.
Parameters
nameName for the function, e.g. "O(log log n)"
opOp's operator() maps a double with the desired complexity function, e.g. log2(log2(n)).
Returns
BigO Error calculation, which is streamable to std::cout.

Definition at line 1146 of file nanobench.h.

Here is the call graph for this function:

◆ complexityBigO() [3/3]

template<typename Op >
BigO ankerl::nanobench::Bench::complexityBigO ( std::string const &  name,
Op  op 
) const

Definition at line 1151 of file nanobench.h.

Here is the call graph for this function:

◆ complexityN()

template<typename T >
Bench & ankerl::nanobench::Bench::complexityN ( b)
noexcept
embed:rst

Sets N for asymptotic complexity calculation, so it becomes possible to calculate `Big O
<https://en.wikipedia.org/wiki/Big_O_notation>`_ from multiple benchmark evaluations.

Use :cpp:func:`ankerl::nanobench::Bench::complexityBigO` when the evaluation has finished. See the tutorial
:ref:`asymptotic-complexity` for details.
Template Parameters
TAny type is cast to double.
Parameters
bLength of N for the next benchmark run, so it is possible to calculate bigO.

Definition at line 1165 of file nanobench.h.

Here is the caller graph for this function:

◆ config()

Bench& ankerl::nanobench::Bench::config ( Config const &  benchmarkConfig)
Here is the caller graph for this function:

◆ doNotOptimizeAway() [1/2]

template<typename Arg >
ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::doNotOptimizeAway ( Arg &&  arg)

Retrieves all benchmark results collected by the bench object so far.

Each call to run() generates a Result that is stored within the Bench instance. This is mostly for advanced users who want to see all the nitty gritty detials.

Returns
All results collected so far.
embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::doNotOptimizeAway`.

◆ doNotOptimizeAway() [2/2]

template<typename Arg >
Bench& ankerl::nanobench::Bench::doNotOptimizeAway ( Arg &&  arg)

Definition at line 1172 of file nanobench.h.

Here is the call graph for this function:

◆ epochIterations()

Bench& ankerl::nanobench::Bench::epochIterations ( uint64_t  numIters)
noexcept

Sets exactly the number of iterations for each epoch.

Ignores all other epoch limits. This forces nanobench to use exactly the given number of iterations for each epoch, not more and not less. Default is 0 (disabled).

Parameters
numItersExact number of iterations to use. Set to 0 to disable.
Here is the caller graph for this function:

◆ epochs()

Bench& ankerl::nanobench::Bench::epochs ( size_t  numEpochs)
noexcept

Controls number of epochs, the number of measurements to perform.

The reported result will be the median of evaluation of each epoch. The higher you choose this, the more deterministic the result be and outliers will be more easily removed. Also the err% will be more accurate the higher this number is. Note that the err% will not necessarily decrease when number of epochs is increased. But it will be a more accurate representation of the benchmarked code's runtime stability.

Choose the value wisely. In practice, 11 has been shown to be a reasonable choice between runtime performance and accuracy. This setting goes hand in hand with minEpocIterations() (or minEpochTime()). If you are more interested in median runtime, you might want to increase epochs(). If you are more interested in mean runtime, you might want to increase minEpochIterations() instead.

Parameters
numEpochsNumber of epochs.
Here is the caller graph for this function:

◆ maxEpochTime()

Bench& ankerl::nanobench::Bench::maxEpochTime ( std::chrono::nanoseconds  t)
noexcept

Upper limit for the runtime of each epoch.

As a safety precausion if the clock is not very accurate, we can set an upper limit for the maximum evaluation time per epoch. Default is 100ms. At least a single evaluation of the benchmark is performed.

See also
minEpochTime(), minEpochIterations()
Parameters
tMaximum target runtime for a single epoch.
Here is the caller graph for this function:

◆ minEpochIterations()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::minEpochIterations ( uint64_t  numIters)
noexcept

Sets the minimum number of iterations each epoch should take.

Default is 1, and we rely on clockResolutionMultiple(). If the err% is high and you want a more smooth result, you might want to increase the minimum number or iterations, or increase the minEpochTime().

See also
minEpochTime(), maxEpochTime(), minEpochIterations()
Parameters
numItersMinimum number of iterations per epoch.
Here is the caller graph for this function:

◆ minEpochTime()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::minEpochTime ( std::chrono::nanoseconds  t)
noexcept

Minimum time each epoch should take.

Default is zero, so we are fully relying on clockResolutionMultiple(). In most cases this is exactly what you want. If you see that the evaluation is unreliable with a high err%, you can increase either minEpochTime() or minEpochIterations().

See also
maxEpochTime(), minEpochIterations()
Parameters
tMinimum time each epoch should take.
Here is the caller graph for this function:

◆ name() [1/2]

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::name ( char const *  benchmarkName)

Name of the benchmark, will be shown in the table row.

Here is the caller graph for this function:

◆ name() [2/2]

Bench& ankerl::nanobench::Bench::name ( std::string const &  benchmarkName)

◆ operator=() [1/2]

Bench& ankerl::nanobench::Bench::operator= ( Bench &&  other)
Here is the caller graph for this function:

◆ operator=() [2/2]

Bench& ankerl::nanobench::Bench::operator= ( Bench const &  other)

◆ output()

ANKERL_NANOBENCH (NODISCARD) std Bench& ankerl::nanobench::Bench::output ( std::ostream *  outstream)
noexcept

Set the output stream where the resulting markdown table will be printed to.

The default is &std::cout. You can disable all output by setting nullptr.

Parameters
outstreamPointer to output stream, can be nullptr.
Here is the caller graph for this function:

◆ performanceCounters()

Bench& ankerl::nanobench::Bench::performanceCounters ( bool  showPerformanceCounters)
noexcept

Enables/disables performance counters.

On Linux nanobench has a powerful feature to use performance counters. This enables counting of retired instructions, count number of branches, missed branches, etc. On default this is enabled, but you can disable it if you don't need that feature.

Parameters
showPerformanceCountersTrue to enable, false to disable.
Here is the caller graph for this function:

◆ relative()

Bench& ankerl::nanobench::Bench::relative ( bool  isRelativeEnabled)
noexcept

Marks the next run as the baseline.

Call relative(true) to mark the run as the baseline. Successive runs will be compared to this run. It is calculated by

\[ 100\% * \frac{baseline}{runtime} \]

  • 100% means it is exactly as fast as the baseline
  • >100% means it is faster than the baseline. E.g. 200% means the current run is twice as fast as the baseline.
  • <100% means it is slower than the baseline. E.g. 50% means it is twice as slow as the baseline.

See the tutorial section "Comparing Results" for example usage.

Parameters
isRelativeEnabledTrue to enable processing
Here is the caller graph for this function:

◆ render()

Bench& ankerl::nanobench::Bench::render ( char const *  templateContent,
std::ostream &  os 
)
embed:rst

Convenience shortcut to :cpp:func:`ankerl::nanobench::render`.
Here is the caller graph for this function:

◆ run() [1/3]

template<typename Op >
Bench & ankerl::nanobench::Bench::run ( char const *  benchmarkName,
Op &&  op 
)

Repeatedly calls op() based on the configuration, and performs measurements.

This call is marked with noinline to prevent the compiler to optimize beyond different benchmarks. This can have quite a big effect on benchmark accuracy.

embed:rst
.. note::

  Each call to your lambda must have a side effect that the compiler can't possibly optimize it away. E.g. add a result to an
  externally defined number (like `x` in the above example), and finally call `doNotOptimizeAway` on the variables the compiler
  must not remove. You can also use :cpp:func:`ankerl::nanobench::doNotOptimizeAway` directly in the lambda, but be aware that
  this has a small overhead.
Template Parameters
OpThe code to benchmark.

Definition at line 1134 of file nanobench.h.

◆ run() [2/3]

template<typename Op >
Bench & ankerl::nanobench::Bench::run ( std::string const &  benchmarkName,
Op &&  op 
)

Definition at line 1140 of file nanobench.h.

◆ run() [3/3]

template<typename Op >
Bench & ankerl::nanobench::Bench::run ( Op &&  op)

Same as run(char const* benchmarkName, Op op), but instead uses the previously set name.

Template Parameters
OpThe code to benchmark.

Definition at line 1112 of file nanobench.h.

Here is the call graph for this function:

◆ title() [1/2]

Bench& ankerl::nanobench::Bench::title ( char const *  benchmarkTitle)

Title of the benchmark, will be shown in the table header.

Changing the title will start a new markdown table.

Parameters
benchmarkTitleThe title of the benchmark.
Here is the caller graph for this function:

◆ title() [2/2]

Bench& ankerl::nanobench::Bench::title ( std::string const &  benchmarkTitle)

◆ unit() [1/2]

Bench& ankerl::nanobench::Bench::unit ( char const *  unit)

Sets the operation unit.

Defaults to "op". Could be e.g. "byte" for string processing. This is used for the table header, e.g. to show ns/byte. Use singular (byte, not bytes). A change clears the currently collected results.

Parameters
unitThe unit name.
Here is the caller graph for this function:

◆ unit() [2/2]

Bench& ankerl::nanobench::Bench::unit ( std::string const &  unit)

◆ warmup()

Bench& ankerl::nanobench::Bench::warmup ( uint64_t  numWarmupIters)
noexcept

Sets a number of iterations that are initially performed without any measurements.

Some benchmarks need a few evaluations to warm up caches / database / whatever access. Normally this should not be needed, since we show the median result so initial outliers will be filtered away automatically. If the warmup effect is large though, you might want to set it. Default is 0.

Parameters
numWarmupItersNumber of warmup iterations.
Here is the caller graph for this function:

Member Data Documentation

◆ mConfig

Config ankerl::nanobench::Bench::mConfig {}
private

Definition at line 924 of file nanobench.h.

◆ mResults

std::vector<Result> ankerl::nanobench::Bench::mResults {}
private

Definition at line 925 of file nanobench.h.


The documentation for this class was generated from the following file: