A C++ testing framework designed for scientific software
Chebyshev is a header-only C++ testing framework specialized in testing scientific software and scientific computing libraries. It was originally developed to test the larger Theoretica project. The framework is composed of three modules: a prec module for precision testing, a benchmark module for benchmarks, and the err module for error checking.
Features
Chebyshev provides a robust and flexible way to ensure the accuracy, performance, and reliability of scientific computing applications:
- Header-only: Chebyshev is a header-only library, making it easy to integrate into existing projects without requiring additional dependencies or build steps.
- Error integral approximation: Chebyshev provides estimators for the precision of functions over their domain.
- Modular design: The framework is composed of three independent modules, allowing developers to use only the components they need.
- Flexible testing: Chebyshev provides a range of testing functions and is implemented using templates, making it easy to write custom tests for specific use cases.
- Many output formats: The framework supports multiple output formats, such as Markdown and LaTeX tables, and is fully extensible with custom formats.
- Multi-platform support: Chebyshev is designed to work on various platforms, including Windows, Linux, and MacOS, and is fully platform-independent.
Interface
The different modules are used through contexts, which handle requests and the underlying logic. You can create a context for a certain module using <module>::make_context(), which will return a <module>_context class which provides all relevant methods. The behavior of a module may be customized and extended by modifying its settings structure inside the context class. When the destructor of a context or terminate() is called, the results of testing are output to standard output and to file in CSV or other formats for easy analysis, manipulation and visualization.
Precision testing
The precision testing module, implemented in the prec namespace, is designed to verify the accuracy of scientific computing algorithms. It provides a set of functions to compare the results of different implementations, ensuring that they produce identical or equivalent results within a specified tolerance. In addition to checking single equivalences, Chebyshev implements precision estimation techniques, which consist in estimating error integrals of functions over a certain domain. This generally consists in estimating, either with deterministic quadrature methods or Monte Carlo methods, error integrals such as the following:
$$\epsilon_{mean} = \frac{1}{\mu(\Omega)} \int_\Omega |\tilde{f}(x) - f(x)| dx$$
$$\epsilon_{max} = \max_{\Omega} |\tilde{f}(x) - f(x)|$$
The implementation is generalized using templates, making it possible to test quite generic types of functions, from real functions to functions of matrices and vectors or complex numbers. The estimates are computed and the single test cases are validated through a fail function, which determines whether the test failed, depending on its results.
Benchmarks
The benchmark module is used to measure the performance of algorithms and functions in general. It provides a set of macros and functions to time and profile the execution of code, allowing developers to optimize their implementations for speed and efficiency. The benchmark() method works by running the function under consideration for multiple runs and iterations, where runs use the same input, while different iterations use different inputs. The average runtime is then computed and registered. The input to feed the function can be fully customized using, for example, randomized input over the domain of the function.
Error checking
The err module makes it possible to test that functions correctly set errno or throw exceptions. This is achieved for example by calling the functions with values outside of their domain, checking that they report the error correctly. The methods check_errno and check_exception() are used for these type of checks.
Output customization
The additional output module, not directly used for testing, makes it possible to customize the output of the tests, such as which fields to print and how to display them. Customization options are available through the settings structure.
Randomized tests
The random module works in conjunction with the three testing modules to randomize test inputs and provide distribution sampling capabilities for your test units.
Getting Started
To use Chebyshev, simply include the relevant header file for the module that you need in your project and start writing tests. You can alternatively include the chebyshev.h header file which automatically includes all functionalities. The framework is designed to be easy to use, with a minimal learning curve and customization options to shape it according to your needs. This example code sets up precision testing for the "example" test unit and estimates the error over a fictitious function with respect to an exact function:
The output of this simple code, using default options, is:
Setup and Usage
Chebyshev is a header-only library, so there is no need to build or install it separately. Simply include the relevant header files in your project and start using the framework straightaway. Only a compiler with C++14 support is needed to use the framework. To use the library, simply include chebyshev.h or a specific <module>.h header (e.g. benchmark.h).
Contributing
Chebyshev is a collaborative open-source project, and contributions are welcome. If you'd like to contribute to the framework, please submit a pull request.
