blob: 00ce0c18447bdba8ee80a8b8855b665e78c4ae0f [file] [log] [blame]
.. _module-pw_compilation_testing:
The pw_compilation_testing module provides for negative compilation (NC)
testing. Negative compilation tests ensure that code that should not compile
does not compile. Negative compilation testing is helpful in a variety of
scenarios, for example:
- Testing for compiler errors, such as ``[[nodiscard]]`` checks.
- Testing that a template cannot be instantiated with certain types.
- Testing that a ``static_assert`` statement is triggered as expected.
- For a ``constexpr`` function, testing that a ``PW_ASSERT`` is triggered as
Negative compilation tests are only supported in GN currently. Negative
compilation tests are not currently supported in GN on Windows due to
`b/241565082 <>`_.
.. warning::
This module is in an early, experimental state. Do not use it unless you have
consulted with the Pigweed team.
Negative compilation test example
.. code-block:: cpp
#include "gtest/gtest.h"
#include "pw_compilation_testing/negative_compilation.h"
template <int kValue>
struct MyStruct {
static_assert(kValue % 2 == 0, "wrong number!");
constexpr int MultiplyOdd(int runtime_value) const {
PW_ASSERT(runtime_value % 2 == 0);
return kValue * runtime_value;
[[maybe_unused]] MyStruct<16> this_one_works;
// NC tests cannot be compiled, so they are created in preprocessor #if or
// #elif blocks. These NC tests check that a static_assert statement fails if
// the code is compiled.
#if PW_NC_TEST(NegativeOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<-1> illegal;
#elif PW_NC_TEST(PositiveOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<5> this_is_illegal;
#endif // PW_NC_TEST
struct Foo {
// Negative compilation tests can go anywhere in a source file.
#if PW_NC_TEST(IllegalValueAsClassMember)
PW_NC_EXPECT("wrong number!");
MyStruct<12> also_illegal;
#endif // PW_NC_TEST
TEST(MyStruct, MultiplyOdd) {
MyStruct<5> five;
EXPECT_EQ(five.MultiplyOdd(3), 15);
// This NC test checks that a specific PW_ASSERT() fails when expected.
// This only works in an NC test if the PW_ASSERT() fails while the compiler
// is executing constexpr code. The test code is used in a constexpr
// statement to force compile-time evaluation.
#if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber)
[[maybe_unused]] constexpr auto fail = [] {
PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);");
MyStruct<3> my_struct;
return my_struct.MultiplyOdd(4); // Even number, PW_ASSERT should fail.
#endif // PW_NC_TEST
Creating a negative compilation test
- Declare a ``pw_cc_negative_compilation_test()`` GN target or set
``negative_compilation_test = true`` in a ``pw_test()`` target.
- Add the test to the build in a toolchain with negative compilation testing
enabled (``pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true``).
- In the test source files, add
``#include "pw_compilation_testing/negative_compilation.h"``.
- Use the ``PW_NC_TEST(TestName)`` macro in a ``#if`` statement.
- Immediately after the ``PW_NC_TEST(TestName)``, provide one or more
Python-style regular expressions with the ``PW_NC_EXPECT()`` macro, one per
- Execute the tests by running the build.
To simplify parsing, all ``PW_NC_TEST()`` and ``PW_NC_EXPECT()`` statements
must fit within a single line. If they are too long for one line, disable
automatic formatting around them with ``// clang-format disable`` or split
``PW_NC_EXPECT()`` statements into multiple statements on separate lines. This
restriction may be relaxed in the future.
Test assertions
Negative compilation tests must have at least one assertion about the
compilation output. The assertion macros must be placed immediately after the
line with the ``PW_NC_TEST()`` or the test will fail.
.. c:macro:: PW_NC_EXPECT(regex_string_literal)
When negative compilation tests are run, checks the compilation output for the
provided regular expression. The argument to the ``PW_NC_EXPECT()`` statement
must be a string literal. The literal is interpreted character-for-character
as a Python raw string literal and compiled as a Python `re
<>`_ regular expression.
For example, ``PW_NC_EXPECT("something (went|has gone) wrong!")`` searches the
failed compilation output with the Python regular expression
``re.compile("something (went|has gone) wrong!")``.
.. c:macro:: PW_NC_EXPECT_GCC(regex_string_literal)
Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with GCC.
.. c:macro:: PW_NC_EXPECT_CLANG(regex_string_literal)
Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with Clang.
.. admonition:: Test expectation tips
:class: tip
Be as specific as possible, but avoid compiler-specific error text. Try
matching against the following:
- ``static_assert`` messages.
- Contents of specific failing lines of source code:
- Comments on affected lines: ``PW_NC_EXPECT("// Cannot construct from
- Function names: ``PW_NC_EXPECT("SomeFunction\(\).*private")``.
Do not match against the following:
- Source file paths.
- Source line numbers.
- Compiler-specific wording of error messages, except when necessary.
The basic flow for negative compilation testing is as follows.
- The user defines negative compilation tests in preprocessor ``#if`` blocks
using the ``PW_NC_TEST()`` and :c:macro:`PW_NC_EXPECT` macros.
- The build invokes the ``pw_compilation_testing.generator`` script. The
generator script:
- finds ``PW_NC_TEST()`` statements and extracts a list of test cases,
- finds all associated :c:macro:`PW_NC_EXPECT` statements, and
- generates build targets for each negative compilation tests,
passing the test information and expectations to the targets.
- The build compiles the test source file with all tests disabled.
- The build invokes the negative compilation test targets, which run the
``pw_compilation_testing.runner`` script. The test runner script:
- invokes the compiler, setting a preprocessor macro that enables the ``#if``
block for the test.
- captures the compilation output, and
- checks the compilation output for the :c:macro:`PW_NC_EXPECT` expressions.
- If compilation failed, and the output matches the test case's
:c:macro:`PW_NC_EXPECT` expressions, the test passes.
- If compilation succeeded or the :c:macro:`PW_NC_EXPECT` expressions did not
match the output, the test fails.
Existing frameworks
Pigweed's negative compilation tests were inspired by Chromium's `no-compile
tests <>`_
tests and a similar framework used internally at Google. Pigweed's negative
compilation testing framework improves on these systems in a few respects:
- Trivial integration with unit tests. Negative compilation tests can easily be
placed alongside other unit tests instead of in separate files.
- Safer, more natural macro-based API for test declarations. Other systems use
``#ifdef`` macro checks to define test cases, which fail silently when there
are typos. Pigweed's framework uses function-like macros, which provide a
clean and natural API, catch typos, and ensure the test is integrated with the
NC test framework.
- More readable, flexible test assertions. Other frameworks place assertions in
comments after test names, while Pigweed's framework uses function-like
macros. Pigweed also supports compiler-specific assertions.
- Assertions are required. This helps ensure that compilation fails for the
expected reason and not for an accidental typo or unrelated issue.