blob: a03bae10de2dc3ca3839019b10521b92f0d96885 [file] [log] [blame]
.. _module-pw_compilation_testing:
The pw_compilation_testing module provides for negative compilation (NC)
testing. Negative compilation tests ensure that code that should not compile
does not compile. Negative compilation testing is helpful in a variety of
scenarios, for example:
- Testing for compiler errors, such as ``[[nodiscard]]`` checks.
- Testing that a template cannot be instantiated with certain types.
- Testing that a ``static_assert`` statement is triggered as expected.
- For a ``constexpr`` function, testing that a ``PW_ASSERT`` is triggered as
Negative compilation tests are only supported in GN currently. Negative
compilation tests are not currently supported in GN on Windows due to
`b/241565082 <>`_.
.. warning::
This module is in an early, experimental state. Do not use it unless you have
consulted with the Pigweed team.
Negative compilation test example
.. code-block:: cpp
#include "gtest/gtest.h"
#include "pw_compilation_testing/negative_compilation.h"
template <int kValue>
struct MyStruct {
static_assert(kValue % 2 == 0, "wrong number!");
constexpr int MultiplyOdd(int runtime_value) const {
PW_ASSERT(runtime_value % 2 == 0);
return kValue * runtime_value;
[[maybe_unused]] MyStruct<16> this_one_works;
// NC tests cannot be compiled, so they are created in preprocessor #if or
// #elif blocks. These NC tests check that a static_assert statement fails if
// the code is compiled.
#if PW_NC_TEST(NegativeOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<-1> illegal;
#elif PW_NC_TEST(PositiveOddNumber)
PW_NC_EXPECT("wrong number!");
[[maybe_unused]] MyStruct<5> this_is_illegal;
#endif // PW_NC_TEST
struct Foo {
// Negative compilation tests can go anywhere in a source file.
#if PW_NC_TEST(IllegalValueAsClassMember)
PW_NC_EXPECT("wrong number!");
MyStruct<12> also_illegal;
#endif // PW_NC_TEST
TEST(MyStruct, MultiplyOdd) {
MyStruct<5> five;
EXPECT_EQ(five.MultiplyOdd(3), 15);
// This NC test checks that a specific PW_ASSERT() fails when expected.
// This only works in an NC test if the PW_ASSERT() fails while the compiler
// is executing constexpr code. The test code is used in a constexpr
// statement to force compile-time evaluation.
#if PW_NC_TEST(MyStruct_MultiplyOdd_AssertsOnOddNumber)
[[maybe_unused]] constexpr auto fail = [] {
PW_NC_EXPECT("PW_ASSERT\(runtime_value % 2 == 0\);");
MyStruct<3> my_struct;
return my_struct.MultiplyOdd(4); // Even number, PW_ASSERT should fail.
#endif // PW_NC_TEST
Creating a negative compilation test
- Declare a ``pw_cc_negative_compilation_test()`` GN target or set
``negative_compilation_test = true`` in a ``pw_test()`` target.
- Add the test to the build in a toolchain with negative compilation testing
enabled (``pw_compilation_testing_NEGATIVE_COMPILATION_ENABLED = true``).
- In the test source files, add
``#include "pw_compilation_testing/negative_compilation.h"``.
- Use the ``PW_NC_TEST(TestName)`` macro in a ``#if`` statement.
- Immediately after the ``PW_NC_TEST(TestName)``, provide one or more
Python-style regular expressions with the ``PW_NC_EXPECT()`` macro, one per
- Execute the tests by running the build.
To simplify parsing, all ``PW_NC_TEST()`` statements must fit on a single line.
``PW_NC_EXPECT()`` statements may span multiple lines, but must contain a single
regular expression as a string literal. The string may be comprised of multiple
implicitly concatenated string literals. The ``PW_NC_EXPECT()`` statement cannot
contain anything else except for ``//``-style comments.
Test assertions
Negative compilation tests must have at least one assertion about the
compilation output. The assertion macros must be placed immediately after the
line with the ``PW_NC_TEST()`` or the test will fail.
.. c:macro:: PW_NC_EXPECT(regex_string_literal)
When negative compilation tests are run, checks the compilation output for the
provided regular expression. The argument to the ``PW_NC_EXPECT()`` statement
must be a string literal. The literal is interpreted character-for-character
as a Python raw string literal and compiled as a Python `re
<>`_ regular expression.
For example, ``PW_NC_EXPECT("something (went|has gone) wrong!")`` searches the
failed compilation output with the Python regular expression
``re.compile("something (went|has gone) wrong!")``.
.. c:macro:: PW_NC_EXPECT_GCC(regex_string_literal)
Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with GCC.
.. c:macro:: PW_NC_EXPECT_CLANG(regex_string_literal)
Same as :c:macro:`PW_NC_EXPECT`, but only applies when compiling with Clang.
.. admonition:: Test expectation tips
:class: tip
Be as specific as possible, but avoid compiler-specific error text. Try
matching against the following:
- ``static_assert`` messages.
- Contents of specific failing lines of source code:
- Comments on affected lines: ``PW_NC_EXPECT("// Cannot construct from
- Function names: ``PW_NC_EXPECT("SomeFunction\(\).*private")``.
Do not match against the following:
- Source file paths.
- Source line numbers.
- Compiler-specific wording of error messages, except when necessary.
The basic flow for negative compilation testing is as follows.
- The user defines negative compilation tests in preprocessor ``#if`` blocks
using the ``PW_NC_TEST()`` and :c:macro:`PW_NC_EXPECT` macros.
- The build invokes the ``pw_compilation_testing.generator`` script. The
generator script:
- finds ``PW_NC_TEST()`` statements and extracts a list of test cases,
- finds all associated :c:macro:`PW_NC_EXPECT` statements, and
- generates build targets for each negative compilation tests,
passing the test information and expectations to the targets.
- The build compiles the test source file with all tests disabled.
- The build invokes the negative compilation test targets, which run the
``pw_compilation_testing.runner`` script. The test runner script:
- invokes the compiler, setting a preprocessor macro that enables the ``#if``
block for the test.
- captures the compilation output, and
- checks the compilation output for the :c:macro:`PW_NC_EXPECT` expressions.
- If compilation failed, and the output matches the test case's
:c:macro:`PW_NC_EXPECT` expressions, the test passes.
- If compilation succeeded or the :c:macro:`PW_NC_EXPECT` expressions did not
match the output, the test fails.
Existing frameworks
Pigweed's negative compilation tests were inspired by Chromium's `no-compile
tests <>`_
tests and a similar framework used internally at Google. Pigweed's negative
compilation testing framework improves on these systems in a few respects:
- Trivial integration with unit tests. Negative compilation tests can easily be
placed alongside other unit tests instead of in separate files.
- Safer, more natural macro-based API for test declarations. Other systems use
``#ifdef`` macro checks to define test cases, which fail silently when there
are typos. Pigweed's framework uses function-like macros, which provide a
clean and natural API, catch typos, and ensure the test is integrated with the
NC test framework.
- More readable, flexible test assertions. Other frameworks place assertions in
comments after test names, while Pigweed's framework uses function-like
macros. Pigweed also supports compiler-specific assertions.
- Assertions are required. This helps ensure that compilation fails for the
expected reason and not for an accidental typo or unrelated issue.