| .. _tracing: |
| |
| Tracing |
| ####### |
| |
| Overview |
| ******** |
| |
| The tracing feature provides hooks that permits you to collect data from |
| your application and allows tools running on a host to visualize the inner-working of |
| the kernel and various subsystems. |
| |
| Every system has application-specific events to trace out. Historically, |
| that has implied: |
| |
| 1. Determining the application-specific payload, |
| 2. Choosing suitable serialization-format, |
| 3. Writing the on-target serialization code, |
| 4. Deciding on and writing the I/O transport mechanics, |
| 5. Writing the PC-side deserializer/parser, |
| 6. Writing custom ad-hoc tools for filtering and presentation. |
| |
| An application can use one of the existing formats or define a custom format by |
| overriding the macros declared in :zephyr_file:`include/tracing/tracing.h`. |
| |
| Different formats, transports and host tools are avialable and supported in |
| Zephyr. |
| |
| In fact, I/O varies greatly from system to system. Therefore, it is |
| instructive to create a taxonomy for I/O types when we must ensure the |
| interface between payload/format (Top Layer) and the transport mechanics |
| (bottom Layer) is generic and efficient enough to model these. See the |
| *I/O taxonomy* section below. |
| |
| |
| Serialization Formats |
| ********************** |
| |
| .. _ctf: |
| |
| Common Trace Format (CTF) Support |
| ================================= |
| |
| Common Trace Format, CTF, is an open format and language to describe trace |
| formats. This enables tool reuse, of which line-textual (babeltrace) and |
| graphical (TraceCompass) variants already exist. |
| |
| CTF should look familiar to C programmers but adds stronger typing. |
| See `CTF - A Flexible, High-performance Binary Trace Format |
| <http://diamon.org/ctf/>`_. |
| |
| |
| CTF allows us to formally describe application specific payload and the |
| serialization format, which enables common infrastructure for host tools |
| and parsers and tools for filtering and presentation. |
| |
| |
| A Generic Interface |
| -------------------- |
| |
| In CTF, an event is serialized to a packet containing one or more fields. |
| As seen from *I/O taxonomy* section below, a bottom layer may: |
| |
| - perform actions at transaction-start (e.g. mutex-lock), |
| - process each field in some way (e.g. sync-push emit, concat, enqueue to |
| thread-bound FIFO), |
| - perform actions at transaction-stop (e.g. mutex-release, emit of concat |
| buffer). |
| |
| CTF Top-Layer Example |
| ---------------------- |
| |
| The CTF_EVENT macro will serialize each argument to a field:: |
| |
| /* Example for illustration */ |
| static inline void ctf_top_foo(uint32_t thread_id, ctf_bounded_string_t name) |
| { |
| CTF_EVENT( |
| CTF_LITERAL(uint8_t, 42), |
| thread_id, |
| name, |
| "hello, I was emitted from function: ", |
| __func__ /* __func__ is standard since C99 */ |
| ); |
| } |
| |
| How to serialize and emit fields as well as handling alignment, can be done |
| internally and statically at compile-time in the bottom-layer. |
| |
| |
| The CTF top layer is enabled using the configuration option |
| :option:`CONFIG_TRACING_CTF` and can be used with the different transport |
| backends both in synchronous and asynchronous modes. |
| |
| |
| SEGGER SystemView Support |
| ========================= |
| |
| Zephyr provides built-in support for `SEGGER SystemView`_ that can be enabled in |
| any application for platforms that have the required hardware support. |
| |
| The payload and format used with SystemView is custom to the application and |
| relies on RTT as a transport. Newer versions of SystemView support other |
| transports, for example UART or using snapshot mode (both still not |
| supported in Zephyr). |
| |
| To enable tracing support with `SEGGER SystemView`_ add the configuration option |
| :option:`CONFIG_SEGGER_SYSTEMVIEW` to your project configuration file and set |
| it to *y*. For example, this can be added to the |
| :ref:`synchronization_sample` to visualize fast switching between threads. |
| SystemView can also be used for post-mortem tracing, which can be enabled with |
| `CONFIG_SEGGER_SYSVIEW_POST_MORTEM_MODE`. In this mode, a debugger can |
| be attached after the system has crashed using ``west attach`` after which the |
| latest data from the internal RAM buffer can be loaded into SystemView:: |
| |
| CONFIG_STDOUT_CONSOLE=y |
| # enable to use thread names |
| CONFIG_THREAD_NAME=y |
| CONFIG_SEGGER_SYSTEMVIEW=y |
| CONFIG_USE_SEGGER_RTT=y |
| CONFIG_TRACING=y |
| # enable for post-mortem tracing |
| CONFIG_SEGGER_SYSVIEW_POST_MORTEM_MODE=n |
| |
| |
| .. figure:: segger_systemview.png |
| :align: center |
| :alt: SEGGER SystemView |
| :figclass: align-center |
| :width: 80% |
| |
| .. _SEGGER SystemView: https://www.segger.com/products/development-tools/systemview/ |
| |
| |
| Recent versions of `SEGGER SystemView`_ come with an API translation table for |
| Zephyr which is incomplete and does not match the current level of support |
| available in Zephyr. To use the latest Zephyr API description table, copy the |
| file available in the tree to your local configuration directory to override the |
| builtin table:: |
| |
| # On Linux and MacOS |
| cp ZEPHYR_BASE/subsys/tracing/sysview/SYSVIEW_Zephyr.txt ~/.config/SEGGER/ |
| |
| Transport Backends |
| ****************** |
| |
| The following backends are currently supported: |
| |
| * UART |
| * USB |
| * File (Using native posix port) |
| * RTT (With SystemView) |
| * RAM (buffer to be retrieved by a debugger) |
| |
| Using Tracing |
| ************* |
| |
| The sample :zephyr_file:`samples/subsys/tracing` demonstrates tracing with |
| different formats and backends. |
| |
| To get started, the simplest way is to use the CTF format with the ``native_posix`` |
| port, build the sample as follows: |
| |
| .. zephyr-app-commands:: |
| :tool: all |
| :app: samples/subsys/tracing |
| :board: native_posix |
| :gen-args: -DCONF_FILE=prj_native_posix_ctf.conf |
| :goals: build |
| |
| You can then run the resulting binary with the option ``-trace-file`` to generate |
| the tracing data:: |
| |
| mkdir data |
| cp $ZEPHYR_BASE/subsys/tracing/ctf/tsdl/metadata data/ |
| ./build/zephyr/zephyr.exe -trace-file=data/channel0_0 |
| |
| The resulting CTF output can be visualized using babeltrace or TraceCompass |
| by pointing the tool to the ``data`` directory with the metadata and trace files. |
| |
| Using RAM backend |
| ================= |
| |
| For devices that do not have available I/O for tracing such as USB or UART but have |
| enough RAM to collect trace datas, the ram backend can be enabled with configuration |
| `CONFIG_TRACING_BACKEND_RAM`. |
| Adjust `CONFIG_RAM_TRACING_BUFFER_SIZE` to be able to record enough traces for your needs. |
| Then thanks to a runtime debugger such as gdb this buffer can be fetched from the target |
| to an host computer:: |
| |
| (gdb) dump binary memory data/channel0_0 <ram_tracing_start> <ram_tracing_end> |
| |
| The resulting channel0_0 file have to be placed in a directory with the ``metadata`` |
| file like the other backend. |
| |
| Visualisation Tools |
| ******************* |
| |
| TraceCompass |
| ============= |
| |
| TraceCompass is an open source tool that visualizes CTF events such as thread |
| scheduling and interrupts, and is helpful to find unintended interactions and |
| resource conflicts on complex systems. |
| |
| See also the presentation by Ericsson, |
| `Advanced Trouble-shooting Of Real-time Systems |
| <https://wiki.eclipse.org/images/0/0e/TechTalkOnlineDemoFeb2017_v1.pdf>`_. |
| |
| |
| |
| Future LTTng Inspiration |
| ************************ |
| |
| Currently, the top-layer provided here is quite simple and bare-bones, |
| and needlessly copied from Zephyr's Segger SystemView debug module. |
| |
| For an OS like Zephyr, it would make sense to draw inspiration from |
| Linux's LTTng and change the top-layer to serialize to the same format. |
| Doing this would enable direct reuse of TraceCompass' canned analyses |
| for Linux. Alternatively, LTTng-analyses in TraceCompass could be |
| customized to Zephyr. It is ongoing work to enable TraceCompass |
| visibility of Zephyr in a target-agnostic and open source way. |
| |
| |
| I/O Taxonomy |
| ============= |
| |
| - Atomic Push/Produce/Write/Enqueue: |
| |
| - synchronous: |
| means data-transmission has completed with the return of the |
| call. |
| |
| - asynchronous: |
| means data-transmission is pending or ongoing with the return |
| of the call. Usually, interrupts/callbacks/signals or polling |
| is used to determine completion. |
| |
| - buffered: |
| means data-transmissions are copied and grouped together to |
| form a larger ones. Usually for amortizing overhead (burst |
| dequeue) or jitter-mitigation (steady dequeue). |
| |
| Examples: |
| - sync unbuffered |
| E.g. PIO via GPIOs having steady stream, no extra FIFO memory needed. |
| Low jitter but may be less efficient (cant amortize the overhead of |
| writing). |
| |
| - sync buffered |
| E.g. ``fwrite()`` or enqueuing into FIFO. |
| Blockingly burst the FIFO when its buffer-waterlevel exceeds threshold. |
| Jitter due to bursts may lead to missed deadlines. |
| |
| - async unbuffered |
| E.g. DMA, or zero-copying in shared memory. |
| Be careful of data hazards, race conditions, etc! |
| |
| - async buffered |
| E.g. enqueuing into FIFO. |
| |
| |
| |
| - Atomic Pull/Consume/Read/Dequeue: |
| |
| - synchronous: |
| means data-reception has completed with the return of the call. |
| |
| - asynchronous: |
| means data-reception is pending or ongoing with the return of |
| the call. Usually, interrupts/callbacks/signals or polling is |
| used to determine completion. |
| |
| - buffered: |
| means data is copied-in in larger chunks than request-size. |
| Usually for amortizing wait-time. |
| |
| Examples: |
| - sync unbuffered |
| E.g. Blocking read-call, ``fread()`` or SPI-read, zero-copying in shared |
| memory. |
| |
| - sync buffered |
| E.g. Blocking read-call with caching applied. |
| Makes sense if read pattern exhibits spatial locality. |
| |
| - async unbuffered |
| E.g. zero-copying in shared memory. |
| Be careful of data hazards, race conditions, etc! |
| |
| - async buffered |
| E.g. ``aio_read()`` or DMA. |
| |
| |
| |
| Unfortunately, I/O may not be atomic and may, therefore, require locking. |
| Locking may not be needed if multiple independent channels are available. |
| |
| - The system has non-atomic write and one shared channel |
| E.g. UART. Locking required. |
| |
| ``lock(); emit(a); emit(b); emit(c); release();`` |
| |
| - The system has non-atomic write but many channels |
| E.g. Multi-UART. Lock-free if the bottom-layer maps each Zephyr |
| thread+ISR to its own channel, thus alleviating races as each |
| thread is sequentially consistent with itself. |
| |
| ``emit(a,thread_id); emit(b,thread_id); emit(c,thread_id);`` |
| |
| - The system has atomic write but one shared channel |
| E.g. ``native_posix`` or board with DMA. May or may not need locking. |
| |
| ``emit(a ## b ## c); /* Concat to buffer */`` |
| |
| ``lock(); emit(a); emit(b); emit(c); release(); /* No extra mem */`` |
| |
| - The system has atomic write and many channels |
| E.g. native_posix or board with multi-channel DMA. Lock-free. |
| |
| ``emit(a ## b ## c, thread_id);`` |
| |
| |
| API |
| *** |
| |
| |
| Common |
| ====== |
| |
| .. doxygengroup:: tracing_apis |
| |
| Threads |
| ======= |
| |
| .. doxygengroup:: thread_tracing_apis |
| |
| |
| Work Queues |
| =========== |
| |
| .. doxygengroup:: work_tracing_apis |
| |
| |
| Poll |
| ==== |
| |
| .. doxygengroup:: poll_tracing_apis |
| |
| Semaphore |
| ========= |
| |
| .. doxygengroup:: sem_tracing_apis |
| |
| Mutex |
| ===== |
| |
| .. doxygengroup:: mutex_tracing_apis |
| |
| Condition Variables |
| =================== |
| |
| .. doxygengroup:: condvar_tracing_apis |
| |
| Queues |
| ====== |
| |
| .. doxygengroup:: queue_tracing_apis |
| |
| FIFO |
| ==== |
| |
| .. doxygengroup:: fifo_tracing_apis |
| |
| LIFO |
| ==== |
| .. doxygengroup:: lifo_tracing_apis |
| |
| Stacks |
| ====== |
| |
| .. doxygengroup:: stack_tracing_apis |
| |
| Message Queues |
| ============== |
| |
| .. doxygengroup:: msgq_tracing_apis |
| |
| Mailbox |
| ======= |
| |
| .. doxygengroup:: mbox_tracing_apis |
| |
| Pipes |
| ====== |
| |
| .. doxygengroup:: pipe_tracing_apis |
| |
| Heaps |
| ===== |
| |
| .. doxygengroup:: heap_tracing_apis |
| |
| Memory Slabs |
| ============ |
| |
| .. doxygengroup:: mslab_tracing_apis |
| |
| Timers |
| ====== |
| |
| .. doxygengroup:: timer_tracing_apis |