Introduction
Aerology is a special purpose debugger for inspecting snapshots of Zephyr and TFM Applications.
At the time of writing it is able to provide convinient access to:
- memory layout, with the
segmentssubcommand - Zephyr DTS, with the
dtssubcommand - Zephyr Config, with the
commandsubcommand - stack usage by thread, with the
stackssubcommand - backtraces by thread, with the
backtracesubcommand - and general purpose queries beginning with global variables, with the
querysubcommand
This book in structured as a case study of a particular bug, and will assume familiarity with both Zephyr and TF-M, where appropriate.
Installation
Aerology is written in Rust, and uses the capstone library for disassembly. Since capstone is written in C++, you will need to ensure that you have a suitable compiler instaled.
If a recent version of Rust in not provided by your OS or it's package manager you may find rustup.rs helpful.
Like many Rust applications, Aerology is built with Rust's build tool and package manager Cargo. Installation is also handled by Cargo, and
$ cargo install --path .
should be sufficient to install Aerology.
Debugging Prep
Aerology's debugging subcommands operate primarily on an application snapshot, called a core, and provides a means of dumping a core.
At the moment, Taking a core is a two step process. If this is found to be too comberson, it may be changed.
Aerology builds a single-file application package, called a Zephyr App
Package or ZAP for short, from a Zephyr build directory.
Aerology takes snapshots using this ZAP.
Some subcommands, such as segments, can work with both a zap and a core.
Packing a ZAP
Aerology builds a ZAP with the pack subcommand.
This subcommand requires a build directory and also accepts an optional zap file
name.
The usage is:
aerology pack <BUILD_DIRECTORY> [OUT_FILE]
For example, I could package my version of the net/dhcpv4_client sample built
for the lpcxpresso55s69_ns platform with the following command:
# Verify that this is indeed a zephyr build directory
$ ls build/lpcxpresso55s69_ns/net/dhcpv4_client/
app compile_commands.json tfm_s_signed.hex
build.ninja Kconfig zephyr
cmake_install.cmake modules zephyr_modules.txt
CMakeCache.txt tfm zephyr_ns_signed.hex
CMakeFiles tfm_merged.hex zephyr_settings.txt
# Package the build directory
$ aerology pack build/lpcxpresso55s69_ns/net/dhcpv4_client/
packed into "dhcpv4_client.zap"
As may be evedent from the above shell session, when no output file is specified, Aerology infers a name from the last component of the build directory.
We can test that this is a working zap by running the segments subcommand.
Don't worry about what the output looks like for now, as we'll cover that in
another chapter.
What we're looking for is a lack of errors:
$ aerology segments -s dhcpv4_client.zap
Note: Not to scale.
Key: r = readable, w = writable, x = executable, z = zeroed on startup
┃ = overlapping section
zephyr tfm_s bl2
300274e0┌──────────┐
│ rwz│
300222a0└──────────┘
3002229c┌──────────┐
│ rw│
30022000└──────────┘
3000ce64 ┌──────────┐
│ rwz│
3000b5fc ┢━━━━━━━━━━┪
┃ rw┃
3000b500 ┡━━━━━━━━━━┩
30007d60 │ │┌──────────┐
│ ││ rwz│
30005560 │ │└──────────┘
30005558 │ │┌──────────┐
│ ││ rw│
30002640 ┢━━━━━━━━━━┪│ │
┃ rw┃│ │
300025c0 ┡━━━━━━━━━━┩│ │
30000840 ┢━━━━━━━━━━┪│ │
┃ rw┃│ │
30000820 ┡━━━━━━━━━━┩│ │
30000400 │ │├──────────┤
│ ││ rwz│
30000000 └──────────┘└──────────┘
10044024┌──────────┐
│ rwx│
10030000└──────────┘
1001df20 ┌──────────┐
│ rx│
100113e0 └──────────┘
1001133c ┌──────────┐
│ rwx│
100060fc │ │┌──────────┐
│ ││ rwx│
10000000 └──────────┘└──────────┘
No errors, so it looks like our ZAP is fine.
Dumping a Core
Now that we have a working ZAP, we can use it to dump a core.
Aerology exposes the dump subcommand to dump a core using the information
in a ZAP.
The usage f the dump command is:
aerology dump <ZAP_FILE> [CORE_FILE]
Similar to the pack subcommand, the output file argument is optional, and
it will infer a name based on the ZAP passed as the first argument.
Also similar to pack, there is very little output.
$ aerology dump dhcpv4_client.zap
Wrote core dump to "dhcpv4_client.core.0"
The core dump that Aerology wrote is self-contained, and can be transfered across machines with different archetectures and operating systems without losing any context or having an different behavior.
Config, Segments, Device Tree
Aerology stores the Zephyr Config and Device Tree in both ZAPs and cores. These files are accessable with a few subcommands.
Config, the config subcommand
The config subcommand dumps the Kconfig that was used for the build of
Zephyr in the ZAP or core.
For example:
$ aerology config dhcpv4_client.zap
--- Lots of config output ---
#
# Boot Options
#
# CONFIG_IS_BOOTLOADER is not set
# CONFIG_BOOTLOADER_MCUBOOT is not set
# CONFIG_BOOTLOADER_BOSSA is not set
# end of Boot Options
#
# Compatibility
#
CONFIG_COMPAT_INCLUDES=y
# end of Compatibility
Further, the config may be queried from a core dump as well. If the core dump and the ZAP have been built from the same build of Zephyr, their configs will be the same.
$ diff <(aerology config dhcpv4_client.zap) <(aerology config dhcpv4_client.core.0)
--- No Output ---
Device Tree, the dts subcommand
Similar to the config subcommand, the dts subcommand dumps the device
tree source used in the build.
For example:
$ aerology dts dhcpv4_client.core.0
--- Lots of DTS output ---
reserved-memory {
#address-cells = < 0x1 >;
#size-cells = < 0x1 >;
ranges;
code: memory@01060000 {
reg = < 0x1060000 0x60000 >;
};
ram: memory@21000000 {
reg = < 0x21000000 0x200000 >;
};
};
};
As with the config subcommand, the dts subcommand works on both ZAPs
and cores.
Memory Layout, the segments subcommand
The memory layout of a Zephyr (and TFM) application can be visually inspected
with the segments subcommand:
$ aerology segments --summary dhcpv4_client.core.0
Note: Not to scale.
Key: r = readable, w = writable, x = executable, z = zeroed on startup
┃ = overlapping section
zephyr tfm_s bl2
31007928┌──────────┐
│ rwz│
31000418└──────────┘
31000414┌──────────┐
│ rw│
31000000└──────────┘
3000ff70 ┌──────────┐
│ rwz│
3000bf38 ┢━━━━━━━━━━┪
┃ rw┃
3000bbc0 ┡━━━━━━━━━━┩
30005f40 │ │┌──────────┐
│ ││ rw│
30003ac0 ┢━━━━━━━━━━┪│ │
┃ rw┃│ │
30003a80 ┡━━━━━━━━━━┩│ │
30002020 ┢━━━━━━━━━━┪│ │
┃ rw┃│ │
30002000 ┡━━━━━━━━━━┩│ │
30000400 │ │├──────────┤
│ ││ rwz│
30000000 └──────────┘└──────────┘
11076384┌──────────┐
│ rwx│
11060000└──────────┘
1105f500 ┌──────────┐
│ rx│
1105f4c0 └──────────┘
11021280 ┌──────────┐
│ rx│
11009fe0 └──────────┘
11009f68 ┌──────────┐
│ rwx│
11000000 └──────────┘
100055e0 ┌──────────┐
│ rwx│
10000000 └──────────┘
The above shows the --summary output, because the unsummarized output is
quite long.
Without summary it looks like:
$ aerology segments dhcpv4_client.core.0
Note: Not to scale.
Key: r = readable, w = writable, x = executable, z = zeroed on startup
┃ = overlapping section
zephyr tfm_s bl2
31007928┌──────────────────────────────┐
│noinit rwz│
31002020├──────────────────────────────┤
│bss rwz│
31000418└──────────────────────────────┘
31000414┌──────────────────────────────┐
│net_l2_area r│
31000404├──────────────────────────────┤
│net_if_dev_area rw│
310003e8├──────────────────────────────┤
│net_if_area rw│
310003a8└──────────────────────────────┘
310003a4┌──────────────────────────────┐
│_net_buf_pool_area rw│
3100034c├──────────────────────────────┤
│k_sem_area rw│
310002ec├──────────────────────────────┤
│k_msgq_area rw│
310002bc├──────────────────────────────┤
│k_mutex_area rw│
3100026c├──────────────────────────────┤
│k_mem_slab_area rw│
31000234├──────────────────────────────┤
│log_dynamic_sections rw│
310001d0├──────────────────────────────┤
│device_states rw│
31000188└──────────────────────────────┘
31000187┌──────────────────────────────┐
│datas rw│
31000000└──────────────────────────────┘
3000ff70 ┌──────────────────────────────┐
│.TFM_BSS rwz│
3000bf40 └──────────────────────────────┘
3000bf38 ┌──────────────────────────────┐
│.TFM_DATA rw│
3000bbc0 ├──────────────────────────────┤
│.TFM_PSA_ROT_LINKER_BSS rwz│
30005f40 │ │┌──────────────────────────────┐
│ ││.heap rwz│
30004f40 │ │├──────────────────────────────┤
│ ││.msp_stack rwz│
30003ac0 ├──────────────────────────────┤│ │
│.TFM_PSA_ROT_LINKER_DATA rw││ │
30003a80 ├──────────────────────────────┤│ │
│.TFM_APP_ROT_LINKER_BSS rwz││ │
30003740 │ │└──────────────────────────────┘
30003728 │ │┌──────────────────────────────┐
│ ││.bss rwz│
30002020 ├──────────────────────────────┤│ │
│.TFM_APP_ROT_LINKER_DATA rw││ │
30002000 ├──────────────────────────────┤│ │
│.heap rwz││ │
30001000 ├──────────────────────────────┤│ │
│.psp_stack rwz││ │
30000800 ├──────────────────────────────┤│ │
│.msp_stack rwz││ │
30000494 │ │├──────────────────────────────┤
│ ││.data rw│
30000400 ├──────────────────────────────┤├──────────────────────────────┤
│.tfm_bl2_shared_data rwz││.tfm_bl2_shared_data rwz│
30000000 └──────────────────────────────┘└──────────────────────────────┘
11076384┌──────────────────────────────┐
│rodata r│
11071bfc├──────────────────────────────┤
│device_handles r│
11071b90├──────────────────────────────┤
│shell_root_cmds_sections r│
11071b38├──────────────────────────────┤
│shell_area r│
11071b08├──────────────────────────────┤
│log_backends_sections r│
11071af8├──────────────────────────────┤
│log_const_sections r│
11071a30├──────────────────────────────┤
│sw_isr_table rw│
11071630├──────────────────────────────┤
│devices r│
11071480├──────────────────────────────┤
│initlevel r│
110713a0├──────────────────────────────┤
│.ARM.exidx r│
11071398├──────────────────────────────┤
│text rx│
11060640├──────────────────────────────┤
│rom_start rwx│
11060000└──────────────────────────────┘
1105f500 ┌──────────────────────────────┐
│.gnu.sgstubs rx│
1105f4c0 └──────────────────────────────┘
11021280 ┌──────────────────────────────┐
│.ARM.exidx r│
11021278 ├──────────────────────────────┤
│.psa_interface_thread_call rx│
11021180 ├──────────────────────────────┤
│.TFM_UNPRIV_CODE rx│
11009fe0 └──────────────────────────────┘
11009f68 ┌──────────────────────────────┐
│.ER_TFM_CODE rx│
110056c0 ├──────────────────────────────┤
│.TFM_APP_ROT_LINKER rx│
11004940 ├──────────────────────────────┤
│.TFM_PSA_ROT_LINKER rx│
11000f40 └──────────────────────────────┘
11000f24 ┌──────────────────────────────┐
│.TFM_SP_LOAD_LIST r│
11000d34 ├──────────────────────────────┤
│.zero.table rw│
11000d1c ├──────────────────────────────┤
│.copy.table rw│
11000cf8 ├──────────────────────────────┤
│.TFM_VECTORS rx│
11000400 └──────────────────────────────┘
100055e0 ┌──────────────────────────────┐
│.zero.table rw│
100055d0 ├──────────────────────────────┤
│.copy.table rw│
100055b8 ├──────────────────────────────┤
│.ARM.exidx r│
100055b0 ├──────────────────────────────┤
│.text rx│
10000000 └──────────────────────────────┘
Stacks
Aerology currently supports 2 subcommands that help with looking through
stacks: stacks for a summary of the filled-level of all thread stacks
Aerology can find, and backtrace for backtracing all stacks in the system.
stacks subcommand
The stacks subcommand renders a pretty version of stack usage in a table format:
$ aerology stacks dhcpv4_client.core.0
Key: █: currently in use ▒: used in the past ░: never used
name used max size
zephyr::logging 32b 32b 768b ░░░
zephyr::shell_uart 32b 32b 2048b ░░░░░░░░
zephyr::idle 00 32b 32b 320b ░
zephyr::main 88b 216b 1024b ░░░░
tfm_s::3000bf40 312b 2440b 8192b █▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░░░░░░░░░░░░░░
tfm_s::3000bf88 192b 192b 2688b ░░░░░░░░░░░
tfm_s::3000bfd0 168b 776b 2048b ▒▒▒░░░░░
tfm_s::3000c018 176b 644b 1664b ▒▒░░░░░
tfm_s::3000c060 120b 256b 1280b ▒░░░░
tfm_s::3000c0a8 72b 300b 1024b ▒░░░
tfm_s::3000c0f0 72b 20b 256b ░
This might be helpful for determining the maximum stack usage of a given thread, for example.
backtrace subcommand
The backtrace subcommand is where Aerology really shines; it backtraces all threads in the system including stacks of TFM partitions:
$ aerology backtrace dhcpv4_client.core.0
Registers
├─ 11008a60 in tfm_access_violation_handler
╞═ Exception Handler Called
├─ 01069bfc in smsc_init
├─ 01069ce3 in eth_init
├─ 01069fe5 in z_sys_init_run_level
├─ 0106a1af in bg_thread_main
├─ 0106bae1 in z_thread_entry
└─ 010645e3 in arch_switch_to_main_thread
Thread zephyr::idle 00
├─ 0106bad4 in z_thread_entry
└─ aaaaaaaa in <unknown>
Thread zephyr::logging
├─ 0106bad4 in z_thread_entry
└─ aaaaaaaa in <unknown>
Thread zephyr::main
├─ 010644f8 in arch_swap
├─ 0106b8dd in k_sys_work_q_init
├─ 01069fe5 in z_sys_init_run_level
├─ 0106a1af in bg_thread_main
├─ 0106bae1 in z_thread_entry
└─ 010645e3 in arch_switch_to_main_thread
Thread zephyr::shell_uart
├─ 0106bad4 in z_thread_entry
└─ aaaaaaaa in <unknown>
Thread tfm_s::sp_crypto
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
Thread tfm_s::sp_initial_attestation
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
Thread tfm_s::sp_ps
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
Thread tfm_s::sp_its
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
Thread tfm_s::sp_platform
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
Thread tfm_s::sp_ns_agent
└─ 11008c85 in tfm_nspm_thread_entry
Thread tfm_s::idle
└─ 11008ca5 in tfm_idle_thread
Optionally, you can have Aerology dump the registers at each stack frame with
the --regs.
The following in an excert from the same backtrace with -r applied:
Thread zephyr::main
├─ 010644f8 in arch_swap
│ R0 00000000 R1 01071470 R2 01071618 R3 01069ce3
│ R4 00000000 R5 00000000 R6 01071480 R7 0106a1a1
│ R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
│ R12 00000000 Sp 210037c8 Lr 0106b8dd Pc 010644f8
│ Psr 61000000
├─ 0106b8dd in k_sys_work_q_init
│ R0 00000000 R1 01071470 R2 01071618 R3 01069ce3
│ R4 01071470 R5 00000000 R6 01071480 R7 0106a1a1
│ R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
│ R12 00000000 Sp 210037e0 Lr 01069fe5 Pc 0106b8dd
│ Psr 61000000
├─ 01069fe5 in z_sys_init_run_level
│ R0 00000000 R1 01071470 R2 01071618 R3 01069ce3
│ R4 0106a1a1 R5 21000e98 R6 21003800 R7 0106a1a1
│ R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
│ R12 00000000 Sp 210037f0 Lr 0106a1af Pc 01069fe5
│ Psr 61000000
├─ 0106a1af in bg_thread_main
│ R0 00000000 R1 01071470 R2 01071618 R3 00000000
│ R4 0106a1a1 R5 21000e98 R6 21003800 R7 0106a1a1
│ R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
│ R12 00000000 Sp 210037f8 Lr 0106bae1 Pc 0106a1af
│ Psr 61000000
├─ 0106bae1 in z_thread_entry
│ R0 00000000 R1 01071470 R2 01071618 R3 00000000
│ R4 0106a1a1 R5 21000e98 R6 21003800 R7 0106a1a1
│ R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
│ R12 00000000 Sp 21003800 Lr 010645e3 Pc 0106bae1
│ Psr 61000000
└─ 010645e3 in arch_switch_to_main_thread
R0 00000000 R1 01071470 R2 01071618 R3 00000000
R4 0106a1a1 R5 21000e98 R6 21003800 R7 0106a1a1
R8 01064cd8 R9 01064cd8 R10 01064cd8 R11 01064cd8
R12 00000000 Sp 21003800 Lr 010645e3 Pc 010645e3
Psr 61000000
Since each stack frame takes up 6 lines instead of 1, all of the backtraces together becomes quite long.
Queries
Aerology's query subcommand is easily it's most general purpse tool.
It provides a means to inspect arbitrary objects (e.g. structs, unions, etc.)
in a C-style fromat or a hexdump.
Syntax
The syntax of a query is inspired by jq, and many of the operators are postfix.
An Aerology query is structured as a global, with optional postfix operators,
followed by zero or more => separated "filters".
Global
Unlike jq, the initial value is provided by global optionally prefixed with a
executable namespace.
For example, to get the zephyr kernel struct, either of the following queries
produce the same result:
_kernel
zephyr::kernel
Executing a query with just the global prints the whole structure C-style:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel
21001ec0: (struct z_kernel) {
.cpus = {
(struct _cpu) {
.nested = (uint32_t) 0 /*0x0*/,
.irq_stack = (char *) 553664832 /*0x21004140*/,
.current = (struct k_thread *) 553651864 /*0x21000e98*/,
.idle_thread = (struct k_thread *) 553651680 /*0x21000de0*/,
.slice_ticks = (int) 0,
.id = (uint8_t) 0 /*0x0*/,
/* .arch is Missing */
},
},
.ready_q = (struct _ready_q) {
.cache = (struct k_thread *) 553651864 /*0x21000e98*/,
.runq = (sys_dlist_t) {
.head = (struct _dnode *) 553651864 /*0x21000e98*/,
.next = (struct _dnode *) 553651864 /*0x21000e98*/,
.tail = (struct _dnode *) 553650440 /*0x21000908*/,
.prev = (struct _dnode *) 553650440 /*0x21000908*/,
},
},
.threads = (struct k_thread *) 553652056 /*0x21000f58*/,
}
Postfix Operators
Once a global is selected, you may follow members with postfix operators. All postfix operators expect a type class (struct, array, pointer, etc.), and will error when the type at that step of the query is of a different class.
The operators are:
.<struct-member>to traverse a struct into a member named "struct-member". This will derefrence pointers to structures before atempting to move to a member if the current type is a pointer to a structure. This will error when the struct does not contain the required member.[<number>]will select array member with index "number". Similar to the struct member operation, this will dereference pointers to arrays before atempting to select the member. This will error if the index is out of bounds, or the array has no size.[]will select all array members, treating the remainder of the query as a query over all members of the array. This will error if the array has no size..*postfix derefrence operator. This will derefrence a pointer.
Struct Member
Continuing with the Zephyr kernel structure, we may select the head of the threads linked list as follows:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.threads
21001ee4: (struct k_thread *) 553652056 /*0x21000f58*/
If we were to have a typo, we would get an error such as:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.thread
Error:
× member not found "thread"
┌─[command-line:1:1]
1 │ zephyr::_kernel.thread
· ───┬──
· └── missing
└────
help: consider replacing with "cpus", "ready_q" or "threads" instead
If we try to select a member of something that's not a struct, such as the cpu array, we get a type error such as:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.cpus.nested
Error:
× type mismatch
┌─[command-line:1:1]
1 │ zephyr::_kernel.cpus.nested
· ───┬──
· └── expected struct or union found struct _cpu[1]
└────
Members may be selected through poiters to structures, without an intevening
dereference operator.
For exmple, if we wanted to select the arch member of the thread struct at
the head of the threads linked list, we may use the following query:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.threads
21001ee4: (struct k_thread *) 553652056 /*0x21000f58*/
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.threads.arch
21001004: (struct _thread_arch) {
.basepri = (uint32_t) 0 /*0x0*/,
.swap_return_value = (uint32_t) 4294967285 /*0xfffffff5*/,
.mode = (uint32_t) 48128 /*0xbc00*/,
.mode_bits = (uint8_t) 0 /*0x0*/,
.mode_exc_return = (uint8_t) 188 /*0xbc*/,
.mode_reserved2 = (uint16_t) 0 /*0x0*/,
}
Note that above the type of zephyr::_kernel.threads is a struct k_thread *,
but we used it as if it were a struct k_thread.
Array Index
Selecting an index in an array works similar to selecting a struct member. For example if we wanted to select the first entry (index 0) of the Zephyr kernel's cpu array we may:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.cpus[0]
21001ec0: (struct _cpu) {
.nested = (uint32_t) 0 /*0x0*/,
.irq_stack = (char *) 553664832 /*0x21004140*/,
.current = (struct k_thread *) 553651864 /*0x21000e98*/,
.idle_thread = (struct k_thread *) 553651680 /*0x21000de0*/,
.slice_ticks = (int) 0,
.id = (uint8_t) 0 /*0x0*/,
/* .arch is Missing */
}
However, if we index past the end of the array, we get an error:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.cpus[11110]
Error:
× Array index out of bounds
┌─[command-line:1:1]
1 │ zephyr::_kernel.cpus[11110]
· ───────
└────
help: This array has a size of 1
Further if we try to index something other than an array, we get a type error such as:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel[6]
Error:
× type mismatch
┌─[command-line:1:1]
1 │ zephyr::_kernel[6]
· ─┬─
· └── expected array found struct z_kernel
└────
Whole Array
You may select a whole array by omitting an array index. For example, we may print the value of the char array that makes up a thread's name with the query:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.threads.name[]
21000fd0: (char) 115 /*0x73*/
21000fd1: (char) 121 /*0x79*/
21000fd2: (char) 115 /*0x73*/
21000fd3: (char) 119 /*0x77*/
21000fd4: (char) 111 /*0x6f*/
21000fd5: (char) 114 /*0x72*/
21000fd6: (char) 107 /*0x6b*/
21000fd7: (char) 113 /*0x71*/
21000fd8: (char) 0 /*0x0*/
21000fd9: (char) 0 /*0x0*/
21000fda: (char) 0 /*0x0*/
21000fdb: (char) 0 /*0x0*/
21000fdc: (char) 0 /*0x0*/
21000fdd: (char) 0 /*0x0*/
21000fde: (char) 0 /*0x0*/
21000fdf: (char) 0 /*0x0*/
21000fe0: (char) 0 /*0x0*/
21000fe1: (char) 0 /*0x0*/
21000fe2: (char) 0 /*0x0*/
21000fe3: (char) 0 /*0x0*/
21000fe4: (char) 0 /*0x0*/
21000fe5: (char) 0 /*0x0*/
21000fe6: (char) 0 /*0x0*/
21000fe7: (char) 0 /*0x0*/
21000fe8: (char) 0 /*0x0*/
21000fe9: (char) 0 /*0x0*/
21000fea: (char) 0 /*0x0*/
21000feb: (char) 0 /*0x0*/
21000fec: (char) 0 /*0x0*/
21000fed: (char) 0 /*0x0*/
21000fee: (char) 0 /*0x0*/
21000fef: (char) 0 /*0x0*/
This query is a tad silly, as aerology is capable of printing strings directly:
$ aerology query dhcpv4_client.core.0 zephyr::_kernel.threads.name
21000fd0: (char[32]) "sysworkq"
Filters
At the moment, 3 filters are supported:
- Postfix operators.
- Reading a linked list with
llnodes - Creating a bactrace with
bt
Postfix operators
Postfix operators may be used as a filter by themselves.
Using an example from the prior section, we may query the name of the head of
the thread list including the filter separator => in a few more places.
$ aerology query dhcpv4_client.core.0 'zephyr::_kernel => .threads.name'
21000fd0: (char[32]) "sysworkq"
$ aerology query dhcpv4_client.core.0 'zephyr::_kernel => .threads => .name'
21000fd0: (char[32]) "sysworkq"
$ aerology query dhcpv4_client.core.0 'zephyr::_kernel.threads => .name'
o21000fd0: (char[32]) "sysworkq"
As may be seen in the exmaple above, these produce the same result.
Linked List Nodes
An important filter in Aerology is the linked list nodes reader.
It's invoked llnodes <postfix-member>.
For example, to get the names of all of the threads in the system, we may
use the following query:
$ aerology query dhcpv4_client.core.0 'zephyr::_kernel.threads => llnodes .next_thread => .name'
21000e58: (char[32]) "idle 00"
210008c8: (char[32]) "logging"
21000f10: (char[32]) "main"
21000980: (char[32]) "shell_uart"
21000fd0: (char[32]) "sysworkq"
Backtrace
Another interesting Aerology filter is the backtrace filter. This filter has a more complex syntax:
bt <reg-name>=<postfix-member>
This filter backtraces after populating the initial register state as described by its arguments.
For example, we can take a backtrace of every thread in TFM with the following query:
$ aerology query dhcpv4_client.core.0 'tfm_s::partition_listhead => llnodes .next => bt pc=.ctx_ctrl.exc_ret psp_s=.ctx_ctrl.sp'
3000bfcc:
╞═ Exception Handler Called
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
3000c014:
╞═ Exception Handler Called
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
3000c05c:
╞═ Exception Handler Called
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
3000c0a4:
╞═ Exception Handler Called
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
3000c0ec:
╞═ Exception Handler Called
├─ 11008e70 in tfm_arch_trigger_pendsv
├─ 11007ee3 in spm_interface_thread_dispatcher
└─ 11021187 in psa_interface_unified_abi
3000c134:
╞═ Exception Handler Called
└─ 11008c85 in tfm_nspm_thread_entry
3000c6e4:
╞═ Exception Handler Called
└─ 11008ca5 in tfm_idle_thread
Note: Zephyr does not store the entire exception payload, so without bitwise operations, we cannot construct a valid value for the pc.
Note: we place the exception payload in pc, not the lr.