From f6c1f67cb9c38f4ddfe292e8d95ff1d61972616c Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Mon, 2 Nov 2026 08:08:32 +0100 Subject: [PATCH 1/6] docs(ctrace): record instruction trace implementation plan --- tools/ctrace/docs/instruction-trace-design.md | 606 ++++++++++++++++++ 1 file changed, 606 insertions(+) create mode 100644 tools/ctrace/docs/instruction-trace-design.md diff --git a/tools/ctrace/docs/instruction-trace-design.md b/tools/ctrace/docs/instruction-trace-design.md new file mode 100644 index 000000000..f12c7668e --- /dev/null +++ b/tools/ctrace/docs/instruction-trace-design.md @@ -0,0 +1,606 @@ +# Instruction Trace Implementation Plan + +Status: minimal implementation agreed as the starting scope; broader design remains a working draft. +The proposed CSV extension is local to this plan and is not yet part of the published CMSIS contract. +Research date: 2026-10-28. Baseline: devtools `17f8f54a`, branch `feat/ctrace-instruction-trace`. +OpenCSD baseline: **v1.8.3**, commit `59f15320637ce6e510604e7bed550d7ad109d15d`. + +This document separates verified library behavior, ctrace implementation gaps, and proposed decisions. +It describes offline decoding of captured files. Target configuration and capture remain producer responsibilities. +The separate ITM-channel-0 change is not part of this branch. + +## 1. Findings and proposed direction + +Instruction trace requires more than enabling another OpenCSD decoder for a `.raw` file: + +1. Identify the protocol and capture-time decoder configuration for every trace source. +2. Supply the actual executable code bytes at their execution addresses. +3. Consume OpenCSD generic instruction ranges, context, exceptions, time information, and discontinuities. +4. Define a semantic event model and an output contract that can retain this information. + +**OpenCSD already contains the instruction decoder needed for program-flow reconstruction.** A separate instruction +codec is not required for that task. An ELF/AXF reader and an address-aware image provider are needed if executable +files are the image source; raw binary images with explicit mappings are another valid input. Assembly text, +symbols, source lines, call graphs, and coverage are separate processing layers. [O2][O3][O4][O5] + +Start with the minimal implementation in section 1.1. The reference capture and its exact protocol still need to +be selected; a verified Cortex-M ETMv4 instruction-only capture from M7 or M33 is the preferred candidate. +Mixed ITM/ETM routing, additional protocols, ELF/AXF support and richer outputs belong to subsequent milestones. +Treat MTB as a separate decoder project. Qualify Armv8.1-M support by feature and test evidence rather than +advertising all M52/M55/M85 captures as supported. + +### 1.1 Minimal implementation — agreed starting point + +The first deliverable decodes **one recorded instruction-trace input, one known ETM protocol and one CPU** to CSV. +Use one matching static binary image at an explicit execution address. Emit one row per decoded instruction range, +with observable gaps and missing-code conditions. This is the starting scope agreed on 2026-10-28. + +```text +RAW -> existing framing -> OpenCSD ETM full decoder -> instruction-range event -> CSV + ^ + matching binary code bytes +``` + +Reuse the existing input convention, for example `Board.TB.raw` with `Board.ctrace-run.yml` in a trace directory. +A freely named `trace.raw` is not currently a standalone CLI input. Adding direct-file CLI support is a later option. +The buffer must already contain valid bytes in chronological order with known framing. [C8] + +The necessary inputs are the actual decoder configuration, architecture/profile, source Trace ID where formatted, +and the matching `.bin` with its execution mapping. A RAW file alone does not establish these facts. Agree the +minimal metadata fields against the first capture; do not infer missing configuration or code bytes. + +Implement these five connected extensions: + +1. **ETM configuration and route:** extend `TraceRunConfig`, `YmlTraceRunConfigReader` and `CtraceRunMeta` to retain + one ETM source, its decoder configuration and the binary-image mapping. ETM metadata is currently ignored. [C2][C4] +2. **Code-memory access:** attach a binary file/region accessor to the OpenCSD tree, supplying the actual bytes at + execution addresses. One static mapping is sufficient for this milestone. An ELF/AXF reader is not required. [O4] +3. **ETM decoder path:** create the selected ETM full decoder and connect it through `DecodePipeline` and + `FileDecodeJob`. Reuse bounded RAW reads, compatible deformatting and the existing tree lifecycle; keep ITM-specific + packet monitoring, recovery and timestamp accumulation out of the ETM path. [C3][C9] +4. **Instruction-element adapter:** consume `INSTR_RANGE` and the state needed to interpret it. The current collector + accepts only `ITMTRACE`. Preserve range/count/execution flags and expose synchronization loss, missing code and + unsupported context transitions; retain valid partial output. [C3][O3] +5. **Events, selection and CSV:** add an instruction-range payload to `TraceEvent`, a selectable event type and its + `CsvRowMapper` mapping. Shared event visitors, including CTF, must handle the added variant; instruction CTF + requests must report unsupported output until that backend is implemented. [C5][C6][C10] + +The minimal CSV extension retains the existing eight columns and introduces `type=instr_range`: + +```csv +cycles,stream,type,index,value,pc,address,note +,18,instr_range,,,0x08000100,0x08000108,"count=3; isa=T32; last_executed=1" +``` + +- `stream`: source Trace ID for this formatted example; preserve existing unformatted stream conventions. +- `pc`: range start; `address`: exclusive range end. +- `note`: instruction count, ISA and last-instruction execution flag; retain additional validity/flow flags as needed. +- `index` and `value`: empty for this record type. +- `cycles`: empty without qualified cycle information. Row order preserves decoded order; no synthetic time is added. + +This is the agreed initial mapping for implementation, not an existing specification claim. Document its type-specific +semantics and diagnostic mapping before exposing it. Existing ITM/DWT row meanings remain unchanged. +The count describes represented instructions; execution flags still govern whether the last instruction executed. + +Completion evidence is an actual capture with complete configuration, matching bytes and independently expected +ranges; correct handling of missing code and synchronization gaps; chunk-boundary-independent results; and unchanged +existing ITM/DWT CSV behavior. The first capture must fit the declared static image/context scope; report unsupported +transitions instead of silently claiming a complete path. + +Subsequent work includes ELF/AXF parsing, assembly text, symbols/DWARF, full context output, additional protocols, +mixed sources/multiple CPUs, time correlation, CTF, coverage and call-graph analysis. OpenCSD's existing instruction +decoder supplies flow decoding for the first milestone; no additional instruction-codec dependency is needed. + +## 2. OpenCSD support and its limits + +### 2.1 Trace protocols + +The pinned README and decoder implementations establish the following support. A protocol being implemented +does not establish correct decoding of every processor, trace option, or instruction extension. [O1][O6] + +| Trace source | OpenCSD v1.8.3 | Consequence for ctrace | +| --- | --- | --- | +| ETMv3 instructions | Full decode, through v3.5 | M3/M4 candidate; qualify M23. | +| ETMv3 data | Packet processing only | Full decoder rejects data and mixed instruction/data modes. | +| ETMv4 instructions | Full decode; v4.6 A/R, v4.4 M advertised | Main candidate; qualify actual features. | +| ETMv4 data | Packet processing/decode are future work | Outside initial scope. | +| PTM v1.1 | Full instruction decode | Future A-profile extension. | +| ETE v1.3 | Full instruction decode | Future processor-trace extension. | +| ITM v1.0, STM v1.1 | Full software-trace decode | Not an executed instruction path. | +| MTB | No built-in decoder | Separate record decoder and input contract. | +| Earlier ETMv1/v2 | No built-in decoder registered | Generic ETM naming does not establish support. | + +OpenCSD supports both packet-processing-only and full-decode configurations. Packet inspection can work without +code images; the desired reconstructed instruction path generally cannot. All decoder families are present in +the OpenCSD source dependency; ctrace currently instantiates only ITM. [O2][O6] + +ETMv3 full decode explicitly accepts only instruction-only mode. A capture configured for combined instruction/data +trace is not handled by simply ignoring its data packets. [O11] + +### 2.2 Cortex-M architecture and trace-source mapping + +The architecture generation and the trace-protocol generation are different axes. The physical device must +actually implement the optional trace source and a usable capture path. A processor name alone is insufficient +decoder configuration. The following matrix is a planning guide, not a device support declaration. [A1][A2][A3] + +| Architecture / processor examples | Relevant instruction-trace source | Assessment for this OpenCSD baseline | +| --- | --- | --- | +| Armv6-M: M0, M1 | No ETM path established here | ISA alone does not establish trace support. | +| Armv6-M: M0+ | Optional MTB [A10] | No built-in MTB decoder. | +| v7-M / v7E-M: M3, M4 | Optional ETMv3 | Protocol implemented; qualify captures/exceptions. | +| Armv7E-M: M7 | ETM-M7, ETMv4 | Upstream M7 fixture exists. | +| v8-M Baseline: M23 | ETM-M23: **ETMv3.5**; or MTB | OpenCSD compatibility risk; see below. | +| v8-M Mainline: M33 | ETM-M33 r0p2: **ETMv4.2** | Within advertised scope; M33-profile fixture exists. | +| v8-M Mainline: M35P | Optional ETMv4.2 / MTB | Protocol within advertised scope; qualify captures. | +| Armv8.1-M: M55, M85 | ETM-M55/M85 r1p1: **ETMv4.5** | Partial evidence; not complete qualification. | +| Armv8.1-M: M52 | ETM-M52 r0p2: **ETMv4.5** | Same Armv8.1-M qualification gate. | + +The M7 mapping is confirmed by ETM-M7 TRM DDI0494D, PDF p.7, and by an upstream snapshot. +ETM-M23 TRM DDI0563C confirms ETMv3.5 on PDF p.11 and p.20. ETM-M55 TRM +101053_0101_03_en and ETM-M85 TRM 101926_0101_07_en identify ETMv4.5 on PDF p.14. +ETM-M33 TRM 100232_0002_06_en identifies ETMv4.2 on PDF p.7. Its OpenCSD snapshot labels the device `ETM4.4`; +that test metadata is not evidence that ETM-M33 hardware implements v4.4. +Arm's M35P introduction identifies ETMv4.2 on PDF p.15; ETM-M52 TRM 102774_0002_05_en identifies ETMv4.5 +on PDF p.14. [A2][A3][A4][A5][A7][A8][A9][O7] + +Two specific OpenCSD gaps affect the design: + +- **M23:** `EtmV3Config::isV7MArch()` requires exactly `ARCH_V7` and `profile_CortexM`. The ETMv3 packet processor + uses this check in M-profile exception handling, whereas the processor-name map associates M23 with `ARCH_V8`. + This is a concrete compatibility risk, not proof that all M23 trace fails. Require M23 exception/security fixtures + and an upstream assessment. Do not misdeclare M23 as Armv7 to bypass the check. [O8][O12][O13] +- **Armv8.1-M:** the README explicitly leaves M-class branch-forward instruction support as future work. + There is already an `armv8_1m_branches` M55 snapshot and Thumb code for low-overhead loops, so support is not + simply absent. Equally, that fixture does not establish complete Armv8.1-M support. A separate disassembler + cannot automatically repair missing OpenCSD path-reconstruction semantics. [O1][O7][O9] + +## 3. Required input: capture, configuration, and code + +### 3.1 Trace-buffer bytes + +Existing discovery already accepts `.TB.raw` and `.TB_.raw`. +The extension `.raw` and the channel name identify a captured input, not its protocol. A buffer can contain +formatted ETM, mixed ETM/ITM, an unformatted single source, or MTB records. [C1] + +For formatter-based captures, the producer must export the valid bytes in chronological order. A raw memory dump +of a circular buffer is insufficient if its write pointer, wrap state, valid length, or alignment is unknown. +The producer must either normalize the buffer or provide enough metadata to do so before deformatting. +Document padding and flush behavior; do not decode unwritten capacity as trace. + +The current ctrace transport accepts 16-byte memory-aligned CoreSight frames, without FSYNC/HSYNC framing. +That is reusable for compatible ETB/ETF/ETR-style exports. Other framing modes require an explicit contract. +The current `TB => formatted` default must not cause an MTB byte array to enter the CoreSight deformatter. [C1][O2] + +### 3.2 Source configuration + +Each source needs a stable source-instance identity, processor association, protocol/revision, architecture/profile, +and an architectural ATB Trace ID when formatted. Unformatted input still needs an explicit source/protocol binding. +Trace ID identifies the source within an input; it is neither an ITM stimulus channel nor a universal CPU number. + +The public OpenCSD configuration structures require the following fields. These names describe software API +inputs, not proposed new YAML keys or instructions to program target registers. [O10] + +| Decoder family | OpenCSD configuration inputs | +| --- | --- | +| ETMv3 | `reg_idr`, `reg_ctrl`, `reg_ccer`, `reg_trc_id`, `arch_ver`, `core_prof` | +| ETMv4I | IDR0/1/2/8–13, CONFIGR, TRACEIDR, architecture/profile | +| ETE, future scope | IDR0/1/2/8, DEVARCH, CONFIGR, TRACEIDR, architecture/profile | + +The latter two rows abbreviate register-backed API fields with their register names; see the complete structures +for their `reg_` field names. ETMv3 and ETE have separate configuration structures. [O14][O15] + +The producer must distinguish unavailable fields from valid zero values. Determine required/unused fields for the +selected decoder configuration explicitly. A list of masked register writes is not necessarily a complete decoder +snapshot: identification registers and the resulting active configuration may be absent. + +The current CMSIS `instructions:` intent and `ctrace-refs` do not by themselves supply this complete contract. +ctrace does not currently retain ETM references, their register state, or code-image descriptions. +Agree ownership with the debugger/pyTS producer before inventing a ctrace-private schema. [C2][A6] + +### 3.3 Executable images + +Full instruction decode combines compressed trace with code bytes. Direct branches and sequential instructions +are reconstructed from those bytes; the trace is not a recording of every instruction opcode. [O2][O3] + +Every image mapping must identify: + +- File/snapshot identity and preferably a capture-time content hash or build identity. +- Processor/source association and Secure/Non-secure address space where applicable. +- Execution address range, file offset and available file-backed length. +- Any explicit relocation, RAM execution mapping, ROM alias, or time-dependent overlay. + +The minimal milestone uses one static binary mapping and a capture within that supported execution context. +For broader captures, include bootloader, application, Secure/Non-secure firmware, and required ROM code when crossed. +A valid ELF with the wrong build can produce plausible but incorrect reconstruction. Source symbols alone do not +provide the code bytes. Modified RAM code, software breakpoint patches, or dynamically generated code require +matching capture-time bytes or a documented limitation. + +## 4. ELF/AXF reader and instruction decoder responsibilities + +### 4.1 What ctrace needs + +The first milestone uses raw `.bin` plus an explicit execution address; OpenCSD can access it without an ELF parser. +For a later ELF/AXF milestone, ctrace needs a **read-only image reader**, not a target programmer or dynamic linker. +Detect the format from its contents. The proposed first ELF support is validated ELF32 Arm `ET_EXEC` images and +file-backed load regions at execution VMAs. A stripped executable can be sufficient; symbols and DWARF are optional. +The ELF-specific rules below apply to that later milestone. [E1][E2] + +Use program-header file offsets and `p_filesz` to expose actual bytes, mapped to the execution address `p_vaddr`. +Do not use flash load address / `p_paddr` as a universal substitute for where the processor executes RAM code. +Validate toolchain-specific AXF/scatter-loading behavior with fixtures; any section-based fallback must be specified. +Do not synthesize missing instruction bytes from `p_memsz`, BSS, or gaps. Reject ambiguous overlaps, unsupported +relocations, and address arithmetic outside the supported domain. An initial implementation can reject `ET_REL` +and dynamically relocated images until their mapping contract is defined. + +No reusable ELF/DWARF reader was found in the current ctrace/devtools C++ path. The implementation decision remains +open between a bounded reader and a portable library. ELFIO is a candidate to evaluate: it is header-only C++ with +an MIT license. Review a pinned version, validation behavior, footprint, and all release platforms before adoption. +A full LLVM dependency is not justified solely by the need to extract ELF code regions. [E3] + +#### 4.1.1 ELFIO licensing and distribution + +ELFIO uses the MIT license. It can be included in an Apache-2.0 project; the Apache Software Foundation explicitly +lists MIT as a Category A license accepted in its products. Our ctrace code and wrapper can remain Apache-2.0; +retain the MIT license and attribution for the ELFIO portions. [L1][L2] + +When integrating ELFIO: + +- Preserve upstream copyright and license notices in the imported headers. +- Include the complete upstream MIT license text, including copyright and warranty disclaimer, in distributed + source packages and binary release packages containing ELFIO code. Use the license from the pinned version. +- Include those notices with binaries even though ELFIO is header-only and has no separately linked library file. + A website link alone is not a substitute for shipping the notices. [L1] + +MIT does not require publication of our source code or modifications, and does not require a file specifically +named `NOTICE`. A practical integration is an ELFIO entry in the third-party component overview plus its original +license in the package's license directory, for example `licenses/ELFIO.txt`. The filename is a packaging choice; +the required notice content must reach recipients. Existing Apache-2.0 license and NOTICE obligations continue +to apply independently. [L1][L3] + +Before distributing a build using ELFIO, verify that the source archive and every platform's binary package contain +its license and attribution. This is a condition of the later ELFIO integration; it adds no dependency to the minimal +binary-image milestone. + +### 4.2 Connecting images to OpenCSD + +OpenCSD offers buffers, binary files, file regions, and callback accessors. File-region access maps a file offset +to an execution start address and length; OpenCSD does not interpret the ELF container for the client. [O2][O4] + +For the minimal single-CPU/static-image milestone, a binary file/region accessor is sufficient. +For the later multi-source design, prefer an image provider behind the Trace-ID-aware callback: + +```text +DecodeTree::addCallbackIDMemAcc(...) + -> read(address, memory_space, trace_id, requested_length) + -> resolve input/source -> processor -> matching image region + -> return available code bytes and actual byte count +``` + +The equivalent C API is `ocsd_dt_add_callback_trcid_mem_acc`. Return a short read or zero for inaccessible bytes; +do not supply fabricated opcodes. Keep backing storage alive for the decode session. + +The built-in mapper is global: it does not select images by CPU/Trace ID and rejects conflicting mappings in the +same memory space. Distinct cores can execute different images at identical VMAs. Therefore a single collection +of overlapping global file accessors is insufficient; ctrace must resolve source/security-aware mappings itself. +Track context changes for future context-dependent mappings. The callback alone does not provide a full process +or overlay history. [O2][O4] + +OpenCSD's memory cache keys reads by Trace ID and address range, which separates static images on different cores. +Memory space, context ID and image generation are not cache keys; a cache hit bypasses the image callback. +ETMv4 invalidates on an updated context packet, but equivalent behavior must not be assumed for every protocol. +Dynamic mappings or code changes at the same source/address need explicit invalidation or disabled caching. +Initially reject unsupported time-dependent mappings; test Secure/Non-secure aliases and context transitions. +The APIs include `InvalidateMemAccCache(traceID)` and `DecodeTree::setMemAccCacheing(...)`. [O4][O19][O23][O24][O25] + +### 4.3 Is another instruction codec needed? + +OpenCSD attaches its built-in `TrcIDecode` to full decoders. Its `IInstrDecode` contract supplies the instruction +length and flow-relevant properties, including branch classification, direct target, and conditional/link/return +information. It supports A32, T32, and A64 internally; the initial Cortex-M scope is T32/Thumb. [O5] + +Keep three responsibilities distinct: + +| Responsibility | Required initially? | Provider | +| --- | --- | --- | +| Read executable bytes at execution addresses | Yes for full path reconstruction | New ctrace image reader/provider | +| Decode flow-relevant instruction properties | Yes | Existing OpenCSD instruction decoder | +| Format assembly text | Optional | Later, separately evaluated disassembler | +| Resolve symbols/source/inline frames | Optional | Later ELF symbol / DWARF layer | + +An optional per-instruction view must walk code using correct 16-/32-bit Thumb instruction lengths. It must also +preserve execution decisions and incomplete-path states; formatting a range as assembly does not establish that +every instruction printed was executed. OpenCSD's instruction-decoder interface can be evaluated for this walk +before adding a dependency. Its missing architecture features remain an upstream integration issue. + +## 5. What OpenCSD outputs + +### 5.1 Generic elements, not an assembly listing + +Full decode calls `ITrcGenElemIn::TraceElemIn` with an input index, source Trace ID, and a +`const OcsdTraceElement&`, derived from `ocsd_generic_trace_elem`. The C API exposes the generic C structure. +The principal instruction result is `OCSD_GEN_TRC_ELEM_INSTR_RANGE`. [O3][O16] + +Important fields are: + +- Start address and exclusive end address: `[st_addr, en_addr)`. +- ISA and `num_instr_range`; mixed Thumb instruction widths prevent deriving count from byte length alone. +- Last instruction size, type/subtype, conditional flag, and `last_instr_exec`. +- Optional cycle-count information, indicated by its validity flag. +- Source and position supplied by the callback; processor/security/context maintained from related elements. + +The upstream M7 fixture contains this generic-element text output: [O22] + +```text +Idx:21; ID:6; OCSD_GEN_TRC_ELEM_INSTR_RANGE(exec range=0x10000140:[0x1000014a] num_i(3) last_sz(2) (ISA=T32) E --- ) +Idx:21; ID:6; OCSD_GEN_TRC_ELEM_EXCEPTION(pref ret addr:0x1000014a; excep num (0x0b) ) +Idx:25; ID:6; OCSD_GEN_TRC_ELEM_TIMESTAMP( [ TS=0x000000000060]; ) +``` + +This is checked-in OpenCSD test output, not a new ctrace execution or a proposed ctrace file syntax. +It represents three Thumb instructions in `[0x10000140, 0x1000014a)`, an exception and a time anchor for source 6. +Range end is the exclusive boundary of the represented code, not a universal branch-destination field. Keep +subsequent addresses and flow metadata rather than inventing a target from `en_addr`. + +### 5.2 Information that must survive normalization + +This is the full target semantics. The minimal milestone retains instruction ranges and the state/diagnostics +needed for its declared scope; richer context/time records follow later. Unsupported transitions must remain visible. + +| Generic element / property | Required interpretation | +| --- | --- | +| `INSTR_RANGE`, last-instruction flags | Preserve range/count and E/N execution decisions. | +| `I_RANGE_NOPATH` | Incomplete path; no complete instruction list or exact coverage. | +| `PE_CONTEXT` | Update ISA/security/context validity; keep unknown fields unknown. | +| `EXCEPTION`, `EXCEPTION_RET` | Preserve addresses, number validity, return and tail-chain information. | +| `TRACE_ON`, `NO_SYNC`, `EO_TRACE` | Preserve start/restart/end and reasons, including gaps and overflow. | +| `ADDR_NACC`, `ADDR_UNKNOWN`, `UNKNOWN` | Expose missing code/address/invalid-decode state as an observable result. | +| `TIMESTAMP`, `CYCLE_COUNT`, `SYNC_MARKER` | Preserve time, association, units and validity. | + +The last instruction of a range can be conditional and unexecuted. `num_instr_range` counts represented +instructions; do not blindly treat it as the number executed without applying the execution flags. [O3][O17] + +The `ADDR_NACC` condition can leave the decoder waiting for a subsequent trace address before it can reconstruct +flow again. Retain partial output and the gap. Decoder success alone does not prove image identity or complete +instruction coverage. Input positions from a formatted stream must keep the existing distinction between exact +offsets and formatter hints. [O3][C3] + +## 6. Time and downstream processing + +Instruction order, covered instruction count, CPU cycle count, and timestamp are different quantities. +Timestamp elements provide source-specific trace-time anchors. Preserve protocol-specific association and any +`SYNC_MARKER` locating an anchor earlier in the stream. They do not assign an exact time to every instruction. +ETMv3/PTM can attach cycle counts to ranges or other elements. ETMv4 has explicit cycle-count elements, and a +cycle count accompanying an ETMv4 timestamp has special association semantics and must not simply be added to +the cumulative count. [O3] + +The semantic model should preserve sequence order, raw time anchors and optional cycle information with an explicit +time-domain identity and quality. Do not feed ETM through the existing ITM local-timestamp accumulator/prescaler. +Do not distribute cycles equally across instructions, treat instruction count as time, or infer cross-source +ITM/ETM clock correlation merely because both sources belong to one processor. + +Suggested processing layers: + +1. **Range preservation:** source, addresses, ISA, counts, flow flags, context, status and timing evidence. +2. **Optional instruction expansion:** read exact bytes, advance by decoded instruction size, retain E/N and gaps. +3. **Optional presentation:** disassembly, symbols and source lines. +4. **Optional analysis:** coverage, branch statistics, call/return analysis and profiling. + +Call-graph construction additionally needs exception/tail-chain, indirect branch, synchronization-loss and return +handling. Trace-buffer wrap or trace filtering can remove the matching entry of a call. Report incomplete stacks +instead of presenting an inferred complete call tree. Coverage needs a definition of executed versus merely covered +or skipped conditional instructions. Time profiling requires qualified time data. + +## 7. Required ctrace extensions + +This section maps the complete target design. Section 1.1 defines the first implementation milestone. + +### 7.1 Reusable boundaries and missing behavior + +| Area | Existing implementation | Required extension | +| --- | --- | --- | +| Discovery / file jobs | TB discovery, bounded reads | Reuse; specify protocol and buffer normalization. | +| Trace-run reader | ETM references discarded | Parse source instances, decoder state and image metadata. | +| Routing | One ITM binding per processor | Allow ITM and ETM sources for the same processor. | +| OpenCSD lifecycle | Named decoder plus `CSConfig` | Reuse; add ETM configs, memory callback and dispatch. | +| Adapter | Only `ITMTRACE` elements consumed | Add instruction generic-element normalization. | +| Semantic pipeline | ITM/DWT-specific | Dispatch by protocol; separate time/state logic. | +| Events / filters | ITM/DWT, sampled PC, `tcyc` | Add ranges, context, gaps and timing evidence. | +| CSV | Eight fixed columns | Minimal `instr_range` mapping; richer schema later if needed. | +| CTF | ITM/DWT profile, clock requirements | Later: instruction/context events and time policy. | +| Diagnostics / tests | Catalog, RAW context, partial results | Add source/image/path diagnostics and fixtures. | + +See [C1] through [C7] for the implementation boundaries. A sampled `pcsample` event is not an instruction path and +must not be repurposed to encode ranges. + +### 7.2 Proposed pipeline + +This is the broader target architecture. The minimal path uses one binary image, one ETM source and CSV as in +section 1.1; ELF/AXF, mixed sources, analysis and instruction CTF are later additions. + +```mermaid +flowchart TD + Capture[TB raw bytes and capture metadata] --> Input[Existing input job and framing] + Images[ELF AXF bin and runtime mappings] --> Memory[Source-aware code image provider] + Input --> Tree[OpenCSD tree and typed source routes] + Memory --> Tree + Tree --> ITM[Existing ITM and DWT adapter] + Tree --> Instr[Instruction generic-element adapter] + ITM --> Events[Backend-independent semantic events] + Instr --> Events + Events --> CSV[CSV output] + Events --> CTF[CTF output] + Events --> Diagnostics[Diagnostics and quality] + Instr --> Later[Optional expansion symbols and analysis] +``` + +Keep the existing single-tree lifecycle constraints and independent per-file state. Mixed ITM/ETM input must not +create competing decode trees or reuse an ITM-specific raw-packet monitor/recovery strategy for ETM. +Use protocol-specific synchronization rules; bounded input reads and formatter state can remain shared. [C3] + +### 7.3 Initial CSV contract and later output options + +Preserve instruction ranges as the primary semantic representation. Expanding every instruction by default can +greatly increase output volume and unnecessarily couples decode to presentation. + +For the first milestone, use the existing eight-column CSV with the new `instr_range` type and mapping in section 1.1. +Define how status/gap records and retained flow flags map to that output, and keep existing ITM/DWT semantics stable. +If later requirements outgrow this mapping, evaluate a separate instruction artifact or an explicitly versioned +common schema for richer context, timing and analysis data. These extensions require specification alignment; +they are not currently part of the published CMSIS contract. + +For the later CTF milestone, define instruction/status/context records and schema/version compatibility. CTF requires +a valid clock frequency for selected streams and timestamps that do not regress. An ordered but untimed range stream +does not automatically meet that contract. Options include deferring CTF until timing is qualified, or defining an +explicit sequence-clock profile with documented viewer limitations. Never label a sequence counter as elapsed +nanoseconds. Existing ITM/DWT CTF and Trace Compass XML contracts must be preserved. [C6][C7] + +## 8. Implementation sequence and validation gates + +| Step | Deliverable | Completion evidence | +| --- | --- | --- | +| 0. Select reference | One protocol/CPU, configuration and matching bin | Reviewed chronological capture bundle. | +| 1. Minimal implementation | Five extensions in section 1.1; range CSV | Expected ranges/gaps; ITM/DWT compatibility. | +| 2. Image formats | ELF/AXF reader and richer mappings | Valid/invalid files; execution-address fixtures. | +| 3. Routing/context | ITM/ETM, multiple CPUs, richer context | Correct routing with overlapping VMAs. | +| 4. CTF/viewers | Qualified instruction/time profile | Independent consumer and viewer validation. | +| 5. Additional coverage | Further protocols; M23/Armv8.1-M qualification | Feature fixtures; upstream gaps addressed. | + +MTB is separate from step 5: it needs its own record-format/circular-buffer contract and decoder, though the image +provider, instruction-range model and output layers should be reusable. PTM/ETE are future scope unless explicitly +selected. Changing the OpenCSD dependency or fixing upstream feature gaps also requires a separate reviewed change. + +Apply validation as the corresponding capability is implemented; section 1.1 defines the minimal milestone's gate. +The complete validation plan should cover: + +- Reproducibly built small firmware with mixed-width Thumb, taken/not-taken branches, direct/indirect control flow, + calls/returns, exceptions, tail-chaining and, where relevant, Security transitions. +- Actual instruction-trace capture, decoder configuration, matching images and independent expected ranges. + Existing `TB-Trace` ITM captures are not evidence of ETM instruction decode. +- Upstream M7/M33 snapshots as integration references, with licensing/provenance reviewed before copying fixtures. +- Variable byte-chunk boundaries through formatter frames and protocol packets; identical semantic output. +- Capture starting mid-stream, buffer wrap, synchronization loss, overflow, incomplete tail and missing code bytes. +- Wrong image identity, corrupted ELF bounds, overlapping mappings, RAM execution and multiple core-local VMAs. +- Preservation of conditional execution, unknown-path ranges and partial results after unavailable memory. +- Timestamp/cycle-count association and explicit absence of timing; no invented cross-source correlation. +- M23 exception encoding and Armv8.1-M features as separate qualification gates rather than generic smoke tests. + +Upstream fixtures demonstrate relevant decoder paths; they are not a substitute for running ctrace against those +paths. This research did not build a new decoder or execute ETM qualification tests. + +## 9. Remaining decisions + +The single-source/bin-to-range-CSV milestone and its initial column mapping are agreed in section 1.1. +Before implementing it, settle: + +1. Which first real capture is available: M7 ETMv4, M33 ETMv4, or a qualified M3/M4 ETMv3 source? +2. Which fields carry its complete decoder configuration and static binary execution mapping in trace-run metadata? +3. Which observable CSV status/gap records and flow flags accompany `instr_range` for this capture? + +For later milestones, decide producer ownership of image identity/security/runtime mappings, the ELF reader, +whether richer CSV needs a new schema, the timing/CTF contract, additional processor qualification and desired +analyses. These decisions do not require an ELF loader, CTF or multi-CPU implementation in the first milestone. + +## 10. Evidence and references + +OpenCSD links below are pinned to the inspected revision. Hardware manuals were fetched and read through the +CMSIS documentation tools; page numbers above refer to PDF pages. Their product revisions matter. +The support matrix is bounded by this evidence, not a claim about every SoC integrating these processor cores. + +- [O1] OpenCSD README, support list and pending features. +- [O2] OpenCSD programming guide: tree, configuration, memory images, callbacks and data flow. +- [O3] Generic-element interpretation guide; [structure and fields][O17]. +- [O4] Memory accessor API; [callback types][O18] and [global mapper implementation][O19]. +- [O5] Instruction-decoder interface; [implementation][O20] and [automatic tree attachment][O21]. +- [O6] Decoder registry and protocol implementation directories. +- [O7] Upstream M7, M33 and Armv8.1-M snapshots. +- [O8] ETMv3 M-profile gate and core architecture map. +- [O9] Thumb low-overhead-loop instruction support. +- [O10] Protocol configuration structures. +- [O11] ETMv3 full-decoder mode validation. +- [O12] ETMv3 M-profile exception packet parsing. +- [O13] Processor-name to architecture/profile map. +- [O14] ETMv3 configuration structure. +- [O15] ETE configuration structure. +- [O16] C++ generic-element callback interface. +- [O22] Checked-in M7 generic-element output example. +- [O23] Memory-cache keys and context assumptions; [cache read implementation][O24]. +- [O25] ETMv4 context update and memory-cache invalidation. +- [A1] Arm Cortex-M resources: architecture and trace documentation map. +- [A2] Arm CoreSight ETM-M7 TRM, DDI0494D, r0p1. +- [A3] Arm CoreSight ETM-M23 TRM, DDI0563C, r0p1. +- [A4] Arm CoreSight ETM-M55 TRM, 101053_0101_03_en, r1p1. +- [A5] Arm CoreSight ETM-M85 TRM, 101926_0101_07_en, r1p1. +- [A6] Published CMSIS experimental trace configuration and output contract. +- [A7] Arm CoreSight ETM-M33 TRM, 100232_0002_06_en, r0p2, issue 06. +- [A8] Arm Cortex-M35P introduction, integrated debug and trace, PDF p.15. +- [A9] Arm China CoreSight ETM-M52 TRM, 102774_0002_05_en, r0p2, issue 05. +- [A10] Arm employee explanation of M0/M0+ trace and MTB. +- [E1] ELF program-header/loading contract. +- [E2] Arm ELF ABI. +- [E3] ELFIO upstream project and license. +- [L1] ELFIO MIT license, copyright and distribution condition. +- [L2] Apache Software Foundation third-party license policy, Category A including MIT. +- [L3] Apache License 2.0, redistribution and NOTICE requirements. +- [C1] ctrace architecture and input constraints. +- [C2] Current trace-run schema and [YAML reader](../src/tracerun/YmlTraceRunConfigReader.cpp). +- [C3] OpenCSD adapter, [tree lifecycle](../src/decode/OpenCsdTreeSession.cpp) + and [pipeline](../src/decode/DecodePipeline.cpp). +- [C4] Current route metadata and normalization. +- [C5] Semantic events and output selectors. +- [C6] CTF output requirements and [CSV row mapping](../src/output/csv/CsvRowMapper.cpp). +- [C7] Current CTF profile and deferred decoder work. +- [C8] Current trace-directory naming and CLI contract. +- [C9] File decode job and pipeline creation. +- [C10] Shared CTF event-payload visitor. + +[O1]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/README.md +[O2]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/docs/prog_guide/prog_guide_main.md +[O3]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/docs/prog_guide/prog_guide_generic_pkts.md +[O4]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/common/ocsd_dcd_tree.h +[O5]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/interfaces/trc_instr_decode_i.h +[O6]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/ocsd_lib_dcd_register.cpp +[O7]: https://github.com/Linaro/OpenCSD/tree/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/tests/snapshots +[O8]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv3/trc_cmp_cfg_etmv3.h +[O9]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/i_dec/trc_idec_arminst.cpp +[O10]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv4/trc_pkt_types_etmv4.h +[O11]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp +[O12]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_proc_etmv3_impl.cpp +[O13]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/trc_core_arch_map.cpp +[O14]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv3/trc_pkt_types_etmv3.h +[O15]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/ete/trc_pkt_types_ete.h +[O16]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/interfaces/trc_gen_elem_in_i.h +[O17]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/trc_gen_elem_types.h +[O18]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/ocsd_if_types.h +[O19]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/mem_acc/trc_mem_acc_mapper.cpp +[O20]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/i_dec/trc_i_decode.cpp +[O21]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/ocsd_dcd_tree.cpp +[O22]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/tests/snapshots/v7m_svc_vector/pkt_proc_logs/v7m_svc_vector.ppl +[O23]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/mem_acc/trc_mem_acc_cache.h +[O24]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/mem_acc/trc_mem_acc_cache.cpp +[O25]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp +[A1]: https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/cortex-m-resources +[A2]: https://documentation-service.arm.com/static/5e906dfe8259fe2368e2ac54 +[A3]: https://documentation-service.arm.com/static/5e90936fc8052b16087620d0 +[A4]: https://documentation-service.arm.com/static/6622c0bbfabc8c11c7b53135 +[A5]: https://documentation-service.arm.com/static/662a6ea9c7622e1c31ae08ee +[A6]: https://open-cmsis-pack.github.io/cmsis-toolbox/Experimental-Features/ +[A7]: https://documentation-service.arm.com/static/66be2251882fec713ef49f7b +[A8]: https://armkeil.blob.core.windows.net/developer/Files/pdf/white-paper/introduction-to-cortex-m35p.pdf +[A9]: https://documentation-service.arm.com/static/65849e9cb52744113be61b26 +[A10]: https://community.arm.com/forums/f/architectures-and-processors-forum/8383/trace-debug-functionality-in-arm-cortex-m0/28550 +[E1]: https://gabi.xinuos.com/elf/07-pheader.html +[E2]: https://github.com/ARM-software/abi-aa/blob/main/aaelf32/aaelf32.rst +[E3]: https://github.com/serge1/ELFIO +[L1]: https://github.com/serge1/ELFIO/blob/main/LICENSE.txt +[L2]: https://www.apache.org/legal/resolved.html#category-a +[L3]: https://www.apache.org/licenses/LICENSE-2.0 +[C1]: architecture.md +[C2]: ../src/tracerun/TraceRunConfig.h +[C3]: ../src/decode/OpenCsdPacketCollector.cpp +[C4]: ../src/tracerun/CtraceRunMeta.cpp +[C5]: ../src/model/TraceEvent.h +[C6]: ../src/output/OutputRequirements.cpp +[C7]: ctf-format.md +[C8]: ../README.md +[C9]: ../src/control/FileDecodeJob.cpp +[C10]: ../src/output/ctf/CtfEncoder.cpp From bb88fffae32e5f548a5adf7dab229d1fc9bd5c7d Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Mon, 9 Nov 2026 13:17:36 +0100 Subject: [PATCH 2/6] docs(ctrace): refine instruction trace design and decoder findings --- tools/ctrace/docs/instruction-trace-design.md | 1096 +++++++++-------- tools/ctrace/docs/opencsd-issues.md | 111 ++ 2 files changed, 692 insertions(+), 515 deletions(-) diff --git a/tools/ctrace/docs/instruction-trace-design.md b/tools/ctrace/docs/instruction-trace-design.md index f12c7668e..53e59f9a1 100644 --- a/tools/ctrace/docs/instruction-trace-design.md +++ b/tools/ctrace/docs/instruction-trace-design.md @@ -1,606 +1,672 @@ # Instruction Trace Implementation Plan -Status: minimal implementation agreed as the starting scope; broader design remains a working draft. -The proposed CSV extension is local to this plan and is not yet part of the published CMSIS contract. -Research date: 2026-10-28. Baseline: devtools `17f8f54a`, branch `feat/ctrace-instruction-trace`. -OpenCSD baseline: **v1.8.3**, commit `59f15320637ce6e510604e7bed550d7ad109d15d`. - -This document separates verified library behavior, ctrace implementation gaps, and proposed decisions. -It describes offline decoding of captured files. Target configuration and capture remain producer responsibilities. -The separate ITM-channel-0 change is not part of this branch. - -## 1. Findings and proposed direction - -Instruction trace requires more than enabling another OpenCSD decoder for a `.raw` file: - -1. Identify the protocol and capture-time decoder configuration for every trace source. -2. Supply the actual executable code bytes at their execution addresses. -3. Consume OpenCSD generic instruction ranges, context, exceptions, time information, and discontinuities. -4. Define a semantic event model and an output contract that can retain this information. - -**OpenCSD already contains the instruction decoder needed for program-flow reconstruction.** A separate instruction -codec is not required for that task. An ELF/AXF reader and an address-aware image provider are needed if executable -files are the image source; raw binary images with explicit mappings are another valid input. Assembly text, -symbols, source lines, call graphs, and coverage are separate processing layers. [O2][O3][O4][O5] - -Start with the minimal implementation in section 1.1. The reference capture and its exact protocol still need to -be selected; a verified Cortex-M ETMv4 instruction-only capture from M7 or M33 is the preferred candidate. -Mixed ITM/ETM routing, additional protocols, ELF/AXF support and richer outputs belong to subsequent milestones. -Treat MTB as a separate decoder project. Qualify Armv8.1-M support by feature and test evidence rather than -advertising all M52/M55/M85 captures as supported. - -### 1.1 Minimal implementation — agreed starting point - -The first deliverable decodes **one recorded instruction-trace input, one known ETM protocol and one CPU** to CSV. -Use one matching static binary image at an explicit execution address. Emit one row per decoded instruction range, -with observable gaps and missing-code conditions. This is the starting scope agreed on 2026-10-28. - -```text -RAW -> existing framing -> OpenCSD ETM full decoder -> instruction-range event -> CSV - ^ - matching binary code bytes +Updated: 2026-11-06. Branch: `feat/ctrace-instruction-trace`. +OpenCSD: **1.8.3**, commit `59f15320637ce6e510604e7bed550d7ad109d15d`. + +## Experimental specification + +Basis: [CMSIS-Toolbox Experimental Features][experimental-spec], `ctrace` and `*.ctrace-run.yml`. + +### Trace + +#### Directory and File Structure + +| Directory or File | Created by | Description | +| :---|:---|:--- | +| `.trace/.ctrace-run.yml` | pyTS / capture tooling | Source configuration and image inventory. | +| `.trace/..raw` | Capture backend | ETM input: formatted `TB` or `TB_`. | +| `.trace/...csv` | ctrace | CSV including instruction trace | + +- One trace-buffer RAW file contains multiple trace sources, identified by their ATB `stream` IDs. +- Decode each source with its own configuration and state; combine all sources in one CSV, retaining `stream`. + +#### File Structure of `*.ctrace-run.yml` + +| `ctrace-run:` | | Content | +| :---|:---|:--- | +|     `ctrace-images:` | Optional | Processor image inventory for instruction reconstruction. | + +| `ctrace-images:` | | Content | +| :---|:---|:--- | +| `- pname:` | **Required** | Unique processor instance name, including single-core captures. | +|     `elf:` | Optional | List of `.elf`/`.axf` paths, relative to `*.ctrace-run.yml`. | +|     `hex:` | Optional | List of Intel HEX paths, relative to `*.ctrace-run.yml`. | + +- Declared image entries need a nonempty file list; processors without images may be omitted. +- Select one format per processor; selection when both lists exist: TBD. HEX production remains open. +- Map matching code to execution addresses; reject invalid files and conflicting overlaps. + +> **Possible extension:** Mixed ELF/HEX files per `pname`. + +ETM source entries: + +| `ctrace-refs:` | | Content | +| :---|:---|:--- | +| `- ref:` | **Required** | Trace-setting reference, for example `CM7/instructions`. | +|     `type:` | **Required** | New trace source type: `etm`. | +|     `pname:` | Conditional | Required with image inventory; agrees with any processor prefix in `ref`. | +|     `stream:` | **Required** | Capture-time CoreSight ATB ID; decoder Trace ID. | +|     `decoder:` | **Required** | Capture-specific ETM decoder configuration. | + +| `decoder:` | | Content | +| :---|:---|:--- | +|     `core:` | **Required** | `Cortex-M4` (ETMv3.5) or `Cortex-M7` (ETMv4.0). | +|     `regs:` | **Required** | Fixed qualified Cortex-M implementation values, written to YAML by pyTS. | +|     `regsToRead:` | **Required** | Variable configuration registers read by pyOCD. | + +- ETM references need no `index`; reject ambiguous processor/source bindings. +- A source's `pname` may have no image entry; this selects image-free fallback, not another processor's images. +- Retain unambiguous legacy processor inference without an inventory and existing DWT `symbol-file` provenance. + +##### Register Accesses + +| Cortex-M profile | `regs` (pyTS) | `regsToRead` (pyOCD) | +| :---|:---|:--- | +| Cortex-M4 / ETMv3.5 | `ETMIDR`, `ETMCCER` | `ETMCR` | +| Cortex-M7 / ETMv4.0 | `TRCIDR0`, `TRCIDR1`, `TRCIDR2`, `TRCIDR8` | `TRCCONFIGR` | + +| `regs:`, `regsToRead:` | | Content | +| :---|:---|:--- | +| `- name:` | **Required** | Symbolic register name. | +|     `value:` | Conditional | Required in `regs`; filled by pyOCD in `regsToRead`. | + +- `ETMCR` / `TRCCONFIGR`: complete effective configuration. + +##### Example + +M7 reference values; existing setup/disable entries omitted. ELF and HEX lists illustrate the pending format selection: + +```yml +ctrace-run: + ctrace-images: + - pname: CM7 + elf: [Bootloader.axf, Firmware.elf] + hex: [Bootloader.hex, Firmware.hex] + ctrace-refs: + - ref: CM7/instructions + pname: CM7 + type: etm + stream: 0x12 + decoder: + core: Cortex-M7 + regs: + - name: TRCIDR0 + value: 0x080006e1 + - name: TRCIDR1 + value: 0x4100f401 + - name: TRCIDR2 + value: 0x00000004 + - name: TRCIDR8 + value: 0x00000001 + regsToRead: + - name: TRCCONFIGR + value: 0x00000001 # Filled by pyOCD ``` -Reuse the existing input convention, for example `Board.TB.raw` with `Board.ctrace-run.yml` in a trace directory. -A freely named `trace.raw` is not currently a standalone CLI input. Adding direct-file CLI support is a later option. -The buffer must already contain valid bytes in chronological order with known framing. [C8] +#### Initial Implementation -The necessary inputs are the actual decoder configuration, architecture/profile, source Trace ID where formatted, -and the matching `.bin` with its execution mapping. A RAW file alone does not establish these facts. Agree the -minimal metadata fields against the first capture; do not infer missing configuration or code bytes. +- ETMv3 and ETMv4: default one PC per line (`instructions`), using ELF or HEX; optional image-free `packets`. -Implement these five connected extensions: +#### Remaining Design Questions -1. **ETM configuration and route:** extend `TraceRunConfig`, `YmlTraceRunConfigReader` and `CtraceRunMeta` to retain - one ETM source, its decoder configuration and the binary-image mapping. ETM metadata is currently ignored. [C2][C4] -2. **Code-memory access:** attach a binary file/region accessor to the OpenCSD tree, supplying the actual bytes at - execution addresses. One static mapping is sufficient for this milestone. An ELF/AXF reader is not required. [O4] -3. **ETM decoder path:** create the selected ETM full decoder and connect it through `DecodePipeline` and - `FileDecodeJob`. Reuse bounded RAW reads, compatible deformatting and the existing tree lifecycle; keep ITM-specific - packet monitoring, recovery and timestamp accumulation out of the ETM path. [C3][C9] -4. **Instruction-element adapter:** consume `INSTR_RANGE` and the state needed to interpret it. The current collector - accepts only `ITMTRACE`. Preserve range/count/execution flags and expose synchronization loss, missing code and - unsupported context transitions; retain valid partial output. [C3][O3] -5. **Events, selection and CSV:** add an instruction-range payload to `TraceEvent`, a selectable event type and its - `CsvRowMapper` mapping. Shared event visitors, including CTF, must handle the added variant; instruction CTF - requests must report unsupported output until that backend is implemented. [C5][C6][C10] +- Whether to offer `--etm-output decoded`: generic OpenCSD output elements, using ELF or HEX. +- ELF/HEX selection when both lists are present; ownership of image-inventory and HEX production. +- Producer/CMSIS schema alignment for these extensions. -The minimal CSV extension retains the existing eight columns and introduces `type=instr_range`: +#### Out-of-Scope -```csv -cycles,stream,type,index,value,pc,address,note -,18,instr_range,,,0x08000100,0x08000108,"count=3; isa=T32; last_executed=1" -``` +- Disassembly and ETM CTF. -- `stream`: source Trace ID for this formatted example; preserve existing unformatted stream conventions. -- `pc`: range start; `address`: exclusive range end. -- `note`: instruction count, ISA and last-instruction execution flag; retain additional validity/flow flags as needed. -- `index` and `value`: empty for this record type. -- `cycles`: empty without qualified cycle information. Row order preserves decoded order; no synthetic time is added. +### `pyTS` Utility -This is the agreed initial mapping for implementation, not an existing specification claim. Document its type-specific -semantics and diagnostic mapping before exposing it. Existing ITM/DWT row meanings remain unchanged. -The count describes represented instructions; execution flags still govern whether the last instruction executed. +- Emit consistent `pname` values on DWT/ETM references and any image entries. +- Populate `decoder.regs` from the qualified Cortex-M implementation; list variable registers in `decoder.regsToRead`. -Completion evidence is an actual capture with complete configuration, matching bytes and independently expected -ranges; correct handling of missing code and synchronization gaps; chunk-boundary-independent results; and unchanged -existing ITM/DWT CSV behavior. The first capture must fit the declared static image/context scope; report unsupported -transitions instead of silently claiming a complete path. +### `ctrace` Utility -Subsequent work includes ELF/AXF parsing, assembly text, symbols/DWARF, full context output, additional protocols, -mixed sources/multiple CPUs, time correlation, CTF, coverage and call-graph analysis. OpenCSD's existing instruction -decoder supplies flow decoding for the first milestone; no additional instruction-codec dependency is needed. +```txt +Usage: + ctrace [options] -## 2. OpenCSD support and its limits +Options: + --csv Export CSV (ETM default: instructions) + --type sel [...] Filter output types; additional ETM selectors: etmv3, etmv4 + --etm-output mode Select ETM mode for --csv (default: instructions) + instructions One PC per line (ELF or HEX) + packets ETM protocol packets (no image) + decoded OpenCSD generic elements (ELF or HEX; TBD) +``` -### 2.1 Trace protocols +- `--csv`: ETM output uses `instructions`. +- Optionally add `--etm-output mode` to select another ETM output mode. -The pinned README and decoder implementations establish the following support. A protocol being implemented -does not establish correct decoding of every processor, trace option, or instruction extension. [O1][O6] +#### `--type` option -| Trace source | OpenCSD v1.8.3 | Consequence for ctrace | -| --- | --- | --- | -| ETMv3 instructions | Full decode, through v3.5 | M3/M4 candidate; qualify M23. | -| ETMv3 data | Packet processing only | Full decoder rejects data and mixed instruction/data modes. | -| ETMv4 instructions | Full decode; v4.6 A/R, v4.4 M advertised | Main candidate; qualify actual features. | -| ETMv4 data | Packet processing/decode are future work | Outside initial scope. | -| PTM v1.1 | Full instruction decode | Future A-profile extension. | -| ETE v1.3 | Full instruction decode | Future processor-trace extension. | -| ITM v1.0, STM v1.1 | Full software-trace decode | Not an executed instruction path. | -| MTB | No built-in decoder | Separate record decoder and input contract. | -| Earlier ETMv1/v2 | No built-in decoder registered | Generic ETM naming does not establish support. | - -OpenCSD supports both packet-processing-only and full-decode configurations. Packet inspection can work without -code images; the desired reconstructed instruction path generally cannot. All decoder families are present in -the OpenCSD source dependency; ctrace currently instantiates only ITM. [O2][O6] - -ETMv3 full decode explicitly accepts only instruction-only mode. A capture configured for combined instruction/data -trace is not handled by simply ignoring its data packets. [O11] - -### 2.2 Cortex-M architecture and trace-source mapping - -The architecture generation and the trace-protocol generation are different axes. The physical device must -actually implement the optional trace source and a usable capture path. A processor name alone is insufficient -decoder configuration. The following matrix is a planning guide, not a device support declaration. [A1][A2][A3] - -| Architecture / processor examples | Relevant instruction-trace source | Assessment for this OpenCSD baseline | -| --- | --- | --- | -| Armv6-M: M0, M1 | No ETM path established here | ISA alone does not establish trace support. | -| Armv6-M: M0+ | Optional MTB [A10] | No built-in MTB decoder. | -| v7-M / v7E-M: M3, M4 | Optional ETMv3 | Protocol implemented; qualify captures/exceptions. | -| Armv7E-M: M7 | ETM-M7, ETMv4 | Upstream M7 fixture exists. | -| v8-M Baseline: M23 | ETM-M23: **ETMv3.5**; or MTB | OpenCSD compatibility risk; see below. | -| v8-M Mainline: M33 | ETM-M33 r0p2: **ETMv4.2** | Within advertised scope; M33-profile fixture exists. | -| v8-M Mainline: M35P | Optional ETMv4.2 / MTB | Protocol within advertised scope; qualify captures. | -| Armv8.1-M: M55, M85 | ETM-M55/M85 r1p1: **ETMv4.5** | Partial evidence; not complete qualification. | -| Armv8.1-M: M52 | ETM-M52 r0p2: **ETMv4.5** | Same Armv8.1-M qualification gate. | - -The M7 mapping is confirmed by ETM-M7 TRM DDI0494D, PDF p.7, and by an upstream snapshot. -ETM-M23 TRM DDI0563C confirms ETMv3.5 on PDF p.11 and p.20. ETM-M55 TRM -101053_0101_03_en and ETM-M85 TRM 101926_0101_07_en identify ETMv4.5 on PDF p.14. -ETM-M33 TRM 100232_0002_06_en identifies ETMv4.2 on PDF p.7. Its OpenCSD snapshot labels the device `ETM4.4`; -that test metadata is not evidence that ETM-M33 hardware implements v4.4. -Arm's M35P introduction identifies ETMv4.2 on PDF p.15; ETM-M52 TRM 102774_0002_05_en identifies ETMv4.5 -on PDF p.14. [A2][A3][A4][A5][A7][A8][A9][O7] - -Two specific OpenCSD gaps affect the design: - -- **M23:** `EtmV3Config::isV7MArch()` requires exactly `ARCH_V7` and `profile_CortexM`. The ETMv3 packet processor - uses this check in M-profile exception handling, whereas the processor-name map associates M23 with `ARCH_V8`. - This is a concrete compatibility risk, not proof that all M23 trace fails. Require M23 exception/security fixtures - and an upstream assessment. Do not misdeclare M23 as Armv7 to bypass the check. [O8][O12][O13] -- **Armv8.1-M:** the README explicitly leaves M-class branch-forward instruction support as future work. - There is already an `armv8_1m_branches` M55 snapshot and Thumb code for low-overhead loops, so support is not - simply absent. Equally, that fixture does not establish complete Armv8.1-M support. A separate disassembler - cannot automatically repair missing OpenCSD path-reconstruction semantics. [O1][O7][O9] - -## 3. Required input: capture, configuration, and code - -### 3.1 Trace-buffer bytes - -Existing discovery already accepts `.TB.raw` and `.TB_.raw`. -The extension `.raw` and the channel name identify a captured input, not its protocol. A buffer can contain -formatted ETM, mixed ETM/ITM, an unformatted single source, or MTB records. [C1] - -For formatter-based captures, the producer must export the valid bytes in chronological order. A raw memory dump -of a circular buffer is insufficient if its write pointer, wrap state, valid length, or alignment is unknown. -The producer must either normalize the buffer or provide enough metadata to do so before deformatting. -Document padding and flush behavior; do not decode unwritten capacity as trace. - -The current ctrace transport accepts 16-byte memory-aligned CoreSight frames, without FSYNC/HSYNC framing. -That is reusable for compatible ETB/ETF/ETR-style exports. Other framing modes require an explicit contract. -The current `TB => formatted` default must not cause an MTB byte array to enter the CoreSight deformatter. [C1][O2] - -### 3.2 Source configuration - -Each source needs a stable source-instance identity, processor association, protocol/revision, architecture/profile, -and an architectural ATB Trace ID when formatted. Unformatted input still needs an explicit source/protocol binding. -Trace ID identifies the source within an input; it is neither an ITM stimulus channel nor a universal CPU number. - -The public OpenCSD configuration structures require the following fields. These names describe software API -inputs, not proposed new YAML keys or instructions to program target registers. [O10] - -| Decoder family | OpenCSD configuration inputs | -| --- | --- | -| ETMv3 | `reg_idr`, `reg_ctrl`, `reg_ccer`, `reg_trc_id`, `arch_ver`, `core_prof` | -| ETMv4I | IDR0/1/2/8–13, CONFIGR, TRACEIDR, architecture/profile | -| ETE, future scope | IDR0/1/2/8, DEVARCH, CONFIGR, TRACEIDR, architecture/profile | +- Selectors `etmv3`, `etmv4` select CSV types `ETMv3`, `ETMv4`. -The latter two rows abbreviate register-backed API fields with their register names; see the complete structures -for their `reg_` field names. ETMv3 and ETE have separate configuration structures. [O14][O15] +#### `--etm-output` option -The producer must distinguish unavailable fields from valid zero values. Determine required/unused fields for the -selected decoder configuration explicitly. A list of masked register writes is not necessarily a complete decoder -snapshot: identification registers and the resulting active configuration may be absent. +| Mode | Required images | Output records | +| :---|:---|:--- | +| `instructions` (default) | ELF or HEX; fallback if absent | PCs, events/status and fallback packets. | +| `packets` | None; no image opened | ETM protocol packets, parser status and diagnostics. | +| `decoded` (TBD) | ELF or HEX | Generic OpenCSD ranges, events and status. | -The current CMSIS `instructions:` intent and `ctrace-refs` do not by themselves supply this complete contract. -ctrace does not currently retain ETM references, their register state, or code-image descriptions. -Agree ownership with the debugger/pyTS producer before inventing a ctrace-private schema. [C2][A6] +- No matching image inventory: warn once per source/input and use packet fallback in `instructions`. +- Image-backed modes: missing/unreadable declared files, invalid images and ambiguous format selection are errors. + `packets` neither validates image files nor selects a format. Unmapped code uses local packet fallback. +- `decoded` remains unavailable until offering it is decided; if offered, absent images are an error. +- `--etm-output` requires CSV output (`--csv` or `--all`); it does not change ITM/DWT decoding. +- Without an output option, validate/decode using the default mode and fallback; write no output file. +- Unknown/unimplemented modes are errors. Fallback retains the selected filename and header. -### 3.3 Executable images +**Example:** -Full instruction decode combines compressed trace with code bytes. Direct branches and sequential instructions -are reconstructed from those bytes; the trace is not a recording of every instruction opcode. [O2][O3] +```bash +ctrace .trace --csv +ctrace .trace --csv --channel TB --etm-output packets +``` -Every image mapping must identify: +#### CSV Format -- File/snapshot identity and preferably a capture-time content hash or build identity. -- Processor/source association and Secure/Non-secure address space where applicable. -- Execution address range, file offset and available file-backed length. -- Any explicit relocation, RAM execution mapping, ROM alias, or time-dependent overlay. +**Common fields:** existing column meanings and relative order, with `packet` added. -The minimal milestone uses one static binary mapping and a capture within that supported execution context. -For broader captures, include bootloader, application, Secure/Non-secure firmware, and required ROM code when crossed. -A valid ELF with the wrong build can produce plausible but incorrect reconstruction. Source symbols alone do not -provide the code bytes. Modified RAM code, software breakpoint patches, or dynamically generated code require -matching capture-time bytes or a documented limitation. +```csv +cycles,stream,type,packet,index,value,pc,address,note +``` -## 4. ELF/AXF reader and instruction decoder responsibilities +| Mode | Additional fields before `note` | Total columns | +| :---|:---|:--- | +| `instructions` | `execution` | 10 | +| `packets` | `packet_size`, `packet_bytes` | 11 | +| `decoded` (TBD) | `instruction_count` | 10 | + +| Column | ETM meaning | +| :---|:--- | +| `cycles` | Attributable CPU-cycle position; otherwise empty. | +| `stream` | CoreSight ATB ID; a processor may own several sources. | +| `type` | `ETMv3` or `ETMv4`; distinct from YAML `type: etm`. | +| `packet` | `INSTRUCTION`, protocol class, generic record kind or parser status. | +| `index` | Architectural exception/vector number when known; otherwise empty. | +| `value` | Instruction opcode, exception transition or valid timing value, in hexadecimal. | +| `pc` | Instruction/range start; empty for protocol-only address reports. | +| `address` | Range end, inaccessible code address, protocol address or preferred exception-return address. | +| `execution` | Predication: `executed`, `not_executed`, `unknown`; not retirement/completion. | +| `instruction_count` | Decoded range count; unknown counts stay empty. | +| `packet_size` | Captured ETM source-byte count, including packet headers. | +| `packet_bytes` | Space-separated hex bytes in trace order, after CoreSight deformatting. | +| `note` | Structured execution reasons, packet/event fields and diagnostics. | + +- One CSV per RAW input. Configured ETM sources select the mode filename/header before row filtering, + even when a file contains no ETM bytes. ITM/DWT rows retain their payload fields; + `packet` and ETM-only fields stay empty. Inputs without configured ETM retain the eight-column CSV. +- Unavailable/inapplicable values stay empty. Preserve per-source order; no cross-source execution-time claim. +- ETM `note`: `key=value` pairs separated by `;`; escape backslash, semicolon and line endings as + `\\`, `\;`, `\r`, `\n`, then apply normal CSV quoting. Free diagnostic text uses `message`. + Absent keys mean unavailable, not zero. Existing ITM/DWT notes keep their format. + +**`instructions`:** + +- Expand complete ranges once, one PC per line; `packet=INSTRUCTION`, opcode in `value`, `index`/`address` empty. + Retain not-executed/unknown PCs and all other decoded events/status; never fabricate missing PCs. +- Explain execution state with `execution_basis` in `note`. CBZ/CBNZ can execute without taking the branch. +- Preserve opcode width in `value`; Thumb-32 uses first then second little-endian halfword: + bytes `2d e9 f0 41` become `value=0xe92d41f0`. +- Uncovered code: preserve valid PC prefixes and gaps, then emit the corresponding complete protocol packet once. + Continue packets through the gap; resume PCs only when code and decoder state permit reconstruction. + A packet may cover several ranges and overlap retained PCs: mark this with `partial=1`. +- Fallback notes contain `record=packet`, `reason=no_image|image_gap`, `packet_size`, `packet_bytes` and packet fields. + Instruction-specific columns stay empty. At a shared origin, decoded rows precede the complete fallback packet. + Fallback preserves observed packets; it is not an independently decodable subtrace. -### 4.1 What ctrace needs +```csv +cycles,stream,type,packet,index,value,pc,address,execution,note +,18,ETMv4,INSTRUCTION,,0x2902,0x08009a1c,,unknown,execution_basis=IT_state_unavailable +,18,ETMv4,INSTRUCTION,,0xd127,0x08009a1e,,not_executed,execution_basis=conditional_branch_predicate +,18,ETMv4,INSTRUCTION,,0x6a74,0x08009a20,,executed,execution_basis=unpredicated_path +``` -The first milestone uses raw `.bin` plus an explicit execution address; OpenCSD can access it without an ELF parser. -For a later ELF/AXF milestone, ctrace needs a **read-only image reader**, not a target programmer or dynamic linker. -Detect the format from its contents. The proposed first ELF support is validated ELF32 Arm `ET_EXEC` images and -file-backed load regions at execution VMAs. A stripped executable can be sufficient; symbols and DWARF are optional. -The ELF-specific rules below apply to that later milestone. [E1][E2] +**`packets`:** one row per protocol packet or parser status; no instruction reconstruction. -Use program-header file offsets and `p_filesz` to expose actual bytes, mapped to the execution address `p_vaddr`. -Do not use flash load address / `p_paddr` as a universal substitute for where the processor executes RAM code. -Validate toolchain-specific AXF/scatter-loading behavior with fixtures; any section-based fallback must be specified. -Do not synthesize missing instruction bytes from `p_memsz`, BSS, or gaps. Reject ambiguous overlaps, unsupported -relocations, and address arithmetic outside the supported domain. An initial implementation can reject `ET_REL` -and dynamically relocated images until their mapping contract is defined. +```csv +cycles,stream,type,packet,index,value,pc,address,packet_size,packet_bytes,note +,18,ETMv4,atom_f1,,,,,1,f7,atoms=E +``` -No reusable ELF/DWARF reader was found in the current ctrace/devtools C++ path. The implementation decision remains -open between a bounded reader and a portable library. ELFIO is a candidate to evaluate: it is header-only C++ with -an MIT license. Review a pinned version, validation behavior, footprint, and all release platforms before adoption. -A full LLVM dependency is not justified solely by the need to extract ELF code regions. [E3] +**`decoded` (TBD):** one row per generic OpenCSD element; ranges remain ranges. -#### 4.1.1 ELFIO licensing and distribution +```csv +cycles,stream,type,packet,index,value,pc,address,instruction_count,note +,18,ETMv4,instr_range,,,0x08000100,0x08000108,3,last_executed=1 +,20,ETMv4,instr_range,,,0x08000100,0x08000104,2,last_executed=1 +``` -ELFIO uses the MIT license. It can be included in an Apache-2.0 project; the Apache Software Foundation explicitly -lists MIT as a Category A license accepted in its products. Our ctrace code and wrapper can remain Apache-2.0; -retain the MIT license and attribution for the ELFIO portions. [L1][L2] +**Record fields:** apply only when valid for the record; other payload columns stay empty. [O3], [O17] + +| Record | CSV mapping | +| :---|:--- | +| `instr_range` | `pc` = start, `address` = exclusive end; `instruction_count`; final flags in `note`. | +| `i_range_nopath` | `pc` = start, `address` = continuation; valid count; no PC expansion. | +| Generic `exception` | Normalized `index`; `value=0x1` only for architectural entry; valid return `address`. | +| `exception_ret` | Return marker in `note`; no inferred destination exception or ITM exit/return transition. | +| `pe_context` | Context/security fields and validity in `note`. | +| `addr_nacc` | `address` = inaccessible code address. | +| `timestamp`, `cycle_count` | Valid complete value in `value`; domain/association and partial fields in `note`. | +| `event`, `sync_marker` | Event/marker kind and value in `note`; an event mask is not an exception index. | +| `trace_on`, `no_sync`, `eo_trace`, `addr_unknown`, `unknown` | Reason/state in `note`. | +| Protocol address | Complete valid address in `address`; partial address bits/mask in `note`. | +| Protocol exception fields | Normalized `index` when known; raw code, event kind and flags in `note`. | +| Other protocol fields | Structured atoms, context, counts, speculation and validity in `note`. | +| Parser status | `record=status`, status-specific reason and available source bytes; no invented packet. | + +- Normalize Cortex-M exceptions using the protocol-specific tables; OpenCSD's generic number is still protocol-raw. + Preserve `exception_raw` and `exception_kind` in `note`. Debug halt, lazy FP push and lockup leave `index`/`value` empty. +- ETMv4 reset has no known return address despite the generic API flag: leave `address` empty. + Preserve the transmitted address only as a protocol field with `address_role=unknown`. [A10, p. 270][A10] +- Preserve valid generic final-instruction flags as `last_executed`, `last_size`, `last_conditional`, `last_type`, + `last_subtype` in `decoded` notes; in `instructions` they are classifier inputs, not range-wide execution claims. +- Copy remaining semantic fields using OpenCSD member names under `packet.` or `element.` in `note`; + omit the constant ISA, duplicate mappings, inactive union members and internal bookkeeping. + Do not parse display strings. +- Preserve ETMv3 cancellation information (`cancel_previous=1`) even when images cover the entire trace. +- `cycles` needs qualified association; raw timestamps/counts do not imply per-PC timing or cross-core synchronization. + Preserve their domain, validity and association in `note`; do not distribute cycles across PCs. + +Packet names and parser status: Appendix B. Exception normalization: section 3.3. + +##### Loopback Requests + +- `ETMv3` / `ETMv4` resolve to `type: etm` references using `solution-set` and `stream`. +- `packet` and ETM exception `index` do not select separate trace settings. +- Recognize the mode suffix in `...csv`; input-wide diagnostics have no source reference. + +#### CTF Format + +- Evaluate CTF eligibility for configured sources after source/type selection. + Selected ETM sources make that input's CTF unsupported, even if no ETM bytes were observed. +- With `--all`, report the CTF error and return nonzero; still generate CSV and process other inputs. + Excluding ETM permits existing ITM/DWT CTF output. + +[experimental-spec]: https://github.com/Open-CMSIS-Pack/cmsis-toolbox/blob/main/docs/Experimental-Features.md + +## 1. Status and scope + +| Item | State | +| --- | --- | +| Production | ITM/DWT supported; ctrace ETM integration pending. Experimental specification above is the contract. | +| Required profiles | Cortex-M4/ETMv3.5 and Cortex-M7/ETMv4.0. | +| Capture evidence | Both profiles decoded with the host helper; PC/classifier validation currently M7 only. | +| Decoder blockers | Pinned OpenCSD ETMv3 cancellation and truncation defects; section 5. | +| Images | ELF/HEX import and parity with archived BIN references remain to be implemented/qualified. | -When integrating ELFIO: +## 2. Code-image provider -- Preserve upstream copyright and license notices in the imported headers. -- Include the complete upstream MIT license text, including copyright and warranty disclaimer, in distributed - source packages and binary release packages containing ELFIO code. Use the license from the pinned version. -- Include those notices with binaries even though ELFIO is header-only and has no separately linked library file. - A website link alone is not a substitute for shipping the notices. [L1] +- Merge selected regions per `pname`; different processors may map different bytes at the same address. +- Static instruction bytes at execution addresses, including RAM code; no relocation, gap padding or dynamic overlays. +- `.bin` is historical reference data, not an input format. Inventory and HEX production ownership remain open. -MIT does not require publication of our source code or modifications, and does not require a file specifically -named `NOTICE`. A practical integration is an ELFIO entry in the third-party component overview plus its original -license in the package's license directory, for example `licenses/ELFIO.txt`. The filename is a packaging choice; -the required notice content must reach recipients. Existing Apache-2.0 license and NOTICE obligations continue -to apply independently. [L1][L3] +| Reader | Import contract | +| --- | --- | +| Intel HEX | Validate record syntax/counts/checksums, EOF and bounds; `00`, `01`, `02`, `04` address/data records. | +| HEX start records | Validate `03`/`05`; do not use them as image bases or trace starting PCs. | +| ELF/AXF | ELF32, little-endian `EM_ARM`, static `ET_EXEC`, usable section headers. | +| ELF code | `SHT_PROGBITS`, `SHF_ALLOC`, `SHF_EXECINSTR`; nonempty bytes at `sh_addr`, any section name. | -Before distributing a build using ELFIO, verify that the source archive and every platform's binary package contain -its license and attribution. This is a condition of the later ELFIO integration; it adds no dependency to the minimal -binary-image milestone. +Validate bounds, address overflow, available data and conflicting overlaps before copying into owned regions. +Reject unsupported compression, relocation and sectionless ELF; leave BSS and holes unmapped. [E2], [E4], [H1], [H2] +For scatter-loaded RAM code, `sh_addr` is the execution address; a segment load address is insufficient. [E1], [E6] -### 4.2 Connecting images to OpenCSD +Direct ELF import candidate: ELFIO, header-only C++, MIT; pin the dependency and retain its notices. [E3], [L1] +Copy bytes before releasing ELFIO storage. HEX and ELF feed the same provider; no HEX conversion is required. [E5] -OpenCSD offers buffers, binary files, file regions, and callback accessors. File-region access maps a file offset -to an execution start address and length; OpenCSD does not interpret the ELF container for the client. [O2][O4] +## 3. Integration boundaries -For the minimal single-CPU/static-image milestone, a binary file/region accessor is sufficient. -For the later multi-source design, prefer an image provider behind the Trace-ID-aware callback: +### 3.1 Source configuration -```text -DecodeTree::addCallbackIDMemAcc(...) - -> read(address, memory_space, trace_id, requested_length) - -> resolve input/source -> processor -> matching image region - -> return available code bytes and actual byte count +| YAML | OpenCSD mapping | +| --- | --- | +| `core: Cortex-M4` | `OCSD_BUILTIN_DCD_ETMV3`, `ARCH_V7`, `profile_CortexM`. | +| `core: Cortex-M7` | `OCSD_BUILTIN_DCD_ETMV4I`, `ARCH_V7`, `profile_CortexM`. | +| `stream` | ETMv3 `reg_trc_id` / ETMv4 `reg_traceidr`. | +| `ETMIDR`, `ETMCCER`, `ETMCR` | `reg_idr`, `reg_ccer`, `reg_ctrl`. | +| `TRCIDR0`, `TRCIDR1`, `TRCIDR2`, `TRCIDR8` | `reg_idr0`, `reg_idr1`, `reg_idr2`, `reg_idr8`. | +| `TRCCONFIGR` | `reg_configr`. | + +- Make `core_prof` configurable; default to `Cortex-M` (`profile_CortexM`). + Its configuration interface is not part of the Experimental specification yet. +- Pending specification update: obtain `TRCCONFIGR` from the pyOCD "write registers" block, + not the "ID regs" block. Keep the Experimental specification unchanged for now. +- `decoder.regs` is metadata; existing reference/disable `regs` retain hardware-write semantics. +- Fixed values belong to the qualified ETM implementation/revision, not every instance of a Cortex-M name. + Offline decoding needs resolved capture-time configuration; pyOCD supplies no static replacement. +- Initialize unused OpenCSD ETMv4 IDR9–13 fields deterministically; they are not measured registers. [O10], [O14] +- Bind `stream -> reference -> pname -> code regions`; preserve DWT symbol-file provenance separately. [C4], [C13] + +M4 source example; the corresponding image entry, when present, uses `pname: CM4`: + +```yaml +- ref: CM4/instructions + pname: CM4 + type: etm + stream: 0x22 + decoder: + core: Cortex-M4 + regs: + - name: ETMIDR + value: 0x4114f250 + - name: ETMCCER + value: 0x18541800 + regsToRead: + - name: ETMCR + value: 0x00000810 # Filled by pyOCD ``` -The equivalent C API is `ocsd_dt_add_callback_trcid_mem_acc`. Return a short read or zero for inaccessible bytes; -do not supply fabricated opcodes. Keep backing storage alive for the decode session. - -The built-in mapper is global: it does not select images by CPU/Trace ID and rejects conflicting mappings in the -same memory space. Distinct cores can execute different images at identical VMAs. Therefore a single collection -of overlapping global file accessors is insufficient; ctrace must resolve source/security-aware mappings itself. -Track context changes for future context-dependent mappings. The callback alone does not provide a full process -or overlay history. [O2][O4] - -OpenCSD's memory cache keys reads by Trace ID and address range, which separates static images on different cores. -Memory space, context ID and image generation are not cache keys; a cache hit bypasses the image callback. -ETMv4 invalidates on an updated context packet, but equivalent behavior must not be assumed for every protocol. -Dynamic mappings or code changes at the same source/address need explicit invalidation or disabled caching. -Initially reject unsupported time-dependent mappings; test Secure/Non-secure aliases and context transitions. -The APIs include `InvalidateMemAccCache(traceID)` and `DecodeTree::setMemAccCacheing(...)`. [O4][O19][O23][O24][O25] +### 3.2 Decoder and packet monitoring -### 4.3 Is another instruction codec needed? - -OpenCSD attaches its built-in `TrcIDecode` to full decoders. Its `IInstrDecode` contract supplies the instruction -length and flow-relevant properties, including branch classification, direct target, and conditional/link/return -information. It supports A32, T32, and A64 internally; the initial Cortex-M scope is T32/Thumb. [O5] - -Keep three responsibilities distinct: - -| Responsibility | Required initially? | Provider | +| Boundary | Responsibility | +| --- | --- | +| CLI / output planning | ETM selectors/mode, fixed schema/name per input; independent CSV/CTF eligibility. | +| YAML / source routing | Neutral configuration fields and source/image binding; no OpenCSD types in YAML models. | +| `FileDecodeJob` / image readers | Owned code regions and Trace-ID-aware memory access. | +| `DecodePipeline` / ETM session | One tree per file, independent ETMv3/ETMv4/ITM sources and diagnostics. | +| ETM adapters / CSV | Structured packets, generic events, range expansion and deterministic fallback merge. | + +- Use `OCSD_TRC_SRC_FRAME_FORMATTED` / `OCSD_DFRMTR_FRAME_MEM_ALIGN` for TB input. [O2] +- `packets` and full-source fallback: `OCSD_CREATE_FLG_PACKET_PROC` plus `OCSD_C_API_CB_PKT_MON`. + MON provides packet bytes and length; `OCSD_C_API_CB_PKT_SINK` alone does not. Add SINK only for flow control. [O30] +- Image-backed `instructions` / proposed `decoded`: `OCSD_CREATE_FLG_FULL_DECODER`, memory accessor before data, + `ocsd_dt_set_gen_elem_outfn` and MON. Copy callback-owned fields/bytes immediately. [O26] +- Use `ocsd_dt_add_callback_trcid_mem_acc` for processor-local mappings; return actual available bytes. + A missing accessor is fatal; an unmapped read returns short/zero and may emit `ADDR_NACC`. [O26], [O27] +- Provider/storage outlive the tree. The cache keys Trace ID + address; dynamic code/security mappings need separate + qualification and invalidation, not just a different callback response. [O19], [O23], [O24], [O25] +- Route identity includes source/ATB ID; `pname` alone is not a route key. [C16] + +**Mixed-output merge:** + +- Retain packets and generic elements per source, with packet order, protocol position and local element order. + Association is many-to-many; decoder positions are not physical RAW offsets. +- Finalize after decoder EOT; spool if necessary to bound memory. Preserve generic-element order and complete + valid PC prefixes. Let the full decoder resolve commit, cancel, mispredict and discard before PC export. +- `ADDR_NACC` opens an image-gap interval at its originating packet. Keep that complete packet and subsequent + packets through the interval; stop before the next successfully reconstructed packet origin. + If that origin also contains a gap, retain its valid PCs and its packet once. +- For one origin: generic rows/expanded PCs first, then its fallback packet. Set `partial=1` when it also produced PCs. + Missing generic output alone does not establish an image gap; canceled/unresolved atoms are not PC rows. +- Only generic output drives instruction/context classification. MON additionally preserves source fields omitted + by generic events, including ETMv3 cancellation; unresolved EOT state remains explicit. + +### 3.3 Cortex-M exception normalization + +`index` uses architectural exception/vector numbers, not protocol encodings or an IPSR sample. +Reset is vector 1; exception-return/tail-chain markers alone do not identify the destination exception. +Read raw structured fields; OpenCSD passes them unchanged into generic `exception_number`. [O25], [O31] + +| Protocol | Raw code | Architectural number | | --- | --- | --- | -| Read executable bytes at execution addresses | Yes for full path reconstruction | New ctrace image reader/provider | -| Decode flow-relevant instruction properties | Yes | Existing OpenCSD instruction decoder | -| Format assembly text | Optional | Later, separately evaluated disassembler | -| Resolve symbols/source/inline frames | Optional | Later ELF symbol / DWARF layer | - -An optional per-instruction view must walk code using correct 16-/32-bit Thumb instruction lengths. It must also -preserve execution decisions and incomplete-path states; formatting a range as assembly does not establish that -every instruction printed was executed. OpenCSD's instruction-decoder interface can be evaluated for this walk -before adding a dependency. Its missing architecture features remain an upstream integration issue. - -## 5. What OpenCSD outputs - -### 5.1 Generic elements, not an assembly listing - -Full decode calls `ITrcGenElemIn::TraceElemIn` with an input index, source Trace ID, and a -`const OcsdTraceElement&`, derived from `ocsd_generic_trace_elem`. The C API exposes the generic C structure. -The principal instruction result is `OCSD_GEN_TRC_ELEM_INSTR_RANGE`. [O3][O16] +| ETMv3 | `0x01..0x07` | `raw + 16` (IRQ1..7). | +| ETMv3 | `0x08` | 16 (IRQ0). | +| ETMv3 | `0x09`, `0x0a`, `0x0d` | 6 (UsageFault), 2 (NMI), 4 (MemManage). | +| ETMv3 | `0x0b`, `0x0c`, `0x0e`, `0x0f` | Unchanged. | +| ETMv3 | `0x11`, `0x13`, `0x15` | 1 (reset), 3 (HardFault), 5 (BusFault). | +| ETMv3 | `0x18..0x1ff` | Unchanged (IRQ8..495). | +| ETMv4 M | `0x01..0x06`, `0x0b`, `0x0c`, `0x0e`, `0x0f` | Unchanged. | +| ETMv4 M | `0x10..0x17` | Unchanged (IRQ0..7). | +| ETMv4 M | `0x208..0x3ef` | `raw - 0x1f0` (IRQ8..495). | +| ETMv4 M | `0x18`, `0x19`, `0x1a` | Empty: debug halt, lazy FP push, lockup. | + +ETMv3 code zero means no exception. Other reserved/implementation-defined codes remain unnormalized; +SecureFault is outside M4/M7. Preserve raw code and event kind. Sources: IHI0014Q Table 7-11, pp. 319–320; +IHI0064H.a Table 6-13, pp. 268–269; DDI0403E.e pp. 517, 525. [A11], [A10], [A12] + +## 4. Instruction execution classification + +- Verify range instruction count, exclusive end and final size against code bytes. M4/M7 use mixed-width Thumb; + neither PC bit zero nor range byte length establishes instruction count. +- `execution` describes the predicate, not completion. Canceled/interrupted instruction facts remain separate events; + no invented completed PC for a partially resumed instruction. +- Track IT state: ordinary taken branches leave an IT block; Armv7-M exception entry starts with ITSTATE=0. + Exception return restores unknown IT state until evidence establishes it. Repeated unchanged context preserves state. + Debug halt, lazy FP push and lockup are not handler-entry indications and do not by themselves clear IT state. +- Omitted Context-ID/VMID fields retain knowledge, including unknown. Gaps/unexplained context changes invalidate IT state. +- ETMv3 E/N normally reports passed/failed predicates; apply instruction-specific exceptions such as CBZ/CBNZ first. + Use resolved endpoints after cancellation. ETMv4 interpretation also depends on enabled features. [A11], [A10] +- `last_executed` describes only the range endpoint. An N atom may mean branch not taken; it does not always mean an + unexecuted instruction. A non-branch endpoint with `last_executed=1` alone does not prove an IT predicate. [O3], [A12] +- Use structured context and protocol fields. The prototype's display-string parser is not the production interface. + +| `execution_basis` | Meaning | +| --- | --- | +| `unpredicated_path` | Known outside IT predication. | +| `conditional_branch_predicate` | Conditional branch result establishes its predicate. | +| `etmv3_atom_predicate` | Resolved E/N predicate after instruction-specific exceptions, including CBZ/CBNZ. | +| `IT_state_unavailable` | IT state cannot be reconstructed. | +| `IT_predicate_not_traced` | IT block known, predicate result absent from trace. | +| `CBZ_CBNZ_comparison` | Comparison executes even when the branch is not taken. | +| `IT_instruction` | IT instruction establishes the following block. | +| `branch_taken` | Observed taken branch establishes that instruction's predicate. | +| `IT_branch_predicate_failed` | Untaken branch in a known IT block has a failed predicate. | +| `missing_branch_outcome` | Branch outcome is unavailable. | +| `branch_predicate_context_unavailable` | Untaken branch without sufficient predicate context. | +| `nonbranch_last_flag_unqualified` | Endpoint flag alone cannot establish a non-branch predicate. | +| `predicate_analysis_profile_unqualified` | No qualified classifier for this profile/ISA. | + +Do not infer missing predicate results from later register snapshots. +ETMv3 PC expansion/classification still needs independent qualification. + +## 5. OpenCSD restrictions and qualification risks + +Details and reproducible defects: [Known OpenCSD issues](opencsd-issues.md). + +| Area | Pinned-version result / consequence | +| --- | --- | +| ETMv3 cancellation | Packed EE followed by cancellation can retain a canceled PC; decoder correction required. | +| ETMv3 IRQ8 field | `cm_irq_n` is 24 instead of 8; normalize from the raw exception code. | +| ETMv3 truncated branch at EOT | Reported as a valid branch with stale fields; truncation handling needs correction. | +| ETMv3 exception metadata | Generic `EXCEPTION` omits the raw Cancel bit; preserve it through packet monitoring. | +| ETMv4 reset address | Generic return-address flag does not establish architectural validity for reset. | +| ETMv4 conditional non-branch trace | Parser recognizes it; full decoder rejects configuration with `HW_CFG_UNSUPP`. | +| Armv8.1-M branch-forward | Not qualified here; fixtures do not establish complete support. | +| Cortex-M23 / ETMv3 | Decoder checks `ARCH_V7`, core map selects `ARCH_V8`; qualification required. | + +- ETMv3 support remains required; correct cancellation and EOT handling before qualifying its outputs. + The CM4 hardware capture contains no cancellation packet and does not cover that defect. +- The M7 reference has conditional non-branch tracing disabled; absent IT predicate results remain `unknown`. + Enabling it requires a decoder that supports the configuration; changing archived metadata cannot recover results. +- Additional cores, security/dynamic mappings and protocol revisions need separate qualification. Unsupported features + must remain explicit; packet-parser support is not full instruction-decoder support. [O1], [O8], [O13], [O25], [O28] + +## 6. Reference evidence + +Companion workspace: `/Users/thode01/work/github/TraceBufferExamples`; captures retain their original schemas. + +| Reference | Evidence | Remaining limit | +| --- | --- | --- | +| [STM32H7B3I-DK / M7][ref-m7] | 4-KiB ETF, ATB `0x12`; packets, ranges and PC validation. | ELF/HEX parity. | +| [STM32H747I-DISCO / M4][ref-m4] | 4-KiB ETF, ATB `0x22`; packets and ranges. | PC/classifier qualification. | +| [ETMv3 cancellation][ref-cancel] | Synthetic packed/split atom comparison. | Host reproduction. | +| [ETMv3 packet fields][ref-fields] | IRQ8 field and truncated-branch cases. | Host reproduction. | -Important fields are: +- Hardware bundles contain matching RAW, images, decoder metadata and validation reports; both capture windows wrap. +- M7 PC evidence: `validation/pc-expansion/README.md`; hardware/project configuration: each board's `TRACE_SETUP.md`. +- Shared host helpers: `STM32H7B3I-DK/tools/`. They establish reference evidence, not production ctrace ETM support. -- Start address and exclusive end address: `[st_addr, en_addr)`. -- ISA and `num_instr_range`; mixed Thumb instruction widths prevent deriving count from byte length alone. -- Last instruction size, type/subtype, conditional flag, and `last_instr_exec`. -- Optional cycle-count information, indicated by its validity flag. -- Source and position supplied by the callback; processor/security/context maintained from related elements. +## Appendix A. Trace representations -The upstream M7 fixture contains this generic-element text output: [O22] +| Representation | Meaning | +| --- | --- | +| Architectural elements | ETMv4 P0: atoms, exceptions, Q, M-profile returns; other elements: state/resolution. | +| Protocol packets | Encodings of elements, potentially several per packet; Appendix B. | +| OpenCSD generic elements | Decoded ranges, events and status; mapped in the CSV specification. | +| Instruction rows | Image-backed expansion of complete generic ranges, with qualified predication. | -```text -Idx:21; ID:6; OCSD_GEN_TRC_ELEM_INSTR_RANGE(exec range=0x10000140:[0x1000014a] num_i(3) last_sz(2) (ISA=T32) E --- ) -Idx:21; ID:6; OCSD_GEN_TRC_ELEM_EXCEPTION(pref ret addr:0x1000014a; excep num (0x0b) ) -Idx:25; ID:6; OCSD_GEN_TRC_ELEM_TIMESTAMP( [ TS=0x000000000060]; ) -``` +ETMv4 element groups: IHI0064H.a Figure 2-3, p. 34. [A10] +One protocol packet is not one range or one instruction; packet names do not imply instruction execution. -This is checked-in OpenCSD test output, not a new ctrace execution or a proposed ctrace file syntax. -It represents three Thumb instructions in `[0x10000140, 0x1000014a)`, an exception and a time anchor for source 6. -Range end is the exclusive boundary of the represented code, not a universal branch-destination field. Keep -subsequent addresses and flow metadata rather than inventing a target from `en_addr`. +## Appendix B. CSV types and packet names -### 5.2 Information that must survive normalization +### B.1 Trace types -This is the full target semantics. The minimal milestone retains instruction ranges and the state/diagnostics -needed for its declared scope; richer context/time records follow later. Unsupported transitions must remain visible. +| Family or existing output | CSV `type` | Status | +| --- | --- | --- | +| ITM / DWT | `itm`, `dwt` | Existing software/data trace. | +| DWT events and PMU | `event`, `pmu` | Existing event categories. | +| Exceptions and PC samples | `exception`, `pcsample` | Existing records; not ETM instruction reconstruction. | +| Timing and diagnostics | `global_ts`, `overflow`, `error`, `info` | Existing output categories. | +| ETMv3 | `ETMv3` | Required; Cortex-M4/ETMv3.5 hardware reference. | +| ETMv4 | `ETMv4` | Required; Cortex-M7/ETMv4.0 hardware reference. | +| PTM / ETE | `ptm`, `ete` | Possible later instruction protocols; OpenCSD decoders exist. | +| STM | `stm` | Possible later software trace; OpenCSD decoder exists. | -| Generic element / property | Required interpretation | -| --- | --- | -| `INSTR_RANGE`, last-instruction flags | Preserve range/count and E/N execution decisions. | -| `I_RANGE_NOPATH` | Incomplete path; no complete instruction list or exact coverage. | -| `PE_CONTEXT` | Update ISA/security/context validity; keep unknown fields unknown. | -| `EXCEPTION`, `EXCEPTION_RET` | Preserve addresses, number validity, return and tail-chain information. | -| `TRACE_ON`, `NO_SYNC`, `EO_TRACE` | Preserve start/restart/end and reasons, including gaps and overflow. | -| `ADDR_NACC`, `ADDR_UNKNOWN`, `UNKNOWN` | Expose missing code/address/invalid-decode state as an observable result. | -| `TIMESTAMP`, `CYCLE_COUNT`, `SYNC_MARKER` | Preserve time, association, units and validity. | +Existing source/event categories retain their meanings. `info` marks skipped input bytes and is not selectable. +[C10], [C11] DWT travels through ITM, not a separate protocol decoder. Minor ETM revisions remain configuration, +not distinct `type` values. Profile qualification and decoder restrictions are in section 5. [O1], [O6] -The last instruction of a range can be conditional and unexecuted. `num_instr_range` counts represented -instructions; do not blindly treat it as the number executed without applying the execution flags. [O3][O17] +### B.2 Packet inventory -The `ADDR_NACC` condition can leave the decoder waiting for a subsequent trace address before it can reconstruct -flow again. Retain partial output and the gap. Decoder success alone does not prove image identity or complete -instruction coverage. Input positions from a formatted stream must keep the existing distinction between exact -offsets and formatter hints. [O3][C3] +Packet classes below include formats beyond the required M4/M7 profiles. +Names use lowercase enum suffixes, normalizing `A_SYNC` to `async`, `CTXT` to `context`, `EXCEPT` to `exception`, +and `EXCEPT_RTN` to `exception_return`. Format/address variants stay distinct. +Availability depends on revision, capabilities and capture configuration; not every core emits every listed packet. -## 6. Time and downstream processing +Sources: IHI0014Q Table 7-1, pp. 301–302 [A11], and IHI0064H.a Table 6-1, pp. 239–241 [A10], checked against +OpenCSD enums/processors. [O10], [O12], [O14], [O28] Appendix A separates architectural and decoded elements. -Instruction order, covered instruction count, CPU cycle count, and timestamp are different quantities. -Timestamp elements provide source-specific trace-time anchors. Preserve protocol-specific association and any -`SYNC_MARKER` locating an anchor earlier in the stream. They do not assign an exact time to every instruction. -ETMv3/PTM can attach cycle counts to ranges or other elements. ETMv4 has explicit cycle-count elements, and a -cycle count accompanying an ETMv4 timestamp has special association semantics and must not simply be added to -the cumulative count. [O3] +#### Shared ETMv3 / ETMv4 packet names -The semantic model should preserve sequence order, raw time anchors and optional cycle information with an explicit -time-domain identity and quality. Do not feed ETM through the existing ITM local-timestamp accumulator/prescaler. -Do not distribute cycles equally across instructions, treat instruction count as time, or infer cross-source -ITM/ETM clock correlation merely because both sources belong to one processor. +| `packet` | Meaning / qualification | +| --- | --- | +| `async` | Alignment synchronization (A-Sync). | +| `timestamp` | Transmitted timestamp; ETMv3.5+ and configured timestamp support. | +| `ignore` | Ignored padding packet; ETMv4.3+ for the ETMv4 instruction stream. | -Suggested processing layers: +Shared names describe roles; encoding and validity remain protocol-specific. Other shared concepts use different +classes: atoms are ETMv3 `p_hdr` versus ETMv4 `atom_f1`–`atom_f6`; context uses ETMv3 `i_sync`/`context_id`/`vmid` +versus ETMv4 `context`/address-with-context. The lists separate encodings, not exclusive capabilities. -1. **Range preservation:** source, addresses, ISA, counts, flow flags, context, status and timing evidence. -2. **Optional instruction expansion:** read exact bytes, advance by decoded instruction size, retain E/N and gaps. -3. **Optional presentation:** disassembly, symbols and source lines. -4. **Optional analysis:** coverage, branch statistics, call/return analysis and profiling. +#### ETMv3-only packet classes -Call-graph construction additionally needs exception/tail-chain, indirect branch, synchronization-loss and return -handling. Trace-buffer wrap or trace filtering can remove the matching entry of a call. Report incomplete stacks -instead of presenting an inferred complete call tree. Coverage needs a definition of executed versus merely covered -or skipped conditional instructions. Time profiling requires qualified time data. +| `packet` | Meaning / qualification | +| --- | --- | +| `branch_address` | Branch address; can also carry exception information. | +| `i_sync`, `i_sync_cycle` | Instruction synchronization, without/with cycle count. | +| `p_hdr` | E/N atoms; retain P-header format 1–4 in the packet fields. | +| `cycle_count` | Standalone cycle count. | +| `trigger` | Trigger indication. | +| `context_id`, `vmid` | Context-ID / VMID update; VMID is an ETMv3.5 feature. | +| `exception_exit` | Exception-return marker. | -## 7. Required ctrace extensions +Instruction-only exception entry uses branch-address fields. Trace enable/overflow are I-Sync reasons; +there are no standalone ETMv3 `trace_on`/`overflow` packets. [A11], [O12] -This section maps the complete target design. Section 1.1 defines the first implementation milestone. +#### ETMv4-only instruction-stream packet classes represented in OpenCSD -### 7.1 Reusable boundaries and missing behavior +| `packet` | Meaning / qualification | +| --- | --- | +| `trace_info`, `trace_on` | Decoder synchronization state / trace becoming active. | +| `discard`, `overflow` | Discarded speculative state / trace overflow. | +| `exception`, `exception_return` | Exception and exception-return indication. | +| `ccnt_f1`, `ccnt_f2`, `ccnt_f3` | Cycle-count formats 1–3. | +| `commit`, `mispredict` | Speculation resolution. | +| `cancel_f1`, `cancel_f1_mispred`, `cancel_f2`, `cancel_f3` | Cancellation formats; F1 can include misprediction. | +| `cond_i_f1`, `cond_i_f2`, `cond_i_f3` | Conditional-instruction formats 1–3. | +| `cond_res_f1`, `cond_res_f2`, `cond_res_f3`, `cond_res_f4` | Conditional-result formats 1–4. | +| `cond_flush` | Conditional trace flush. | +| `event` | Trace event indication. | +| `context` | Processor context, including implemented/transmitted Context-ID and VMID fields. | +| `addr_ctxt_l_32is0`, `addr_ctxt_l_32is1` | Long 32-bit address with context, instruction-set encoding IS0/IS1. | +| `addr_ctxt_l_64is0`, `addr_ctxt_l_64is1` | Long 64-bit address with context, IS0/IS1. | +| `addr_match` | Exact match with a retained address. | +| `addr_s_is0`, `addr_s_is1` | Short address, IS0/IS1. | +| `addr_l_32is0`, `addr_l_32is1` | Long 32-bit address, IS0/IS1. | +| `addr_l_64is0`, `addr_l_64is1` | Long 64-bit address, IS0/IS1. | +| `q` | Q packet; preserve its subtype and valid fields. | +| `atom_f1`, `atom_f2`, `atom_f3` | E/N atom formats 1–3. | +| `atom_f4`, `atom_f5`, `atom_f6` | E/N atom formats 4–6. | +| `ts_marker` | Timestamp marker; ETMv4.6+ with the feature implemented. | + +Enum membership does not establish hardware capability or full-decoder support (section 5). +IS0/IS1 are protocol instruction-set encodings whose meaning depends on processor state. [O28], [A10] + +#### Additional parsed ETMv4 classes outside the initial M4/M7 instruction-only profile + +| `packet` | Scope | +| --- | --- | +| `func_ret` | Function-return packet; additional profile/revision qualification required. | +| `num_ds_mkr`, `unnum_ds_mkr` | Numbered/unnumbered data-synchronization markers; data-trace association. | -| Area | Existing implementation | Required extension | -| --- | --- | --- | -| Discovery / file jobs | TB discovery, bounded reads | Reuse; specify protocol and buffer normalization. | -| Trace-run reader | ETM references discarded | Parse source instances, decoder state and image metadata. | -| Routing | One ITM binding per processor | Allow ITM and ETM sources for the same processor. | -| OpenCSD lifecycle | Named decoder plus `CSConfig` | Reuse; add ETM configs, memory callback and dispatch. | -| Adapter | Only `ITMTRACE` elements consumed | Add instruction generic-element normalization. | -| Semantic pipeline | ITM/DWT-specific | Dispatch by protocol; separate time/state logic. | -| Events / filters | ITM/DWT, sampled PC, `tcyc` | Add ranges, context, gaps and timing evidence. | -| CSV | Eight fixed columns | Minimal `instr_range` mapping; richer schema later if needed. | -| CTF | ITM/DWT profile, clock requirements | Later: instruction/context events and time policy. | -| Diagnostics / tests | Catalog, RAW context, partial results | Add source/image/path diagnostics and fixtures. | - -See [C1] through [C7] for the implementation boundaries. A sampled `pcsample` event is not an instruction path and -must not be repurposed to encode ranges. - -### 7.2 Proposed pipeline - -This is the broader target architecture. The minimal path uses one binary image, one ETM source and CSV as in -section 1.1; ELF/AXF, mixed sources, analysis and instruction CTF are later additions. - -```mermaid -flowchart TD - Capture[TB raw bytes and capture metadata] --> Input[Existing input job and framing] - Images[ELF AXF bin and runtime mappings] --> Memory[Source-aware code image provider] - Input --> Tree[OpenCSD tree and typed source routes] - Memory --> Tree - Tree --> ITM[Existing ITM and DWT adapter] - Tree --> Instr[Instruction generic-element adapter] - ITM --> Events[Backend-independent semantic events] - Instr --> Events - Events --> CSV[CSV output] - Events --> CTF[CTF output] - Events --> Diagnostics[Diagnostics and quality] - Instr --> Later[Optional expansion symbols and analysis] -``` +These classes have parser handlers; omission from the initial profile is not an OpenCSD parser defect. [O28] -Keep the existing single-tree lifecycle constraints and independent per-file state. Mixed ITM/ETM input must not -create competing decode trees or reuse an ITM-specific raw-packet monitor/recovery strategy for ETM. -Use protocol-specific synchronization rules; bounded input reads and formatter state can remain shared. [C3] +#### Additional ETMv4 architecture packets absent from the pinned OpenCSD processor -### 7.3 Initial CSV contract and later output options +| Proposed `packet` | Architecture / status | +| --- | --- | +| `branch_future_flush` | Added in ETMv4.5; branch-future state handling is not implemented here. | +| `resynchronization` | Added in ETMv4.5; no corresponding packet class/handler in this OpenCSD pin. | -Preserve instruction ranges as the primary semantic representation. Expanding every instruction by default can -greatly increase output volume and unnecessarily couples decode to presentation. +These architectural entries require decoder work; see IHI0064H Table 6-1 and the ETMv4.5 change summary, p. 536. +[A10], [O1], [O28] -For the first milestone, use the existing eight-column CSV with the new `instr_range` type and mapping in section 1.1. -Define how status/gap records and retained flow flags map to that output, and keep existing ITM/DWT semantics stable. -If later requirements outgrow this mapping, evaluate a separate instruction artifact or an explicitly versioned -common schema for richer context, timing and analysis data. These extensions require specification alignment; -they are not currently part of the published CMSIS contract. +### B.3 Parser status -For the later CTF milestone, define instruction/status/context records and schema/version compatibility. CTF requires -a valid clock frequency for selected streams and timestamps that do not regress. An ordered but untimed range stream -does not automatically meet that contract. Options include deferring CTF until timing is qualified, or defining an -explicit sequence-clock profile with documented viewer limitations. Never label a sequence counter as elapsed -nanoseconds. Existing ITM/DWT CTF and Trace Compass XML contracts must be preserved. [C6][C7] +- Map `NOTSYNC`, `INCOMPLETE_EOT`, `BAD_SEQUENCE`, `BAD_TRACEMODE`, `RESERVED`, `RESERVED_CFG` to + `notsync`, `incomplete_eot`, `bad_sequence`, `bad_tracemode`, `reserved`, `reserved_cfg` with `record=status`. + They are parser states, not on-wire packets. Preserve reason and available bytes. +- `NOERROR` / `NO_ERR_TYPE` are sentinels; `BRANCH_OR_BYPASS_EOT` and `EXTENSION` are internal dispatch states, + not exported packet classes. Extension headers resolve to concrete types. +- The pinned ETMv3 processor can report a truncated branch at EOT as a valid branch with stale fields; + see [Known OpenCSD issues](opencsd-issues.md#etmv3-unfinished-packet-emitted-without-eot-error-status). +- Decoder diagnostics retain existing `error`, `overflow`, `info` categories and source attribution when known. +- ETE-only source-address, transaction and PE-reset entries in the shared header are outside this proposal. -## 8. Implementation sequence and validation gates +## Sources -| Step | Deliverable | Completion evidence | -| --- | --- | --- | -| 0. Select reference | One protocol/CPU, configuration and matching bin | Reviewed chronological capture bundle. | -| 1. Minimal implementation | Five extensions in section 1.1; range CSV | Expected ranges/gaps; ITM/DWT compatibility. | -| 2. Image formats | ELF/AXF reader and richer mappings | Valid/invalid files; execution-address fixtures. | -| 3. Routing/context | ITM/ETM, multiple CPUs, richer context | Correct routing with overlapping VMAs. | -| 4. CTF/viewers | Qualified instruction/time profile | Independent consumer and viewer validation. | -| 5. Additional coverage | Further protocols; M23/Armv8.1-M qualification | Feature fixtures; upstream gaps addressed. | - -MTB is separate from step 5: it needs its own record-format/circular-buffer contract and decoder, though the image -provider, instruction-range model and output layers should be reusable. PTM/ETE are future scope unless explicitly -selected. Changing the OpenCSD dependency or fixing upstream feature gaps also requires a separate reviewed change. - -Apply validation as the corresponding capability is implemented; section 1.1 defines the minimal milestone's gate. -The complete validation plan should cover: - -- Reproducibly built small firmware with mixed-width Thumb, taken/not-taken branches, direct/indirect control flow, - calls/returns, exceptions, tail-chaining and, where relevant, Security transitions. -- Actual instruction-trace capture, decoder configuration, matching images and independent expected ranges. - Existing `TB-Trace` ITM captures are not evidence of ETM instruction decode. -- Upstream M7/M33 snapshots as integration references, with licensing/provenance reviewed before copying fixtures. -- Variable byte-chunk boundaries through formatter frames and protocol packets; identical semantic output. -- Capture starting mid-stream, buffer wrap, synchronization loss, overflow, incomplete tail and missing code bytes. -- Wrong image identity, corrupted ELF bounds, overlapping mappings, RAM execution and multiple core-local VMAs. -- Preservation of conditional execution, unknown-path ranges and partial results after unavailable memory. -- Timestamp/cycle-count association and explicit absence of timing; no invented cross-source correlation. -- M23 exception encoding and Armv8.1-M features as separate qualification gates rather than generic smoke tests. - -Upstream fixtures demonstrate relevant decoder paths; they are not a substitute for running ctrace against those -paths. This research did not build a new decoder or execute ETM qualification tests. - -## 9. Remaining decisions - -The single-source/bin-to-range-CSV milestone and its initial column mapping are agreed in section 1.1. -Before implementing it, settle: - -1. Which first real capture is available: M7 ETMv4, M33 ETMv4, or a qualified M3/M4 ETMv3 source? -2. Which fields carry its complete decoder configuration and static binary execution mapping in trace-run metadata? -3. Which observable CSV status/gap records and flow flags accompany `instr_range` for this capture? - -For later milestones, decide producer ownership of image identity/security/runtime mappings, the ELF reader, -whether richer CSV needs a new schema, the timing/CTF contract, additional processor qualification and desired -analyses. These decisions do not require an ELF loader, CTF or multi-CPU implementation in the first milestone. - -## 10. Evidence and references - -OpenCSD links below are pinned to the inspected revision. Hardware manuals were fetched and read through the -CMSIS documentation tools; page numbers above refer to PDF pages. Their product revisions matter. -The support matrix is bounded by this evidence, not a claim about every SoC integrating these processor cores. - -- [O1] OpenCSD README, support list and pending features. -- [O2] OpenCSD programming guide: tree, configuration, memory images, callbacks and data flow. -- [O3] Generic-element interpretation guide; [structure and fields][O17]. -- [O4] Memory accessor API; [callback types][O18] and [global mapper implementation][O19]. -- [O5] Instruction-decoder interface; [implementation][O20] and [automatic tree attachment][O21]. -- [O6] Decoder registry and protocol implementation directories. -- [O7] Upstream M7, M33 and Armv8.1-M snapshots. -- [O8] ETMv3 M-profile gate and core architecture map. -- [O9] Thumb low-overhead-loop instruction support. -- [O10] Protocol configuration structures. -- [O11] ETMv3 full-decoder mode validation. -- [O12] ETMv3 M-profile exception packet parsing. -- [O13] Processor-name to architecture/profile map. -- [O14] ETMv3 configuration structure. -- [O15] ETE configuration structure. -- [O16] C++ generic-element callback interface. -- [O22] Checked-in M7 generic-element output example. -- [O23] Memory-cache keys and context assumptions; [cache read implementation][O24]. -- [O25] ETMv4 context update and memory-cache invalidation. -- [A1] Arm Cortex-M resources: architecture and trace documentation map. -- [A2] Arm CoreSight ETM-M7 TRM, DDI0494D, r0p1. -- [A3] Arm CoreSight ETM-M23 TRM, DDI0563C, r0p1. -- [A4] Arm CoreSight ETM-M55 TRM, 101053_0101_03_en, r1p1. -- [A5] Arm CoreSight ETM-M85 TRM, 101926_0101_07_en, r1p1. -- [A6] Published CMSIS experimental trace configuration and output contract. -- [A7] Arm CoreSight ETM-M33 TRM, 100232_0002_06_en, r0p2, issue 06. -- [A8] Arm Cortex-M35P introduction, integrated debug and trace, PDF p.15. -- [A9] Arm China CoreSight ETM-M52 TRM, 102774_0002_05_en, r0p2, issue 05. -- [A10] Arm employee explanation of M0/M0+ trace and MTB. -- [E1] ELF program-header/loading contract. -- [E2] Arm ELF ABI. -- [E3] ELFIO upstream project and license. -- [L1] ELFIO MIT license, copyright and distribution condition. -- [L2] Apache Software Foundation third-party license policy, Category A including MIT. -- [L3] Apache License 2.0, redistribution and NOTICE requirements. -- [C1] ctrace architecture and input constraints. -- [C2] Current trace-run schema and [YAML reader](../src/tracerun/YmlTraceRunConfigReader.cpp). -- [C3] OpenCSD adapter, [tree lifecycle](../src/decode/OpenCsdTreeSession.cpp) - and [pipeline](../src/decode/DecodePipeline.cpp). -- [C4] Current route metadata and normalization. -- [C5] Semantic events and output selectors. -- [C6] CTF output requirements and [CSV row mapping](../src/output/csv/CsvRowMapper.cpp). -- [C7] Current CTF profile and deferred decoder work. -- [C8] Current trace-directory naming and CLI contract. -- [C9] File decode job and pipeline creation. -- [C10] Shared CTF event-payload visitor. +OpenCSD links identify the inspected pin; architecture editions/pages are stated at each use. [O1]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/README.md [O2]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/docs/prog_guide/prog_guide_main.md [O3]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/docs/prog_guide/prog_guide_generic_pkts.md -[O4]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/common/ocsd_dcd_tree.h -[O5]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/interfaces/trc_instr_decode_i.h [O6]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/ocsd_lib_dcd_register.cpp -[O7]: https://github.com/Linaro/OpenCSD/tree/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/tests/snapshots [O8]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv3/trc_cmp_cfg_etmv3.h -[O9]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/i_dec/trc_idec_arminst.cpp [O10]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv4/trc_pkt_types_etmv4.h -[O11]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp [O12]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_proc_etmv3_impl.cpp [O13]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/trc_core_arch_map.cpp [O14]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/etmv3/trc_pkt_types_etmv3.h -[O15]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/ete/trc_pkt_types_ete.h -[O16]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/interfaces/trc_gen_elem_in_i.h [O17]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/trc_gen_elem_types.h -[O18]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/ocsd_if_types.h [O19]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/mem_acc/trc_mem_acc_mapper.cpp -[O20]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/i_dec/trc_i_decode.cpp -[O21]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/ocsd_dcd_tree.cpp -[O22]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/tests/snapshots/v7m_svc_vector/pkt_proc_logs/v7m_svc_vector.ppl [O23]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/mem_acc/trc_mem_acc_cache.h [O24]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/mem_acc/trc_mem_acc_cache.cpp [O25]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp -[A1]: https://developer.arm.com/community/arm-community-blogs/b/architectures-and-processors-blog/posts/cortex-m-resources -[A2]: https://documentation-service.arm.com/static/5e906dfe8259fe2368e2ac54 -[A3]: https://documentation-service.arm.com/static/5e90936fc8052b16087620d0 -[A4]: https://documentation-service.arm.com/static/6622c0bbfabc8c11c7b53135 -[A5]: https://documentation-service.arm.com/static/662a6ea9c7622e1c31ae08ee -[A6]: https://open-cmsis-pack.github.io/cmsis-toolbox/Experimental-Features/ -[A7]: https://documentation-service.arm.com/static/66be2251882fec713ef49f7b -[A8]: https://armkeil.blob.core.windows.net/developer/Files/pdf/white-paper/introduction-to-cortex-m35p.pdf -[A9]: https://documentation-service.arm.com/static/65849e9cb52744113be61b26 -[A10]: https://community.arm.com/forums/f/architectures-and-processors-forum/8383/trace-debug-functionality-in-arm-cortex-m0/28550 +[O26]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/c_api/opencsd_c_api.h +[O27]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/common/trc_pkt_decode_base.h +[O28]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv4/trc_pkt_proc_etmv4i.cpp + +[A10]: https://documentation-service.arm.com/static/60017fbb3f22832ff1d6872b +[A11]: https://documentation-service.arm.com/static/5f90158b4966cd7c95fd5b5e +[A12]: https://documentation-service.arm.com/static/606dc36485368c4c2b1bf62f + [E1]: https://gabi.xinuos.com/elf/07-pheader.html [E2]: https://github.com/ARM-software/abi-aa/blob/main/aaelf32/aaelf32.rst [E3]: https://github.com/serge1/ELFIO +[E4]: https://gabi.xinuos.com/elf/03-sheader.html +[E5]: https://github.com/serge1/ELFIO/blob/main/elfio/elfio_section.hpp +[E6]: https://developer.arm.com/documentation/100070/0612 + +[H1]: https://www.keil.com/support/docs/1584/default.htm +[H2]: https://onlinedocs.microchip.com/oxy/GUID-C3F66E96-7CDD-47A0-9AB7-9068BADB46C0-en-US-4/GUID-DF9E479D-6BA8-49E3-A2A5-997BBA49D34D.html + [L1]: https://github.com/serge1/ELFIO/blob/main/LICENSE.txt -[L2]: https://www.apache.org/legal/resolved.html#category-a -[L3]: https://www.apache.org/licenses/LICENSE-2.0 -[C1]: architecture.md -[C2]: ../src/tracerun/TraceRunConfig.h -[C3]: ../src/decode/OpenCsdPacketCollector.cpp + [C4]: ../src/tracerun/CtraceRunMeta.cpp -[C5]: ../src/model/TraceEvent.h -[C6]: ../src/output/OutputRequirements.cpp -[C7]: ctf-format.md -[C8]: ../README.md -[C9]: ../src/control/FileDecodeJob.cpp -[C10]: ../src/output/ctf/CtfEncoder.cpp +[C10]: ../src/model/TraceSelection.h +[C11]: ../src/output/csv/CsvRowMapper.cpp +[C13]: ../src/tracerun/YmlTraceRunConfigReader.cpp +[C16]: ../src/tracerun/CtraceRunMeta.h + +[O30]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/c_api/ocsd_c_api_types.h +[O31]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp + +[ref-m7]: ../../../../TraceBufferExamples/ReferenceTraces/STM32H7B3I-DK-ETMv4/README.md +[ref-m4]: ../../../../TraceBufferExamples/ReferenceTraces/STM32H747I-DISCO-CM4-ETMv3/README.md +[ref-cancel]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-Cancellation/README.md +[ref-fields]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-PacketFields/README.md diff --git a/tools/ctrace/docs/opencsd-issues.md b/tools/ctrace/docs/opencsd-issues.md index 2f4857558..27c085eb3 100644 --- a/tools/ctrace/docs/opencsd-issues.md +++ b/tools/ctrace/docs/opencsd-issues.md @@ -43,3 +43,114 @@ the patch and pass afterward. The patch is applied idempotently without resetting the submodule or discarding unrelated local changes. Once OpenCSD releases the fix, update the pinned submodule revision and remove the local patch. + +## ETMv3 canceling exceptions after packed atoms + +OpenCSD **1.8.3**, pinned at `59f15320637ce6e510604e7bed550d7ad109d15d`, +can emit the final instruction of a P-header even when the following exception +cancels it. A synthetic Cortex-M4 instruction-only trace reproduced the defect +with a freshly built host adapter on 2026-11-06: + +| Atoms before the same canceling SysTick exception | Expected PCs | Observed PCs | +| --- | --- | --- | +| `88`: EE | `0x1000` | `0x1000`, **`0x1002`** | +| `84 66 84`: E, IGNORE, E | `0x1000` | `0x1000` | + +Both decoder runs return success without diagnostics. The packet processor +correctly reports `Cancel prev instr` in both cases. [Arm IHI0014Q][etmv3-spec], section 7.3.1, +p.303 permits equivalent P-header encodings; pp.323 and 338 specify cancellation +for interrupted or continued M-profile instructions during instruction-only trace. + +The [pending-list guard][etmv3-pending] uses `numPend >= getNumElem()`. +When multiple ranges are queued, `pendLastNElem(1)` therefore retains none. +The [ETMv3 P-header decoder][etmv3-retain] calls this method to protect the last +range, but the later [exception cancellation][etmv3-cancel] can remove only +pending elements. The canceled PC has already been exported. + +Durable [evidence and reproduction][cancel-evidence] are in the separate TraceBufferExamples +workspace: README, reproduction script, +text fixtures, packet/element output, results and provenance. No executable or +build directory is archived. The existing CM4 hardware reference contains no +canceling packets, so its successful decode does not validate this case. + +No ETMv3 fix is applied. Correct cancellation and packing-invariant output are +required before qualifying the default ETMv3 instruction output. This host test +covers the OpenCSD full decoder, not the future production ctrace classifier. + +## ETMv3 IRQ8 structured packet field + +In the same OpenCSD 1.8.3 pin, an M-profile exception with raw code `0x18` +sets `exception.bits.cm_irq_n` to **24**, although the correct IRQ number is 8. +The [parser conversion][etmv3-irq] subtracts `0x10` only when the raw value is +strictly greater than `0x18`. IRQ7 and IRQ9 controls produce their correct values. +The raw exception code remains correct, and the [independent text formatter][etmv3-irq-text] +correctly prints `Exception=IRQ8`; this is a structured-field defect, not a text +formatting error. IHI0014Q Table 7-11, pp.319–320 defines the raw-code mapping. + +The minimal packet `95 49 90 01`, following A-Sync and I-Sync, reproduces the +error through `OCSD_C_API_CB_PKT_MON`. Consumers must normalize the raw exception +code using its protocol-specific mapping rather than trust this derived field +until it is corrected and qualified. + +## ETMv3 unfinished packet emitted without EOT error status + +The same pin's [ETMv3 EOT handler][etmv3-eot] emits pending bytes without changing +the packet type to `ETM3_PKT_INCOMPLETE_EOT`. A trace ending with only `95` after +A-Sync and I-Sync at `0x1000` produces a normal `BRANCH_ADDRESS` packet with +`err_type=NOERROR`; its display retains the previous address `0x1000`. +IHI0014Q Table 7-8, p.314 requires another address byte when bit 7 is set, so +this packet is incomplete. The probe nevertheless returns success. + +The captured byte is preserved in the monitor callback; the defect is missing +truncation status and apparently valid stale packet fields, not lost input bytes. +Packet and mixed-fallback output must retain the byte as incomplete and suppress +unverified address fields. The design already identified this qualification risk; +the synthetic case now confirms it without a source patch. + +Both packet-field cases, including IRQ7/IRQ9 controls, have [evidence and reproduction][field-evidence] +in the separate TraceBufferExamples workspace, with the C++ probe, reproduction script, fixtures, callback output, +results and provenance. No compiled products are archived. + +## ETMv3 cancellation metadata in generic output + +This is an API information limitation, separate from the incorrect PC above. +Even when cancellation correctly removes the last range, +[ETMv3 generic EXCEPTION construction][etmv3-exception] preserves the exception +number but omits the cancellation reason. An instruction-output adapter that +promises to explain incomplete paths must retain this information through its +packet monitor, including when all code bytes are available. + +A nonzero `Resume` field is not required for the initial instruction-only profile: +IHI0014Q p.323 specifies that exception information byte 2 is not emitted in this +mode. Preserve the actual cancellation indication without inventing complete +execution of a paused instruction. + +## ETMv4 reset exception address validity + +The pinned [ETMv4 exception decoder][etmv4-exception] sets `excep_ret_addr=1` +for non-tail-chained exception elements, including PE reset. This flag alone is +insufficient to establish an architectural return address: [IHI0064H.a][etmv4-spec], +p.270 defines the address accompanying a PE Reset exception element as UNKNOWN. +The transmitted address still updates protocol address registers; that does not +make it a valid execution or return address. + +Consumers must suppress the generic reset return-address claim and retain any +transmitted address only with its protocol role and unknown exception-address +semantics. This is an interface-validity limitation identified from the specification +and decoder source, not a reproduced wrong-PC claim. + +[etmv3-pending]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/common/ocsd_gen_elem_list.h#L124-L140 +[etmv3-retain]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp#L645-L651 +[etmv3-cancel]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp#L486-L490 +[etmv3-exception]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp#L529-L534 +[etmv3-spec]: https://documentation-service.arm.com/static/5f90158b4966cd7c95fd5b5e + +[etmv3-irq]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_proc_etmv3_impl.cpp#L989-L1013 +[etmv3-irq-text]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_elem_etmv3.cpp#L669-L674 +[etmv3-eot]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_proc_etmv3_impl.cpp#L129-L139 + +[cancel-evidence]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-Cancellation/README.md +[field-evidence]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-PacketFields/README.md + +[etmv4-exception]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp#L1690-L1704 +[etmv4-spec]: https://documentation-service.arm.com/static/60017fbb3f22832ff1d6872b From c16ad0b13d6d7103641961fb209a756900e962ec Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Tue, 10 Nov 2026 10:19:24 +0100 Subject: [PATCH 3/6] docs(ctrace): define instruction trace completion and qualification --- tools/ctrace/docs/instruction-trace-design.md | 338 +++++++++++++++--- 1 file changed, 296 insertions(+), 42 deletions(-) diff --git a/tools/ctrace/docs/instruction-trace-design.md b/tools/ctrace/docs/instruction-trace-design.md index 53e59f9a1..e72499650 100644 --- a/tools/ctrace/docs/instruction-trace-design.md +++ b/tools/ctrace/docs/instruction-trace-design.md @@ -1,6 +1,6 @@ # Instruction Trace Implementation Plan -Updated: 2026-11-06. Branch: `feat/ctrace-instruction-trace`. +Updated: 2026-11-10. Branch: `feat/ctrace-instruction-trace`. OpenCSD: **1.8.3**, commit `59f15320637ce6e510604e7bed550d7ad109d15d`. ## Experimental specification @@ -35,6 +35,7 @@ Basis: [CMSIS-Toolbox Experimental Features][experimental-spec], `ctrace` and `* - Declared image entries need a nonempty file list; processors without images may be omitted. - Select one format per processor; selection when both lists exist: TBD. HEX production remains open. - Map matching code to execution addresses; reject invalid files and conflicting overlaps. +- Optional image verification uses `--crc `; see [section 2.1](#21-image-integrity). > **Possible extension:** Mixed ELF/HEX files per `pname`. @@ -46,13 +47,12 @@ ETM source entries: |     `type:` | **Required** | New trace source type: `etm`. | |     `pname:` | Conditional | Required with image inventory; agrees with any processor prefix in `ref`. | |     `stream:` | **Required** | Capture-time CoreSight ATB ID; decoder Trace ID. | -|     `decoder:` | **Required** | Capture-specific ETM decoder configuration. | +|     `decoder:` | **Required** | ETM decoder configuration supplied by pyTS. | | `decoder:` | | Content | | :---|:---|:--- | |     `core:` | **Required** | `Cortex-M4` (ETMv3.5) or `Cortex-M7` (ETMv4.0). | -|     `regs:` | **Required** | Fixed qualified Cortex-M implementation values, written to YAML by pyTS. | -|     `regsToRead:` | **Required** | Variable configuration registers read by pyOCD. | +|     `regs:` | **Required** | All decoder register values, prescribed by pyTS for the Cortex-M profile. | - ETM references need no `index`; reject ambiguous processor/source bindings. - A source's `pname` may have no image entry; this selects image-free fallback, not another processor's images. @@ -60,17 +60,22 @@ ETM source entries: ##### Register Accesses -| Cortex-M profile | `regs` (pyTS) | `regsToRead` (pyOCD) | -| :---|:---|:--- | -| Cortex-M4 / ETMv3.5 | `ETMIDR`, `ETMCCER` | `ETMCR` | -| Cortex-M7 / ETMv4.0 | `TRCIDR0`, `TRCIDR1`, `TRCIDR2`, `TRCIDR8` | `TRCCONFIGR` | +| Cortex-M profile | `regs` (fixed values supplied by pyTS) | +| :---|:--- | +| Cortex-M4 / ETMv3.5 | `ETMIDR`, `ETMCCER`, `ETMCR` | +| Cortex-M7 / ETMv4.0 | `TRCIDR0`, `TRCIDR1`, `TRCIDR2`, `TRCIDR8`, `TRCCONFIGR` | -| `regs:`, `regsToRead:` | | Content | +| `regs:` | | Content | | :---|:---|:--- | | `- name:` | **Required** | Symbolic register name. | -|     `value:` | Conditional | Required in `regs`; filled by pyOCD in `regsToRead`. | +|     `value:` | **Required** | Complete register value supplied by pyTS. | -- `ETMCR` / `TRCCONFIGR`: complete effective configuration. +- Initial implementation: pyTS supplies fixed decoder presets for each supported Cortex-M architecture/profile. + This includes `ETMCR` / `TRCCONFIGR`; no decoder register is read from hardware initially. +- Presets must match the selected ETM implementation and capture setup. `ETMCR` / `TRCCONFIGR` contain the complete + prescribed configuration, including all enabled options. +- Possible later extension: pyOCD reads `ETMCR` / `TRCCONFIGR` back from hardware. Its interface remains to be defined; + readback is not required for the initial implementation. ##### Example @@ -98,9 +103,8 @@ ctrace-run: value: 0x00000004 - name: TRCIDR8 value: 0x00000001 - regsToRead: - name: TRCCONFIGR - value: 0x00000001 # Filled by pyOCD + value: 0x00000001 # Initially prescribed by pyTS ``` #### Initial Implementation @@ -111,6 +115,7 @@ ctrace-run: - Whether to offer `--etm-output decoded`: generic OpenCSD output elements, using ELF or HEX. - ELF/HEX selection when both lists are present; ownership of image-inventory and HEX production. +- `--crc` value syntax, association when multiple images are selected, and behavior when no image is used. - Producer/CMSIS schema alignment for these extensions. #### Out-of-Scope @@ -120,7 +125,8 @@ ctrace-run: ### `pyTS` Utility - Emit consistent `pname` values on DWT/ETM references and any image entries. -- Populate `decoder.regs` from the qualified Cortex-M implementation; list variable registers in `decoder.regsToRead`. +- Populate all of `decoder.regs` from fixed presets for the supported Cortex-M architecture/profile, + including `ETMCR` / `TRCCONFIGR`. Keep these values consistent with the generated capture setup. ### `ctrace` Utility @@ -131,6 +137,7 @@ Usage: Options: --csv Export CSV (ETM default: instructions) --type sel [...] Filter output types; additional ETM selectors: etmv3, etmv4 + --crc Optionally verify an image against the supplied CRC32 --etm-output mode Select ETM mode for --csv (default: instructions) instructions One PC per line (ELF or HEX) packets ETM protocol packets (no image) @@ -144,6 +151,12 @@ Options: - Selectors `etmv3`, `etmv4` select CSV types `ETMv3`, `ETMv4`. +#### `--crc` option + +- `--crc ` optionally supplies the expected image CRC32; details in [section 2.1](#21-image-integrity). +- Without this option, accept the supplied images without CRC verification or a missing-checksum warning. + Normal image-format and bounds validation still applies. + #### `--etm-output` option | Mode | Required images | Output records | @@ -154,7 +167,8 @@ Options: - No matching image inventory: warn once per source/input and use packet fallback in `instructions`. - Image-backed modes: missing/unreadable declared files, invalid images and ambiguous format selection are errors. - `packets` neither validates image files nor selects a format. Unmapped code uses local packet fallback. + `packets` neither validates image files nor selects a format. Unmapped code uses local packet fallback with a warning; + fallback alone does not make the invocation fail. See [diagnostics and completion](#diagnostics-and-completion). - `decoded` remains unavailable until offering it is decided; if offered, absent images are an error. - `--etm-output` requires CSV output (`--csv` or `--all`); it does not change ITM/DWT decoding. - Without an output option, validate/decode using the default mode and fallback; write no output file. @@ -207,13 +221,15 @@ cycles,stream,type,packet,index,value,pc,address,note **`instructions`:** -- Expand complete ranges once, one PC per line; `packet=INSTRUCTION`, opcode in `value`, `index`/`address` empty. - Retain not-executed/unknown PCs and all other decoded events/status; never fabricate missing PCs. +- Expand confirmed ranges once, one PC per line; `packet=INSTRUCTION`, opcode in `value`, `index`/`address` empty. + Confirmation requires decoder resolution and image validation as defined in section 3.2. + Retain not-executed/unknown PCs, other confirmed events and observed status; never fabricate missing PCs. + A known unresolved tail is reported on `eo_trace`, not as an `INSTRUCTION` with `execution=unknown`. - Explain execution state with `execution_basis` in `note`. CBZ/CBNZ can execute without taking the branch. - Preserve opcode width in `value`; Thumb-32 uses first then second little-endian halfword: bytes `2d e9 f0 41` become `value=0xe92d41f0`. - Uncovered code: preserve valid PC prefixes and gaps, then emit the corresponding complete protocol packet once. - Continue packets through the gap; resume PCs only when code and decoder state permit reconstruction. + Continue packets through the gap; resume PCs only when both code and sufficient decoder state permit reconstruction. A packet may cover several ranges and overlap retained PCs: mark this with `partial=1`. - Fallback notes contain `record=packet`, `reason=no_image|image_gap`, `packet_size`, `packet_bytes` and packet fields. Instruction-specific columns stay empty. At a shared origin, decoded rows precede the complete fallback packet. @@ -227,13 +243,16 @@ cycles,stream,type,packet,index,value,pc,address,execution,note ``` **`packets`:** one row per protocol packet or parser status; no instruction reconstruction. +A complete terminal packet remains valid even when instruction reconstruction would need later cancellation resolution. +Do not infer pending ranges or `INCOMPLETE_EOT` from that condition, or require images/full decoding in this mode. ```csv cycles,stream,type,packet,index,value,pc,address,packet_size,packet_bytes,note ,18,ETMv4,atom_f1,,,,,1,f7,atoms=E ``` -**`decoded` (TBD):** one row per generic OpenCSD element; ranges remain ranges. +**`decoded` (TBD):** one row per confirmed generic OpenCSD element or observed status; ranges remain ranges. +Apply the same confirmation and unresolved-tail policy as `instructions`; forced EOT output is not an exception. ```csv cycles,stream,type,packet,index,value,pc,address,instruction_count,note @@ -253,7 +272,8 @@ cycles,stream,type,packet,index,value,pc,address,instruction_count,note | `addr_nacc` | `address` = inaccessible code address. | | `timestamp`, `cycle_count` | Valid complete value in `value`; domain/association and partial fields in `note`. | | `event`, `sync_marker` | Event/marker kind and value in `note`; an event mask is not an exception index. | -| `trace_on`, `no_sync`, `eo_trace`, `addr_unknown`, `unknown` | Reason/state in `note`. | +| `trace_on`, `no_sync`, `addr_unknown`, `unknown` | Reason/state in `note`. | +| `eo_trace` | End reason and any known unresolved reconstruction state in `note`; no instruction payload. | | Protocol address | Complete valid address in `address`; partial address bits/mask in `note`. | | Protocol exception fields | Normalized `index` when known; raw code, event kind and flags in `note`. | | Other protocol fields | Structured atoms, context, counts, speculation and validity in `note`. | @@ -269,6 +289,11 @@ cycles,stream,type,packet,index,value,pc,address,instruction_count,note omit the constant ISA, duplicate mappings, inactive union members and internal bookkeeping. Do not parse display strings. - Preserve ETMv3 cancellation information (`cancel_previous=1`) even when images cover the entire trace. +- For known unresolved reconstruction at normal EOT, annotate the existing `eo_trace` row with `unresolved=1` and + `unresolved_reason`; ETMv3 missing cancellation lookahead uses `unresolved_reason=etmv3_cancel_lookahead_missing`. + Add `pending_ranges` only when the exact pending range count is known. Leave `pc`, `address`, `value`, `index` and + `execution`/`instruction_count` empty. These are reconstruction-state annotations, not new packets or parser errors. + Absence of an unresolved annotation does not certify completeness when the decoder cannot expose that state. - `cycles` needs qualified association; raw timestamps/counts do not imply per-PC timing or cross-core synchronization. Preserve their domain, validity and association in `note`; do not distribute cycles across PCs. @@ -280,6 +305,52 @@ Packet names and parser status: Appendix B. Exception normalization: section 3.3 - `packet` and ETM exception `index` do not select separate trace settings. - Recognize the mode suffix in `...csv`; input-wide diagnostics have no source reference. +#### Diagnostics and completion + +Structured diagnostic impact determines invocation failure independently of output filtering. The Error conditions +in this ETM contract have failing impact; Warning and Info conditions do not fail the invocation by themselves. +Existing ITM/DWT and producer diagnostics retain their established impact. [C17] + +| Condition | Diagnostic | Required effect | +| --- | --- | --- | +| No image or an image gap with valid packet fallback | Warning | No failure by itself. | +| Initial unsynchronized prefix followed by usable sync | Info | Preserve skipped-byte status; no failure. | +| Configured source without payload bytes | None for this condition | No missing-sync error. | +| Normal EOT without evidence of a defect | None for this condition | Export confirmed results; no extra failure. | +| Known unresolved tail at normal EOT | Info | Retain confirmed prefix and unresolved state; no failure. | +| Valid terminal DISCARD after usable sync | Info | Retain prefix and discontinuity; no failure. | +| Payload received, but no usable synchronization by EOT | Error | Nonzero exit; retain other confirmed results. | +| Incomplete or invalid packet, unsupported configuration | Error | Nonzero exit; retain safe results. | +| Range inconsistent with the image bytes | Error | Nonzero exit; no PCs from the invalid range. | +| Fatal input or decoder error | Error | Nonzero exit; safe CSV prefix and global abort marker. | +| Spool I/O error | Error | Nonzero exit; retain only reliably replayable confirmed prefixes. | +| CSV write or close error | Error | Nonzero exit; remove the unreliable CSV. | + +- Usable synchronization is assessed for the active decoder mode; an absent code image is not a synchronization error. + Do not require a generic `TRACE_ON` element: the M7 reference reconstructs instructions without one. + Remember whether usable synchronization was reached; a terminal DISCARD does not turn it into a never-synchronized source. + Normal EOT does not prove that every observed trace element could be reconstructed; see section 3.2. +- A valid terminal DISCARD and its `NO_SYNC(discard)` describe one protocol discontinuity. Retain observed status and + confirmed results; do not issue a global abort marker or demand subsequent synchronization. + Unresolved reconstruction alone is likewise nonfailing; neither condition clears an earlier failing diagnostic. + Actual malformed/incomplete packets and overflow retain their own diagnosed cause and impact. +- For each of these terminal conditions, report one nonfailing CLI Info per source/input, visible at normal verbosity + with input/source attribution independently of output filters. In image-backed modes, the CSV carries the state in + `no_sync` or `eo_trace`; packet mode retains the observed DISCARD packet without inferring reconstruction state. + Do not add `type=info` CSV rows for these conditions; existing skipped-byte annotations and their filter rules remain. +- Parser-status rows retain their ETM type and the mappings in Appendix B.3. Associated warning/error CSV diagnostics + retain the existing `error`/`overflow` categories; there is no new `warning` type or severity column. +- On a fatal input abort, preserve confirmed results from all sources, including buffered results, when reliably + available and subject to normal output selection. Append one input-wide `type=error` row whose `note` starts with + `Decode aborted; trace incomplete`. + Leave all other fields empty. This marker bypasses type/stream filters, following the existing output policy. [C17] +- A failed spool may prevent retrieval of buffered results. Retain only a reliably replayable prefix per source; + do not silently append later records across an unreadable segment. Publish a safely finalized partial CSV with the + abort marker; if writing or closing that CSV fails, remove it. Never publish an apparently complete truncated output. + Failures before CSV startup create no CSV. Source-local recovery and input abort are specified in section 3.2. +- Incomplete CTF artifacts are removed and excluded from target XML. Continue other inputs after an input-local failure + when the remaining input/output infrastructure is usable. [C17] + #### CTF Format - Evaluate CTF eligibility for configured sources after source/type selection. @@ -296,8 +367,11 @@ Packet names and parser status: Appendix B. Exception normalization: section 3.3 | Production | ITM/DWT supported; ctrace ETM integration pending. Experimental specification above is the contract. | | Required profiles | Cortex-M4/ETMv3.5 and Cortex-M7/ETMv4.0. | | Capture evidence | Both profiles decoded with the host helper; PC/classifier validation currently M7 only. | -| Decoder blockers | Pinned OpenCSD ETMv3 cancellation and truncation defects; section 5. | +| Decoder configuration | All register values initially prescribed by pyTS; optional later pyOCD readback. | +| Decoder blockers | ETMv3 cancellation, pending EOT output and truncation require corrections; section 5. | +| Local decoder correction | Thumb-16 short-read regressions pass; complete ETM output qualification remains pending. | | Images | ELF/HEX import and parity with archived BIN references remain to be implemented/qualified. | +| Completion and recovery | Required behavior specified below; production acceptance gates in section 6.2 remain open. | ## 2. Code-image provider @@ -315,10 +389,32 @@ Packet names and parser status: Appendix B. Exception normalization: section 3.3 Validate bounds, address overflow, available data and conflicting overlaps before copying into owned regions. Reject unsupported compression, relocation and sectionless ELF; leave BSS and holes unmapped. [E2], [E4], [H1], [H2] For scatter-loaded RAM code, `sh_addr` is the execution address; a segment load address is insufficient. [E1], [E6] +The local OpenCSD Thumb-fetch patch permits a complete 16-bit instruction at a region end without invented padding. +Image-provider qualification must also cover incomplete Thumb-32 instructions and adjacent regions; section 5. +For both hardware references, qualify the actual AXF import against the archived BIN mapping at every reconstructed +instruction address, including `not_executed` and `unknown` positions. Compare existing bytes at execution addresses, +not entire files or BIN padding; preserve ELF holes. These checks do not establish target-flash identity or qualify +other ELF/HEX layouts. Reference-input preparation and the required output sequences are specified in section 6. Direct ELF import candidate: ELFIO, header-only C++, MIT; pin the dependency and retain its notices. [E3], [L1] Copy bytes before releasing ELFIO storage. HEX and ELF feed the same provider; no HEX conversion is required. [E5] +### 2.1 Image integrity + +Image verification is opt-in through `--crc `. The caller supplies the expected CRC32 when verification +is wanted. Without the option, ctrace accepts the supplied image bytes without a CRC check or matching-image guarantee. +No CRC or expected-length metadata is required in the image inventory. + +- Use zlib-compatible CRC32 over the complete original file bytes, in binary mode, separately for each ELF/AXF or HEX. + The check value for ASCII `123456789` is `0xcbf43926`. [I1] +- With `--crc`, compare the CRC32 against the supplied value before instruction reconstruction, + using the same owned file bytes that are imported. A mismatch is an error. +- `packets` continues to open no images and calculate no image checksums. Define the handling of `--crc` + when no image is used, and its association when multiple images are selected, before implementation. +- This identifies the exact file artifact: even metadata or HEX line-ending changes can cause a mismatch. + It does not establish ELF/HEX code equivalence or verify the target's actual programmed memory. +- CRC32 is a non-cryptographic checksum for accidental-change detection; collisions remain possible. [I1] + ## 3. Integration boundaries ### 3.1 Source configuration @@ -334,12 +430,13 @@ Copy bytes before releasing ELFIO storage. HEX and ELF feed the same provider; n - Make `core_prof` configurable; default to `Cortex-M` (`profile_CortexM`). Its configuration interface is not part of the Experimental specification yet. -- Pending specification update: obtain `TRCCONFIGR` from the pyOCD "write registers" block, - not the "ID regs" block. Keep the Experimental specification unchanged for now. +- pyTS is the initial source of all decoder register values, using fixed Cortex-M architecture/profile presets. + `ETMCR` / `TRCCONFIGR` are supplied in `decoder.regs` as complete values, without reading hardware or deriving + decoder metadata from the pyOCD register-write block. Later pyOCD readback is an optional extension. - `decoder.regs` is metadata; existing reference/disable `regs` retain hardware-write semantics. -- Fixed values belong to the qualified ETM implementation/revision, not every instance of a Cortex-M name. - Offline decoding needs resolved capture-time configuration; pyOCD supplies no static replacement. -- Initialize unused OpenCSD ETMv4 IDR9–13 fields deterministically; they are not measured registers. [O10], [O14] +- pyTS owns preset selection and consistency with the supported ETM implementation/revision and capture setup. +- Initialize unused OpenCSD ETMv4 IDR9–13 fields to zero for these instruction-only profiles; + no hardware readback is needed. The M7 reference's historical IDR12 value does not affect this decode path. [O10], [O25] - Bind `stream -> reference -> pname -> code regions`; preserve DWT symbol-file provenance separately. [C4], [C13] M4 source example; the corresponding image entry, when present, uses `pname: CM4`: @@ -356,9 +453,8 @@ M4 source example; the corresponding image entry, when present, uses `pname: CM4 value: 0x4114f250 - name: ETMCCER value: 0x18541800 - regsToRead: - name: ETMCR - value: 0x00000810 # Filled by pyOCD + value: 0x00000810 # Initially prescribed by pyTS ``` ### 3.2 Decoder and packet monitoring @@ -367,9 +463,9 @@ M4 source example; the corresponding image entry, when present, uses `pname: CM4 | --- | --- | | CLI / output planning | ETM selectors/mode, fixed schema/name per input; independent CSV/CTF eligibility. | | YAML / source routing | Neutral configuration fields and source/image binding; no OpenCSD types in YAML models. | -| `FileDecodeJob` / image readers | Owned code regions and Trace-ID-aware memory access. | -| `DecodePipeline` / ETM session | One tree per file, independent ETMv3/ETMv4/ITM sources and diagnostics. | -| ETM adapters / CSV | Structured packets, generic events, range expansion and deterministic fallback merge. | +| `FileDecodeJob` / image readers | Owned code regions, Trace-ID-aware access, and distinct normal/abort completion. | +| `DecodePipeline` / ETM session | One tree per file, independent sources, transaction checks and local recovery. | +| ETM adapters / CSV | Confirmed records, bounded buffering, range expansion and deterministic fallback merge. | - Use `OCSD_TRC_SRC_FRAME_FORMATTED` / `OCSD_DFRMTR_FRAME_MEM_ALIGN` for TB input. [O2] - `packets` and full-source fallback: `OCSD_CREATE_FLG_PACKET_PROC` plus `OCSD_C_API_CB_PKT_MON`. @@ -382,19 +478,63 @@ M4 source example; the corresponding image entry, when present, uses `pname: CM4 qualification and invalidation, not just a different callback response. [O19], [O23], [O24], [O25] - Route identity includes source/ATB ID; `pname` alone is not a route key. [C16] -**Mixed-output merge:** +#### Confirmed results and mixed output - Retain packets and generic elements per source, with packet order, protocol position and local element order. Association is many-to-many; decoder positions are not physical RAW offsets. -- Finalize after decoder EOT; spool if necessary to bound memory. Preserve generic-element order and complete - valid PC prefixes. Let the full decoder resolve commit, cancel, mispredict and discard before PC export. +- Treat callback output as provisional until the synchronous decoder operation returns and its diagnostics have been + classified. Accept only valid records outside any discarded transaction suffix; callback arrival alone is insufficient. + Retain observed parser-status bytes and diagnostics even when reconstruction output from that transaction is discarded. +- A confirmed instruction range is complete, no longer speculative or cancelable under the qualified decoder contract, + and validated against the image bytes as specified in section 4. Let the decoder resolve commit, cancel, mispredict + and discard before PC export. Confirmation concerns reconstruction, not architectural retirement or completion. +- Preserve generic-element order and confirmed PC prefixes. A source's packet monitor additionally preserves fields + omitted by generic elements, including ETMv3 cancellation. Only generic output drives instruction/context classification. - `ADDR_NACC` opens an image-gap interval at its originating packet. Keep that complete packet and subsequent packets through the interval; stop before the next successfully reconstructed packet origin. If that origin also contains a gap, retain its valid PCs and its packet once. - For one origin: generic rows/expanded PCs first, then its fallback packet. Set `partial=1` when it also produced PCs. Missing generic output alone does not establish an image gap; canceled/unresolved atoms are not PC rows. -- Only generic output drives instruction/context classification. MON additionally preserves source fields omitted - by generic events, including ETMv3 cancellation; unresolved EOT state remains explicit. +- An image gap does not create synchronization or reset another source. Resume PCs only when code and sufficient + decoder state support reconstruction. A range inconsistent with available image bytes is an Error, not `image_gap`. + +#### Source-local recovery + +- Classify errors after the active decoder operation returns. When recovery is qualified for the affected source, + discard only its invalid transaction suffix and reset only its decoder chain and dependent classification state. + Preserve confirmed results, the formatter state and unaffected sources; do not re-feed consumed input bytes. +- Resume the affected source only at a real protocol synchronization point with sufficient state for the selected mode. + If safe source-local recovery is unavailable, or a formatter/input-wide failure prevents it, abort that input. + Recovery does not clear a diagnostic's failing impact. + +#### EOT, abort and buffering + +- On normal input exhaustion, process decoder EOT, classify its output and diagnostics, then finalize the merge. + EOT must not artificially confirm outstanding speculative or still-cancelable elements. Export only confirmed results + and observed status. Previously confirmed records that were merely buffered remain exportable; callback timing alone + does not establish confirmation, so do not reject all EOT callbacks indiscriminately. +- ETMv3 requires cancellation lookahead. At normal EOT, withhold any reconstruction elements still pending that decision, + including instruction ranges and exception-return markers. Annotate `eo_trace` as specified in the CSV contract; + do not label the pending result canceled, not-executed, an image gap or an incomplete packet. + The qualified decoder integration must expose the pending/confirmed distinction across EOT as structured state and + prevent forced promotion from being accepted as confirmation. Pinned `onEOT()` calls `commitAllPendElem()`; correcting + the packed-atom pending guard alone does not satisfy this contract. [O31] +- The M4 reference therefore requires 7,933 confirmed instruction positions and one unresolved terminal range. + Only the last occurrence at `0x08100e34` is withheld; earlier occurrences of that address remain eligible. + The historical 7,934-range export includes the EOT-promoted range and is not the confirmed-output oracle; section 6.1. +- Distinguish normal EOT from an `INCOMPLETE_EOT` parser status: preserve that status and available bytes without + manufacturing a complete packet or trusting stale payload fields. The known ETMv3 defect must be corrected first. +- Report unresolved state only when supported by observed packets or qualified decoder output; do not invent missing + PCs, counts or resolution reasons. Pinned ETMv4 can discard uncommitted P0 elements at EOT without reporting their + count or identity. Its generic `EO_TRACE` therefore does not certify complete reconstruction. [O25], [O17] +- Keep an abort-completion path independent of decoder EOT. After a fatal error, stop feeding the decoder and finalize + already confirmed records under normal output selection before the global abort marker. Do not call EOT, reset or commit + on the failed decoder to recover buffered output. Discard provisional results rather than turn them into PC rows. +- Bound memory independently of input length. Spill retained records to an owned temporary spool when its explicit + memory limit is reached, and stream expansion/output without accumulating every PC in memory. +- Spool creation, write, flush, read and close failures are Errors. Replay only the reliable confirmed prefix per source; + do not skip unreadable segments or promise recovery of an inaccessible spool. Apply the CSV failure policy above. + Close and remove temporary spools after normal completion or abort; report cleanup failures. ### 3.3 Cortex-M exception normalization @@ -423,6 +563,8 @@ IHI0064H.a Table 6-13, pp. 268–269; DDI0403E.e pp. 517, 525. [A11], [A10], [A1 - Verify range instruction count, exclusive end and final size against code bytes. M4/M7 use mixed-width Thumb; neither PC bit zero nor range byte length establishes instruction count. +- A failed complete-range validation is an Error. Emit no PCs from that range; retain earlier confirmed ranges and + apply the recovery/abort rules in section 3.2. Unavailable code follows the separate image-gap fallback contract. - `execution` describes the predicate, not completion. Canceled/interrupted instruction facts remain separate events; no invented completed PC for a partially resumed instruction. - Track IT state: ordinary taken branches leave an IT block; Armv7-M exception entry starts with ITSTATE=0. @@ -460,36 +602,136 @@ Details and reproducible defects: [Known OpenCSD issues](opencsd-issues.md). | Area | Pinned-version result / consequence | | --- | --- | +| Thumb instruction at image end | Four-byte fetch rejects complete Thumb-16; corrected by the local Thumb-fetch patch. | | ETMv3 cancellation | Packed EE followed by cancellation can retain a canceled PC; decoder correction required. | +| ETMv3 pending output at EOT | Pending results are promoted without trace resolution; preserve unresolved state. | | ETMv3 IRQ8 field | `cm_irq_n` is 24 instead of 8; normalize from the raw exception code. | | ETMv3 truncated branch at EOT | Reported as a valid branch with stale fields; truncation handling needs correction. | | ETMv3 exception metadata | Generic `EXCEPTION` omits the raw Cancel bit; preserve it through packet monitoring. | +| ETMv4 unresolved EOT state | Uncommitted P0 may be discarded without a count/identity; EOT is not completeness proof. | | ETMv4 reset address | Generic return-address flag does not establish architectural validity for reset. | | ETMv4 conditional non-branch trace | Parser recognizes it; full decoder rejects configuration with `HW_CFG_UNSUPP`. | | Armv8.1-M branch-forward | Not qualified here; fixtures do not establish complete support. | | Cortex-M23 / ETMv3 | Decoder checks `ARCH_V7`, core map selects `ARCH_V8`; qualification required. | -- ETMv3 support remains required; correct cancellation and EOT handling before qualifying its outputs. - The CM4 hardware capture contains no cancellation packet and does not cover that defect. +- ETMv3 support remains required; correct cancellation, pending-output handling and packet truncation before qualifying + its outputs. The CM4 hardware capture has no cancellation packet, but ends in packed EE with a pending final range. + An isolated pending-guard correction releases 7,933 ranges before EOT and the last range only during EOT. + Synthetic cancellation/noncancellation extensions distinguish an unresolved tail from a confirmed one; they do not + prove that the captured hardware instruction was actually canceled. A production correction remains required. [O31] +- The Thumb-fetch correction covers ETMv3, ETMv4 and PTM fetch sites, including shared ETE source-address code. + It does not establish ETE qualification or replace ELF/HEX import and end-to-end instruction-output tests. - The M7 reference has conditional non-branch tracing disabled; absent IT predicate results remain `unknown`. Enabling it requires a decoder that supports the configuration; changing archived metadata cannot recover results. - Additional cores, security/dynamic mappings and protocol revisions need separate qualification. Unsupported features must remain explicit; packet-parser support is not full instruction-decoder support. [O1], [O8], [O13], [O25], [O28] -## 6. Reference evidence +## 6. Evidence and acceptance criteria + +### 6.1 Reference evidence Companion workspace: `/Users/thode01/work/github/TraceBufferExamples`; captures retain their original schemas. | Reference | Evidence | Remaining limit | | --- | --- | --- | -| [STM32H7B3I-DK / M7][ref-m7] | 4-KiB ETF, ATB `0x12`; packets, ranges and PC validation. | ELF/HEX parity. | -| [STM32H747I-DISCO / M4][ref-m4] | 4-KiB ETF, ATB `0x22`; packets and ranges. | PC/classifier qualification. | +| [STM32H7B3I-DK / M7][ref-m7] | 4-KiB ETF, ATB `0x12`; packets, ranges and PC validation. | AXF import parity and production terminal-DISCARD handling. | +| [STM32H747I-DISCO / M4][ref-m4] | 4-KiB ETF, ATB `0x22`; packets and ranges. | AXF import parity, confirmed-tail handling and PC/classifier qualification. | | [ETMv3 cancellation][ref-cancel] | Synthetic packed/split atom comparison. | Host reproduction. | | [ETMv3 packet fields][ref-fields] | IRQ8 field and truncated-branch cases. | Host reproduction. | - Hardware bundles contain matching RAW, images, decoder metadata and validation reports; both capture windows wrap. - M7 PC evidence: `validation/pc-expansion/README.md`; hardware/project configuration: each board's `TRACE_SETUP.md`. - Shared host helpers: `STM32H7B3I-DK/tools/`. They establish reference evidence, not production ctrace ETM support. +- Fresh host runs with the local Thumb-fetch patch reproduce both archived generic/packet exports at input chunk sizes + 16 and 65,536. M7 also reproduces with unused IDR9–13 zeroed. These runs use BIN mappings, not the proposed AXF reader. + +#### Prepared ctrace reference inputs + +Prepare one isolated input directory per reference; preserve the original bundles and archived outputs unchanged. +Copy `capture.TB.raw` and the matching `InstructionTrace.axf`, verify their archived lengths/hashes as fixture provenance, +and add `capture.ctrace-run.yml`. The historical JSON is evidence for this conversion, not an additional ctrace input format. + +| Reference input | ETM reference / processor | Stream | Image inventory | +| --- | --- | --- | --- | +| M7 | `ref: CM7/instructions`, `pname: CM7` | `0x12` | `pname: CM7`, `elf: [InstructionTrace.axf]` | +| M4 | `ref: CM4/instructions`, `pname: CM4` | `0x22` | `pname: CM4`, `elf: [InstructionTrace.axf]` | + +Each YAML contains `ctrace-run.ctrace-images` and `ctrace-run.ctrace-refs`, with `type: etm`, the matching `decoder.core` +and the fixed register values shown in the source examples. Keep image paths relative to that YAML. Use only the ELF +list; the M4 `Boot_CM7.axf` is bootstrap code, not a CM4 code image or an additional trace source. +Invoke each directory separately with `ctrace --csv --channel TB`; discovery is not recursive. [C18] +Repeat with `--etm-output packets`. The resulting `capture.TB.instructions.csv` and `capture.TB.packets.csv` are new outputs. +Run these cases without `--crc`; no missing-checksum diagnostic is expected. Fixture hash verification above is test +provenance, not a new runtime checksum requirement. These are two single-source runs, not a combined multi-source test. + +#### Expected results for the two RAW files + +The following are acceptance targets for the future implementation, not completed ctrace qualification. Compare semantic +sequences, source identity, packet bytes and status; the historical CSV schemas and total row counts differ from this contract. +Both RAW files are 4,096-byte memory-aligned inputs with an initial unsynchronized prefix and later usable synchronization. +With the matching AXF, neither should encounter an image gap, invalid/incomplete packet or global abort. +Both references must exit zero in `instructions` and `packets`. The table and reconstruction annotations below apply +only to image-backed `instructions` and, if offered, equivalent confirmed ranges/status in `decoded`. +`packets` preserves the observed packet sequence and parser status without asserting pending reconstruction state. + +| Reference | Required instruction result | Required completion | +| --- | --- | --- | +| M7 | 3,809 positions from 1,076 ranges. | Terminal discontinuity retained; nonfailing Info. | +| M4 | 7,933 confirmed positions. | One unresolved terminal range; nonfailing Info. | + +- M7's PC/opcode/width sequence must match the reference. Its observed packet tail is DebugHalt, COMMIT, DISCARD; + the instruction output retains the DebugHalt event, `no_sync` with the discard reason, and `eo_trace`, plus one + discontinuity CLI Info. Do not add COMMIT/DISCARD packet rows to image-covered instruction output solely for this test. +- M7 classification must retain 3,531 `executed`, 145 `not_executed` and 133 `unknown` positions with their exact assignments. + The unknown reasons are one unavailable initial IT state and 132 untraced IT predicates. Check the absence of generic + `TRACE_ON`: successful synchronization must not depend on that element being present. +- M4's historical export has 7,934 single-instruction ranges. Exclude only its final range occurrence + `[0x08100e34, 0x08100e36)` at decoder position 4,071 (`sequence=8095` in `capture.decoded.csv`), not all matching PCs. + Preserve earlier confirmed output from the same packet origin. The instruction-mode `eo_trace` must carry + `unresolved=1`, `unresolved_reason=etmv3_cancel_lookahead_missing` and `pending_ranges=1`. + The final complete P-header `88` (EE) remains in packet output without pending-range annotations. + Neither the packet nor the unresolved reconstruction is `INCOMPLETE_EOT`. +- Independently qualify the M4 PC widths/bytes and execution classifier; the expected count alone is insufficient. + The reference contains 499 CBZ/CBNZ ranges, including 238 N outcomes. Check that the comparison still executes; + do not turn every `last_executed=0` into `execution=not_executed`. +- Confirm AXF/BIN code-byte parity as required in section 2. M7's existing LLVM validation checks BIN bytes and + instruction boundaries within decoder-provided ranges; it does not validate AXF import or independently prove + trace control flow/predication. Neither reference qualifies general ELF/HEX layouts or multiple sources by itself. + +### 6.2 Production qualification + +The following ctrace acceptance gates remain open. Existing host/reference results and the local Thumb-fetch dependency +regressions do not replace them. Cover both required profiles and vary input chunk boundaries where applicable. +Each test checks exact records and source order, packet bytes/association, diagnostics and exit status. + +| Scenario | Required evidence | +| --- | --- | +| Prepared M4/M7 AXF inputs | Section 6.1 semantic sequences, completion and AXF/BIN parity. | +| ETMv3 tail and resolving/canceling continuations | Only confirmed results; faithful packets. | +| Terminal M7 DISCARD | Prefix retained, nonfailing Info; no missing-sync error or abort. | +| Valid prefix, image gap, then reconstruction | Prefix retained; PCs resume only with code and decoder state. | +| Multiple ranges and a gap sharing one packet origin | Packet emitted once after generic rows; accurate `partial=1`. | +| Packed/split atoms, cancellation and a later abort | Packing-invariant confirmed PCs; no canceled or speculative PC. | +| Normal EOT, including a parsed but unresolved trace | Confirmed results only; no invented resolution/counts. | +| Truncated packet and malformed formatter tail | Bytes/status retained; no stale fields; Error and safe output. | +| Fatal failure after a confirmed prefix | Buffered prefix retained without decoder EOT; exactly one global abort row. | +| Two sources in one formatted RAW, local failure/recovery | Other source unchanged; source state and images isolated. | +| Initial unsynchronized bytes, never-synchronized source, empty source | Distinct Info/Error/no-error outcomes. | +| Synchronization loss followed by recovery | Earlier failing impact retained; confirmed prefix preserved. | +| Input larger than the buffer limit | Bounded peak memory; spooled and in-memory output byte-identical. | +| Injected spool and CSV I/O failures | Error exit; trustworthy partial CSV or removal; temporary resources cleaned up. | + +Use different code bytes at the same addresses for the two-source case. Include missing-image packet fallback and +recovery with an image gap; a source with no mapped code alone does not validate resumption. Inject failures at spool +creation/write/flush/read/close and CSV write/close, including failure while publishing an abort marker. +Exercise an unreadable middle spool segment: no later source records may silently bridge it. Repeat fatal-abort tests +with type/stream filters; data rows obey selection and the single global abort marker remains visible. +Exercise ETMv3 pending ranges and exception-return markers, as well as confirmed output buffered until EOT; +only the unresolved elements are withheld. Use the real M4 terminal EE and clearly labeled synthetic continuations: +a following noncanceling packet confirms its tail, while a canceling exception removes it. Keep the original RAW unchanged; +synthetic extensions are decoder tests, not evidence of what happened on the board. Compare packing/chunk variants. +Cancellation, pending-EOT and ETMv3 truncation gates require the decoder corrections in section 5 +before qualification can pass. ## Appendix A. Trace representations @@ -616,7 +858,15 @@ These architectural entries require decoder work; see IHI0064H Table 6-1 and the not exported packet classes. Extension headers resolve to concrete types. - The pinned ETMv3 processor can report a truncated branch at EOT as a valid branch with stale fields; see [Known OpenCSD issues](opencsd-issues.md#etmv3-unfinished-packet-emitted-without-eot-error-status). -- Decoder diagnostics retain existing `error`, `overflow`, `info` categories and source attribution when known. +- Initial `notsync` is not an immediate Error while first usable synchronization may still follow. Assess missing + synchronization at EOT per source; a configured source without bytes is not a missing-sync error. +- Synchronization loss after a synchronized period follows its diagnosed cause and recovery policy. Later + resynchronization does not clear a diagnostic's failing impact. Valid terminal DISCARD/`NO_SYNC(discard)` is the + nonfailing discontinuity specified above; it is not a parser error or failure to achieve initial synchronization. + A complete terminal packet awaiting reconstruction lookahead is not `incomplete_eot`. +- `incomplete_eot`, invalid sequences, reserved/unsupported encodings and configurations produce Error diagnostics. + Preserve source attribution when known. CSV categories, failure impact and retained output follow + [diagnostics and completion](#diagnostics-and-completion); recovery follows section 3.2. - ETE-only source-address, transaction and PE-reset entries in the shared header are outside this proposal. ## Sources @@ -657,11 +907,15 @@ OpenCSD links identify the inspected pin; architecture editions/pages are stated [L1]: https://github.com/serge1/ELFIO/blob/main/LICENSE.txt +[I1]: https://docs.python.org/3/library/zlib.html#zlib.crc32 + [C4]: ../src/tracerun/CtraceRunMeta.cpp [C10]: ../src/model/TraceSelection.h [C11]: ../src/output/csv/CsvRowMapper.cpp [C13]: ../src/tracerun/YmlTraceRunConfigReader.cpp [C16]: ../src/tracerun/CtraceRunMeta.h +[C17]: constraints.md#observable-behavior-and-output-safety +[C18]: ../src/tracerun/TraceRunDiscovery.cpp [O30]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/c_api/ocsd_c_api_types.h [O31]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp From 4c9c63a223aa1e8f8e4b755aa9a1393cc61787bb Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Tue, 10 Nov 2026 10:37:42 +0100 Subject: [PATCH 4/6] fix(ctrace): patch OpenCSD Thumb-16 boundary reads --- .../cmake/dependencies/ConfigureOpenCSD.cmake | 10 +- .../dependencies/OpenCSD-thumb-fetch.patch | 130 +++++++++ tools/ctrace/docs/THIRD_PARTY_NOTICES.md | 12 +- tools/ctrace/docs/opencsd-issues.md | 40 ++- tools/ctrace/test/unit/CMakeLists.txt | 1 + .../OpenCsdInstructionBoundaryTests.cpp | 263 ++++++++++++++++++ 6 files changed, 448 insertions(+), 8 deletions(-) create mode 100644 tools/ctrace/cmake/dependencies/OpenCSD-thumb-fetch.patch create mode 100644 tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp diff --git a/tools/ctrace/cmake/dependencies/ConfigureOpenCSD.cmake b/tools/ctrace/cmake/dependencies/ConfigureOpenCSD.cmake index 23132f016..092d83e50 100644 --- a/tools/ctrace/cmake/dependencies/ConfigureOpenCSD.cmake +++ b/tools/ctrace/cmake/dependencies/ConfigureOpenCSD.cmake @@ -10,9 +10,9 @@ if(NOT EXISTS "${CTRACE_OPENCSD_SOURCE_DIR}/CMakeLists.txt") ) endif() -function(ctrace_apply_opencsd_patch) +function(ctrace_apply_opencsd_patch patch_name) find_package(Git REQUIRED QUIET) - set(patch_file "${CMAKE_CURRENT_FUNCTION_LIST_DIR}/OpenCSD.patch") + set(patch_file "${CMAKE_CURRENT_FUNCTION_LIST_DIR}/${patch_name}") set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${patch_file}") # Reconfiguration must preserve both the patch and unrelated local edits. @@ -34,7 +34,7 @@ function(ctrace_apply_opencsd_patch) ERROR_VARIABLE patch_error ) if(NOT patch_result EQUAL 0) - message(FATAL_ERROR "Failed to apply the ctrace OpenCSD patch: ${patch_error}") + message(FATAL_ERROR "Failed to apply the ctrace OpenCSD patch ${patch_name}: ${patch_error}") endif() endfunction() @@ -61,7 +61,9 @@ set(BUILD_TESTING OFF CACHE BOOL "Build the testing tree." FORCE) set(ctrace_saved_install_prefix_default "${CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT}") set(CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT FALSE) -ctrace_apply_opencsd_patch() +# Apply independently so an existing ITM-patched checkout can receive later fixes. +ctrace_apply_opencsd_patch(OpenCSD.patch) +ctrace_apply_opencsd_patch(OpenCSD-thumb-fetch.patch) add_subdirectory( "${CTRACE_OPENCSD_SOURCE_DIR}" "${CMAKE_BINARY_DIR}/external/OpenCSD" diff --git a/tools/ctrace/cmake/dependencies/OpenCSD-thumb-fetch.patch b/tools/ctrace/cmake/dependencies/OpenCSD-thumb-fetch.patch new file mode 100644 index 000000000..bb5ddc1b9 --- /dev/null +++ b/tools/ctrace/cmake/dependencies/OpenCSD-thumb-fetch.patch @@ -0,0 +1,130 @@ +diff --git a/decoder/source/ocsd_code_follower.cpp b/decoder/source/ocsd_code_follower.cpp +--- a/decoder/source/ocsd_code_follower.cpp ++++ b/decoder/source/ocsd_code_follower.cpp +@@ -35,5 +35,6 @@ + + #include "common/ocsd_code_follower.h" ++#include "i_dec/trc_idec_arminst.h" + + OcsdCodeFollower::OcsdCodeFollower() + { +@@ -141,2 +142,2 @@ +- uint32_t opcode; // buffer for opcode ++ uint32_t opcode = 0; // buffer for opcode, including a partial read + +@@ -147,7 +148,9 @@ + if(err != OCSD_OK) + return err; + +- if(bytesReq == 4) // check that we got all memory requested. ++ // A complete Thumb-16 instruction needs only the first halfword. ++ if((bytesReq == 4) || ((bytesReq >= 2) && (m_instr_info.isa == ocsd_isa_thumb2) && ++ !is_wide_thumb((uint16_t)opcode))) + { + m_instr_info.opcode = opcode; + err = m_pIDecode->first()->DecodeInstruction(&m_instr_info); +diff --git a/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp b/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp +--- a/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp ++++ b/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp +@@ -37,6 +37,7 @@ + #include "opencsd/etmv4/trc_pkt_decode_etmv4i.h" + + #include "common/trc_gen_elem.h" ++#include "i_dec/trc_idec_arminst.h" + + + #define DCD_NAME "DCD_ETMV4" +@@ -1772,13 +1773,14 @@ + // walk iCount instructions + for (int i = 0; i < iCount; i++) + { +- uint32_t opcode; ++ uint32_t opcode = 0; + uint32_t bytesReq = 4; + + err = accessMemory(m_instr_info.instr_addr, getCurrMemSpace(), &bytesReq, (uint8_t *)&opcode); + if (err != OCSD_OK) break; + +- if (bytesReq == 4) // got data back ++ if ((bytesReq == 4) || ((bytesReq >= 2) && (m_instr_info.isa == ocsd_isa_thumb2) && ++ !is_wide_thumb((uint16_t)opcode))) + { + m_instr_info.opcode = opcode; + err = instrDecode(&m_instr_info); +@@ -1845,7 +1847,7 @@ + ocsd_err_t err = OCSD_OK; + TrcStackElemAddr *pElem = dynamic_cast(m_P0_stack.back()); // get the address element + etmv4_addr_val_t srcAddr = pElem->getAddr(); +- uint32_t opcode, bytesReq = 4; ++ uint32_t opcode = 0, bytesReq = 4; + ocsd_vaddr_t currAddr = m_instr_info.instr_addr; // get the latest decoded address. + instr_range_t out_range; + bool bSplitRangeOnN = getComponentOpMode() & ETE_OPFLG_PKTDEC_SRCADDR_N_ATOMS ? true : false; +@@ -1858,7 +1860,8 @@ + return err; + } + +- if (bytesReq != 4) ++ if ((bytesReq != 4) && ((bytesReq < 2) || (m_instr_info.isa != ocsd_isa_thumb2) || ++ is_wide_thumb((uint16_t)opcode))) + { + // can't access - no bytes returned - output nacc. + err = m_out_elem.addElemType(pElem->getRootIndex(), OCSD_GEN_TRC_ELEM_ADDR_NACC); +@@ -1917,6 +1920,7 @@ + while ((instr.instr_addr < out_range.en_addr) && !bMemAccErr) + { + bytesReq = 4; ++ opcode = 0; + err = accessMemory(instr.instr_addr, getCurrMemSpace(), &bytesReq, (uint8_t *)&opcode); + if (err != OCSD_OK) + { +@@ -1924,7 +1928,8 @@ + return err; + } + +- if (bytesReq == 4) ++ if ((bytesReq == 4) || ((bytesReq >= 2) && (instr.isa == ocsd_isa_thumb2) && ++ !is_wide_thumb((uint16_t)opcode))) + { + instr.opcode = opcode; + err = instrDecode(&instr); +@@ -2023,10 +2028,12 @@ + { + // start off by reading next opcode; + bytesReq = 4; ++ opcode = 0; + err = accessMemory(m_instr_info.instr_addr, getCurrMemSpace(),&bytesReq,(uint8_t *)&opcode); + if(err != OCSD_OK) break; + +- if(bytesReq == 4) // got data back ++ if((bytesReq == 4) || ((bytesReq >= 2) && (m_instr_info.isa == ocsd_isa_thumb2) && ++ !is_wide_thumb((uint16_t)opcode))) + { + m_instr_info.opcode = opcode; + err = instrDecode(&m_instr_info); +diff --git a/decoder/source/ptm/trc_pkt_decode_ptm.cpp b/decoder/source/ptm/trc_pkt_decode_ptm.cpp +--- a/decoder/source/ptm/trc_pkt_decode_ptm.cpp ++++ b/decoder/source/ptm/trc_pkt_decode_ptm.cpp +@@ -34,6 +34,7 @@ + + #include + #include "opencsd/ptm/trc_pkt_decode_ptm.h" ++#include "i_dec/trc_idec_arminst.h" + + #define DCD_NAME "DCD_PTM" + +@@ -633,11 +634,13 @@ + { + // start off by reading next opcode; + bytesReq = 4; ++ opcode = 0; + curr_op_address = m_instr_info.instr_addr; // save the start address for the current opcode + err = accessMemory(m_instr_info.instr_addr,mem_space,&bytesReq,(uint8_t *)&opcode); + if(err != OCSD_OK) break; + +- if(bytesReq == 4) // got data back ++ if((bytesReq == 4) || ((bytesReq >= 2) && (m_instr_info.isa == ocsd_isa_thumb2) && ++ !is_wide_thumb((uint16_t)opcode))) + { + m_instr_info.opcode = opcode; + err = instrDecode(&m_instr_info); diff --git a/tools/ctrace/docs/THIRD_PARTY_NOTICES.md b/tools/ctrace/docs/THIRD_PARTY_NOTICES.md index 8cad9fa7b..7262af613 100644 --- a/tools/ctrace/docs/THIRD_PARTY_NOTICES.md +++ b/tools/ctrace/docs/THIRD_PARTY_NOTICES.md @@ -26,9 +26,15 @@ ctrace consumes the repository-wide cxxopts and yaml-cpp targets configured by d `tools/ctrace/cmake/dependencies/ConfigureOpenCSD.cmake` builds OpenCSD's upstream minimal static target, enables large trace-source IDs, and adjusts its -MSVC build options. It applies `tools/ctrace/cmake/dependencies/OpenCSD.patch`, -which replaces an invalid empty-vector subscript in the ITM sync search with -`std::vector::data()`. See the [OpenCSD issue notes](opencsd-issues.md). +MSVC build options. It applies two local patches independently: + +- `tools/ctrace/cmake/dependencies/OpenCSD.patch` replaces an invalid empty-vector + subscript in the ITM sync search with `std::vector::data()`. +- `tools/ctrace/cmake/dependencies/OpenCSD-thumb-fetch.patch` permits complete + Thumb-16 instructions after short memory reads in the ETMv3, ETMv4 and PTM + fetch paths, while retaining inaccessible-code handling for partial instructions. + +See the [OpenCSD issue notes](opencsd-issues.md). The OpenCSD minimal static target still contains upstream decoders other than ITM because OpenCSD 1.8.3 does not expose a supported ITM-only CMake target. diff --git a/tools/ctrace/docs/opencsd-issues.md b/tools/ctrace/docs/opencsd-issues.md index 27c085eb3..58da15991 100644 --- a/tools/ctrace/docs/opencsd-issues.md +++ b/tools/ctrace/docs/opencsd-issues.md @@ -44,6 +44,44 @@ The patch is applied idempotently without resetting the submodule or discarding unrelated local changes. Once OpenCSD releases the fix, update the pinned submodule revision and remove the local patch. +## Complete Thumb-16 instruction rejected after a short memory read + +OpenCSD **1.8.3**, pinned at `59f15320637ce6e510604e7bed550d7ad109d15d`, +requests four bytes before decoding an instruction and treats a shorter read as +inaccessible code, even when the returned two bytes form a complete Thumb-16 +instruction. This affects the ETMv3 code follower, the ETMv4 instruction-fetch +paths (including Q and shared ETE source-address handling), and PTM. + +A synthetic Cortex-M4/ETMv3.5 trace reproduced the defect on 2026-11-09: +`00 bf fd e7` at `0x1000` contains `NOP; B 0x1000`. Six E atoms should produce +`0x1000, 0x1002` three times. The decoder instead emits only `0x1000`, then +`ADDR_NACC(0x1002)`, and still returns success. Adding two unexecuted bytes after +the branch restores all six PCs. Padding images is not a valid remedy: the +branch is already complete, and the image provider must preserve genuine gaps. + +The ctrace build applies +[`OpenCSD-thumb-fetch.patch`](../cmake/dependencies/OpenCSD-thumb-fetch.patch) +independently of the ITM patch. Each affected fetch accepts a short read only +when at least two bytes are available, the ISA is Thumb, and the existing +`is_wide_thumb` predicate identifies a complete 16-bit instruction. Partial +Thumb-32, A32 and A64 instructions remain inaccessible. The opcode buffer is +cleared before each read; instruction decoding and its state changes occur only +after this completeness check. + +The dependency regressions in +[`OpenCsdInstructionBoundaryTests.cpp`](../test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp) +exercise full ETMv3, ETMv4 atom, ETMv4 Q and PTM decoders with an exact memory +callback. They cover two- and three-byte Thumb-16 reads, zero/one-byte reads, +all three incomplete Thumb-32 prefix classes, complete Thumb-32 instructions, +and partial/complete A32 and A64 controls. On macOS arm64 Release, eight valid +Thumb-16 cases fail before the patch; all 21 cases pass afterward, together +with all seven ctrace CTest entries. Reconfiguration leaves the patched source +unchanged. Shared ETE source-address paths were source-reviewed, not runtime-qualified. + +This dependency correction does not implement ctrace ETM output or qualify ETE +support. Cancellation, packet truncation and the IRQ8 field issue below remain +separate defects. + ## ETMv3 canceling exceptions after packed atoms OpenCSD **1.8.3**, pinned at `59f15320637ce6e510604e7bed550d7ad109d15d`, @@ -73,7 +111,7 @@ text fixtures, packet/element output, results and provenance. No executable or build directory is archived. The existing CM4 hardware reference contains no canceling packets, so its successful decode does not validate this case. -No ETMv3 fix is applied. Correct cancellation and packing-invariant output are +No cancellation fix is applied. Correct cancellation and packing-invariant output are required before qualifying the default ETMv3 instruction output. This host test covers the OpenCSD full decoder, not the future production ctrace classifier. diff --git a/tools/ctrace/test/unit/CMakeLists.txt b/tools/ctrace/test/unit/CMakeLists.txt index 9f24584f1..69a7fdbfc 100644 --- a/tools/ctrace/test/unit/CMakeLists.txt +++ b/tools/ctrace/test/unit/CMakeLists.txt @@ -38,6 +38,7 @@ add_executable(CtraceUnitTests src/decode/OpenCsdItmDecoderTests.cpp src/decode/OpenCsdPacketCollectorTests.cpp src/decode/OpenCsdTreeSessionTests.cpp + src/decode/OpenCsdInstructionBoundaryTests.cpp src/diagnostics/DiagnosticsTests.cpp src/diagnostics/RawDiagnosticSinkTests.cpp src/model/TraceSelectionTests.cpp diff --git a/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp b/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp new file mode 100644 index 000000000..9c69762e5 --- /dev/null +++ b/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp @@ -0,0 +1,263 @@ +/* + * Copyright (c) 2026 Arm Limited. All rights reserved. + * + * SPDX-License-Identifier: Apache-2.0 + * Generated with AI + */ + +#include "TestSupport.h" + +#include + +#include "common/ocsd_dcd_tree.h" +#include "common/trc_gen_elem.h" +#include "interfaces/trc_gen_elem_in_i.h" +#include "interfaces/trc_tgt_mem_access_i.h" +#include "opencsd/etmv3/trc_cmp_cfg_etmv3.h" +#include "opencsd/etmv4/trc_cmp_cfg_etmv4.h" +#include "opencsd/ptm/trc_cmp_cfg_ptm.h" + +#include +#include +#include +#include +#include + +/** @brief Exercises the dependency's instruction reads with an exact, unpadded code image. */ +class OpenCsdInstructionBoundaryDecoder final : public ITrcGenElemIn, public ITargetMemAccess { +public: + enum class Protocol { Etmv3, Etmv4, Etmv4Q, Ptm }; + enum class InstructionSet { Thumb, Arm, AArch64 }; + + explicit OpenCsdInstructionBoundaryDecoder(Protocol protocol, std::vector image, + InstructionSet instructionSet = InstructionSet::Thumb) + : m_image(std::move(image)), + m_tree(DecodeTree::CreateDecodeTree(OCSD_TRC_SRC_SINGLE, 0U), DecodeTree::DestroyDecodeTree) + { + require(m_tree != nullptr, "create instruction-boundary decode tree"); + m_tree->setGenTraceElemOutI(this); + m_tree->setMemAccessI(this); + createDecoder(protocol, instructionSet); + const auto raw = traceBytes(protocol, instructionSet); + std::uint32_t processed = 0U; + checkResponse(m_tree->TraceDataIn(OCSD_OP_DATA, 0U, static_cast(raw.size()), raw.data(), &processed)); + require(processed == raw.size(), "consume complete instruction-boundary trace"); + checkResponse(m_tree->TraceDataIn(OCSD_OP_EOT, raw.size(), 0U, nullptr, nullptr)); + } + + /** @brief Copies generic output while the callback-owned element is available. */ + ocsd_datapath_resp_t TraceElemIn(ocsd_trc_index_t, std::uint8_t, const OcsdTraceElement& element) override + { + m_elements.push_back(element); + return OCSD_RESP_CONT; + } + + /** @brief Returns only the bytes present in the image, including short reads. */ + ocsd_err_t ReadTargetMemory(ocsd_vaddr_t address, std::uint8_t, ocsd_mem_space_acc_t, + std::uint32_t* size, std::uint8_t* output) override + { + if (address < CodeAddress || address - CodeAddress >= m_image.size()) { + *size = 0U; + } else { + const auto offset = static_cast(address - CodeAddress); + *size = static_cast(std::min(*size, m_image.size() - offset)); + std::copy_n(m_image.data() + offset, *size, output); + } + return OCSD_OK; + } + + /** @brief The test image is immutable and has no memory cache. */ + void InvalidateMemAccCache(std::uint8_t) override {} + + std::vector elementsOfType(ocsd_gen_trc_elem_t type) const + { + std::vector selected; + for (const auto& element : m_elements) { + if (element.getType() == type) { + selected.push_back(element); + } + } + return selected; + } + + static constexpr ocsd_vaddr_t CodeAddress = 0x1000U; + +private: + void createDecoder(Protocol protocol, InstructionSet instructionSet) + { + if (protocol == Protocol::Etmv3) { + ocsd_etmv3_cfg registers{}; + registers.arch_ver = ARCH_V7; + registers.core_prof = instructionSet == InstructionSet::Thumb ? profile_CortexM : profile_CortexA; + registers.reg_idr = 0x4114f250U; + registers.reg_ctrl = 0x810U; + registers.reg_ccer = 0x18541800U; + registers.reg_trc_id = 6U; + const EtmV3Config config(®isters); + require(m_tree->createDecoder(OCSD_BUILTIN_DCD_ETMV3, OCSD_CREATE_FLG_FULL_DECODER, &config) == OCSD_OK, + "create ETMv3 instruction-boundary decoder"); + } else if (protocol == Protocol::Ptm) { + ocsd_ptm_cfg registers{}; + registers.arch_ver = ARCH_V7; + registers.core_prof = profile_CortexA; + registers.reg_idr = 0x4100f310U; + registers.reg_trc_id = 6U; + const PtmConfig config(®isters); + require(m_tree->createDecoder(OCSD_BUILTIN_DCD_PTM, OCSD_CREATE_FLG_FULL_DECODER, &config) == OCSD_OK, + "create PTM instruction-boundary decoder"); + } else { + ocsd_etmv4_cfg registers{}; + registers.arch_ver = instructionSet == InstructionSet::AArch64 ? ARCH_V8 : ARCH_V7; + registers.core_prof = instructionSet == InstructionSet::Thumb ? profile_CortexM : profile_CortexA; + registers.reg_idr0 = protocol == Protocol::Etmv4Q ? 0x080186e1U : 0x080006e1U; + registers.reg_idr1 = 0x4100f401U; + registers.reg_idr2 = instructionSet == InstructionSet::AArch64 ? 8U : 4U; + registers.reg_idr8 = 1U; + registers.reg_configr = 1U; + registers.reg_traceidr = 6U; + const EtmV4Config config(®isters); + require(m_tree->createDecoder(OCSD_BUILTIN_DCD_ETMV4I, OCSD_CREATE_FLG_FULL_DECODER, &config) == OCSD_OK, + "create ETMv4 instruction-boundary decoder"); + } + } + + static std::vector traceBytes(Protocol protocol, InstructionSet instructionSet) + { + const std::uint8_t thumbBit = instructionSet == InstructionSet::Thumb ? 1U : 0U; + if (protocol == Protocol::Ptm) { + // A-Sync; I-Sync at 0x1000; one executed waypoint. + return {0U, 0U, 0U, 0U, 0U, 0x80U, 0x08U, thumbBit, 0x10U, 0U, 0U, 0x20U, 0x80U}; + } + if (protocol == Protocol::Etmv3) { + // A-Sync; I-Sync at 0x1000; one executed instruction. + return {0U, 0U, 0U, 0U, 0U, 0x80U, 0x08U, 0x20U, thumbBit, 0x10U, 0U, 0U, 0x84U}; + } + // A-Sync; Trace Info; address and context at 0x1000. + std::vector raw = { + 0U, 0U, 0U, 0U, 0U, 0U, 0U, 0U, 0U, 0U, 0U, 0x80U, + 0x01U, 0x01U, 0U}; + if (instructionSet == InstructionSet::Thumb) { + raw.insert(raw.end(), {0x83U, 0U, 0x10U, 0U, 0U, 0x03U}); + } else if (instructionSet == InstructionSet::Arm) { + raw.insert(raw.end(), {0x82U, 0U, 0x08U, 0U, 0U, 0x03U}); + } else { + raw.insert(raw.end(), {0x85U, 0U, 0x08U, 0U, 0U, 0U, 0U, 0U, 0U, 0x13U}); + } + if (protocol == Protocol::Etmv4Q) { + // Q with one instruction and continuation address 0x1000. + raw.insert(raw.end(), {0xabU, 0U, 0x10U, 0U, 0U, 0x01U}); + } else { + raw.push_back(0xf7U); // One executed branch atom. + } + raw.insert(raw.end(), {0x2dU, 0x01U}); // Explicitly commit the atom or Q element. + return raw; + } + + void checkResponse(ocsd_datapath_resp_t response) + { + for (unsigned attempt = 0U; OCSD_DATA_RESP_IS_WAIT(response) && attempt < 8U; ++attempt) { + response = m_tree->TraceDataIn(OCSD_OP_FLUSH, 0U, 0U, nullptr, nullptr); + } + require(!OCSD_DATA_RESP_IS_FATAL(response) && !OCSD_DATA_RESP_IS_WAIT(response), + "complete instruction-boundary decoder operation"); + } + + std::vector m_image; + std::vector m_elements; + std::unique_ptr m_tree; +}; + +class OpenCsdInstructionBoundaryTests : public ::testing::TestWithParam { +protected: + using Decoder = OpenCsdInstructionBoundaryDecoder; + + void expectInstruction(const std::vector& image, std::uint32_t width) + { + const Decoder decoder(GetParam(), image); + const auto ranges = decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE); + ASSERT_EQ(ranges.size(), 1U); + EXPECT_EQ(ranges.front().st_addr, Decoder::CodeAddress); + EXPECT_EQ(ranges.front().en_addr, Decoder::CodeAddress + width); + EXPECT_EQ(ranges.front().num_instr_range, 1U); + EXPECT_EQ(ranges.front().last_instr_sz, width); + EXPECT_TRUE(decoder.elementsOfType(OCSD_GEN_TRC_ELEM_ADDR_NACC).empty()); + EXPECT_TRUE(decoder.elementsOfType(OCSD_GEN_TRC_ELEM_I_RANGE_NOPATH).empty()); + } + + void expectUnavailable(const std::vector& image) + { + const Decoder decoder(GetParam(), image); + EXPECT_TRUE(decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE).empty()); + const auto type = GetParam() == Decoder::Protocol::Etmv4Q ? OCSD_GEN_TRC_ELEM_I_RANGE_NOPATH + : OCSD_GEN_TRC_ELEM_ADDR_NACC; + const auto unavailable = decoder.elementsOfType(type); + ASSERT_EQ(unavailable.size(), 1U); + EXPECT_EQ(unavailable.front().st_addr, Decoder::CodeAddress); + } +}; + +TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb16AtImageEnd) +{ + expectInstruction({0xfeU, 0xe7U}, 2U); // B 0x1000. +} + +TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb16WithOneTrailingByte) +{ + expectInstruction({0xfeU, 0xe7U, 0xffU}, 2U); +} + +TEST_P(OpenCsdInstructionBoundaryTests, RejectsMissingOrPartialFirstHalfword) +{ + expectUnavailable({}); + expectUnavailable({0xfeU}); +} + +TEST_P(OpenCsdInstructionBoundaryTests, RejectsIncompleteThumb32ForEveryWidePrefix) +{ + for (const auto prefix : {0xe8U, 0xf0U, 0xf8U}) { + SCOPED_TRACE(prefix); + expectUnavailable({0U, static_cast(prefix)}); + expectUnavailable({0U, static_cast(prefix), 0xffU}); + } +} + +TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb32AtImageEnd) +{ + expectInstruction({0xffU, 0xf7U, 0xfeU, 0xbfU}, 4U); // B.W 0x1000. +} + +TEST(CtraceUnitTests, testOpenCsdFixedWidthInstructionsStillRequireFourBytes) +{ + using Decoder = OpenCsdInstructionBoundaryDecoder; + for (const auto& [protocol, instructionSet] : { + std::pair{Decoder::Protocol::Etmv3, Decoder::InstructionSet::Arm}, + std::pair{Decoder::Protocol::Etmv4, Decoder::InstructionSet::Arm}, + std::pair{Decoder::Protocol::Ptm, Decoder::InstructionSet::Arm}, + std::pair{Decoder::Protocol::Etmv4, Decoder::InstructionSet::AArch64}}) { + SCOPED_TRACE(static_cast(protocol)); + SCOPED_TRACE(static_cast(instructionSet)); + for (const auto& bytes : {std::vector{0xfeU, 0xe7U}, + std::vector{0xfeU, 0xe7U, 0xffU}}) { + const Decoder decoder(protocol, bytes, instructionSet); + EXPECT_TRUE(decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE).empty()); + const auto missing = decoder.elementsOfType(OCSD_GEN_TRC_ELEM_ADDR_NACC); + ASSERT_EQ(missing.size(), 1U); + EXPECT_EQ(missing.front().st_addr, Decoder::CodeAddress); + } + const auto branch = instructionSet == Decoder::InstructionSet::Arm + ? std::vector{0xfeU, 0xffU, 0xffU, 0xeaU} + : std::vector{0U, 0U, 0U, 0x14U}; + const Decoder decoder(protocol, branch, instructionSet); + const auto ranges = decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE); + ASSERT_EQ(ranges.size(), 1U); + EXPECT_EQ(ranges.front().st_addr, Decoder::CodeAddress); + EXPECT_EQ(ranges.front().en_addr, Decoder::CodeAddress + 4U); + EXPECT_EQ(ranges.front().last_instr_sz, 4U); + } +} + +INSTANTIATE_TEST_SUITE_P(FullDecoders, OpenCsdInstructionBoundaryTests, + ::testing::Values(OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3, + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4, + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q, + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm)); From 60e04b117e7d98305e1f25f40a680c5fadff1489 Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Tue, 10 Nov 2026 11:02:41 +0100 Subject: [PATCH 5/6] docs(ctrace): make external trace references portable --- tools/ctrace/docs/instruction-trace-design.md | 18 +++++++----------- tools/ctrace/docs/opencsd-issues.md | 17 +++++++---------- 2 files changed, 14 insertions(+), 21 deletions(-) diff --git a/tools/ctrace/docs/instruction-trace-design.md b/tools/ctrace/docs/instruction-trace-design.md index e72499650..4118528af 100644 --- a/tools/ctrace/docs/instruction-trace-design.md +++ b/tools/ctrace/docs/instruction-trace-design.md @@ -630,14 +630,15 @@ Details and reproducible defects: [Known OpenCSD issues](opencsd-issues.md). ### 6.1 Reference evidence -Companion workspace: `/Users/thode01/work/github/TraceBufferExamples`; captures retain their original schemas. +Reference artifacts are maintained in the separate `TraceBufferExamples` workspace and are not included in this repository. +Locations below are relative to its `ReferenceTraces/` directory; captures retain their original schemas. -| Reference | Evidence | Remaining limit | +| Reference location | Evidence | Remaining limit | | --- | --- | --- | -| [STM32H7B3I-DK / M7][ref-m7] | 4-KiB ETF, ATB `0x12`; packets, ranges and PC validation. | AXF import parity and production terminal-DISCARD handling. | -| [STM32H747I-DISCO / M4][ref-m4] | 4-KiB ETF, ATB `0x22`; packets and ranges. | AXF import parity, confirmed-tail handling and PC/classifier qualification. | -| [ETMv3 cancellation][ref-cancel] | Synthetic packed/split atom comparison. | Host reproduction. | -| [ETMv3 packet fields][ref-fields] | IRQ8 field and truncated-branch cases. | Host reproduction. | +| `STM32H7B3I-DK-ETMv4/README.md` | M7, 4-KiB ETF, ATB `0x12`; packets/ranges/PCs. | AXF import, DISCARD. | +| `STM32H747I-DISCO-CM4-ETMv3/README.md` | M4, 4-KiB ETF, ATB `0x22`; packets/ranges. | AXF, EOT, PC/classifier. | +| `OpenCSD-ETMv3-Cancellation/README.md` | Synthetic packed/split atom comparison. | Host reproduction. | +| `OpenCSD-ETMv3-PacketFields/README.md` | IRQ8 field and truncated-branch cases. | Host reproduction. | - Hardware bundles contain matching RAW, images, decoder metadata and validation reports; both capture windows wrap. - M7 PC evidence: `validation/pc-expansion/README.md`; hardware/project configuration: each board's `TRACE_SETUP.md`. @@ -919,8 +920,3 @@ OpenCSD links identify the inspected pin; architecture editions/pages are stated [O30]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/include/opencsd/c_api/ocsd_c_api_types.h [O31]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_decode_etmv3.cpp - -[ref-m7]: ../../../../TraceBufferExamples/ReferenceTraces/STM32H7B3I-DK-ETMv4/README.md -[ref-m4]: ../../../../TraceBufferExamples/ReferenceTraces/STM32H747I-DISCO-CM4-ETMv3/README.md -[ref-cancel]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-Cancellation/README.md -[ref-fields]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-PacketFields/README.md diff --git a/tools/ctrace/docs/opencsd-issues.md b/tools/ctrace/docs/opencsd-issues.md index 58da15991..f102eee57 100644 --- a/tools/ctrace/docs/opencsd-issues.md +++ b/tools/ctrace/docs/opencsd-issues.md @@ -105,10 +105,10 @@ The [ETMv3 P-header decoder][etmv3-retain] calls this method to protect the last range, but the later [exception cancellation][etmv3-cancel] can remove only pending elements. The canceled PC has already been exported. -Durable [evidence and reproduction][cancel-evidence] are in the separate TraceBufferExamples -workspace: README, reproduction script, -text fixtures, packet/element output, results and provenance. No executable or -build directory is archived. The existing CM4 hardware reference contains no +Evidence and reproduction are retained in the separate `TraceBufferExamples` workspace: +`ReferenceTraces/OpenCSD-ETMv3-Cancellation/README.md` describes the reproduction script, +text fixtures, packet/element output, results and provenance. These artifacts are not included in this repository. +No executable or build directory is archived. The existing CM4 hardware reference contains no canceling packets, so its successful decode does not validate this case. No cancellation fix is applied. Correct cancellation and packing-invariant output are @@ -145,9 +145,9 @@ Packet and mixed-fallback output must retain the byte as incomplete and suppress unverified address fields. The design already identified this qualification risk; the synthetic case now confirms it without a source patch. -Both packet-field cases, including IRQ7/IRQ9 controls, have [evidence and reproduction][field-evidence] -in the separate TraceBufferExamples workspace, with the C++ probe, reproduction script, fixtures, callback output, -results and provenance. No compiled products are archived. +Both packet-field cases, including IRQ7/IRQ9 controls, are documented in the separate `TraceBufferExamples` workspace +at `ReferenceTraces/OpenCSD-ETMv3-PacketFields/README.md`. The C++ probe, reproduction script, fixtures, callback output, +results and provenance are maintained there, outside this repository. No compiled products are archived. ## ETMv3 cancellation metadata in generic output @@ -187,8 +187,5 @@ and decoder source, not a reproduced wrong-PC claim. [etmv3-irq-text]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_elem_etmv3.cpp#L669-L674 [etmv3-eot]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv3/trc_pkt_proc_etmv3_impl.cpp#L129-L139 -[cancel-evidence]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-Cancellation/README.md -[field-evidence]: ../../../../TraceBufferExamples/ReferenceTraces/OpenCSD-ETMv3-PacketFields/README.md - [etmv4-exception]: https://github.com/Linaro/OpenCSD/blob/59f15320637ce6e510604e7bed550d7ad109d15d/decoder/source/etmv4/trc_pkt_decode_etmv4i.cpp#L1690-L1704 [etmv4-spec]: https://documentation-service.arm.com/static/60017fbb3f22832ff1d6872b From a5e008c57a1dd47de5f1eef0aa6c574951e75845 Mon Sep 17 00:00:00 2001 From: Thorsten de Buhr Date: Tue, 10 Nov 2026 12:10:52 +0100 Subject: [PATCH 6/6] test(ctrace): use direct boundary test registration --- .../OpenCsdInstructionBoundaryTests.cpp | 173 +++++++++++++++--- 1 file changed, 144 insertions(+), 29 deletions(-) diff --git a/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp b/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp index 9c69762e5..9419dd61c 100644 --- a/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp +++ b/tools/ctrace/test/unit/src/decode/OpenCsdInstructionBoundaryTests.cpp @@ -167,13 +167,44 @@ class OpenCsdInstructionBoundaryDecoder final : public ITrcGenElemIn, public ITa std::unique_ptr m_tree; }; -class OpenCsdInstructionBoundaryTests : public ::testing::TestWithParam { -protected: +class OpenCsdInstructionBoundaryChecks { +public: using Decoder = OpenCsdInstructionBoundaryDecoder; - void expectInstruction(const std::vector& image, std::uint32_t width) + static void expectCompleteThumb16AtImageEnd(Decoder::Protocol protocol) { - const Decoder decoder(GetParam(), image); + expectInstruction(protocol, {0xfeU, 0xe7U}, 2U); // B 0x1000. + } + + static void expectCompleteThumb16WithOneTrailingByte(Decoder::Protocol protocol) + { + expectInstruction(protocol, {0xfeU, 0xe7U, 0xffU}, 2U); + } + + static void expectMissingOrPartialFirstHalfword(Decoder::Protocol protocol) + { + expectUnavailable(protocol, {}); + expectUnavailable(protocol, {0xfeU}); + } + + static void expectIncompleteThumb32ForEveryWidePrefix(Decoder::Protocol protocol) + { + for (const auto prefix : {0xe8U, 0xf0U, 0xf8U}) { + SCOPED_TRACE(prefix); + expectUnavailable(protocol, {0U, static_cast(prefix)}); + expectUnavailable(protocol, {0U, static_cast(prefix), 0xffU}); + } + } + + static void expectCompleteThumb32AtImageEnd(Decoder::Protocol protocol) + { + expectInstruction(protocol, {0xffU, 0xf7U, 0xfeU, 0xbfU}, 4U); // B.W 0x1000. + } + +private: + static void expectInstruction(Decoder::Protocol protocol, const std::vector& image, std::uint32_t width) + { + const Decoder decoder(protocol, image); const auto ranges = decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE); ASSERT_EQ(ranges.size(), 1U); EXPECT_EQ(ranges.front().st_addr, Decoder::CodeAddress); @@ -184,46 +215,136 @@ class OpenCsdInstructionBoundaryTests : public ::testing::TestWithParam& image) + static void expectUnavailable(Decoder::Protocol protocol, const std::vector& image) { - const Decoder decoder(GetParam(), image); + const Decoder decoder(protocol, image); EXPECT_TRUE(decoder.elementsOfType(OCSD_GEN_TRC_ELEM_INSTR_RANGE).empty()); - const auto type = GetParam() == Decoder::Protocol::Etmv4Q ? OCSD_GEN_TRC_ELEM_I_RANGE_NOPATH - : OCSD_GEN_TRC_ELEM_ADDR_NACC; + const auto type = protocol == Decoder::Protocol::Etmv4Q ? OCSD_GEN_TRC_ELEM_I_RANGE_NOPATH + : OCSD_GEN_TRC_ELEM_ADDR_NACC; const auto unavailable = decoder.elementsOfType(type); ASSERT_EQ(unavailable.size(), 1U); EXPECT_EQ(unavailable.front().st_addr, Decoder::CodeAddress); } }; -TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb16AtImageEnd) +TEST(CtraceUnitTests, testOpenCsdEtmv3DecodesCompleteThumb16AtImageEnd) { - expectInstruction({0xfeU, 0xe7U}, 2U); // B 0x1000. + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3); } -TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb16WithOneTrailingByte) +TEST(CtraceUnitTests, testOpenCsdEtmv3DecodesCompleteThumb16WithOneTrailingByte) { - expectInstruction({0xfeU, 0xe7U, 0xffU}, 2U); + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16WithOneTrailingByte( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3); } -TEST_P(OpenCsdInstructionBoundaryTests, RejectsMissingOrPartialFirstHalfword) +TEST(CtraceUnitTests, testOpenCsdEtmv3RejectsMissingOrPartialFirstHalfword) { - expectUnavailable({}); - expectUnavailable({0xfeU}); + OpenCsdInstructionBoundaryChecks::expectMissingOrPartialFirstHalfword( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3); } -TEST_P(OpenCsdInstructionBoundaryTests, RejectsIncompleteThumb32ForEveryWidePrefix) +TEST(CtraceUnitTests, testOpenCsdEtmv3RejectsIncompleteThumb32ForEveryWidePrefix) { - for (const auto prefix : {0xe8U, 0xf0U, 0xf8U}) { - SCOPED_TRACE(prefix); - expectUnavailable({0U, static_cast(prefix)}); - expectUnavailable({0U, static_cast(prefix), 0xffU}); - } + OpenCsdInstructionBoundaryChecks::expectIncompleteThumb32ForEveryWidePrefix( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv3DecodesCompleteThumb32AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb32AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4DecodesCompleteThumb16AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4); } -TEST_P(OpenCsdInstructionBoundaryTests, DecodesCompleteThumb32AtImageEnd) +TEST(CtraceUnitTests, testOpenCsdEtmv4DecodesCompleteThumb16WithOneTrailingByte) { - expectInstruction({0xffU, 0xf7U, 0xfeU, 0xbfU}, 4U); // B.W 0x1000. + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16WithOneTrailingByte( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4RejectsMissingOrPartialFirstHalfword) +{ + OpenCsdInstructionBoundaryChecks::expectMissingOrPartialFirstHalfword( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4RejectsIncompleteThumb32ForEveryWidePrefix) +{ + OpenCsdInstructionBoundaryChecks::expectIncompleteThumb32ForEveryWidePrefix( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4DecodesCompleteThumb32AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb32AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4QDecodesCompleteThumb16AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4QDecodesCompleteThumb16WithOneTrailingByte) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16WithOneTrailingByte( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4QRejectsMissingOrPartialFirstHalfword) +{ + OpenCsdInstructionBoundaryChecks::expectMissingOrPartialFirstHalfword( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4QRejectsIncompleteThumb32ForEveryWidePrefix) +{ + OpenCsdInstructionBoundaryChecks::expectIncompleteThumb32ForEveryWidePrefix( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q); +} + +TEST(CtraceUnitTests, testOpenCsdEtmv4QDecodesCompleteThumb32AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb32AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q); +} + +TEST(CtraceUnitTests, testOpenCsdPtmDecodesCompleteThumb16AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm); +} + +TEST(CtraceUnitTests, testOpenCsdPtmDecodesCompleteThumb16WithOneTrailingByte) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb16WithOneTrailingByte( + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm); +} + +TEST(CtraceUnitTests, testOpenCsdPtmRejectsMissingOrPartialFirstHalfword) +{ + OpenCsdInstructionBoundaryChecks::expectMissingOrPartialFirstHalfword( + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm); +} + +TEST(CtraceUnitTests, testOpenCsdPtmRejectsIncompleteThumb32ForEveryWidePrefix) +{ + OpenCsdInstructionBoundaryChecks::expectIncompleteThumb32ForEveryWidePrefix( + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm); +} + +TEST(CtraceUnitTests, testOpenCsdPtmDecodesCompleteThumb32AtImageEnd) +{ + OpenCsdInstructionBoundaryChecks::expectCompleteThumb32AtImageEnd( + OpenCsdInstructionBoundaryDecoder::Protocol::Ptm); } TEST(CtraceUnitTests, testOpenCsdFixedWidthInstructionsStillRequireFourBytes) @@ -255,9 +376,3 @@ TEST(CtraceUnitTests, testOpenCsdFixedWidthInstructionsStillRequireFourBytes) EXPECT_EQ(ranges.front().last_instr_sz, 4U); } } - -INSTANTIATE_TEST_SUITE_P(FullDecoders, OpenCsdInstructionBoundaryTests, - ::testing::Values(OpenCsdInstructionBoundaryDecoder::Protocol::Etmv3, - OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4, - OpenCsdInstructionBoundaryDecoder::Protocol::Etmv4Q, - OpenCsdInstructionBoundaryDecoder::Protocol::Ptm));