Repository navigation
Conversation
Add a tick source for machines without a usable PMU (most cloud VMs): the recorded program counts its own ticks. A compiler pass (for LLVM or GCC) places a tick so that no instrumented instruction executes twice in the same stack frame without a tick in between (at function entries, loop headers and after returns_twice calls), which is what (ticks, registers) needs to identify a point in the execution. A tick decrements a countdown at a fixed address in the preload thread-locals page and traps when it reaches zero; include/rr/softticks.h is the tracee ABI. Instrumented modules carry a .note.rrsoftticks note. When the program rr starts carries the note, rr records with the new TicksSemantics value `software` and needs no PMU: - PerfCounters programs the countdown instead of perf events (skid 1). The countdown is per address space; rr runs one task at a time, so start() takes it over for the task it resumes after counting the ticks of the task that had it. - Task::did_waitpid turns a tick trap into the TIME_SLICE_SIGNAL stop the PMU interrupt would have caused, so scheduling, signal recording and replay work unchanged. When recording, a task stopped after a tick's decrement and before the end of its sequence is moved to the end: whether the rest runs depends on the countdown's value, which replay programs differently. - The trace header records the ABI version and the countdown's address (Header.softwareTicks). Replay reads the header before constructing the session, so these traces replay without a PMU too. - The preload thread-locals segment is sized to cover the countdown (the kernel does not keep what is written to a shared mapping's page beyond the end of its file). rr parks the countdown when it creates the page, so instrumented programs also work with PMU ticks, and memory checksums ignore it. rr reads and programs the countdown only in the page it created, while that is mapped and writable: a tracee may map over it, and a task that has not exec'd yet may still have the page of an outer rr. - The syscallbuf's force_tick() also executes the tick sequence, so that a task killed between a record's commit and that point is recognized in replay as with the PMU (bufferedSyscallForcedTick). With PMU ticks the countdown is parked and that tick never traps. - The PMU-only checks (ticks before the initial exec, CPUID-bug detection) are skipped. Built for software ticks (__RR_SOFTTICKS__, which the compiler passes predefine), tests with hand-written loops tick in them, mmap_replace_most_mappings keeps the countdown's page and rdtsc_loop2 keeps its rdtsc patchable; otherwise the tests are unchanged. Ticks only come from instrumented code: in code that is not instrumented (e.g. libc, ld.so) the registers alone must tell repetitions apart, which makes reverse execution through long uninstrumented stretches slow or ambiguous. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea of adding a soft-ticks mode to rr has been discussed many times. I think the main reason it never really went anywhere is that there's too much variation in how people think the necessary instrumentation should get into the instrumented binary. So this is my attempt at adding the feature without having to go into the exact specifics of how the instrumentation happens. We simply define the ABI and whatever and however the user wants to satisfy that is not our problem.
The ABI defined is as follows. There is a counter word in rr thread-local preload space (instrumented binaries are responsible for allocating this if the tracer is not attached). The count word is used as follows:
(size_t)-1).i386 is slightly more complicated because the trap word can overflow (underflow) during record, which would give a spurious SIGTRAP if rr is not attached to handle it. To fix this, x86 gets a separate guard word to disable the trap unless rr is attached. Another option would be to use a 64 bit counter there as well (well internally in rr, it's always 64 bits, but it detects overflow and adds), at the cost of one extra instruction per tick increment, but that didn't seem worth it.
This deliberately does not define when ticks happen. The only invariant is that the same register state cannot recur between ticks. A safe invariant is thus to tick on every function entry, backedge and after any returns_twice functions, but instrumenting compilers are permitted to optimize as long as they uphold the invariant.
To decide whether to use soft-ticks mode, rr recognizes a special ELF section. The idea is that this would be set by either a binary that was compiled with appropriate compiler instrumentation or a dynamic binary instrumentation runner that understands the tick format.
Implementation is AI-generated and I need to do a lot more testing, but I wanted to open this early for feedback.