Fixed-dt runs: keep the clock on t_step*dt so restarts continue bitwise

[sbryngelson]

Description

With a fixed dt, a run restarted from step N does not continue the uninterrupted run exactly. For a flapping-wing case (patch_ib%kin_model = 1) on Frontier, a restart from step 27000 differed from the uninterrupted run by 5.9e-4 in rho*u after 270 steps. The difference was deterministic, independent of rank count, and confined to a few cells at the wing root. A pitch-up case (kin_model = 2) happened to agree to 1e-11.

Root cause. Three things make the restarted run and the uninterrupted run use slightly different times:

1. Clock (src/simulation/m_start_up.fpp:656). The uninterrupted run advances mytime = mytime + dt, a running sum that drifts from t_step*dt. A restart starts the clock at t_step*dt (p_main.fpp:55) and evaluates prescribed IB kinematics at t_step_start*dt (m_ibm.fpp:101). With dt = 6.17e-4 the sum is 4.6e-12 (1290 ulps) ahead at step 27000. So for the whole restarted run, the kinematics, inflow ramps and forcing are all evaluated at times offset by that amount.
2. Stage times (m_time_steppers.fpp:845-846). Each RK stage evaluates the prescribed kinematics at mytime + dt (or mytime + dt/2). That is not bitwise (t_step + 1)*dt, which is what a restart evaluates.
3. Last-step dt trim (m_start_up.fpp:610-613). The last step of every fixed-dt run sets dt = finaltime - mytime. That changes the step's dt by the accumulated drift, a step the uninterrupted run takes with the true dt. This alone makes even a restart with no IB differ, at 1e-15.

On their own these are round-off. A moving IB amplifies them: whether a cell is inside the body, or which cell an image point lands in, is a near-tie that a sub-ulp shift can flip. The pitch-ramp test comment in cases.py documents a 5e-13 perturbation moving the field by 5e-3. Whether a restart shows a visible difference therefore depends on whether such a near-tie cell exists near the body at that moment, not on the kinematic model; the flapping and pitch-ramp models use the same code path.

All other state is restored exactly:

• The restart file holds the full-precision conservative fields.
• For kin_model > 0, s_ibm_setup re-evaluates the kinematics analytically at t_init, overriding ib_state_<step>.dat (which is full real(wp) anyway).
• ib_offset_<step>.dat is written ES24.16, which round-trips doubles exactly.

Fix. For a fixed dt only (cfl_dt runs are unchanged):

• advance the clock as mytime = (t_step + 1)*dt;
• form the IB stage times as (t_step + 1)*dt and (t_step + 0.5)*dt;
• drop the last-step trim, since the step count already lands on t_step_stop (the now-unused finaltime is removed).

Is the old behaviour a bias? No. It is a one-time round-off perturbation per restart, amplified locally; it is not a systematic drift. In the production case the integrated force differed by 3e-6 relative over the 270 steps. The cost is that restarted runs are not bit-reproducible continuations, which makes restart checks hard to interpret.

Verification

These runs were on a Frontier CPU compute node (GNU 12.3 + Cray MPICH, Release, 1 rank), with dt = 5e-4 and serial restart I/O. Each compares a straight 0→40 run with a 0→20 run restarted 20→40, bit for bit at step 40:

case	master	this branch
3D pitch ramp (test D7E7DE04's case)	38664 values differ, max 1.8e-15	bitwise identical
same, 100x pitch amplitude	41538 values differ, max 1.6e-14	bitwise identical
same case, static IB	not run	bitwise identical
same case, no IB	not run	bitwise identical

With only the clock change, and before dropping the trim, the IB cases still differed; the no-IB and static-IB cases traced the remainder to the last-step trim. Saving every step versus every 20 versus only at 40 makes no difference, so saving itself perturbs nothing.

Regression: 114 tests that use time-dependent features (all IBM tests including moving and prescribed kinematics, acoustic sources, synthetic turbulence, body forces, restart roundtrips) pass against their existing goldens. Results change only at round-off, so I did not change any goldens. simulation compiles with CCE 19 (CPU) and GNU 12.3. Precheck passes, apart from two test_thermochem cases that fail on the Frontier login node because they compile with the system /usr/bin/gfortran (addressed by #1943).

Not addressed. With cfl_dt, a restart sets mytime = t_save*n_start, while the saved state is at the actual mytime, which can differ from that by up to one dt. This is a separate and larger restart inconsistency.

Contribution Policy

We do not accept pull requests generated primarily by AI without genuine understanding or real-world usage context.

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

---

Acknowledgement

• I confirm this PR meets the above expectations and reflects my own understanding and real-world context.

---

This PR was prepared with the assistance of an AI tool (Claude Code). We found this in restart tests of production runs on Frontier; the fix was exercised on the small cases above.

PR template credit: junegunn

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

• The new comment says “Land exactly on t_stop”, but the following logic only performs… · New
• The comment embeds very specific numeric details (“1290 ulps by step 27000”) that are likely to… · New
• For kind-safety/clarity, consider making the fixed-dt stage-time expressions explicitly… · New

What changed in this PR

This PR makes fixed-dt runs bitwise reproducible across restarts by aligning the simulation clock and immersed-boundary (IB) stage-time evaluation to exact t_step*dt, and by removing last-step dt trimming that depended on accumulated floating-point drift.

Changes:

• Update fixed-dt clock advancement to mytime = (t_step + 1)*dt (keep cfl_dt behavior unchanged).
• Compute IB Runge–Kutta stage times from (t_step + 1)*dt / (t_step + 0.5)*dt for fixed-dt, and pass t_step into propagation routine.
• Remove finaltime and the fixed-dt “last-step trim” logic.

File	Description
src/​simulation/​p_main.fpp	Removes finaltime initialization tied to t_step_stop*dt.
src/​simulation/​m_time_steppers.fpp	Makes IB stage times restart-consistent for fixed-dt by using t_step*dt forms and threads t_step into IB propagation.
src/​simulation/​m_start_up.fpp	Removes fixed-dt last-step dt trim and advances mytime via (t_step + 1)*dt for fixed-dt.
src/​simulation/​m_global_parameters.fpp	Removes global finaltime parameter.

🧠 Review effort: Lite

---

Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_time_steppers.fpp	886	+5
src/simulation/m_global_parameters.fpp	792	-1
src/simulation/m_start_up.fpp	1243	-1
src/simulation/p_main.fpp	73	-1

Directory	Lines	Diff
simulation	28056	+2
total	47027	+2

[codecov]

Codecov Report

❌ Patch coverage is 60.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.65%. Comparing base (3dc5b2f) to head (00c3622).
⚠️ Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
src/simulation/m_time_steppers.fpp	57.14%	0 Missing and 3 partials ⚠️
src/simulation/m_start_up.fpp	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1951   +/-   ##
=======================================
  Coverage   62.64%   62.65%           
=======================================
  Files          86       86           
  Lines       22425    22437   +12     
  Branches     3325     3327    +2     
=======================================
+ Hits        14048    14057    +9     
  Misses       6119     6119           
- Partials     2258     2261    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.