Skip to content

Execution list single query - #96

Merged
alexanderoster merged 3 commits into
alexanderoster:developfrom
krixkrix:execution-list-single-query
Nov 5, 2026
Merged

alexanderoster merged 3 commits into
alexanderoster:developfrom
krixkrix:execution-list-single-query

Conversation

@krixkrix

Copy link
Copy Markdown

Summary

GET /api/executions (the Build History list, full and ?since= deltas) ran about six SQL queries per execution. It now runs one query per request, whatever the list length.

 CAPIHandler_Executions::handleListExecutionsRequest
   ListJobExecutions / ListJobExecutionsChangedSince        # 1 query
   for each execution
-    GetStatus, GetStatusString, GetDescription             # 1 query each
-    GetEndTimeStampInMicroseconds, ComputeElapsedTime...   # 1 query each
-    RetrieveJob -> HasThumbnailStream                      # 1+ query
+    writeExecutionListEntryToJSON                          # no query
+      GetLoadedStatus, GetLoadedDescription, ...           # values loaded by the list query
  • LibMCData: the execution SELECT now also loads status, description, end time stamp and the job's thumbnail. New getters return them: GetLoadedStatus, GetLoadedStatusString, GetLoadedDescription, GetLoadedEndTimeStampInMicroseconds, ComputeLoadedElapsedTimeInMicroseconds, JobHasThumbnailStream and GetJobThumbnailStreamUUID. The existing getters still read fresh from the database, so plugins see no change.
  • The SELECT was copied into five queries whose column order had to match by hand. It is now one shared CBuildJobExecution::selectQueryWithoutConditions().
  • A damaged job row (missing row, unparseable status, malformed thumbnail UUID) used to fail the whole list and every delta. It now affects only its own entry.
  • DevTools/executionlist_perf/ seeds N executions into a scratch MinimalSystem server and measures the list, so the numbers below can be reproduced.

Evidence

executionlist_perf.py run -n <N>, same seeded database before and after, -O0 dev build, median of 5:

before after
full list, 2000 executions 1212 ms 63 ms
6 concurrent full lists, 2000 executions 8.3 s 0.18 s
100-row ?since= delta, 2000 executions 65 ms 5 ms
full list, 100 executions 23 ms 5 ms
list with one orphaned execution (--damaged 1) request fails 200 OK
  • Before: the cost grows with the length of the list, so a few clients opening Build History on a device with a long history keep amc_server busy for seconds.
    After: the /api/executions response bodies (full, since=0, since=head-10, since=head) are byte-identical to before.
  • New unit tests in LibMCData_BuildJobs all pass, as does the rest of amc_unittest (487 of 487):
    • ListExecsLoadedValues: every execution query loads the right values.
    • ListExecsLoadedIsSnapshot: the loaded getters keep the listed values, while the fresh getters re-read the database.
    • ListExecsMalformedJobRow and ListExecsOrphanedExecution: a damaged row degrades only its own entry.
    • ListExecsJSONEntry: the entry JSON written from the loaded values.

Merge Danger

Door: two-way

The ACT changes only add methods. There is no schema migration, and the JSON format is unchanged.

Blast Radius: small

It touches every place that builds execution objects: ListJobExecutions, ListJobExecutionsChangedSince, RetrieveJobExecution and RetrieveBuildJobExecutions[ByStatus]. A bug in the shared SELECT would therefore show up in all of them. The unit tests cover each of those paths. Plugins that hold long-lived execution objects keep the fresh-reading getters.

@krixkrix
krixkrix force-pushed the execution-list-single-query branch from cf97158 to db44520 Compare November 4, 2026 09:04
DevTools/executionlist_perf seeds N build jobs and executions into a
scratch MinimalSystem server and measures GET /api/executions: the full
list, concurrent loads, ?since= deltas and the /api/frontend snapshot,
with latency, response size and amc_server CPU. It gives repeatable
before/after numbers for the following execution list changes.
Standard library Python only.
Every BuildJobExecution getter used by an execution list (status,
description, end time stamp, elapsed time, and the job's thumbnail via
RetrieveJob) runs its own query, so a list of N executions costs about
6N queries.

The execution SELECT now also loads these values, and new getters
return them without querying again: GetLoadedStatus,
GetLoadedStatusString, GetLoadedDescription,
GetLoadedEndTimeStampInMicroseconds,
ComputeLoadedElapsedTimeInMicroseconds, JobHasThumbnailStream and
GetJobThumbnailStreamUUID. The existing getters still read fresh,
because long-lived plugin objects rely on that.

The SELECT was copied into five queries that had to match the column
order by hand. It now lives in
CBuildJobExecution::selectQueryWithoutConditions().

A damaged job row (missing, unparseable status, malformed thumbnail
UUID) used to make the whole list and every ?since= delta throw. It is
now reported as job status Deleted or as no thumbnail, for its own
entry only.

Unit tests: LibMCData_BuildJobs/ListExecs*.
/api/executions wrote each entry with fresh getters (GetStatus,
GetStatusString, GetDescription, GetEndTimeStamp, ComputeElapsedTime)
and a RetrieveJob for the thumbnail: about six queries per execution,
for the full list and for every ?since= delta.

Each entry is now written by writeExecutionListEntryToJSON from the
values the list query already loaded, so a list request is one query
whatever its length. The JSON is unchanged, and an execution whose job
row is missing no longer fails the request.

At 2000 executions (DevTools/executionlist_perf, debug build) one full
list drops from 1.2 s to 63 ms, six concurrent loads from 8.3 s to
0.18 s, and a 100-row delta from 65 ms to 5 ms. The response bodies
are byte-identical before and after.

Unit test: LibMCData_BuildJobs/ListExecsJSONEntry.
@krixkrix
krixkrix force-pushed the execution-list-single-query branch from db44520 to 162b893 Compare November 5, 2026 15:45
@alexanderoster
alexanderoster merged commit 7301935 into alexanderoster:develop Nov 5, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants