Skip to content

Index Variable.measure_id to keep analysis creation fast (179D-310 backport of #894) - #895

Merged
brianlball merged 1 commit into
179D-310from
fix/variable-measure-id-index-179D-310
Nov 3, 2026
Merged

brianlball merged 1 commit into
179D-310from
fix/variable-measure-id-index-179D-310

Conversation

@brianlball

Copy link
Copy Markdown
Contributor

Cherry-pick of #894 (by @anchapin) onto the 179D-310 rollback branch, which serves the 3.10.0-179D-test images where the slowdown was observed. Same two-file change; only the target branch differs. Credit and the deploy note belong to the original PR.

What broke

POST /projects/:id/analyses.json gets slower with every analysis on the server. In a 179D run of 3648 batches it went from ~14 s to ~90 s per batch by batch 1560.

Measure.create_from_os_json calls measure.variables.include?(new_var) once per measure argument. On a has_many proxy that resolves to Ruby's Enumerable#include?, which loads the whole relation with a {measure_id: X} filter (no _id, no limit). variables had no index on measure_id, so every call was a COLLSCAN over the full collection (441 ms at 601k variables on the live server).

What changed

  • server/app/models/variable.rb: belongs_to :measure, optional: true, index: true (Mongoid declares {measure_id: 1}).
  • server/spec/models/variable_spec.rb (new): asserts the measure_id index is declared.

Existing databases pick the index up from rake db:mongoid:create_indexes, which docker/server/start-server.sh runs at every web boot. If the index was already created by hand under a name other than Mongoid's default measure_id_1, drop the hand-made one first or create_indexes will log an IndexOptionsConflict on each boot.

How verified

  • Live server (Index Variable.measure_id to keep analysis creation fast #894): explain went from COLLSCAN 441 ms to IXSCAN 2 ms after adding the index by hand; request time ~90 s to ~5 s.
  • Query shape confirmed locally with Mongoid 8.1.10 query logging: measure.variables.include? issues find variables {measure_id: ...} with no _id; explain on that shape is COLLSCAN without the index.
  • The new spec is collected by the linux-test / macos-test unit-test step (rspec from server/ with tag filters only, no file list; see Index Variable.measure_id to keep analysis creation fast #894 discussion).
  • Branch diff vs develop is OS 3.10 pin + CI/deploy wiring only, so the cherry-pick applied clean.

Measure.create_from_os_json calls measure.variables.include? for each measure
argument. Without an index on measure_id, each call scans the whole variables
collection, so creating an analysis gets slower as analyses build up. With
601k variables, POST /projects/:id/analyses.json took about 90 s; with the
index it takes about 5 s.

(cherry picked from commit eb1e985)
@brianlball

Copy link
Copy Markdown
Contributor Author

@anchapin this look good to also get into the '179D-310' branch?

@brianlball
brianlball merged commit f5035e3 into 179D-310 Nov 3, 2026
5 checks passed
@brianlball
brianlball deleted the fix/variable-measure-id-index-179D-310 branch November 3, 2026 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant