Skip to content

Bound concurrency across locale and domain merges - #453

Draft
oliver-kriska wants to merge 1 commit into
elixir-gettext:mainfrom
oliver-kriska:bound-gettext-merge-concurrency
Draft

oliver-kriska wants to merge 1 commit into
elixir-gettext:mainfrom
oliver-kriska:bound-gettext-merge-concurrency

Conversation

@oliver-kriska

Copy link
Copy Markdown
Contributor

Why

Bound the number of simultaneously merging catalogs when gettext.merge processes every locale in a backend directory. Currently it nests a per-domain Task.async_stream inside a per-locale stream; with S schedulers, that can run up to S² file merges at once, each with its own parsed catalogs and merge state.

Flatten the locale/domain pairs into one stream using the existing scheduler-count default. Single-locale and two-file calls keep their existing behavior. This is independent of #451: it limits simultaneously live catalogs rather than changing fuzzy matching.

Measurements

Apple M4 Pro, Elixir 1.20.4 / OTP 29.0.5, MIX_ENV=prod, explicitly limited to four or eight schedulers. Medians of seven alternating fresh-VM runs after a discarded warmup round. Each run starts from fresh input copies and invokes the real merge task implementation (baseline and candidate modules renamed for comparison). Wall time measures the task; peak RSS is /usr/bin/time -l for the entire VM, including startup, loading the comparison modules, fixture copies and output hashing. RSS is not cumulative allocation.

Workload Schedulers Current time → flat time Current peak RSS → flat peak RSS
Real application replay: 176 POs, 16 locales, 11 uneven domains 4 6.369 → 6.391 s 257.6 → 242.4 MiB
Balanced: 8 locales × 8 domains × 2,000 messages, 10 edited messages/domain 4 3.153 → 3.998 s 324.2 → 194.9 MiB
Same balanced workload 8 2.539 → 2.507 s 749.6 → 271.9 MiB
Balanced, both sides also using the fuzzy-pruning patch from PR 451 4 1.638 → 1.971 s 286.5 → 190.4 MiB
Balanced, both sides also using PR 451 8 1.215 → 1.203 s 675.1 → 268.4 MiB

Every run produced identical PO bytes and merge statistics. The private application catalog is not included. The balanced workload repeats an application-derived 2,000-message catalog across eight equally sized domains to expose concurrent catalog memory, rather than relying only on the application's one large domain and ten small ones. Neither workload reproduces the unknown catalog in issue 440.

This is a memory/throughput tradeoff, not a universal speedup: the four-scheduler balanced case is about 27% slower without pruning. I also measured a 2×scheduler flat budget. It recovered that slowdown (2.947 s / 258.3 MiB), but increased the real replay's peak RSS to 319.4 MiB (5.774 s). This draft keeps the ordinary Task default rather than introducing a multiplier tuned to one workload.

Behavior and verification

  • Concurrency is bounded per backend directory, not globally across multiple backends processed concurrently by extract/generate. The eager job list contains locale/path pairs; its metadata grows with locales × domains.
  • Orphan-PO warnings now run after all file jobs succeed, instead of after each locale finishes. On success the warnings are preserved, but grouped after writes. If another locale fails, warnings that previously could have appeared before the failure may now be absent. There is no transactional-write guarantee in either version.
  • Extended the all-locales integration test to cover multiple domains, missing PO/LC_MESSAGES creation, and byte-for-byte preservation of a nonempty orphan PO. Added coverage for warnings when no POT files exist.
  • Full suite: 194 passed on Elixir 1.20.4/OTP 29 and Elixir 1.19.5/OTP 28, with --warnings-as-errors; formatting and whitespace checks pass.
  • Independent read-only review found no blocking defect and called out the warning-timing change documented above.

Use one task stream per backend directory instead of nested locale and domain streams. This limits live parsed catalogs to the scheduler-count worker budget, with a throughput tradeoff on some workloads.

Keep single-locale merging unchanged and scan orphan PO files after all merges complete. Cover missing catalogs and directories, nonempty orphan preservation, and warnings without POT files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants