Repository navigation
Bound concurrency across locale and domain merges - #453
Draft
oliver-kriska wants to merge 1 commit into
Draft
oliver-kriska wants to merge 1 commit into
oliver-kriska wants to merge 1 commit into
Conversation
Use one task stream per backend directory instead of nested locale and domain streams. This limits live parsed catalogs to the scheduler-count worker budget, with a throughput tradeoff on some workloads. Keep single-locale merging unchanged and scan orphan PO files after all merges complete. Cover missing catalogs and directories, nonempty orphan preservation, and warnings without POT files.
josevalim
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bound the number of simultaneously merging catalogs when
gettext.mergeprocesses every locale in a backend directory. Currently it nests a per-domainTask.async_streaminside a per-locale stream; with S schedulers, that can run up to S² file merges at once, each with its own parsed catalogs and merge state.Flatten the locale/domain pairs into one stream using the existing scheduler-count default. Single-locale and two-file calls keep their existing behavior. This is independent of #451: it limits simultaneously live catalogs rather than changing fuzzy matching.
Measurements
Apple M4 Pro, Elixir 1.20.4 / OTP 29.0.5,
MIX_ENV=prod, explicitly limited to four or eight schedulers. Medians of seven alternating fresh-VM runs after a discarded warmup round. Each run starts from fresh input copies and invokes the real merge task implementation (baseline and candidate modules renamed for comparison). Wall time measures the task; peak RSS is/usr/bin/time -lfor the entire VM, including startup, loading the comparison modules, fixture copies and output hashing. RSS is not cumulative allocation.Every run produced identical PO bytes and merge statistics. The private application catalog is not included. The balanced workload repeats an application-derived 2,000-message catalog across eight equally sized domains to expose concurrent catalog memory, rather than relying only on the application's one large domain and ten small ones. Neither workload reproduces the unknown catalog in issue 440.
This is a memory/throughput tradeoff, not a universal speedup: the four-scheduler balanced case is about 27% slower without pruning. I also measured a 2×scheduler flat budget. It recovered that slowdown (2.947 s / 258.3 MiB), but increased the real replay's peak RSS to 319.4 MiB (5.774 s). This draft keeps the ordinary Task default rather than introducing a multiplier tuned to one workload.
Behavior and verification
--warnings-as-errors; formatting and whitespace checks pass.