[tamird]

After #542, a rule's dependency labels were still copied into both the traversal cache and the final output. A large fan-out makes those copies expensive even though parents only read the cached digest. Share the list until traversal finishes, then move it into the output whenever there's one owner. When a rule has one sorted, unique list, reuse its allocation and skip building a hash set to deduplicate it.

In a release build on macOS ARM64, generating hashes and edges for a synthetic graph with 2.16 million edges took 1.294 s before and 1.093 s after, with median peak RSS falling from 840 MB to 639 MB. Both JSON files were byte-identical in all six runs per revision. This measures the CLI with frozen Bazel replies; it does not measure Bazel query execution.

With #564 as the baseline, this change reduced the same fixture’s median from 0.801 s to 0.678 s and median peak RSS from 658 MB to 477 MB. The hash and edge files matched byte for byte in all 12 counterbalanced runs.

The Rust and CLI tests and formatting check pass. A regression covers a raw rule and generated alias that both need the same cached edge list.

Reproduce the benchmark

Build bazel build --config=release //:bazel-diff at 1f26908 and this PR's head. Keep both executables. Save the following as bench.py in an empty temporary directory and run python3 bench.py /path/to/before /path/to/after with Python 3.11 or later on macOS or Linux. It prints median elapsed seconds, median native maximum RSS (bytes on macOS, KiB on Linux), and the SHA-256 of both output files.

import hashlib
import json
import os
import statistics
import sys
import time
from pathlib import Path

root = Path(__file__).resolve().parent
groups = [(17, 65848), (896, 1164)]
def varint(n):
    b = bytearray()
    while n >= 128:
        b.append(128 | (n & 127))
        n >>= 7
    b.append(n)
    return bytes(b)
def field(n, s):
    content = s.encode() if isinstance(s, str) else s
    return varint(n * 8 + 2) + varint(len(content)) + content
def frame(data):
    return varint(len(data)) + data
root_map, generated = {}, {}
with (root / 'repos.pb').open('wb') as f:
    for number, (hubs, spokes) in enumerate(groups):
        prefix = f'+extension{number}+'
        for kind, count in [('hub', hubs), ('spoke', spokes)]:
            for i in range(count):
                short = f'{kind}{number}_{i:05}'
                name = prefix + short
                generated[short] = name
                if kind == 'hub':
                    root_map[short] = name
                f.write(frame(field(1, name) + field(2, 'http_archive')))
(root / 'root.json').write_text(json.dumps(root_map, separators=(',', ':')) + '\n')
(root / 'generated.json').write_text(json.dumps(generated, separators=(',', ':')) + '\n')
rule = field(1, '//:root') + field(2, 'test_rule') + b''.join(field(5, '@'+hub+'//:lib') for hub in root_map)
(root / 'query.pb').write_bytes(frame(b'\x08\x01' + field(2, rule)))

replay = root / 'bazel-replay'
replay.write_text(r'''#!/bin/sh
set -eu
d=$(dirname "$0")
case "$1:${2:-}" in
  "version:") echo "Build label: 8.8.2" ;;
  "mod:dump_repo_mapping")
    if [ "$3" = "" ]; then cat "$d/root.json"; else cat "$d/generated.json"; fi ;;
  "mod:graph") echo '{"name":"bench","dependencies":[]}' ;;
  "mod:show_repo") cat "$d/repos.pb" ;;
  "info:output_base") echo "$d" ;;
  query:*)
    while [ "$#" -gt 1 ]; do
      if [ "$1" = "--output_file" ]; then cp "$d/query.pb" "$2"; exit; fi
      shift
    done
    exit 2 ;;
  *) exit 2 ;;
esac
''')
replay.chmod(0o755)

before, after = map(lambda path: Path(path).resolve(), sys.argv[1:])
measurements = {before: [], after: []}
expected = None
for binary in [before, after, after, before] * 3:
    os.environ['RAYON_NUM_THREADS'] = '8'
    command = [str(binary), 'generate-hashes', f'--workspacePath={root}',
               f'--bazelPath={replay}', '--includeTargetType',
               f'--depEdgesFile={root}/edges.json', str(root / 'hashes.json')]
    start = time.monotonic()
    pid = os.fork()
    if pid == 0:
        with open(os.devnull, 'wb') as log:
            os.dup2(log.fileno(), 1)
            os.dup2(log.fileno(), 2)
            os.execv(command[0], command)
    _, status, usage = os.wait4(pid, 0)
    assert os.waitstatus_to_exitcode(status) == 0
    measurements[binary].append((time.monotonic() - start, usage.ru_maxrss))
    sums = []
    for name in ('hashes.json', 'edges.json'):
        with (root / name).open('rb') as result:
            sums.append(hashlib.file_digest(result, 'sha256').hexdigest())
    if expected is None:
        expected = sums
    assert sums == expected
for binary, values in measurements.items():
    print(binary.name, *(statistics.median(column) for column in zip(*values)))
print(*expected)

Codex for Tamir.