From 1fa74a3603ee023e048aec636440632497402486 Mon Sep 17 00:00:00 2001 From: Rishabh Manoj Date: Sat, 3 Oct 2026 14:33:48 +0000 Subject: [PATCH] chore: repo hygiene, CI consistency and documentation fixes No functional code changes. ruff/pyink pass unchanged; make deps_table_check_updated now passes. README (broken commands / wrong flags): - Fix `per_device_batch_size=.0.25` typo in the Wan 2.1 T2V inference command (fails float()). - Fix `gs:/` -> `gs://` in two output_dir args; remove a shell comment placed after a `\` continuation and a dangling trailing `\` in the Wan LoRA command. - Close the unterminated quote in `pip install "transformer_engine[jax]` and pin ==2.1.0 (as setup.sh). - Point `pip install -r requirements.txt` at the real generated requirements path (no root file). - Fix clone URL org (google -> AI-Hypercomputer). - `cudnn_te_flash` -> `cudnn_flash_te` (registered kernel name); replace non-existent `ici_fsdp_batch_parallelism` with `ici_data_parallelism`. - Remove duplicated `class_prompt=` from the Dreambooth command. - Fix Flux v6e block-size links (#L79-L89 / #L85-L95) and note flux_dev's block is already active. - v5p-128 -> v5p-256 for the 128-chip example (matches the earlier XPK example). - Correct remat policy mention, synthetic-data instructions, test-directory references, TOC (add Flux.2-Klein, Ulysses/Ring/Caching/Tile-size entries), What's-new ordering, and misc typos. docs/: - docs/README.md: drop dangling train_README.md link; index dgx_spark/profiling/metrics/attention docs. - profiling.md: replace Google-internal pantheon.corp.google.com URL with console.cloud.google.com. - first_run.md / run_maxdiffusion_via_xpk.md / data_README.md / dgx_spark.md: typos, wrong clone URL, stale "Tensorflow >= 2.12" requirement, pipeline count (5, add synthetic row). - configs/README.md: document all 22 configs (15 were missing) grouped by model family. Repo hygiene: - Remove stray 1.3 MB test_lightning.png from repo root (tests use tests/images/ copy). - Remove unused _typos.toml and .github/actions/setup-miniconda (uses deprecated actions/cache@v2). - Trim Makefile to targets that exist (diffusers-inherited targets referenced missing dirs/scripts). - Regenerate dependency_versions_table.py so it matches utils/update_dependency_table.py; fix its docstring. - .gitignore: narrow `.gemini/` to `.gemini/*` + `!.gemini/commands/` so the tracked review-command files are no longer shadowed; move Gemini.md entry. CI / tooling: - Workflows touched by this change must pass the org's zizmor security scan, which now enforces hash-pinning as a blanket policy. In the 7 touched workflow files: pin every third-party `uses:` to a commit SHA with a `# vX.Y.Z` comment (same SHAs/format as MaxText; versions are unchanged except where noted); add least-privilege `permissions: contents: read` to CPUTests/UnitTests/UploadDockerImages; replace `secrets: inherit` in gemini-dispatch.yml with the three secrets the called workflows actually use (declared on their workflow_call). - CPUTests.yml: checkout@v3/setup-python@v4 -> v5 (GitHub-hosted runner); fix `pend_to_end` typo. - UnitTests.yml: drop unused isort install and dead commented block referencing AddLabel.yml. - UploadDockerImages.yml: fix copy-pasted header comment. - pre-commit: stop excluding .github/ from whitespace/EOF hooks; normalize existing files. - code_style.sh: warn when local pyink != 23.10.0 (the CI/pre-commit version). - pyproject.toml: JAX-accurate description/keywords; quality/dev extras use pyink/pylint/ruff instead of unused black/isort/hf-doc-builder; ruff line-length aligned to 125. Scripts / Docker: - docker_build_dependency_image.sh: replace header copy-pasted from docker_upload_runner.sh. - maxdiffusion_gpu_dependencies.Dockerfile: dockerfile:1 syntax, typo, remove `RUN ls .` and duplicate WORKDIR. - setup.sh: remove dead commented block; fix "libtpu" in GPU log line. - unit_test_and_lint.sh: remove dead commented pylint line. --- .github/actions/setup-miniconda/action.yml | 146 ----------- .github/workflows/CPUTests.yml | 11 +- .github/workflows/UnitTests.yml | 15 +- .github/workflows/UploadDockerImages.yml | 11 +- .github/workflows/XLML.yml | 2 +- .github/workflows/gemini-dispatch.yml | 20 +- .github/workflows/gemini-invoke.yml | 13 +- .github/workflows/gemini-review.yml | 17 +- .github/workflows/utils/setup_runner.sh | 2 +- .gitignore | 7 +- .pre-commit-config.yaml | 2 - Makefile | 95 ++----- README.md | 107 ++++---- _typos.toml | 13 - code_style.sh | 9 + docker_build_dependency_image.sh | 12 +- docs/README.md | 16 +- docs/data_README.md | 11 +- docs/dgx_spark.md | 12 +- docs/getting_started/first_run.md | 10 +- .../run_maxdiffusion_via_xpk.md | 6 +- docs/profiling.md | 2 +- maxdiffusion_gpu_dependencies.Dockerfile | 9 +- pyproject.toml | 14 +- setup.sh | 11 +- src/maxdiffusion/configs/README.md | 49 +++- src/maxdiffusion/dependency_versions_table.py | 234 +++++++++++++++--- test_lightning.png | Bin 1343016 -> 0 bytes unit_test_and_lint.sh | 2 +- utils/update_dependency_table.py | 2 +- 30 files changed, 438 insertions(+), 422 deletions(-) delete mode 100644 .github/actions/setup-miniconda/action.yml delete mode 100644 _typos.toml delete mode 100644 test_lightning.png diff --git a/.github/actions/setup-miniconda/action.yml b/.github/actions/setup-miniconda/action.yml deleted file mode 100644 index 5f85af7a2..000000000 --- a/.github/actions/setup-miniconda/action.yml +++ /dev/null @@ -1,146 +0,0 @@ -name: Set up conda environment for testing - -description: Sets up miniconda in your ${RUNNER_TEMP} environment and gives you the ${CONDA_RUN} environment variable so you don't have to worry about polluting non-empeheral runners anymore - -inputs: - python-version: - description: If set to any value, dont use sudo to clean the workspace - required: false - type: string - default: "3.10" - miniconda-version: - description: Miniconda version to install - required: false - type: string - default: "4.12.0" - environment-file: - description: Environment file to install dependencies from - required: false - type: string - default: "" - -runs: - using: composite - steps: - # Use the same trick from https://github.com/marketplace/actions/setup-miniconda - # to refresh the cache daily. This is kind of optional though - - name: Get date - id: get-date - shell: bash - run: echo "today=$(/bin/date -u '+%Y%m%d')d" >> $GITHUB_OUTPUT - - name: Setup miniconda cache - id: miniconda-cache - uses: actions/cache@v2 - with: - path: ${{ runner.temp }}/miniconda - key: miniconda-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }} - - name: Install miniconda (${{ inputs.miniconda-version }}) - if: steps.miniconda-cache.outputs.cache-hit != 'true' - env: - MINICONDA_VERSION: ${{ inputs.miniconda-version }} - shell: bash -l {0} - run: | - MINICONDA_INSTALL_PATH="${RUNNER_TEMP}/miniconda" - mkdir -p "${MINICONDA_INSTALL_PATH}" - case ${RUNNER_OS}-${RUNNER_ARCH} in - Linux-X64) - MINICONDA_ARCH="Linux-x86_64" - ;; - macOS-ARM64) - MINICONDA_ARCH="MacOSX-arm64" - ;; - macOS-X64) - MINICONDA_ARCH="MacOSX-x86_64" - ;; - *) - echo "::error::Platform ${RUNNER_OS}-${RUNNER_ARCH} currently unsupported using this action" - exit 1 - ;; - esac - MINICONDA_URL="https://repo.anaconda.com/miniconda/Miniconda3-py39_${MINICONDA_VERSION}-${MINICONDA_ARCH}.sh" - curl -fsSL "${MINICONDA_URL}" -o "${MINICONDA_INSTALL_PATH}/miniconda.sh" - bash "${MINICONDA_INSTALL_PATH}/miniconda.sh" -b -u -p "${MINICONDA_INSTALL_PATH}" - rm -rf "${MINICONDA_INSTALL_PATH}/miniconda.sh" - - name: Update GitHub path to include miniconda install - shell: bash - run: | - MINICONDA_INSTALL_PATH="${RUNNER_TEMP}/miniconda" - echo "${MINICONDA_INSTALL_PATH}/bin" >> $GITHUB_PATH - - name: Setup miniconda env cache (with env file) - id: miniconda-env-cache-env-file - if: ${{ runner.os }} == 'macOS' && ${{ inputs.environment-file }} != '' - uses: actions/cache@v2 - with: - path: ${{ runner.temp }}/conda-python-${{ inputs.python-version }} - key: miniconda-env-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }}-${{ hashFiles(inputs.environment-file) }} - - name: Setup miniconda env cache (without env file) - id: miniconda-env-cache - if: ${{ runner.os }} == 'macOS' && ${{ inputs.environment-file }} == '' - uses: actions/cache@v2 - with: - path: ${{ runner.temp }}/conda-python-${{ inputs.python-version }} - key: miniconda-env-${{ runner.os }}-${{ runner.arch }}-${{ inputs.python-version }}-${{ steps.get-date.outputs.today }} - - name: Setup conda environment with python (v${{ inputs.python-version }}) - if: steps.miniconda-env-cache-env-file.outputs.cache-hit != 'true' && steps.miniconda-env-cache.outputs.cache-hit != 'true' - shell: bash - env: - PYTHON_VERSION: ${{ inputs.python-version }} - ENV_FILE: ${{ inputs.environment-file }} - run: | - CONDA_BASE_ENV="${RUNNER_TEMP}/conda-python-${PYTHON_VERSION}" - ENV_FILE_FLAG="" - if [[ -f "${ENV_FILE}" ]]; then - ENV_FILE_FLAG="--file ${ENV_FILE}" - elif [[ -n "${ENV_FILE}" ]]; then - echo "::warning::Specified env file (${ENV_FILE}) not found, not going to include it" - fi - conda create \ - --yes \ - --prefix "${CONDA_BASE_ENV}" \ - "python=${PYTHON_VERSION}" \ - ${ENV_FILE_FLAG} \ - cmake=3.22 \ - conda-build=3.21 \ - ninja=1.10 \ - pkg-config=0.29 \ - wheel=0.37 - - name: Clone the base conda environment and update GitHub env - shell: bash - env: - PYTHON_VERSION: ${{ inputs.python-version }} - CONDA_BASE_ENV: ${{ runner.temp }}/conda-python-${{ inputs.python-version }} - run: | - CONDA_ENV="${RUNNER_TEMP}/conda_environment_${GITHUB_RUN_ID}" - conda create \ - --yes \ - --prefix "${CONDA_ENV}" \ - --clone "${CONDA_BASE_ENV}" - # TODO: conda-build could not be cloned because it hardcodes the path, so it - # could not be cached - conda install --yes -p ${CONDA_ENV} conda-build=3.21 - echo "CONDA_ENV=${CONDA_ENV}" >> "${GITHUB_ENV}" - echo "CONDA_RUN=conda run -p ${CONDA_ENV} --no-capture-output" >> "${GITHUB_ENV}" - echo "CONDA_BUILD=conda run -p ${CONDA_ENV} conda-build" >> "${GITHUB_ENV}" - echo "CONDA_INSTALL=conda install -p ${CONDA_ENV}" >> "${GITHUB_ENV}" - - name: Get disk space usage and throw an error for low disk space - shell: bash - run: | - echo "Print the available disk space for manual inspection" - df -h - # Set the minimum requirement space to 4GB - MINIMUM_AVAILABLE_SPACE_IN_GB=4 - MINIMUM_AVAILABLE_SPACE_IN_KB=$(($MINIMUM_AVAILABLE_SPACE_IN_GB * 1024 * 1024)) - # Use KB to avoid floating point warning like 3.1GB - df -k | tr -s ' ' | cut -d' ' -f 4,9 | while read -r LINE; - do - AVAIL=$(echo $LINE | cut -f1 -d' ') - MOUNT=$(echo $LINE | cut -f2 -d' ') - if [ "$MOUNT" = "/" ]; then - if [ "$AVAIL" -lt "$MINIMUM_AVAILABLE_SPACE_IN_KB" ]; then - echo "There is only ${AVAIL}KB free space left in $MOUNT, which is less than the minimum requirement of ${MINIMUM_AVAILABLE_SPACE_IN_KB}KB. Please help create an issue to PyTorch Release Engineering via https://github.com/pytorch/test-infra/issues and provide the link to the workflow run." - exit 1; - else - echo "There is ${AVAIL}KB free space left in $MOUNT, continue" - fi - fi - done diff --git a/.github/workflows/CPUTests.yml b/.github/workflows/CPUTests.yml index 2bc75f136..f47818890 100644 --- a/.github/workflows/CPUTests.yml +++ b/.github/workflows/CPUTests.yml @@ -9,6 +9,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true +permissions: + contents: read + jobs: cpu: name: "CPU tests" @@ -18,9 +21,9 @@ jobs: os: [ubuntu-latest] python-version: ['3.12'] steps: - - uses: actions/checkout@v3 + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v4 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: ${{ matrix.python-version }} - name: Install Dependencies @@ -38,7 +41,7 @@ jobs: - name: Analysing the code with pylint in end_to_end/ run: | pylint --fail-under=7 end_to_end/ && \ - echo 'PyLint check on pend_to_end/ is successful' || { echo \ + echo 'PyLint check on end_to_end/ is successful' || { echo \ 'PyLint check has failed. Please run bash code_style.sh to fix issues'; exit 20; } - name: Analysing the code with pyink in maxdiffusion/ run: | @@ -46,5 +49,3 @@ jobs: - name: Analysing the code with pyink in end_to_end/ run: | pyink end_to_end --check --diff --color --pyink-indentation=2 --line-length=125 - - diff --git a/.github/workflows/UnitTests.yml b/.github/workflows/UnitTests.yml index 56d8b2239..98e41ee08 100644 --- a/.github/workflows/UnitTests.yml +++ b/.github/workflows/UnitTests.yml @@ -30,6 +30,9 @@ concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true +permissions: + contents: read + jobs: build: strategy: @@ -39,9 +42,9 @@ jobs: name: "TPU test (${{ matrix.tpu-type }})" runs-on: ["self-hosted","${{ matrix.tpu-type }}"] steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - name: Set up Python 3.12 - uses: actions/setup-python@v5 + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: '3.12' - name: Install dependencies @@ -49,7 +52,6 @@ jobs: bash setup.sh MODE=stable export PATH=$PATH:$HOME/.local/bin pip install ruff - pip install isort pip install pytest - name: Analysing the code with ruff run: | @@ -62,10 +64,3 @@ jobs: run: | #--deselect=src/maxdiffusion/tests/input_pipeline_interface_test.py export LIBTPU_INIT_ARGS='--xla_tpu_scoped_vmem_limit_kib=65536' HF_HUB_CACHE=/mnt/disks/github-runner-disk/ HF_HOME=/mnt/disks/github-runner-disk/ TOKENIZERS_PARALLELISM=false python3 -m pytest --ignore=src/maxdiffusion/kernels/ --deselect=src/maxdiffusion/tests/ltx_transformer_step_test.py -x -# add_pull_ready -# if: github.ref != 'refs/heads/main' -# permissions: -# checks: read -# pull-requests: write -# needs: build -# uses: ./.github/workflows/AddLabel.yml diff --git a/.github/workflows/UploadDockerImages.yml b/.github/workflows/UploadDockerImages.yml index 26bf37056..e20f16ccc 100644 --- a/.github/workflows/UploadDockerImages.yml +++ b/.github/workflows/UploadDockerImages.yml @@ -12,8 +12,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -# This workflow will install Python dependencies, run tests and lint with a variety of Python versions -# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python +# This workflow builds the MaxDiffusion stable and nightly dependency/runner Docker images +# and pushes them to GCR via .github/workflows/build_and_upload_images.sh. name: Build Images @@ -21,14 +21,17 @@ on: schedule: # Run the job daily at 12AM UTC - cron: '0 0 * * *' - + workflow_dispatch: +permissions: + contents: read + jobs: build-image: runs-on: ["self-hosted", "e2", "cpu"] steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 - name: Cleanup old docker images run: docker system prune --all --force - name: build maxdiffusion stable image diff --git a/.github/workflows/XLML.yml b/.github/workflows/XLML.yml index 37f320787..1a4b249e6 100644 --- a/.github/workflows/XLML.yml +++ b/.github/workflows/XLML.yml @@ -19,4 +19,4 @@ jobs: -H "Authorization: token $GITHUB_TOKEN" \ -H "Accept: application/vnd.github.v3+json" \ "https://api.github.com/repos/${{ github.repository }}/issues/$PR_NUMBER/comments" \ - -d '{ "body": "e2e testgrid: https://8bcf50593faf4ea38060e236169827e5-dot-us-central1.composer.googleusercontent.com/dags/maxdiffusion_tpu_e2e/grid" }' \ No newline at end of file + -d '{ "body": "e2e testgrid: https://8bcf50593faf4ea38060e236169827e5-dot-us-central1.composer.googleusercontent.com/dags/maxdiffusion_tpu_e2e/grid" }' diff --git a/.github/workflows/gemini-dispatch.yml b/.github/workflows/gemini-dispatch.yml index 091c947f2..fedf0f521 100644 --- a/.github/workflows/gemini-dispatch.yml +++ b/.github/workflows/gemini-dispatch.yml @@ -20,7 +20,7 @@ defaults: jobs: debugger: # Debug mode: with a repository variable called DEBUG to true - if: |- + if: |- ${{ fromJSON(vars.DEBUG || vars.ACTIONS_STEP_DEBUG || false) }} runs-on: 'ubuntu-latest' permissions: @@ -35,7 +35,7 @@ jobs: DEBUG_event__pull_request__author_association: '${{ github.event.pull_request.author_association }}' DEBUG_event__review__author_association: '${{ github.event.review.author_association }}' DEBUG_event: '${{ toJSON(github.event) }}' - run: |- + run: |- env | grep '^DEBUG_' dispatch: @@ -67,7 +67,7 @@ jobs: id: 'mint_identity_token' if: |- ${{ vars.APP_ID }} - uses: 'actions/create-github-app-token@v2' + uses: 'actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349' # v2.2.2 with: app-id: '${{ vars.APP_ID }}' private-key: '${{ secrets.APP_PRIVATE_KEY }}' @@ -77,7 +77,7 @@ jobs: - name: 'Extract command' id: 'extract_command' - uses: 'actions/github-script@v8' + uses: 'actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd' # v8.0.0 env: EVENT_TYPE: '${{ github.event_name }}.${{ github.event.action }}' REQUEST: '${{ github.event.comment.body || github.event.review.body || github.event.issue.body }}' @@ -126,7 +126,10 @@ jobs: pull-requests: 'write' with: additional_context: '${{ needs.dispatch.outputs.additional_context }}' - secrets: 'inherit' + secrets: + APP_PRIVATE_KEY: '${{ secrets.APP_PRIVATE_KEY }}' + GEMINI_API_KEY: '${{ secrets.GEMINI_API_KEY }}' + GOOGLE_API_KEY: '${{ secrets.GOOGLE_API_KEY }}' invoke: needs: 'dispatch' @@ -140,7 +143,10 @@ jobs: pull-requests: 'write' with: additional_context: '${{ needs.dispatch.outputs.additional_context }}' - secrets: 'inherit' + secrets: + APP_PRIVATE_KEY: '${{ secrets.APP_PRIVATE_KEY }}' + GEMINI_API_KEY: '${{ secrets.GEMINI_API_KEY }}' + GOOGLE_API_KEY: '${{ secrets.GOOGLE_API_KEY }}' fallthrough: needs: @@ -159,7 +165,7 @@ jobs: id: 'mint_identity_token' if: |- ${{ vars.APP_ID }} - uses: 'actions/create-github-app-token@v2' + uses: 'actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349' # v2.2.2 with: app-id: '${{ vars.APP_ID }}' private-key: '${{ secrets.APP_PRIVATE_KEY }}' diff --git a/.github/workflows/gemini-invoke.yml b/.github/workflows/gemini-invoke.yml index 0db244858..94476a030 100644 --- a/.github/workflows/gemini-invoke.yml +++ b/.github/workflows/gemini-invoke.yml @@ -7,6 +7,13 @@ on: type: 'string' description: 'Any additional context from the request' required: false + secrets: + APP_PRIVATE_KEY: + required: false + GEMINI_API_KEY: + required: false + GOOGLE_API_KEY: + required: false concurrency: # any single pull request, only one invoke runs at a time @@ -30,7 +37,7 @@ jobs: id: 'mint_identity_token' if: |- ${{ vars.APP_ID }} - uses: 'actions/create-github-app-token@v2' + uses: 'actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349' # v2.2.2 with: app-id: '${{ vars.APP_ID }}' private-key: '${{ secrets.APP_PRIVATE_KEY }}' @@ -39,9 +46,9 @@ jobs: permission-pull-requests: 'write' - name: 'Run Gemini CLI' - # Trigger Gemini with context + # Trigger Gemini with context id: 'run_gemini' - uses: 'google-github-actions/run-gemini-cli@main' + uses: 'google-github-actions/run-gemini-cli@f77273f4c914e4bf38440cf36a0369cb64a37489' # v0.1.22 env: TITLE: '${{ github.event.pull_request.title || github.event.issue.title }}' DESCRIPTION: '${{ github.event.pull_request.body || github.event.issue.body }}' diff --git a/.github/workflows/gemini-review.yml b/.github/workflows/gemini-review.yml index b1d41e3ef..fe1f0b497 100644 --- a/.github/workflows/gemini-review.yml +++ b/.github/workflows/gemini-review.yml @@ -7,6 +7,13 @@ on: type: 'string' description: 'Any additional context from the request' required: false + secrets: + APP_PRIVATE_KEY: + required: false + GEMINI_API_KEY: + required: false + GOOGLE_API_KEY: + required: false concurrency: # any single pull request, only one review runs at a time @@ -31,7 +38,7 @@ jobs: id: 'mint_identity_token' if: |- ${{ vars.APP_ID }} - uses: 'actions/create-github-app-token@v2' + uses: 'actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349' # v2.2.2 with: app-id: '${{ vars.APP_ID }}' private-key: '${{ secrets.APP_PRIVATE_KEY }}' @@ -41,7 +48,7 @@ jobs: - name: 'Checkout repository' # downloads the code to be analyzed - uses: 'actions/checkout@v5' + uses: 'actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09' # v5.1.0 - name: 'Prepare prompt context' shell: 'bash' @@ -58,8 +65,8 @@ jobs: '{repository: $repo, pull_request_number: $pr, additional_context: $context}' > .gemini/context.json - name: 'Run Gemini pull request review' - # reviews code with detailed set of instructions for the Gemini - uses: 'google-github-actions/run-gemini-cli@main' + # reviews code with detailed set of instructions for the Gemini + uses: 'google-github-actions/run-gemini-cli@f77273f4c914e4bf38440cf36a0369cb64a37489' # v0.1.22 id: 'gemini_pr_review' env: GEMINI_CLI_TRUST_WORKSPACE: 'true' @@ -69,7 +76,7 @@ jobs: PULL_REQUEST_NUMBER: '${{ github.event.pull_request.number || github.event.issue.number }}' REPOSITORY: '${{ github.repository }}' ADDITIONAL_CONTEXT: '${{ inputs.additional_context }}' - with: + with: gcp_location: '${{ vars.GOOGLE_CLOUD_LOCATION }}' gcp_project_id: '${{ vars.GOOGLE_CLOUD_PROJECT }}' gcp_service_account: '${{ vars.SERVICE_ACCOUNT_EMAIL }}' diff --git a/.github/workflows/utils/setup_runner.sh b/.github/workflows/utils/setup_runner.sh index c320a1589..59d3bf054 100644 --- a/.github/workflows/utils/setup_runner.sh +++ b/.github/workflows/utils/setup_runner.sh @@ -17,7 +17,7 @@ # Heavily influenced by # https://github.com/openxla/iree/tree/main/build_tools/github_actions/runner/config -# This file sets up a tpu vm to be used as a github runner for testing. +# This file sets up a tpu vm to be used as a github runner for testing. # It creates a user runner without sudo permissions to # run the config file and authenticate to github diff --git a/.gitignore b/.gitignore index 7106e4f98..44e9114e9 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,6 @@ __pycache__/ *$py.class # C extensions *.so -Gemini.md # tests and logs tests/fixtures/cached_*_text.txt @@ -179,8 +178,10 @@ tags wandb -# Gemini CLI -.gemini/ +# Gemini CLI (local state only; .gemini/commands/ is tracked and used by the review workflows) +.gemini/* +!.gemini/commands/ +Gemini.md gha-creds-*.json # JAX cache diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 8fec6736f..a382cb5ad 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -5,9 +5,7 @@ repos: rev: v5.0.0 hooks: - id: trailing-whitespace - exclude: ^\.github/ - id: end-of-file-fixer - exclude: ^\.github/ - id: check-yaml - id: check-added-large-files args: ['--maxkb=5000'] diff --git a/Makefile b/Makefile index e44ee3c38..895f94730 100644 --- a/Makefile +++ b/Makefile @@ -1,96 +1,31 @@ -.PHONY: deps_table_update modified_only_fixup extra_style_checks quality style fixup fix-copies test test-examples +.PHONY: deps_table_update deps_table_check_updated style quality test -# make sure to test the local checkout in scripts and not the pre-installed one (don't use quotes!) +# Make sure to test the local checkout in scripts and not the pre-installed one (don't use quotes!) export PYTHONPATH = src -check_dirs := examples scripts src tests utils - -modified_only_fixup: - $(eval modified_py_files := $(shell python utils/get_modified_files.py $(check_dirs))) - @if test -n "$(modified_py_files)"; then \ - echo "Checking/fixing $(modified_py_files)"; \ - black $(modified_py_files); \ - ruff $(modified_py_files); \ - else \ - echo "No library .py files were modified"; \ - fi - -# Update src/maxdiffusion/dependency_versions_table.py +# Update src/maxdiffusion/dependency_versions_table.py from the generated requirements. deps_table_update: - @python utils/update_dependency_table.py + @python3 utils/update_dependency_table.py deps_table_check_updated: @md5sum src/maxdiffusion/dependency_versions_table.py > md5sum.saved - @python utils/update_dependency_table.py - @md5sum -c --quiet md5sum.saved || (printf "\nError: the version dependency table is outdated.\nPlease run 'make fixup' or 'make style' and commit the changes.\n\n" && exit 1) - @rm md5sum.saved - -# autogenerating code - -autogenerate_code: deps_table_update - -# Check that the repo is in a good state - -repo-consistency: - python utils/check_dummies.py - python utils/check_repo.py - python utils/check_inits.py - -# this target runs checks on all files - -quality: - black --check $(check_dirs) - ruff $(check_dirs) - doc-builder style src/maxdiffusion docs/source --max_len 119 --check_only --path_to_docs docs/source - python utils/check_doc_toc.py - -# Format source code automatically and check is there are any problems left that need manual fixing - -extra_style_checks: - python utils/custom_init_isort.py - doc-builder style src/maxdiffusion docs/source --max_len 119 --path_to_docs docs/source - python utils/check_doc_toc.py --fix_and_overwrite + @python3 utils/update_dependency_table.py + @md5sum -c --quiet md5sum.saved || (rm -f md5sum.saved; printf "\nError: the version dependency table is outdated.\nPlease run 'make deps_table_update' and commit the changes.\n\n"; exit 1) + @rm -f md5sum.saved -# this target runs checks on all files and potentially modifies some of them +# Format source code with pyink and lint with pylint (same tools as CI). style: - black $(check_dirs) - ruff $(check_dirs) --fix - ${MAKE} autogenerate_code - ${MAKE} extra_style_checks + bash code_style.sh -# Super fast fix and check target that only works on relevant modified files since the branch was made +# Check formatting/lint without modifying files. -fixup: modified_only_fixup extra_style_checks autogenerate_code repo-consistency - -# Make marked copies of snippets of codes conform to the original - -fix-copies: - python utils/check_copies.py --fix_and_overwrite - python utils/check_dummies.py --fix_and_overwrite +quality: + bash code_style.sh --check + ruff check . -# Run tests for the library +# Run the unit tests (CI additionally skips kernels/ and a few TPU-only tests; see .github/workflows/UnitTests.yml). test: - python -m pytest -n auto --dist=loadfile -s -v ./tests/ - -# Run tests for examples - -test-examples: - python -m pytest -n auto --dist=loadfile -s -v ./examples/ - - -# Release stuff - -pre-release: - python utils/release.py - -pre-patch: - python utils/release.py --patch - -post-release: - python utils/release.py --post_release - -post-patch: - python utils/release.py --post_release --patch + python3 -m pytest src/maxdiffusion/tests diff --git a/README.md b/README.md index 911d859b7..3b735eb18 100755 --- a/README.md +++ b/README.md @@ -28,8 +28,8 @@ - **`2026/01/29`**: Wan LoRA for inference is now supported - **`2026/01/15`**: Wan2.1 and Wan2.2 Img2vid generation is now supported - **`2025/11/11`**: Wan2.2 txt2vid generation is now supported -- **`2025/10/10`**: Wan2.1 txt2vid training and generation is now supported. - **`2025/10/14`**: NVIDIA DGX Spark Flux support. +- **`2025/10/10`**: Wan2.1 txt2vid training and generation is now supported. - **`2025/08/14`**: LTX-Video img2vid generation is now supported. - **`2025/07/29`**: LTX-Video text2vid generation is now supported. - **`2025/04/17`**: Flux Finetuning. @@ -71,7 +71,7 @@ MaxDiffusion supports - [Overview](#overview) - [Table of Contents](#table-of-contents) - [Getting Started](#getting-started) - - [Getting Started:](#getting-started-1) + - [Getting Started](#getting-started-1) - [NVIDIA DGX Spark](#nvidia-dgx-spark) - [Training](#training) - [Wan2.1](#wan-21-training) @@ -83,10 +83,15 @@ MaxDiffusion supports - [Dreambooth](#dreambooth) - [Inference](#inference) - [Wan](#wan-models) + - [Ulysses Attention](#ulysses-attention) + - [Caching Mechanisms](#caching-mechanisms) + - [Ring Attention](#ring-attention) + - [Automatic Tile-Size Search](#automatic-tile-size-search) - [LTX-Video](#ltx-video) - [LTX-2 Video](#ltx-2-video) - [Flux](#flux) - - [Fused Attention for GPU](#fused-attention-for-gpu) + - [Flux.2-Klein](#flux2-klein-4b--9b) + - [Fused Attention for GPU](#fused-attention-for-gpu) - [SDXL](#stable-diffusion-xl) - [SD 2 base](#stable-diffusion-2-base) - [SD 2.1](#stable-diffusion-21) @@ -106,11 +111,11 @@ MaxDiffusion supports We recommend starting with a single TPU host and then moving to multihost. -Minimum requirements: Ubuntu Version 22.04, Python 3.12 and Tensorflow >= 2.12.0. +Minimum requirements: Ubuntu Version 22.04 and Python 3.12. -## Getting Started: +## Getting Started -For your first time running Maxdiffusion, we provide specific [instructions](docs/getting_started/first_run.md). +For your first time running MaxDiffusion, we provide specific [instructions](docs/getting_started/first_run.md). ## NVIDIA DGX Spark @@ -122,7 +127,7 @@ After installation completes, run the training script. ## Wan 2.1 Training - in the first part, we'll run on a single host VM to get familiar with the workflow, then run on xpk for large scale training. + In the first part, we'll run on a single host VM to get familiar with the workflow, then run on xpk for large scale training. Although not required, attaching an external disk is recommended as weights take up a lot of disk space. [Follow these instructions if you would like to attach an external disk](https://cloud.google.com/tpu/docs/attach-durable-block-storage). @@ -267,18 +272,18 @@ After installation completes, run the training script. ``` It is important to note a couple of things: - - per_device_batch_size can be a fractional, but must be a whole number when multiplied by number of devices. In this example, 0.25 * 4 (devices) = effective global batch size = 1. - - The step time in v5p-8 with global batch size = 1 is large due to using `FULL` remat. On larger number of chips we can run larger batch sizes greatly increasing MFU, as we will see in the next session of deploying with xpk. + - per_device_batch_size can be fractional, but must be a whole number when multiplied by number of devices. In this example, 0.25 * 4 (devices) = effective global batch size = 1. + - The step time in v5p-8 with global batch size = 1 is large due to the aggressive remat policy (`HIDDEN_STATE_WITH_OFFLOAD`). On a larger number of chips we can run larger batch sizes, greatly increasing MFU, as we will see in the next section on deploying with xpk. - To enable eval during training set `eval_every` to a value > 0. - In Wan2.1, the ici_fsdp_parallelism axis is used for sequence parallelism, the ici_tensor_parallelism axis is used for head parallelism. - You can enable both, keeping in mind that Wan2.1 has 40 heads and 40 must be evenly divisible by ici_tensor_parallelism. - For Sequence parallelism, the code pads the sequence length to evenly divide the sequence. Try out different ici_fsdp_parallelism numbers, but we find 2 and 4 to be the best right now. - - For use on GPU it is recommended to enable the cudnn_te_flash attention kernel for optimal performance. - - Best performance is achieved with the use of batch parallelism, which can be enabled by using the ici_fsdp_batch_parallelism axis. Note that this parallelism strategy does not support fractional batch sizes. - - ici_fsdp_batch_parallelism and ici_fsdp_parallelism can be combined to allow for fractional batch sizes. However, padding is not currently supported for the cudnn_te_flash attention kernel and it is therefore required that the sequence length is divisible by the number of devices in the ici_fsdp_parallelism axis. + - For use on GPU it is recommended to enable the `cudnn_flash_te` attention kernel for optimal performance. + - Best performance is achieved with the use of batch parallelism, which can be enabled by using the ici_data_parallelism axis. Note that this parallelism strategy does not support fractional batch sizes. + - ici_data_parallelism and ici_fsdp_parallelism can be combined to allow for fractional batch sizes. However, padding is not currently supported for the `cudnn_flash_te` attention kernel and it is therefore required that the sequence length is divisible by the number of devices in the ici_fsdp_parallelism axis. - For benchmarking training performance on multiple data dimension input without downloading/re-processing the dataset, the synthetic data iterator is supported. - - Set dataset_type='synthetic' and synthetic_num_samples=null to enable the synthetic data iterator. - - The following overrides on data dimensions are supported: + - Set dataset_type='synthetic' to enable the synthetic data iterator. + - `synthetic_num_samples` (null for infinite) and the data-dimension overrides below are commented out in the base configs; uncomment the ones you need in your YAML (command-line overrides only work for keys present in the YAML): - synthetic_override_height: 720 - synthetic_override_width: 1280 - synthetic_override_num_frames: 85 @@ -312,9 +317,9 @@ After installation completes, run the training script. ### Deploying with XPK - This assumes the user has already created an xpk cluster, installed all dependencies and the also created the dataset from the step above. For getting started with MaxDiffusion and xpk see [this guide](docs/getting_started/run_maxdiffusion_via_xpk.md). + This assumes the user has already created an xpk cluster, installed all dependencies and also created the dataset from the step above. For getting started with MaxDiffusion and xpk see [this guide](docs/getting_started/run_maxdiffusion_via_xpk.md). - Using v5p-256 Then the command to run on xpk is as follows: + Using v5p-256, the command to run on xpk is as follows: ```bash RUN_NAME=jfacevedo-wan-v5p-8-${RANDOM} @@ -431,7 +436,7 @@ After installation completes, run the training script. ### Multi-Host Training with XPK - For large-scale multi-host training across TPU pods or clusters (e.g. v5p, v6e, v7x). The following example is configured for a 128-chip slice (such as `v5p-128`, `v6e-128`, or `tpu7x-4x4x4`): + For large-scale multi-host training across TPU pods or clusters (e.g. v5p, v6e, v7x). The following example is configured for a 128-chip slice (such as `v5p-256`, `v6e-128`, or `tpu7x-4x4x4`): ```bash python3 ~/xpk/xpk.py workload create \ @@ -553,12 +558,13 @@ After installation completes, run the training script. Supported models are **Stable Diffusion 1.x,2.x** ```bash - python src/maxdiffusion/dreambooth/train_dreambooth.py src/maxdiffusion/configs/base14.yml class_data_dir= instance_data_dir= instance_prompt="a photo of ohwx dog" class_prompt="photo of a dog" max_train_steps=150 jax_cache_dir= class_prompt="a photo of a dog" activations_dtype=bfloat16 weights_dtype=float32 per_device_batch_size=1 enable_profiler=False precision=DEFAULT cache_dreambooth_dataset=False learning_rate=4e-6 num_class_images=100 run_name= output_dir=gs:// + python src/maxdiffusion/dreambooth/train_dreambooth.py src/maxdiffusion/configs/base14.yml class_data_dir= instance_data_dir= instance_prompt="a photo of ohwx dog" max_train_steps=150 jax_cache_dir= class_prompt="a photo of a dog" activations_dtype=bfloat16 weights_dtype=float32 per_device_batch_size=1 enable_profiler=False precision=DEFAULT cache_dreambooth_dataset=False learning_rate=4e-6 num_class_images=100 run_name= output_dir=gs:// ``` ## Inference To generate images, run the following command: + ## Stable Diffusion XL Single and Multi host inference is supported with sharding annotations: @@ -604,7 +610,7 @@ To generate images, run the following command: The following command will run LTX-2 T2V: - ```bash + ```bash HF_HUB_CACHE=/mnt/disks/external_disk/maxdiffusion_hf_cache/ \ LIBTPU_INIT_ARGS="--xla_tpu_enable_async_collective_fusion=true \ --xla_tpu_enable_async_collective_fusion_fuse_all_reduce=true \ @@ -655,13 +661,13 @@ To generate images, run the following command: width=1280 \ height=720 \ jax_cache_dir=gs://jfacevedo-maxdiffusion/jax_cache/ \ - per_device_batch_size=.0.25 \ + per_device_batch_size=0.25 \ ici_data_parallelism=2 \ ici_context_parallelism=2 \ flow_shift=5.0 \ enable_profiler=True \ run_name=wan-inference-testing-720p \ - output_dir=gs:/jfacevedo-maxdiffusion \ + output_dir=gs://jfacevedo-maxdiffusion \ fps=16 \ flash_min_seq_length=0 \ flash_block_sizes='{"block_q" : 3024, "block_kv_compute" : 1024, "block_kv" : 2048, "block_q_dkv": 3024, "block_kv_dkv" : 2048, "block_kv_dkv_compute" : 2048, "block_q_dq" : 3024, "block_kv_dq" : 2048 }' \ @@ -676,7 +682,7 @@ To generate images, run the following command: ### Ulysses Attention - MaxDiffusion supports Ulysses attention for WAN TPU inference. Enable it by setting `attention="ulysses"`. + MaxDiffusion supports Ulysses attention for Wan TPU inference. Enable it by setting `attention="ulysses"`. Internally, this follows the Ulysses sequence-parallel attention pattern and trades sequence shards for head shards around the local TPU splash kernel. For background, see [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://arxiv.org/abs/2309.14509). @@ -755,22 +761,22 @@ To generate images, run the following command: use_cfg_cache=True \ ... -# Example: enable MagCache for Wan 2.2 T2V -python src/maxdiffusion/generate_wan.py \ - src/maxdiffusion/configs/base_wan_27b.yml \ - use_magcache=True \ - magcache_thresh=0.04 \ - magcache_K=2 \ - ... + # Example: enable MagCache for Wan 2.2 T2V + python src/maxdiffusion/generate_wan.py \ + src/maxdiffusion/configs/base_wan_27b.yml \ + use_magcache=True \ + magcache_thresh=0.04 \ + magcache_K=2 \ + ... -# Example: enable MagCache for Wan 2.2 I2V -python src/maxdiffusion/generate_wan.py \ - src/maxdiffusion/configs/base_wan_i2v_27b.yml \ - use_magcache=True \ - magcache_thresh=0.06 \ - magcache_K=2 \ - ... -``` + # Example: enable MagCache for Wan 2.2 I2V + python src/maxdiffusion/generate_wan.py \ + src/maxdiffusion/configs/base_wan_i2v_27b.yml \ + use_magcache=True \ + magcache_thresh=0.06 \ + magcache_K=2 \ + ... + ``` ### Ring Attention We added ring attention support for Wan models. Below are the stats for one `720p` (81 frames) video generation (with CFG DP): @@ -820,7 +826,7 @@ The optimal attention tile sizes (`block_q` / `block_kv`) depend on the sequence python src/maxdiffusion/generate_flux.py src/maxdiffusion/configs/base_flux_dev.yml jax_cache_dir=/tmp/cache_dir run_name=flux_test output_dir=/tmp/ prompt="photograph of an electronics chip in the shape of a race car with trillium written on its side" per_device_batch_size=1 ``` - If you are using a TPU v6e (Trillium), you can use optimized flash block sizes for faster inference. Uncomment Flux-dev [config](src/maxdiffusion/configs/base_flux_dev.yml#60) and Flux-schnell [config](src/maxdiffusion/configs/base_flux_schnell.yml#68) + If you are using a TPU v6e (Trillium), you can use optimized flash block sizes for faster inference. They are already enabled in the Flux-dev [config](src/maxdiffusion/configs/base_flux_dev.yml#L79-L89); for Flux-schnell, uncomment the `flash_block_sizes` block in its [config](src/maxdiffusion/configs/base_flux_schnell.yml#L85-L95). To keep text encoders, vae and transformer on HBM memory at all times, the following command shards the model across devices. @@ -861,15 +867,15 @@ The optimal attention tile sizes (`block_q` / `block_kv`) depend on the sequence ```bash python src/maxdiffusion/generate_flux2klein.py src/maxdiffusion/configs/base_flux2klein_9B.yml run_name=flux2klein_9b_kv_edit prompt="change the lighting to evening" image_paths="['src/maxdiffusion/tests/images/flux2klein/ref_flux2klein_9b.png']" use_kv=True ``` - ## Fused Attention for GPU: + ## Fused Attention for GPU Fused Attention for GPU is supported via TransformerEngine. Installation instructions: ```bash cd maxdiffusion pip install -U "jax[cuda12]" - pip install -r requirements.txt + pip install -r dependencies/requirements/generated_requirements/requirements.txt pip install --upgrade torch torchvision - pip install "transformer_engine[jax] + pip install "transformer_engine[jax]==2.1.0" pip install . ``` @@ -882,7 +888,7 @@ The optimal attention tile sizes (`block_q` / `block_kv`) depend on the sequence Disclaimer: not all LoRA formats have been tested. Currently supports ComfyUI and AI Toolkit formats. If there is a specific LoRA that doesn't load, please let us know. - First create a copy of the relevant config file eg: `src/maxdiffusion/configs/base_wan_{*}.yml`. Update the prompt and LoRA details in the config. Make sure to set `enable_lora: True`. Then run the following command: + First create a copy of the relevant config file, e.g. `src/maxdiffusion/configs/base_wan_{*}.yml`. Update the prompt and LoRA details in the config. Make sure to set `enable_lora: True`. Then run the following command (replace `base_wan_i2v_14b.yml` with your copy): ```bash HF_HUB_CACHE=/mnt/disks/external_disk/maxdiffusion_hf_cache/ \ @@ -893,15 +899,15 @@ The optimal attention tile sizes (`block_q` / `block_kv`) depend on the sequence --xla_enable_async_all_reduce=true" \ HF_HUB_ENABLE_HF_TRANSFER=1 \ python src/maxdiffusion/generate_wan.py \ - src/maxdiffusion/configs/base_wan_i2v_14b.yml \ # --> Change to your copy + src/maxdiffusion/configs/base_wan_i2v_14b.yml \ jax_cache_dir=gs://jfacevedo-maxdiffusion/jax_cache/ \ per_device_batch_size=.125 \ ici_data_parallelism=2 \ ici_context_parallelism=2 \ run_name=wan-lora-inference-testing-720p \ - output_dir=gs:/jfacevedo-maxdiffusion \ + output_dir=gs://jfacevedo-maxdiffusion \ seed=118445 \ - enable_lora=True \ + enable_lora=True ``` Loading multiple LoRAs is supported as well. @@ -972,21 +978,21 @@ ZONE= PROJECT_ID= gcloud compute tpus tpu-vm ssh $TPU_NAME --zone=$ZONE --project $PROJECT_ID --worker=all --command=" export LIBTPU_INIT_ARGS="" -git clone https://github.com/google/maxdiffusion +git clone https://github.com/AI-Hypercomputer/maxdiffusion cd maxdiffusion pip3 install jax[tpu] -f https://storage.googleapis.com/jax-releases/libtpu_releases.html -pip3 install -r requirements.txt +pip3 install -r dependencies/requirements/generated_requirements/requirements.txt pip3 install . python -m src.maxdiffusion.train src/maxdiffusion/configs/base_2_base.yml run_name=my_run output_dir=gs://your-bucket/" ``` # Comparison to Alternatives -MaxDiffusion started as a fork of [Diffusers](https://github.com/huggingface/diffusers), a Hugging Face diffusion library written in Python, Pytorch and Jax. MaxDiffusion is compatible with Hugging Face Jax models. MaxDiffusion is more complex and was designed to run distributed across TPU Pods. +MaxDiffusion started as a fork of [Diffusers](https://github.com/huggingface/diffusers), a Hugging Face diffusion library written in Python, PyTorch and JAX. MaxDiffusion is compatible with Hugging Face JAX models. MaxDiffusion is more complex and was designed to run distributed across TPU Pods. # Development -Whether you are forking MaxDiffusion for your own needs or intending to contribute back to the community, a full suite of tests can be found in `tests` and `src/maxdiffusion/tests`. +Whether you are forking MaxDiffusion for your own needs or intending to contribute back to the community, a full suite of unit tests can be found in `src/maxdiffusion/tests` and end-to-end tests in `end_to_end/`. To run unit tests simply run: ```bash @@ -1028,8 +1034,7 @@ bash code_style.sh This script will automatically format your code with `pyink` and help you identify any remaining style issues. - -The full suite of -end-to end tests is in `tests` and `src/maxdiffusion/tests`. We run them with a nightly cadance. +The full suite of end-to-end tests is in `end_to_end/`. We run them with a nightly cadence. ## Profiling To learn how to enable ML Diagnostics and XProf profiling for your runs, please see our [ML Diagnostics Guide](docs/profiling.md). diff --git a/_typos.toml b/_typos.toml deleted file mode 100644 index 551099f98..000000000 --- a/_typos.toml +++ /dev/null @@ -1,13 +0,0 @@ -# Files for typos -# Instruction: https://github.com/marketplace/actions/typos-action#getting-started - -[default.extend-identifiers] - -[default.extend-words] -NIN="NIN" # NIN is used in scripts/convert_ncsnpp_original_checkpoint_to_diffusers.py -nd="np" # nd may be np (numpy) -parms="parms" # parms is used in scripts/convert_original_stable_diffusion_to_diffusers.py - - -[files] -extend-exclude = ["_typos.toml"] diff --git a/code_style.sh b/code_style.sh index f99c4588c..ac8ac88c8 100755 --- a/code_style.sh +++ b/code_style.sh @@ -19,6 +19,15 @@ set -e # Exit immediately if any command fails FOLDERS_TO_FORMAT=("src/maxdiffusion" "end_to_end/tpu") LINE_LENGTH=$(grep -E "^max-line-length=" pylintrc | cut -d '=' -f 2) +# Keep in sync with .pre-commit-config.yaml and .github/workflows/CPUTests.yml. +EXPECTED_PYINK_VERSION="23.10.0" + +# Different pyink versions can format differently; warn if the local version won't match CI. +INSTALLED_PYINK_VERSION=$(pyink --version 2>/dev/null | head -n 1 | grep -oE '[0-9]+\.[0-9]+\.[0-9]+' | head -n 1) +if [[ "${INSTALLED_PYINK_VERSION}" != "${EXPECTED_PYINK_VERSION}" ]]; then + echo -e "\e[33mWARNING: CI formats with pyink ${EXPECTED_PYINK_VERSION} but you have '${INSTALLED_PYINK_VERSION:-none}'." \ + "Results may differ from CI. Install the pinned version with: pip install pyink==${EXPECTED_PYINK_VERSION}\e[0m" +fi # Check for --check flag CHECK_ONLY_PYINK_FLAGS="" diff --git a/docker_build_dependency_image.sh b/docker_build_dependency_image.sh index ab5512139..a34760415 100644 --- a/docker_build_dependency_image.sh +++ b/docker_build_dependency_image.sh @@ -14,14 +14,16 @@ # See the License for the specific language governing permissions and # limitations under the License. -# This scripts takes a docker image that already contains the MaxDiffusion dependencies, copies the local source code in and -# uploads that image into GCR. Once in GCR the docker image can be used for development. +# This script builds a local docker image (maxdiffusion_base_image) containing all MaxDiffusion dependencies for the +# requested MODE/DEVICE. It does not push anything; use "bash docker_upload_runner.sh" afterwards to layer in the local +# source code and upload the result to GCR. -# Each time you update the base image via a "bash docker_maxdiffusion_image_upload.sh", there will be a slow upload process -# (minutes). However, if you are simply changing local code and not updating dependencies, uploading just takes a few seconds. +# Rebuilding this base image is slow (minutes) but only needed when dependencies change. If you are only changing local +# code, re-run docker_upload_runner.sh instead, which takes a few seconds. -# bash docker_build_dependency_image.sh MODE=stable JAX_VERSION=0.4.13 +# bash docker_build_dependency_image.sh MODE=stable JAX_VERSION=0.9.0 # bash docker_build_dependency_image.sh MODE=stable +# bash docker_build_dependency_image.sh MODE=nightly DEVICE=gpu set -e diff --git a/docs/README.md b/docs/README.md index 543b95943..5c41a595c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,7 +5,8 @@ This folder contains documentation for getting started with and using MaxDiffusi ## Getting Started * **[First Run](getting_started/first_run.md)** - Provides instructions for setting up and running MaxDiffusion for the first time. -* **[Running MaxDiffusion via XPK](getting_started/run_maxdiffusion_via_xpk.md)** - Explains how to run MaxDiffusion using the XPK format. +* **[Running MaxDiffusion via XPK](getting_started/run_maxdiffusion_via_xpk.md)** - Explains how to run MaxDiffusion on GKE using XPK. +* **[NVIDIA DGX Spark](dgx_spark.md)** - Explains how to run MaxDiffusion on an NVIDIA DGX Spark. ## Contributing & Community @@ -14,8 +15,17 @@ This folder contains documentation for getting started with and using MaxDiffusi ## Training -* **[Common Training Guide](train_README.md)** - Provides a comprehensive guide to training MaxDiffusion models, including script usage, configuration options, and sharding strategies. +* **[Training Guide](../README.md#training)** - Per-model training walkthroughs (Wan 2.1 / 2.2, Flux, SDXL, SD 2 base, SD 1.4, Dreambooth) live in the main README. ## Data Input -* **[Common Data Input Guide](data_README.md)** - Provides a comprehensive guide to data input pipelines. +* **[Common Data Input Guide](data_README.md)** - Provides a comprehensive guide to data input pipelines. + +## Observability + +* **[Profiling](profiling.md)** - How to enable ML Diagnostics and XProf profiling for your runs. +* **[Metrics](metrics.md)** - How to enable ML Diagnostics metrics tracking for your runs. + +## Internals + +* **[Attention Block Sizes](attention_blocks_flowchart.md)** - Explains the flash-attention `block_*` tiling parameters and how they relate to each other. diff --git a/docs/data_README.md b/docs/data_README.md index 02c11ef09..42613e35f 100644 --- a/docs/data_README.md +++ b/docs/data_README.md @@ -1,13 +1,14 @@ # Data Input Guide ## Overview -Currently MaxDiffusion supports 3 data input pipelines, controlled by the flag `dataset_type` +Currently MaxDiffusion supports 5 data input pipelines, controlled by the flag `dataset_type` | Pipeline | Dataset Location | Dataset formats | Features or limitations | | -------- | ---------------- | --------------- | ----------------------- | | HuggingFace (hf) | datasets in HuggingFace Hub or local/Cloud Storage | Formats supported in HF Hub: parquet, arrow, json, csv, txt | data are not loaded in memory but streamed from the saved location, good for big dataset | -| tf | dataset will be downloaded form HuggingFace Hub to disk | Formats supported in HF Hub: parquet, arrow, json, csv, txt | Will read the whole dataset into memory, works for small dataset | +| tf | dataset will be downloaded from HuggingFace Hub to disk | Formats supported in HF Hub: parquet, arrow, json, csv, txt | Will read the whole dataset into memory, works for small dataset | | tfrecord | local/Cloud Storage | TFRecord | data are not loaded in memory but streamed from the saved location, good for big dataset | | Grain | local/Cloud Storage | ArrayRecord (or any random access format) | data are not loaded in memory but streamed from the saved location, good for big dataset, supports global shuffle and data iterator checkpoint for determinism (see details in [doc](https://github.com/AI-Hypercomputer/maxtext/blob/main/getting_started/Data_Input_Pipeline.md#grain-pipeline---for-determinism)) | +| synthetic | n/a (generated on device) | n/a | random tensors of the configured shape; no I/O, useful for performance benchmarking and pipeline debugging (see the `synthetic_*` keys in the Wan configs) | ## Usage examples @@ -54,7 +55,7 @@ grain_train_files: gs:////*.arrayrecord # match the file patter ## Best Practice ### Multihost Dataloading -In multihost environment, if use a streaming type of input pipeline and the data format only supports sequential reads (dataset_type in (hf, tfrecord in MaxDiffusion)), the most performant way is to have each data file only accessed by one host, and each host access a subset of data files (shuffle is within the subset of files). This requires (# of data files) > (# of hosts loading data). We recommand users to reshard the dataset if this requirement is not met. +In a multihost environment, if using a streaming type of input pipeline and the data format only supports sequential reads (dataset_type in (hf, tfrecord in MaxDiffusion)), the most performant way is to have each data file only accessed by one host, and each host access a subset of data files (shuffle is within the subset of files). This requires (# of data files) > (# of hosts loading data). We recommend users reshard the dataset if this requirement is not met. #### HuggingFace pipeline when streaming from Hub -* When (# of data files) >= (# of hosts loading data), assign files to each host as evenly as possible, some host may ended up with 1 file more than the others. When a host run out of data, it will automatically start another epoch. Since each host run out of data at different speed, different host come to next epoch at different time. -* When (# of data files) < (# of hosts loading data), files are read sequentially with multiple hosts accessing each file, perf can degrade quickly as # of host increases. +* When (# of data files) >= (# of hosts loading data), files are assigned to each host as evenly as possible; some hosts may end up with 1 file more than the others. When a host runs out of data, it will automatically start another epoch. Since each host runs out of data at a different speed, different hosts reach the next epoch at different times. +* When (# of data files) < (# of hosts loading data), files are read sequentially with multiple hosts accessing each file, and perf can degrade quickly as the # of hosts increases. diff --git a/docs/dgx_spark.md b/docs/dgx_spark.md index c0b3efa02..1bd5e74e9 100644 --- a/docs/dgx_spark.md +++ b/docs/dgx_spark.md @@ -1,4 +1,4 @@ -# MaxDiffusion on Nvidia DGX Spark GPU: A complete User Guide +# MaxDiffusion on NVIDIA DGX Spark GPU: A complete User Guide This guide provides a detailed step-by-step walkthrough for setting up and running the maxdiffusion library within a custom Docker environment on an ARM-based machine with NVIDIA GPU support. We will cover everything from building the optimized Docker image to generating your first image and retrieving it successfully. @@ -15,13 +15,13 @@ Before you begin, ensure you have the following: The foundation of a smooth workflow is a well-built Docker image. The following Dockerfile is optimized for build speed by caching dependencies, ensuring that code changes don't require a full reinstall of all libraries. -### Step1: Create the Dockerfile +### Step 1: Create the Dockerfile In the root directory of your maxdiffusion project, create a file named box.Dockerfile and paste the following content into it. ```docker -# Nvidia Base image for ARM64 with CUDA support -# As JAX AI Image as it currently doesn't support ARM builds. +# NVIDIA base image for ARM64 with CUDA support. +# Used instead of the JAX AI Image, which currently doesn't support ARM builds. FROM nvcr.io/nvidia/cuda-dl-base@sha256:3631d968c12ef22b1dfe604de63dbc71a55f3ffcc23a085677a6d539d98884a4 # Set environment variables (these rarely change) @@ -56,7 +56,7 @@ RUN pip install . CMD ["/bin/bash"] ``` -### Step2: Build the Image +### Step 2: Build the Image Open your terminal on DGX Spark, navigate to the root directory of the maxdiffusion project, and run the build command: @@ -101,7 +101,7 @@ huggingface-cli login You will be prompted to paste a Hugging Face User Access Token. -1. Go to[ huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) in your web browser. +1. Go to [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) in your web browser. 2. Copy your token (or create a new one with write permissions). diff --git a/docs/getting_started/first_run.md b/docs/getting_started/first_run.md index 54531a36e..ccc452557 100644 --- a/docs/getting_started/first_run.md +++ b/docs/getting_started/first_run.md @@ -8,9 +8,9 @@ We recommend starting with a single host first and then moving to multihost. Local development is a convenient way to run MaxDiffusion on a single host. It doesn't scale to multiple hosts. -1. [Create and SSH to a single-host TPU (v6-8). ](https://cloud.google.com/tpu/docs/users-guide-tpu-vm#creating_a_cloud_tpu_vm_with_gcloud) -* You can find here [here](https://cloud.google.com/tpu/docs/regions-zones) the list of zones that support the v6(Trillium) TPUs -* We recommend using the base VM image "v2-alpha-tpuv6e", which meets the version requirements: Ubuntu Version 22.04, Python 3.12 and Tensorflow >= 2.12.0 +1. [Create and SSH to a single-host TPU (v6e-8). ](https://cloud.google.com/tpu/docs/users-guide-tpu-vm#creating_a_cloud_tpu_vm_with_gcloud) +* You can find [here](https://cloud.google.com/tpu/docs/regions-zones) the list of zones that support the v6e (Trillium) TPUs +* We recommend using the base VM image "v2-alpha-tpuv6e", which meets the version requirements: Ubuntu Version 22.04 and Python 3.12 1. Clone MaxDiffusion in your TPU VM. ``` @@ -24,13 +24,13 @@ cd maxdiffusion bash setup.sh MODE=stable DEVICE=tpu ``` -1. Active your virtual environment: +1. Activate your virtual environment: ``` # Replace with your virtual environment name if not using this default name venv_name="maxdiffusion_venv" source ~/$venv_name/bin/activate ``` -## Getting Starting: Multihost development +## Getting Started: Multihost development [GKE, recommended] [Running MaxDiffusion with xpk](run_maxdiffusion_via_xpk.md) - Quick Experimentation and Production support diff --git a/docs/getting_started/run_maxdiffusion_via_xpk.md b/docs/getting_started/run_maxdiffusion_via_xpk.md index 78204a481..35452a5ab 100644 --- a/docs/getting_started/run_maxdiffusion_via_xpk.md +++ b/docs/getting_started/run_maxdiffusion_via_xpk.md @@ -50,8 +50,8 @@ after which log out and log back in to the machine. 1. Git clone MaxDiffusion locally ```shell - git clone https://github.com/google/MaxDiffusion.git - cd MaxDiffusion + git clone https://github.com/AI-Hypercomputer/maxdiffusion.git + cd maxdiffusion ``` 2. Build local MaxDiffusion docker image @@ -74,7 +74,7 @@ after which log out and log back in to the machine. gcloud config set project $PROJECT_ID gcloud config set compute/zone $ZONE - # See instructions in README.me to create below buckets. + # See instructions in README.md to create the buckets below. BASE_OUTPUT_DIR=gs://output_bucket/ DATASET_PATH=gs://dataset_bucket/ diff --git a/docs/profiling.md b/docs/profiling.md index 3a8c010bc..e74ce2a24 100644 --- a/docs/profiling.md +++ b/docs/profiling.md @@ -31,4 +31,4 @@ If permissions are not configured correctly, your job will fail with an error si ## 4. Viewing Your Runs Once your job is running with diagnostics enabled, you can monitor the profiles, execution times, and metrics in the Cluster Director console here: -🔗 **https://pantheon.corp.google.com/cluster-director/diagnostics** +🔗 **https://console.cloud.google.com/cluster-director/diagnostics** diff --git a/maxdiffusion_gpu_dependencies.Dockerfile b/maxdiffusion_gpu_dependencies.Dockerfile index ba369cf52..0fd83efe8 100644 --- a/maxdiffusion_gpu_dependencies.Dockerfile +++ b/maxdiffusion_gpu_dependencies.Dockerfile @@ -1,5 +1,5 @@ -# syntax=docker/dockerfile:experimental -# Note: This pulls in the lastest of jax:base +# syntax=docker/dockerfile:1 +# Note: This pulls in the latest of jax:base ARG BASEIMAGE=ghcr.io/nvidia/jax:base FROM $BASEIMAGE @@ -18,8 +18,6 @@ RUN apt-get update && \ # Set environment variables for Google Cloud SDK ENV PATH="/usr/local/google-cloud-sdk/bin:${PATH}" - - ARG MODE ENV ENV_MODE=$MODE @@ -36,9 +34,6 @@ WORKDIR /deps # Copy all files from local workspace into docker container COPY . . -RUN ls . RUN echo "Running command: bash setup.sh MODE=$ENV_MODE JAX_VERSION=$ENV_JAX_VERSION DEVICE=${ENV_DEVICE}" RUN --mount=type=cache,target=/root/.cache/pip bash setup.sh MODE=${ENV_MODE} JAX_VERSION=${ENV_JAX_VERSION} DEVICE=${ENV_DEVICE} - -WORKDIR /deps diff --git a/pyproject.toml b/pyproject.toml index 25d1cc31e..413ac8914 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,8 +25,8 @@ dynamic = ["version", "dependencies"] requires-python = ">=3.12" readme = "README.md" license = "Apache-2.0" -description = "State-of-the-art diffusion in PyTorch and JAX." -keywords = ["deep learning", "diffusion", "jax", "pytorch", "stable diffusion", "audioldm"] +description = "High-performance diffusion model training and inference in JAX for Cloud TPUs and GPUs." +keywords = ["deep learning", "diffusion", "jax", "flax", "tpu", "stable diffusion", "video generation"] authors = [ {name = "Google LLC", email = "shahrokhi@google.com"}, ] @@ -43,8 +43,7 @@ classifiers = [ ] [project.optional-dependencies] -quality = ["urllib3", "black", "isort", "ruff", "hf-doc-builder"] -docs = ["hf-doc-builder"] +quality = ["pyink==23.10.0", "pylint", "ruff"] training = ["accelerate", "datasets", "protobuf", "tensorboard", "Jinja2"] test = [ "compel", @@ -67,7 +66,7 @@ test = [ torch = ["torch", "accelerate"] flax = ["jax", "jaxlib", "flax"] dev = [ - "urllib3", "black", "isort", "ruff", "hf-doc-builder", "compel", "datasets", "Jinja2", + "pyink==23.10.0", "pylint", "ruff", "compel", "datasets", "Jinja2", "invisible-watermark", "k-diffusion", "librosa", "omegaconf", "parameterized", "pytest", "pytest-timeout", "pytest-xdist", "requests-mock", "safetensors", "sentencepiece", "scipy", "torchvision", "accelerate", "protobuf", "tensorboard", "torch", "jax", "jaxlib", "flax" @@ -91,8 +90,9 @@ files = ["dependencies/requirements/generated_requirements/requirements.txt"] packages = ["src/maxdiffusion", "src/install_maxdiffusion_extra_deps"] [tool.ruff] -# Never enforce `E501` (line length violations). -line-length = 119 +# Keep in sync with pylintrc (max-line-length) and the pyink --line-length used in CI / pre-commit. +# E501 (line length) is not enforced by ruff; pyink is the formatter of record. +line-length = 125 [tool.ruff.lint] ignore = ["C901", "E501", "E741", "F402", "F823", "E402", "I001"] diff --git a/setup.sh b/setup.sh index 6a04cd55c..b59599dbe 100644 --- a/setup.sh +++ b/setup.sh @@ -37,14 +37,7 @@ if ! python3 -c 'import sys; assert sys.version_info >= (3, 12)' 2>/dev/null; th if [[ $REPLY =~ ^[Yy]$ ]]; then # Check if uv is installed first; if not, install uv if ! command -v uv &> /dev/null; then - # echo -e "\n'uv' command not found. Installing it now via the official installer..." - # curl -LsSf https://astral.sh/uv/install.sh | sh - - # echo -e "\n\e[33m'uv' has been installed.\e[0m" - # echo "The installer likely printed instructions to update your shell's PATH." - # echo "Please open a NEW terminal session (or 'source ~/.bashrc') and re-run this script." - # exit 1 - pip install uv + python3 -m pip install uv fi maxdiffusion_dir=$(pwd) cd @@ -139,7 +132,7 @@ if [[ "$MODE" == "stable" || ! -v MODE ]]; then echo "Installing stable jax, jaxlib ${JAX_VERSION}" python3 -m uv pip install -U "jax[cuda12]==${JAX_VERSION}" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html else - echo "Installing stable jax, jaxlib, libtpu for NVIDIA gpu" + echo "Installing stable jax, jaxlib for NVIDIA gpu" python3 -m uv pip install "jax[cuda12]" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html fi export NVTE_FRAMEWORK=jax diff --git a/src/maxdiffusion/configs/README.md b/src/maxdiffusion/configs/README.md index 376f69594..fe5f27ea0 100644 --- a/src/maxdiffusion/configs/README.md +++ b/src/maxdiffusion/configs/README.md @@ -1,8 +1,13 @@ # Model Configs -This directory contains model configuration for different Stable Diffusion models. +This directory contains the base YAML configuration for every model family supported by MaxDiffusion +(Stable Diffusion, SDXL, Flux, Flux.2-Klein, Z-Image, Wan and LTX video models). Pass a config as the first +positional argument to a `train_*.py` / `generate_*.py` script and override any key on the command line, +e.g. `python src/maxdiffusion/generate_wan.py src/maxdiffusion/configs/base_wan_14b.yml run_name=my_run`. -## Stable Diffusion 1.5 +## Stable Diffusion 1.4 / 1.5 + +base14.yml - used for training (including Dreambooth) and inference using [stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4). base15.yml - used for training and inference using [stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5). The upstream checkpoint ships PyTorch weights only, so this config sets `from_pt: True`; point @@ -19,7 +24,7 @@ base_2_base.yml - used for training and inference using [stable-diffusion-2-base ## Stable Diffusion XL & SDXL Lightning -base_xl.yml - used to run inference using [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) +base_xl.yml - used for training and inference using [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) base_xl_lightning.yml - used to run inference using [SDXL-Lightning](https://huggingface.co/ByteDance/SDXL-Lightning) @@ -27,4 +32,42 @@ base_xl_lightning.yml - used to run inference using [SDXL-Lightning](https://hug base_flux_dev.yml - used for training and inference using [Flux Dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) +base_flux_dev_multi_res.yml - used with `generate_flux_multi_res.py` for multi-resolution inference using [Flux Dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) + base_flux_schnell.yml - used for training and inference using [Flux Schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell) + +## Flux.2-Klein + +base_flux2klein.yml - used with `generate_flux2klein.py` for text-to-image and image editing using [FLUX.2-klein-4B](https://huggingface.co/black-forest-labs/FLUX.2-klein-4B) + +base_flux2klein_9B.yml - same as above for [FLUX.2-klein-9B](https://huggingface.co/black-forest-labs/FLUX.2-klein-9B) + +## Z-Image + +base_zimage.yml - used with `generate_zimage.py` for inference using [Z-Image](https://huggingface.co/Tongyi-MAI/Z-Image) + +base_zimage_turbo.yml - used with `generate_zimage.py` for inference using [Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) + +## Wan 2.1 + +base_wan_1_3b.yml - used for text-to-video inference using [Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers) + +base_wan_14b.yml - used for text-to-video training (`train_wan.py`) and inference (`generate_wan.py`) using [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B-Diffusers) + +base_wan_i2v_14b.yml - used for image-to-video inference using [Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P-Diffusers) + +## Wan 2.2 + +base_wan_27b.yml - used for dual-expert text-to-video training (`train_wan.py`) and inference (`generate_wan.py`) using [Wan2.2-T2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers) + +base_wan_i2v_27b.yml - used for image-to-video inference using [Wan2.2-I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers) + +base_wan_animate.yml - used with `generate_wan_animate.py` for inference using [Wan2.2-Animate-14B](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B-Diffusers) + +## LTX-Video + +ltx_video.yml - used with `generate_ltx_video.py` for inference using [LTX-Video](https://huggingface.co/Lightricks/LTX-Video) + +ltx2_video.yml - used with `generate_ltx2.py` for inference using [LTX-2](https://huggingface.co/Lightricks/LTX-2) + +ltx2_3_video.yml - used with `generate_ltx2.py` for inference using [LTX-2.3](https://huggingface.co/dg845/LTX-2.3-Diffusers) diff --git a/src/maxdiffusion/dependency_versions_table.py b/src/maxdiffusion/dependency_versions_table.py index 241035ac3..f1c6b6e13 100644 --- a/src/maxdiffusion/dependency_versions_table.py +++ b/src/maxdiffusion/dependency_versions_table.py @@ -1,42 +1,206 @@ # THIS FILE HAS BEEN AUTOGENERATED. To update: -# 1. modify the requirements in dependencies/requirements/base_requirements/requirements.txt +# 1. modify the requirements in dependencies/requirements/generated_requirements/requirements.txt # 2. run `make deps_table_update` or `python utils/update_dependency_table.py` deps = { - "Jinja2": "Jinja2", - "Pillow": "Pillow", - "absl-py": "absl-py", - "aqtp": "aqtp", - "datasets": "datasets", - "einops": "einops", - "flax": "flax", - "ftfy": "ftfy", - "google-cloud-storage": "google-cloud-storage", - "grain": "grain", - "hf_transfer": "hf_transfer", - "huggingface_hub": "huggingface_hub", - "imageio": "imageio", - "imageio-ffmpeg": "imageio-ffmpeg", - "jax": "jax", - "jaxlib": "jaxlib", - "opencv-python-headless": "opencv-python-headless", - "optax": "optax", - "orbax-checkpoint": "orbax-checkpoint", - "parameterized": "parameterized", - "pyink": "pyink", - "pylint": "pylint", - "pytest": "pytest", + "absl-py": "absl-py>=2.3.1", + "accelerate": "accelerate>=1.13.0", + "aiofiles": "aiofiles>=25.1.0", + "aiohappyeyeballs": "aiohappyeyeballs>=2.6.1", + "aiohttp": "aiohttp>=3.13.3", + "aiosignal": "aiosignal>=1.4.0", + "annotated-types": "annotated-types>=0.7.0", + "anyio": "anyio>=4.13.0", + "aqtp": "aqtp>=0.9.0", + "array-record": "array-record>=0.8.3 ; sys_platform != 'win32'", + "astroid": "astroid>=4.0.4", + "astunparse": "astunparse>=1.6.3", + "attrs": "attrs>=25.4.0", + "auditwheel": "auditwheel>=6.6.0", + "av": "av>=17.0.1", + "black": "black>=25.12.0", + "build": "build>=1.4.0", + "certifi": "certifi>=2026.1.4", + "cffi": "cffi>=2.0.0 ; platform_python_implementation != 'PyPy'", + "charset-normalizer": "charset-normalizer>=3.4.4", + "cheroot": "cheroot>=11.1.2", + "chex": "chex>=0.1.91", + "click": "click>=8.3.1", + "cloudpickle": "cloudpickle>=3.1.2", + "colorama": "colorama>=0.4.6", + "contourpy": "contourpy>=1.3.3", + "cryptography": "cryptography>=46.0.6", + "cycler": "cycler>=0.12.1", + "dataclasses-json": "dataclasses-json>=0.6.7", + "datasets": "datasets>=4.8.4", + "decorator": "decorator>=5.2.1", + "dill": "dill>=0.4.1", + "dm-tree": "dm-tree>=0.1.9", + "docstring-parser": "docstring-parser>=0.17.0", + "einops": "einops>=0.8.2", + "einshape": "einshape>=1.0", + "etils": "etils>=1.13.0", + "execnet": "execnet>=2.1.2", + "filelock": "filelock>=3.20.3", + "flatbuffers": "flatbuffers>=25.12.19", + "flax": "flax>=0.12.6", + "fonttools": "fonttools>=4.61.1", + "frozenlist": "frozenlist>=1.8.0", + "fsspec": "fsspec>=2026.1.0", + "ftfy": "ftfy>=6.3.1", + "gast": "gast>=0.7.0", + "gcsfs": "gcsfs>=2026.1.0", + "google-api-core": "google-api-core>=2.30.0", + "google-auth": "google-auth>=2.49.1", + "google-auth-oauthlib": "google-auth-oauthlib>=1.3.0", + "google-cloud-core": "google-cloud-core>=2.5.0", + "google-cloud-storage": "google-cloud-storage>=3.10.1", + "google-cloud-storage-control": "google-cloud-storage-control>=1.11.0", + "google-crc32c": "google-crc32c>=1.8.0", + "google-pasta": "google-pasta>=0.2.0", + "google-resumable-media": "google-resumable-media>=2.8.0", + "googleapis-common-protos": "googleapis-common-protos>=1.73.1", + "grain": "grain>=0.2.16", + "grpc-google-iam-v1": "grpc-google-iam-v1>=0.14.3", + "grpcio": "grpcio>=1.76.0", + "grpcio-status": "grpcio-status>=1.76.0", + "gviz-api": "gviz-api>=1.10.0", + "h11": "h11>=0.16.0", + "h5py": "h5py>=3.15.1", + "hf-transfer": "hf-transfer>=0.1.9", + "hf-xet": "hf-xet>=1.4.2 ; platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'", + "httpcore": "httpcore>=1.0.9", + "httpx": "httpx>=0.28.1", + "huggingface-hub": "huggingface-hub>=0.36.2", + "humanize": "humanize>=4.15.0", + "hypothesis": "hypothesis>=6.142.1", + "idna": "idna>=3.11", + "imageio": "imageio>=2.37.3", + "imageio-ffmpeg": "imageio-ffmpeg>=0.6.0", + "immutabledict": "immutabledict>=4.3.1", + "importlib-resources": "importlib-resources>=6.5.2", + "iniconfig": "iniconfig>=2.3.0", + "isort": "isort>=8.0.1", + "jaraco-functools": "jaraco-functools>=4.4.0", + "jax": "jax>=0.9.0", + "jaxlib": "jaxlib>=0.9.0", + "jaxopt": "jaxopt>=0.8.5", + "jaxtyping": "jaxtyping>=0.3.9", + "jinja2": "jinja2>=3.1.6", + "keras": "keras>=3.13.1", + "kiwisolver": "kiwisolver>=1.4.9", + "lazy-loader": "lazy-loader>=0.5", + "libclang": "libclang>=18.1.1", + "libtpu": "libtpu>=0.0.42.1 ; platform_machine == 'x86_64' and sys_platform == 'linux'", + "markdown": "markdown>=3.10.1", + "markdown-it-py": "markdown-it-py>=4.0.0", + "markupsafe": "markupsafe>=3.0.3", + "marshmallow": "marshmallow>=3.26.2", + "matplotlib": "matplotlib>=3.10.8", + "mccabe": "mccabe>=0.7.0", + "mdurl": "mdurl>=0.1.2", + "ml-dtypes": "ml-dtypes>=0.5.4", + "more-itertools": "more-itertools>=10.8.0", + "mpmath": "mpmath>=1.3.0", + "msgpack": "msgpack>=1.1.2", + "multidict": "multidict>=6.7.1", + "multiprocess": "multiprocess>=0.70.19", + "mypy-extensions": "mypy-extensions>=1.1.0", + "namex": "namex>=0.1.0", + "networkx": "networkx>=3.6.1", + "numpy": "numpy>=2.0.2", + "numpy-typing-compat": "numpy-typing-compat>=20251206.2.0", + "nvidia-cuda-cccl": "nvidia-cuda-cccl>=13.1.115", + "oauthlib": "oauthlib>=3.3.1", + "opencv-python-headless": "opencv-python-headless>=4.13.0.92", + "opt-einsum": "opt-einsum>=3.4.0", + "optax": "optax>=0.2.8", + "optree": "optree>=0.18.0", + "optype": "optype>=0.15.0", + "orbax-checkpoint": "orbax-checkpoint>=0.11.33", + "orbax-export": "orbax-export>=0.0.8", + "packaging": "packaging>=26.0", + "pandas": "pandas>=3.0.1", + "parameterized": "parameterized>=0.9.0", + "pathspec": "pathspec>=1.0.4", + "pillow": "pillow>=12.1.0", + "platformdirs": "platformdirs>=4.9.4", + "pluggy": "pluggy>=1.6.0", + "portpicker": "portpicker>=1.6.0", + "promise": "promise>=2.3", + "propcache": "propcache>=0.4.1", + "proto-plus": "proto-plus>=1.27.2", + "protobuf": "protobuf>=6.33.6", + "psutil": "psutil>=7.2.1", + "pyarrow": "pyarrow>=23.0.1", + "pyasn1": "pyasn1>=0.6.3", + "pyasn1-modules": "pyasn1-modules>=0.4.2", + "pycparser": "pycparser>=3.0 ; implementation_name != 'PyPy' and platform_python_implementation != 'PyPy'", + "pydantic": "pydantic>=2.12.5", + "pydantic-core": "pydantic-core>=2.41.5", + "pyelftools": "pyelftools>=0.32", + "pygments": "pygments>=2.19.2", + "pyink": "pyink>=25.12.0", + "pylint": "pylint>=4.0.5", + "pyparsing": "pyparsing>=3.3.2", + "pyproject-hooks": "pyproject-hooks>=1.2.0", + "pytest": "pytest>=8.4.2", + "pytest-xdist": "pytest-xdist>=3.8.0", + "python-dateutil": "python-dateutil>=2.9.0.post0", + "pytokens": "pytokens>=0.4.1", + "pyyaml": "pyyaml>=6.0.3", "qwix": "qwix==0.1.8", - "ruff": "ruff", - "scikit-image": "scikit-image", - "sentencepiece": "sentencepiece", - "tensorboard": "tensorboard", - "tensorboard-plugin-profile": "tensorboard-plugin-profile", - "tensorboardx": "tensorboardx", - "tensorflow": "tensorflow", - "tensorflow-datasets": "tensorflow-datasets", - "tokamax": "tokamax @ https://github.com/openxla/tokamax/archive/main.zip", - "tokenizers": "tokenizers", + "regex": "regex>=2026.2.28", + "requests": "requests>=2.32.5", + "requests-oauthlib": "requests-oauthlib>=2.0.0", + "rich": "rich>=14.2.0", + "ruff": "ruff>=0.15.8", + "safetensors": "safetensors>=0.7.0", + "scikit-image": "scikit-image>=0.26.0", + "scipy": "scipy>=1.17.0", + "scipy-stubs": "scipy-stubs>=1.17.0.1", + "sentencepiece": "sentencepiece>=0.2.1", + "setuptools": "setuptools>=80.10.1", + "simple-parsing": "simple-parsing>=0.1.8", + "simplejson": "simplejson>=3.20.2", + "six": "six>=1.17.0", + "sortedcontainers": "sortedcontainers>=2.4.0", + "sympy": "sympy>=1.14.0", + "tensorboard": "tensorboard>=2.20.0", + "tensorboard-data-server": "tensorboard-data-server>=0.7.2", + "tensorboard-plugin-profile": "tensorboard-plugin-profile>=2.22.0", + "tensorboardx": "tensorboardx>=2.6.4", + "tensorflow": "tensorflow>=2.20.0", + "tensorflow-datasets": "tensorflow-datasets>=4.9.9", + "tensorflow-metadata": "tensorflow-metadata>=1.17.3", + "tensorstore": "tensorstore>=0.1.80", + "termcolor": "termcolor>=3.3.0", + "tifffile": "tifffile>=2026.3.3", + "tokamax": "tokamax>=0.0.10", + "tokenizers": "tokenizers>=0.22.2", + "toml": "toml>=0.10.2", + "tomlkit": "tomlkit>=0.14.0", + "toolz": "toolz>=1.1.0", "torch": "torch @ https://download.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl", + "torchax": "torchax>=0.0.11", "torchvision": "torchvision @ https://download.pytorch.org/whl/cpu/torchvision-0.25.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl", - "transformers": "transformers", + "tqdm": "tqdm>=4.67.3", + "transformers": "transformers>=4.57.6", + "treescope": "treescope>=0.1.10", + "typeguard": "typeguard>=2.13.3", + "typing-extensions": "typing-extensions>=4.15.0", + "typing-inspect": "typing-inspect>=0.9.0", + "typing-inspection": "typing-inspection>=0.4.2", + "tzdata": "tzdata>=2025.3 ; sys_platform == 'emscripten' or sys_platform == 'win32'", + "urllib3": "urllib3>=2.6.3", + "uvloop": "uvloop>=0.22.1", + "wadler-lindig": "wadler-lindig>=0.1.7", + "wcwidth": "wcwidth>=0.6.0", + "werkzeug": "werkzeug>=3.1.5", + "wheel": "wheel>=0.46.2", + "wrapt": "wrapt>=2.1.2", + "xprof": "xprof>=2.22.0", + "xxhash": "xxhash>=3.6.0", + "yarl": "yarl>=1.23.0", + "zipp": "zipp>=3.23.0", + "zstandard": "zstandard>=0.25.0", } diff --git a/test_lightning.png b/test_lightning.png deleted file mode 100644 index 36e844cc573bbb6e3f89cf1dfc77ec19cb4f2f09..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1343016 zcmV)DK*7I>P)8%97jY(qN>|97c9-o&g}pH!_vO&5>3zDy)=!x3uYyV2zOIu-UkLE z<`LO_jWWaC&D2x{FgOf_Z{PeDgxL->0wl}~06+tX@rri=!tCl3|B&=+c>PTSNWuvC zjY$|l5dIG1zBD6YMlwE)PyCO%BLM@azsBQ12x!Lml8L|T&%3%y{z@?Bi+rYy@l8)O zp9Db|VK4$PG6!j9{^m4X^JmxniF_8X|Frp8Fq#1}W<{GeK!Cib`C7*aU8oUY^1T^Mq6))$gd>5QM2OkZ7_9Y?Bl0B~nnQINQ2-a$mCi3>ZqRSI; z`Exx(Qc5C`fGHssa-*K>9Bp(~cSZmx0vn$TC4?Tq5O=I!08ogcqFm|tHyAXiHRM2?#paio(`w2h^N;-QLDU-S44ro0DWA}ipgPlp z#djxawUmi;+9)~Z^0jDWrmIP0PHgen`(JqF1acvLbb``VHdfk+V-Wk*c1=T?nSx1E zCI*m0ZXoY{^_BQrg0D?|&vSzfSM$mhxc<#1eL7Jf>p7uXT)Ga;LpZaMFcZ3Xq%jqk z#8gUO4C>59@)?K_C}KAWg_U{)A(Np7bM7-3)C{R{f~cEKa$}-=QlKbGvl8jKy016I z#~MVzRIMoyV5u{OFxAp~0EAG3_@{Yy7g?c?EEQ=HN9|)Qfu*pe!DRZ(#*kSY2SUvV z>^mUho2_OPSm?gN*Qzj3UG4ajOq`(&P?!OxP$a{efu>rEmi+F_?^@PU8;x)2&{_Uu zV?MzpLryB?izR8KVy+bp$sG3?ZyHPtmgEIOYH^YZY|<{AFiYLvrr%0Dm)yG2T3;ZO z@yen7n&cQmk7XY?@A{H@;+LPrU%Bpcef`O5Gxdd^Ty(>rEETUl(;JmsrjQeY5+Lo9`v4bqF}mlDW5va6xe zf`LM#4(XBzZB`yu5}ea>&87!k@6q&|vqrA^j6cbIkj~e~4aT$K@bPK}zHXL}m%8Z{ zorJ(M1FgXN)k?|LpT-~_<{EN-aH*-+>uA)7O3rlDCY_Hgou@W{@niXZ*xTj0txV8F% zTW!OjJS%N&mW-9!eh?3@ER9(xvgB@Q4AgUcxbp<2>0D9672RTHnOEdGNz^O4n?z5E zjaM}Pt8((opAx0ZWRu^Gf5veZcAs66v<1%f`l32BIRtB_x z!u*kVDP_|j&-@EH3{t@9DI^mlrPPVrC6uc1BcZ<(N#}%s!B3$+CvrA(-ZQ!~CrgxB zpuUpi1hlknCRU3xfjW07WAZZoZ4;Aq z7b2wYl_8hwQ>;2f5K=?DnrX=xUcc%4N&;f;w(%Pcx^urwB#k|T_bs@CH1ZVy8>mP1@|_!}6^ku2iK*v7JJ>U~QO*tEg9fJuTgJeYg~ibV6d zM4Ia(lb0+OePZ-!d96MgyqPyi5fl)8ayNuGFFaYkzXEqvh9rvFrRtBBE%K`G>X@*_ zouNfDmK10C?;sr1hxOf5KsLXLFZWMi^*c-^Q~q^O(|q!>t5?V zn{HLs`D-WjJPMyl!GKBSIqz^{vg_AmeI)v|WP%bNvxxfQD8{3%Z~oaY$X%Fq*!4F^ zly{{jHe5EdPtKWnj6CnjN#le?36dSAM@jb0MZCHHW216g|r`h zI6{gzK98N1^nWx8XoBIyYou`K4sq}z#Gc<}Tro7L(V&j?htf?yjK!AfzN-OZ5D}27 zCJ?nP^={a>7P>@I!p6Ef9FfVP?7Ahb(Xt6$g|OZ)x^jgl@C9^#p&;~SuGm27fPsQ^ za>H4pG&B-2mV7LZtu2kaB*si4upvH*Y(Mi0enDyt6l4i7IG*d|?SJ^jo4va2ufDE@ zpWjz)5U11HYy+lvh&N(6YA;~vdn8n2#b#8)l(5rEknkW=T8vQNF=&vJ4PNu-kZtfW z6Av(faC)6ZSi9h|C#lnrpg|}IHl$`~Q790C)<}ZBCgMCHG&6ComU<{mmA)Db7ln(v z@->cyi_wTd2UG!I5sy$)F#%j!ggnsnD}ykhs%3BsE)E)@=8ju}X1fIzbERSmK$=(* zt~1xX4De*wn1Qngi2`pA{Ob4A;t)Th2yiJ z*DJunl;AqU=~93g$|=o^98l5+gVH3uyk-^@HC_cI9lyH&GiY^2Au%LJ)CkcI{Zw2*pq4L||Z zxmw}xHDIshqU$CQ8eVHSw~YNVb$3!NI6;M~S?nSu3;~+y03W_?g0i?FvDInmh_8vF z2x9CrGPZ3MP4~;3tR=;`6NS0`10dz+Sk(BFnF)Y|2AcDv!jZKrm@@o)oYN>njhIJH zhGha=3Dqk)pMh@7p5jKn7Jt@Shha~81OOcGVeA2rSZIoa1MraMq{l+<#GTD5Dm&OS zSIJT~hdw6(2ifai_#=3|w|W*A)f=5;ocK&@d2W`u3+bYd#Votq*7tUXK$4VGg7 zT2jI%?%~1|sFP1K={BF5*hNz7DV=HS-VDJn498n)5zkhjxX?0lG~?szO$eifg!TAu zGoerHH2ycLRCrZiWW~+hh83Nl-AWWUHFl`(|XqwC~8)5FhbQRggV!yj8@as zha#c?O_4T(o;EX{>jDe+i? zGm(#8Va=~s3M*;KIRpkJMNLUVBX+7)-dl(P_pzFwHqc;7qYF3+OcQ^*x~F>(EO*E4 zaF|3IkOxXw)->?LSu}1zx+KdTe9FC>TjRBu^C3YnLn1S}nky=b{0F=9!&^-B4GD1Ux zGQun|Bge;Pnh>KTiPBe^EVc;sXy%eJY)IeK6gN!SCcXitofM&(I;9b!rmZD{OXH-0 z^z4@)zCbrk1WlbTe3l}f_aYDlxML6G>#J@OL{zz&0MW4Th$Nz5rBDPzj|aKn5fRmhOo4811WGWZjbUk1qMTU8yUwSAGDimpvXn33PA@WZkt<<-hrb@?Xy4;- z`CwBYMZQwia1c?!xP;?A5Xn(EX?T8MtjU0TMTWyqMWY53(MbZQYDKa_B8wp`%@mcc zU6fqj%y}B|vliGgaYZK1iz@>Kzv8agW2W?}2!(`#Me5)fKbW~swxq?bp9T%HlnUfPO=It`BTLos#0NkOPQVo35Jk!#T<(= zJYh(~&e!r=&E0rr;toCr@=R9=gmh`-@nUovE(rR-FLX#1oOx;rS2;^p8@dD#HNkWzH zPEx1_Ge=Xdeluj}%1J_GGpUiP8n_|JW)iA8)YAYHsrzv~F3SOPj5CSW1-Sy**AfzKH1oLS9KrwMZw2N>dAZ9`g+_QRqbh7|pN?dT>LSZYlFP zD{EhGHLBUlDg>J5MF!>skpTn~m<&x38Eyqh-JJx1mkB8LCZcK$+Gu2%+K_~2ED4nb z1Wsxq5J-sqynEm}Qd6<`Vo--)Q{7ol?52?vXQenaRB!4w$KvQwOXV>hW_u}TPU}gt-2u13$^#2v#Aaen>Vm}Gc%;{ZA|WvaEarQTSJ8Y@31jpCGlG%{;5D9nBYn8;Tb-0C zLv>#%#fdas5AHN6Ul39u8P#Bo&6y#GKN8zf)N8`xWwnEjf12fPM?kf zU<=&>vnB*Wl!-X=S);g`2-Y9~Z4?nv+Zg_kSRAKy2^4HLT=E1mj;DK*2GCR373u+n za#+Z;*^Igqme@(Jlbeyg{^4jx+MH%0J{JH`hC&?lkje}2`&($+A$So+cKE-`XXlj! z0f+;?_^v7ZjS9E=4O|b45-=4|H5~6`W-n+sm{9U9<$ER28$6|7itOD%5K+o0<_tkX z`H{ND5TFWm(T0Sqn(!`2$bt|fPa?FV8B93PPT=Gs`}6_j(s?-{cckhbIwGGjc(157 z*PrPYNMeRTec#>eD?bh+s8Ur%L}r=>n#&J{iXYDVpei#>sF;*h!*Xv$fVeSY<|Bs` zO9VU5&x5hzxf!{#`H;LpzEXd?dI))wX%4iJJ(;?1lI_UgQd(|SX+#cP4AyhYb6v_% zx;7jWhsOULzU|tg++!;do~C>ZFc$;m0&=K|TI9sLKBl2yrm47d%pXpQWGYKVu8fhL zsCmOAtFghgRIwq0s>0ne?PD0$V<2f)`Ln1=ny7uKIhc9C*SWVSFKHu9L2I1SOn;Iw%SdYFY0hfWXw*-AIvebJBgBLQt#)H^DXbS61c2h{?I(UXsL0orX zNf^2YQ1>x+%V@FX);WEO`1|k(Kqd>Akm^|w6g8D7?glZO+`)<=VG(k_oa;$2Bn=Fn zWAIcd2n$RJHQ!knl#s+5MY4gwuHIv0pKDkt;gDqRM96v=%F84v&I54D&|XfcOG5W)^R8;ooXq+Xbz+2)Ypu^D$1`?j=K0Xo|1g-c))=%0$qh{b&+O?ZA!(` zj7q|EH$$VlRHt=h_&a@#BoZ!^0b1PnaCfOQ4&jOS0&^AvK571PgGA$QnrH#kp#}nFq3AU^DwxBPhFDIRx(dFsG9?1W{Zd^z z%7tV~27$~h3BNH!4Le-Os}zaH3G1Sa7>d9k(jI>6i$f6O= zY>yNmhGUreR1AqDQW|9Qz4r8@L-PfHlG4>4iLqMjVukaojUZ5hfKzO5+q<*GA8Llh~ z4|7EEq;8Q^Z$tzsQ|C)KysX6FSmDXlFc?3cJ7x9FntKuu>CI%p#aPCkjC_@L5LZlj zk!1nbY<;h4;yK76T?3sJFXWE;U%m??l4?aXOty^e$YqQvGj`2QPltb@b^w%T(FIiu zv2x8=DkODY+yo)4$`lyOofKfP@m2<|!39)qR%FuG4!F_|+RFYRQFK3cCq|MPVNBA` z_a{avBR+JcyKw^mBl3LU8VHDNV;6!JPDcaOBsD0hU4*VA$WBKr)|;4`EAr-zmBP)b z0&9XMn!+Gb7t?rTWal`cHF}N$U=*mT11tBCHvx$5mH%?#VrK33Nj1*bHr8P{W}VGsJFUO7BM=lYF@ER~?IA)h?PLq`313?Rbu3PQvn zSuC4_52Y23XN%~ZRJP0miXJ>>?BepzZP)PP8X9FcHPEO+QJ8r)p162WF}27? zLPJxCV$WSRt1%b=iV4DU842idu|@l}q2_(5HIPiwFM;M6coeG68Q{4sMZ=tha4Cbl z&}PK&ihdFt00S;o+oVa6mzH6wYhFKb{F0jA#o7!MFfmA~dJl^yZEGkOi)e_F;!w%r zI}$?P05zKA)?SMZ^JY`Dh&l&Fz$g^0TBM9Nx*N>n!!4V_`TWF_iwp+jrY3`N-UEZ} z0_nX2+DN~+QFh%3h(Ku4n}1he(waDzTh)od$7lv;gDn)0>m?Y<5lvNhO`(fyW@^wZ z6b}4rFoUWGtf7WYYz#gGb$=Da)Ab2fAD2R9opFQ0wyVnEo7ZX&4WVK1kE6Bi6+7!Vg3l4 z$^qdfia#>^nCfEyLoo>*U*aR=pUw}~2rd=x zg^$M)^`_T#QeZZQ66p|`4HbJhgn#VFP{Oklp_bhe*A3;G51O7`U~FtuJR%}9wd@>V zF3l_}jevB_F#jLV%t;~lEZn1G_Vb>NT_j6N)r&z;9{B2$IP_b%KVtyTX|nYzaXp}E zpkw*`TFK3x1@8d4tT1YDNRcN@J^TBDhYP`uNjp*O;;Ibvz$B8`U&fi7l@ z?0gfCOv9*}rgLaIL|X=ms%>h3iTf`|g1i6eT1;K%G2|y*VnIzULK=a|0yVP24lJJW zYdmzZ`woXM-qkT}*U0<;65n_+T;ZKTkWoenz!RZkr!~odgCoeY5jaXzk@xWe33p%( zR>34Y1{UHljSX)3N%9ZVt7L;5gKCgy1Cc9&>_KNGl2Ry^6ib;JzYR$eYdn-F2L1&A zN{T}(N=#`h?wr=(?G$zc(D>(3MpjxR4F-RCd>|@pky4{H8i}Y;L<_$@5<8UUr2#5z zERN41mR}T19nbNf^AG8DFU>iKK%*#>&L|+@s4LP`x}ZgR23mMbvp7I0VIhkHdw-Pb z*f$FaXKE0W-ISnp$I|dD7ZIv8hqwx;=jteTPnLD1?YZCK2^8}pVv(#e7MkZ5!IX)i z;b>C5nX1;lM3jILaT3X>p@xfuF!l8dfeGgVcQ=;N%~GJTfu*%3{lWs`pM5mzZA$`0 z?3Wu@Ow3~}2P3zvlX7` z8(`4gAQZ7^5Un<35Td|(7Jju?Ois+bC`z*M?F9v>qzaJXou!+jUS$9aMQ3TlIF_BAK~kx<;%g)O(%H(=vF-oVK;t1!$xPXtTr;`2vsGC=cNnEtJjwKS z7K0|rub1VhQ1^nbP%GB4=q`(Ly7DT`vB#{ZJlh15?*=_yw-~os>7r`Nwzvc_O7^HbHKs!59Wg&uvE`nrF zDmJ4@_f-Fd&CFf$3Zs&nx-ZkDvn!#XA>shiKn*0Y)64_JCekeYJ{3HIcTpCD<9M}n zJXl28ysWZ$PX&q^8oQc;Vx|(}M@bDyJh~imRBtD?$>caUwwQiUCG7bqys& zhZvrlp}6SvX_T-~n7620nFJ_IPg5ZZ(u>|~upcJ* zWj|i(qDmDgv%d;6ssXQQbsgfa67}XV5g8s)rVIuGbVdtKNwDp%$s%BHHIvlzN{N*v zmst?SD4~d%o}IkNFHy6Um4lorhu3RwTy$nVT`qxx!DOhqqZM%RKN154oOJ$i9LXW? z;a}G|)g}mu7-4LPd*#TVH!?(gh?sMWDUeap}ED7h4;a)_a%MwS^5}vaZge1dIL|UjKWd(rnzzqdA zu;V02LYBB1$mN?<+_>A77WIWl_%;;L)m-BLEM>RrkMhTWD{-?NZ0>OR4vL0qdn_Y4 zund7%D*SQR0TJro@(5Ti`!p)T&~q#pz+(bzVkz1+I`u(x4%qB#F;PFTt9d52Gsp7<-MG^>YH(2J{*cfizdX>DilC{HG8<#|E!CizE zhjq3~w8b&8QQ&1vy&fwTO0~2Y23OXa*ywDX#zV{~Z5;PieR1Ti&*?7OPkuH4N45z}1( zrU@Y2boXgPqitou{SbbULJ-kBRw5#6n6VCDAf?WI}o-5ME{M*$l zm6)X_E|n4LJ+`{ELQ$_lBVBAVLh>o8+3Wfg%=LCs?C^<3%15O6n7Mfqs%^vl960z*FRF@6j4vuh-M!L70T*D+ADvf|aC8UREuq0Noyk84Ga_|~KCxBuEdJ8*S zghDo9qroSZ8<ljkQ6DDjyM0)VQac`zTQ-fbx0VR?yzWrDAP zD(m<3prX=rn9UXo5?5F@>6%bK3pwtb%ay`qh|&=XLh1M~jQ8`w`_Hs( z{qICp8Q$<^4mZZ@XN+r$-e5M(C!IaM@inKk8Cc#Sz$$%QVsqV1AE^ZfFmsDy#7LXb zGQkGZLgXY%C5DouWI?>4M_Rn7?MWq$gpzV=2Z{7UFvN|^GFQUsp-rALM(ZBQf&^=6 z0Qi!MsxUK>$wruJGi%~<--Oyl`qD(KH|atFI<+_Ly>-~S$SSt3tu-Ex?YQuCI`zIB zPs_T<;n1bGzAnP31q7Pyo0(oVeYx1>Wq-^EL(H zWEhVid_uGmgMfB zJ-|^G7;3ao(OvUQ55gOO4lSBgr8H^H^Vw;vOrn{i;Dz{u3d@olkho|xaR%Tu`kvoD zDV~gx4br)SDTx5g5(o{UUe^Fo+@dV;9L__VzrhF;xD^nmSO^(;5z)nA8cVt4W0=7z zp0aKs5psDVdUb_2Ld>`vNQX8C2aH}0Ok)OVDvPi{*zhh@EiWLA$RZ>GJFR)%4~>-8 zrIU@eHXKBb4d=7mAN$R@-JI6Pr{mqB-#?vBH@Bzz)9H9P+??8>wPg|MY|-`Fn^J6- z%?!5hdU@7;yF71~kC)4bj~_mK`sK%$zy9^(k3T;D@Z-x5KVN?MY5RDwZPUwU=B;u> z(CN(@I%L~72WL%Whc()!qQIu^$Trt@Om|MAB_iuB0@lI6XlOwO1%-))NV9g{SJm08 zQuovZ(oUJawT;|$55bV32ODvqx&_W{jNv^RhR9x_N930aV^^y3|v|B!?E#LpszZ zGiK;l%b@s|*t>o*jyaxgI^_8qE)-P4xs0QB%#;Fm!c>ld$s9-P{Pg(Z67P(Rc_H381Oimd$LAkASG#T zHDo@wK6C$>zS}`45oP08UMtF&Pn5AlBS;=+93*6=4sxdM&321Jq-8Y}%#d9Kl@yBR zhDvO!S0co6MU!$Lf`Fv86h)vk5vsN(UjDlXg|_tG#RR%8(i=Lk66?~Iz8nrb9oyZl z+}|EF1yS`r$8s`S9akKm7H_Pe1+i^5I!OT%^getM&z|s%E-2HR|pO4AF#i zqxOg_M?E1uGa-E4)*Cd=4FHKfv|y~4@3z5$CR2zKu&7W~4wKY9H0q2uf&7%2;b#dV z4^Me?3$YX#-rL84uL8(wkO}=KD->r0DV78oOsWP(=Qv?>lur>16o{VjQHm*_e9&Ab zLaJ+qXi4E>z)EDt^zuNN8pV)2$rT~PG!iXQ^Ob_}A^9dR1gEFWoCu<7UlBn82w`I7 zGh&uKNbo+Hl{vnG=}`#?=6Id3Ch3K;C>-|%Y(fw%JoO}07%9O*&re{S24E^)Kq(@` z+GJfsU~5BfICfb@4#(xNw8M(Sx~{7ndh1K;U6$V6lNyJHH7U=LQsKw!d6-+iX+a6`QK6?wjr#6so)J8+N5qcExT`)yt;aF1zZkwu`7jciW;hW;VJf zOuem0Ml}U=Uhv&fN3W5H$DnnPX)ywTXq{geKEJ@RssB?gmkM#d6Hi4@bMM`(z6W;# z$U`Ku>w>C9!rBzgN73!{7J+oIrSQL2CrJPrwW_9BsDRW%3EN9~hWkc5IUeqbW@gk4 z-33QkK4eCy(N{(($f!>)kGESu!ma@(9N8$+*F!;C1b}6m!$cSi^+*QfjdC?tX>RB? zk6!_#NwX$s#s+Hwn065nti(z?A9;IP?oa*cwmrRF9`A1M9!?LB_cu>(jyLD^_P!s_ z?QkP~Veb%97Kj)LI_M=}t7mDM+J1rUy6@OkFTd#JL>3{qW~M|NNI9UjF()KfF-UcZD@7-30BntC|WxgCC-|n+gcb9N@TW@f4fcr*%^D{sxC+~USA)23nN4n&4+;c7|f=B zuQ@3!fy9!T8dQ2Do}5k@Mm}mxR6=eH068OD))!yDEHv8qv1ZMP;za}g}mqM|j5h0&^<;ryb7YeO;MIdE*uJLN2 zhV_*rN!qF!fHCR`Nz^_iet6|~`SQ!If63*!3XM23)bBQyMiN0JMsRz_%9gXrE(q&q z0^B1e;&4I2jCTcaHToo(VL7l4ez8O(OyyBaHmR0W zbo??!&)m;+ocWhBBOP>yWq>_XMt^ZK&v@|@UnHEU(c)96OJ{EolcjNOSX=Llh*%fv zT^6y^($1^i-1OV?^6=*L_;mmD{^92F{^sdvy}e)Vp89&~hcnuZ$q`;SEFIQ{0EtFN zyX_mzU>kYCez9%iNoy!k;ts2ffAnBeb7|aph3R@-UQgcei_jgN(Pj z`br*QvBEVc7DvUAiu)U>gzSx@x)iEBsqj7 zOc7Yi654y|Ix7J{+RzeeVLDR+ks5~~SD16-Qzjb07!#}TMo)SW^vbS0$0 z{nx$8@woPeg}RC?g2Q5`liaM!X_fP7Js+0iu^ra+cseevCRvYV>zyKY-p1=|J%E*sV+ zmkpO4YS>gag-R4!isGVqs4&93F|({2_5hC2yu+jOnKdsL3NrF4d`s08wAIA@^ln#& zOdR^D$YWNDWSHzJ-T@$Ux{!e-W94rQN^FLdKerz{SvK+W9kOl0z^}{Qi$W{^^e&{`%ANPapY<;{4YOrM2*kHIq zoOWgam}}=wEOcm!>XJ zkv5QUw@!2)FbfsxPnCjMdC%~_(NAhLF))uQ^^ApFTr^4}8X?4?vC)<$$BwmeZMLq~ zJJ;S0s~kISZ`G7-EH}Buvef{?K%{Rx#Z~N(?AJ5Vc+K-eK(tt)bS1q6b4cL7r z#X7J@-yL%_xx>64_h$CI+Q#j2*gl;vKR;bwzWe2;=b!%a;fFu}@gINxumA9efBKgn z{`7A@{`|q7U)mzg!@Zrl?W!h?+)Myb=uR_h; zJ9f0ZYh`j7^3-temyr9P3Lc&tq5#v2hjQ;(sGAZEPE8Z?wkQs^S%4<+F0qpiZYOcg zQ$NR*uU<190+t;=2Co3_@QpXb3)4rn3nJi{F;5P2*33$Pms`S9K1!yg9!b5F5rEG} zT5qNV7%5!|K7HdFBR`1&_(hpvjhR7HpF%itq6gwf^g)NANi46Blt!p`?G!*7x}b{) zO(^c~?_GKqAv)!_im0AMPDi=BUGE?I&3U~!9qw+{(`h*#kB7A{hu)X9wS)A|<{h;p zdQ7J#auKAuSK?`8`FLW_ZZF%mZ~M#3%k%U8>G|c;r|t55`Sj`HR>8LGw(HA_Zkuk~ zzNu}ST?D(b!@6)cZi-E@@79#Nim18MUX5F-;(a$ENKqIY8f4ovQE_`4jC{t?;7k_hA20~qroB!Li8!3^1q ze1ciDj*adNPLVoIuB=!O9KkrMe4@5~BI)Tj37>PVgLh@-q*eu{8o@&ttCNyH*UVH% zM*pB{1r%z-N17>Vo-ay^^Bie$Dg|f|TK8K3B52XT3Z2r3waKCLxU^%J+oQaB>R&yb zzWe&_{nz&oUp?KueYZZm#o@jk&X6Pe(WIe~Y<@qA!Nf54hI1yR#o4ERf~>Ekk~@~& zc9r91`rPhb?B)6J_QT=R&$k~xzWwQ!?|=O1r$7JYhd=!Ir$7Dt`yW33@R#SGpZV$8 zcJk6>7qdp)Y*UCpHje}A<_>aq#KZ=Ma*vf!Ew4oGN;(a_E+z<`^lF>BEd4|Y2GVd3 z4mQUaloWk5HjE?@A453rn-@30DDXotUGw-o!@-yIT0ep>U}*^mxWg=33+Ppz1puaB zD%{M@FnK{>oDlDsXP1+gfx!ls3@c@VloL5#zYGbXV!?H~SE$`#Wam^T^k01U#oMFw z(d^?7Xg>^%@x@5Ean5ULCV2vR>a}vsPb;4(7f7L95fMX1Qe!wMWH7Hx49l*E;TOkk z-N+l4TXq_-`CF6kBSA~mMUOSVdh)A33e|&e-~6U-_t|w<3x1U+_9d`!{k_j0hxBA$ z4=jT&19I53E^)QeqTcTM@m2=dkT=Zz$n&oP9i^{$bdf5kZ^*M_#{d(v5K&81HbU;= zM3G7*kgc+9PznGdV(Yn}oIV={4?kZYcU?Jpp%gNVy>%Cz3LWK+o3NP-Bq_egslQp+-EF_37HY_q;5bV;}Gj% zqGpjw9<`P8Ek+$q`{1d<`V&)JM+l3|2@8!DnXO*Ln1`uV9fD;84MxwNbSKgCa0zei zQ|8eMtdO|IY-^5UrxA~b76>l8TqI4iL?9@gH74jdO*7oOLCduR{$$2X71yW92V z{(N(PyY{}G4*jr5@4c^VP1@pKsjyX~!C|2RnXTc4iR!7$wwsylrnc#}>AtJ#5E zw##MPcD+1rpFX{O`uOtv>E+X>=jZ3``DNR-ecN-0BDAN*WLTB6T5fo^{fMRHt{}QPSpydTgW7~u}S_9vF|adkY!fQI08l4s@sh5Z3UyG3L^Gb$Y-;}&FOE=3RWbVNBxCd&o0 zXK%E0+-TseG?PQWIBsJeu97L3C~k_Z!f5dZ;7n)*gz z^wLFfeyAsH3^n2>T;!@i2udjhuk`glDP}_Li*&K$($}RQJ8nB3@0Ry(m*0GQ`}MbP zzWz_|PY-XGhqrRN!*W798-(ouY2iaOC09iwc2ac2lXhliIzX5hGy_YHm3AQex?!X4 zwmoxy*)N}Ld)}Xa-amfWKm79i^Upv3@Y4@}`r(iN{O3RX;pad8>G`i8_m3Mk^oxhQ zJfCQTnr^Bu+pbyVqndqRuAIk85b9_DIYFAYKaEZRMWe%#31MQ{&Bv#pW^p`20*sPm znozbpyxEY)%v%O|7kC=X_TW4*tFsTXGYOpA7%XXKKm@=^Kvew{KJFgH2n7+=g(b+a zR3WK;yub(`j2d~4&eUzAfx4J^G9c=!vz#7w&Ft|0(#wG8hzJ`#TT9OX;D>`H%@%nV z=S*n4bj)lcOt!plcIePi7JOwqm|g#g5Ia6jS1PZbSgLut$+e!#0nI09O%q7j@!2w! zq?_hD!|@Rx0EU%vb)nBFMc0_`uVVESJa6CrcBtzwuG2u_lOg*bzFwi|ym4CK3DTzp z7)DnPaI7MY(MyD37vzehM#+VOLmgiv?{w_NY?#EprLUKXTRAA(a`ZJ4|399LieY(u zJR&r%G!t!MByctuf|4g2G-ooy4FqfPlGoLAW1f9fVQXx_5v>> z+;fSw$lZVCdPcM2w9wyJqZ+T{P>68|LfoV!~gjI{?kAHk01a1!}jTg8``G(WjBRwmy4?SF)`l#5GGrcMVfje zps*02sv3tP!Nb#g#E60H8S3p3!zCU(DptD)?4}6+xrL_$vfFNvXe99wqBZ6aUvQ`r zjXnt&;b*_&s)vaw!`qz{RD$qY7-kF?iQ-e5B@(KZL@1%=ga!8$P2$huoHwwNY+gx> z5+fFs0F`0sep0tUt`hk^f4!F8ujSn&k|T$zviW@}=ye~g2x~0Vl+j_3=Op{5ro&fdaC-noQyL+* zyOtesuSpX|hH;8*|9*7GBxlm_>!UAGHqi*DzZH|P5{mds+NlaHukk5#5yry4M zFJ%U4GAY>b7AH^$4V{8LS@#vh2tp+RuO6L7fznN3nS@mnSrcwdxvrpjSy5_s!$`Zw`0& zINb4ggVxcGhEC}aFCbhpKG?$*NwGk9;SmoZ@0$if)RQtTWNze5nOqq_cqMKpcy}dL(Z5K7X zY`X2bshJn$`56bQ>DY1qL6C@K5?IC&VTfZYv=FLAlO>c&51e2gi=K1Y^VVW1c|&j| zY6mT={0AdA%146!JnS!}00D?szPX*CDG*^)vwinuCs%WBb;Q>+GH1*HhRV97l_V<;9z8ZaAt zG2Pe0E1@vozO*W9b`vxmXg@JSUYk0yNP4_;wMNeC?(B=1X=2(hymHFZJ527I5x6ao{>z}h0CvuQk@`>>W6axXop&u_u$+cO$dQ$?{h}6DaPeAWhuUQd%gu7QTW)TSclY;i@4x+#69C0dGVU-zht2>Ig!H1~Tr#HOz$UU@DQt1M)Ud^ zg9w-AO?0&ri%9Eej4YtTDr@JuaBaLjE|2Hs&BNj0@%;GqbaOu5KHjaz^Ljo>Kg#0H zQ5r?08_-*$bdctK&fUrjBm(cbA5#HgDll^rb|ZD)Y~Qf$x?NN+wk_M`WxHJLV0!MC zm*wSo{rG(P`0?e#hv$zUo!~}>qINY$iC{9em$bk$F1`)@g zj3c*fBA3uhQa2d#CYv^Z@HCkK(g2J=bHCXf(;cx6qba$xOq*p&Oe;iza3Q&Rbp8); z;7n=oBJ#M2ORG>`2fPIWZXiVY+CSw*gRf(d7M zuLL7ih-L!Q-b4hgL7>Yb$7SiJ=ibgIdAODT^qYtO^xfU}zk754{_E4zH|_Mm^^D~P z(hQAU$rdMFw+{9zOu!7k)yzp+&wX%#AZgk;Qc0;Q2eTdKM6zMOm~LnnS?Tnh9I* zSS0xbp2&uHBt+9Q8{`UQ_Bmw8bd1ffmFZ*rh4lm+bJXQ0zoN`X>rLlh|TR1P31 zFP1hlQeu0P$4S=*-Y-0SNbQLARt{esPw)G~ACGU}zWvR+um0bE|9|}L@Bi@6A3wd| zMX|MRsw7Gy&58}k(w?y>}t&Qn*6fKVe>4f3%@8WoYbxSpqX(IP6skEU7`8NTb)9jK&t3T z7JA4xf&dBhT!>0m0+E>u5t+*>WEb&>S#K1I(3Zw!kwyC2*=cK+)7o#2a(}bFd2@Wa zJ3T$!+&!Mp_vhoyt(?x%4$HEzEueU&hKR%=qZCT>EQ9P$Ee<0<@Af@)-3wP``2el8 z9cDDu?PB}I_O0)ieY@zk?JrwzYhPa0lOAvO6Z`BpFUkKo3^$s zU0#Hn>Scp83bi-B$R-;oBAaeItW!j`U13x;fr|9a)cjN?!uMvlZ?Vvj%bH=rVJf5{OIC zdJqsj$sAxe3FmxoUfd#!v~}faZRg(3E5CYo`0e*kzx&T$ef8~|+c!_^-8(tmaXGOa zAPb}kJq47cfPLDe+^et`!z-2u2taxet^`C*!rO666vqNC&q*$n6+GH< z8so5H1uT-L!VOcVSL{X6hB-kX`tcPYag)vrS)Wcz>mrWSiiUC=z|M3ea`*~*bD+4R z`DuJ~L)+qHK!(H2t6Uq*{4~vSoYugLZ(LNHSI<|AP&4Ei1_lE&XV8w{j|hwDY#|;SXwD?e8&1yNzX8S((&;`XO_~=2|-aeo}MbqLGPM892s} zX;$%m5^gnhH)9ue(_PWf7RcWEsU2>X)A4*d-QFz^Z~Nc=zy9^_{_e*gf2ML_V`IbS zOiU$SZ13U$SpgtMjZF-#>WO3;l{86dMX0H;om%O&U>xof0Ss~I41$5xyDP!1^lL%M zN;t+4DpPz2SN3}>ibN54ep7}#;{!IgieKt4kFc<`*$vtit5 zX4zNojB7DFVqU$j1nwkZEp(OCJ>qhv$|B$iBv!&RW=#e;rTi}W=pi31?fskk_4al> zpW5N5?I6pMghALk*`zfQX6YTl=BIWCC=xHAcSS6vKnw;pO2p|+qNJK;X3D;C-(Wko z7i^pCoAj5yZSvCkzV&50EbVey&!_d5(|Y{*<1asb{P^ibKB0?t)0a&J5H&H|+=&V0 z=G`onO)il>X8?AHnzlHho4YAs4c;o(GC(=%cRfoX<(<`LPMaS-kAREz%7{#@9*d#i zIL~AvK`YpF4GtzUorrzZ@q`gEJUgCDkyfGQQNh!#A~J7RPl}BDV)bHe4s$yl#g434 z{;k34h$GU799?pydk-8&JN3BlUqJ>9!{cwsKgIV}iAn|A03{b|2gm`-Q4Y_C!(lm{4>zat?fLfp_;~-vfBL)q-~ROR zr=PW>Z;c!H0+Xdvw~Oj-YLMva<&1xku)#x1*laHb-g;j^|fo=2Hk>%j6 z5}iUc)xbD2%Yb{9ypcZBSGfOkqjw^K()P1PNpyM*DN_Mmi3rpxCT<*0okuw-^_Uhz z`3~0?lCM)VOqe#y%qE~4nMEACuK_=c0LCn>P-=>XAyp(F7Z0i~uK4_V4oZAA-?3yi zEm203W{rO(J!_GCk7fhl^-iV;E3lA~_f^U9s~5+L%wU28YRg%&{-)~tz^ns3pXTEviqO;KuF||N5;BDOA=3RW*2ngxbVF4xbX3=J)U`g zlBb(?b85$f99LaCm!`|AZSDQg5BIm{H}6kxzgr%^>bLK4d_X^2JA)lu!4;&}^ku-T z<@Q-=ncB(N>7tNV{Q3Hi;9*0{sXDiKVVwbDN_H{neZ7sY8*Snkk?qU7v?d{{6 z^UcHg?(O4xzUjxCz8+MhwN+Xt05-N}Y-F4q5aB6$M^s6{pMZ&;YB>Bv!9PJkp&itA z0FcJMz_c%`?YnLlTQ0h7JRJ7ROM7`<`qulU_1>1{a6TMPhoAoXp)D_;KE3n?z6dXd z7Zu<#YK~El>x(Ju?d<26t>1$hWti5_hs`i1BbH8%GcqbY_br@&N0(jsfx`K4rL`X1SGQ z0?-p<4f>Cc=)N^bSQ(P3D4UFi-_BDu?lF8rr8uls@il2&L4|jxTZd{H)xU zyq%TkBNCp()v6j%KVE&&691=Lh2aU~^sEr_!~1EbNE^UhttnuUfuAvk;N$9sV%!BS zL^A-Z6F^34l0R3iGMUVvjs?xKv9CJC1o{spU_hJ(C;?p}QCne?5I0MFsL@O(*E7!Z ziri1{f)buvA(zH3TpG>`ZdX3sw1?aFc+01o_ITUQ2RW|2FVfq#tmsWB+78R%?*8=t zo5R!h{r)T7ys^U*`WZYzjsU>s`Teujt!P>H+m_nI;+LXW*r2rSs))|Mtfpcf7C@wnKIp)C3gTW`KCXlgR|8UA3sHi0s!K zB{@!~q%6Gxp(5U6pdHid*kXL zmsCa9LTb`e5ry~vytAHP9aHf2vgiuWafhg~HZU9exa4wcnTFm#potp*jA#JS8+wxt z>y4du=v)^$E^_F&JuP>K{`TGJt2f6tPv`T)?aiAv%gwE@LI#D{$R9M zF^cW`?g#W*KvGk4s-x!NtQ<PNAF_8^H_e=&7OBDaYYZTe*yo~BE zHD$T~O)1f!F7bUx16LLT^V~Lxl6Moq(kR5bw4>l=wa2^m^_%r~zkB%hx9{%XzBxX< z>$mq@Z_rPa1KL6^Ak%zp#)c%!%n0S@K+Q?nvPWT2^_5&eWlzc2+0r2aIy}O$8GzP^ z4(-NfvPxT*wzg&Ihjl%)(|Nr;|NbBU@y9>^`O}AOYfV0?VG+Y(`@SncKMUR{#2!{C zL&F+pdSdR1MaN86%HHtJ41nfd`z~wPJ{ALa-_R=pAgMuV31b3O6zM1Q$?{7=1h#tz zeFVxDhVPIX?qmyPl;&dMHjRcw?mrq~H_HXKE5c6H5(|^Uwm?oYssY~9FOrlpv_Cs> zocV`UkgMx)z;?56l;$sU2StUPAX(x3NcnT&~eJfJW35$ zH+GuK_a@!yk%8QS@TfRb8Q~|5X}M<-VF1%0d|`s$rs=ByXq>F#no4>@{Ob}pCi9Zn zEK_5Hdpyrp%pKTX+UuwARJ`49=kIr6?>uxaz|DfETY0*Xw-4*ngFM{e?%WTHEen@b zq=~dnYG0OqJoNj=GW-I5=sg47+duTL}@T4iQpyHvl~H_`bEn&GGb`!_gqWZ2#x~_}f3~v*&VkQVF`d zxKmZ4gmU*b9&|UWGpa-xy7M9pTD?uu*-S)LcR1Bh0n?&h%jsk`Q<(z6dlR%B(cbc! z*P1ccBJpkF@sU&PpGQ%!hJ!#|-D3@;{sAD^Q){TF~cq>+Fhw%Y({ z6rgwS!HM3`nC0@sp>YwB#zSjM$0Bm>{r1=%ZkDIVwY2|-XJqo>g;s}#^NOGvq?_$vV_#%hY`?H~ zwpIIz%cp(2te4)}(%Prids{xX-rA4&@bQzhO@J4Ws_dF#*22Z#KxpLhckCv4m=bMQ zHHc8vwTkD*$!V8a8%