Repository navigation
Conversation
PARMETIS_ANISOTROPY_WEIGHT (default YES) weights the ParMETIS graph edges by cell aspect ratio, gives the edges that chain points into wall-normal lines the largest weight, and moves each line to the rank holding most of its points after partitioning. Lines are then not split between ranks, which keeps the block-Jacobi ILU and line-based agglomeration from seeing fragments of lines. On stretched boundary-layer meshes this turned stalled or diverging parallel runs into converging ones (2D shear layer, 2D bump, 3D bump, flat plate with multigrid) and made iteration counts match the serial run more closely, with no case that converged without it failing with it. The load imbalance stays within 1.01-1.05 in 2D and up to about 1.3 on a large 3D mesh. The option has no effect on isotropic meshes or without ParMETIS; a warning is printed when it is set explicitly in a build without ParMETIS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Author
|
I bumped black up a version and precommit removed a couple of empty lines in some python files. |
…eights In AD builds MPI_DOUBLE maps to AMPI_ADOUBLE, which expects active su2double buffers. The passivedouble buffers were overrun, corrupting the heap and crashing the AD regression cases that partition with ParMETIS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ode/su2 into feature_parmetis_anisotropy
Every rank holds a copy of the pressure drop, so the adjoint extracted on one rank was only its local contribution and Adjoint_SolExtra depended on the partition. The extracted adjoint is now summed over the ranks, and the seed is divided by the number of ranks because the reverse Allreduce of the massflow sums the seeds of all copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TransposeInPlace turns the blocks of halo rows into owned rows of the transposed matrix. Boundary conditions modify only owned rows (symmetry planes project out the normal momentum, walls delete velocity, energy and turbulence rows), so wherever a partition boundary met such a boundary the adjoint linear system had spurious couplings. The primal never reads halo rows and was unaffected. On da_sp_pinArray_cht_2d_mf at CFL 100 this made the adjoint diverge on some partitions (2 ranks with anisotropic ParMETIS weights, 3 and 4 ranks without), with the adjoint linear solver reaching only 1e-1.4 instead of 1e-3. All partitions from 1 to 4 ranks now give the same results. The reference values of that case are updated, they include the pressure drop adjoint fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bigfooted
changed the title
TransposeInPlace now takes the geometry and copies the owner rows into the halo rows itself, so the two callers cannot forget it. Comments shortened and the new code run through clang-format. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Contributor
Author
|
there was also an mpi error in AD that popped up with this PR, it is fixed with 50639a4 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR splits off the parmetis improvements from #2931. It turns out that this parmetis improvement is 'almost' always better, and in the worst case performs equally well. It improves MPI performance for cases with inflation layers.
PARMETIS_ANISOTROPY_WEIGHT (default YES) weights the ParMETIS graph edges by cell aspect ratio, gives the edges that chain points into wall-normal lines the largest weight, and moves each line to the rank holding most of its points after partitioning. Lines are then not split between ranks, which keeps the block-Jacobi ILU and line-based agglomeration from seeing fragments of lines.
On stretched boundary-layer meshes this turned stalled or diverging parallel runs into converging ones (2D shear layer, 2D bump, 3D bump, flat plate with multigrid) and made iteration counts match the serial run more closely, with no case that converged without it failing with it. The load imbalance stays within 1.01-1.05 in 2D and up to about 1.3 on a large 3D mesh. The option has no effect on isotropic meshes or without ParMETIS; a warning is printed when it is set explicitly in a build without ParMETIS.
Proposed Changes
Give a brief overview of your contribution here in a few sentences.
Related Work
Resolve any issues (bug fix or feature request), note any related PRs, or mention interactions with the work of others, if any.
PR Checklist
Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.
pre-commit run --allto format old commits.