Skip to content

Preserve small values in Yeo-Johnson inverse transformation - #1101

Open
ktz03 wants to merge 1 commit into
feature-engine:mainfrom
ktz03:fix/yeojohnson-inverse-near-zero
Open

ktz03 wants to merge 1 commit into
feature-engine:mainfrom
ktz03:fix/yeojohnson-inverse-near-zero

Conversation

@ktz03

@ktz03 ktz03 commented Oct 12, 2026

Copy link
Copy Markdown

A fitted YeoJohnsonTransformer preserves small nonzero values in transform, but inverse_transform rounds them to zero. For example, fitting on [-3, -1, 0, 1, 3] and round-tripping [-1e-20, 0, 1e-20] currently returns [0, 0, 0]. The inverse formulas lose precision when adding or subtracting 1.

Use the mathematically equivalent log1p / expm1 formulas on both signs, including the lambda 0 and 2 branches. Add regressions for learned lambdas, near-zero and ordinary values, index preservation, untouched columns, and unchanged inputs.

Closes #1100.

Validation on Windows with Python 3.12.14, NumPy 2.2.6, pandas 2.3.3, SciPy 1.16.3 and scikit-learn 1.7.2:

  • Reproduced the public round-trip failure on release 1.9.4 and main d866312874a66364c6dee3c247e69476b478395c.
  • The 17 new regression cases give 10 precision failures and 7 passing ordinary-value controls on unmodified main; all 17 pass with the fix.
  • pytest tests: 2134 passed. The initial baseline run had three dataset-cache permission failures in addition to the 10 new regressions; those same three baseline cases pass with a writable cache containing the standard California housing dataset.
  • Full flake8 feature_engine tests, changed-file isort checks and sphinx-build -W -b html pass. Black reports the same existing missing comma in the old test at line 126 on both versions; the production file is clean. Full mypy reports the same two existing errors in selection/shuffle_features.py and datetime/datetime_subtraction.py, with none in the changed production file.

No other platform or dependency matrix was run. This is a numerical correction to the current pandas implementation; it does not change the separate narwhals migration in #965, whose current inverse formulas have the same cancellation.

@ktz03

ktz03 commented Oct 12, 2026

Copy link
Copy Markdown
Author

The Python 3.9 job fails in 52 existing sparse-input estimator checks (2082 other tests pass). The same 52 case IDs reproduce on both unchanged main d866312 and this PR head 3d256d1 in a separate Windows environment with Python 3.9.25, NumPy 2.0.2, pandas 2.3.3, SciPy 1.13.1 and scikit-learn 1.6.1. Their failure messages and traces are identical: they stop in the unchanged test adapter while validating sparse input in fit, before inverse_transform.

In that environment, the Yeo-Johnson test module has 10 precision failures and 17 passing controls on main, and all 27 tests pass with the fix. The official job log is truncated and does not expose its complete dependency versions, so this is a compatible local reproduction rather than an exact copy of the CI environment. The official Python 3.9 job remains failed; no tests or CI configuration were changed to suppress it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

YeoJohnsonTransformer inverse_transform loses small nonzero values

1 participant