Skip to content

Feat/issue 167 unit normalizer - #466

Merged
rosspeili merged 9 commits into
ARPAHLS:mainfrom
rizzoMartin:feat/issue-167-unit-normalizer
Nov 5, 2026
Merged

rosspeili merged 9 commits into
ARPAHLS:mainfrom
rizzoMartin:feat/issue-167-unit-normalizer

Conversation

@rizzoMartin

@rizzoMartin rizzoMartin commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

Adds data_engineering/unit_normalizer v0.1.0, the from-scratch replacement for #166 agreed in #167.

Models handle a single 10 km -> miles conversion fine. They fail in pipelines. A batch of mixed-unit records either burns output tokens or needs one tool call per value. 5 ft 11 in gets read as 5.11 ft. 20 tons gets converted under a reading chosen silently from three that are up to 12% apart. This skill does the arithmetic locally and deterministically, one call per batch, and records every reading it had to choose.

Scope is the approved directive (comment 6285849865): section 3 plus Exciters 1-6. The items I flagged as proposals in my follow-up comment are not included and are documented as v1 limits: Exciter 7 custom_units, text inside batch records, prefix/composed trace fields, and unit products such as oz-in. They can follow in separate PRs if approved.

Acceptance criteria mapping

Directive item Where
3.1 normalize as a batch of one skill.py execute(): normalize builds a one-record list and shares _convert_record. Test: test_normalize_scalar
3.2 normalize_batch, non-blocking errors with record id skill.py _convert_record. Tests: test_batch_issue_example, test_batch_partial_failure
3.3 parse_quantity: compound, fractions, rates parser.py. Tests: TestParser, test_parse_quantity_action
3.4 uniform {factor, offset} units.py table and convert(). Test: test_temperature_offsets
3.5 raw value plus display_value, default 6 dp skill.py _display, precision 0-12. Test: test_precision
3.6 dimension_mismatch without crashing skill.py _check_dimensions. Test: test_dimension_mismatch
Exciter 1 canonicalize / omitted to_unit Test: test_canonicalize
Exciter 2 assume-and-declare with risk_variance units.py AMBIGUOUS, validate_assume. Tests: TestAmbiguity, test_ambiguous_row_is_warning_not_ready
Exciter 3 normalize_table, row and columnar Tests: test_table_rows, test_table_columnar
Exciter 4 compact: true Tests: test_compact_bounds_output, test_compact_table_omits_table
Exciter 5 zero dependencies requirements: [], stdlib only
Exciter 6 boundary with telemetry_normalizer instructions.md "When not to invoke", catalog page, category hub pipeline line
Manifest v2 as specified manifest.yaml. Test: test_manifest_metadata
20+ tests 52 bundle tests
Docs, 5 providers, hubs, CHANGELOG docs/skills/data_engineering/unit_normalizer.md, both catalog rows, sitemap, CHANGELOG.md
Runnable demo (review request) examples/unit_normalizer_demo.py, registered in examples/README.md, the agent loops matrix and tests/test_examples_smoke.py

Design notes for reviewers

  • Angle is its own dimension. SI treats the radian as dimensionless; here angle is a fifth slot in the dimension vector, so deg never validates against anything that is not an angle. rpm and dps compose as angle per time and canonicalize to rad/s (REP 103).
  • SI prefixes are case-sensitive and limited to an allowlist (m, g, s, L, Pa, J, W, N, bar), applied only after an exact alias lookup misses. So min is a minute, MPa and mPa stay 10^9 apart, and MM is unknown_unit. Case-insensitive lookup applies only to long names (Kilometers, Celsius).
  • Compound sums are rejected across dimensions or with offset units. 20 C 5 F is unparseable, not a number. A fraction with a decimal denominator (1/2.3") is refused, because it is an optical sensor-format designation, not a length.
  • An ambiguous unit follows the other side's dimension and stays declared. 8 oz -> mL reads fluid ounces, ces, and 45° 30' reads arcminutes.
  • Rows locked with assume come back as clean ready rows with no assumption fields, as agreed in the thread.
  • target: [software, hardware] follows the spec. As a result SkillContext emits SkillwareHardwareSafetyre_execution=True, even though the skill never actuates anything. This is documented on the catalog page.
  • Skill history cites 770fd3f, the first branch commit after the rebase onto current main, the same way # feat(networking): add geo_ip_rotator skill and networking category (#437) #464 did. The SHA will change again on squash and can be reconciled then.
  • SPDX headers. Every Python file in the bundle (skill.py, units.py, parser.py, test_skill.py, __in-License-Identifier: MIT, matching telemetry_normalizer.

@rin-katsuragi, the hardware-facing pieces are angle handling, rpm/dps, prefixes, and the '/" arcminute/pinned in TestUnits, TestAmbiguity and test_dms_without_spaces. The offer to pin your real datasheetstrings as fixtures still stands.

Type of Change

  • New Skill — new registry bundle under skills/
  • Skill Upgrade — changes to an existing skill under skills/
  • Bug Fix — incorrect runtime or framework behavior
  • Documentation — docs, README, CONTRIBUTING only
  • Framework Feature — skillware/core/ loader, env, adapters
  • CLI — skillware/cli.py, docs/usage/cli.md
  • Examples — examples/*.py, agent loops, examples/README.md
  • Packaging — PyPI wheel, pyproject.toml, MANIFEST.in
  • RFC / meta — templates, labels, CI, or large design doc

Checklist (all PRs)

  • Linked GitHub issue (Fixes #… or Refs #…)
  • Scope matches the issue — no unrelated refactors
  • Contribution complies with [Legal notice and code ownership](CONTRIBUTING.md#legal-notice-and-code-ownershse grant, no malicious logic)
  • python -m black --check ., flake8, and ruff check . pass locally (or CI-equivalent subset). black -; ruff` is not installed locally, so I am leaving that to CI.
  • pytest skills/ and pytest tests/ pass locally when relevant. All 52 bundle tests, the new demo smoke test, and the registry, doc, card, issuer, identity, manifest and extras guards pass. The full suite has 21 local failures (async tests, the
    bio demo, max_skills_cap), and the same 21 fail on main in my environment: the venv is missing `pytest-async
  • CHANGELOG.md updated under [Unreleased] when user-visible behavior changes
  • examples/README.md updated if this PR adds, renames, or removes a runnable script
  • Ran pytest tests/test_registry_docs.py when skills, examples index, or agent-loops matrix changed
  • Ran pytest tests/test_skill_docs.py when catalog Usage Examples or provider snippets changed

New or updated skill

Bundle and metadata

  • Skill at skills/<category>/<skill_name>/ (from templates/python_skill/ or equivalent)
  • manifest.yaml: name (full ID), version, description, parameters, constitution, real issuer
  • Optional: short_description, issuer.github, issuer.org, requirements, env_vars

Effect, Directive, Assurance

  • Deterministic skill.py (Effect; no ad-hoc LLM-generated execution paths)
  • instructions.md (Directive) explains when and how to use the skill
  • card.json (Presentation) issuer matches manifest when present
  • test_skill.py (Assurance) covers execution and schema expectations
  • SkillLoader.load_skill("<category>/<skill_name>") succeeds (or deps documented)

Documentation and catalog

  • docs/skills/<category>/<skill_name>.md, category hub row, row in docs/skills/README.md, and `docs/site
  • Usage Examples for Gemini, Claude, OpenAI, DeepSeek, Ollama per [skill usage template](docs/usage/skill_us

Constitution and safety (skills only)

  • Offline and deterministic. Exact bundled constants; no network, LLM, environment variables or dependencies
  • Never a silent guess. A value with a choice behind it always carries assumed, assumption, alternatives the caller locked the reading with assume`.
  • Hard dimension guard. No conversion across dimensions, and compound strings that mix dimensions or offset
  • A bad record never aborts a batch. Every failure is a structured error with a stable code and the record sults are rejected, and execute()` returns an envelope for any input instead of raising.

Related Issues

Fixes #167

@rosspeili rosspeili left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rizzoMartin This implementation of a new skill is top tier! Well done <3

Before merge, please address these two small items:

  1. Rebase on main: PR #465 and a badge palette chore recently merged, causing a merge conflict in CHANGELOG.md under ### Added. Please rebase onto current main and keep both unreleased entries.
  2. SPDX Headers: Add # SPDX-License-Identifier: MIT to line 1 of skill.py, units.py, parser.py, and test_skill.py to conform with repo licensing conventions.
  3. (Optional but recommended): Add a standalone runnable demo examples/unit_normalizer_demo.py showcasing batch conversion, compound parsing, and table normalization to match the other registry skills.

Once rebased, this is ready to merge. Excellent work on this bundle.

Adds the directive, output card and fixture, catalog page with five-provider usage examples, category and library rows, sitemap, install extras, agent loops row, and changelog entry. Fixes ARPAHLS#167
Rejects non-finite parsed numbers and results, lets an ambiguous source follow an ambiguous target (gal to oz), skips blank and NaN table cells, parses degree-minute-second strings without spaces, and returns an error envelope for non-object params. Refs ARPAHLS#167
@rizzoMartin
rizzoMartin force-pushed the feat/issue-167-unit-normalizer branch from 2f8003e to 29012de Compare November 3, 2026 14:14

@rosspeili rosspeili left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM @rizzoMartin! This is a textbook level design. Thanks <3

Merging as is.

@rosspeili
rosspeili merged commit c6507b6 into ARPAHLS:main Nov 5, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[New Skill]: data_engineering/unit_normalizer — structured unit normalization for agent pipelines

2 participants