Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ All notable changes to this project are documented in this file.

### Added

- **CLI YAML parse and config validation errors now report `file:line:column` source locations and `SKILL.md` remediation links.** `ingests.from_yaml_with_locations` and `ingests.to_sections_with_locations` preserve exact YAML coordinates (including fields inherited from `template:`), `YamlParseError` surfaces malformed YAML coordinates (`yaml-parse-error`), and `SKILL.md` documents step-by-step fixes linked from `flatten_pydantic_error` and validation exceptions. ([#280](https://github.com/SkyeAv/Tablassert/issues/280))
- **NDJSON (JSONL) is a fifth table source kind.** `kind: ndjson` scans newline-delimited JSON lazily with `pl.scan_ndjson` and renames its columns positionally to `column_1..` in first-seen key order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Ragged records scan missing keys as nulls instead of dropping rows; an empty file fails fast with a coded `ndjson-empty` error instead of polars' opaque `Cannot infer NDJSON types` compute error. ([#283](https://github.com/SkyeAv/Tablassert/pull/283), [#262](https://github.com/SkyeAv/Tablassert/issues/262))
- **Every pull request must carry a new changelog entry.** A stdlib-only `scripts/check_changelog.py` requires at least one new bullet under the `## Unreleased` heading of `CHANGELOG.md` relative to the pull request's merge base, failing with remediation guidance when whitespace, subheadings, or released-history edits are the only additions; a CI job runs it on every pull request including forks, release-bump and dependency-bump pull requests are exempt by their `chore(release):` / `chore(deps):` title or commit subject, documentation-only pull requests (every changed path is markdown or under `docs/`) are exempt via `--exempt-docs-only`, and a pre-push hook mirrors the same check locally. ([#284](https://github.com/SkyeAv/Tablassert/pull/284), [#282](https://github.com/SkyeAv/Tablassert/issues/282))
- **Arrow IPC (Feather v2) is a fourth table source kind.** `kind: ipc` scans a `.arrow` file lazily with `pl.scan_ipc` and renames its columns positionally to `column_1..` in schema order, matching the `parquet` reader contract so `method: column` encodings keep using Excel-style letters. Only IPC file layout is supported; a stream-layout file fails fast with a coded `ipc-stream-unsupported` error naming the conversion fix instead of polars' opaque `InvalidFooter` compute error. ([#281](https://github.com/SkyeAv/Tablassert/pull/281), [#264](https://github.com/SkyeAv/Tablassert/issues/264))
Expand Down
62 changes: 62 additions & 0 deletions SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Tablassert Remediation Guide (`SKILL.md`)

Quick remediation steps for common Tablassert CLI and configuration errors.

## `yaml-parse-error` {#yaml-parse-error}

The YAML file could not be parsed at the reported `file:line:column`.

1. Open the file at the reported line and column.
2. Check for unquoted colons (`:`), mismatched indentation, stray tabs, or unclosed quotes/brackets.
3. Re-run `tablassert validate <config> --schema graph` (or `--schema table`).

## `graph-validation-failed` {#graph-validation-failed}

One or more fields in the graph configuration failed schema validation.

1. Check the field path and reported `file:line:column` in the error message.
2. Verify required keys (`name`, `version`, `tables`, `fullmap`, `rig`) against [Graph Configuration](docs/configuration/graph.md).
3. Re-run `tablassert validate <graph.yaml> --schema graph`.

## `section-validation-failed` {#section-validation-failed}

One or more fields in a table configuration section failed schema validation.

1. Check the reported `file:line:column` (including inherited `template:` lines).
2. Verify `source`, `statement`, `provenance`, and `annotations` against [Table Configuration](docs/configuration/table.md).
3. Re-run `tablassert validate <table.yaml> --schema table`.

## `rig-legacy-keys` {#rig-legacy-keys}

Top-level `description`, `contributions`, `ui_explanation`, or `infores` keys moved under the `rig:` section.

1. Move `description` -> `rig.source_info.description`, `infores` -> `rig.source_info.infores_id`, `contributions` -> `rig.provenance_info.contributions`, and `ui_explanation` -> `rig.ui_explanation`.
2. Re-run `tablassert validate <graph.yaml> --schema graph`.

## `taxon-from-conflict` {#taxon-from-conflict}

`taxon_from` and an explicit `taxon` cannot both be set on the same node encoding.

1. Remove either `taxon` or `taxon_from` on the reported encoding.
2. Re-run `tablassert validate <table.yaml> --schema table`.

## `encoding-bad-excel-column` {#encoding-bad-excel-column}

`method: column` expects an Excel-style column letter (`A` through `ZZ`).

1. Replace the `encoding` value with the 1-based column letter (`A`, `B`, ..., `AA`).
2. Re-run `tablassert validate <table.yaml> --schema table`.

## `provenance-missing-publication` {#provenance-missing-publication}

Derived provenance requires `provenance.publication` unless `provenance.override` is set.

1. Provide `provenance.publication` (with `provenance.repo`), or configure `provenance.override`.
2. Re-run `tablassert validate <table.yaml> --schema table`.

## `qualifier-bad-value` {#qualifier-bad-value}

An enum-ranged qualifier literal is outside the allowed Biolink vocabulary.

1. Use one of the permitted Biolink enum values listed in the error message.
2. Re-run `tablassert validate <table.yaml> --schema table`.
32 changes: 17 additions & 15 deletions src/tablassert/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -538,29 +538,31 @@ def validate_pipeline(table_configuration_file: Path, progress: PipelineProgress
Raises:
SectionValidationError: If any section fails Pydantic validation.
"""
from tablassert.ingests import from_yaml, to_sections
from tablassert.ingests import from_yaml_with_locations, to_sections_with_locations
from tablassert.lib import Tcode
from tablassert.utils import STORE, mkhash

# Stage 1/3: load tables.
progress.stage("Loading Tables")
r: object = from_yaml(table_configuration_file)
r, table_locs = from_yaml_with_locations(table_configuration_file)

# Stage 2/3: extract sections.
progress.stage("Extracting Sections")
sections: list[dict[str, Any]] = to_sections(r, table_configuration_file) # pyright: ignore
sections, section_locs = to_sections_with_locations(r, table_configuration_file, table_locs) # pyright: ignore
n: int = len(sections)

# Stage 3/3: validate section syntax.
progress.stage("Validating Section Syntax")
start, advance, _ = progress.section_loop(n, "Validate")
for s in sections:
for s, locs in zip(sections, section_locs, strict=True):
h: str = mkhash(s)
start(f"{Path(str(s['config'])).stem} · {h[:8]}")
try:
Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")})
except pydantic.ValidationError as e:
raise SectionValidationError(table_configuration_file, h, flatten_pydantic_error(e)) from e
raise SectionValidationError(
table_configuration_file, h, flatten_pydantic_error(e, source_file=table_configuration_file, locations=locs)
) from e
advance()

logger.info("Validated {n} sections from {config}", n=n, config=table_configuration_file.name)
Expand All @@ -581,37 +583,37 @@ def validate_graph_pipeline(configuration_file: Path, progress: PipelineProgress
GraphValidationError: If the graph YAML fails Pydantic validation.
SectionValidationError: If any referenced table section fails validation.
"""
from tablassert.ingests import from_yaml, to_sections
from tablassert.ingests import from_yaml_with_locations, to_sections_with_locations
from tablassert.lib import Tcode
from tablassert.models import Graph
from tablassert.utils import STORE, mkhash

# Stage 1/2: validate the graph config.
progress.stage("Validating Graph")
r: object = from_yaml(configuration_file)
r, graph_locs = from_yaml_with_locations(configuration_file)
try:
g: Graph = Graph.model_validate(r)
except pydantic.ValidationError as e:
raise GraphValidationError(configuration_file, flatten_pydantic_error(e)) from e
raise GraphValidationError(configuration_file, flatten_pydantic_error(e, source_file=configuration_file, locations=graph_locs)) from e

# Stage 2/2: validate every referenced table's sections.
progress.stage("Validating Tables")
# Expand each referenced table into sections up front so the bar total is known.
table_sections: list[tuple[Path, dict[str, Any]]] = []
table_sections: list[tuple[Path, dict[str, Any], dict[tuple[str | int, ...], tuple[Path, int, int]]]] = []
for table in g.tables:
raw_table: object = from_yaml(table)
sections: list[dict[str, Any]] = to_sections(raw_table, table) # pyright: ignore
for section in sections:
table_sections.append((table, section))
raw_table, table_locs = from_yaml_with_locations(table)
sections, sec_locs = to_sections_with_locations(raw_table, table, table_locs) # pyright: ignore
for section, locs in zip(sections, sec_locs, strict=True):
table_sections.append((table, section, locs))
n: int = len(table_sections)
start, advance, _ = progress.section_loop(n, "Validate")
for table, s in table_sections:
for table, s, locs in table_sections:
h: str = mkhash(s)
start(f"{Path(str(s['config'])).stem} · {h[:8]}")
try:
Tcode.model_validate({**s, "store": (STORE / f"{h}.parquet")})
except pydantic.ValidationError as e:
raise SectionValidationError(table, h, flatten_pydantic_error(e)) from e
raise SectionValidationError(table, h, flatten_pydantic_error(e, source_file=table, locations=locs)) from e
advance()

logger.info("Validated graph {name}: {n} sections across {tables} tables", name=g.name, n=n, tables=len(g.tables))
Expand Down
54 changes: 43 additions & 11 deletions src/tablassert/errors.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
from __future__ import annotations

import re
from collections.abc import Sequence
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import TYPE_CHECKING, Literal

if TYPE_CHECKING:
from pydantic import ValidationError # pyright: ignore[reportMissingImports]

DOCS_URL: str = "https://tablassert.readthedocs.io/errors/"
SKILL_URL: str = "https://github.com/SkyeAv/Tablassert/blob/main/SKILL.md#"

_SECRET_ASSIGNMENT_RE: re.Pattern[str] = re.compile(
r"""(?ix)
Expand Down Expand Up @@ -44,6 +45,7 @@ def redact_secrets(text: str, secrets: Sequence[str] = ()) -> str:
"missing-extra",
"graph-validation-failed",
"section-validation-failed",
"yaml-parse-error",
"babel-download-failed",
"source-file-unreadable",
"ipc-stream-unsupported",
Expand Down Expand Up @@ -272,33 +274,63 @@ def __init__(self, missing: Sequence[str] = ()) -> None:
self.missing: tuple[str, ...] = tuple(missing)


def flatten_pydantic_error(e: ValidationError) -> str:
"""Flatten a pydantic ``ValidationError`` into a single-line, ``;``-joined summary.
def _lookup_source_location(
loc: tuple[str | int, ...], locations: Mapping[tuple[str | int, ...], tuple[Path, int, int]] | None
) -> tuple[Path, int, int] | None:
"""Return the most specific ``(file, line, column)`` for a Pydantic error path."""
if not locations or not loc:
return None
for end in range(len(loc), 0, -1):
prefix: tuple[str | int, ...] = loc[:end]
if prefix in locations:
return locations[prefix]
return None


Each fragment is ``"<loc>: <msg> [<kind>]"``. ``<kind>`` is the stable
Tablassert error code when the error is coded (recovered from
``ctx["error"].code``), else the pydantic ``type``. Coded messages already
embed their docs URL via ``_Coded.__str__``.
def flatten_pydantic_error(
e: ValidationError, *, source_file: Path | None = None, locations: Mapping[tuple[str | int, ...], tuple[Path, int, int]] | None = None
) -> str:
"""Flatten a pydantic ``ValidationError`` into a ``;``-joined diagnostic summary.

Each fragment includes ``file:line:column`` when known (never fabricated when absent),
``"<loc>: <msg> [<kind>]"``, and a ``SKILL.md#<anchor>`` fix link when available.
"""
parts: list[str] = []
for err in e.errors():
loc: str = ".".join(str(p) for p in err.get("loc", ())) or "<root>"
loc_tuple: tuple[str | int, ...] = tuple(p for p in err.get("loc", ()) if isinstance(p, (str, int)))
loc: str = ".".join(str(p) for p in loc_tuple) or "<root>"
msg: str = str(err.get("msg", "")).replace("\n", " ").replace("|", "/").strip()
ctx: object = err.get("ctx") or {}
code: str | None = getattr(ctx.get("error"), "code", None) if isinstance(ctx, dict) else None
kind: str = code if code is not None else str(err.get("type", ""))
parts.append(f"{loc}: {msg} [{kind}]")
coords: tuple[Path, int, int] | None = _lookup_source_location(loc_tuple, locations)
where: str = f"{coords[0]}:{coords[1]}:{coords[2]} " if coords is not None else (f"{source_file} " if source_file is not None else "")
fix_link: str = f" (fix: {SKILL_URL}{code})" if code is not None else ""
parts.append(f"{where}{loc}: {msg} [{kind}]{fix_link}")
return "; ".join(parts)


class YamlParseError(TablassertError):
"""Coded error for malformed YAML, carrying ``file:line:column`` when available."""

def __init__(self, config: Path, line: int | None, column: int | None, problem: str) -> None:
where: str = f"{config}:{line}:{column}" if line is not None and column is not None else f"{config}"
super().__init__(f"YAML parse error at {where}: {problem}\nFix guide: {SKILL_URL}yaml-parse-error", code="yaml-parse-error")


class GraphValidationError(TablassertError):
def __init__(self, config: Path, detail: str) -> None:
super().__init__(f"Graph config validation failed: {config}\n{detail}", code="graph-validation-failed")
super().__init__(
f"Graph config validation failed: {config}\n{detail}\nFix guide: {SKILL_URL}graph-validation-failed", code="graph-validation-failed"
)


class SectionValidationError(TablassertError):
def __init__(self, config: Path, section_hash: str, detail: str) -> None:
super().__init__(f"Section validation failed: {config} (hash {section_hash[:8]})\n{detail}", code="section-validation-failed")
super().__init__(
f"Section validation failed: {config} (hash {section_hash[:8]})\n{detail}\nFix guide: {SKILL_URL}section-validation-failed",
code="section-validation-failed",
)


class BabelDownloadError(TablassertError):
Expand Down
79 changes: 78 additions & 1 deletion src/tablassert/ingests.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@
import yaml
from yaml import CSafeLoader

from tablassert.errors import YamlParseError

SourceLocation = tuple[Path, int, int]
LocationMap = dict[tuple[str | int, ...], SourceLocation]


def fastmerge(a: list[Any] | dict[str, Any], b: list[Any] | dict[str, Any]) -> Any:
"""Recursively merge ``b`` into ``a`` in place and return the result.
Expand Down Expand Up @@ -42,6 +47,51 @@ def fastmerge(a: list[Any] | dict[str, Any], b: list[Any] | dict[str, Any]) -> A
return b


def _collect_yaml_locations(node: yaml.Node | None, path: Path, prefix: tuple[str | int, ...], out: LocationMap) -> None:
"""Recursively record 1-based ``(path, line, column)`` for every YAML mapping/sequence node."""
if node is None:
return
if prefix:
out[prefix] = (path, node.start_mark.line + 1, node.start_mark.column + 1)
if isinstance(node, yaml.MappingNode):
for key_node, value_node in node.value:
key: object = key_node.value
if isinstance(key, (str, int)):
child_prefix: tuple[str | int, ...] = (*prefix, key)
out[child_prefix] = (path, value_node.start_mark.line + 1, value_node.start_mark.column + 1)
_collect_yaml_locations(value_node, path, child_prefix, out)
elif isinstance(node, yaml.SequenceNode):
for idx, item_node in enumerate(node.value):
_collect_yaml_locations(item_node, path, (*prefix, idx), out)


def from_yaml_with_locations(p: Path) -> tuple[object, LocationMap]:
"""Read a YAML config file and return ``(parsed_data, location_map)``.

Raises:
YamlParseError: With exact ``file:line:column`` when PyYAML reports a ``problem_mark``.
"""
text: str = p.read_text(encoding="utf-8")
try:
data: object = yaml.load(text, Loader=CSafeLoader)
except yaml.constructor.ConstructorError:
raise
except yaml.YAMLError as exc:
mark = getattr(exc, "problem_mark", None)
line: int | None = mark.line + 1 if mark is not None else None
col: int | None = mark.column + 1 if mark is not None else None
problem: str = str(getattr(exc, "problem", None) or exc)
raise YamlParseError(p, line, col, problem) from exc
locations: LocationMap = {}
loader = yaml.SafeLoader(text)
try:
root: yaml.Node | None = loader.get_single_node()
_collect_yaml_locations(root, p, (), locations)
finally:
loader.dispose()
return data, locations


def from_yaml(p: Path) -> object:
"""Read a YAML config file into a Python object.

Expand All @@ -55,7 +105,16 @@ def from_yaml(p: Path) -> object:
# CSafeLoader (not CLoader): config files may be untrusted, and CLoader honors
# tags like !!python/object/apply that construct arbitrary Python objects -> RCE.
# CSafeLoader keeps libyaml's C speed while refusing unsafe object construction.
return yaml.load(f, Loader=CSafeLoader)
try:
return yaml.load(f, Loader=CSafeLoader)
except yaml.constructor.ConstructorError:
raise
except yaml.YAMLError as exc:
mark = getattr(exc, "problem_mark", None)
line: int | None = mark.line + 1 if mark is not None else None
col: int | None = mark.column + 1 if mark is not None else None
problem: str = str(getattr(exc, "problem", None) or exc)
raise YamlParseError(p, line, col, problem) from exc


class _IndentedSafeDumper(yaml.SafeDumper):
Expand Down Expand Up @@ -106,3 +165,21 @@ def to_sections(instructions: dict[str, Any], table: Path) -> list[dict[str, Any
for index, section in enumerate(merged, start=1):
section["section_index"] = index
return merged


def to_sections_with_locations(instructions: dict[str, Any], table: Path, locations: LocationMap) -> tuple[list[dict[str, Any]], list[LocationMap]]:
"""Expand a table config and preserve per-section YAML source locations.

Template-inherited keys keep their ``template.<path>`` coordinates in ``table``;
section-level overrides overwrite with their ``sections.<i>.<path>`` coordinates.
"""
merged: list[dict[str, Any]] = to_sections(instructions, table)
template_locs: LocationMap = {key[1:]: loc for key, loc in locations.items() if len(key) > 1 and key[0] == "template"}
section_locs: list[LocationMap] = []
for idx in range(len(merged)):
loc_map: LocationMap = dict(template_locs)
for key, loc in locations.items():
if len(key) > 2 and key[0] == "sections" and key[1] == idx:
loc_map[key[2:]] = loc
section_locs.append(loc_map)
return merged, section_locs
Loading