diff --git a/CHANGELOG.md b/CHANGELOG.md index fe6a75f..4cb18f0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ - `in2lambda draft field replace FIELD OLD NEW` changes the wording inside a field that is already written, for the faults only an edit can fix - a brace the OCR dropped out of some maths, which no range of the source says correctly. OLD has to occur in the field exactly once, or the command is refused saying how many times it occurs; `--regex` reads it as a regular expression and NEW as what to replace it with. The field is left quoting the lines it was taken from, at the layer that wrote it, but recorded as edited and by whoever replaced the wording, so the change can be shown against the source. - `in2lambda spec run SPEC` runs a YAML file of selectors over the frozen source: it says which blocks are questions, parts and solutions, which to ignore, what to strip off the front of each one, and which of the four filters lays the solutions out. It fills in the draft's fields with the markdown of the lines each was taken from, records the spec's name and hash in the log so a replay runs the same file, and reports every block it made nothing of. Running an edited spec over a draft it has already filled in is refused, as freezing a document that has changed is: `in2lambda source add --start-over` begins the draft again. Reading a spec needs pyyaml, which the `convert` extra now installs alongside panflute. See [the spec page](https://lambda-feedback.github.io/in2lambda/spec.html) for the selectors and layouts. - A field quoted out of a list item is now dedented as commonmark reads the item: the marker comes off the first line and as much of the same width off every line under it. So a question written `1. ` no longer carries its number, a continuation line no longer arrives indented far enough to be rendered as a code block, and a spec's `strip` is left with what pandoc does not read as a marker. Values written by `in2lambda spec run`, `in2lambda draft question add`, `in2lambda draft part add` and `in2lambda draft question solution` change accordingly; the ranges behind them still name the same source lines. -- `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, parts nothing answers, and fields holding nothing. Each finding names the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood. It also checks over the set the draft describes, as a converted document is checked at export - maths delimiters, what KaTeX will not render, images the export would not carry, and the compile Lambda Feedback's PDF generator does where pandoc and xelatex are installed, with a warning saying what to install where they are not - and reports each of those against the draft field the text is written in, so that `in2lambda build` refuses them as it refuses anything else in the report. -- `in2lambda build` writes the draft in this directory out as a Lambda Feedback set: one question per `qN.text` field, holding the parts written for it and the worked solutions, with the images those fields refer to under `media/`, as `in2lambda convert` writes a set - a field naming an image that is not beside the draft is refused saying which file is missing, since the checks read the draft and not the folder it is in, and a question's own solution written beside a solution for every part it has becomes a part of its own holding just that solution, as `convert` pairs them up. It is refused unless `in2lambda validate` has been run since the draft last changed - every command that changes one drops its report - and found nothing, and the refusal prints what the report says so it can be acted on without opening the draft. `in2lambda render` writes each question as a PDF instead, compiled as Lambda Feedback's own PDF generator compiles it, which needs pandoc and xelatex; it is gated on nothing, since looking at a draft is how what the checks found gets fixed. Both take `-o/--out`, as `convert` does. +- `in2lambda validate` checks a draft over as a whole and writes what it finds into it as a `report`: source blocks in no field and not marked ignore, two fields taken from the same lines, gaps in the numbering of the questions or their parts, and fields holding nothing, each at level `error`; a part, or a question written without parts, that nothing in the draft answers is reported at level `warning` instead. Each finding names the level, the field and the lines it is about, so it can be acted on without reading the draft. Finding something is not a failure and the command still exits 0; the report is replaced by the next run of the checks and dropped by the next command that changes the draft, since it describes the draft as it stood. It also checks over the set the draft describes, as a converted document is checked at export - maths delimiters, what KaTeX will not render, images the export would not carry, and the compile Lambda Feedback's PDF generator does where pandoc and xelatex are installed, with a warning saying what to install where they are not - and reports each of those against the draft field the text is written in, at level `error`, so that `in2lambda build` refuses them as it refuses anything else at that level. +- `in2lambda build` writes the draft in this directory out as a Lambda Feedback set: one question per `qN.text` field, holding the parts written for it and the worked solutions, with the images those fields refer to under `media/`, as `in2lambda convert` writes a set - a field naming an image that is not beside the draft is refused saying which file is missing, since the checks read the draft and not the folder it is in, and a question's own solution written beside a solution for every part it has becomes a part of its own holding just that solution, as `convert` pairs them up. It is refused unless `in2lambda validate` has been run since the draft last changed - every command that changes one drops its report - and found nothing at level `error`, and the refusal prints those findings so they can be acted on without opening the draft. A finding at level `warning` - a part or question nothing in the draft answers - does not stop it: half the sheets there are keep their solutions in another file or have none at all, so the warning is printed and the set written all the same, rather than a solution having to be invented to quiet it. `in2lambda render` writes each question as a PDF instead, compiled as Lambda Feedback's own PDF generator compiles it, which needs pandoc and xelatex; it is gated on nothing, since looking at a draft is how what the checks found gets fixed. Both take `-o/--out`, as `convert` does. - Importing `in2lambda.katex_convert` no longer writes a file called `log` into the working directory. What it has to say about a converted expression goes to the `in2lambda.katex_convert` logger, which is silent unless the application configures logging. - The Python API is unchanged: `in2lambda.main.runner` and everything under `in2lambda.api` take the same arguments and return the same objects. diff --git a/in2lambda/draft/export.py b/in2lambda/draft/export.py index 5e9b38a..f077576 100644 --- a/in2lambda/draft/export.py +++ b/in2lambda/draft/export.py @@ -5,8 +5,10 @@ is the one place that reads the one as the other, so both what is written out and what is rendered for review come from the same reading of the draft. -:func:`build` refuses a draft the checks have not looked at, or have something to say -about. There is no timestamp in that: every command that changes a draft takes its +:func:`build` refuses a draft the checks have not looked at, or have found an error in; +what they found at level warning - a question or part nothing answers - it says and +exports anyway, since a sheet whose solutions are in another file or nowhere is still a +sheet. There is no timestamp in that: every command that changes a draft takes its report with it, so a draft holding one has been checked since it last changed, and `in2lambda.source.frozen` refuses one whose source has moved on underneath it. :func:`render` is gated on nothing, since looking at a draft is how what the checks @@ -191,25 +193,38 @@ def build(directory: str = ".", output_dir: str = "out") -> Path: Raises: NotValidated: the draft has not been checked since it last changed, or the - checks found something. Either way what would be uploaded is not what + checks found an error in it. Either way what would be uploaded is not what anybody has looked at. MissingImage: a field refers to an image file that is not beside the draft. SourceError: the draft is missing, is not one of ours, or was written from markdown that has changed since. + + Warns: + UserWarning: once per finding the checks made at level warning, which is a + question or part the draft has no solution for. The set is written with it. """ + # Here rather than at the top of the module: `report` checks the set this writes, so + # it imports this, and only what reads a report - this one function - needs it back. + from in2lambda.draft.report import errors + draft, _ = frozen(directory) if "report" not in draft: raise NotValidated( f"{DRAFT} has not been validated since it last changed, so what it would " "export is what nothing has checked. Run in2lambda validate." ) - if draft["report"]: + if refusing := errors(draft["report"]): raise NotValidated( - "\n".join(finding["message"] for finding in draft["report"]) + "\n".join(finding["message"] for finding in refusing) + f"\n{DRAFT} is not exported while its report says this. Fix what it " "names, or mark the blocks it is about as ignored, and run in2lambda " "validate again." ) + for finding in draft["report"]: + # Said rather than refused: a sheet whose solutions are elsewhere or absent is + # one to export as it stands, and writing one in to quiet this would put wording + # into the set that no source of it says. + warnings.warn(finding["message"], stacklevel=2) exported = as_set(draft, directory) # The export carries every image a field refers to into media/, which is the only # place Lambda Feedback looks for one, so a file that is not there is not something diff --git a/in2lambda/draft/report.py b/in2lambda/draft/report.py index 92ce094..b31f806 100644 --- a/in2lambda/draft/report.py +++ b/in2lambda/draft/report.py @@ -13,6 +13,15 @@ what the text of a question says is `in2lambda.validation`'s: the set the draft describes is exported and checked over as well, so that maths Lambda Feedback will not render is reported against the field it is written in rather than found after uploading. + +Each finding carries the level it is found at, which is what `in2lambda.draft.export` +goes by. An error is the draft contradicting its own source or its own export - lines +nothing accounts for, two fields quoting the same ones, a numbering with a hole in it, a +quotation of nothing, maths that will not render - and there is no sheet those are right +about. A warning is something that may well be right: half the sheets there are write +their solutions in another file, or have none, so a question nothing answers is said to +whoever is building the set rather than stopping them - inventing a solution to quiet it +is the one thing nobody wanted. """ import re @@ -25,20 +34,30 @@ from in2lambda.validation import pdf Finding = dict[str, Any] -"""One thing a check found: ``{"check", "field", "ranges", "message"}``. +"""One thing a check found: ``{"check", "level", "field", "ranges", "message"}``. ``check`` is which check found it - ``problem`` where it was `in2lambda.validation`, -over the set the draft describes - ``field`` the block id or field key it is about, -``ranges`` the lines in question as ``[[start, end], ...]``, and ``message`` a sentence -naming all of that, so that a line of the report can be acted on by itself. +over the set the draft describes - ``level`` :data:`ERROR` or :data:`WARNING`, ``field`` +the block id or field key it is about, ``ranges`` the lines in question as +``[[start, end], ...]``, and ``message`` a sentence naming all of that, so that a line of +the report can be acted on by itself. """ +ERROR = "error" +"""A finding the draft cannot be exported over: it says something its source does not.""" + +WARNING = "warning" +"""A finding the export says and goes on past: it may be what the sheet really is.""" + _NUMBERED = re.compile(r"((?:q\d+\.p)|q)(\d+)\.text") """A question's or a part's text, split into what numbers it and the number.""" _PART = re.compile(r"(q\d+)\.p\d+\.text") """A part's text, and the question it belongs to.""" +_QUESTION = re.compile(r"(q\d+)\.text") +"""A question's text, and the question it is.""" + _UNPLACED = float("inf") """Where a finding about no particular line sorts: after every finding about one.""" @@ -120,6 +139,7 @@ def uncovered(draft: dict[str, Any]) -> list[Finding]: found.append( { "check": "uncovered", + "level": ERROR, "field": block["id"], "ranges": free, "message": f"{block['id']}{_where(free)} is in no field and not " @@ -140,6 +160,7 @@ def _overlaps(draft: dict[str, Any]) -> list[Finding]: return [ { "check": "overlap", + "level": ERROR, "field": key, "ranges": fields[key]["ranges"], "message": f"{key}{_where(fields[key]['ranges'])} and {other}" @@ -164,6 +185,7 @@ def _gaps(draft: dict[str, Any]) -> list[Finding]: return [ { "check": "gap", + "level": ERROR, "field": f"{prefix}{missing}.text", "ranges": [], "message": f"There is no {prefix}{missing}.text, though " @@ -176,28 +198,47 @@ def _gaps(draft: dict[str, Any]) -> list[Finding]: def _without_solutions(draft: dict[str, Any]) -> list[Finding]: - """Parts that nothing in the draft answers. + """Parts, and questions written without any, that nothing in the draft answers. A part is answered by its own solution or by the solution of the question it belongs - to, since a sheet often writes one worked solution covering every part at once. + to, since a sheet often writes one worked solution covering every part at once. A + question with parts is answered through them and is not reported itself; one with + none is a question in its own right, and is reported where nothing answers it. """ fields = draft["fields"] found = [] for key in sorted(fields): - if (named := _PART.fullmatch(key)) is None: - continue - part = key.removesuffix(".text") - if f"{part}.solution" in fields or f"{named[1]}.solution" in fields: - continue - found.append( - { - "check": "no-solution", - "field": part, - "ranges": fields[key]["ranges"], - "message": f"{part}{_where(fields[key]['ranges'])} has no solution: " - f"neither {part}.solution nor {named[1]}.solution is written.", - } - ) + if named := _PART.fullmatch(key): + part = key.removesuffix(".text") + if f"{part}.solution" in fields or f"{named[1]}.solution" in fields: + continue + found.append( + { + "check": "no-solution", + "level": WARNING, + "field": part, + "ranges": fields[key]["ranges"], + "message": f"{part}{_where(fields[key]['ranges'])} has no solution: " + f"neither {part}.solution nor {named[1]}.solution is written.", + } + ) + elif named := _QUESTION.fullmatch(key): + question = named[1] + if f"{question}.solution" in fields or any( + (belongs := _PART.fullmatch(other)) and belongs[1] == question + for other in fields + ): + continue + found.append( + { + "check": "no-solution", + "level": WARNING, + "field": question, + "ranges": fields[key]["ranges"], + "message": f"{question}{_where(fields[key]['ranges'])} has no " + f"solution: {question}.solution is not written, and it has no parts.", + } + ) return found @@ -206,6 +247,7 @@ def _empty(draft: dict[str, Any]) -> list[Finding]: return [ { "check": "empty", + "level": ERROR, "field": key, "ranges": field["ranges"], "message": f"{key}{_where(field['ranges'])} is empty.", @@ -224,7 +266,9 @@ def checks(draft: dict[str, Any]) -> list[Finding]: Returns: One :data:`Finding` per thing found, earliest line first and then by what it is about, with the findings about no particular line last. An empty list means the - draft covers its source once each, with nothing missing from its numbering. + draft covers its source once each, with nothing missing from its numbering; a + list holding only warnings is one `in2lambda.draft.export.build` says and + exports over. Examples: >>> from in2lambda.draft.report import checks @@ -253,6 +297,25 @@ def _order(finding: Finding) -> tuple[int | float, str]: ) +def errors(findings: list[Finding]) -> list[Finding]: + """The findings of a report that a draft cannot be exported over. + + Args: + findings: A report, as :func:`checks` or :func:`validate` writes one. + + Returns: + Those at level :data:`ERROR`, in the order they were reported. The rest are + warnings, which `in2lambda.draft.export.build` says and exports anyway. + + Examples: + >>> from in2lambda.draft.report import errors + >>> report = [{"level": "warning"}, {"level": "error", "check": "gap"}] + >>> errors(report) + [{'level': 'error', 'check': 'gap'}] + """ + return [finding for finding in findings if finding["level"] == ERROR] + + def problems(draft: dict[str, Any], directory: str = ".") -> list[Finding]: """What `in2lambda.validation` finds in the set the draft describes. @@ -270,6 +333,8 @@ def problems(draft: dict[str, Any], directory: str = ".") -> list[Finding]: rather than by the question and part of the export, so that a line of it can be acted on with `field replace`. A problem about no one field - the set as a whole failing to compile - keeps the validator's own naming of where it is. + All of them are at level :data:`ERROR`: what Lambda Feedback will not render is + not something to upload. Warns: UserWarning: pandoc or xelatex is not installed, so the set was not compiled. @@ -306,6 +371,7 @@ def problems(draft: dict[str, Any], directory: str = ".") -> list[Finding]: rest = problem.location[len(location) :] finding = { "check": "problem", + "level": ERROR, "field": key, "ranges": ranges, "message": f"{key}{_where(ranges)}{rest}: {problem.message}", @@ -313,6 +379,7 @@ def problems(draft: dict[str, Any], directory: str = ".") -> list[Finding]: else: finding = { "check": "problem", + "level": ERROR, "field": "", "ranges": [], "message": str(problem), diff --git a/in2lambda/main.py b/in2lambda/main.py index 35bb26f..853aaab 100644 --- a/in2lambda/main.py +++ b/in2lambda/main.py @@ -52,6 +52,23 @@ def _message_not_traceback() -> Iterator[None]: raise click.ClickException(str(error)) from None +@contextmanager +def _warnings_said() -> Iterator[None]: + """Echoes whatever is warned inside it as a line, as `runner` says its problems. + + What `build` and `render` warn about is something they wrote out anyway - a question + nothing answers, a question xelatex gave up on - so it belongs beside what they + wrote, and is said even where the command goes on to refuse for another reason. + """ + with warnings.catch_warnings(record=True) as said: + warnings.simplefilter("always") + try: + yield + finally: + for warning in said: + click.echo(f"Warning: {warning.message}") + + def docx_to_md(docx_file: str) -> str: """Converts .docx files to markdown. @@ -423,18 +440,24 @@ def validate() -> None: """Checks the draft in this directory over and writes the report into it. Reports source blocks in no field and not marked ignore, two fields taken from the - same lines, gaps in the numbering of the questions or their parts, parts nothing - answers, and fields holding nothing. The set the draft describes is checked over as - well - maths delimiters, what KaTeX will not render, images the export would not - carry, and the compile Lambda Feedback's PDF generator does where pandoc and xelatex - are installed - each against the field it is written in. Finding something is not a - failure: the report is written into draft.json either way, and replaced by the next - one. + same lines, gaps in the numbering of the questions or their parts, and fields holding + nothing. The set the draft describes is checked over as well - maths delimiters, what + KaTeX will not render, images the export would not carry, and the compile Lambda + Feedback's PDF generator does where pandoc and xelatex are installed - each against + the field it is written in. All of those in2lambda build refuses; a question or part + nothing answers is reported as a warning, which it builds over. Finding something is + not a failure: the report is written into draft.json either way, and replaced by the + next one. """ with _message_not_traceback(): report = in2lambda.draft.report.validate() for finding in report: - click.echo(finding["message"]) + # Marked as such, since the two are acted on differently and the report is often + # read off the terminal rather than out of the draft. + if finding["level"] == in2lambda.draft.report.WARNING: + click.echo(f"Warning: {finding['message']}") + else: + click.echo(finding["message"]) if not report: click.echo("Nothing to report.") @@ -457,9 +480,11 @@ def build(output_dir: str) -> None: """Writes the draft in this directory out as a Lambda Feedback set. Refused unless in2lambda validate has been run since the draft last changed and - found nothing, so that what is uploaded is what the checks have been over. + found no error, so that what is uploaded is what the checks have been over. What it + found at level warning - a question or part with no solution written for it - is + said, and the set written all the same. """ - with _message_not_traceback(): + with _message_not_traceback(), _warnings_said(): written = in2lambda.draft.export.build(output_dir=output_dir) click.echo(f"Wrote {written}") @@ -473,14 +498,10 @@ def render(output_dir: str) -> None: needs pandoc and xelatex. What the checks have to say about the draft is not asked: a draft is rendered to look at, including one there is something to fix in. """ - with _message_not_traceback(): - # As `runner` does: a question xelatex complains about is still written out, and - # what it refused is a line to read rather than a traceback. - with warnings.catch_warnings(record=True) as refused: - warnings.simplefilter("always") - written = in2lambda.draft.export.render(output_dir=output_dir) - for warning in refused: - click.echo(f"Warning: {warning.message}") + # A question xelatex complains about is still written out, and what it refused is a + # line to read rather than a traceback. + with _message_not_traceback(), _warnings_said(): + written = in2lambda.draft.export.render(output_dir=output_dir) for pdf in written: click.echo(f"Wrote {pdf}") diff --git a/tests/fixtures/drafts/README.md b/tests/fixtures/drafts/README.md index 82f2cbf..b6cd92d 100644 --- a/tests/fixtures/drafts/README.md +++ b/tests/fixtures/drafts/README.md @@ -3,7 +3,9 @@ Each folder here is one run: a `source.md` to freeze, the `commands.json` to apply to the draft of it, the `expected.json` those commands should leave in the draft's `fields`, and the `report.json` that `in2lambda validate` should then find in it - a folder with no `report.json` -is a draft with nothing wrong with it. The test freezes the source, applies each command, checks +is a draft with nothing wrong with it, and one whose findings are all at level `warning` is a +draft `in2lambda build` says them over and exports anyway, where a `level` of `error` refuses +it. The test freezes the source, applies each command, checks the draft over, compares the fields and the report, and then replays the draft from its log and checks the file is unchanged byte for byte - so a folder covers both what a command writes and that it can be rebuilt from what it recorded. @@ -32,8 +34,11 @@ it as a part of its own rather than dropping the wording. `nested_list` is a num range because only the top-level item is a block: each field is dedented by its own depth, four spaces for the question and eight for the parts, while its range still names the source lines. `question_without_parts` is a question -and nothing else, which the checks have nothing to say about: it is here because a question with +and nothing else, which the checks warn has no solution: it is here because a question with no parts is what the export has to write out as an empty part rather than as the template's. +`questions_only` is the sheet half the corpus is - a title, two questions, two parts under the +first of them and no solution written anywhere - so every one of its findings is a warning, and +it is the folder that says a draft with nothing answering it is still exported. `degrees` writes `^\circ` into the maths of both a question and the one worked solution answering its two parts, and is the one folder whose report comes from `in2lambda.validation` over the set the draft describes rather than from the checks over the draft itself: the solution is reported diff --git a/tests/fixtures/drafts/degrees/report.json b/tests/fixtures/drafts/degrees/report.json index 32b880f..0f31f76 100644 --- a/tests/fixtures/drafts/degrees/report.json +++ b/tests/fixtures/drafts/degrees/report.json @@ -2,6 +2,7 @@ { "check": "problem", "field": "q1.text", + "level": "error", "message": "q1.text (lines 3-3): ^\\circ does not display; write the degree sign \u00b0 instead", "ranges": [ [ @@ -13,6 +14,7 @@ { "check": "problem", "field": "q1.solution", + "level": "error", "message": "q1.solution (lines 9-9): ^\\circ does not display; write the degree sign \u00b0 instead", "ranges": [ [ diff --git a/tests/fixtures/drafts/empty_field/report.json b/tests/fixtures/drafts/empty_field/report.json index 015677f..c435686 100644 --- a/tests/fixtures/drafts/empty_field/report.json +++ b/tests/fixtures/drafts/empty_field/report.json @@ -1,7 +1,20 @@ [ + { + "check": "no-solution", + "field": "q2", + "level": "warning", + "message": "q2 (lines 2-2) has no solution: q2.solution is not written, and it has no parts.", + "ranges": [ + [ + 2, + 2 + ] + ] + }, { "check": "empty", "field": "q2.text", + "level": "error", "message": "q2.text (lines 2-2) is empty.", "ranges": [ [ @@ -9,5 +22,17 @@ 2 ] ] + }, + { + "check": "no-solution", + "field": "q1", + "level": "warning", + "message": "q1 (lines 3-3) has no solution: q1.solution is not written, and it has no parts.", + "ranges": [ + [ + 3, + 3 + ] + ] } ] diff --git a/tests/fixtures/drafts/field_replace/report.json b/tests/fixtures/drafts/field_replace/report.json index 30d6ccd..d36a251 100644 --- a/tests/fixtures/drafts/field_replace/report.json +++ b/tests/fixtures/drafts/field_replace/report.json @@ -2,6 +2,7 @@ { "check": "uncovered", "field": "b1", + "level": "error", "message": "b1 (lines 1-1) is in no field and not marked ignore.", "ranges": [ [ @@ -13,6 +14,7 @@ { "check": "uncovered", "field": "b3", + "level": "error", "message": "b3 (lines 6-6) is in no field and not marked ignore.", "ranges": [ [ diff --git a/tests/fixtures/drafts/mark_ignore/report.json b/tests/fixtures/drafts/mark_ignore/report.json index d23d4f2..08cd014 100644 --- a/tests/fixtures/drafts/mark_ignore/report.json +++ b/tests/fixtures/drafts/mark_ignore/report.json @@ -2,6 +2,7 @@ { "check": "uncovered", "field": "b2", + "level": "error", "message": "b2 (lines 3-3) is in no field and not marked ignore.", "ranges": [ [ @@ -13,6 +14,7 @@ { "check": "uncovered", "field": "b3", + "level": "error", "message": "b3 (lines 5-5) is in no field and not marked ignore.", "ranges": [ [ @@ -24,6 +26,7 @@ { "check": "uncovered", "field": "b4", + "level": "error", "message": "b4 (lines 6-6) is in no field and not marked ignore.", "ranges": [ [ @@ -35,6 +38,7 @@ { "check": "uncovered", "field": "b5", + "level": "error", "message": "b5 (lines 8-10) is in no field and not marked ignore.", "ranges": [ [ @@ -46,6 +50,7 @@ { "check": "uncovered", "field": "b6", + "level": "error", "message": "b6 (lines 12-12) is in no field and not marked ignore.", "ranges": [ [ diff --git a/tests/fixtures/drafts/nested_list/report.json b/tests/fixtures/drafts/nested_list/report.json index 6c931f3..c582b7d 100644 --- a/tests/fixtures/drafts/nested_list/report.json +++ b/tests/fixtures/drafts/nested_list/report.json @@ -2,6 +2,7 @@ { "check": "uncovered", "field": "b2", + "level": "error", "message": "b2 (lines 5-5, 8-8) is in no field and not marked ignore.", "ranges": [ [ @@ -17,6 +18,7 @@ { "check": "no-solution", "field": "q1.p1", + "level": "warning", "message": "q1.p1 (lines 6-7) has no solution: neither q1.p1.solution nor q1.solution is written.", "ranges": [ [ @@ -28,6 +30,7 @@ { "check": "no-solution", "field": "q1.p2", + "level": "warning", "message": "q1.p2 (lines 9-9) has no solution: neither q1.p2.solution nor q1.solution is written.", "ranges": [ [ diff --git a/tests/fixtures/drafts/part_without_solution/report.json b/tests/fixtures/drafts/part_without_solution/report.json index 49e68ab..02dde55 100644 --- a/tests/fixtures/drafts/part_without_solution/report.json +++ b/tests/fixtures/drafts/part_without_solution/report.json @@ -2,6 +2,7 @@ { "check": "no-solution", "field": "q1.p1", + "level": "warning", "message": "q1.p1 (lines 5-5) has no solution: neither q1.p1.solution nor q1.solution is written.", "ranges": [ [ diff --git a/tests/fixtures/drafts/question_without_parts/report.json b/tests/fixtures/drafts/question_without_parts/report.json new file mode 100644 index 0000000..b701bcd --- /dev/null +++ b/tests/fixtures/drafts/question_without_parts/report.json @@ -0,0 +1,14 @@ +[ + { + "check": "no-solution", + "field": "q1", + "level": "warning", + "message": "q1 (lines 1-1) has no solution: q1.solution is not written, and it has no parts.", + "ranges": [ + [ + 1, + 1 + ] + ] + } +] diff --git a/tests/fixtures/drafts/questions_only/commands.json b/tests/fixtures/drafts/questions_only/commands.json new file mode 100644 index 0000000..3f2b3c0 --- /dev/null +++ b/tests/fixtures/drafts/questions_only/commands.json @@ -0,0 +1,39 @@ +[ + { + "args": { + "block": "b1" + }, + "by": "tests", + "command": "mark ignore" + }, + { + "args": { + "text": "b2" + }, + "by": "tests", + "command": "question add" + }, + { + "args": { + "question": "q1", + "text": "b3" + }, + "by": "tests", + "command": "part add" + }, + { + "args": { + "question": "q1", + "text": "b4" + }, + "by": "tests", + "command": "part add" + }, + { + "args": { + "text": "b5" + }, + "by": "tests", + "command": "question add" + } +] diff --git a/tests/fixtures/drafts/questions_only/expected.json b/tests/fixtures/drafts/questions_only/expected.json new file mode 100644 index 0000000..b141282 --- /dev/null +++ b/tests/fixtures/drafts/questions_only/expected.json @@ -0,0 +1,62 @@ +{ + "b1.ignore": { + "by": "tests", + "edited": false, + "layer": 3, + "ranges": [ + [ + 1, + 1 + ] + ], + "value": true + }, + "q1.p1.text": { + "by": "tests", + "edited": false, + "layer": 3, + "ranges": [ + [ + 5, + 5 + ] + ], + "value": "Find the time of flight." + }, + "q1.p2.text": { + "by": "tests", + "edited": false, + "layer": 3, + "ranges": [ + [ + 7, + 7 + ] + ], + "value": "Find the horizontal range." + }, + "q1.text": { + "by": "tests", + "edited": false, + "layer": 3, + "ranges": [ + [ + 3, + 3 + ] + ], + "value": "A ball is thrown at speed $u$ at an angle $\\theta$ to the horizontal." + }, + "q2.text": { + "by": "tests", + "edited": false, + "layer": 3, + "ranges": [ + [ + 9, + 9 + ] + ], + "value": "A train brakes uniformly from $30$ m/s to rest in $20$ s. Find the deceleration." + } +} diff --git a/tests/fixtures/drafts/questions_only/report.json b/tests/fixtures/drafts/questions_only/report.json new file mode 100644 index 0000000..f70fb6b --- /dev/null +++ b/tests/fixtures/drafts/questions_only/report.json @@ -0,0 +1,38 @@ +[ + { + "check": "no-solution", + "field": "q1.p1", + "level": "warning", + "message": "q1.p1 (lines 5-5) has no solution: neither q1.p1.solution nor q1.solution is written.", + "ranges": [ + [ + 5, + 5 + ] + ] + }, + { + "check": "no-solution", + "field": "q1.p2", + "level": "warning", + "message": "q1.p2 (lines 7-7) has no solution: neither q1.p2.solution nor q1.solution is written.", + "ranges": [ + [ + 7, + 7 + ] + ] + }, + { + "check": "no-solution", + "field": "q2", + "level": "warning", + "message": "q2 (lines 9-9) has no solution: q2.solution is not written, and it has no parts.", + "ranges": [ + [ + 9, + 9 + ] + ] + } +] diff --git a/tests/fixtures/drafts/questions_only/source.md b/tests/fixtures/drafts/questions_only/source.md new file mode 100644 index 0000000..07c803d --- /dev/null +++ b/tests/fixtures/drafts/questions_only/source.md @@ -0,0 +1,9 @@ +# Projectile and braking problems + +A ball is thrown at speed $u$ at an angle $\theta$ to the horizontal. + +(a) Find the time of flight. + +(b) Find the horizontal range. + +A train brakes uniformly from $30$ m/s to rest in $20$ s. Find the deceleration. diff --git a/tests/test_draft.py b/tests/test_draft.py index ed09b48..88fe43f 100644 --- a/tests/test_draft.py +++ b/tests/test_draft.py @@ -3,7 +3,9 @@ Each folder in ``fixtures/drafts`` is a document, the commands to run against its draft and the fields they should write, so covering another command means adding a folder rather than a test. Each is also what `in2lambda build` and `in2lambda render` make of -it: the one with no ``report.json`` is exported, the rest are refused. The remainder is +it: one whose ``report.json`` holds a finding at level error is refused, and one holding +only warnings - a question or part nothing answers - is exported with them said. The +remainder is what the command line does when a replay cannot be trusted - a source that has moved on, a log naming a command nothing has, a draft edited by hand - which is not something a fixture can say. @@ -476,6 +478,8 @@ def test_a_draft_edited_into_an_overlap_or_a_gap_is_reported( ("overlap", "q1.p1.text"), ("gap", "q2.text"), ] + # Both are the draft contradicting its own source, so neither is one to export over. + assert [finding["level"] for finding in report] == ["error", "error"] # Both sides of the overlap, so that either field can be looked at without the draft. assert "q1.text" in report[0]["message"] assert result.output == f"{report[0]['message']}\n{report[1]['message']}\n" @@ -633,23 +637,31 @@ def test_build_refuses_a_draft_that_has_not_been_validated( @pytest.mark.parametrize("folder", DRAFTS, ids=lambda path: path.name) def test_build_follows_the_report(folder: Path, tmp_path: Path, monkeypatch) -> None: - """A draft is exported once the checks have been over it and found nothing.""" + """A draft is exported once the checks have found no error in it. + + A warning does not stop it: a sheet whose solutions are in another file or nowhere at + all is a sheet to export, and is exported with what was found said beside it. + """ monkeypatch.setenv("COLUMNS", "200") monkeypatch.chdir(tmp_path) _built(folder, tmp_path) fields = json.loads((folder / "expected.json").read_text()) + report = _reported(folder) result = CliRunner().invoke(cli, ["build"]) - if report := _reported(folder): + if refusing := [finding for finding in report if finding["level"] == "error"]: assert result.exit_code != 0, result.output - # Every finding, so that what is left to do can be read off the refusal itself. - for finding in report: + # Every error, so that what is left to do can be read off the refusal itself. + for finding in refusing: assert finding["message"] in result.output assert not (tmp_path / "out").exists() return assert result.exit_code == 0, result.output + # And each warning said, since the set was written over it rather than without it. + for finding in report: + assert finding["message"] in result.output exported = tmp_path / "out" / "set.zip" assert exported.is_file()