Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,32 @@ because it turns other people's test suites red.

## [Unreleased]

### Breaking

- **An HTML page with no label has a title that says Page.** The smallest
page and every page written with `--clean` carried an empty title, which
the HTML standard does not allow and the Nu HTML Checker reports as an
error. They now carry `<title>Page</title>`, a word that says nothing
about this tool, since `--clean` is there to keep it out of the file.
Pages with a label are unchanged. The smallest HTML page grows from 118 B
to 122 B, so 118 B to 121 B are now refused, and `tfg preset eject
empty-and-minimal` prints 122 where it printed 118 - a manifest from a run
of that preset records a different `recipe_hash`.
- **The catalogue of every element renames its forms section and its hidden
field.** With `elements=every` the forms section is `id="tfg-forms"` and
the hidden field is `name="tfg-hidden"`, the way the list of links has
always been `tfg-links`. DOMPurify removes an id or a name that shadows a
property of the document, so after sanitising, the two links to the forms
section led nowhere and the hidden field left a submitted form without its
name. The smallest catalogue grows from 6924 B to 6944 B.
- **A newsletter's page says its language.** With `layout=newsletter` the
HTML part opens with `<html lang="en">`, as the pages of the `html` format
do, and the line in Polish and Japanese marks each half with its own
language, `<span lang="pl">` and `<span lang="ja">`, so a screen reader
does not read them as English. The smallest newsletter grows by 10 B in
7bit, and by 57 B to 78 B with 8bit, quoted-printable and base64, which
carry that line. A message laid out `simple` is unchanged.

### Added

- **Films: the `webm` format.** A WebM video with an AV1 picture and no
Expand Down Expand Up @@ -312,6 +338,13 @@ because it turns other people's test suites red.

### Changed

- **The `eml` settings say what Thunderbird and MimeKit do with them.**
Measured with Thunderbird 157.0.1 and MimeKit 4.18.1. `charset` says that
MimeKit with no code page provider tries UTF-8 and then Latin-1, and reads
Latin-2 and Windows-1250 right once one is registered. `message_depth` says that Thunderbird shows 32 levels in full,
and past that shows only the headers of the 32nd message inside, without a
word. `filename_style` says that Thunderbird calls an attachment with no
name Part 1.2, Part 1.3 and so on.
- **`empty-and-minimal` now includes the films, GeoJSON and mail.** The preset builds the
smallest file of every format this build has, so with `formats` left at
`all` it gains a `webm`, an `mp4`, a `geojson` and an `eml` target, and the recipe
Expand Down
4 changes: 2 additions & 2 deletions internal/format/eml/message.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ func newMessage(s settings, seed uint64, children, pics []child) *message {
return &message{
s: s, id: id, children: children, pictures: pics, seed: seed,
plainLine: s.internationalLine("", ""),
htmlLine: s.internationalLine("<p>", "</p>"),
htmlLine: s.htmlInternationalLine(),
images: imageLines(s, pics),
frame: newFrame(s, id),
}
Expand Down Expand Up @@ -189,7 +189,7 @@ func (m *message) plain(o *out) {
// the pictures before them, in its frame.
func (m *message) html(o *out) {
eol := o.eol
o.put(m.frame.htmlBefore, "<html>")
o.put(m.frame.htmlBefore, m.frame.htmlTag)
if m.parity {
o.put(" ")
}
Expand Down
16 changes: 11 additions & 5 deletions internal/format/eml/newsletter.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,15 @@ type frame struct {
headers string
// plainOpen and plainClose stand before and after the plain text.
plainOpen, plainClose string
// htmlBefore stands before <html>, htmlOpen after it and htmlClose after
// the paragraph of words.
htmlBefore, htmlOpen, htmlClose string
// htmlBefore stands before the html element's start tag, htmlTag is that
// tag, htmlOpen stands after it and htmlClose after the paragraph of words.
htmlBefore, htmlTag, htmlOpen, htmlClose string
}

// simpleFrame is the frame of a message laid out simple: the body element and
// nothing else, the bytes this format has always written.
func simpleFrame(s settings) frame {
return frame{htmlOpen: "<body>" + s.textEOL(), htmlClose: "</body></html>"}
return frame{htmlTag: "<html>", htmlOpen: "<body>" + s.textEOL(), htmlClose: "</body></html>"}
}

// newsletterFrame is the frame of a newsletter, unsubscribe being the https
Expand All @@ -67,7 +67,13 @@ func newsletterFrame(s settings, unsubscribe string) frame {
"You receive this because you subscribed to "+newsletterName+".") +
frameLine(s, "Unsubscribe ("+unsubscribe+")"),
htmlBefore: frameLines(s, "<!DOCTYPE html>"),
htmlOpen: frameLines(s, newsletterHead...),
// The page says its language, as the pages of the html format do. The
// Nu HTML Checker asks for it, and a screen reader reads the page in
// the voice it names (until 2026-10-15 the newsletter had none). In the
// frame rather than in the writer, so quoted-printable writes its = as
// =3D like every other attribute.
htmlTag: frameLine(s, `<html lang="en">`),
Comment thread
coderabbitai[bot] marked this conversation as resolved.
htmlOpen: frameLines(s, newsletterHead...),
htmlClose: frameLines(s, newsletterFoot(unsubscribe)...) +
frameLine(s, "</html>"),
}
Expand Down
7 changes: 4 additions & 3 deletions internal/format/eml/settings.go
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ func charsetProperty() format.Property {
Default: CharsetUTF8,
Detail: "The character set of the text when text_encoding is not 7bit. iso-8859-2 and windows-1250 hold Polish and no Japanese, " +
"and they write ą, ś and ź as different bytes, so a program that takes one for the other shows the wrong letters. " +
"MimeKit with no code page provider registered reads both as Latin-1, and Go's standard library converts no character set at all. " +
"MimeKit with no code page provider registered tries UTF-8 and then Latin-1, which shows the Polish letters of both wrong, and reads them right once one is. Go's standard library converts no character set at all. " +
"With 7bit the text is plain ASCII and says us-ascii, whatever this is set to.",
}
}
Expand Down Expand Up @@ -341,7 +341,7 @@ func filenameStyle() format.Property {
Detail: "How the names of attached files and pictures are written. rfc2231 is the standard way. " +
"rfc2047 is the encoded form RFC 2047 forbids inside a name, and many readers decode it anyway. " +
"both writes the two, and content-type puts the name only where old mail programs looked for it. " +
"none writes no name at all, and a program that needs one has to make one up. " +
"none writes no name at all, and a program that needs one has to make one up - Thunderbird 157 calls them Part 1.2, Part 1.3 and so on. " +
"Go's standard library reads no name from content-type and the encoded text itself from rfc2047. " +
"Unless headers is encoded a name is written as it is and only where it goes changes, " +
"so rfc2047 and both write the same bytes.",
Expand All @@ -357,7 +357,8 @@ func depthProperty() format.Property {
Default: "1",
Detail: "How many messages deep the message goes. 1 is a message alone. 2 carries a forwarded message as an attachment, " +
"and each step adds one more inside the last, the smallest message each can be. " +
"Python 3.14's email package runs out of recursion at around 490, and MimeKit stops taking them apart past 513 without a word.",
"Python 3.14's email package runs out of recursion at around 490, and MimeKit stops taking them apart past 513 without a word. " +
"Thunderbird 157 shows up to 32 in full. Past that the 32nd message inside shows only its headers and nothing under it appears, without a word.",
Group: attachmentsGroup,
}
}
34 changes: 32 additions & 2 deletions internal/format/eml/text.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ const (
// Windows-1250 write as different bytes.
const (
polish = "Zażółć gęślą jaźń."
international = polish + " 日本語のテキストです。"
japanese = "日本語のテキストです。"
international = polish + " " + japanese
)

// singleByte is how ISO-8859-2 and Windows-1250 write the letters of the
Expand Down Expand Up @@ -97,7 +98,36 @@ func (s settings) internationalLine(open, close string) string {
if !s.international() {
return ""
}
line := open + s.encodedLine() + close
return s.textLine(open + s.encodedLine() + close)
}

// htmlInternationalLine is the line as the HTML part writes it. A newsletter's
// page says it is English (<html lang="en">), so there each half says which
// language it is in a span of its own - the Polish half, and the Japanese one
// where the character set holds it - or a screen reader reads both as English
// (review of #190, as the html format marks its paragraphs in other scripts).
// A message laid out simple says no language anywhere and keeps the bytes it
// always wrote.
func (s settings) htmlInternationalLine() string {
if s.layout != Newsletter {
return s.internationalLine("<p>", "</p>")
}
if !s.international() {
return ""
}
// Two lines rather than one where the Japanese half comes: the spans make
// the line 83 characters, past the 78 RFC 5322 asks for, and 8bit has no
// soft line break to keep it under (measured by the structural checker).
// A line break inside a paragraph is white space, as the space it replaces.
if _, single := singleByte[s.charset]; single {
return s.textLine(`<p><span lang="pl">` + s.encodedLine() + `</span></p>`)
}
return s.textLine(`<p><span lang="pl">`+polish+`</span>`) + s.textLine(`<span lang="ja">`+japanese+`</span></p>`)
}

// textLine is one line of the text in the encoding the text is written in,
// with the text's line ending.
func (s settings) textLine(line string) string {
if s.encoding == QuotedPrintable {
line = quotedPrintable(line, s.eol)
}
Expand Down
15 changes: 11 additions & 4 deletions internal/format/htmlfile/catalogue.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,15 +42,22 @@ const (
`<meta name="description" content="Every element of the HTML standard, each one at least once">` + "\n" +
"<style>\nmark { background: #ffeb3b }\n</style>\n"

// catalogueBody is the elements themselves. The forms article is
// tfg-forms rather than forms for the reason the links list is tfg-links:
// an id that shadows a document property is taken out by DOMPurify 3.4.17,
// and the two links to it then lead nowhere (O282, until 2026-10-15). The
// hidden field is named tfg-hidden for the same reason - a name hidden
// shadows document.hidden, DOMPurify took it out, and the field left a
// submitted form without one (measured 2026-10-15).
catalogueBody = `<header>
<hgroup><h1>Every element</h1><p>Every element of the HTML standard, each one at least once.</p></hgroup>
<nav><ul><li><a href="#text">Text</a></li><li><a href="#tables">Tables</a></li><li><a href="#forms">Forms</a></li><li><a href="#media">Media</a></li></ul></nav>
<nav><ul><li><a href="#text">Text</a></li><li><a href="#tables">Tables</a></li><li><a href="#tfg-forms">Forms</a></li><li><a href="#media">Media</a></li></ul></nav>
<search><form><label>Search <input type="search" name="q"></label> <button type="submit">Find</button></form></search>
</header>
<main>
<article id="text">
<h2>Text</h2>
<p>Plain <a href="#forms">link</a>, <abbr title="HyperText Markup Language">HTML</abbr>, <b>bold</b>, <bdi>name</bdi>, <bdo dir="rtl">reversed</bdo>, <cite>a title</cite>, <code>code()</code>, <data value="42">forty two</data>, <del>removed</del>, <dfn>term</dfn>, <em>stressed</em>, <i>alternate</i>, <ins>added</ins>, <kbd>Ctrl</kbd>, <mark>marked</mark>, <q>quoted</q>, <s>no longer accurate</s>, <samp>output</samp>, <small>fine print</small>, <span>span</span>, <strong>important</strong>, H<sub>2</sub>O, x<sup>2</sup>, <time datetime="2026-10-14">14 October 2026</time>, <u>annotated</u>, <var>n</var>, long<wbr>word, a line<br>break.</p>
<p>Plain <a href="#tfg-forms">link</a>, <abbr title="HyperText Markup Language">HTML</abbr>, <b>bold</b>, <bdi>name</bdi>, <bdo dir="rtl">reversed</bdo>, <cite>a title</cite>, <code>code()</code>, <data value="42">forty two</data>, <del>removed</del>, <dfn>term</dfn>, <em>stressed</em>, <i>alternate</i>, <ins>added</ins>, <kbd>Ctrl</kbd>, <mark>marked</mark>, <q>quoted</q>, <s>no longer accurate</s>, <samp>output</samp>, <small>fine print</small>, <span>span</span>, <strong>important</strong>, H<sub>2</sub>O, x<sup>2</sup>, <time datetime="2026-10-14">14 October 2026</time>, <u>annotated</u>, <var>n</var>, long<wbr>word, a line<br>break.</p>
<p><ruby>base<rp>(</rp><rt>annotation</rt><rp>)</rp></ruby> and <math><mi>x</mi><mo>=</mo><mn>2</mn></math> and <slot name="note">slot fallback</slot> and <tfg-note>a custom element</tfg-note>.</p>
<h3>Third level</h3>
<h4>Fourth level</h4>
Expand Down Expand Up @@ -78,12 +85,12 @@ const (
<tfoot><tr><td colspan="3">footer row</td></tr></tfoot>
</table>
</article>
<article id="forms">
<article id="tfg-forms">
<h2>Forms</h2>
<form>
<fieldset>
<legend>Every type of input</legend>
<label>hidden <input type="hidden" name="hidden" value="h"></label>
<label>hidden <input type="hidden" name="tfg-hidden" value="h"></label>
<label>text <input type="text" name="text"></label>
<label>search <input type="search" name="search"></label>
<label>tel <input type="tel" name="tel"></label>
Expand Down
2 changes: 1 addition & 1 deletion internal/format/htmlfile/html.go
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ func labelledHead(s settings, r format.Request, min int64) (head, heading string
// does - one for the tab and one for the reader. A fragment has no head
// to put a title in, so it carries only the heading, and the label is
// still visible because a heading is.
head = strings.Replace(plain, emptyTitle, "<title>"+label+"</title>", 1)
head = strings.Replace(plain, placeholderTitle, "<title>"+label+"</title>", 1)
extra = int64(len(head) - len(plain))
}
if extra+int64(len(heading))+min <= r.Bytes {
Expand Down
7 changes: 4 additions & 3 deletions internal/format/htmlfile/markup.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,14 +78,15 @@ const (

// plain is how a page has been spelled since 2026-08-01, the golden hashes
// holding it there. The skeleton is in two halves so the catalogue and the
// style sheet can go into the head between them, and the title is empty until a
// label fills it - the empty pair is what the smallest document carries.
// style sheet can go into the head between them, and the title says Page until
// a label fills it - what the smallest document and every --clean one carry
// (an empty title until 2026-10-15, O277).
var plain = markup{
headOpen: `<!DOCTYPE html>` + "\n" +
`<html lang="en">` + "\n" +
`<head>` + "\n" +
`<meta charset="utf-8">` + "\n" +
emptyTitle + "\n",
placeholderTitle + "\n",
headClose: `</head>` + "\n" +
`<body>` + "\n",
bodyClose: "</body>\n</html>\n",
Expand Down
9 changes: 7 additions & 2 deletions internal/format/htmlfile/page.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,13 @@ package htmlfile
// The markup itself - the skeleton, the label, the levels and the blocks - is
// spelled in markup.go, in every layout and every way of carrying styles.

// emptyTitle is the title a page carries until a label fills it in.
const emptyTitle = `<title></title>`
// placeholderTitle is the title a page carries when no label fills it in: at
// the smallest sizes, and in every page under --clean. Not empty, because the
// standard asks a title for text and the Nu HTML Checker calls an empty one an
// error (O277). A word that says nothing about this tool, because --clean is
// there to keep the tool's description out of the file
// (docs/HTML-6-2026-10-15.md).
const placeholderTitle = `<title>Page</title>`

// byteOrderMark is the UTF-8 mark, the only one a page may carry: the standard
// requires UTF-8 (docs/HTML-STRUCTURE-2026-09-08.md section 1). Written as its
Expand Down
10 changes: 10 additions & 0 deletions internal/guard/emlnewsletter_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,16 @@ func TestTheEMLCheckRefusesANewsletterBrokenOnPurpose(t *testing.T) {
".frame { width: 600px; background: #ffffff; }\r\n"+
".button { background: #1a73e8; color: #ffffff; text-decoration: none; }\r\n", "<style>\r\n"),
"layout=newsletter and the HTML lacks: style sheet in its head"},
// The page's language, which the frame writes since 2026-10-15 and
// nothing read before (docs/HTML-6-2026-10-15.md).
{newsletter, newsletter, replaced(`<html lang="en">`, "<html>"),
"layout=newsletter and the HTML lacks: language of the page"},
// Each half of the line in Polish and Japanese says its language, since
// the page says English (review of #190).
{newsletter, newsletter, replaced(`<span lang="pl">`, "<span>"),
"layout=newsletter and the HTML lacks: the Polish line marked as Polish"},
{newsletter, newsletter, replaced(`<span lang="ja">`, "<span>"),
"layout=newsletter and the HTML lacks: the Japanese line marked as Japanese"},
}
dir := t.TempDir()
for i, c := range cases {
Expand Down
7 changes: 7 additions & 0 deletions internal/guard/generatorbytes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,13 @@ func pageGoldenCases() map[string]engine.Target {
Properties: map[string]string{"structure": "fragment", "bom": "true", "nesting": "2"}},
"html_64kib_everything": {ID: "g", Format: "html", Sizes: engine.Uniform(1, 65536), Label: true,
Properties: map[string]string{"nesting": "600", "characters": "international", "elements": "every", "bom": "true"}},
// The two pages no label fills the title of, which carry the
// placeholder: the smallest document, where the label has no room, and
// a page under --clean. No case held either until 2026-10-15, when the
// placeholder went from empty to Page and moved bytes nothing pinned
// (O277, docs/HTML-6-2026-10-15.md).
"html_smallest_page": {ID: "g", Format: "html", Sizes: engine.Uniform(1, 122), Label: true},
"html_4kib_clean": {ID: "g", Format: "html", Sizes: engine.Uniform(1, 4096), Label: false},
// Pictures inside the page: the default format, and an odd size in a
// fragment inside levels, with a format whose own bytes are text. Each
// picture is the file its format writes alone, so these also pin the
Expand Down
16 changes: 15 additions & 1 deletion internal/guard/htmlfragment_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,8 @@ func TestEachHTMLShapeAnswersForItsOwnMinimum(t *testing.T) {
// clothes of a feature. Saying nothing and saying "document" out loud are two
// different routes through the parser and have to meet.
func TestTheDefaultHTMLShapeIsTheBytesItAlwaysWrote(t *testing.T) {
for _, size := range []int64{118, 119, 400, 4096} {
smallest := htmlMinimum(t)
for _, size := range []int64{smallest, smallest + 1, 400, 4096} {
silent := writeEncoded(t, "html", size, nil)
spoken := writeEncoded(t, "html", size, shapeProps(shapeDocument))
if !bytes.Equal(silent, spoken) {
Expand Down Expand Up @@ -215,3 +216,16 @@ func TestTheStructuralCheckIsToldWhichHTMLShapeToExpect(t *testing.T) {
}
}
}

// htmlMinimum is the smallest page the registry declares, asked rather than
// written down. The two tests that ask for it were written with 118 B and 119 B
// in them, and went red on 2026-10-15 when the minimum moved to 122 B for a
// reason that had nothing to do with what they hold (O277).
func htmlMinimum(t *testing.T) int64 {
t.Helper()
d, err := format.Get("html")
if err != nil {
t.Fatal(err)
}
return d.MinBytes
}
3 changes: 2 additions & 1 deletion internal/guard/htmlsettings_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -229,7 +229,8 @@ func TestEveryHTMLSettingIsRecordedInTheManifest(t *testing.T) {
func TestTheDefaultHTMLSettingsAreTheBytesItAlwaysWrote(t *testing.T) {
spoken := map[string]string{"nesting": "0", "characters": "ascii", "elements": "basic", "bom": "false",
"formatting": "lines", "styles": "none", "links": "0", "inline": "0", "inline_format": "png", "inline_size": "8kb"}
for _, size := range []int64{118, 119, 4096} {
smallest := htmlMinimum(t)
for _, size := range []int64{smallest, smallest + 1, 4096} {
if !bytes.Equal(writeEncoded(t, "html", size, nil), writeEncoded(t, "html", size, spoken)) {
t.Errorf("at %d B: saying nothing and saying every default produce different bytes", size)
}
Expand Down
Loading
Loading