From 3932bd8d60f6f540c226d55ed0eebe1b706357e7 Mon Sep 17 00:00:00 2001 From: "Seb H." <138793908+SebHenn@users.noreply.github.com> Date: Thu, 19 Nov 2026 19:36:21 +0100 Subject: [PATCH 1/3] M28: plan for toward 1.0 Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Seb H. <138793908+SebHenn@users.noreply.github.com> --- docs/plan-m28-toward-1-0.md | 162 ++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/plan-m28-toward-1-0.md diff --git a/docs/plan-m28-toward-1-0.md b/docs/plan-m28-toward-1-0.md new file mode 100644 index 0000000..440b83e --- /dev/null +++ b/docs/plan-m28-toward-1-0.md @@ -0,0 +1,162 @@ +# Plan for M28: toward 1.0 + +Status: **plan, no code yet.** Three parts, in this order, each its own pull request. The plan goes to the `privacy-reviewer` agent +before part 1 starts, as M27 did, because part 1 brings material into the archive from outside and part 2 adds public addresses. + +## What is wanted + +1. **Commonage reads its own export**, so two archives can be merged and a group can move into a fresh installation. It uses + the import's dry run and changes nothing until a curator confirms. +2. **Dublin Core and a JSON alternate** for every public item, so other tools can read a record without scraping the page. +3. **A 1.0 checklist** that is written down and checked, not remembered. + +Not planned: reading another system's export (Omeka, Mukurtu, ...); OAI-PMH (it stays in "Later"); restoring users, settings, +the activity log or visitor notes from an export (a backup does that, see `operations.md`). + +## Part 1: reading Commonage's own export + +The input is the ZIP the export writes: `archive.json` with `"format": "commonage-export"` and the files beside it. Versions 4 +and later are read (version 4 is the first with tags). Anything else, or a file that is not a Commonage export, is refused with a +plain message that names the version found and the versions understood. + +**What arrives, and as what** + +- Every item arrives as a **draft**, whatever it was in the source. Reading an export never makes anything public by itself + (rule 1). Its `mediaLevel` and `publicFrom` are kept, so a curator who held something back in one archive does not lose that + choice by moving it; it is shown in the dry run so it is not a surprise. +- Collections are matched by title and otherwise created. Tags are matched by name and kind and otherwise created. The date is + taken as the text written (`date.text`) and read again by the same parser, so a date in a vocabulary this version lacks is + reported in the dry run, not guessed. +- Files are taken from the ZIP by the path `archive.json` names, and are stored through `MediaUploadService`, the same path as an + upload, so size limits, decoding and derivatives are never skipped. The checksum in the export is checked against the file; a + mismatch is reported and the file is left out. +- Alt text, caption, position, rotation, rights URI, credit, description, `needsTranscription` and `helpWanted` come with the item. + A table of every property of the export's item, file, collection, tag and settings records, each marked read, dropped or + dropped-because, is part of the code, and a reflection test fails when the exporter gains a field the table lacks. + +**What does not arrive, and why** + +- **Contributor details and consent text** (name, email, reference code, consent). Consent was given to the other archive for + that archive; carrying it over would claim a consent this archive never collected (rule 3). The item keeps its credit line and + rights statement as text. Left out of the dry run's result with that reason, and said in the handbook. +- Visitor notes, activity log, users, settings, the texts of the pages, the bin, revisions, `internalNotes`, `receivedFrom` and + the other catalogue fields that name a donor. The `notes` and `contribution` parts of the JSON are never deserialised into + storable objects, never logged and never shown. +- The status, the published date and the ids (the same reason as the CSV import: no id or status column on purpose). + +**Merging and repeating.** Duplicates are found as the CSV import finds them: by (collection, title, date) and by the content of a +file, so a second run adds nothing. This needs one decision (see "Decisions"): what to do with an item that is a duplicate but +whose description differs. Proposed: skip it and list it, never overwrite (rule 4's spirit: nothing is lost at once). + +**Safety.** The ZIP is from outside, so it is handled as a contribution is: a size and entry-count limit, a name inside the ZIP is +only a key and never a path, a ZIP made to expand without end is stopped, and the JSON is read as a stream with a depth limit and +a cap on the number of items per run. It waits in `import-staging/` and is removed after the run or a day. One import at a time, +as now. It needs a curator, and is listed under `Administer` like the export. Every import is in the activity log with how many +items and files came in, and names no contributor. + +**Tests.** Export, import into an empty database and export again: the two `archive.json` agree except for the fields listed above +(a round trip test, which also catches a field added to the export and forgotten here). A version 4 file, a file with a wrong +checksum, a ZIP with `../` names, a file with a future version, a second run, and a draft with `mediaLevel` and `publicFrom`. +Against the export test that forces a decision for each new field: the importer gets the same table. + +## Part 2: Dublin Core and a JSON alternate + +- `/items/{id}.json` and, on the item page, `` and `` tags + (`DC.title`, `DC.description`, `DC.date`, `DC.creator` only when the credit is public as it is on the page, `DC.rights` from the + rights URI, `DC.language`, `DC.subject` from tags, `DC.identifier` the page address, `DC.type`). +- The JSON has the same fields as the page shows, nothing more: no ids of private things, no contributor details, no original file + names, no personal data. Files are listed with their public `/media` addresses **only when the item's media are visible**; a + record-only item lists none. +- Both start from `VisibleItems` and `VisibleMedia`. A draft, an embargoed or a binned item answers 404, exactly like its page, so + the address cannot be used to find out that something exists. The JSON address goes into `VisibilityMatrixTests.Addresses` + and `EndpointAccessTests` on purpose, and the guard test covers the new file. +- Cached for a short time only (ten minutes at most, as the feed), so an item taken down does not stay in a cache for long. Said + in `security.md`. +- Dates keep their meaning: the text as written plus the earliest and latest day, as the export does, because "about 1950" + must not become a day. + +## Part 3: the 1.0 checklist + +A page `docs/release-1.0.md` that lists each condition with its evidence, and a test or script where a condition can be +checked by a machine: + +| Condition | How it is checked | +|---|---| +| A real group has run it for a season | Written note from the group (their name only if they agree); not something a test can show | +| The upgrade works from every 0.x minor | `scripts/` upgrade test run against each earlier minor in CI, restoring a backup made by that version | +| Export and import round trip | Part 1's test | +| A support policy | `SUPPORT.md`: which versions get security fixes, for how long, how to report | +| A second person can release | Written handover or second approver, as `handover.md` starts | +| No open `LEAK` from the privacy review, no known accepted limit missing from `security.md` | The last review, dated | +| Accessibility and translations | Browser tests green, no untranslated text in the resource test | + +Nothing in this part changes behaviour, so it comes last and is the only part that needs the user and the group, not code. + +## Decisions to make + +1. **Duplicate with a different description:** skip and list (proposed), or offer to add the other text as a revision. +2. **Contributor details:** left out (proposed), or optional behind a checkbox saying the curator holds that consent. +3. **`publicFrom` and `mediaLevel` kept on import** (proposed) or reset, so everything arrives plain. +4. **Is "a real group running it" a condition for 1.0**, or only for calling it supported? + +## Changes after the review + +The review found no leak in the plan as such, but fourteen gaps. What they change: + +**Part 1** + +- **Sharing consent is not dropped silently.** An item without a contribution always travels (previews, feed, and in part 2 the + JSON). So an item whose source says `consentCoversSharing: false` is **not imported**: it is listed in the dry run with that + reason. Storing the restriction would need a new field and a migration; it waits until a group asks. Test: such an item is + absent after import. +- **Status.** `Rejected` and `Withdrawn` items are skipped and listed, so a withdrawal is not undone. The importer has no + path that writes `Published` or `PublicFrom` beyond copying a validated future-or-past date; a source-file test checks it. +- **Insert only.** Nothing existing is updated, removed or merged, no `Remove` or `ExecuteDelete`. An item that sits in the bin is + not seen by the duplicate check; this is stated in the handbook and tested. +- **Unknown values are issues, not exceptions.** An unknown `status`, `mediaLevel` or tag `kind` is one line in the dry run for that + item. A `publicFrom` must be a real date. +- **Keys.** Titles, tags and collections are compared in NFC with an ordinal, culture-free comparison, so a Mac-written (NFD) + title and a server in a Turkish culture do not make duplicates. Items with files are matched by the checksum first; items + without files can be duplicated by a changed title, and the handbook says so. `coverFileId`, collection position and sort are not + trusted. Collections created per run are capped, and creating one has its own `AuditAction`. +- **Dates.** The exported `from` and `to` are used when present and sane, `text` stays as the display text, and a mismatch is + reported, not fatal. Tests with Chinese, right-to-left and Cyrillic titles and a date text the parser cannot read. +- **The ZIP.** `archive.json` is read through a counting stream with an upper limit (256 MB), `MaxDepth` set, and caps on items, + files per item, tags per item, collections and the length of each field (the limits of `ItemImport`). A total uncompressed + budget covers all entries, not only each one. The dry run decodes outside the request or within a limit, so it cannot time out + and leave staging behind. Tests: a highly compressible `archive.json` and an entry whose `Length` lies. +- **File paths** in `archive.json` must match `files//-...`, be unique per file, and never name `archive.json`, `items.csv`, + `branding/` or a README. No fallback to matching by bare name. `contentType` and `sizeBytes` are untrusted and the type is + detected again. Files the export itself marked `missing` or with a bad checksum are skipped and counted. +- **Where.** The page is for admins like the export, listed in `Program.cs`, `EndpointAccessTests` and `AccessibilityTests`, and the + dry run answers `no-store` because it names other items' titles. +- **Failing midway.** The audit entry is written first and updated at the end; a second run resumes because duplicates are + skipped. "One import at a time" is a lock released in `finally`. +- **`ImportStaging.DeleteOld`** catches only `IOException` today. If it runs in a background loop, a permission error stops the app. + Fix and test that before part 1 adds more staging. +- **Tests across versions.** A round trip with `includesPersonalData=true` leaves no contributor name, email or consent text in + the database, the activity log, the logs or the dry-run page. Exports of versions 4 to 8 and a published-only export. + +**Part 2** + +- **Same rules as the preview.** `DC.description`, the file list and the JSON description are left out when the item may not + travel; the JSON and DC follow the preview rules and not "what the page shows". With search engines off, the JSON answers + `X-Robots-Tag: noindex` and the DC tags are left out. The JSON address is not in the sitemap. +- **A dedicated record with an allow-list**, never a serialised entity. `DC.creator` is exactly what the page prints. Never + `OriginalFileName`, `Sha256`, `SizeBytes`, keys, contribution, next or previous links or totals. A test serialises an item that + has every private field filled and asserts none appears. +- **Hidden means the same answer.** Draft, embargoed, binned, rejected, withdrawn and unknown ids get one identical 404 (same body + and headers as the page's), never 401 or 403. The `` is rendered only inside the visible result. The 200 + has `Cache-Control: public, max-age=600` and no `ETag` or `Last-Modified` from `UpdatedAt`, which would show edit times made + during an embargo; the 404 is `no-store`. Test: embargoed until tomorrow is 404 today and 200 tomorrow with the clock moved. + +**Part 3** + +- Where the dated review output is kept is named, with no contributor details or real archive material in it. +- Every part has its `CHANGELOG.md` entry and `operations.md` update, and `security.md` gets rows for the ZIP import, the public JSON + and the consent decision. + +## Decisions to make (updated) + +Decision 1 to 4 above stand. The review adds one: **items whose source did not allow sharing are skipped (proposed)**, or a field +is added to remember it. From 34a3dbf879d9637a5ecd562c2ae076a30fd0e35f Mon Sep 17 00:00:00 2001 From: "Seb H." <138793908+SebHenn@users.noreply.github.com> Date: Thu, 19 Nov 2026 19:31:00 +0100 Subject: [PATCH 2/3] M27: cover notes, recordings, PDF pages and decades for hidden items The note-form test passed without testing anything, because visitor notes are off by default and every address answered 404. It now switches them on and checks that a shown item opens. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Seb H. <138793908+SebHenn@users.noreply.github.com> --- .../Commonage.Tests/VisibilityMatrixTests.cs | 130 +++++++++++++++++- 1 file changed, 128 insertions(+), 2 deletions(-) diff --git a/tests/Commonage.Tests/VisibilityMatrixTests.cs b/tests/Commonage.Tests/VisibilityMatrixTests.cs index 509ce8e..2f4f94c 100644 --- a/tests/Commonage.Tests/VisibilityMatrixTests.cs +++ b/tests/Commonage.Tests/VisibilityMatrixTests.cs @@ -2,6 +2,9 @@ using System.Text; using Commonage.Web.Data; using Commonage.Web.Domain; +using Commonage.Web.Media; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; namespace Commonage.Tests; @@ -106,12 +109,17 @@ public async Task The_pictures_of_hidden_items_are_not_served_to_visitors() [Fact] public async Task A_visitor_cannot_open_the_note_form_of_a_hidden_item() { + var shown = await SeedAsync("noteshown", (i, now) => i.Publish(now)); var draft = await SeedAsync("notedraft", (_, _) => { }); var embargoed = await SeedAsync("noteembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }); using var visitor = app.CreateClient(); - foreach (var hidden in new[] { draft, embargoed }) - Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/items/{hidden.ItemId}/tell")).StatusCode); + await WithVisitorNotesOnAsync(async () => + { + foreach (var hidden in new[] { draft, embargoed }) + Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/items/{hidden.ItemId}/tell")).StatusCode); + Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/items/{shown.ItemId}/tell")).StatusCode); + }); } [Fact] @@ -181,6 +189,124 @@ await TextAsync(visitor, $"/search?q={word}"), Assert.DoesNotContain(title, text); } + [Fact] + public async Task A_visitor_cannot_send_a_note_about_a_hidden_item() + { + var shown = await SeedAsync("sendshown", (i, now) => i.Publish(now)); + var draft = await SeedAsync("senddraft", (_, _) => { }); + var embargoed = await SeedAsync("sendembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }); + using var visitor = app.CreateClient(); + + await WithVisitorNotesOnAsync(async () => + { + foreach (var hidden in new[] { draft, embargoed }) + { + var response = await Browser.SendFormAsync(visitor, $"/items/{hidden.ItemId}/tell", + new() { ["Input.Kind"] = "Information", ["Input.Message"] = $"Note {hidden.Word}" }, $"/items/{shown.ItemId}/tell"); + Assert.Equal(HttpStatusCode.NotFound, response.StatusCode); + } + }); + + Assert.False(await app.WithDbAsync(db => db.VisitorNotes.AnyAsync(n => n.ArchiveItemId == draft.ItemId || n.ArchiveItemId == embargoed.ItemId))); + } + + [Fact] + public async Task The_recordings_and_pdf_pages_of_hidden_and_record_only_items_are_not_served_to_visitors() + { + var storage = app.Services.GetRequiredService(); + async Task<(int Audio, int Pdf)> SeedFilesAsync(string kind, Action state) + { + var marked = await SeedAsync(kind, state); + return await app.WithDbAsync(async db => + { + var item = await db.ArchiveItems.IgnoreQueryFilters().SingleAsync(i => i.Id == marked.ItemId); + var audio = new MediaFile + { + ArchiveItemId = item.Id, OriginalFileName = "a.mp3", ContentType = "audio/mpeg", Sha256 = new string('b', 64), + StorageId = Guid.NewGuid(), Status = ProcessingStatus.Ready, UploadedAt = DateTime.UtcNow, + }; + var pdf = new MediaFile + { + ArchiveItemId = item.Id, OriginalFileName = "a.pdf", ContentType = "application/pdf", Sha256 = new string('c', 64), + StorageId = Guid.NewGuid(), Status = ProcessingStatus.Ready, PageCount = 1, UploadedAt = DateTime.UtcNow, + }; + db.AddRange(audio, pdf); + await db.SaveChangesAsync(); + await storage.SaveAsync(audio.AudioKey, new MemoryStream([1, 2, 3])); + await storage.SaveAsync(pdf.PageKey(1), new MemoryStream([1, 2, 3])); + return (audio.Id, pdf.Id); + }); + } + var shown = await SeedFilesAsync("filesshown", (i, now) => i.Publish(now)); + var hidden = new[] + { + await SeedFilesAsync("filesdraft", (_, _) => { }), + await SeedFilesAsync("filesbinned", (i, now) => { i.Publish(now); i.MoveToBin(now); }), + await SeedFilesAsync("filesembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }), + await SeedFilesAsync("filesrecord", (i, now) => { i.Publish(now); i.MediaLevel = MediaLevel.RecordOnly; }), + }; + using var visitor = app.CreateClient(); + + // The shown item proves the addresses work, so the refusals below are about visibility and not missing files. + Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/media/{shown.Audio}/audio")).StatusCode); + Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/media/{shown.Pdf}/page/1")).StatusCode); + foreach (var files in hidden) + { + Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/media/{files.Audio}/audio")).StatusCode); + Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/media/{files.Pdf}/page/1")).StatusCode); + } + } + + [Fact] + public async Task The_decade_timeline_does_not_count_a_hidden_item_with_a_date() + { + var word = TestData.UniqueWord("decades"); + await app.WithDbAsync(async db => + { + var now = DateTime.UtcNow; + var collection = new Collection { Title = $"Box {word}", CreatedAt = now }; + ArchiveItem Make(string date, Action state) + { + var item = new ArchiveItem { Title = $"Dated {word}", Collection = collection, CreatedAt = now, UpdatedAt = now }; + item.TrySetDate(date); + state(item); + return item; + } + db.AddRange(collection, + Make("1930", i => i.Publish(now)), + Make("1950", i => i.Publish(now)), + Make("1810", _ => { }), + Make("1820", i => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }), + Make("1830", i => { i.Publish(now); i.MoveToBin(now); })); + await db.SaveChangesAsync(); + return 0; + }); + using var visitor = app.CreateClient(); + + var text = await TextAsync(visitor, $"/search?q={word}"); + + Assert.Contains("1930", text); + foreach (var decade in new[] { "1810", "1820", "1830" }) + Assert.DoesNotContain(decade, text); + } + + private async Task WithVisitorNotesOnAsync(Func body) + { + async Task Set(bool on) => await app.WithDbAsync(async db => + { + var settings = await db.ArchiveSettings.FirstOrDefaultAsync(); + if (settings is null) + db.ArchiveSettings.Add(settings = new ArchiveSettings()); + settings.AcceptVisitorNotes = on; + var saved = await db.SaveChangesAsync(); + app.Services.GetRequiredService().Forget(); + return saved; + }); + await Set(true); + try { await body(); } + finally { await Set(false); } + } + private static async Task TextAsync(HttpClient client, string address) { var response = await client.GetAsync(address); From 342b22d9e4e7697853dca34307333e5bfdec2a81 Mon Sep 17 00:00:00 2001 From: "Seb H." <138793908+SebHenn@users.noreply.github.com> Date: Thu, 19 Nov 2026 19:38:42 +0100 Subject: [PATCH 3/3] M28: record the decisions and mark M27 done Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Seb H. <138793908+SebHenn@users.noreply.github.com> --- docs/plan-m28-toward-1-0.md | 8 ++++---- docs/roadmap.md | 5 +++-- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/plan-m28-toward-1-0.md b/docs/plan-m28-toward-1-0.md index 440b83e..835c92a 100644 --- a/docs/plan-m28-toward-1-0.md +++ b/docs/plan-m28-toward-1-0.md @@ -1,6 +1,6 @@ # Plan for M28: toward 1.0 -Status: **plan, no code yet.** Three parts, in this order, each its own pull request. The plan goes to the `privacy-reviewer` agent +Status: **plan, no code yet; decisions made (see "Decisions").** Three parts, in this order, each its own pull request. The plan goes to the `privacy-reviewer` agent before part 1 starts, as M27 did, because part 1 brings material into the archive from outside and part 2 adds public addresses. ## What is wanted @@ -92,7 +92,7 @@ checked by a machine: Nothing in this part changes behaviour, so it comes last and is the only part that needs the user and the group, not code. -## Decisions to make +## Decisions (taken as proposed) 1. **Duplicate with a different description:** skip and list (proposed), or offer to add the other text as a revision. 2. **Contributor details:** left out (proposed), or optional behind a checkbox saying the curator holds that consent. @@ -156,7 +156,7 @@ The review found no leak in the plan as such, but fourteen gaps. What they chang - Every part has its `CHANGELOG.md` entry and `operations.md` update, and `security.md` gets rows for the ZIP import, the public JSON and the consent decision. -## Decisions to make (updated) +## Decisions added by the review -Decision 1 to 4 above stand. The review adds one: **items whose source did not allow sharing are skipped (proposed)**, or a field +Decisions 1 to 3 stand as proposed. Decision 4 was open: a real group running it is a condition for calling 1.0 supported, and the checklist says so, but the version number does not wait for it. The review adds one: **items whose source did not allow sharing are skipped (proposed)**, or a field is added to remember it. diff --git a/docs/roadmap.md b/docs/roadmap.md index 3a5f3ec..7251a04 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -182,7 +182,7 @@ real pace over M7 and M8 before promising dates. Each milestone is a minor relea | **M24 Helpers and accounts** | 3 | A helper describes and uploads but cannot publish, delete or see contributors' email addresses. Accounts have display names and a last login. | | **M25 Oral histories** | 4 | A curator uploads an interview. Visitors play it with its transcript beside it, and the file carries no metadata. | | **M26 Description history** | 3 | A curator sees who changed which field and when, and restores an earlier description. | -| **M27 Visibility levels** | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. | +| **M27 Visibility levels** ✅ | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. | | **M28 Toward 1.0** | 3 | Commonage reads its own export, so archives can be merged. Dublin Core and a JSON alternate exist. A 1.0 checklist is met. | M22 to M28 were planned in November 2026, after M21, from the "Later" list. There are still no reports from groups or @@ -529,7 +529,7 @@ shows the years of its published items. M19 is complete. documents a local whisper.cpp workflow, as "Not planned" promises. **M26 Description history** | 3 | A curator sees who changed which field and when, and restores an earlier description. | -| **M27 Visibility levels** | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. | +| **M27 Visibility levels** ✅ | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. | | **M28 Toward 1.0** | 3 | Commonage reads its own export, so archives can be merged. Dublin Core and a JSON alternate exist. A 1.0 checklist is met. | M22 to M28 were planned in November 2026, after M21, from the "Later" list. There are still no reports from groups or @@ -898,6 +898,7 @@ shows the years of its published items. M19 is complete. **M28 Toward 1.0** +- The plan is in [plan-m28-toward-1-0.md](plan-m28-toward-1-0.md). - Reading Commonage's own export (versions 4 and later) with the import's dry run. Dublin Core and `/items/{id}.json` as an alternate link. A 1.0 checklist: a real group running it, the upgrade tested from every 0.x minor version, a support policy, and a second release approver or a written handover.