diff --git a/docs/plan-m28-toward-1-0.md b/docs/plan-m28-toward-1-0.md
new file mode 100644
index 0000000..835c92a
--- /dev/null
+++ b/docs/plan-m28-toward-1-0.md
@@ -0,0 +1,162 @@
+# Plan for M28: toward 1.0
+
+Status: **plan, no code yet; decisions made (see "Decisions").** Three parts, in this order, each its own pull request. The plan goes to the `privacy-reviewer` agent
+before part 1 starts, as M27 did, because part 1 brings material into the archive from outside and part 2 adds public addresses.
+
+## What is wanted
+
+1. **Commonage reads its own export**, so two archives can be merged and a group can move into a fresh installation. It uses
+ the import's dry run and changes nothing until a curator confirms.
+2. **Dublin Core and a JSON alternate** for every public item, so other tools can read a record without scraping the page.
+3. **A 1.0 checklist** that is written down and checked, not remembered.
+
+Not planned: reading another system's export (Omeka, Mukurtu, ...); OAI-PMH (it stays in "Later"); restoring users, settings,
+the activity log or visitor notes from an export (a backup does that, see `operations.md`).
+
+## Part 1: reading Commonage's own export
+
+The input is the ZIP the export writes: `archive.json` with `"format": "commonage-export"` and the files beside it. Versions 4
+and later are read (version 4 is the first with tags). Anything else, or a file that is not a Commonage export, is refused with a
+plain message that names the version found and the versions understood.
+
+**What arrives, and as what**
+
+- Every item arrives as a **draft**, whatever it was in the source. Reading an export never makes anything public by itself
+ (rule 1). Its `mediaLevel` and `publicFrom` are kept, so a curator who held something back in one archive does not lose that
+ choice by moving it; it is shown in the dry run so it is not a surprise.
+- Collections are matched by title and otherwise created. Tags are matched by name and kind and otherwise created. The date is
+ taken as the text written (`date.text`) and read again by the same parser, so a date in a vocabulary this version lacks is
+ reported in the dry run, not guessed.
+- Files are taken from the ZIP by the path `archive.json` names, and are stored through `MediaUploadService`, the same path as an
+ upload, so size limits, decoding and derivatives are never skipped. The checksum in the export is checked against the file; a
+ mismatch is reported and the file is left out.
+- Alt text, caption, position, rotation, rights URI, credit, description, `needsTranscription` and `helpWanted` come with the item.
+ A table of every property of the export's item, file, collection, tag and settings records, each marked read, dropped or
+ dropped-because, is part of the code, and a reflection test fails when the exporter gains a field the table lacks.
+
+**What does not arrive, and why**
+
+- **Contributor details and consent text** (name, email, reference code, consent). Consent was given to the other archive for
+ that archive; carrying it over would claim a consent this archive never collected (rule 3). The item keeps its credit line and
+ rights statement as text. Left out of the dry run's result with that reason, and said in the handbook.
+- Visitor notes, activity log, users, settings, the texts of the pages, the bin, revisions, `internalNotes`, `receivedFrom` and
+ the other catalogue fields that name a donor. The `notes` and `contribution` parts of the JSON are never deserialised into
+ storable objects, never logged and never shown.
+- The status, the published date and the ids (the same reason as the CSV import: no id or status column on purpose).
+
+**Merging and repeating.** Duplicates are found as the CSV import finds them: by (collection, title, date) and by the content of a
+file, so a second run adds nothing. This needs one decision (see "Decisions"): what to do with an item that is a duplicate but
+whose description differs. Proposed: skip it and list it, never overwrite (rule 4's spirit: nothing is lost at once).
+
+**Safety.** The ZIP is from outside, so it is handled as a contribution is: a size and entry-count limit, a name inside the ZIP is
+only a key and never a path, a ZIP made to expand without end is stopped, and the JSON is read as a stream with a depth limit and
+a cap on the number of items per run. It waits in `import-staging/` and is removed after the run or a day. One import at a time,
+as now. It needs a curator, and is listed under `Administer` like the export. Every import is in the activity log with how many
+items and files came in, and names no contributor.
+
+**Tests.** Export, import into an empty database and export again: the two `archive.json` agree except for the fields listed above
+(a round trip test, which also catches a field added to the export and forgotten here). A version 4 file, a file with a wrong
+checksum, a ZIP with `../` names, a file with a future version, a second run, and a draft with `mediaLevel` and `publicFrom`.
+Against the export test that forces a decision for each new field: the importer gets the same table.
+
+## Part 2: Dublin Core and a JSON alternate
+
+- `/items/{id}.json` and, on the item page, `` and `` tags
+ (`DC.title`, `DC.description`, `DC.date`, `DC.creator` only when the credit is public as it is on the page, `DC.rights` from the
+ rights URI, `DC.language`, `DC.subject` from tags, `DC.identifier` the page address, `DC.type`).
+- The JSON has the same fields as the page shows, nothing more: no ids of private things, no contributor details, no original file
+ names, no personal data. Files are listed with their public `/media` addresses **only when the item's media are visible**; a
+ record-only item lists none.
+- Both start from `VisibleItems` and `VisibleMedia`. A draft, an embargoed or a binned item answers 404, exactly like its page, so
+ the address cannot be used to find out that something exists. The JSON address goes into `VisibilityMatrixTests.Addresses`
+ and `EndpointAccessTests` on purpose, and the guard test covers the new file.
+- Cached for a short time only (ten minutes at most, as the feed), so an item taken down does not stay in a cache for long. Said
+ in `security.md`.
+- Dates keep their meaning: the text as written plus the earliest and latest day, as the export does, because "about 1950"
+ must not become a day.
+
+## Part 3: the 1.0 checklist
+
+A page `docs/release-1.0.md` that lists each condition with its evidence, and a test or script where a condition can be
+checked by a machine:
+
+| Condition | How it is checked |
+|---|---|
+| A real group has run it for a season | Written note from the group (their name only if they agree); not something a test can show |
+| The upgrade works from every 0.x minor | `scripts/` upgrade test run against each earlier minor in CI, restoring a backup made by that version |
+| Export and import round trip | Part 1's test |
+| A support policy | `SUPPORT.md`: which versions get security fixes, for how long, how to report |
+| A second person can release | Written handover or second approver, as `handover.md` starts |
+| No open `LEAK` from the privacy review, no known accepted limit missing from `security.md` | The last review, dated |
+| Accessibility and translations | Browser tests green, no untranslated text in the resource test |
+
+Nothing in this part changes behaviour, so it comes last and is the only part that needs the user and the group, not code.
+
+## Decisions (taken as proposed)
+
+1. **Duplicate with a different description:** skip and list (proposed), or offer to add the other text as a revision.
+2. **Contributor details:** left out (proposed), or optional behind a checkbox saying the curator holds that consent.
+3. **`publicFrom` and `mediaLevel` kept on import** (proposed) or reset, so everything arrives plain.
+4. **Is "a real group running it" a condition for 1.0**, or only for calling it supported?
+
+## Changes after the review
+
+The review found no leak in the plan as such, but fourteen gaps. What they change:
+
+**Part 1**
+
+- **Sharing consent is not dropped silently.** An item without a contribution always travels (previews, feed, and in part 2 the
+ JSON). So an item whose source says `consentCoversSharing: false` is **not imported**: it is listed in the dry run with that
+ reason. Storing the restriction would need a new field and a migration; it waits until a group asks. Test: such an item is
+ absent after import.
+- **Status.** `Rejected` and `Withdrawn` items are skipped and listed, so a withdrawal is not undone. The importer has no
+ path that writes `Published` or `PublicFrom` beyond copying a validated future-or-past date; a source-file test checks it.
+- **Insert only.** Nothing existing is updated, removed or merged, no `Remove` or `ExecuteDelete`. An item that sits in the bin is
+ not seen by the duplicate check; this is stated in the handbook and tested.
+- **Unknown values are issues, not exceptions.** An unknown `status`, `mediaLevel` or tag `kind` is one line in the dry run for that
+ item. A `publicFrom` must be a real date.
+- **Keys.** Titles, tags and collections are compared in NFC with an ordinal, culture-free comparison, so a Mac-written (NFD)
+ title and a server in a Turkish culture do not make duplicates. Items with files are matched by the checksum first; items
+ without files can be duplicated by a changed title, and the handbook says so. `coverFileId`, collection position and sort are not
+ trusted. Collections created per run are capped, and creating one has its own `AuditAction`.
+- **Dates.** The exported `from` and `to` are used when present and sane, `text` stays as the display text, and a mismatch is
+ reported, not fatal. Tests with Chinese, right-to-left and Cyrillic titles and a date text the parser cannot read.
+- **The ZIP.** `archive.json` is read through a counting stream with an upper limit (256 MB), `MaxDepth` set, and caps on items,
+ files per item, tags per item, collections and the length of each field (the limits of `ItemImport`). A total uncompressed
+ budget covers all entries, not only each one. The dry run decodes outside the request or within a limit, so it cannot time out
+ and leave staging behind. Tests: a highly compressible `archive.json` and an entry whose `Length` lies.
+- **File paths** in `archive.json` must match `files//-...`, be unique per file, and never name `archive.json`, `items.csv`,
+ `branding/` or a README. No fallback to matching by bare name. `contentType` and `sizeBytes` are untrusted and the type is
+ detected again. Files the export itself marked `missing` or with a bad checksum are skipped and counted.
+- **Where.** The page is for admins like the export, listed in `Program.cs`, `EndpointAccessTests` and `AccessibilityTests`, and the
+ dry run answers `no-store` because it names other items' titles.
+- **Failing midway.** The audit entry is written first and updated at the end; a second run resumes because duplicates are
+ skipped. "One import at a time" is a lock released in `finally`.
+- **`ImportStaging.DeleteOld`** catches only `IOException` today. If it runs in a background loop, a permission error stops the app.
+ Fix and test that before part 1 adds more staging.
+- **Tests across versions.** A round trip with `includesPersonalData=true` leaves no contributor name, email or consent text in
+ the database, the activity log, the logs or the dry-run page. Exports of versions 4 to 8 and a published-only export.
+
+**Part 2**
+
+- **Same rules as the preview.** `DC.description`, the file list and the JSON description are left out when the item may not
+ travel; the JSON and DC follow the preview rules and not "what the page shows". With search engines off, the JSON answers
+ `X-Robots-Tag: noindex` and the DC tags are left out. The JSON address is not in the sitemap.
+- **A dedicated record with an allow-list**, never a serialised entity. `DC.creator` is exactly what the page prints. Never
+ `OriginalFileName`, `Sha256`, `SizeBytes`, keys, contribution, next or previous links or totals. A test serialises an item that
+ has every private field filled and asserts none appears.
+- **Hidden means the same answer.** Draft, embargoed, binned, rejected, withdrawn and unknown ids get one identical 404 (same body
+ and headers as the page's), never 401 or 403. The `` is rendered only inside the visible result. The 200
+ has `Cache-Control: public, max-age=600` and no `ETag` or `Last-Modified` from `UpdatedAt`, which would show edit times made
+ during an embargo; the 404 is `no-store`. Test: embargoed until tomorrow is 404 today and 200 tomorrow with the clock moved.
+
+**Part 3**
+
+- Where the dated review output is kept is named, with no contributor details or real archive material in it.
+- Every part has its `CHANGELOG.md` entry and `operations.md` update, and `security.md` gets rows for the ZIP import, the public JSON
+ and the consent decision.
+
+## Decisions added by the review
+
+Decisions 1 to 3 stand as proposed. Decision 4 was open: a real group running it is a condition for calling 1.0 supported, and the checklist says so, but the version number does not wait for it. The review adds one: **items whose source did not allow sharing are skipped (proposed)**, or a field
+is added to remember it.
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 3a5f3ec..7251a04 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -182,7 +182,7 @@ real pace over M7 and M8 before promising dates. Each milestone is a minor relea
| **M24 Helpers and accounts** | 3 | A helper describes and uploads but cannot publish, delete or see contributors' email addresses. Accounts have display names and a last login. |
| **M25 Oral histories** | 4 | A curator uploads an interview. Visitors play it with its transcript beside it, and the file carries no metadata. |
| **M26 Description history** | 3 | A curator sees who changed which field and when, and restores an earlier description. |
-| **M27 Visibility levels** | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. |
+| **M27 Visibility levels** ✅ | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. |
| **M28 Toward 1.0** | 3 | Commonage reads its own export, so archives can be merged. Dublin Core and a JSON alternate exist. A 1.0 checklist is met. |
M22 to M28 were planned in November 2026, after M21, from the "Later" list. There are still no reports from groups or
@@ -529,7 +529,7 @@ shows the years of its published items. M19 is complete.
documents a local whisper.cpp workflow, as "Not planned" promises.
**M26 Description history** | 3 | A curator sees who changed which field and when, and restores an earlier description. |
-| **M27 Visibility levels** | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. |
+| **M27 Visibility levels** ✅ | 5 | An item can be public as a record without its media, or embargoed until a date. Every public query goes through `VisibleItems`. |
| **M28 Toward 1.0** | 3 | Commonage reads its own export, so archives can be merged. Dublin Core and a JSON alternate exist. A 1.0 checklist is met. |
M22 to M28 were planned in November 2026, after M21, from the "Later" list. There are still no reports from groups or
@@ -898,6 +898,7 @@ shows the years of its published items. M19 is complete.
**M28 Toward 1.0**
+- The plan is in [plan-m28-toward-1-0.md](plan-m28-toward-1-0.md).
- Reading Commonage's own export (versions 4 and later) with the import's dry run. Dublin Core and `/items/{id}.json` as an
alternate link. A 1.0 checklist: a real group running it, the upgrade tested from every 0.x minor version, a support policy,
and a second release approver or a written handover.
diff --git a/tests/Commonage.Tests/VisibilityMatrixTests.cs b/tests/Commonage.Tests/VisibilityMatrixTests.cs
index 509ce8e..2f4f94c 100644
--- a/tests/Commonage.Tests/VisibilityMatrixTests.cs
+++ b/tests/Commonage.Tests/VisibilityMatrixTests.cs
@@ -2,6 +2,9 @@
using System.Text;
using Commonage.Web.Data;
using Commonage.Web.Domain;
+using Commonage.Web.Media;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.DependencyInjection;
namespace Commonage.Tests;
@@ -106,12 +109,17 @@ public async Task The_pictures_of_hidden_items_are_not_served_to_visitors()
[Fact]
public async Task A_visitor_cannot_open_the_note_form_of_a_hidden_item()
{
+ var shown = await SeedAsync("noteshown", (i, now) => i.Publish(now));
var draft = await SeedAsync("notedraft", (_, _) => { });
var embargoed = await SeedAsync("noteembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); });
using var visitor = app.CreateClient();
- foreach (var hidden in new[] { draft, embargoed })
- Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/items/{hidden.ItemId}/tell")).StatusCode);
+ await WithVisitorNotesOnAsync(async () =>
+ {
+ foreach (var hidden in new[] { draft, embargoed })
+ Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/items/{hidden.ItemId}/tell")).StatusCode);
+ Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/items/{shown.ItemId}/tell")).StatusCode);
+ });
}
[Fact]
@@ -181,6 +189,124 @@ await TextAsync(visitor, $"/search?q={word}"),
Assert.DoesNotContain(title, text);
}
+ [Fact]
+ public async Task A_visitor_cannot_send_a_note_about_a_hidden_item()
+ {
+ var shown = await SeedAsync("sendshown", (i, now) => i.Publish(now));
+ var draft = await SeedAsync("senddraft", (_, _) => { });
+ var embargoed = await SeedAsync("sendembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); });
+ using var visitor = app.CreateClient();
+
+ await WithVisitorNotesOnAsync(async () =>
+ {
+ foreach (var hidden in new[] { draft, embargoed })
+ {
+ var response = await Browser.SendFormAsync(visitor, $"/items/{hidden.ItemId}/tell",
+ new() { ["Input.Kind"] = "Information", ["Input.Message"] = $"Note {hidden.Word}" }, $"/items/{shown.ItemId}/tell");
+ Assert.Equal(HttpStatusCode.NotFound, response.StatusCode);
+ }
+ });
+
+ Assert.False(await app.WithDbAsync(db => db.VisitorNotes.AnyAsync(n => n.ArchiveItemId == draft.ItemId || n.ArchiveItemId == embargoed.ItemId)));
+ }
+
+ [Fact]
+ public async Task The_recordings_and_pdf_pages_of_hidden_and_record_only_items_are_not_served_to_visitors()
+ {
+ var storage = app.Services.GetRequiredService();
+ async Task<(int Audio, int Pdf)> SeedFilesAsync(string kind, Action state)
+ {
+ var marked = await SeedAsync(kind, state);
+ return await app.WithDbAsync(async db =>
+ {
+ var item = await db.ArchiveItems.IgnoreQueryFilters().SingleAsync(i => i.Id == marked.ItemId);
+ var audio = new MediaFile
+ {
+ ArchiveItemId = item.Id, OriginalFileName = "a.mp3", ContentType = "audio/mpeg", Sha256 = new string('b', 64),
+ StorageId = Guid.NewGuid(), Status = ProcessingStatus.Ready, UploadedAt = DateTime.UtcNow,
+ };
+ var pdf = new MediaFile
+ {
+ ArchiveItemId = item.Id, OriginalFileName = "a.pdf", ContentType = "application/pdf", Sha256 = new string('c', 64),
+ StorageId = Guid.NewGuid(), Status = ProcessingStatus.Ready, PageCount = 1, UploadedAt = DateTime.UtcNow,
+ };
+ db.AddRange(audio, pdf);
+ await db.SaveChangesAsync();
+ await storage.SaveAsync(audio.AudioKey, new MemoryStream([1, 2, 3]));
+ await storage.SaveAsync(pdf.PageKey(1), new MemoryStream([1, 2, 3]));
+ return (audio.Id, pdf.Id);
+ });
+ }
+ var shown = await SeedFilesAsync("filesshown", (i, now) => i.Publish(now));
+ var hidden = new[]
+ {
+ await SeedFilesAsync("filesdraft", (_, _) => { }),
+ await SeedFilesAsync("filesbinned", (i, now) => { i.Publish(now); i.MoveToBin(now); }),
+ await SeedFilesAsync("filesembargoed", (i, now) => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }),
+ await SeedFilesAsync("filesrecord", (i, now) => { i.Publish(now); i.MediaLevel = MediaLevel.RecordOnly; }),
+ };
+ using var visitor = app.CreateClient();
+
+ // The shown item proves the addresses work, so the refusals below are about visibility and not missing files.
+ Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/media/{shown.Audio}/audio")).StatusCode);
+ Assert.Equal(HttpStatusCode.OK, (await visitor.GetAsync($"/media/{shown.Pdf}/page/1")).StatusCode);
+ foreach (var files in hidden)
+ {
+ Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/media/{files.Audio}/audio")).StatusCode);
+ Assert.Equal(HttpStatusCode.NotFound, (await visitor.GetAsync($"/media/{files.Pdf}/page/1")).StatusCode);
+ }
+ }
+
+ [Fact]
+ public async Task The_decade_timeline_does_not_count_a_hidden_item_with_a_date()
+ {
+ var word = TestData.UniqueWord("decades");
+ await app.WithDbAsync(async db =>
+ {
+ var now = DateTime.UtcNow;
+ var collection = new Collection { Title = $"Box {word}", CreatedAt = now };
+ ArchiveItem Make(string date, Action state)
+ {
+ var item = new ArchiveItem { Title = $"Dated {word}", Collection = collection, CreatedAt = now, UpdatedAt = now };
+ item.TrySetDate(date);
+ state(item);
+ return item;
+ }
+ db.AddRange(collection,
+ Make("1930", i => i.Publish(now)),
+ Make("1950", i => i.Publish(now)),
+ Make("1810", _ => { }),
+ Make("1820", i => { i.Publish(now); i.SetPublicFrom(DateOnly.FromDateTime(now).AddDays(3), now); }),
+ Make("1830", i => { i.Publish(now); i.MoveToBin(now); }));
+ await db.SaveChangesAsync();
+ return 0;
+ });
+ using var visitor = app.CreateClient();
+
+ var text = await TextAsync(visitor, $"/search?q={word}");
+
+ Assert.Contains("1930", text);
+ foreach (var decade in new[] { "1810", "1820", "1830" })
+ Assert.DoesNotContain(decade, text);
+ }
+
+ private async Task WithVisitorNotesOnAsync(Func body)
+ {
+ async Task Set(bool on) => await app.WithDbAsync(async db =>
+ {
+ var settings = await db.ArchiveSettings.FirstOrDefaultAsync();
+ if (settings is null)
+ db.ArchiveSettings.Add(settings = new ArchiveSettings());
+ settings.AcceptVisitorNotes = on;
+ var saved = await db.SaveChangesAsync();
+ app.Services.GetRequiredService().Forget();
+ return saved;
+ });
+ await Set(true);
+ try { await body(); }
+ finally { await Set(false); }
+ }
+
private static async Task TextAsync(HttpClient client, string address)
{
var response = await client.GetAsync(address);