Rotate the audit log by size (#161)

CHANGELOG.md

@@ -7,6 +7,22 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ## [Unreleased]
 
+### Changed
+
+- **The audit log is rotated by size (#161); a change in default behaviour.** Before a line would
+  take `audit.log` past `audit.maxBytes` (default `10MB`), it moves to `audit.log.1` and the older
+  ones up by one; past `audit.keep` (default 10) the oldest is deleted, so the log takes at most
+  about 110 MB. `audit.keep=0` deletes nothing, `audit.maxBytes=0` never rotates (the old
+  behaviour). An existing `audit.log` larger than the limit is rotated on the first line after the
+  upgrade; nothing is deleted then. The Audit view reads the files newest first, from their ends,
+  and stops as soon as it has its lines, instead of reading the whole log for every request.
+
+### Fixed
+
+- The Audit view's *since* compares times as times. Times are written with 3, 6 or 9 fraction
+  digits, and compared as text a later line could sort before an earlier one and end the list too
+  soon. A partly typed time (`2026-10-07`, `2026-10-07T09:30`) is the start of that period, in UTC.
+
 ### Removed
 
 - docs/hosted.md, the hosted-mode design note of 2026-10-05: its body described the design before it

bin/smoke-hosted.sh

@@ -41,6 +41,8 @@ setup.roots=$WORK
 agents=ci:idm-deployers;reader:
 agent.ci.token=ci-agent-token-0123456789
 agent.reader.token=reader-agent-token-0123456789
+audit.maxBytes=2KB
+audit.keep=0
 ENV
 IDMWEB_AGENT_KIT="$HERE/hosted/agent-kit" IDM_ENVIRONMENTS="$WORK/data/environments.properties" "$HERE/bin/idmweb" serve --hosted "$WORK/hosted.properties" > "$WORK/log" 2>&1 &
 PID=$!
@@ -389,8 +391,19 @@ try:
     s, r, _ = bob.call("POST", "/api/setup/directory", {"project": "outside", "path": "/etc"})
     check("a deployer may attach only under setup.roots (#41)", s == 400 and "outside" in r.get("error", ""))
     # #42: the audit log, deployers only
+    import datetime
+    def when(t):   # 3, 6 or 9 fraction digits: compare as times, not text (#161)
+        return datetime.datetime.fromisoformat(t.replace("Z", "+00:00"))
     s, a, _ = bob.call("GET", "/api/audit?limit=50")
-    check("a deployer reads the audit log (newest first, sign-ins among them)", s == 200 and len(a["entries"]) > 0 and any(e["event"] == "sign-in" for e in a["entries"]) and a["entries"][0]["time"] >= a["entries"][-1]["time"])
+    check("a deployer reads the audit log (newest first, sign-ins among them)", s == 200 and len(a["entries"]) > 0 and any(e["event"] == "sign-in" for e in a["entries"]) and when(a["entries"][0]["time"]) >= when(a["entries"][-1]["time"]))
+    # #161: rotated by size (2KB here, all kept); the view reads across the files, newest first
+    logs = sorted(f for f in os.listdir(os.path.join(work, "data")) if f.startswith("audit.log"))
+    current = len(open(os.path.join(work, "data", "audit.log")).read().splitlines())
+    whole = all(l.startswith("{") and l.endswith("}") for f in logs for l in open(os.path.join(work, "data", f)).read().splitlines())
+    check("the audit log rotates at audit.maxBytes, every line whole", "audit.log.1" in logs and whole
+          and all(os.path.getsize(os.path.join(work, "data", f)) <= 2 * 1024 for f in logs))
+    s, a, _ = bob.call("GET", "/api/audit?limit=2000")
+    check("the Audit view reads past the current file into the rotated ones", s == 200 and len(a["entries"]) > current)
     s, a, _ = bob.call("GET", "/api/audit?event=sign-in&user=alice")
     check("filtered by event and user", s == 200 and a["entries"] and all(e["event"] == "sign-in" and e["user"] == "alice" for e in a["entries"]))
     s, a, _ = bob.call("GET", "/api/audit?event=git-token")
@@ -418,8 +431,12 @@ finally:
     check("sessions.json is the owner's only", oct(os.stat(os.path.join(work, "data2", "sessions.json")).st_mode)[-3:] == "600")
     p2.terminate()
 
-with open(os.path.join(work, "data", "audit.log")) as f:
-    lines = [json.loads(l) for l in f if l.strip()]
+# the whole log, oldest first: audit.log.N … audit.log.1, then audit.log (rotated, #161)
+rotated = sorted((int(f.rsplit(".", 1)[1]) for f in os.listdir(os.path.join(work, "data")) if f.startswith("audit.log.")), reverse=True)
+lines = []
+for f in ["audit.log.%d" % n for n in rotated] + ["audit.log"]:
+    with open(os.path.join(work, "data", f)) as fh:
+        lines += [json.loads(l) for l in fh if l.strip()]
 events = [(l["user"], l["event"], l["outcome"]) for l in lines]
 check("the audit log has sign-ins, the workspace, the edit, the commit and the sign-out, per user",
       ("alice", "sign-in", "ok") in events and ("alice", "workspace.open", "ok") in events and ("alice", "op/gcv.set", "ok") in events and ("alice", "commit+push", "ok") in events and ("bob", "workspace.open", "ok") in events and ("alice", "sign-out", "ok") in events and ("agent:ci", "workspace.open", "ok") in events)

docs/hosted-install.md

@@ -152,7 +152,9 @@ ssh debian@192.168.103.99 'cd ~/idmweb && sed -i "s#^IDMWEB_IMAGE=.*#IDMWEB_IMAG
 - The `idmweb-data` volume holds the workspaces, `audit.log`, `sessions.json`, each person's
   files (`users/<subject>/`: vault credentials, Git tokens, traces), clone tokens (`secrets/`) and
   archived projects (`archive/`); the configuration is in `./config`. Back up both.
-- `audit.log` is one JSON line per event; ship it where the customer keeps logs.
+- `audit.log` is one JSON line per event, rotated by size to `audit.log.1` … (`audit.maxBytes`,
+  `audit.keep`: [hosted-mode.md](hosted-mode.md#the-audit-log)). Ship the numbered files where the
+  customer keeps logs; set `audit.keep=0` if nothing may be deleted.
 - Sessions are kept in `<data.dir>/sessions.json` (owner-only): a restart signs nobody out, and
   each person gets their last project and branch back on their next session.
 - `bin/smoke-container.sh` runs the image end to end on a workstation with Docker.

docs/hosted-mode.md

@@ -19,7 +19,7 @@ deploys come from the base branch only, as the repository holds it. The server h
 definitions (`environments.file`, no secrets) with the driver secrets beside them, and each
 person's bind DN and password in their own file under `data.dir/users/<subject>/`
 (docs/multi-user.md); it writes an audit line per sign-in, write, deploy, driver action, trace
-change, commit and push (`data.dir/audit.log`, not rotated yet: issue #161). Roles come from groups: *readers* look; *editors* read, edit and commit;
+change, commit and push (`data.dir/audit.log`, rotated by size: [the audit log](#the-audit-log)). Roles come from groups: *readers* look; *editors* read, edit and commit;
 *deployers* also deploy, control drivers, set trace levels and read the engine's trace file, and
 change the shared environments and driver secrets. Watching a live trace is open to anyone
 signed in.
@@ -51,8 +51,18 @@ roles.deployer=idm-deployers
 session.hours=8                         # 0: never (the one-person install); sessions survive a restart
 release.check=true                      # once a day: is a newer release out? (the top bar says so; false: never ask)
 setup.roots=/srv,/var/lib/idmweb      # where path-based setup may point (default: data.dir and /srv)
+audit.maxBytes=10MB                     # audit.log moves to audit.log.1 before a line would take it past this (KB, MB, GB; 0: never)
+audit.keep=10                           # rotated files kept (audit.log.1 … .10), the oldest deleted (0: keep them all)
 ```
 
+**The audit log.** One JSON line per sign-in, write, deploy, driver action, trace change, commit
+and push, in `data.dir/audit.log`. Before a line would take it past `audit.maxBytes`, the file
+moves to `audit.log.1` and the older ones up by one (`.1` to `.2` …); the ones past `audit.keep`
+are deleted. With the defaults that is at most about 110 MB: the current file and ten 10 MB
+ones. A line is never split between files. The Audit view (deployers) reads them newest first, and
+stops as soon as it has the lines it shows. To keep everything, set `audit.keep=0` and ship the
+rotated files to where your logs are kept; each one is complete once it has a number.
+
 **Roles and the host.** An *editor* reads, edits, commits and pushes, and adds a project by
 uploading a Designer project or export, exporting from a vault, or cloning a remote. Attaching a
 path of the host — a checkout in place, a bare repository, a directory of files — typing a

docs/security.md

@@ -14,7 +14,7 @@ configured for one person and one for a team.
 | Roles | none: the one person may do everything | reader, editor, deployer (`roles.*`, docs/hosted-mode.md) |
 | Vault credentials | the `idm` CLI's environments file (`IDM_ENVIRONMENTS`, `./environments.properties`, `~/.idm/environments.properties`), secrets included | the project's definitions in `environments.file` (no secrets; the server will not start with any), each person's bind DN and password in their own `data.dir/users/<subject>/environments.properties` |
 | Where a typed password is kept | the macOS Keychain, the owner-only file, or a variable, command or secret-manager reference | the person's own owner-only file; a deployer may also keep a reference (below). The Keychain is macOS only, so never in the container |
-| Audit log | none | `data.dir/audit.log`, readable by deployers |
+| Audit log | none | `data.dir/audit.log` (owner-only), readable by deployers; rotated by size to `audit.log.1` …, the oldest past `audit.keep` deleted (`audit.keep=0` keeps all) |
 | Typed secrets cross | the loopback connection | the connection to the server: TLS only when `public.url` is https (a team server behind its proxy; the local container stays on the host's loopback) |
 
 The rest holds in both modes unless it names one.