feat(deps): add Azure DevOps support to deps import - #52
Conversation
Vendoring could fetch only from github.com, because it delegated to
`gh`. Canonical domain models also live in private Azure DevOps Git
repositories, which is the live case ADR-0007's bar asked for before a
second transport.
`ParseSource` now reads a `dev.azure.com` blob URL alongside a github.com
one:
https://dev.azure.com/<org>/<project>/_git/<repo>?path=...&version=GB<branch>
The GB/GT/GC version prefixes map to the API's branch/tag/commit version
type, `--ref` overrides the URL's ref and lets the API auto-detect the
type, a URL with no version is refused before a header can be stamped,
and a legacy `*.visualstudio.com` URL is intercepted with a pointer to its
`dev.azure.com` address.
The fetch still delegates to an external CLI over an argv array with no
shell, so modelith holds no HTTP client, TLS configuration, or credential
handling (ADR-0011). Azure DevOps delegates to `az rest`, which resolves
its own token from the user's `az login` session; requests carry the
well-known Azure DevOps resource ID so `az` asks for the right audience,
and content is written with `--output-file` because `az rest` appends a
newline to a raw body on stdout, which would move the copy's digest off
canonical.
A delegated command that spawns helpers inheriting its pipes can hold
Wait() open past a deadline, so on Unix the process group is killed and
cmd.WaitDelay bounds the pipe drain; Windows, which has no POSIX process
groups, kills the direct child under the same WaitDelay bound. A --timeout
decorator gives each command its own deadline and reports a fired deadline
separately from a caller's cancel.
`deps check` and `deps update` still reach the origin through gh, so a copy
vendored from Azure DevOps is refused up front with an error naming the
host and the remedy rather than failing as a malformed URL.
Recorded as an amendment to ADR-0015, which had named `gh` as the only
transport.
Refs stacklok#44.
Signed-off-by: blevins darrin <darrinb765@gmail.com>
|
Thanks for the contribution—there is one functional blocker before this can merge.
I do not have a Windows machine available to validate the runtime process-handling behavior at the moment—I am traveling—so I can only confirm the Windows build cross-compiles. Could you please test the Windows |
|
One non-blocking suggestion for follow-up, or for this PR if you want to make Azure DevOps support first class: preserve the Azure DevOps ref type in the provenance header and implement refresh for that origin. GitHub refresh can record only I suggest an optional header field: # modelith-ref-type: branchUse The header already has enough location data for refresh: This is not a merge requirement for me. But if we are adding Azure DevOps support, I would like it to become a first-class origin with check and update support rather than an import-only path. |
|
I pushed the documentation follow-up on a branch based on this PR: It documents both import hosts, their CLI authentication, You can pull it in with: git fetch https://github.com/stacklok/modelith.git docs/ado-vendoring-followups
git cherry-pick 875ef26Alternatively, enable Allow edits and access to secrets by maintainers on this PR and I can push the commit directly to your branch. |
Signed-off-by: Joe Beda <joe@stacklok.com>
…class Addresses the review on the pull request, and closes the import-only scope the earlier revision recorded as follow-up. download=true: fetchContentADO built the Items API request without it, and the endpoint defaults it to false — so the body was a JSON GitItem describing the item, which the model parser then rejected. The fake runner now mirrors that contract (returning the metadata body unless download=true is present), so the omission is a test failure rather than a surprise against a real endpoint, and the endpoint test asserts the parameter. refresh: sourceFromHeader now dispatches on the origin host, rebuilding a dev.azure.com origin into a typed ADO address and a github.com one into a blob URL as before; fetchContent/fetchCommit became fetchContentFor/fetchCommitFor, which the refresh path shares with Import. A copy from a host with no transport is still refused per file, naming the origin. ref-type: the header gains an optional modelith-ref-type key. An ADO version has a type — GB a branch, GT a tag, GC a commit — and letting the API infer it is not equivalent when a branch and a tag share a name. Import records branch, tag, or commit, or auto for an unprefixed URL or a --ref override. The key is omitted for GitHub, so a GitHub header written earlier is byte-identical to one written now, and provenance validates the value against the closed set. Docs and the ADR-0015 amendment are updated accordingly, and fold in the maintainer's docs commit. Signed-off-by: blevins darrin <darrinb765@gmail.com>
…update Addresses the architecture review of 7f28d9c. Re-pin (blocker). adoSourceFromHeader applied the recorded ref-type prefix to the overridden ref, so re-pinning a branch-typed copy to a tag asked Azure DevOps for a branch named after the tag: "version=v1.0.0&versionType=branch" 404s, and the copy could not be re-pinned at all. The prefix is now dropped when --ref changes the ref, matching what an import override already does, and recordedRefType replaces the type in the header — a bare refresh keeps the recorded type, a re-pin records auto. Pinned by TestRefresh_ADORepinDropsTheRecordedType and TestRefresh_ADOBareRefreshKeepsTheRecordedType, the first of which fails on the old code with the header still claiming a branch. az authentication. unauthenticated() matched only gh's text, so an expired az session was not classified as ErrToolUnavailable and a batch repeated the same paragraph once per copy instead of stopping. It now also matches az login and AADSTS, and the sentinel reads "the delegated CLI is unavailable" rather than naming gh. --timeout on check and update. The bound existed only on import, so a stalled gh or az wedged a scheduled CI check indefinitely. CheckOptions and UpdateOptions carry a Timeout that decorates the runner in survey, and both commands expose --timeout; their help text no longer claims gh is the only delegated CLI. Signed-off-by: blevins darrin <darrinb765@gmail.com>
Two references still named gh as the only delegated CLI: Check's doc comment (saying the run stops when "gh was unusable") and the CLI reference, which documented --timeout for deps import but not for the deps check and deps update commands it now bounds. Signed-off-by: blevins darrin <darrinb765@gmail.com>
…dment ADR-0015 is upstream-owned and the repo rule is that decisions are never edited in place — a new ADR supersedes the old one. The previous revision appended a 101-line amendment to ADR-0015, which broke that rule and read as a fork edit to an upstream record. ADR-0015 is now byte-identical to upstream except a status note pointing at the superseding record, and the decision lives in its own ADR-0019. The note keeps what ADR-0015 decided at the time while telling a reader where the transport decision moved. Also adds TestADR_0019_ (pinning that the new ref-type key is recorded for ADO and omitted for GitHub), cites ADR-0019 from docs/10-vendoring, and repoints the deps.go comment. Signed-off-by: blevins darrin <darrinb765@gmail.com>
Verified against real endpoints (az 2.88.0): the Azure DevOps Git Items API does not infer a version's type. An untyped versionDescriptor.version is read as a *branch*, so a 40-hex commit fails with "The version descriptor <Branch: ...> could not be resolved" — reproduced on api-version 7.1, 7.1-preview.1, and 7.2-preview.1, and equivalently for a tag name. The refs filter shows the same default: filter=heads/main returns refs/heads/main, filter=main returns nothing. That makes the "auto" value a fiction. Nothing is inferred, so a copy pinned to a tag or a commit could not be fetched while its type was unset — which covered both override paths (--ref on import and on update) and any header written before the key existed. Dropping the type on a repin, the previous revision of this fix, only helped the tag -> branch direction; branch -> tag resolved to the same branch-typed lookup and still 404ed. So the type is now resolved before the fetch when it is not named: a git object id (40 or 64 hex) is a commit, and anything else is looked up under refs/heads/ and then refs/tags/. A branch wins a name collision, because that is the API's own default and a tag is reachable from the URL's GT form. The header's ref-type value set drops "auto": an unknown type is omitted, and a refresh resolves it, so an older header is self-healing rather than rejected. The az fake now models all of this — the refs endpoint, the versionType default to branch, and a 404 for an unresolvable typed version — because the previous fake let the untyped request pass, which is how the bug reached a real endpoint. Signed-off-by: blevins darrin <darrinb765@gmail.com>
…able A refs lookup goes through az like any fetch, so an absent or unauthenticated session there must end the run rather than be reported once per copy as if each file were separately at fault. The comment wording that still implied the API infers a type is corrected in the ADR, refresh.go, and a test. Signed-off-by: blevins darrin <darrinb765@gmail.com>
|
Thanks for the review — all three points are addressed, and one of them turned up something worth flagging. Head is now 1.
|
versionDescriptor.versionType |
result |
|---|---|
| omitted | FAIL TF401175: The version descriptor <Branch: 1b04973a…> could not be resolved |
| branch | FAIL |
| tag | FAIL |
| commit | OK |
Same repository, file, and 40-hex sha; identical on api-version 7.1, 7.1-preview.1, and 7.2-preview.1. So recording auto would have made any tag- or commit-pinned copy unfetchable — and in practice both override paths (deps import --ref, deps update --ref) failed with it.
Reworked (bfc6258): ref-type still records the three real types, but an unknown type is now resolved before the fetch — a 40/64-hex value is a commit; otherwise a refs lookup under refs/heads/<ref> then refs/tags/<ref>, a branch winning a collision to match the API's own default. auto is gone from the value set; a header that omits the key is resolved on the next refresh, so nothing needs migrating. Verified end to end: deps import --ref <sha> and deps update --ref <sha> now succeed where they returned TF401175 before, and the header records commit.
The fake now models the refs endpoint and the default-to-Branch as well — the same lenient-fake gap as the download=true case above.
3. Windows runtime check — done
You asked for the az timeout/cancellation path on Windows. I ran it on a Windows machine using a fake az that hangs, and one that leaves a child holding the inherited pipes:
| unix | Windows | |
|---|---|---|
plain hang, --timeout 5s |
5.3s | 9.4s |
| helper holding the pipes | 5.1s | 10.1s |
Both abandon at the bound with the friendly did not finish within … message and no bare signal: killed. The extra time on Windows is cmd.WaitDelay = 5s: with no POSIX process groups the helper survives the direct-child kill, so the bound is soft by up to WaitDelay. I'd treat that as an accepted cost of the platform rather than something to document on the flag — flagging it in case you disagree.
Also landed
--timeoutondeps checkanddeps update(previously import-only), so a stalled CLI can't wedge a scheduled check.unauthenticated()now recognisesaz loginandAADSTS, so an expiredazsession stops a batch instead of repeating the same error once per copy.- ADR: converted to a standalone
ADR-0019, per the repo rule that decisions aren't edited in place; ADR-0015 now differs frommainonly by a "partially superseded by ADR-0019" status note. That answers the ADR question I left on Azure DevOps support fordeps import#44.
Remaining offer
The exec_unix.go/exec_windows.go split means CI (ubuntu-only) can't catch a Windows-only break while goreleaser ships two Windows targets — a GOOS=windows go build ./... && go vet ./... step would. Happy to add it here or leave it to you. I also have a small script that drives the timeout check on both platforms if it would be useful.
|
Merging! Thank you! |
Adds Azure DevOps (
dev.azure.com) as a second, first-class origin formodelith deps. Upstream issue: #44.What it does
ParseSourcenow reads adev.azure.comblob URL alongsidegithub.com:GB/GT/GCversion prefixes map to the API'sbranch/tag/commitversion type. A bareversion=(no prefix) leaves the type to be resolved before the fetch.--refoverrides the URL's ref; the type is then resolved for the new ref rather than assuming the old one.--refis refused before a header can be stamped, so no copy records an empty ref.*.visualstudio.comURL is intercepted with a pointer to itsdev.azure.comaddress.deps checkanddeps updatedispatch on the origin's host, so an Azure DevOps copy is not just importable but refreshable — see Optional header field below.The ADR-0011 boundary is unchanged
The fetch still delegates to an external CLI over an argv array with no shell, so
modelithholds no HTTP client, TLS configuration, or credential handling. GitHub goes throughgh api; Azure DevOps goes throughaz rest, which resolves its own token from the user's existingaz loginsession.Two Azure-DevOps-specific facts are encoded, both about the
azCLI rather than configuration modelith invents:--resource 499b84ac-1321-427f-aa17-267ca6975798(the well-known Azure DevOps first-party app ID), becauseaz restcannot derive that audience from adev.azure.comURL.download=true. Without it the endpoint returns a JSONGitItemdescribing the item rather than its content, which the model parser then rejects. Content is then written with--output-filerather than stdout, becauseaz restappends a newline to a body on stdout — one byte that would move the vendored copy's digest off canonical.An earlier revision fetched through
curlwith a token read fromaz account get-access-token; it was reverted. Routing a credential through the binary's argv breaks the no-credential property ADR-0011 keeps, and nocurlinvocation restores it.Optional header field:
modelith-ref-typeAn Azure DevOps version has a type (
GBbranch,GTtag,GCcommit), and the API takes the type alongside the value. The API does not infer a missing type — an untypedversionDescriptor.versionis read as a branch, so a copy pinned to a tag or a commit cannot be fetched while the type is unset (verified against api-version 7.1, 7.1-preview.1, and 7.2-preview.1).So an Azure DevOps import records:
# modelith-ref-type: branchwith
branch,tag, orcommit. The key is omitted for GitHub, whose API resolves an untyped ref on its own, so a GitHub header written by an earlier release is byte-identical to one written now.When the type is not named — a bare
version=, a--refoverride, or a header written before the key existed — it is resolved before the fetch: a git object id (40 or 64 hex) is a commit, and anything else is looked up with the refs API underrefs/heads/<ref>and thenrefs/tags/<ref>, a branch winning a name collision to match the API's own default. A header that omits the key is resolved on the next refresh, so nothing needs migrating.Process lifecycle
A delegated command that spawns helpers inheriting its pipes can hold
Wait()open past a context deadline (observed with macOSazhelpers). The fix is split by platform:exec_unix.go(!windows):Setpgid+ a negative-PIDSIGKILLso the whole group dies, withcmd.WaitDelaybounding the pipe drain if a helper escapes.exec_windows.go(windows): no POSIX process groups, so the direct child is killed andWaitDelaystill bounds the wait.--timeoutdecorator gives each delegated command its own deadline ondeps import,deps check, anddeps update, and reports a fired deadline separately from a caller's cancel (e.g. Ctrl+C) rather than mislabelling both as a timeout. On Windows the bound is soft by up toWaitDelay(5s).A delegated CLI that is absent or unauthenticated stops the run rather than failing once per copy, for both
ghandaz.ADR
Recorded as a standalone ADR-0019, which supersedes ADR-0015's
ghis the only transport decision. ADR-0015 now differs frommainonly by a status note pointing at ADR-0019 — it is not edited in place, per.claude/rules/adr.md.Companion change worth considering
Because this introduces the
exec_unix.go/exec_windows.gosplit, it's worth aGOOS=windows go build ./...step (andgo vetalongside it) inci.yml. CI builds on ubuntu only today, while.goreleaser.yamlshipswindows/amd64andwindows/arm64, so a Windows-only compile break would otherwise surface at release time.Verification
task checkgreen locally;go build,go vet,staticcheck@v0.7.0,golangci-lintclean;GOOS=windows GOARCH=amd64|arm64build andgo vetpass.go test -race -count=1 ./...— 8/8 packages pass.az2.88.0): import,check,update, and a--refre-pin to a commit all succeed, and a re-import is byte-identical to the committed copy (matching digest).TestImport_ADO_*(end to end, the GB/GT/GC prefixes,--refresolution,download=trueon the items request),TestRefresh_ADOCopyIsFirstClass,TestRefresh_ADORepinRetypesTheRef,TestRefresh_ADORepinToACommitTypesItAsACommit,TestRefresh_ADOHeaderWithoutRefTypeResolvesIt,TestSurvey_ResolutionAbortsOnUnusableCLI, theTestTimeoutRunner_*set, andTestADR_0019_*.azfake mirrors the real API contract for each gap found: it returns the JSONGitItemunlessdownload=true, serves the refs endpoint, and defaults an untyped version to Branch rather than inferring one.The maintainer's
docs/ado-vendoring-followupscommit (875ef26) is folded in (589318a), with the refresh-limitation note corrected now that refresh is supported.