Skip to content

feat(deps): add Azure DevOps support to deps import - #52

Merged
jbeda merged 8 commits into
stacklok:mainfrom
gosharplite:feat/ado-deps-import-upstream
Sep 27, 2026
Merged

jbeda merged 8 commits into
stacklok:mainfrom
gosharplite:feat/ado-deps-import-upstream

Conversation

@gosharplite

@gosharplite gosharplite commented Sep 26, 2026 •

Copy link
Copy Markdown

Adds Azure DevOps (dev.azure.com) as a second, first-class origin for modelith deps. Upstream issue: #44.

What it does

ParseSource now reads a dev.azure.com blob URL alongside github.com:

https://dev.azure.com/<org>/<project>/_git/<repo>?path=docs/payments.modelith.yaml&version=GB<branch>
  • The GB/GT/GC version prefixes map to the API's branch/tag/commit version type. A bare version= (no prefix) leaves the type to be resolved before the fetch.
  • --ref overrides the URL's ref; the type is then resolved for the new ref rather than assuming the old one.
  • A URL with no version and no --ref is refused before a header can be stamped, so no copy records an empty ref.
  • A legacy *.visualstudio.com URL is intercepted with a pointer to its dev.azure.com address.

deps check and deps update dispatch on the origin's host, so an Azure DevOps copy is not just importable but refreshable — see Optional header field below.

The ADR-0011 boundary is unchanged

The fetch still delegates to an external CLI over an argv array with no shell, so modelith holds no HTTP client, TLS configuration, or credential handling. GitHub goes through gh api; Azure DevOps goes through az rest, which resolves its own token from the user's existing az login session.

Two Azure-DevOps-specific facts are encoded, both about the az CLI rather than configuration modelith invents:

  • Requests pass --resource 499b84ac-1321-427f-aa17-267ca6975798 (the well-known Azure DevOps first-party app ID), because az rest cannot derive that audience from a dev.azure.com URL.
  • The Items API is asked for the file's bytes with download=true. Without it the endpoint returns a JSON GitItem describing the item rather than its content, which the model parser then rejects. Content is then written with --output-file rather than stdout, because az rest appends a newline to a body on stdout — one byte that would move the vendored copy's digest off canonical.

An earlier revision fetched through curl with a token read from az account get-access-token; it was reverted. Routing a credential through the binary's argv breaks the no-credential property ADR-0011 keeps, and no curl invocation restores it.

Optional header field: modelith-ref-type

An Azure DevOps version has a type (GB branch, GT tag, GC commit), and the API takes the type alongside the value. The API does not infer a missing type — an untyped versionDescriptor.version is read as a branch, so a copy pinned to a tag or a commit cannot be fetched while the type is unset (verified against api-version 7.1, 7.1-preview.1, and 7.2-preview.1).

So an Azure DevOps import records:

# modelith-ref-type: branch

with branch, tag, or commit. The key is omitted for GitHub, whose API resolves an untyped ref on its own, so a GitHub header written by an earlier release is byte-identical to one written now.

When the type is not named — a bare version=, a --ref override, or a header written before the key existed — it is resolved before the fetch: a git object id (40 or 64 hex) is a commit, and anything else is looked up with the refs API under refs/heads/<ref> and then refs/tags/<ref>, a branch winning a name collision to match the API's own default. A header that omits the key is resolved on the next refresh, so nothing needs migrating.

Process lifecycle

A delegated command that spawns helpers inheriting its pipes can hold Wait() open past a context deadline (observed with macOS az helpers). The fix is split by platform:

  • exec_unix.go (!windows): Setpgid + a negative-PID SIGKILL so the whole group dies, with cmd.WaitDelay bounding the pipe drain if a helper escapes.
  • exec_windows.go (windows): no POSIX process groups, so the direct child is killed and WaitDelay still bounds the wait.
  • A --timeout decorator gives each delegated command its own deadline on deps import, deps check, and deps update, and reports a fired deadline separately from a caller's cancel (e.g. Ctrl+C) rather than mislabelling both as a timeout. On Windows the bound is soft by up to WaitDelay (5s).

A delegated CLI that is absent or unauthenticated stops the run rather than failing once per copy, for both gh and az.

ADR

Recorded as a standalone ADR-0019, which supersedes ADR-0015's gh is the only transport decision. ADR-0015 now differs from main only by a status note pointing at ADR-0019 — it is not edited in place, per .claude/rules/adr.md.

Companion change worth considering

Because this introduces the exec_unix.go/exec_windows.go split, it's worth a GOOS=windows go build ./... step (and go vet alongside it) in ci.yml. CI builds on ubuntu only today, while .goreleaser.yaml ships windows/amd64 and windows/arm64, so a Windows-only compile break would otherwise surface at release time.

Verification

  • task check green locally; go build, go vet, staticcheck@v0.7.0, golangci-lint clean; GOOS=windows GOARCH=amd64|arm64 build and go vet pass.
  • go test -race -count=1 ./... — 8/8 packages pass.
  • Exercised against real Azure DevOps endpoints (az 2.88.0): import, check, update, and a --ref re-pin to a commit all succeed, and a re-import is byte-identical to the committed copy (matching digest).
  • New/updated tests: TestImport_ADO_* (end to end, the GB/GT/GC prefixes, --ref resolution, download=true on the items request), TestRefresh_ADOCopyIsFirstClass, TestRefresh_ADORepinRetypesTheRef, TestRefresh_ADORepinToACommitTypesItAsACommit, TestRefresh_ADOHeaderWithoutRefTypeResolvesIt, TestSurvey_ResolutionAbortsOnUnusableCLI, the TestTimeoutRunner_* set, and TestADR_0019_*.
  • The az fake mirrors the real API contract for each gap found: it returns the JSON GitItem unless download=true, serves the refs endpoint, and defaults an untyped version to Branch rather than inferring one.

The maintainer's docs/ado-vendoring-followups commit (875ef26) is folded in (589318a), with the refresh-limitation note corrected now that refresh is supported.

Vendoring could fetch only from github.com, because it delegated to
`gh`. Canonical domain models also live in private Azure DevOps Git
repositories, which is the live case ADR-0007's bar asked for before a
second transport.

`ParseSource` now reads a `dev.azure.com` blob URL alongside a github.com
one:

    https://dev.azure.com/<org>/<project>/_git/<repo>?path=...&version=GB<branch>

The GB/GT/GC version prefixes map to the API's branch/tag/commit version
type, `--ref` overrides the URL's ref and lets the API auto-detect the
type, a URL with no version is refused before a header can be stamped,
and a legacy `*.visualstudio.com` URL is intercepted with a pointer to its
`dev.azure.com` address.

The fetch still delegates to an external CLI over an argv array with no
shell, so modelith holds no HTTP client, TLS configuration, or credential
handling (ADR-0011). Azure DevOps delegates to `az rest`, which resolves
its own token from the user's `az login` session; requests carry the
well-known Azure DevOps resource ID so `az` asks for the right audience,
and content is written with `--output-file` because `az rest` appends a
newline to a raw body on stdout, which would move the copy's digest off
canonical.

A delegated command that spawns helpers inheriting its pipes can hold
Wait() open past a deadline, so on Unix the process group is killed and
cmd.WaitDelay bounds the pipe drain; Windows, which has no POSIX process
groups, kills the direct child under the same WaitDelay bound. A --timeout
decorator gives each command its own deadline and reports a fired deadline
separately from a caller's cancel.

`deps check` and `deps update` still reach the origin through gh, so a copy
vendored from Azure DevOps is refused up front with an error naming the
host and the remedy rather than failing as a malformed URL.

Recorded as an amendment to ADR-0015, which had named `gh` as the only
transport.

Refs stacklok#44.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
@gosharplite
gosharplite requested a review from jbeda as a code owner September 26, 2026 12:18
@jbeda

jbeda commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution—there is one functional blocker before this can merge.

fetchContentADO builds the Git Items API URI without download=true (internal/deps/deps.go:598-605). Azure DevOps defaults that parameter to false, so the endpoint returns a GitItem JSON metadata response rather than the raw model bytes. az rest --output-file then faithfully writes that JSON, and modelith attempts to parse it as YAML. Please add download=true to the /items request and extend TestImport_ADO_CallsAzWithTheExpectedEndpoints to assert it.

I do not have a Windows machine available to validate the runtime process-handling behavior at the moment—I am traveling—so I can only confirm the Windows build cross-compiles. Could you please test the Windows az timeout/cancellation path, if you have access to one?

@jbeda

jbeda commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

One non-blocking suggestion for follow-up, or for this PR if you want to make Azure DevOps support first class: preserve the Azure DevOps ref type in the provenance header and implement refresh for that origin.

GitHub refresh can record only modelith-ref because its APIs resolve an untyped branch, tag, or commit value. Azure DevOps takes both versionDescriptor.version and versionDescriptor.versionType; the current importer correctly derives branch, tag, or commit from GB, GT, and GC, but the provenance header records only the ref. A later refresh would have to auto-detect it, which is not equivalent when a branch and tag share a name.

I suggest an optional header field:

# modelith-ref-type: branch

Use branch, tag, commit, or auto. Preserve it for ordinary Azure DevOps imports and refreshes. Record auto for an unprefixed version= URL or an explicit --ref override, matching the current choice to let the Azure DevOps API infer the type. Omit the field for GitHub so existing headers retain their shape.

The header already has enough location data for refresh: origin contains the Azure DevOps organization, project, and repository, while path and ref identify the item and version. sourceFromHeader can dispatch by origin host and build a typed Azure DevOps Source directly, then deps check/deps update can dispatch to the existing ADO content and commit fetchers instead of rejecting the host.

This is not a merge requirement for me. But if we are adding Azure DevOps support, I would like it to become a first-class origin with check and update support rather than an import-only path.

@jbeda

jbeda commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

I pushed the documentation follow-up on a branch based on this PR: stacklok/modelith:docs/ado-vendoring-followups at 875ef26.

It documents both import hosts, their CLI authentication, --timeout, and the current Azure DevOps refresh limitation. It assumes the download=true fix is included.

You can pull it in with:

git fetch https://github.com/stacklok/modelith.git docs/ado-vendoring-followups
git cherry-pick 875ef26

Alternatively, enable Allow edits and access to secrets by maintainers on this PR and I can push the commit directly to your branch.

jbeda and others added 7 commits September 27, 2026 05:28
Signed-off-by: Joe Beda <joe@stacklok.com>
…class

Addresses the review on the pull request, and closes the import-only scope
the earlier revision recorded as follow-up.

download=true: fetchContentADO built the Items API request without it, and the
endpoint defaults it to false — so the body was a JSON GitItem describing the
item, which the model parser then rejected. The fake runner now mirrors that
contract (returning the metadata body unless download=true is present), so the
omission is a test failure rather than a surprise against a real endpoint, and
the endpoint test asserts the parameter.

refresh: sourceFromHeader now dispatches on the origin host, rebuilding a
dev.azure.com origin into a typed ADO address and a github.com one into a blob
URL as before; fetchContent/fetchCommit became fetchContentFor/fetchCommitFor,
which the refresh path shares with Import. A copy from a host with no transport
is still refused per file, naming the origin.

ref-type: the header gains an optional modelith-ref-type key. An ADO version has
a type — GB a branch, GT a tag, GC a commit — and letting the API infer it is
not equivalent when a branch and a tag share a name. Import records branch, tag,
or commit, or auto for an unprefixed URL or a --ref override. The key is omitted
for GitHub, so a GitHub header written earlier is byte-identical to one written
now, and provenance validates the value against the closed set.

Docs and the ADR-0015 amendment are updated accordingly, and fold in the
maintainer's docs commit.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
…update

Addresses the architecture review of 7f28d9c.

Re-pin (blocker). adoSourceFromHeader applied the recorded ref-type prefix to
the overridden ref, so re-pinning a branch-typed copy to a tag asked Azure
DevOps for a branch named after the tag: "version=v1.0.0&versionType=branch"
404s, and the copy could not be re-pinned at all. The prefix is now dropped when
--ref changes the ref, matching what an import override already does, and
recordedRefType replaces the type in the header — a bare refresh keeps the
recorded type, a re-pin records auto. Pinned by
TestRefresh_ADORepinDropsTheRecordedType and
TestRefresh_ADOBareRefreshKeepsTheRecordedType, the first of which fails on the
old code with the header still claiming a branch.

az authentication. unauthenticated() matched only gh's text, so an expired az
session was not classified as ErrToolUnavailable and a batch repeated the same
paragraph once per copy instead of stopping. It now also matches az login and
AADSTS, and the sentinel reads "the delegated CLI is unavailable" rather than
naming gh.

--timeout on check and update. The bound existed only on import, so a stalled
gh or az wedged a scheduled CI check indefinitely. CheckOptions and
UpdateOptions carry a Timeout that decorates the runner in survey, and both
commands expose --timeout; their help text no longer claims gh is the only
delegated CLI.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
Two references still named gh as the only delegated CLI: Check's doc comment
(saying the run stops when "gh was unusable") and the CLI reference, which
documented --timeout for deps import but not for the deps check and deps update
commands it now bounds.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
…dment

ADR-0015 is upstream-owned and the repo rule is that decisions are never edited
in place — a new ADR supersedes the old one. The previous revision appended a
101-line amendment to ADR-0015, which broke that rule and read as a fork edit
to an upstream record.

ADR-0015 is now byte-identical to upstream except a status note pointing at the
superseding record, and the decision lives in its own ADR-0019. The note keeps
what ADR-0015 decided at the time while telling a reader where the transport
decision moved. Also adds TestADR_0019_ (pinning that the new ref-type key is
recorded for ADO and omitted for GitHub), cites ADR-0019 from docs/10-vendoring,
and repoints the deps.go comment.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
Verified against real endpoints (az 2.88.0): the Azure DevOps Git Items API does
not infer a version's type. An untyped versionDescriptor.version is read as a
*branch*, so a 40-hex commit fails with "The version descriptor <Branch: ...>
could not be resolved" — reproduced on api-version 7.1, 7.1-preview.1, and
7.2-preview.1, and equivalently for a tag name. The refs filter shows the same
default: filter=heads/main returns refs/heads/main, filter=main returns nothing.

That makes the "auto" value a fiction. Nothing is inferred, so a copy pinned to a
tag or a commit could not be fetched while its type was unset — which covered
both override paths (--ref on import and on update) and any header written
before the key existed. Dropping the type on a repin, the previous revision of
this fix, only helped the tag -> branch direction; branch -> tag resolved to the
same branch-typed lookup and still 404ed.

So the type is now resolved before the fetch when it is not named: a git object
id (40 or 64 hex) is a commit, and anything else is looked up under refs/heads/
and then refs/tags/. A branch wins a name collision, because that is the API's
own default and a tag is reachable from the URL's GT form. The header's
ref-type value set drops "auto": an unknown type is omitted, and a refresh
resolves it, so an older header is self-healing rather than rejected.

The az fake now models all of this — the refs endpoint, the versionType default
to branch, and a 404 for an unresolvable typed version — because the previous
fake let the untyped request pass, which is how the bug reached a real endpoint.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
…able

A refs lookup goes through az like any fetch, so an absent or unauthenticated
session there must end the run rather than be reported once per copy as if each
file were separately at fault. The comment wording that still implied the API
infers a type is corrected in the ADR, refresh.go, and a test.

Signed-off-by: blevins darrin <darrinb765@gmail.com>
@gosharplite

gosharplite commented Sep 26, 2026 •

Copy link
Copy Markdown
Author

Thanks for the review — all three points are addressed, and one of them turned up something worth flagging. Head is now 52a5544.

1. download=true — fixed

You were right, and it was worse than a dropped byte: without it the endpoint returned a GitItem, so deps import couldn't parse anything. Added to the items request, and the test asserts it. I also fixed the reason it slipped through: the fake az now returns the metadata body unless download=true is present, so the omission fails in tests rather than against a real endpoint.

2. ref-type and first-class refresh — implemented, with one correction

Implemented as you specified: # modelith-ref-type: records branch/tag/commit, omitted for GitHub; sourceFromHeader dispatches on host and rebuilds a typed ADO address; deps check/deps update route through the ADO fetchers. Your 875ef26 is folded in (589318a), with the refresh-limitation note corrected since it no longer applies.

But the "let the API infer it" part doesn't hold. Exercising this against real endpoints (az 2.88.0), the Azure DevOps API does not infer a version's type — omitted is read as Branch:

versionDescriptor.versionType result
omitted FAIL TF401175: The version descriptor <Branch: 1b04973a…> could not be resolved
branch FAIL
tag FAIL
commit OK

Same repository, file, and 40-hex sha; identical on api-version 7.1, 7.1-preview.1, and 7.2-preview.1. So recording auto would have made any tag- or commit-pinned copy unfetchable — and in practice both override paths (deps import --ref, deps update --ref) failed with it.

Reworked (bfc6258): ref-type still records the three real types, but an unknown type is now resolved before the fetch — a 40/64-hex value is a commit; otherwise a refs lookup under refs/heads/<ref> then refs/tags/<ref>, a branch winning a collision to match the API's own default. auto is gone from the value set; a header that omits the key is resolved on the next refresh, so nothing needs migrating. Verified end to end: deps import --ref <sha> and deps update --ref <sha> now succeed where they returned TF401175 before, and the header records commit.

The fake now models the refs endpoint and the default-to-Branch as well — the same lenient-fake gap as the download=true case above.

3. Windows runtime check — done

You asked for the az timeout/cancellation path on Windows. I ran it on a Windows machine using a fake az that hangs, and one that leaves a child holding the inherited pipes:

unix Windows
plain hang, --timeout 5s 5.3s 9.4s
helper holding the pipes 5.1s 10.1s

Both abandon at the bound with the friendly did not finish within … message and no bare signal: killed. The extra time on Windows is cmd.WaitDelay = 5s: with no POSIX process groups the helper survives the direct-child kill, so the bound is soft by up to WaitDelay. I'd treat that as an accepted cost of the platform rather than something to document on the flag — flagging it in case you disagree.

Also landed

  • --timeout on deps check and deps update (previously import-only), so a stalled CLI can't wedge a scheduled check.
  • unauthenticated() now recognises az login and AADSTS, so an expired az session stops a batch instead of repeating the same error once per copy.
  • ADR: converted to a standalone ADR-0019, per the repo rule that decisions aren't edited in place; ADR-0015 now differs from main only by a "partially superseded by ADR-0019" status note. That answers the ADR question I left on Azure DevOps support for deps import #44.

Remaining offer

The exec_unix.go/exec_windows.go split means CI (ubuntu-only) can't catch a Windows-only break while goreleaser ships two Windows targets — a GOOS=windows go build ./... && go vet ./... step would. Happy to add it here or leave it to you. I also have a small script that drives the timeout check on both platforms if it would be useful.

@jbeda

jbeda commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Merging! Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants