Flagsmith SDK behavioural spec

A language-agnostic, versioned specification of how Flagsmith SDKs behave, written in Gherkin and executed against every SDK.

• features/ — the spec. One .feature file per behaviour.
• runner/ — executes the spec (pytest-bdd). Step definitions live here, written once for every SDK.
• compose.yml — the Flagsmith API (pinned) the spec runs against.
• harness/ — one harness per language, running an SDK's steps file (HARNESS.md).
• .github/workflows/run-spec.yml — reusable workflow SDKs call.
• .github/workflows/compatibility.yml — runs the spec against every onboarded SDK on pull requests and nightly.

How it works

 ┌────────┐  Admin API: seed & inspect        ┌───────────┐
 │ runner │ ────────────────────────────────► │ Flagsmith │
 │        │                                   └───────────┘
 │        │  stdio  ┌──────────────────────────────┐  ▲
 │        │ ──────► │ harness ─► SDK steps file    │──┘ SDK API calls
 └────────┘         │ (SDK image, built from source)│
                    └──────────────────────────────┘

Each scenario gets its own Flagsmith project and environment, and the steps split three ways:

• Given steps set up Flagsmith through the Admin API. The runner implements them.
• When steps are the application using the SDK. Every SDK implements every When step in its steps file.
• Then steps check what the SDK returned and Flagsmith's resulting state. The runner implements them.

The feature files are therefore the whole contract: the When steps in features/ are the steps SDKs implement. The runner creates the scenario's SDK client just before its first When step.

Writing scenarios

• Describe behaviour, not wire details: assert on what the application receives and what Flagsmith ends up with.
• Table cells and value arguments are JSON literals: "text", 42, true, null.
• Tag every feature with a capability (@core, …). SDKs select capabilities with a tag expression. Register new tags under markers in runner/pyproject.toml.
• Prefer reusing existing steps in runner/tests/conftest.py.
• A new When step is registered with one sdk_step(...) line in the runner. It is a minor version: SDKs that have not implemented it go red with "The SDK does not implement the step: …". Rewording a When step is a major version.
• The compatibility matrix runs with missing-steps: xfail: scenarios using a step an SDK has not implemented yet are expected failures, reported as warnings, so a pull request adding a step stays green unless something else breaks. SDK repositories keep the default, fail.

Versioning

The spec is released as semver tags (vX.Y.Z). SDKs pin a tag, so spec changes never break an SDK until it opts in.

Each release also pins the Flagsmith version in compose.yml that it was verified against.

Onboarding an SDK

1. Add sdk-spec/steps.<ext> to the SDK repository: a function creating the SDK client, and one function per When step in features/. Mark each with a comment directly above it:

   # Create the SDK client
   def create_client(environment_key, api_url):
       return Flagsmith(environment_key=environment_key, api_url=api_url)
   
   
   # When the application gets the environment flags
   def get_environment_flags(flagsmith):
       return as_spec_flags(flagsmith.get_environment_flags())

   • Create the SDK client receives a server-side environment key and the Flagsmith API URL, and returns a client created with default options.
   • A When step receives the client, then the step's placeholders in order, then its data table. A <key> | value table arrives as a map, e.g. {"plan": "premium", "age": 30}.
   • Values are JSON values. Pass flag and trait values through with their types intact.
   • Steps that get flags return a list of {"feature": …, "enabled": …, "value": …}.
   • Let SDK errors propagate: the scenario fails with the error.

   Keep it to SDK calls. The steps file doubles as a working, CI-checked usage example.

2. Add sdk-spec/Dockerfile, which builds the SDK from source and runs the steps file under the language's harness. See harness/<language>/README.md. If the language has no harness yet, add one (see HARNESS.md).

3. Add a workflow:

   name: SDK spec
   on:
     pull_request:
     push:
       branches: [main]
   jobs:
     sdk-spec:
       uses: Flagsmith/sdk-spec/.github/workflows/run-spec.yml@v0.3.0
       with:
         spec-ref: v0.3.0  # keep in sync with `uses:`
         language: python
         tags: core        # optional: select capabilities

4. Let Renovate bump both refs together:

   {
     "customManagers": [
       {
         "customType": "regex",
         "managerFilePatterns": ["/^\\.github/workflows/sdk-spec\\.yml$/"],
         "matchStrings": ["spec-ref: (?<currentValue>v[\\d.]+)"],
         "depNameTemplate": "Flagsmith/sdk-spec",
         "datasourceTemplate": "github-tags"
       }
     ],
     "packageRules": [
       { "matchPackageNames": ["Flagsmith/sdk-spec"], "groupName": "SDK spec" }
     ]
   }

5. Add the SDK to the matrix in compatibility.yml.

Running locally

# 1. Start Flagsmith
docker compose up --detach --wait

# 2. Build the SDK image, in the SDK repository
docker build -t sdk-spec-sdk -f sdk-spec/Dockerfile --build-context harness=<path to sdk-spec>/harness/python .

# 3. Run the spec
cd runner
FLAGSMITH_SDK_URL=http://flagsmith:8000 \
SDK_COMMAND="docker run -i --rm --network sdk-spec_default sdk-spec-sdk" \
uv run pytest -v

Variable	Default	Purpose
SDK_COMMAND	(required)	Starts the SDK under its harness
MISSING_STEPS	fail	xfail makes missing steps expected failures
FLAGSMITH_URL	http://localhost:8000	Where the runner reaches Flagsmith
FLAGSMITH_SDK_URL	$FLAGSMITH_URL	Where the SDK reaches Flagsmith
FLAGSMITH_PORT	8000	Host port for Flagsmith (compose.yml)
FLAGSMITH_IMAGE	pinned version	Override the Flagsmith image (compose.yml)

SDK_COMMAND can be any command, so you can also run a harness straight against an SDK checkout, without Docker, e.g. python ../harness/python/harness.py ../../flagsmith-python-client/sdk-spec/steps.py.

Implementing a new spec version

A spec version is implemented in three phases:

1. Spec update. A pull request to this repository is merged, and the new spec version is tagged by a maintainer (see Versioning).
2. Spec implementation. Renovate proposes the new version to every SDK onboarded in the compatibility matrix, as a bump pull request in the SDK's repository. A bump whose CI is red is fixed by implementing the spec in it. Fixed bumps are reviewed by the SDK's code owners and merged.
3. SDK releases. Maintainers release each SDK version that includes its merged bump.

A spec version is implemented once every onboarded SDK has released the new spec version.