diff --git a/docs/docs/experimentation/index.md b/docs/docs/experimentation/index.md index 8ee24c4afb35..18109747c2b0 100644 --- a/docs/docs/experimentation/index.md +++ b/docs/docs/experimentation/index.md @@ -19,6 +19,13 @@ Experiments are scoped to an [**environment**](/flagsmith-concepts/data-model#en ## Set up and run an experiment +:::tip + +Using Claude Code or another coding agent? [One prompt](/experimentation/set-up-with-a-coding-agent) creates the flag, +metric and experiment, and instruments your application. + +::: + 1. **[Connect a warehouse](/experimentation/connect-a-warehouse)**: enable the managed Flagsmith warehouse, or connect your own ClickHouse or Databricks. Self-hosted Flagsmith uses your own. Done once per environment. 2. **[Create metrics](/experimentation/create-metrics)**: define the outcomes you measure, computed from the events your diff --git a/docs/docs/experimentation/set-up-with-a-coding-agent.mdx b/docs/docs/experimentation/set-up-with-a-coding-agent.mdx new file mode 100644 index 000000000000..750017a600a2 --- /dev/null +++ b/docs/docs/experimentation/set-up-with-a-coding-agent.mdx @@ -0,0 +1,132 @@ +--- +title: Set Up an Experiment with a Coding Agent +sidebar_label: Set Up with an Agent +sidebar_position: 1 +description: A copy-paste prompt that has a coding agent create an experiment and instrument your application to run it. +--- + +# Set Up an Experiment with a Coding Agent + +:::info Beta + +Experimentation is in beta. + +::: + +Run the prompt below from your application's repository in [Claude Code](https://claude.com/claude-code), or any coding +agent that can run shell commands and edit code. It takes you from an idea to a running A/B test: + +1. Asks what you want to test and what counts as success, and finds the matching places in your code. +2. Checks that your stack can send events. +3. Creates a [multivariate flag](/managing-flags/core-management) with a keyed variation, a + [metric](/experimentation/create-metrics) and an [experiment](/experimentation/create-an-experiment). +4. Shows you the planned code changes and, once you confirm, adds the SDK calls that + [record exposures and conversion events](/experimentation/run-an-experiment) and starts the experiment. +5. Optionally sends test traffic and refreshes the results, so you can see the experiment working within minutes. + +No application to hand? The agent can build a small demo page or script instead. + +## Prerequisites + +- A Flagsmith instance with a [connected warehouse](/experimentation/connect-a-warehouse): either Flagsmith Cloud, or a + local instance set up with the [local setup prompt](/getting-started/run-locally-with-a-coding-agent). +- An application using an SDK that supports experimentation: JavaScript or React (`@flagsmith/flagsmith` 12.4.0+), + Python (`flagsmith` 6.3.0+), Java (`com.flagsmith:flagsmith-java-client` 8.2.0+), Node.js (`@flagsmith/nodejs` + 9.1.0+), Go (`flagsmith-go-client/v5` v5.3.0+) or Flutter (`flagsmith` 6.2.0+). See + [Run an Experiment](/experimentation/run-an-experiment). +- The [Flagsmith CLI](/integrating-with-flagsmith/CLI) v2 or later. The agent installs it if it's missing. +- Claude Code or another coding agent. + +## The prompt + +```text title="Prompt: set up an experiment" +Help me run an A/B test in this codebase with Flagsmith Experimentation. Work step by step and verify each step. +If something fails, stop and show me the exact error; don't guess or work around it. + +1. Ask me these questions first and wait for my answers: + - What do you want to test? (For example, "the copy on the checkout button".) Find the component or file that + renders it. + - What counts as success? (For example, "a completed purchase".) Find where that happens in the code; the + conversion event goes there. + - Which Flagsmith are we using? Flagsmith Cloud, or a local instance (API http://localhost:8000, event ingestion + service http://localhost:8080)? For a local instance, ask me for the path to the `flagsmith.env` file the local + setup wrote; it holds the CLI credentials. + If there is no app to change, offer to build a tiny demo instead (a single HTML page or a script) that simulates + users, so results show up within minutes. + +2. Check the stack. Read https://docs.flagsmith.com/experimentation/run-an-experiment and follow its tab for this + app's language from here on; it has the install, initialise, exposure and conversion code for each SDK. The SDKs + that can send events are JavaScript and React (`@flagsmith/flagsmith` 12.4.0+), Python (`flagsmith` 6.3.0+), Java + (`com.flagsmith:flagsmith-java-client` 8.2.0+), Node.js (`@flagsmith/nodejs` 9.1.0+; the older `flagsmith-nodejs` + package can't), Go (`flagsmith-go-client/v5` v5.3.0+) and Flutter (`flagsmith` 6.2.0+). If the app uses none of + them, tell me and stop. Find how the app identifies users: exposures and conversions are joined per identity, so + both must use the same identifier. + +3. Connect the Flagsmith CLI. If `flagsmith --version` doesn't report v2 or later, install it with + `curl -fsSL https://get.flagsmith.com | sh` or `brew install Flagsmith/tap/flagsmith`; use + `npm install -g @flagsmith/cli` only as a fallback. For Cloud, ask me to run `flagsmith login`; for a local + instance, run `source `. Don't copy that file or its admin API key into this repository, + and never print the key. Ask me which project and environment to use (`flagsmith project list`, + `flagsmith environment list --project `) and bind this repository to them with + `flagsmith init --project --environment `. Make the API calls below with + `flagsmith api ` (`--input -` reads a JSON body from stdin). Don't use the Flagsmith MCP server: it can't + start experiments. The environment must have a warehouse: if + `GET api/v1/environments//warehouse-connections/` returns none, send me to + https://docs.flagsmith.com/experimentation/connect-a-warehouse and stop. + +4. Create a multivariate flag: `flagsmith feature create --value control --enabled`, then add each variation + with `flagsmith feature variant add --value --key --weight 0`. Every variation needs a key: + one without a key records no exposures. `control` is reserved for the flag's own value, and `--variants` on + `feature create` can't set keys. Check with `flagsmith feature get --json`. + +5. Create the metric: `POST api/v1/environments//experiment-metrics/` with + {"name": "", "aggregation": "occurrence", "direction": "up", "definition": {"version": 1, "event": ""}} + The event name is case-sensitive and must match what the code sends. + +6. Create the experiment: `POST api/v1/environments//experiments/` with + {"feature": , "name": "", "hypothesis": "", + "metrics": [{"metric": , "expected_direction": "increase"}], + "experiment_rollout": {"enabled": true, "rollout_percentage": 100, + "feature_state_value": {"type": "string", "value": "control"}, + "multivariate_feature_state_values": [{"multivariate_feature_option": , "percentage_allocation": 50}]}} + Each variation gets its `percentage_allocation` and control gets the rest. Don't start it yet: starting locks the + flag and turns on the rollout. + +7. Plan the code changes and show them to me. Don't edit anything until I confirm. Then: + - Install or upgrade the SDK and initialise it with events enabled, using the environment key the app already + uses for flags. For a local instance, also set the page's two self-hosted options: the API URL + http://localhost:8000/api/v1/ and the event ingestion service URL http://localhost:8080/. + - Identify the user, then get the experiment flag (for example `getExperimentFlag`, or `useExperiment` in React) + where the user actually sees the variation, and render by its value. + - Track the conversion event where success happens. + - Send buffered events before the app exits, as the page shows for this SDK. + Keep keys out of source control, following how the app already handles configuration. Once the changes are in, + start the experiment with `POST api/v1/environments//experiments//start/` and check its + status is `running`. + +8. Offer to send test traffic with a short script that simulates a few hundred users, each with its own identifier: + get the experiment flag for each one and track the event for some of them. The SDKs send events in batches, so + send what is left before the script exits: JavaScript `await flushEvents()`, Python + `EventProcessorConfig(max_buffer_items=1)`, Node.js `await close()`, Java `close()`, Go `FlushEvents(ctx)`, + Flutter `await flushEvents()`. About a minute later, call + `POST api/v1/environments//experiments//results/refresh/` (allowed once every 5 minutes) and + read `GET .../experiments//results/`. Finish with a link to the experiment: + /project//environment//experiments/, where the dashboard is + https://app.flagsmith.com on Cloud or http://localhost:8000 locally. +``` + +## After it runs + +Open the experiment in the dashboard to watch exposures arrive and read the results; see +[Analyse an Experiment](/experimentation/analyse-an-experiment). The flag is locked while the experiment runs. + +If the agent stops on an error, the most common causes are: + +- **No exposures recorded**: a variation has no key, the user isn't identified before the experiment flag is requested, + or the process exited before buffered events were sent. +- **Conversions not counted**: the event name doesn't exactly match the metric's event, or the event was sent with a + different identifier from the exposure. +- **Events not arriving on a local instance**: the SDK is still sending to Flagsmith Cloud. Set the event ingestion + service URL to `http://localhost:8080/`. + +See [Troubleshooting](/experimentation/run-an-experiment#troubleshooting) for more checks. diff --git a/docs/docs/getting-started/run-locally-with-a-coding-agent.mdx b/docs/docs/getting-started/run-locally-with-a-coding-agent.mdx new file mode 100644 index 000000000000..434cbce1528a --- /dev/null +++ b/docs/docs/getting-started/run-locally-with-a-coding-agent.mdx @@ -0,0 +1,98 @@ +--- +title: Run Flagsmith Locally with a Coding Agent +sidebar_label: Run Locally with an Agent +sidebar_position: 2.5 +description: + A copy-paste prompt that has a coding agent run Flagsmith with experimentation on your machine using Docker. +--- + +# Run Flagsmith Locally with a Coding Agent + +Paste the prompt below into [Claude Code](https://claude.com/claude-code), or any coding agent that can run shell +commands, and it will bring up a complete Flagsmith instance on your machine with Docker: + +1. Checks that Docker is installed and running, and that the ports Flagsmith needs are free. +2. Downloads the Flagsmith [Docker Compose file](/deployment-self-hosting/hosting-guides/docker), starts it and waits + for it to become healthy. +3. Creates your admin account, an organisation, a project, and Development and Production environments. +4. Connects the bundled ClickHouse warehouse to both environments, so you can run [experiments](/experimentation) + straight away. +5. Installs the [Flagsmith CLI](/integrating-with-flagsmith/CLI) and authenticates it against your instance. +6. Prints a summary with your login, environment keys and URLs. + +## Prerequisites + +- [Docker](https://docs.docker.com/get-docker/) with Docker Compose v2, and at least 4 GB of memory allocated to Docker. +- Ports 8000, 8001 and 8080 free on your machine. +- Claude Code or another coding agent. + +## The prompt + +```text title="Prompt: run Flagsmith locally" +Set up a local Flagsmith instance with experimentation for me, using Docker. Work step by step. After each step, +verify that it worked. If something fails, stop and show me the exact error; don't guess or work around it. + +1. Prerequisites. Check that Docker is running (`docker info`) and Docker Compose v2 is available + (`docker compose version`), and that ports 8000, 8001 and 8080 are free. If anything is missing or a port is + taken, tell me how to fix it and stop. Never stop or kill anything that is already running. + +2. Start Flagsmith. In a new `flagsmith` directory, download + https://raw.githubusercontent.com/Flagsmith/flagsmith/main/docker-compose.yml and run `docker compose up -d`. + The first run pulls several images and can take a few minutes. Wait until + http://localhost:8000/health/liveness/ and http://localhost:8080/health return 200, and `docker compose ps -a` + shows the one-off services (`migrate-db`, `kafka-init`, `clickhouse-init`) exited with code 0. If a service + fails, show me its last log lines. + +3. Create the admin account. Ask me for my name and email address (or use Admin User / admin@example.com if I + don't mind) and generate a strong password. There is no CLI bootstrap: the dashboard's onboarding screen does + this while `GET http://localhost:8000/version/` reports `self_hosted_data.has_users: false`, and you do the same + through the API: + - `POST /api/v1/auth/users/` with `{"email", "password", "re_password", "first_name", "last_name", + "superuser": true}`. `superuser` is only accepted while no users exist. The response's `key` authenticates + the next calls as `Authorization: Token `. If `has_users` is already true, ask me for the existing + credentials and get a key from `POST /api/v1/auth/login/` with `{"email", "password"}` instead. + - Create an organisation (`POST /api/v1/organisations/` with `{"name"}`), a project (`POST /api/v1/projects/` + with `{"name", "organisation"}`) and two environments, Development and Production + (`POST /api/v1/environments/` with `{"name", "project"}`). Reuse any that already exist. + +4. Connect the bundled ClickHouse warehouse to each environment with + `POST /api/v1/environments//warehouse-connections/` and this body: + {"warehouse_type": "clickhouse", "config": {"host": "clickhouse", "port": 8123, "database": "flagsmith_exp", + "username": "flagsmith", "secure": false}, "credentials": {"password": "password"}} + Each should come back with `"status": "connected"`. To check settings without saving, POST the same body to + `/api/v1/environments//warehouse-connections/test-warehouse-connection/`. If `/version/` reports + `experimentation_enabled: false`, skip this step and tell me. + +5. Set up the Flagsmith CLI. If `flagsmith --version` doesn't report v2 or later, install it with + `curl -fsSL https://get.flagsmith.com | sh`, which puts it in `$HOME/.local/bin`, or with + `brew install Flagsmith/tap/flagsmith`; use `npm install -g @flagsmith/cli` only as a fallback. Check the version + again. Authenticate the CLI with an organisation API key, not `flagsmith login`: create one with + `POST /api/v1/organisations//master-api-keys/` and `{"name": "Flagsmith CLI", "is_admin": true}`; its `key` + is only returned once. Against a self-hosted instance the CLI only reads host-scoped keys, so write these two + lines to `flagsmith/flagsmith.env`, with file permissions 600: + export FLAGSMITH_API_URL=http://localhost:8000 + export FLAGSMITH_API_KEY_localhost_8000= + Then load the file and check that `flagsmith auth status` and `flagsmith project list` work. + +6. Finish with a summary: the dashboard URL (http://localhost:8000) with my email and password; the organisation, + project and environment names with each environment's key; the event ingestion service URL for SDKs + (http://localhost:8080); how to load the CLI credentials (`source flagsmith/flagsmith.env`); and how to stop + Flagsmith (`docker compose down`) or delete everything (`docker compose down -v`). +``` + +## What you get + +| Service | URL | +| ----------------------- | ----------------------- | +| Dashboard and API | `http://localhost:8000` | +| Task processor | `http://localhost:8001` | +| Event ingestion service | `http://localhost:8080` | + +The instance stores its data in Docker volumes, so it survives restarts until you run `docker compose down -v`. Treat it +as a local playground: the Compose file uses default secrets that you should change before exposing it anywhere. For +production deployments, see [Deployment and Self-hosting](/deployment-self-hosting). + +## Next steps + +To run your first A/B test against this instance, open your application's repository and use the +[experiment set-up prompt](/experimentation/set-up-with-a-coding-agent).