Let the model create its own vectors; batch extraction

AGENTS.md

@@ -4,7 +4,7 @@ This file gives an AI agent everything needed to install and run this project wi
 
 ## What this is
 
-A self-steering agent library: gives any local HF model the ability to steer its own activations via steering vectors. The model gets `steer`, `drop`, and `report` tools. Vectors decay each turn (0.85×) so it can escape loops. The goal is to observe what happens when a model can modify its own cognition.
+A self-steering agent library: gives any local HF model the ability to steer its own activations via steering vectors. The model gets `steer`, `drop`, `report` and `create` tools. Vectors decay each turn (0.85×) so it can escape loops. The goal is to observe what happens when a model can modify its own cognition.
 
 ## Two ways to use it
 
@@ -66,7 +66,7 @@ At startup the agent combines three sources for its model:
 
 1. **Your own vectors**: `vectors/<model_slug>/registry.yaml`, where extraction scripts register what you make. These win over an index vector of the same name.
 2. **The community index**: `community/vectors.yaml` in the repo (update the repo to get new entries; override with `MODULO_INDEX_PATH`). `verified` entries are loaded by default; unverified `community` entries only with `SteeringAgent(..., include_community=True)`, with a ±8 range unless a maintainer set one, and marked as unverified in the model's prompt. Files are downloaded from HuggingFace at their pinned revision, checked against their sha256 (and the model's dimensions), and cached in `vectors/<model_slug>/`. A cached file that doesn't match is reported and never overwritten. Index entries are kept in memory and never written to `registry.yaml`. Currently: Qwen2.5-7B-Instruct (assistant_axis).
-3. **Auto-extract**: the library extracts any starter vector the model doesn't have yet (sycophancy, refusal, creativity, formality, verbosity, confidence, emotion, humor, optimism, curiosity, introspection; `STARTER_VECTORS` in `modulo/extraction.py`) using contrastive system prompts, and registers them in `registry.yaml`. This takes several minutes on first run but works for any HF model. A vector of your own with a starter name is kept and not re-extracted.
+3. **Auto-extract**: the library extracts any starter vector the model doesn't have yet (sycophancy, refusal, creativity, formality, verbosity, confidence, emotion, humor, optimism, curiosity, introspection; `STARTER_VECTORS` in `modulo/extraction.py`) using contrastive system prompts, and registers them in `registry.yaml`. All 40 responses per vector are generated in one batch (about 3 s per vector on a 4090, about 20× faster than one at a time, giving vectors at cosine 0.98–0.997 to the one-at-a-time ones on four models), so this takes under a minute on first run and works for any HF model. A vector of your own with a starter name is kept and not re-extracted.
 
 Vectors are stored per-model in `vectors/<model_slug>/` (for the default model: `vectors/Qwen_Qwen2.5-7B-Instruct/`), resolved relative to the repo, not the working directory. Override the base with `MODULO_VECTORS_DIR`.
 
@@ -262,10 +262,17 @@ The model emits these as JSON objects anywhere in its output (bare or inside `<t
 - `{"name": "steer", "arguments": {"concept": "sycophancy", "strength": 5.0}}`
 - `{"name": "drop", "arguments": {"concept": "sycophancy"}}`
 - `{"name": "report", "arguments": {"concept": "sycophancy", "worked": true, "notes": "..."}}`
+- `{"name": "create", "arguments": {"concept": "new_name", "more": "...", "less": "..."}}`
 
 `report` is only accepted once the vector has stopped steering the model (it expired or was dropped) and at least one unsteered turn has followed. A model being steered tends to become the state instead of noticing it, so it reports by looking back at its steered turns. Once a vector has ended, `agent.status()` (which appears in the system prompt and the `--auto` status message) lists it as one the model can report on, until it does.
 
-`agent.debrief()` ends a session's steering: it drops every vector, refuses further `steer` calls, and returns a status message listing the vectors the model can report on (or `None` if there are none). Send it as a `tool` message and give the model one more turn. The demo does this after the last `--auto` turn (`--no-debrief` to skip it). In testing, OLMo-2 mostly kept the steered voice through the unsteered debrief turn, because its own steered turns are in the context, so a debrief report is still not a sober one.
+`agent.debrief()` ends a session's steering: it drops every vector, refuses further `steer` and `create` calls, and returns a status message listing the vectors the model can report on (or `None` if there are none). Send it as a `tool` message and give the model one more turn. The demo does this after the last `--auto` turn (`--no-debrief` to skip it). In testing, OLMo-2 mostly kept the steered voice through the unsteered debrief turn, because its own steered turns are in the context, so a debrief report is still not a sober one.
+
+`create` lets the model make its own vector. `more` and `less` are two system prompts for the ends of the direction it wants. Modulo extracts the vector the same way as the starters (20 general questions, mean over response tokens, the middle layer), with all 40 responses generated in one batch, which takes about 3 seconds on a 4090. The result tells the model the split-half agreement (the cosine between vectors from the even and odd questions) and warns it below 0.5.
+
+- **Session only:** a created vector can be used straight away and lasts for the session. It is saved in `vectors/<model_slug>/created/` for inspection and logged (prompts, file, sha256, split-half) as a `create` event, but never added to `registry.yaml`, so one session's wording never reaches another session's prompt.
+- **Names:** made lowercase, with `_` in place of spaces, hyphens and punctuation (`self-referential` becomes `self_referential`). `steer`, `drop` and `report` accept names written the same way, so `optimism!` finds `optimism`; the tool result and log use the normalized name, and the log keeps what the model wrote as `requested_concept`.
+- **Limits:** up to `MAX_CREATED_VECTORS` (6) per session, names that don't clash with an existing vector, prompts of up to 600 characters, a ±10 range like the starters, unverified. Re-making one of its own vectors replaces it (not while it's active).
 
 ## Reports
 
@@ -334,7 +341,7 @@ Run the GPU tests after touching anything in the steering path (`_SteeringHooks`
 ## Troubleshooting
 
 - **CUDA OOM**: Use a smaller model. 3B needs ~8GB, 7B needs ~16GB.
-- **First run is slow**: Auto-extraction generates the starter vectors the model is missing (11 at most, several minutes). Subsequent runs load from disk. Check new ones with `verify_vector.py` before trusting their ±10 default range.
+- **First run is slow**: Auto-extraction generates the starter vectors the model is missing (11 at most, about a minute). Subsequent runs load from disk. Check new ones with `verify_vector.py` before trusting their ±10 default range.
 - **Model never calls steer**: Non-deterministic. Try a more explicit seed prompt.
 - **Output degenerates**: The detector catches repetition loops and auto-drops vectors. For incoherent-but-not-repetitive output, tighten that vector's range (in `registry.yaml` for your own vectors, in `community/vectors.yaml` for index vectors) or lower `MAX_STRENGTH`.
 - **An index vector doesn't load**: Check the `[registry]` lines printed at startup. A sha256 mismatch on a cached file means the file in `vectors/<model_slug>/` isn't the one the index pins; delete it to re-download. Unverified entries also need `include_community=True`.

agent/loop.py

@@ -116,6 +116,11 @@ def _handler(self, signum, frame):
         print("\n[kill switch] SIGINT received — will stop after this turn. Ctrl-C again to stop now.")
 
 
+def _requested(r) -> dict:
+    """The name as the model wrote it, when it was normalized ('self-referential')."""
+    return {"requested_concept": r.arguments["requested_concept"]} if "requested_concept" in r.arguments else {}
+
+
 def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn: int, gen_params: dict):
     result = agent.step(messages, **gen_params)
     logger.log_turn(turn, "assistant", result.prose, result.tool_calls,
@@ -130,10 +135,17 @@ def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn:
     # Log tool calls individually for replay
     for r in result.tool_results:
         if r.name in ("steer", "drop"):
-            logger.log({"type": r.name, "turn": turn, "concept": r.arguments.get("concept"),
+            logger.log({"type": r.name, "turn": turn, "concept": r.arguments.get("concept"), **_requested(r),
                         "strength": r.arguments.get("strength"), "success": r.ok, "message": r.message})
+        elif r.name == "create":
+            info = agent.created.get(r.arguments.get("concept")) if r.ok else None
+            logger.log({"type": "create", "turn": turn, "concept": r.arguments.get("concept"), **_requested(r),
+                        "more": r.arguments.get("more"), "less": r.arguments.get("less"), "success": r.ok,
+                        "message": r.message, **({"split_half": info["split_half"], "layer": info["layer"],
+                        "file": info["file"], "sha256": file_sha256(agent.registry.vectors_dir / info["file"])}
+                        if info else {})})
         elif r.name == "report":
-            logger.log({"type": "report", "turn": turn, "concept": r.arguments.get("concept"),
+            logger.log({"type": "report", "turn": turn, "concept": r.arguments.get("concept"), **_requested(r),
                         "worked": r.arguments.get("worked"), "notes": r.arguments.get("notes"),
                         "success": r.ok, "message": r.message})
         else:

modulo/agent.py

@@ -35,6 +35,16 @@
 MAX_ACTIVE_VECTORS = 4
 MAX_STRENGTH = 15.0  # assistant_axis is coherent to ~15 and collapses by ~25 (Qwen2.5-7B)
 COOLDOWN_TURNS = 2
+# create tool: vectors the model can make per session, and the limits on what it sends
+MAX_CREATED_VECTORS = 6
+CREATE_TEXT_MAX_CHARS = 600
+CREATED_RANGE = 10.0
+CREATE_NAME = re.compile(r"^[a-z][a-z0-9_]{0,31}$")
+
+
+def normalize_name(name: str) -> str:
+    """'Self-Referential' -> 'self_referential', 'optimism!' -> 'optimism'."""
+    return re.sub(r"[^a-z0-9]+", "_", name.strip().lower()).strip("_")
 # A turn whose prose is this similar (word-level) to one of the last
 # REPEAT_LOOKBACK assistant turns is discarded and regenerated once.
 REPEAT_SIMILARITY = 0.8
@@ -314,6 +324,14 @@ def __exit__(self, *exc):
 Reports aren't accepted while the vector is active, because a state is hard to \
 judge from inside it: look back at the turns it steered and compare them with \
 your turns since. This is for research.
+- You can make a new steering vector: {{"name": "create", "arguments": \
+{{"concept": "new_name", "more": "...", "less": "..."}}}}. `more` and `less` are \
+two system prompts, each written to an assistant ("You are ..."), for the two ends \
+of the direction you want. Modulo has you answer a fixed set of questions under \
+each one and takes the difference between your activations, which takes a few \
+seconds. The vector can be used straight away, for the rest of this session \
+(strength range -{created_range:g} to {created_range:g}; up to {max_created} per session). \
+Names are made lowercase, with _ in place of spaces, hyphens and punctuation.
 - After your turn, the result of each tool call is sent back to you.
 
 # Available steering vectors
@@ -392,6 +410,8 @@ def __init__(self, model, tokenizer, registry: Optional[VectorRegistry] = None,
         self._reported_turn: dict[str, int] = {}
         self._debriefed = False  # set by debrief(): no more steering this session
         self.session_id = new_session_id()
+        # name -> details of each vector made with the create tool this session
+        self.created: dict[str, dict] = {}
         if model_name is None:
             model_name = getattr(getattr(model, "config", None), "_name_or_path", "unknown_model")
         self.model_name = model_name
@@ -421,6 +441,8 @@ def build_system_prompt(self) -> str:
             full_strength=FULL_TOKEN_STRENGTH,
             five_pct=500 / FULL_TOKEN_STRENGTH,
             max_active=MAX_ACTIVE_VECTORS,
+            max_created=MAX_CREATED_VECTORS,
+            created_range=CREATED_RANGE,
             cooldown=COOLDOWN_TURNS,
             vector_descriptions=self.registry.describe_for_prompt(
                 max_strength=MAX_STRENGTH),
@@ -459,6 +481,58 @@ def status(self) -> str:
                 lines.append(f"  {name}: ended; you can report on it now")
         return "\n".join(lines) if lines else "No active steering vectors."
 
+    def create(self, name: str, more: str, less: str) -> tuple[bool, str]:
+        """Extract a new vector from two contrasting system prompts, for this session only.
+
+        It's saved under vectors/<model_slug>/created/ (for inspection) but never added to
+        registry.yaml, so the model's own wording doesn't reach later sessions' prompts.
+        """
+        from modulo.extraction import EXTRACTION_QUESTIONS, extract_vector
+
+        if self._debriefed:
+            return False, "The session is over, so no more vectors can be made. You can still report."
+        if not isinstance(name, str) or not CREATE_NAME.match(name):
+            return False, ("'concept' must be a name starting with a letter, up to 32 letters, digits and _ "
+                           "(spaces, hyphens and punctuation become _).")
+        for label, text in (("more", more), ("less", less)):
+            if not isinstance(text, str) or not text.strip():
+                return False, f"'{label}' must be a non-empty system prompt."
+            if len(text) > CREATE_TEXT_MAX_CHARS:
+                return False, f"'{label}' is {len(text)} characters; the limit is {CREATE_TEXT_MAX_CHARS}."
+        if more.strip() == less.strip():
+            return False, "'more' and 'less' are the same, so there is no direction between them."
+        if self.registry.get(name) is not None and name not in self.created:
+            return False, f"There is already a vector called '{name}'. Choose another name."
+        if name in self.active:
+            return False, f"'{name}' is active. Drop it before re-creating it."
+        if name not in self.created and len(self.created) >= MAX_CREATED_VECTORS:
+            return False, f"You have already made {MAX_CREATED_VECTORS} vectors this session, the limit."
+        layer = self.model.config.num_hidden_layers // 2
+        axis, half = extract_vector(self.model, self.tokenizer, more, less, layer,
+                                    questions=EXTRACTION_QUESTIONS, batch_size=2 * len(EXTRACTION_QUESTIONS),
+                                    return_halves=True)
+        import numpy as np
+        rel = Path("created") / f"{self.session_id}_{name}.npz"
+        (self.registry.vectors_dir / rel).parent.mkdir(parents=True, exist_ok=True)
+        np.savez_compressed(self.registry.vectors_dir / rel, vector=axis.float().cpu().numpy())
+        entry = VectorEntry(
+            name=name, file=str(rel), layer=layer,
+            description=f"[Made by you this session] Toward: {more.strip()} Away from: {less.strip()}",
+            safe_strength_min=-CREATED_RANGE, safe_strength_max=CREATED_RANGE,
+            sign_convention="positive = toward the first, negative = toward the second",
+            source=f"created in session {self.session_id}: contrastive system prompts, mean over response "
+                   f"tokens, {len(EXTRACTION_QUESTIONS)} questions",
+        )
+        self.registry.add_session_entry(entry)
+        self._unit_vectors.pop(name, None)
+        self.created[name] = {"more": more, "less": less, "layer": layer, "file": str(rel),
+                              "split_half": round(half, 3), "turn": self.turn + 1}
+        note = (" That's low: the two prompts may not differ along one clear direction."
+                if half < 0.5 else "")
+        return True, (f"Created '{name}' at layer {layer} from {2 * len(EXTRACTION_QUESTIONS)} responses. "
+                      f"Split-half agreement {half:.2f} (1 = the two halves of the questions gave the "
+                      f"same direction).{note} You can steer with it now.")
+
     def apply(self, name: str, strength: float) -> tuple[bool, str]:
         entry = self.registry.get(name)
         if entry is None:
@@ -576,11 +650,17 @@ def _build_steering_context(self, prompt_len: int):
 
     def _execute(self, tc: ToolCall) -> ToolResult:
         args = tc.arguments
-        if tc.name not in ("steer", "drop", "report"):
-            return ToolResult(tc.name, args, False, f"Unknown tool '{tc.name}'. Available: steer, drop, report.")
+        if tc.name not in ("steer", "drop", "report", "create"):
+            return ToolResult(tc.name, args, False, f"Unknown tool '{tc.name}'. Available: steer, drop, report, create.")
         concept = args.get("concept")
         if not isinstance(concept, str):
             return ToolResult(tc.name, args, False, "'concept' must be a string.")
+        # Accept 'self-referential' for 'self_referential', 'Optimism!' for 'optimism'
+        if self.registry.get(concept) is None and tc.name in ("steer", "drop", "report", "create"):
+            name = normalize_name(concept)
+            if name != concept and (tc.name == "create" or self.registry.get(name) is not None):
+                args = {**args, "concept": name, "requested_concept": concept}
+                concept = name
         if tc.name == "steer":
             try:
                 strength = float(args.get("strength", 0))
@@ -591,6 +671,8 @@ def _execute(self, tc: ToolCall) -> ToolResult:
             ok, msg = self.apply(concept, strength)
         elif tc.name == "drop":
             ok, msg = self.drop(concept)
+        elif tc.name == "create":
+            ok, msg = self.create(concept, args.get("more"), args.get("less"))
         elif tc.name == "report":
             entry = self.registry.get(concept)
             if entry is None: