diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/README.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/README.md new file mode 100644 index 0000000..5cb844f --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/README.md @@ -0,0 +1,16 @@ +# Multi-Agent Problems in World Models + +Meeting: 2026-10-26 · Yuxin Wang · before-meeting + +Open `presentation.html` directly in a desktop browser for the Chinese slides. Keep `assets/` beside it. Text, figures, fonts and controls work offline; paper links require an internet connection. No embedded videos are required. + +- Arrow keys / Page Up / Page Down: navigate. Home / End: first / last slide. +- O: slide directory. N: speaker notes. F: full screen. Escape: close an overlay. +- Hover, focus or click dotted-underlined terms to read definitions. +- Click a paper figure to enlarge it; use the detail button to inspect it by scrolling. + +The English companion pages use the repository stylesheet. Serve the repository root locally to preview them with site navigation. `references.md` and `idea-map.md` remain readable without a server. Figure attribution is in [figure-credits.html](figure-credits.html). + +The preparation includes 43 slides and 15 papers: seven core readings, five supporting comparisons, two brief latent-action backgrounds, and one brief distributed-generation perspective. Literature cutoff: 2026-10-23. No meeting outcomes or assigned follow-ups are claimed. + +The revised slides contain 14 original teaching diagrams and 21 reproduced paper figures. Each of the six task definitions has a concrete scene example, and each of the seven core methods has a simplified explanatory diagram. Schematic examples are explicitly distinguished from paper experiments. diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/action-items.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/action-items.md new file mode 100644 index 0000000..064672f --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/action-items.md @@ -0,0 +1,11 @@ +# Action Items + +Multi-Agent Problems in World Models · 2026-10-26 + +## Status before the meeting + +No follow-up actions have been agreed. There are no assigned owners, deadlines, or commitments to run experiments. + +The open questions and possible diagnostic in the [topic summary](summary.html) are prompts for discussion, not an action list. Any agreed follow-up work can be recorded here after the meeting, together with its actual owner, expected outcome, and agreed timing. + +[Meeting note](meeting-note.html) · [Discussion overview](./) diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/.gitkeep b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/CREDITS.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/CREDITS.md new file mode 100644 index 0000000..640a9c3 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/CREDITS.md @@ -0,0 +1,50 @@ +# Figure credits + +## Original teaching diagrams + +These fourteen editable SVG diagrams were created for this presentation to explain task conditions and simplified method flows. Characters, actions, trajectories, object arrangements and decisions are illustrative; they are not paper figures, experimental outputs or additional empirical evidence. Method references identify the concepts being explained. Original paper figures are listed separately below. + +| File | Explanatory purpose | Method basis | +|---|---|---| +| [teaching-a-lam-basics.svg](teaching-a-lam-basics.svg) | LAM input and training goal | FLAM §3–4; PlaySlot §3 | +| [teaching-a-action-learning.svg](teaching-a-action-learning.svg) | Two concurrent visual changes without per-agent action labels | FLAM §3–4 | +| [teaching-a-flam-mechanism.svg](teaching-a-flam-mechanism.svg) | Factor-wise action inference and whole-scene prediction | FLAM §4.1–4.2 | +| [teaching-a-control-binding.svg](teaching-a-control-binding.svg) | Correct and swapped assignments of supplied controls | ActionParty §1, §3.1 | +| [teaching-a-actionparty-mechanism.svg](teaching-a-actionparty-mechanism.svg) | Action–subject and subject–space associations | ActionParty §3.1–3.3 | +| [teaching-b-other-actions.svg](teaching-b-other-actions.svg) | Supplied actions versus unspecified other-agent behavior | COMBAT / WorldMind task framing | +| [teaching-b-combat.svg](teaching-b-combat.svg) | Other-agent behavior contained in predicted video | COMBAT §3 | +| [teaching-b-worldmind.svg](teaching-b-worldmind.svg) | Visual state, skill history, explicit decisions and video feedback | WorldMind §3 | +| [teaching-b-cooperation.svg](teaching-b-cooperation.svg) | Own action choices under uncertain partner intent | COMBO §3, §5 | +| [teaching-b-combo.svg](teaching-b-combo.svg) | Candidate video futures used to choose an own action | COMBO §4–5 | +| [teaching-c-shared-world.svg](teaching-c-shared-world.svg) | One opening-door event from two viewpoints | ME-World §1–3 task framing | +| [teaching-c-me-flow.svg](teaching-c-me-flow.svg) | View-specific actions, shared history and joint generation | ME-World §3.1–3.4 | +| [teaching-c-count.svg](teaching-c-count.svg) | From two controlled agents to four agents | Gamma-World agent-count task framing | +| [teaching-c-gamma-flow.svg](teaching-c-gamma-flow.svg) | A finite identity pool and shared hub communication | Gamma-World §3.2–3.3 | + +## Reproduced paper figures + +The figures below are reproduced unmodified from the linked paper HTML pages. License labels are transcribed from the source articles; they do not imply that all third-party content has the same license. + +| File | Paper | Version | Figure | Original image | Source article license label | +|---|---|---|---|---|---| +| [control-actionparty-examples.png](control-actionparty-examples.png) | [ActionParty](https://arxiv.org/html/2604.02330v2) | 2604.02330v2 | Fig. 6 | [Original image](https://arxiv.org/html/2604.02330v2/qual_v8.png) | arXiv.org perpetual non-exclusive license | +| [control-actionparty-method.png](control-actionparty-method.png) | [ActionParty](https://arxiv.org/html/2604.02330v2) | 2604.02330v2 | Fig. 2 | [Original image](https://arxiv.org/html/2604.02330v2/arch_v11.png) | arXiv.org perpetual non-exclusive license | +| [control-incantation-examples.png](control-incantation-examples.png) | [Incantation](https://arxiv.org/html/2605.18601v2) | 2605.18601v2 | Fig. 1 / teaser | [Original image](https://arxiv.org/html/2605.18601v2/fig_teaser_optm.png) | arXiv.org perpetual non-exclusive license | +| [lam-flam-edit-blue.png](lam-flam-edit-blue.png) | [FLAM](https://arxiv.org/html/2602.16229v2) | 2602.16229v2 | Fig. 6, blue-agent panel | [Original image](https://arxiv.org/html/2602.16229v2/manipulate_blue_agent.png) | CC BY-NC-SA 4.0 | +| [lam-flam-method.svg](lam-flam-method.svg) | [FLAM](https://arxiv.org/html/2602.16229v2) | 2602.16229v2 | Fig. 3 | [Original image](https://arxiv.org/html/2602.16229v2/pipeline.svg) | CC BY-NC-SA 4.0 | +| [lam-lpwm-method.png](lam-lpwm-method.png) | [LPWM](https://arxiv.org/html/2603.04553v1) | 2603.04553v1 | Fig. 3 | [Original image](https://arxiv.org/html/2603.04553v1/figs/arch_main_ltr.png) | arXiv.org perpetual non-exclusive license | +| [lam-playslot-method.png](lam-playslot-method.png) | [PlaySlot](https://arxiv.org/html/2502.07600v2) | 2502.07600v2 | Fig. 2(a), training pipeline panel | [Original image](https://arxiv.org/html/2502.07600v2/imgs/main/main_train_v3.png) | CC BY 4.0 | +| [world-combat-action-ratio.png](world-combat-action-ratio.png) | [COMBAT](https://arxiv.org/html/2603.00825v1) | 2603.00825v1 | Figure 5 | [Original image](https://arxiv.org/html/2603.00825v1/figures/Relative_Action_Consistency.png) | CC BY 4.0 | +| [world-combat-method.png](world-combat-method.png) | [COMBAT](https://arxiv.org/html/2603.00825v1) | 2603.00825v1 | Figure 2a | [Original image](https://arxiv.org/html/2603.00825v1/figures/ICCV-T3-DIAG2.png) | CC BY 4.0 | +| [world-combo-composition.png](world-combo-composition.png) | [COMBO](https://arxiv.org/html/2404.10775v3) | 2404.10775v3 | Figure 2 | [Original image](https://arxiv.org/html/2404.10775v3/CWM_v5.png) | CC BY 4.0 | +| [world-combo-method.png](world-combo-method.png) | [COMBO](https://arxiv.org/html/2404.10775v3) | 2404.10775v3 | Figure 3 | [Original image](https://arxiv.org/html/2404.10775v3/method_v4.png) | CC BY 4.0 | +| [world-combo-planning.png](world-combo-planning.png) | [COMBO](https://arxiv.org/html/2404.10775v3) | 2404.10775v3 | Figure 5 | [Original image](https://arxiv.org/html/2404.10775v3/scale.png) | CC BY 4.0 | +| [world-gamma-four-agent.png](world-gamma-four-agent.png) | [Gamma-World](https://arxiv.org/html/2605.28816v1) | 2605.28816v1 | Figure 5 | [Original image](https://arxiv.org/html/2605.28816v1/4agent_visualization.png) | CC BY 4.0 | +| [world-gamma-method.png](world-gamma-method.png) | [Gamma-World](https://arxiv.org/html/2605.28816v1) | 2605.28816v1 | Figure 2 | [Original image](https://arxiv.org/html/2605.28816v1/multiagent_method.png) | CC BY 4.0 | +| [world-me-method.png](world-me-method.png) | [ME-World](https://arxiv.org/html/2610.12299v1) | 2610.12299v1 | Figure 3 | [Original image](https://arxiv.org/html/2610.12299v1/main_architecture.png) | CC BY 4.0 | +| [world-me-motivation.png](world-me-motivation.png) | [ME-World](https://arxiv.org/html/2610.12299v1) | 2610.12299v1 | Figure 2 | [Original image](https://arxiv.org/html/2610.12299v1/motivation.png) | CC BY 4.0 | +| [world-me-results.png](world-me-results.png) | [ME-World](https://arxiv.org/html/2610.12299v1) | 2610.12299v1 | Figure 5 | [Original image](https://arxiv.org/html/2610.12299v1/main_qual.png) | CC BY 4.0 | +| [world-multics-method.png](world-multics-method.png) | [Multics](https://arxiv.org/html/2610.17201v1) | 2610.17201v1 | Figure 2 | [Original image](https://arxiv.org/html/2610.17201v1/pipeline.png) | arXiv perpetual non-exclusive license | +| [world-overseer-method.png](world-overseer-method.png) | [OverSeer](https://arxiv.org/html/2610.28554v1) | 2610.28554v1 | Figure 2 | [Original image](https://arxiv.org/html/2610.28554v1/method.png) | arXiv perpetual non-exclusive license | +| [world-worldmind-cases.png](world-worldmind-cases.png) | [WorldMind](https://arxiv.org/html/2608.21439v1) | 2608.21439v1 | Figure 4 | [Original image](https://arxiv.org/html/2608.21439v1/qualitative.png) | CC BY 4.0 | +| [world-worldmind-method.png](world-worldmind-method.png) | [WorldMind](https://arxiv.org/html/2608.21439v1) | 2608.21439v1 | Figure 3 | [Original image](https://arxiv.org/html/2608.21439v1/framework.png) | CC BY 4.0 | diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-examples.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-examples.png new file mode 100644 index 0000000..7e8fb8f Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-examples.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-method.png new file mode 100644 index 0000000..2d4a959 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-actionparty-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-incantation-examples.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-incantation-examples.png new file mode 100644 index 0000000..c205d26 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/control-incantation-examples.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-edit-blue.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-edit-blue.png new file mode 100644 index 0000000..b1c1cc9 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-edit-blue.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-method.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-method.svg new file mode 100644 index 0000000..b6f83a2 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-flam-method.svg @@ -0,0 +1,317 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-lpwm-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-lpwm-method.png new file mode 100644 index 0000000..389c417 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-lpwm-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-playslot-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-playslot-method.png new file mode 100644 index 0000000..c98d539 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/lam-playslot-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-action-learning.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-action-learning.svg new file mode 100644 index 0000000..9af0650 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-action-learning.svg @@ -0,0 +1 @@ +Separate action-learning taskIllustrative example; not a reported experiment.输入：前一帧AB输入：后一帧AABB希望：分别表示两路变化A 的运动变化B 的运动变化而非只用一份混合表示观察者能看出：A 左移、B 跳起；模型只得到视频，没有这些动作标签。 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-actionparty-mechanism.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-actionparty-mechanism.svg new file mode 100644 index 0000000..cfaec2d --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-actionparty-mechanism.svg @@ -0,0 +1 @@ +ActionParty simplified binding mechanismTeaching schematic of action-subject and subject-space associations, not an actual model rollout. Directions are illustrative.输入：画面 + 分角色动作ABA ←B →动作与该主体状态关联A：状态 + A 的动作B：状态 + B 的动作再用空间信息关联视频区域共同预测：画面与位置AABB框示意 A / B 的位置预测 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-control-binding.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-control-binding.svg new file mode 100644 index 0000000..4653bb9 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-control-binding.svg @@ -0,0 +1 @@ +Action binding task exampleIllustrative example; not a reported experiment.给定：A 左移，B 右移ABA ←B →目标：各做各的动作AABB错误：指令作用到别人AB画面流畅，也可能控制错人这里的动作已知；要检验的是“哪个角色执行了哪条指令”。 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-flam-mechanism.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-flam-mechanism.svg new file mode 100644 index 0000000..ab1c294 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-flam-mechanism.svg @@ -0,0 +1 @@ +FLAM simplified training exampleSimplified explanatory flow based on FLAM. Colors do not imply supervised identities or guaranteed factor-agent alignment.训练输入：前后视频帧AB前AB后场景分解后，分路推断因子 1z₁因子 2z₂前后对应的表示用于推断各自的 z一起预测下一帧当前场景 + z₁、z₂AB输出仍是一幅完整画面 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-lam-basics.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-lam-basics.svg new file mode 100644 index 0000000..d6ab28e --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-a-lam-basics.svg @@ -0,0 +1 @@ +LAM task exampleOriginal teaching schematic; not a paper experiment.观察到两帧：人向左移动AA当前帧后续帧从变化中推断 z潜动作 z不是人工给的“向左”标签当前帧 + z → 预测后续AA用实际后续帧检验预测 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combat.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combat.svg new file mode 100644 index 0000000..f2e5f9c --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combat.svg @@ -0,0 +1 @@ +输入：最近画面AB外部动作条件：A 向前 / 出拳没有给定 B 的动作条件动作条件视频世界模型联合生成画面变化把互动模式学入预测输出：下一段视频ABB 的后退已经在生成画面里没有单独输出“B 后退”指令原创教学示意 · 反应仅用于说明输出形式，不代表论文中的指定测试 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combo.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combo.svg new file mode 100644 index 0000000..92f68c8 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-combo.svg @@ -0,0 +1 @@ +同一组输入取物区AB伙伴行为假设B 去拿面包自身的两个选择候选 1A 也拿面包B 去拿面包（假设）候选 2A 去拿盘子B 去拿面包（假设）想象各自的未来备餐台AB?备餐台AB比较 / 选择示意后果盘子仍未准备示意后果两种物品到位→ A 去拿盘子原创教学示意 · 视频输出候选后果；规划选择自身动作；伙伴预测与生成都可能出错 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-cooperation.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-cooperation.svg new file mode 100644 index 0000000..6ea6191 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-cooperation.svg @@ -0,0 +1 @@ +合作目标：把面包和盘子送到备餐台取物区AB我控制 A；B 独立行动对 B 的假设B 可能去拿面包根据观察推测，并非已知A 应该选择什么？也去拿面包可能重复分工转而去拿盘子可能形成互补需要比较共同后果原创教学示意 · 用于解释规划目标；不对应 COMBO 的具体实验轨迹 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-other-actions.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-other-actions.svg new file mode 100644 index 0000000..59e06c7 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-other-actions.svg @@ -0,0 +1 @@ +刚才：双方动作都给定A：向前B：后退AB按指定动作生成双方现在：只给 A 的动作A：向前B：未指定ABB 的下一步从哪里来？输出仍要包含 BB后退？B迎击？都是待生成的可能行为原创教学示意 · 不是论文实验；不指定唯一正确反应 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-worldmind.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-worldmind.svg new file mode 100644 index 0000000..4cb2d32 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-b-worldmind.svg @@ -0,0 +1 @@ +1 形成当前状态AB从画面估计：双方接近从动作记录计算例：冲刺还需冷却2 显式选 NPC 动作状态 + 技能说明语言模型决策示意：B 后退3 对齐时间1234动作 → 生成条件在哪段时间执行4 生成视频AB执行 B 的动作新画面 + 新动作记录 → 下一轮状态原创教学示意 · 技能与动作是假设例子；几何估计和技能记录来源不同 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-count.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-count.svg new file mode 100644 index 0000000..4ea0a49 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-count.svg @@ -0,0 +1 @@ +从两主体训练到四主体使用的任务示意A/B/C/D各自拥有动作条件与视角输出。自绘任务示意，颜色、动作和布局均非论文实验结果。训练：2个主体A向左A的视角B向右B的视角测试：4个主体，各有输入与输出A向左A的视角B向右B的视角C转身C的视角D抬头D的视角不是多复制两路画面，而是增加两个能被分别控制、参与交互的主体。 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-gamma-flow.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-gamma-flow.svg new file mode 100644 index 0000000..f87ca22 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-gamma-flow.svg @@ -0,0 +1 @@ +Gamma-World身份池与共享hub的示意依据Gamma-World方法自绘，身份池仅展示六个抽象编码并非固定数量。各主体的独立输入与输出通过共享hub交换信息，不是平均成相同视频。预设身份池：为各主体分配不同编码e1e2e3e4e5e6训练随机分配；更多主体用更多编码（6个仅为示意）。A：身份 e3自己的观察＋开门A的后续视频B：身份 e1自己的观察＋移动B的后续视频C：身份 e6自己的观察＋转身C的后续视频D：身份 e4自己的观察＋抬头D的后续视频共享 hub：少量共享表示，交换跨主体信息生成过程中交换信息，各人仍保留自己的表示与输出。 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-me-flow.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-me-flow.svg new file mode 100644 index 0000000..7b96670 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-me-flow.svg @@ -0,0 +1 @@ +ME-World联合生成的输入、条件组织和多视角输出依据ME-World方法的自绘简图，省略内部层。开门仅作说明，不是论文实验。动作在目标视角中表达，共同环境历史参与联合生成，A/B保留不同视频输出。各自的观察与动作A：当前视角＋开门B：当前视角＋移动动作来自外部给定AB按目标视角表达动作A视角 ← A、B动作B视角 ← A、B动作共同环境历史行动前的观察门仍关闭联合生成A视频表示B视频表示共同更新、交换信息初始观察也作为生成条件A的后续视频BB的后续视频A \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-shared-world.svg b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-shared-world.svg new file mode 100644 index 0000000..ad0c38e --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/teaching-c-shared-world.svg @@ -0,0 +1 @@ +一扇打开的门，在两个不同视角中保持同一状态自绘任务示意，不是论文实验。A开门后，A和B的画面不同，但门应同为打开状态。同一次事件：A开门AB门已经打开A的视角B看到门的一侧，以及BB的视角A看到门的另一侧，以及A共同事实：同一扇门已经打开。画面可以不同；若B的画面仍显示门关闭，就与同一次事件冲突。 \ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-action-ratio.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-action-ratio.png new file mode 100644 index 0000000..c89fbab Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-action-ratio.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-method.png new file mode 100644 index 0000000..4a639c9 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combat-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-composition.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-composition.png new file mode 100644 index 0000000..833cf50 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-composition.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-method.png new file mode 100644 index 0000000..f82614b Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-planning.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-planning.png new file mode 100644 index 0000000..f80527c Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-combo-planning.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-four-agent.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-four-agent.png new file mode 100644 index 0000000..c99c42e Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-four-agent.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-method.png new file mode 100644 index 0000000..138a9e5 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-gamma-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-method.png new file mode 100644 index 0000000..292a4f5 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-motivation.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-motivation.png new file mode 100644 index 0000000..d3b44f0 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-motivation.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-results.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-results.png new file mode 100644 index 0000000..9f94133 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-me-results.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-multics-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-multics-method.png new file mode 100644 index 0000000..16bde97 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-multics-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-overseer-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-overseer-method.png new file mode 100644 index 0000000..388cc40 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-overseer-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-cases.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-cases.png new file mode 100644 index 0000000..3d787c3 Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-cases.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-method.png b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-method.png new file mode 100644 index 0000000..57a29ba Binary files /dev/null and b/discussions/2026-10-26-yuxin-multi-agent-world-models/assets/world-worldmind-method.png differ diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/figure-credits.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/figure-credits.html new file mode 100644 index 0000000..c28938c --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/figure-credits.html @@ -0,0 +1,74 @@ + + + + + + + Figure credits · Multi-Agent Problems in World Models · CausalVideoGroup + + + + +
+ CausalVideoGroup + +
+
+

Reading materials · 2026-10-26

Figure credits

Multi-Agent Problems in World Models

+
+

Original teaching diagrams

These fourteen editable SVG diagrams were created for this presentation to explain task conditions and simplified method flows. Characters, actions, trajectories, object arrangements and decisions are illustrative; they are not paper figures, experimental outputs or additional empirical evidence. Method references identify the concepts being explained. Original paper figures are listed separately below.

+ + + + + + + + + + + + + +
FileExplanatory purposeMethod basis
teaching-a-lam-basics.svgLAM input and training goalFLAM §3–4; PlaySlot §3
teaching-a-action-learning.svgTwo concurrent visual changes without per-agent action labelsFLAM §3–4
teaching-a-flam-mechanism.svgFactor-wise action inference and whole-scene predictionFLAM §4.1–4.2
teaching-a-control-binding.svgCorrect and swapped assignments of supplied controlsActionParty §1, §3.1
teaching-a-actionparty-mechanism.svgAction–subject and subject–space associationsActionParty §3.1–3.3
teaching-b-other-actions.svgSupplied actions versus unspecified other-agent behaviorCOMBAT / WorldMind task framing
teaching-b-combat.svgOther-agent behavior contained in predicted videoCOMBAT §3
teaching-b-worldmind.svgVisual state, skill history, explicit decisions and video feedbackWorldMind §3
teaching-b-cooperation.svgOwn action choices under uncertain partner intentCOMBO §3, §5
teaching-b-combo.svgCandidate video futures used to choose an own actionCOMBO §4–5
teaching-c-shared-world.svgOne opening-door event from two viewpointsME-World §1–3 task framing
teaching-c-me-flow.svgView-specific actions, shared history and joint generationME-World §3.1–3.4
teaching-c-count.svgFrom two controlled agents to four agentsGamma-World agent-count task framing
teaching-c-gamma-flow.svgA finite identity pool and shared hub communicationGamma-World §3.2–3.3

Reproduced paper figures

+

The 21 paper figures below are reproduced unmodified from the linked paper HTML pages. License labels are transcribed from the source articles; they do not imply that all third-party content has the same license.

+

Open the Chinese presentation · Discussion overview

+
+ + + + + + + + + + + + + + + + + + + + + + + + + +
FilePaperVersionFigureOriginal imageSource article license label
control-actionparty-examples.pngActionParty2604.02330v2Fig. 6Original imagearXiv.org perpetual non-exclusive license
control-actionparty-method.pngActionParty2604.02330v2Fig. 2Original imagearXiv.org perpetual non-exclusive license
control-incantation-examples.pngIncantation2605.18601v2Fig. 1 / teaserOriginal imagearXiv.org perpetual non-exclusive license
lam-flam-edit-blue.pngFLAM2602.16229v2Fig. 6, blue-agent panelOriginal imageCC BY-NC-SA 4.0
lam-flam-method.svgFLAM2602.16229v2Fig. 3Original imageCC BY-NC-SA 4.0
lam-lpwm-method.pngLPWM2603.04553v1Fig. 3Original imagearXiv.org perpetual non-exclusive license
lam-playslot-method.pngPlaySlot2502.07600v2Fig. 2(a), training pipeline panelOriginal imageCC BY 4.0
world-combat-action-ratio.pngCOMBAT2603.00825v1Figure 5Original imageCC BY 4.0
world-combat-method.pngCOMBAT2603.00825v1Figure 2aOriginal imageCC BY 4.0
world-combo-composition.pngCOMBO2404.10775v3Figure 2Original imageCC BY 4.0
world-combo-method.pngCOMBO2404.10775v3Figure 3Original imageCC BY 4.0
world-combo-planning.pngCOMBO2404.10775v3Figure 5Original imageCC BY 4.0
world-gamma-four-agent.pngGamma-World2605.28816v1Figure 5Original imageCC BY 4.0
world-gamma-method.pngGamma-World2605.28816v1Figure 2Original imageCC BY 4.0
world-me-method.pngME-World2610.12299v1Figure 3Original imageCC BY 4.0
world-me-motivation.pngME-World2610.12299v1Figure 2Original imageCC BY 4.0
world-me-results.pngME-World2610.12299v1Figure 5Original imageCC BY 4.0
world-multics-method.pngMultics2610.17201v1Figure 2Original imagearXiv perpetual non-exclusive license
world-overseer-method.pngOverSeer2610.28554v1Figure 2Original imagearXiv perpetual non-exclusive license
world-worldmind-cases.pngWorldMind2608.21439v1Figure 4Original imageCC BY 4.0
world-worldmind-method.pngWorldMind2608.21439v1Figure 3Original imageCC BY 4.0
+
+
+
+ + + diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/idea-map.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/idea-map.md new file mode 100644 index 0000000..1a93abb --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/idea-map.md @@ -0,0 +1,101 @@ +# Idea Map + +Multi-Agent Problems in World Models · 2026-10-26 + +This map organizes research questions, not an implemented sequence of systems. The terms agent, passive object, learned slot or particle, and camera viewpoint refer to different things. + +## 1. Learn separate actions from video + +**Given:** visual transitions without per-agent action labels. + +**Need:** latent representations that separate changes associated with different entities. + +**Core:** FLAM. + +**Supporting comparisons:** PlaySlot uses object slots and action prototypes plus continuous variation; LPWM models local stochastic transitions through particles. + +**Brief background:** OTF-LAM factorizes local effects and then aggregates a single action representation; GDLAM learns latent groups. Neither establishes that every group is an active agent. + +**Boundary:** Decomposition does not guarantee a one-to-one correspondence with agents or true physical commands. + +**Next question:** If actions are already supplied, how are they applied to the correct subjects? + +## 2. Bind controls to the correct subjects + +**Given:** each subject's action or instruction. + +**Need:** video in which the correct subject executes that action. + +**Core:** ActionParty. + +**Supporting comparison:** Incantation uses per-entity language action slots and explores action-semantic reuse across entities. + +**Boundary:** These methods use action supervision. Learning separate latent actions and binding known controls are different tasks. + +**Next question:** If only some actions are supplied, where do the other characters' actions come from? + +## 3. Generate other agents' behavior + +**Given:** observation history and externally supplied actions for only part of the scene. + +**Need:** behavior for agents whose actions were not specified. + +**Core sequence:** COMBAT, then WorldMind. + +**Contrast:** COMBAT generates opponent behavior within conditional video prediction. WorldMind explicitly connects compact state estimation, NPC decisions, action conditioning, and video generation. + +**Boundary:** An implicit continuation is not an already trained transferable policy. Explicit decisions require suitable state and skill interfaces; the two methods are not a controlled performance ranking. + +**Next question:** How can predictions of behavior and consequences help an agent choose its own cooperative action? + +## 4. Use imagined futures for cooperative planning + +**Given:** partial observations, candidate own actions, and inferred partner actions. + +**Need:** a choice among the acting agent's candidate actions. + +**Core:** COMBO. + +**Mechanism:** Estimate the visual world, generate joint-action consequences, evaluate and search, then execute the selected own action. + +**Boundary:** The planner need not control all partners. Different action candidates are different possible futures and can legitimately disagree. + +**Next question:** For one chosen world evolution, how can several participants' videos agree on shared facts? + +## 5. Maintain one world across views and time + +**Given:** multiple local views, action conditions, and history. + +**Need:** compatible identities, object states, and event outcomes across viewpoints and revisits. + +**Core:** ME-World, with joint generation, shared action conditioning, and shared geometry-aligned memory. + +**Supporting comparison:** OverSeer generates fixed auxiliary views as a persistent common reference. + +**Brief distributed perspective:** WorldCast coordinates local client generation through shared player state and scene memory. + +**Boundary:** A camera is not another agent. Generated auxiliary views are not omniscient observations. Agreement does not by itself demonstrate physical correctness. + +**Next question:** Do these identity and information structures still work with more agents? + +## 6. Generalize to more agents + +**Given:** training scenes with fewer agents than a target inference setting. + +**Need:** distinguishable identities and relevant cross-agent information at the larger population. + +**Core:** Gamma-World, with a preset identity pool and shared hub communication. + +**Supporting comparison:** Multics uses player-group-global communication and dynamic grouping for concurrent interactions. + +**Boundary:** Tested population generalization is not unlimited scale, arbitrary online entry and exit, or guaranteed long-term state persistence. + +## Cross-cutting comparison axes + +- What is supplied: unlabelled video, actions, identity, geometry, skill rules, or history? +- What is inferred: latent actions, binding, behavior, candidate outcomes, shared state, or relations? +- What is the output for: reconstruction, control, decision-making, synchronized views, or population transfer? +- What is evaluated: targeted control, behavioral response, cooperative success, shared events, persistence, or agent-count transfer? +- What remains conditional: training supervision, action format, world coverage, identity capacity, and tested environments? + +The six open discussion questions and a clearly labelled diagnostic example appear in the [English summary](summary.html). All fifteen papers are linked and analyzed in [References](references.html). diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/index.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/index.html new file mode 100644 index 0000000..7686098 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/index.html @@ -0,0 +1,49 @@ + + + + + + + Multi-Agent Problems in World Models · CausalVideoGroup + + + +
+ CausalVideoGroup + +
+
+
+

Topic discussion · 2026-10-26

+

Multi-Agent Problems in World Models

+

Led by Yuxin Wang

+

before-meeting Chinese presentation · English reading materials

+
+
+

What changes when a video world model must represent several agents? This survey follows six questions: learning separate action representations, applying controls to the correct agents, generating others' behavior, planning with imagined futures, maintaining shared facts across viewpoints, and handling more agents than seen during training.

+

The Chinese slides include concrete illustrated examples for all six questions and simplified diagrams for the seven core methods. Teaching schematics are labeled separately from reproduced paper evidence.

+

The discussion emphasizes the actual inputs, outputs, and information assumptions of each method. Fifteen papers provide seven core readings, five supporting comparisons, two brief latent-action backgrounds, and one distributed-generation perspective.

+
+ Open Chinese presentation + Read English summary + Analyzed references +
+

Reading route

+
    +
  1. Action learning: FLAM, with PlaySlot and LPWM; OTF-LAM and GDLAM as background.
    2. +
  2. Control binding: ActionParty, with Incantation.
    3. +
  3. Other-agent behavior: COMBAT followed by WorldMind.
    4. +
  4. Cooperative planning: COMBO.
    5. +
  5. Shared worlds and persistence: ME-World, with OverSeer and a brief WorldCast perspective.
    6. +
  6. Agent-count generalization: Gamma-World, with Multics.
    7. +
+

These are connected research questions, not components of an already integrated system. Objects, particles, active agents, and camera views are distinguished throughout.

+

Problem map and open questions · Meeting note · Figure sources

+

Materials cover literature checked through 2026-10-23. The meeting has not yet taken place; no discussion conclusions or follow-up assignments are recorded.

+
+
+ + + diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.html new file mode 100644 index 0000000..d7a31ab --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.html @@ -0,0 +1,36 @@ + + + + + + + Meeting Note · Multi-Agent Problems in World Models · CausalVideoGroup + + + +
+ CausalVideoGroup + +
+
+

2026-10-26 · Yuxin Wang

Meeting Note

Multi-Agent Problems in World Models

+

Multi-Agent Problems in World Models · 2026-10-26 · Yuxin Wang

+

Meeting status

+

The meeting has not yet taken place. This is the before-meeting record. No participant feedback, changed views, consensus, experimental outcomes, or follow-up decisions are recorded.

+

Discussion material

+

The prepared literature survey covers six questions: action learning, control binding, other-agent behavior, cooperative planning, shared-world consistency and persistence, and agent-count generalization.

+ +

The questions and diagnostic example in the summary are literature-based discussion prompts. They are not meeting conclusions or agreed experiments.

+

Post-meeting record

+

After the meeting, this page can record revised views, concrete critiques, useful connections, and any agreed next steps. At present there are no such outcomes to report. No raw transcript is included.

+

Action status

+

No follow-up tasks, owners, or deadlines have been agreed. The source action-items file therefore contains no assignments.

+

Discussion overview

+
+ + + diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.md new file mode 100644 index 0000000..634fe66 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.md @@ -0,0 +1,27 @@ +# Meeting Note + +Multi-Agent Problems in World Models · 2026-10-26 · Yuxin Wang + +## Meeting status + +The meeting has not yet taken place. This is the before-meeting record. No participant feedback, changed views, consensus, experimental outcomes, or follow-up decisions are recorded. + +## Discussion material + +The prepared literature survey covers six questions: action learning, control binding, other-agent behavior, cooperative planning, shared-world consistency and persistence, and agent-count generalization. + +- [Chinese presentation](presentation.html) +- [English summary, problem map, and open questions](summary.html) +- [Analyzed references](references.html) + +The questions and diagnostic example in the summary are literature-based discussion prompts. They are not meeting conclusions or agreed experiments. + +## Post-meeting record + +After the meeting, this page can record revised views, concrete critiques, useful connections, and any agreed next steps. At present there are no such outcomes to report. No raw transcript is included. + +## Action status + +No follow-up tasks, owners, or deadlines have been agreed. The source action-items file therefore contains no assignments. + +[Discussion overview](./) diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/metadata.yaml b/discussions/2026-10-26-yuxin-multi-agent-world-models/metadata.yaml new file mode 100644 index 0000000..517255e --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/metadata.yaml @@ -0,0 +1,82 @@ +title: "Multi-Agent Problems in World Models" +date: 2026-10-26 +topic_slug: multi-agent-world-models +leader: + name: "Yuxin Wang" + short_name: yuxin +status: before-meeting +summary: >- + A problem-oriented survey of multi-agent video world models: learning separate + action representations, binding controls, generating other agents' behavior, + planning with imagined futures, maintaining a shared world across views, + and generalizing to more agents. +tags: + - world-model + - multi-agent + - video-generation + - latent-action + - embodied-intelligence +related_projects: [] +papers: + - title: "Factored Latent Action World Models" + url: https://arxiv.org/abs/2602.16229 + role: method + note: Core reading; separate latent action channels for scene factors. + - title: "ActionParty: Multi-Subject Action Binding in Generative Video Games" + url: https://arxiv.org/abs/2604.02330 + role: method + note: Core reading; bind supplied actions to the correct subjects. + - title: "COMBAT: Conditional World Models for Behavioral Agent Training" + url: https://arxiv.org/abs/2603.00825 + role: method + note: Core reading; generate opponent behavior under one-sided action conditioning. + - title: "WorldMind: Decoupled Game World Model for State-Aware NPC Behavior" + url: https://arxiv.org/abs/2608.21439 + role: system + note: Core reading; connect estimated state, explicit NPC decisions, and video generation. + - title: "COMBO: Compositional World Models for Embodied Multi-Agent Cooperation" + url: https://arxiv.org/abs/2404.10775 + role: system + note: Core reading; compare imagined joint-action futures for cooperative planning. + - title: "Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction" + url: https://arxiv.org/abs/2610.12299 + role: method + note: Core reading; joint egocentric generation with shared action conditions and memory. + - title: "Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players" + url: https://arxiv.org/abs/2605.28816 + role: method + note: Core reading; identity encoding and shared communication for agent-count generalization. + - title: "PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning" + url: https://arxiv.org/abs/2502.07600 + role: method + note: Supporting comparison; object slots with discrete action prototypes and continuous variation. + - title: "Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling" + url: https://arxiv.org/abs/2603.04553 + role: method + note: Supporting comparison; particle-level stochastic latent actions. + - title: "Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models" + url: https://arxiv.org/abs/2605.18601 + role: method + note: Supporting comparison; per-entity language action interfaces. + - title: "OverSeer: Multi-Agent World Models Beyond What Agents See" + url: https://arxiv.org/abs/2610.28554 + role: method + note: Supporting comparison; generated auxiliary views for off-screen state continuity. + - title: "Multics: Generative Multi-Player World Model with Hierarchical Communication" + url: https://arxiv.org/abs/2610.17201 + role: method + note: Supporting comparison; player-group-global communication for concurrent interactions. + - title: "Latent Actions from Factorized Transition Effects under Agent Ambiguity" + url: https://arxiv.org/abs/2606.30544 + role: method + note: Brief background; local transition effects are aggregated into one latent action. + - title: "GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining" + url: https://arxiv.org/abs/2609.26118 + role: method + note: Brief background; grouped latent representations do not guarantee agent-level decomposition. + - title: "WorldCast: Distributed Multiplayer World Models" + url: https://arxiv.org/abs/2610.12412 + role: system + note: Brief distributed perspective; local video generation coordinated through shared state and scene memory. +external_videos: [] +slides: presentation.html diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/presentation.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/presentation.html new file mode 100644 index 0000000..779682f --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/presentation.html @@ -0,0 +1,23 @@ +世界模型背景下的多主体问题 · Yuxin Wang
导入MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

世界模型背景下的多主体问题

CAUSALVIDEOGROUP · READING GROUP

世界模型背景下的
多主体问题

Multi-Agent Problems in World Models

Yuxin Wang2026.10.26
本次讲述结构／概念示意
01 / 43
导入MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

今天的六个部分

从 LAM 切入，逐步看视频世界模型中的多主体问题。

01多主体动作学习
02多主体控制
03他者行为与反应
04视频想象辅助合作规划
05共享世界与持续性
06主体数量泛化与扩展
七篇重点论文；其他工作用于补充不同思路。文献截止：2026.10.23。
本次讲述结构／概念示意
02 / 43
导入MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

先用一页理解 LAM 在学什么

视频有画面变化，却往往没有对应的动作标签。

任务目标从视频中学习能表达动作／动态变化的潜在表示，用于预测与控制。
原创任务示意 · 先用一个角色的移动理解输入与训练目标
是学出的内部表示；训练时可用后续帧推断变化，再用当前帧与这份表示预测后续。
如果画面里有多个主体同时行动，这份表示该怎样分开？
01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

一个向左走，一个跳起来：动作怎样分别学？

已知：前后视频帧。未知：每个主体自己的动作标签。

原创任务示意 · 颜色和A/B只帮助读图，不代表训练时给定了身份或动作标签

要解决的问题

两路变化同时发生，能否分别表示，而不是全部混在一个潜动作中？

目标与边界

希望得到可分别改变的表示；但学出的因子未必一一对应真实主体。

01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

FLAM：让这个例子走完训练流程

视频里发生的两路变化，怎样经过分解、动作推断，再回到画面？

原创方法简化图 · 依据FLAM的训练流程；人形与轨迹不是论文实验结果
关键关系前后因子表示帮助推断各自的 z；预测时仍使用完整当前场景与这些 z，输出一幅共同画面。
图中按两路画出便于理解；模型可能合并相关实体，因子不保证等于agent。
FLAM 2026 ｜方法简化；§4.1–4.2
05 / 43
01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

FLAM：在原方法图里找到这三步

把刚才的分解、潜动作和共同预测，对应到论文的实际模块。

01分解画面得到场景 slots
02推断动作前后 slots → 各因子潜动作
03预测画面当前场景 ＋ 各潜动作 → 下一帧
FLAM Fig.3 · 原方法图，可点击放大

分别表示，不是彼此隔离

提供分开的表示位置；动力学仍可参考全部当前slots，保留交互信息。

训练：未来帧用于推断潜动作。
输出：各因子表示及重建／预测的未来画面。
01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

看一个动作通道是否能单独改变

FLAM 的定性证据：改变蓝色实体的潜动作，观察生成轨迹。

FLAM Fig.6 的蓝色实体面板 · 原始轨迹与修改后的轨迹

这个例子说明什么

选定的动作表示可以改变对应部分的运动，提供“分别建模”的直观证据。

读图时保留一个条件

其他实体沿用原视频推断的潜动作；这还不是其他agent自主决定反应。

分解也有边界：共同运动的实体可能进入同一个factor。
01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

补充两种分解单位：对象与粒子

共同点是分别建模视觉变化；分开的单位和动作表示并不一样。

PlaySlot · 对象级

对象slots → 推断各对象潜动作 → 预测slots并解码视频。
动作表示：离散原型 ＋ 连续变化量。

PlaySlot Fig.2 · 对象与潜动作学习流程

LPWM · 局部粒子级

→ 随机潜动作 → 未来粒子与视频。
补充思想：通过采样表示局部运动的不确定性。

LPWM · 原方法图：局部表示与随机动态
PlaySlot的多实体证据包含简单运动对象；LPWM的粒子不保证对应完整主体。
01 动作学习MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

变化可以分组，但分组不一定是多主体

两篇背景帮助我们看清边界，不作为独立多agent控制的主要证据。

背景工作实际处理方式与本节的关系
OTF-LAM局部视觉变化 → 共享原语 → 状态条件聚合最后仍汇总成一个潜动作
GDLAM多组潜表示 ＋ 空间门控，分担变化分组不保证对应各agent
本节结论从“整场景一个动作”走向“多个变化通道”，但通道与主体的对应需要具体判断。
下一步：如果各人的动作已经给定，怎样控制到正确角色？
02 多主体控制MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

指令没变，为什么动起来的是另一个人？

与上一节不同：现在明确给定A和B各自的动作，不再从视频里发现动作。

原创任务示意 · 左右只用于说明指令归属；不是论文游戏的实际输出
需要检查A的指令是否一直作用于A，B的是否作用于B；“两个角色都动了”并不足够。
输入是历史画面、各自动作和方法所需主体信息；输出是各角色正确执行控制的未来画面。
02 多主体控制MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

ActionParty：沿着一个角色追踪它的指令

动作条件先关联主体状态，再利用空间信息影响视频中的相应角色。

原创方法简化图 · 颜色串起同一角色的动作、状态与输出；框示意位置预测
两个关联“这是谁的动作”由动作与主体状态联系起来；“他在画面哪里”由空间信息联系起来。
输出同时包括视频与主体位置。图中箭头是讲解用的条件传递，不表示把两个角色独立生成后贴回画面。
02 多主体控制MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

ActionParty：原图里的两种绑定

动作与主体、主体与空间的联系，落实在视频和状态的共同预测中。

01主体状态为每个角色保留状态表示
02动作绑定各自动作关联到各自状态
03空间绑定用位置信息影响对应区域
ActionParty Fig.2 · 视频tokens与主体状态tokens共同预测

核心流程

已知动作与当前主体状态 → 联合更新主体状态和视频 → 下一段继续保持对应。

训练含动作与坐标监督。
不是从无标签视频自动发现的真实身份。
02 多主体控制MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

看结果时，同时看动作与角色位置

同一张图中，动作标注、角色编号与预测位置需要一起读。

ActionParty Fig.6 · 不同游戏设置下的生成帧与主体位置预测

读图顺序

先选一个角色 → 找它的动作标注 → 观察后续画面与位置预测。

主要证据是什么

这些实例展示多主体控制和状态预测；不是只靠图像质量评价动作是否执行正确。

不同游戏和人数设置不等于“只用少人数训练、任意人数推理”的验证。
02 多主体控制MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

Incantation：把动作接口写成逐角色语言

补充问题：指令怎样表达，动作语义能否跨角色复用？

Incantation · 逐实体语言控制与跨实体动作示例
01输入视频历史 ＋ 各角色动作文本
02输出按角色条件继续生成视频

与 ActionParty 的分工

ActionParty突出“动作控制对人”；这里突出“动作如何表达与复用”。

逐角色语言槽使动作条件更明确；训练仍使用动作监督。
如果只给自己的动作，其他角色的下一步行为从哪里来？
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

只给自己的动作，其他人怎么办？

条件改变了：外部不再指定每个角色的下一步动作。

原创教学示意 · A向前时，B的动作从外部给定变成了需要产生的行为
共同目标输入仍有自己的动作，但输出还要包含未指定的他者行为。B可以有多种合理反应。
两条路线：把行为学进视频预测；或显式理解状态、决定动作。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBAT：B 的行为就在生成画面里

看一个简化例子：只告诉模型A向前／出拳，B的反应随视频一起产生。

原创教学示意 · 用虚构的互动场景说明COMBAT的输入与输出形式
核心机制从视频中的互动模式学习预测；生成器直接输出包含双方行为的画面。
图中“B后退”只是一个可能结果；不是必然反应，也不是额外输出的动作标签。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBAT：在预测视频时隐式产生对手行为

Tekken 3：只以 Player 1 的动作作为动作控制条件。

01输入画面历史 ＋ 己方动作
02视频预测生成下一段游戏画面
03他者行为对手的行动随画面产生
COMBAT Fig.2a · 动作条件视频模型的训练流程

关键选择

不单独预测并输出一个对手动作指令，再把它交给生成器。对手行为被包含在视觉动态中。

原文包含RGB与RGB＋pose版本。
未来帧是训练目标，对手动作没有作为策略条件输入。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBAT 的行为证据，具体测了什么？

作者使用动作数量与拳／腿比例，比较生成片段和原始游戏片段。

COMBAT Fig.5 · 生成与原始片段的拳／腿动作比例之比

怎样读这张图

横轴是训练阶段，纵轴比较动作比例。接近1表示拳／腿比例更接近原始片段。

它支持的范围

能观察行为统计是否相近；仅凭这个指标，还不能证明对手学会了通用战术决策。

另有动作总量与血量变化等评价。将RL用于目标导向策略训练仍列为后续工作。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

WorldMind：把“B 做什么”显式选出来

同样没有外部指定B的下一步；这次先选动作，再让视频模型执行。

原创教学示意 · 画面估计与动作记录共同形成状态，决策后进入视频生成闭环
为什么拆开位置从画面判断；技能是否可用从记录计算。决策选出的动作，再转换为有时间对应的生成条件。
这里的技能与后退动作是说明接口的假设例子，不对应论文中的一条实测轨迹。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

WorldMind：先决定 NPC 动作，再生成画面

把“当前是什么状态、NPC该做什么、画面如何变化”分别处理。

01理解状态画面 ＋ 技能历史
02决定动作状态 ＋ 技能机制 → NPC计划
03转换条件把动作对齐到生成时间段
04生成画面结果再回到状态理解
WorldMind Fig.3 · 理解—决策—控制—生成的闭环

两种状态来源

几何关系从生成画面估计；技能历史和冷却由动作记录计算。

决策由语言模型结合技能信息完成。几何分支训练使用引擎状态标注；推理不直接读取真实引擎几何。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

沿一个案例读完状态到行为的全过程

关注信息怎样传递，不必把整张图的术语一次记住。

WorldMind Fig.4 · 原文状态、动作选择、语言条件与生成结果案例
读图顺序看空间与技能状态 → 看所选NPC动作 → 看生成条件 → 看下一段画面。
案例帮助理解接口；是否做出合适动作，还依赖状态估计、技能描述和生成执行。
03 他者行为MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

两种行为来源，各自把什么交给模型？

COMBAT 与 WorldMind 都保留为重点，比较任务组织与信息条件。

比较项COMBATWorldMind
他者动作在哪里产生隐含在未来视频中显式决策模块先选动作
主要额外接口己方动作条件；可加入pose紧凑状态、技能机制、动作转换
重点理解视频预测能学到哪些行为模式状态理解与决策怎样接入生成
这是机制对照，不是同数据、同环境的性能排名。两者也不构成必须先后使用的系统。
知道他者可能怎样行动之后，怎样帮助自己选择合作动作？
04 合作规划MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

能预测后果之后，还要选择自己的动作

自己只看到局部场景，也不能预先知道伙伴的全部行动。

原创教学示意 · 两人准备物品：预测伙伴可能做什么，再决定自己做什么
问题所在不是替伙伴下指令，而是在不完全知道伙伴行动时，选择自己的下一步。
COMBO：将视频世界模型作为合作规划中的预测工具。
04 合作规划MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBO：让两种选择各自“发生一次”

沿同一个备物场景，区分预测未来的输出和规划选择的输出。

原创教学示意 · 对每个自身候选动作想象共同后果，再比较并选择
图里的两个输出视频模型给出两份候选后果；评价与搜索帮助A选择自己的动作。
伙伴动作是推测条件，画面也是预测。真实执行后要重新观察，不能把想象当成保证。
04 合作规划MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBO：组合多个动作条件，生成共同后果

当前视觉场景 ＋ 各主体文本动作 → 一份未来视频。

COMBO Fig.2 · 各主体动作条件的组合生成

核心做法

把联合动作拆为逐主体文本条件，组合对应的扩散预测，生成这些动作共同作用的后果。

发生在生成过程内部。
训练也包含联合动作条件阶段，不是完全免去交互数据。
输出用途这份视频接下来用于评价候选行动，帮助选择计划。
04 合作规划MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

COMBO：让视频预测进入选择循环

观察、提出候选、推测伙伴、想象结果，最后执行自己的动作。

COMBO Fig.3 · 从局部RGBD观察到世界估计、视频想象与树搜索
核心流程局部观察 → 场景估计 → 候选／伙伴动作 → 未来视频 → 评价与搜索 → 自身动作
场景估计采用投影和补全的顶视表示；生成模型是规划的一环，不负责独自完成所有决策。
04 合作规划MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

看一个规划例子：想象怎样影响选择

原文用不同搜索预算下的计划，展示想象与选择的关系。

COMBO Fig.5 · 搜索预算与计划案例

读图重点

从任务目标看：候选计划如何变化？想象的后果为什么会影响自己的选择？

要区分的两个输出

视频模型输出可能后果；规划过程输出要执行的动作。

这里无需展开低层机器人控制。我们关心的是视频预测如何被用于合作决策。
不同候选动作可以有不同未来；同一次未来的多个视角则要相容。
05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

A打开了门，B看到的门也应当打开

固定同一次行动，从两个人的视角生成两份视频。

自绘任务示意 · 不对应论文中的具体实验；不同视角应描述同一次开门事件
这里的一致不是两张画面相同，而是共同事实相容：同一扇门、同一时刻、同样的开关状态。
输入：各主体的初始观察与动作。输出：各自视角下的后续视频。多视角相容不等于所有物理过程都正确。
05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

从一扇门，扩展到共享世界的三类检查

论文检查动作、环境与互动后的状态能否跨视角对应。

ME-World Fig.2 · 动作、环境与状态更新的不一致类型

动作一致性

同一主体在各视角中的动作应当对应。

环境一致性

共同区域的布局与场景应当对应。

状态更新一致性

互动后产生的变化，应在各视角中相容。

05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

ME-World：共同生成，而非事后对答案

每一路都保留自己的视角，同时参考所有主体的动作与共同历史。

自绘方法简图 · 依据 ME-World §3；开门仅用于解释条件怎样进入生成过程
核心流程把动作表达在各个目标视角中，再结合共享环境历史，联合更新多路视频。
不意味着输出同一张图。未来身体／头部动作是给定条件，不是由该方法自主选择。
05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

共同知道“他在做什么”和“那里原本有什么”

在论文原图中，对应两条进入联合生成过程的信息来源。

ME-World Fig.3 · 联合生成、共享动作与共享记忆

所有人的动作 → 各目标视角

生成A的视角时，不仅知道A的动作，也接收B运动在这个视角中的表达。

共同历史 → 环境记忆

利用几何对齐的历史画面，为当前视角提供场景参考。

一个约束当前交互，一个帮助延续环境状态；两者不能互相替代。
05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

用同步视角检查共享事实

看两路视频中的人物、场景和互动结果，而不只看单路清晰度。

ME-World Fig.5 · 真实与合成场景的同步视角定性比较
读图建议先看 GT 与 Ours 两列，在同一时刻对齐两路视角，再观察人物与共同场景。
论文另以环境、状态更新与身份等指标评价。这里的实例限作者测试场景，不推断任意人数能力。
05 共享世界MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

补充：离屏状态与分布式生成

它们都服务于共享世界，但改变的是不同条件。

OverSeer Fig.2 · 联合生成固定辅助视角与主体视角

OverSeer：增加固定参考

例如角色转身后门暂时离开画面：额外生成固定辅助视角，帮助延续离屏变化与重访状态。

辅助相机不是额外agent或动作流；其画面也是预测。ME-World本身已有共享记忆。
WorldCast各客户端生成本地视频，交换玩家状态与场景记忆；是同一目标下的分布式实现路线。
少量参与者共享世界以后，人数增加还能保持这些能力吗？
06 数量泛化MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

训练只有两个人，使用时来了四个人

新增C、D之后，每个人仍须对应自己的动作、身份和视角。

自绘任务示意 · 2→4用于解释人数变化；下列动作与画面不是论文实验截图
希望保持A的动作仍属于A；C、D有自己的控制与输出，同时大家仍能发生交互。
人数指主动主体数，不是多加几台相机。难点先是身份和条件能否扩展，然后才是计算成本。
06 数量泛化MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

Gamma-World：给新主体身份，也给它交流通道

每个人保留不同的输入与输出，通过共享表示交换相关信息。

自绘方法简图 · 依据 Gamma-World §3；身份数量与分配结果仅为示意
两件事从预设身份池分配不同编码；主体间通过共享hub交换信息，避免全部视频tokens两两直连。
训练随机指派池内身份；测试更多主体使用更多池内身份。共享通道不是额外agent，也不是把各人的内容平均成一份。
06 数量泛化MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

回到原图：身份编码与共享hub各做什么

一个区分主体，一个组织跨主体信息交换。

Gamma-World Fig.2 · 主体编码与共享hub信息交换

身份表示

对称关系的身份编码：区分不同人，同时避免把固定槽位绑定成固定角色。

跨主体交流

在少量共享表示中汇集与传递信息，各主体仍保留自己的表示。

身份池受编码维度限制，并非无限；也不据此宣称任意主体实时加入退出。
06 数量泛化MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

人数泛化的一个实际测试：2 → 4

作者报告：两主体训练后，生成四主体的同步视角视频。

Gamma-World Fig.5 · 四主体生成例子，行对应主体，列沿时间推进
主要看什么新增主体是否有自己的视角与控制，同时仍能处在共同场景中。
这是论文测试范围内的例子。计算成本随人数变化的曲线，不等于各人数的生成质量都已验证。
06 数量泛化MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

Multics：人数多了，信息也要有组织

例如A、B一起开门，C、D一起搬箱；两件事可先各自处理，再交换相关信息。

Multics Fig.2 · 玩家—分组—全局的信息结构
01玩家保留各自信息
02分组组织相关交互
03全局交换跨组共同信息

分组会随交互调整

如果搬箱者准备穿门，两个事件变得相关，就需要交换跨组信息。此处为解释分组的自拟例子。

人数扩展既依赖可泛化的身份表示，也依赖有效的信息组织。
讨论与阅读MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

现在再把六个问题放在一起

关键区别：外部给了什么，模型需要补出什么，结果用来做什么。

问题主要已知条件需要得到什么重点阅读
动作学习视频；缺逐主体动作标签分别表示各实体动作／运动FLAM
多主体控制各人的动作或指令正确角色执行各自控制ActionParty
他者行为己方控制；他者未完全指定产生他者后续行为COMBAT / WorldMind
合作规划局部观察与合作目标借助预测选择自身动作COMBO
共享世界同一世界、各视角及动作多路视频中的共同事实相容ME-World
数量泛化训练少人，测试更多人保持身份、控制与信息交流Gamma-World
这是本次按问题组织的阅读地图，不是一套已经串接完成的技术系统。
本次讲述结构／概念示意
39 / 43
讨论与阅读MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

哪些问题值得下一轮继续深入？

先选择问题，再决定要精读、补数据，还是设计更合适的评价。

01 动作表示

学出的通道怎样对应主动agent，而不只对应被动物体或局部变化？

02 控制绑定

遮挡、接触和角色交换后，怎样让动作仍作用于正确主体？

03 他者行为

隐式生成与显式决策，各需要哪些数据和接口？

04 预测用途

哪类预测准确性真正影响合作决策，而不只是改善画面？

05 共同状态

多视角怎样记住离屏变化，并在重访时核对共同历史？

06 人数变化

哪些身份表示和通信结构能复用，哪些仍受训练规模限制？

这些是读书会讨论入口，不预先断言都是尚无人研究的空白。
本次讲述结构／概念示意
40 / 43
讨论与阅读MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

一个具体讨论：控制串人，还是合理交互？

仅作为尚未实施的验证思路，不是本次已完成的实验。

设置一：双方动作都给定

固定初态与B的动作，改变A的动作。

检查指令是否绑定正确，以及共同物体的后果是否在两视角一致。

设置二：B有自己的响应机制

只指定A的动作，让B依据当前状态产生行为。

再检查模型是否能描述这种响应，而非只重放固定动作。

关键区分固定B的动作不代表B的状态不能受接触影响；合理交互与错误的指令归属需要分开看。
选一个感兴趣的问题，把输入条件、目标能力和评价对象先说清楚。
本次讲述结构／概念示意
41 / 43
讨论与阅读MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

七篇重点阅读入口

按讲述顺序阅读；第三组两篇都作为重点。

FLAM · 2026

分别学习场景因子的潜动作

ActionParty · 2026

给定动作与主体的持续绑定

COMBAT · 2026

视频预测中隐式产生对手行为

WorldMind · 2026

状态理解、NPC决策与生成闭环

COMBO · 2024 / ICLR 2025

视频想象辅助合作规划

ME-World · 2026

多主体视角共享动作与环境历史

Gamma-World · 2026

身份表示与人数泛化

链接需要联网；本地正文与原图可离线使用。详细分析见同目录 references.html。
本次讲述结构／概念示意
42 / 43
讨论与阅读MULTI-AGENT WORLD MODELS · CAUSALVIDEOGROUP

补充工作保留各自的作用

五篇补充对照、两篇LAM背景、一篇分布式简述。

PlaySlot · 2025

对象级潜动作：原型与连续变化

LPWM · 2026

粒子级随机潜动作与未来

Incantation · 2026

逐角色语言动作与语义复用

OverSeer · 2026

固定辅助视角与离屏状态

Multics · 2026

分层通信与相关事件分组

OTF-LAM · 2026

背景：局部变化最终聚合为一个动作

GDLAM · 2026

背景：分组表示不保证等于多agent

WorldCast · 2026

简述：共享状态驱动的各端生成

本次讲述结构／概念示意
43 / 43
\ No newline at end of file diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/references.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/references.html new file mode 100644 index 0000000..26df412 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/references.html @@ -0,0 +1,175 @@ + + + + + + + Key References · Multi-Agent Problems in World Models · CausalVideoGroup + + + +
+ CausalVideoGroup + +
+
+

2026-10-26 · Yuxin Wang

Key References

Multi-Agent Problems in World Models

+

Multi-Agent Problems in World Models · Yuxin Wang · 2026-10-26

+

Literature checked through 2026-10-23. The collection contains seven core readings, five supporting comparisons, two brief latent-action backgrounds, and one distributed-generation perspective. “Read” below means the relevant original problem, method, evaluation, and limitation passages, not a claim that every appendix was read or that results were reproduced. Evidence summaries describe the authors' reported experiments.

+

The six-part organization is a synthesis for this discussion. It does not imply that the papers form an implemented pipeline. Dates refer to the initial public version where established; a later formal venue is stated separately. Titles and canonical identifiers distinguish papers from similarly named systems.

+

1. Factored Latent Action World Models — FLAM

+ +

2. ActionParty: Multi-Subject Action Binding in Generative Video Games

+ +

3. COMBAT: Conditional World Models for Behavioral Agent Training

+ +

4. WorldMind: Decoupled Game World Model for State-Aware NPC Behavior

+ +

5. COMBO: Compositional World Models for Embodied Multi-Agent Cooperation

+ +

6. Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction — ME-World

+ +

7. Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players

+ +

8. PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning

+ +

9. Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling — LPWM

+ +

10. Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models

+ +

11. OverSeer: Multi-Agent World Models Beyond What Agents See

+ +

12. Multics: Generative Multi-Player World Model with Hierarchical Communication

+ +

13. Latent Actions from Factorized Transition Effects under Agent Ambiguity — OTF-LAM

+ +

14. GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining

+ +

15. WorldCast: Distributed Multiplayer World Models

+ +

Reading connections

+

The additional readings broaden mechanisms within the six questions; they do not add separate presentation chapters. Compare assumptions before comparing outcomes: unlabelled transitions versus supplied controls, implicit behavior versus explicit decisions, candidate futures versus synchronized views, and tested population sizes versus unrestricted dynamic populations.

+

English summary and open questions · Chinese presentation · Discussion overview

+
+ + + diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/references.md b/discussions/2026-10-26-yuxin-multi-agent-world-models/references.md new file mode 100644 index 0000000..3b0133a --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/references.md @@ -0,0 +1,163 @@ +# Key References + +Multi-Agent Problems in World Models · Yuxin Wang · 2026-10-26 + +Literature checked through 2026-10-23. The collection contains seven core readings, five supporting comparisons, two brief latent-action backgrounds, and one distributed-generation perspective. “Read” below means the relevant original problem, method, evaluation, and limitation passages, not a claim that every appendix was read or that results were reproduced. Evidence summaries describe the authors' reported experiments. + +The six-part organization is a synthesis for this discussion. It does not imply that the papers form an implemented pipeline. Dates refer to the initial public version where established; a later formal venue is stated separately. Titles and canonical identifiers distinguish papers from similarly named systems. + +## 1. Factored Latent Action World Models — FLAM + +- **Source and date:** [arXiv:2602.16229](https://arxiv.org/abs/2602.16229), February 2026; [ICML 2026 proceedings](https://proceedings.mlr.press/v306/wang26gs.html). Method discussed: [v2](https://arxiv.org/html/2602.16229v2). +- **Type and role:** Core method; Part 1, multi-agent action learning. +- **Actual problem:** A single scene-level latent action mixes changes from several entities. Can different factors receive separate action channels that support selective manipulation? +- **Mechanism:** Factorize frame features into slots, infer a continuous latent action for each slot from a transition, and predict future features before recombining them. The inverse and forward models can access all current slots, preserving interaction context. +- **Evidence:** Selected-entity manipulation in MultiGrid, prediction in Procgen and nuPlan, and analyses of factor allocation. +- **Boundary:** Correlated entities may share a factor. Some predictive demonstrations infer latent actions from observed future frames; they do not establish autonomous decisions by every entity. +- **Why important / discussion question:** A direct entry into separate action channels. When do the learned factors align with the agents one wants to control? +- **Read:** v2 Sections 2–5 and the relevant control, factorization, and architecture passages of Appendix C. + +## 2. ActionParty: Multi-Subject Action Binding in Generative Video Games + +- **Source and date:** [arXiv:2604.02330](https://arxiv.org/abs/2604.02330), April 2026; method discussed: [v2](https://arxiv.org/html/2604.02330v2). +- **Type and role:** Core method; Part 2, control binding. +- **Actual problem:** Each player's action is already supplied, but a video generator may apply it to the wrong character or lose the action-to-subject association during interaction. +- **Mechanism:** Subject-state tokens connect actions and coordinates to persistent subject representations. Spatial conditioning helps route controls to the correct regions; the model predicts subject coordinates together with video. +- **Evidence:** Multi-subject control in MeltingPot, including subject-count and long-horizon checks. +- **Boundary:** Training uses action and coordinate supervision. This is not discovery of separate actions from unlabelled video; controllability also does not determine an uncontrolled character's policy. +- **Why important / discussion question:** Isolates the difference between representing an action and binding it to the right actor. How should identity binding be evaluated during occlusion or close interaction? +- **Read:** Sections 1 and 3.1–3.3, coordinate-supervision details, and the relevant subject-count and long-horizon results. + +## 3. COMBAT: Conditional World Models for Behavioral Agent Training + +- **Source and date:** [arXiv:2603.00825](https://arxiv.org/abs/2603.00825), 2026; method discussed: [v1](https://arxiv.org/html/2603.00825v1). +- **Type and role:** Core method; Part 3, implicit other-agent behavior. +- **Actual problem:** In a two-player fighting game, only Player 1's actions condition the prediction. The model must still generate what the opponent does next. +- **Mechanism:** Condition future video on visual history and Player 1's actions. Opponent behavior is generated within the predicted video, with RGB and RGB-plus-pose variants. +- **Evidence:** Tekken 3 experiments include damage trajectories, action frequencies, and human judgments of responsiveness, beyond image-quality checks. +- **Boundary:** The setting is fixed one-versus-one interaction. Behavioral plausibility is not proof of identifying arbitrary opponent policies or training a transferable standalone policy; policy learning inside the model is discussed as future work. +- **Why important / discussion question:** Shows that leaving an agent's actions unspecified changes the prediction task. Which behavioral tests separate meaningful responses from visually plausible continuation? +- **Read:** Section 3 architecture and conditioning, Section 4 behavioral evaluation, and Section 6 limitations and future work. + +## 4. WorldMind: Decoupled Game World Model for State-Aware NPC Behavior + +- **Source and date:** [arXiv:2608.21439](https://arxiv.org/abs/2608.21439), August 2026; method discussed: [v1](https://arxiv.org/html/2608.21439v1). +- **Type and role:** Core system; Part 3, explicit state and decision interfaces. +- **Actual problem:** An NPC's next action should depend on the evolving scene and game mechanics, rather than emerge only as an implicit visual continuation. +- **Mechanism:** Estimate compact state from frames and skill history; use that state and skill mechanisms to choose NPC actions; convert decisions to temporally aligned controls for video generation and feed the generated observation back. +- **Evidence:** State-estimation, mechanism-intervention, and closed-loop comparisons in the paper. +- **Boundary:** Training uses engine-state supervision for geometry; skill history and mechanics provide additional information. Not all state is recovered from pixels. Component transfer should not be presented as demonstrated generalization of the entire closed-loop system. +- **Why important / discussion question:** Complements COMBAT by making the behavioral interface inspectable. Which state variables are necessary for useful decisions? +- **Read:** Section 3 interfaces and the relevant Section 4 experiments. + +## 5. COMBO: Compositional World Models for Embodied Multi-Agent Cooperation + +- **Source and date:** [arXiv:2404.10775](https://arxiv.org/abs/2404.10775), April 2024; [ICLR 2025 proceedings](https://proceedings.iclr.cc/paper_files/paper/2025/hash/7d03c6bf9f07acb4038eea96c63db52d-Abstract-Conference.html). Method discussed: [v3](https://arxiv.org/html/2404.10775v3). +- **Type and role:** Core system; Part 4, cooperative planning. +- **Actual problem:** An agent with partial observations must anticipate partners and compare the consequences of its own actions before choosing how to cooperate. +- **Mechanism:** Build and complete a top-down visual estimate from RGBD history; propose own actions and infer partner actions; compose action-conditioned diffusion predictions; evaluate imagined outcomes with tree search; execute the selected own action. +- **Evidence:** Cooperative tasks in TDW-Game, TDW-Cook, and 2D-FetchQ, with component comparisons. +- **Boundary:** The agent does not directly command all partners. Composition occurs within diffusion prediction, not by overlaying separately generated videos, and joint-action conditioning appears in training. Planning can exploit prediction artifacts. +- **Why important / discussion question:** Connects video prediction to decisions. Which prediction failures actually change cooperative success? +- **Read:** Sections 4–5, Section 6.1, and relevant task definitions in Appendix D. + +## 6. Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction — ME-World + +- **Source and date:** [arXiv:2610.12299](https://arxiv.org/abs/2610.12299), October 2026; method discussed: [v1](https://arxiv.org/html/2610.12299v1). +- **Type and role:** Core method; Part 5, shared worlds and persistence. +- **Actual problem:** Different participants see the same interaction from different positions. Their generated videos must agree on bodies, objects, and the effects of actions. +- **Mechanism:** Jointly generate multiple egocentric streams with one diffusion transformer; project all agents' action conditions into each target view; use geometry-aligned history and reference memory. +- **Evidence:** Synthetic and real two-person interactions, with evaluations of interaction, identity, environment, and state updates. +- **Boundary:** Experiments mainly involve two agents and supplied pose controls. A general formulation for N agents does not establish arbitrary population sizes. Consistent views alone do not prove physically correct dynamics. +- **Why important / discussion question:** Provides a concrete shared-world interface. How should agreement on an event be distinguished from superficial visual similarity? +- **Read:** Sections 3.1–3.4, 4.1–4.3, and limitations. + +## 7. Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players + +- **Source and date:** [arXiv:2605.28816](https://arxiv.org/abs/2605.28816), May 2026; method discussed: [v1](https://arxiv.org/html/2605.28816v1). +- **Type and role:** Core method; Part 6, agent-count generalization. +- **Actual problem:** A model trained with few players should distinguish and coordinate more players at inference without depending on a fixed semantic meaning for each identity slot. +- **Mechanism:** Use distinguishable, symmetrically related identity encodings from a preset pool, randomly assign active identities during training, and organize cross-agent information through shared hub tokens. +- **Evidence:** The paper reports two-player training followed by four-player inference, with identity-encoding comparisons and computational scaling analyses. +- **Boundary:** The identity pool is finite. A scaling curve at a larger population is not, by itself, evidence of generation quality at that population. More-agent inference does not establish arbitrary online entry and exit. +- **Why important / discussion question:** Frames population size as a representation and information-routing problem. What transfers when the number and interaction pattern of agents change? +- **Read:** Sections 3.2–3.3, relevant Section 4 results, and limitations. + + +## 8. PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning + +- **Source and date:** [arXiv:2502.07600](https://arxiv.org/abs/2502.07600), February 2025; [ICML 2025 proceedings](https://proceedings.mlr.press/v267/villar-corrales25a.html). Relevant version: [v2](https://arxiv.org/html/2502.07600v2). +- **Role:** Supporting comparison in Part 1. +- **Problem and mechanism:** Infer controllable object changes from transitions using frozen SAVi slots. Latent actions combine a discrete prototype with continuous variation. The method includes a scene-level variant and a per-slot variant. +- **Evidence and boundary:** GridShapes tests several moving objects; robot examples largely use a single-action setting. Multiple moving shapes should not be described as multiple autonomous robots. +- **Read:** Sections 3.1–3.5 and 4.1–4.4, with relevant Appendix A/B.2 passages. +- **Comparison question:** What does object-level decomposition add when compared with one global transition code? + +## 9. Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling — LPWM + +- **Source and date:** [arXiv:2603.04553](https://arxiv.org/abs/2603.04553), March 2026; [ICLR 2026 proceedings](https://proceedings.iclr.cc/paper_files/paper/2026/hash/cf7ba4b2d14e0f6a0e8247af77745094-Abstract-Conference.html). Relevant version: [v1](https://arxiv.org/html/2603.04553v1). +- **Role:** Supporting comparison in Part 1. +- **Problem and mechanism:** Represent distinct stochastic motions locally through latent particles, per-particle action posteriors, and state-conditioned priors from which future motion can be sampled. +- **Evidence and boundary:** Video prediction and control examples include games and robot-object interaction. A particle is not necessarily a whole object, agent, or independent control command; control demonstrations are mainly single-robot settings. +- **Read:** Sections 1–2, 4, 5.1–5.2, and 6. +- **Comparison question:** How do local uncertainty and agent-specific control differ? + +## 10. Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models + +- **Source and date:** [arXiv:2605.18601](https://arxiv.org/abs/2605.18601), 2026; relevant version: [v2](https://arxiv.org/html/2605.18601v2). +- **Role:** Supporting comparison in Part 2. +- **Problem and mechanism:** Express heterogeneous character actions through per-entity structured language slots, with temporally localized text conditioning for video generation. +- **Evidence and boundary:** Multi-character game examples and cross-entity action reuse. Labels come from game animation/state information, not unsupervised action discovery. Long-term game logic uses additional event tracking and transition mechanisms. +- **Read:** Section 3.1 and action-interface results; Appendix A6 on labels and A16 on long-term logic. +- **Comparison question:** Which aspects of an action meaning transfer across bodies, and which remain tied to game mechanics? + +## 11. OverSeer: Multi-Agent World Models Beyond What Agents See + +- **Source and date:** [arXiv:2610.28554](https://arxiv.org/abs/2610.28554), October 2026; relevant version: [v1](https://arxiv.org/html/2610.28554v1). +- **Role:** Supporting comparison in Part 5. +- **Problem and mechanism:** Moving local cameras may lose common state. Jointly generate egocentric streams and fixed auxiliary views, using compressed changes relative to an anchor. +- **Evidence and boundary:** Revisit and regrouping cases in Minecraft and Doom, plus multi-camera examples. Auxiliary streams are generated at inference, not reads of ground-truth hidden state; coverage and scale remain limitations. +- **Read:** Sections 3.1–3.2, relevant Section 4 evaluation, and Appendix E. +- **Comparison question:** What can a generated auxiliary view preserve, and how can its own drift be detected? + +## 12. Multics: Generative Multi-Player World Model with Hierarchical Communication + +- **Source and date:** [arXiv:2610.17201](https://arxiv.org/abs/2610.17201), October 2026; relevant version: [v1](https://arxiv.org/html/2610.17201v1). +- **Role:** Supporting comparison in Part 6. +- **Problem and mechanism:** With several concurrent interactions, unrestricted global information exchange can mix unrelated events. Player-group-global communication and dynamic grouping organize relevant interactions. +- **Evidence and boundary:** Multiplayer control and interference checks in OpenCS2 and EgoTeam. Communication improvements do not establish persistent shared state over arbitrary durations. +- **Read:** Method overview, relevant Section 3.2–3.3 mechanisms, Sections 4.2–4.4, and Appendix D. +- **Comparison question:** Which interactions should share a local group, and when does information need global access? + +## 13. Latent Actions from Factorized Transition Effects under Agent Ambiguity — OTF-LAM + +- **Source and date:** [arXiv:2606.30544](https://arxiv.org/abs/2606.30544), June 2026; relevant version: [v2](https://arxiv.org/html/2606.30544v2). +- **Role:** Brief latent-action background in Part 1. +- **Problem and mechanism:** Reduce entanglement between observed transitions, appearance, and embodiment. Learn shared local transition primitives and aggregate them with state and position into one compact continuous latent action. +- **Evidence and boundary:** Appearance and morphology transfer, plus action decoding/control. Decomposing local effects does not produce separate actions for several agents; the final representation remains aggregated. +- **Read:** v2 problem, method, experiments, conclusion, and relevant Appendix B.1–B.3 passages. +- **Comparison question:** What is gained by decomposing effects even when the final action representation is global? + +## 14. GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining + +- **Source and date:** [arXiv:2609.26118](https://arxiv.org/abs/2609.26118), 2026; [original text](https://arxiv.org/html/2609.26118). Only the year is stated because the recorded first-release metadata is inconsistent. +- **Role:** Brief latent-action background in Part 1. +- **Problem and mechanism:** Separate mixed visual changes into latent groups through multiple stochastic bottlenecks and spatial routing, with sparsity and separation objectives. +- **Evidence and boundary:** Group interventions and swaps, robot-action representations, and downstream transfer. The learned groups need not correspond to agents or identified causal sources; separate multi-agent control is not established. +- **Read:** Grouping/routing method, experiment settings, and relevant limitation and metric passages. +- **Comparison question:** Which evidence would distinguish useful grouping from true agent-aligned actions? + +## 15. WorldCast: Distributed Multiplayer World Models + +- **Source and date:** [arXiv:2610.12412](https://arxiv.org/abs/2610.12412), October 2026; relevant version: [v1](https://arxiv.org/html/2610.12412v1). +- **Role:** Brief distributed-generation perspective within Part 5. +- **Problem and mechanism:** Let separate clients generate local views while exchanging the player state and generated scene memory needed for a common environment. +- **Evidence and boundary:** Multiplayer Counter-Strike 2 generation and long-horizon checks. Training uses position and map geometry; state estimates can drift. Block-synchronous evaluation does not demonstrate tolerance to arbitrary delays, missing packets, or disconnections. +- **Read:** Relevant system and evaluation passages in Section 5 and Appendix F. +- **Comparison question:** What shared information is sufficient for local generators to agree on the same event? + +## Reading connections + +The additional readings broaden mechanisms within the six questions; they do not add separate presentation chapters. Compare assumptions before comparing outcomes: unlabelled transitions versus supplied controls, implicit behavior versus explicit decisions, candidate futures versus synchronized views, and tested population sizes versus unrestricted dynamic populations. + +[English summary and open questions](summary.html) · [Chinese presentation](presentation.html) · [Discussion overview](./) diff --git a/discussions/2026-10-26-yuxin-multi-agent-world-models/summary.html b/discussions/2026-10-26-yuxin-multi-agent-world-models/summary.html new file mode 100644 index 0000000..2f92446 --- /dev/null +++ b/discussions/2026-10-26-yuxin-multi-agent-world-models/summary.html @@ -0,0 +1,88 @@ + + + + + + + Multi-Agent Problems in World Models · Topic Summary + + + + +
+ CausalVideoGroup + +
+
+
+

2026-10-26 · Yuxin Wang · Before meeting

+

Multi-Agent Problems in World Models

+

Separate the questions by what actions are known, what the model must infer, and how its predictions are used.

+
+
+

Scope and background

+

This reading-group survey focuses on world models connected to video generation and embodied interaction. An active agent makes choices; a manipulated object need not do so, and an additional camera is not an additional agent. Object, particle, and motion-factor representations remain useful related approaches, with these distinctions preserved.

+

A latent action model infers a compact representation of a transition from before-and-after observations, then uses that representation with the current observation to predict the next one. The representation need not recover a true command or intention. Multiple moving entities raise a further question: can their changes be represented separately?

+

Recent work provides several answers to parts of the broader problem. The papers below differ in action labels, geometry, state access, environments, and outputs; their reported results do not form a single performance ranking. The survey covers fifteen papers checked through 2026-10-23, with relevant original method and evaluation sections read. It is not a reproduction study.

+ +

Six questions and their reading route

+
+ + + + + + + + + + +
QuestionGiven informationWhat the model must provideReading entry
1. Action learningVideo transitions without per-agent action labelsSeparate latent action or motion representationsFLAM; PlaySlot, LPWM; OTF-LAM and GDLAM as background
2. Control bindingActions or instructions for named subjectsA future video in which the correct subjects execute themActionParty; Incantation
3. Other-agent behaviorHistory and externally specified actions for only part of the sceneThe behavior of agents whose actions were not suppliedCOMBAT; WorldMind
4. Cooperative planningPartial observations and candidate actionsImagined consequences that help select the acting agent's next actionCOMBO
5. Shared world and persistenceMultiple views, action conditions, and historyDifferent videos that agree on shared entities, events, and state changesME-World; OverSeer; WorldCast briefly
6. Agent-count generalizationTraining with fewer agents than may be present at inferenceUsable identities and information exchange for more agentsGamma-World; Multics
+
+ +

Mechanisms and boundaries

+

Separate representations. FLAM assigns latent action channels to learned scene factors, while the dynamics can still use the full current scene. PlaySlot adds object-slot actions with discrete prototypes and continuous variation; LPWM models stochastic particle-level transitions. OTF-LAM decomposes local effects but aggregates them into one latent action, and GDLAM learns grouped representations. Neither a factor nor a group is guaranteed to be an autonomous agent.

+

Known controls. ActionParty links each subject's action and state to spatial locations and predicts video together with subject coordinates. Incantation provides per-entity language action slots. These approaches use action supervision; they are not action-free latent-action discovery.

+

Unspecified behavior. COMBAT conditions on one player's actions and implicitly generates the opponent's behavior in video. WorldMind explicitly connects state estimation, NPC decisions, action conditions, and generation. Its state and decision interfaces use geometry supervision during training and skill information; not all state is recovered directly from pixels. Implicit behavioral generation is not evidence that a transferable policy has already been trained.

+

Choosing actions. COMBO builds a visual world estimate from partial observations, predicts partners' actions, imagines joint-action consequences, and searches over its own choices. Different candidates deliberately produce different possible futures. It does not require the planner to directly control every partner.

+

Agreeing on one world. ME-World jointly generates egocentric streams using shared action conditions and geometry-aligned history. OverSeer adds generated fixed auxiliary views for continuity beyond moving local views; these are not access to a true omniscient state. WorldCast illustrates distributed local generation coordinated through shared state and scene memory. Its reported block-synchronous evaluation is not a guarantee under arbitrary network delays.

+

More agents. Gamma-World uses symmetric identity encoding from a preset identity pool and shared hub communication, including a reported two-agent-training to four-agent-inference setting. Multics organizes information through player, group, and global levels. Testing more agents does not establish unlimited population size or arbitrary online entry and exit.

+

The sequence connects research questions rather than implemented systems: ActionParty does not consume FLAM outputs, and COMBO is not a downstream integration of COMBAT or WorldMind.

+ +

Research-quality prompts

+
+
Central claim: “Multi-agent world modeling” covers distinct tasks. Which agents' actions are given, whether outputs support decisions or multiple viewpoints, and whether agent count changes are essential to interpreting a result.
+
Hidden assumption: Identity labels, action annotations, coordinates, geometry, skill rules, and history may be available at different stages. Comparing methods without stating these information conditions can obscure the actual difficulty.
+
Failure boundary: Decomposed latents need not align with agents; controllability need not supply other agents' decisions; cross-view agreement need not imply physical correctness; population generalization remains bounded by the tested setting and representation.
+
Connection: The literature connects latent dynamics, controllable video generation, embodied planning, and persistent multi-view generation through a common concern: how actions and shared state are represented and checked across interacting entities.
+
Concrete experiment: A possible public-literature diagnostic is to hold an initial two-character scene and character B's action fixed while changing A's action, then inspect action binding and shared-object updates across two views. A separate condition lets B follow a response policy. This distinguishes misplaced controls from legitimate interaction effects. This is a discussion example; no experiment or result is reported here.
+
Discussion seed: Which information interface is the bottleneck for a given task: action representation, subject binding, other-agent decisions, shared state, or scalable communication?
+
+ +

Open questions

+
    +
  1. When do learned action factors track an active agent rather than an object, body part, or local visual change?
    2. +
  2. How can action-to-identity binding survive occlusion, close interaction, and role or position exchanges?
    3. +
  3. When is implicit video prediction sufficient for useful other-agent behavior, and when is an explicit state and decision interface needed?
    4. +
  4. Which prediction errors change cooperative action choices, rather than only the perceived quality of a generated video?
    5. +
  5. How should cross-view event agreement and off-screen persistence be tested separately from visual similarity?
    6. +
  6. Which identity and communication structures transfer to more agents, and which depend on a preset population or interaction pattern?
    7. +
+

These are discussion questions synthesized from the literature, not established novelty claims or assigned projects. See the reference analyses for the seven core readings and all eight supporting or background entries.

+
+
+ + + diff --git a/discussions/index.html b/discussions/index.html index f5f5677..f7e55f1 100644 --- a/discussions/index.html +++ b/discussions/index.html @@ -2,7 +2,8 @@
CausalVideoGroup

Weekly archive

Topic discussions

Background prepared before each meeting and technical conclusions recorded afterward, organized by date, leader, and topic.

-

2026-10-12 · Yichang Jian

Video Reasoning

A chronological walk through 26 video-reasoning papers (2025-09 to 2026-09), organized around four contested questions: who does the reasoning, what counts as correct, where the reasoning happens, and whether it can be taught.

before-meeting video-generation video-reasoning benchmark evaluation world-model

Read discussion →
+

2026-10-26 · Yuxin Wang

Multi-Agent Problems in World Models

A problem-oriented survey of multi-agent video world models: learning separate action representations, binding controls, generating other agents' behavior, planning with imagined futures, maintaining a shared world across views, and generalizing to more agents.

before-meeting world-model multi-agent video-generation latent-action embodied-intelligence

Read discussion →
+

2026-10-12 · Yichang Jian

Video Reasoning

A chronological walk through 26 video-reasoning papers (2025-09 to 2026-09), organized around four contested questions: who does the reasoning, what counts as correct, where the reasoning happens, and whether it can be taught.

before-meeting video-generation video-reasoning benchmark evaluation world-model

Read discussion →

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

A survey of video editing mechanisms and controllable changes, with method deep dives, interactive examples, and open problems in object interaction, editing history, world models, agents, and evaluation.

before-meeting video-editing action-editing world-model agents benchmark

Read discussion →

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
diff --git a/index.html b/index.html index 01ab3b0..bfb01e8 100644 --- a/index.html +++ b/index.html @@ -31,9 +31,9 @@

Video generation from a causal perspective.

Weekly reading group

Topic discussions

Each entry prepares the necessary background before the meeting, then records the assumptions, failure cases, and experiments identified during discussion.

-

2026-10-12 · Yichang Jian

Video Reasoning

A chronological walk through 26 video-reasoning papers (2025-09 to 2026-09), organized around four contested questions: who does the reasoning, what counts as correct, where the reasoning happens, and whether it can be taught.

before-meeting video-generation video-reasoning benchmark evaluation world-model

Read discussion →
-

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

A survey of video editing mechanisms and controllable changes, with method deep dives, interactive examples, and open problems in object interaction, editing history, world models, agents, and evaluation.

before-meeting video-editing action-editing world-model agents benchmark

Read discussion →
-

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
+

2026-10-26 · Yuxin Wang

Multi-Agent Problems in World Models

A problem-oriented survey of multi-agent video world models: learning separate action representations, binding controls, generating other agents' behavior, planning with imagined futures, maintaining a shared world across views, and generalizing to more agents.

before-meeting world-model multi-agent video-generation latent-action embodied-intelligence

Read discussion →
+

2026-10-12 · Yichang Jian

Video Reasoning

A chronological walk through 26 video-reasoning papers (2025-09 to 2026-09), organized around four contested questions: who does the reasoning, what counts as correct, where the reasoning happens, and whether it can be taught.

before-meeting video-generation video-reasoning benchmark evaluation world-model

Read discussion →
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

A survey of video editing mechanisms and controllable changes, with method deep dives, interactive examples, and open problems in object interaction, editing history, world models, agents, and evaluation.

before-meeting video-editing action-editing world-model agents benchmark

Read discussion →
Meeting guidelinesHow to contribute
diff --git a/people/index.html b/people/index.html index db6d04a..d5a3a97 100644 --- a/people/index.html +++ b/people/index.html @@ -5,7 +5,7 @@

Group Leader · guangyi

Guangyi Chen

Email

Discussion Leaders

- +
diff --git a/sitemap.xml b/sitemap.xml index f9061ff..653710d 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -10,12 +10,16 @@ https://causalvideogroup.github.io/tags/ https://causalvideogroup.github.io/projects/dog-bone-showcase/ https://causalvideogroup.github.io/projects/video-action-editing/ + https://causalvideogroup.github.io/discussions/2026-10-26-yuxin-multi-agent-world-models/ https://causalvideogroup.github.io/discussions/2026-10-12-yichang-video-reasoning/ https://causalvideogroup.github.io/discussions/2026-09-28-peiyuan-video-editing-frontiers/ https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/ https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/ https://causalvideogroup.github.io/discussions/2026-07-20-jinyuan-multi-shot-video-generation/ https://causalvideogroup.github.io/discussions/2026-07-13-yifan-forcing-ar-video-distillation/ + https://causalvideogroup.github.io/discussions/2026-10-26-yuxin-multi-agent-world-models/summary.html + https://causalvideogroup.github.io/discussions/2026-10-26-yuxin-multi-agent-world-models/references.html + https://causalvideogroup.github.io/discussions/2026-10-26-yuxin-multi-agent-world-models/meeting-note.html https://causalvideogroup.github.io/discussions/2026-10-12-yichang-video-reasoning/summary.html https://causalvideogroup.github.io/discussions/2026-10-12-yichang-video-reasoning/references.html https://causalvideogroup.github.io/discussions/2026-10-12-yichang-video-reasoning/meeting-note.html diff --git a/tags/index.html b/tags/index.html index 53b33c5..ffa5fb0 100644 --- a/tags/index.html +++ b/tags/index.html @@ -10,16 +10,19 @@

1 discussion(s) · 0 project(s)

causal-generation

1 discussion(s) · 0 project(s)

controllable-generation

0 discussion(s) · 1 project(s)

disentanglement

+

1 discussion(s) · 0 project(s)

embodied-intelligence

3 discussion(s) · 0 project(s)

evaluation

+

1 discussion(s) · 0 project(s)

latent-action

1 discussion(s) · 0 project(s)

long-video-generation

+

1 discussion(s) · 0 project(s)

multi-agent

1 discussion(s) · 0 project(s)

multi-shot-video

1 discussion(s) · 0 project(s)

physics-consistency

0 discussion(s) · 1 project(s)

showcase

1 discussion(s) · 0 project(s)

streaming-generation

1 discussion(s) · 0 project(s)

video-distillation

1 discussion(s) · 2 project(s)

video-editing

-

5 discussion(s) · 0 project(s)

video-generation

+

6 discussion(s) · 0 project(s)

video-generation

1 discussion(s) · 0 project(s)

video-reasoning

-

4 discussion(s) · 0 project(s)

world-model

+

5 discussion(s) · 0 project(s)

world-model