🧠 Multi-Agent Debate Strategies to Enhance Requirements Engineering with Large Language Models

This repository is a research-focused project for creating AI assistants based on Large Language Models (LLMs) that engage in structured discussions to improve the performance of requirements engineering tasks, such as requirements classification.

---

🚀 Project Structure

llm-discussion/
│
├── code/               # Python modules for assistant creation, interaction, and evaluation
│   ├── llm/            # Logic for interacting with OpenAI API
│   ├── evaluation/     # Scripts for generating baselines and computing metrics
│   └── ...
│
├── data/               # All data-related resources
│   ├── assistants/     # Metadata and files related to created assistants
│   ├── config/         # Configuration files defining assistant roles and coordination logic
│   ├── prompts/        # System and user prompts for various assistant roles
│   ├── datasets/       # Input CSV datasets (e.g., PROMISE dataset)
│   ├── output/         # Output generated by assistants (raw responses, predictions)
│   └── evaluation/     # Evaluation metrics, confusion matrices, and summaries
│
└── requirements.txt    # Python dependencies

---

⚙️ Setup

1. Clone the repository

git clone https://github.com/nlp4se/llm-discussion.git
cd llm-discussion

2. Add your API keys in a .env file

OPENAI_API_KEY=your_openai_api_key_here
# Add other keys if needed

3. Install dependencies

pip install -r requirements.txt

---

🔍 Requirements Classification (Binary) - Functional and non-functional debaters

This pattern defines three assistants:

• Functional (F) debater: argues why the given requirement is functional.
• Non-Functional (NF) debater: argues why the given requirement is non-functional.
• Judge: analyses the response from the debaters and leads to a classification decision (F/NF).

🪄 Step 1: Create the binary classification assistants

python -m code.llm.openai.create_assistant_openai --assistant_name non-functional-debater --prompt_file_path ./data/prompts/discussant/non-functional-debater-prompt.txt

python -m code.llm.openai.create_assistant_openai --assistant_name functional-debater --prompt_file_path ./data/prompts/discussant/functional-debater-prompt.txt

python -m code.llm.openai.create_assistant_openai --assistant_name judge-debater --prompt_file_path ./data/prompts/coordinator/judge-prompt.txt --schema_file_path ./data/prompts/coordinator/requirements-schema-binary.json

---

✏️ Step 2: Test the assistant on a sample prompt

python -m code.llm.openai.ask_assistant_openai --assistant_name functional-debater --prompt_file ./data/prompts/discussant/requirement-example-prompt.txt

---

🧪 Step 3: Generate ground-truth baseline from dataset

python -m code.evaluation.baseline --input_path ./data/datasets/PROMISE/nfr.csv --output_folder ./data/output/baseline/binary/  --batch_size 10  --assistant_name persona_classifier_binary  --model gpt-4o

---

📊 Step 4: Evaluate classification performance for baseline

python -m code.evaluation.metrics --input ./data/output/baseline/binary/nfr.csv --output ./data/evaluation/binary

---

🔢 Step 5: Classification with discussion

Run classification using multi-agent debate strategy:

python -m code.llm.coordinator --input ./data/datasets/PROMISE/nfr.csv --output ./data/output/judge/binary/nfr.csv --assistants-file ./data/config/assistants.json --judge-file ./data/config/judge.json --temperature 0 --iterations 1 &

Explanation of parameters:

• --input: Input CSV file path
• --output: Output CSV file path
• --assistants-file: JSON file with assistant configs
• --judge-file: JSON file with judge config
• --temperature: Temperature for generation (default: 0)
• --n: Number of requirements to sample per class (optional)
• --iterations: Number of debate iterations between the debaters (default: 0)

---

📊 Step 6: Evaluate classification performance for debate results

python -m code.evaluation.metrics --input ./data/output/judge/binary/nfr.csv  --output ./data/evaluation/judge/binary

---

📄 License

This project is licensed under the GNU General Public License v3.0.

You are free to use, modify, and distribute this software under the terms of the GPLv3 License.