> For the complete documentation index, see [llms.txt](https://mixedvoices.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mixedvoices.gitbook.io/docs/getting-started/agent-evaluation.md).

# Agent Evaluation

MixedVoices provides the ability to evaluate agents through simulated text-to-text conversations. This allows you to test your agent's behavior across different scenarios before deploying to production.

### Quick Start

```python
import mixedvoices as mv
from mixedvoices.metrics import empathy, Metric

# Create project with metrics
hangup_metric = Metric(
    name="call_hangup",
    definition="FAILS if the bot faces problems in ending the call",
    scoring="binary"
)
project = mv.create_project("dental_clinic", metrics=[empathy, hangup_metric])

# Create version
v1 = project.create_version(
    "v1", 
    prompt="You are a friendly dental receptionist...",
    metadata={"model": "gpt-4", "deployment_date": "2024-01-15"}
)

# Generate test cases
test_generator = mv.TestCaseGenerator(v1.prompt)
test_cases = test_generator.add_from_transcripts([existing_conversation])
                          .add_edge_cases(2)
                          .add_from_descriptions(["An elderly patient", "A rushed parent"])
                          .generate()

# Create and run evaluator
evaluator = project.create_evaluator(test_cases, metric_names=["empathy", "call_hangup"])
evaluator.run(v1, MyAgent, agent_starts=False)
```

### Test Case Generation

The `TestCaseGenerator` class provides multiple methods to create diverse test cases:

```python
test_generator = mv.TestCaseGenerator(agent_prompt)

# Add from existing conversations
test_generator.add_from_transcripts([transcript1, transcript2])

# Add from audio recordings
test_generator.add_from_recordings(["call1.wav", "call2.wav"], user_channel="left")

# Add edge cases
test_generator.add_edge_cases(count=2)

# Add from user descriptions
test_generator.add_from_descriptions([
    "An old lady from India with hearing difficulties",
    "A young man from New York in a hurry"
])

# Add from existing project/version paths
test_generator.add_from_project(project)
test_generator.add_from_version(version)

# Generate all test cases
test_cases = test_generator.generate()
```

### Implementing Your Agent

Create a class that inherits from `BaseAgent`:

```python
from typing import Tuple
import mixedvoices as mv

class MyDentalAgent(mv.BaseAgent):
    def __init__(self, model="gpt-4", temperature=0.7):
        self.agent = YourAgentImplementation(
            model=model,
            temperature=temperature
        )

    def respond(self, input_text: str) -> Tuple[str, bool]:
        """Generate agent response for each conversation turn.
        
        Args:
            input_text (str): User input (empty string if agent starts)
            
        Returns:
            Tuple[str, bool]: (response_text, has_conversation_ended)
        """
        response = self.agent.get_response(input_text)
        has_ended = check_conversation_ended(response)
        return response, has_ended
```

### Running Evaluations

```python
# Create evaluator with specific metrics
evaluator = project.create_evaluator(
    test_cases,
    metric_names=["empathy", "call_hangup"]  # or omit for all project metrics
)

# Run evaluation
eval_run = evaluator.run(
    version=v1,
    agent_class=MyDentalAgent,
    agent_starts=False,  # False: user starts, True: agent starts, None: random
    verbose=True,        # Print conversations and scores
    model="gpt-4",      # Additional args passed to agent
    temperature=0.7
)

# Access results
results = eval_run.results  # List of results per test case
status = eval_run.status()  # IN PROGRESS/COMPLETED/FAILED/INTERRUPTED
```

### Best Practices

1. **Test Case Diversity**
   * Mix different generation methods
   * Include edge cases and failure scenarios
   * Use real conversation transcripts when available
2. **Agent Implementation**
   * Handle empty input for conversation starts
   * Implement clear conversation ending logic
   * Return accurate conversation end status
3. **Metrics Selection**
   * Choose metrics relevant to your use case
   * Balance binary and continuous metrics
   * Consider including custom metrics
4. **Evaluation Strategy**
   * Run evaluations before deployment
   * Test specific conversation paths
   * Validate fixes for known issues
   * Monitor evaluation runs for completion