Evals:给 Agent 打分,代替"肉眼测试"

为什么需要 Evals

• 换模型:从 4o-mini 切到 Haiku,怎么确认没退化?
• 改 prompt:加一句"要用中文",是不是牵一发动全身?
• 加 tool:新工具是不是反而让 Agent 乱调?
• 升级框架版本:底层行为变了没?

人眼抽查十条可能没事,一个月后发现某场景全线塌方。Eval 让每次改动都有客观数据支撑。

Metric 分类

挂 Metric 到 Agent

import { Agent } from '@mastra/core/agent';
import {
  AnswerRelevancyMetric,
  ToxicityMetric,
  FaithfulnessMetric,
} from '@mastra/evals/llm';
import { openai } from '@ai-sdk/openai';

const judge = openai('gpt-4o-mini');

export const qaAgent = new Agent({
  name: 'qa',
  instructions: '忠于文档回答问题,不会就说不知道',
  model: openai('gpt-4o-mini'),
  evals: {
    relevancy: new AnswerRelevancyMetric(judge),
    toxicity:  new ToxicityMetric(judge),
    faithful:  new FaithfulnessMetric(judge),
  },
});

挂上后,Playground 每条对话下方自动出现评分卡片;也可以程序里直接取:

const { text, evals } = await qaAgent.generate('如何部署到 Cloudflare?');
console.log(evals.relevancy.score);  // 0.0 ~ 1.0
console.log(evals.relevancy.info);   // 解释

跑测试集

// evals.test.ts(Vitest)
import { describe, it, expect } from 'vitest';
import { qaAgent } from './agents';
import { AnswerRelevancyMetric } from '@mastra/evals/llm';

const cases = [
  { q: 'Cloudflare 怎么部署?', mustInclude: ['wrangler', 'mastra deploy'] },
  { q: '怎么开启 memory?', mustInclude: ['Memory', 'storage'] },
];

describe('qaAgent', () => {
  const metric = new AnswerRelevancyMetric(openai('gpt-4o-mini'));

  cases.forEach(({ q, mustInclude }) => {
    it(q, async () => {
      const { text } = await qaAgent.generate(q);
      const { score } = await metric.measure(q, text);
      expect(score).toBeGreaterThan(0.7);
      mustInclude.forEach(kw => expect(text).toContain(kw));
    });
  });
});

CI 里跑

# .github/workflows/eval.yml
name: Mastra Evals
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - run: pnpm install
      - run: pnpm test:eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

自定义 Metric

import { Metric, type MetricResult } from '@mastra/core/eval';

export class ChineseOnlyMetric extends Metric {
  async measure(input: string, output: string): Promise<MetricResult> {
    const cnChars = output.match(/[一-龥]/g)?.length ?? 0;
    const total = output.replace(/\s/g, '').length;
    const ratio = total ? cnChars / total : 0;
    return {
      score: ratio,
      info: { reason: `中文占比 ${(ratio*100).toFixed(1)}%` },
    };
  }
}

RAG 三指标

import {
  FaithfulnessMetric,
  ContextualRecallMetric,
  ContextualPrecisionMetric,
} from '@mastra/evals/llm';

const faith = new FaithfulnessMetric(judge, { context: retrievedChunks });
const r = await faith.measure(userQ, agentAnswer);

Eval 面板

Playground 的 Eval 面板可以:

• 导入 CSV/JSON 测试集
• 一键跑当前 Agent 所有 metric
• 得分直方图 + 排序找 worst case
• 对比两次运行的 diff(调 prompt 后的回归对比)

常见坑

本章小结