测试 Skill:让能力可验证

三层测试体系

第一层:脚本单元测试

scripts/ 就是普通 Python/Node 代码,用常规单测框架。

# tests/test_detect_fields.py
import json, subprocess
from pathlib import Path

def test_detect_simple_form():
    pdf = Path("tests/fixtures/simple-form.pdf")
    r = subprocess.run(
        ["python", "scripts/detect_fields.py", str(pdf)],
        capture_output=True, text=True, check=True,
    )
    fields = json.loads(r.stdout)
    assert len(fields) == 5
    assert {"name": "FirstName", "type": "/Tx", "value": ""} in fields

Fixtures(测试用的 PDF、CSV 等)放 tests/fixtures/,跟 Skill 一起提交。

第二层:激活测试

目的:让 Claude "对什么请求激活本 Skill,对什么不激活"可预测。

# tests/test_activation.py
from anthropic import Anthropic

SKILL_ID = "skill_abc123"  # 预上传
client = Anthropic()

POSITIVE = [
    "帮我填 invoice.pdf",
    "fill the form in report.pdf with this data",
    "把这些字段签到合同 PDF 里",
]
NEGATIVE = [
    "帮我把 PDF 转成 Word",
    "解释一下什么是 PDF",
    "扫描这张发票",
]

def activated(prompt: str) -> bool:
    r = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
        skills=[{"id": SKILL_ID}],
        tools=[{"type": "bash_20250124", "name": "bash"}],
    )
    # 启发式:看 Claude 是否 read 了 SKILL.md 或提到了 Skill 名
    text = str(r.content).lower()
    return "pdf-form-filler" in text or "skill.md" in text

def test_activation():
    for p in POSITIVE:
        assert activated(p), f"should activate: {p}"
    for p in NEGATIVE:
        assert not activated(p), f"should NOT activate: {p}"

第三层:Eval 场景测试

端到端跑一遍完整任务,用 golden 输出做比对。

# tests/eval/cases.yaml
- name: basic_form
  prompt: |
    填一下 tests/fixtures/expense.pdf,数据:
    - 姓名: 张三
    - 金额: 1200
    - 日期: 2026-04-15
  expect:
    must_call_script: scripts/fill.py
    output_contains:
      - "expense_filled.pdf"
    fields_filled:
      FirstName: "张三"
      Amount: "1200.00"   # 注意 Skill 要求小数位

- name: edge_signature_field
  prompt: "填 tests/fixtures/contract.pdf,包括签名"
  expect:
    must_mention: "signature-guide.md"   # 必须读边界指南
    does_not: "直接填充签字域"

# tests/eval/runner.py — 简版
import yaml, json
from anthropic import Anthropic

def run_case(case):
    # 调用 Claude API,拿到 tool_use 列表和最终文本
    r = call_claude(case["prompt"])
    return {
        "scripts_called": [t.input["command"] for t in r.tool_uses if t.name == "bash"],
        "final_text": r.final_text,
    }

def check(case, result):
    exp = case["expect"]
    if (s := exp.get("must_call_script")):
        assert any(s in c for c in result["scripts_called"])
    for s in exp.get("output_contains", []):
        assert s in result["final_text"]

Few-shot 纠偏:在 SKILL.md 里教 Claude

测试跑出来 Claude 总在"金额格式"上栽跟头?最直接的纠偏不是改脚本,是在 SKILL.md 里加 few-shot。

## 示例

### ✅ 正确

用户:"填 1200 元"
Claude:
- 读到 Amount 字段
- 依照 references/amount-format.md,格式化为 "1200.00"
- 填入

### ❌ 错误

用户:"填 1200 元"
Claude:直接填 "1200"(缺小数位)

few-shot 比规则更有效,Claude 遇到相似场景会模仿正例。

评估指标

CI 集成

# .github/workflows/skill-ci.yml
name: Skill CI
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Lint SKILL.md
        run: python scripts/lint_skill.py SKILL.md
      - name: Unit test scripts
        run: pytest tests/ -q
      - name: Upload skill to Anthropic (staging)
        run: python scripts/upload_skill.py --env staging
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_STAGING_KEY }}
      - name: Run eval cases
        run: python tests/eval/runner.py
      - name: Gate on metrics
        run: python scripts/check_thresholds.py

lint_skill.py 检查 frontmatter 必填字段、description 长度、相对路径有效性;upload 失败不 merge;eval 不达标拒绝发版。

灰度发布

Skill 改了不要直接替换生产的 skill_id:

1. 新 Skill 上传获得 skill_v2_xyz
2. 线上 10% 流量改用 v2,其余继续用 v1
3. 观测关键指标(激活率、产出达标率)不降
4. 1-3 天后全量,老 skill_id 保留 1 周再删

用户反馈回流

生产环境里,让 Claude 在任务末尾问一句"产出是否满足要求?"——把反馈记入日志,每周汇总添加到 eval 场景集。

本章小结