CD Context Delta

Packet-quality eval

Measure whether the packet had the context the task actually needed.

Token savings are a cost signal. Eval scores are the quality signal: useful-context density, impact coverage, omission rate, forbidden inclusions, and packet size.

Run the demo eval.

Use the bundled demo workspace and labeled gold set.

npm run eval -- --workspace examples/demo-workspace --cases ../eval/gold-set.json

Use the release-gated multi-shape suite before changing ranking or presets.

npm run eval:multi

Reports are written locally under .contextdelta/eval/.

Label the minimum sufficient packet.

Each case names required paths, acceptable alternates, forbidden paths, and pass criteria.

{
  "id": "auth-refresh-rotation",
  "task": "Add refresh token rotation for admin users",
  "minimum_context": {
    "paths": ["specs/004-admin-auth/spec.md", "src/auth/service.ts"],
    "acceptable_paths": ["specs/004-admin-auth/tasks.md"]
  },
  "forbidden_paths": ["docs/legacy-oauth.md"]
}

Scores that survive skeptical review.

  • Useful-context density: useful paths divided by included paths.
  • Impact coverage: required paths included divided by required paths.
  • Omission rate: required paths missed divided by required paths.
  • Forbidden included: paths the packet should have avoided.
  • Packet size: estimated packed tokens.

Use eval before correctness claims.

Public copy should say Context Delta makes context visible, replayable, and measurable. Narrow "better context" claims should wait for labeled eval results. Token reduction is a cost signal; eval quality is the correctness signal.