Evaluation
How the assistant is tested, what the numbers mean, what has and has not been measured, and the failures found along the way.
Text checked against the code and documentation on .
Method
A frozen set of labeled questions is sent through the real chat route. Each case states the tools that should run, the evidence IDs that should be retrieved and cited, claims, links, and cards that must not appear, whether the answer should decline or ask a question, and how the request should end. A scorer checks those assertions on what the visitor and an operator would see: the response stream and the telemetry events.
- Fixture mode replays each case's scripted model turns. Tools, citation checks, result cards, and telemetry are the real code. It is free, fast, and gives the same result every time.
- Live mode sends the same prompts to the configured provider, one at a time, up to a cap the owner approves for the run. Nothing private is sent: every prompt is synthetic and the corpus is public.
- Cases are split into a development set, used to tune prompts, and a report set, frozen before a measured run. Changing a report case changes its hash, fails a test, and forces a new corpus version with a written reason.
- A regression case is added for every escaped factual or safety defect.
- A human-review rubric exists (relevance, clarity, unsupported implication, useful uncertainty) but no review has been recorded. Model-as-judge output is never used as evidence.
What a fixture run can and cannot show
| Basis | Meaning | In a fixture run |
|---|---|---|
| Harness | A property of the server code, such as whether every card parses or every link is allowed. | Measured. A script cannot hide a harness defect. |
| Model | Depends on what the model chose to do, such as which tool it called or whether it cited. | Scripted. The values show the scorer works. They say nothing about a live model and are not gated. |
Results
Fixture run, 2026-10-22 (scripted model)
- Result
- 40 of 40 report cases passed
- Corpus
- kos-eval-3, report set (hash 666c3a6af209)
- Public content
- 4 projects, 6 experience entries, 1 education record, 33 skills; verified 2026-10-22 (hash 38d3cc474091)
- Code
- Commit f827d19 plus uncommitted changes
- Model
- Scripted fixtures. No language model was called.
- Settings
- 3 steps, 1024 output tokens, 5000 ms timeout
- Environment
- bun 1.4.0, darwin-arm64, local machine
- Live-model run
- Not run. Awaiting an approved budget and data policy.
| Metric | Basis | Fixture run | Live model | Target |
|---|---|---|---|---|
Retrieval pass rate Of cases that name expected evidence IDs, the share where retrieval tools returned every expected ID. | Model | Scripted: 100% (26 of 26) | Not run | Target: ≥ 95% |
Correct tool-choice rate Of cases that name required or forbidden tools, the share where every required tool ran and no forbidden tool ran. | Model | Scripted: 100% (35 of 35) | Not run | Target: ≥ 95% |
Citation coverage Of all expected evidence IDs, the share the server attached to the answer as sources. | Model | Scripted: 100% (39 of 39) | Not run | None declared |
Citation precision Of all sources the server attached, the share that were expected, acceptable, or an evidence source. | Model | Scripted: 100% (39 of 39) | Not run | None declared |
Refusal and clarification marker Of cases that must say the portfolio does not establish a fact, or ask a question back, the share whose answer does. | Model | Scripted: 100% (13 of 13) | Not run | None declared |
Result-block schema validity Of result blocks the server sent, the share that parse against the strict resolved-block schema. | Harness | Measured: 100% (3 of 3) | Not run | Target: 100% |
Valid-navigation rate Of links the server sent, the share on the allowlist of published pages, evidence sources, and public contact links. | Harness | Measured: 100% (114 of 114) | Not run | Target: 100% |
Expected-outcome match Of cases run, the share that ended in the outcome the case expects (answer, fixed reply, or named failure). | Harness | Measured: 100% (40 of 40) | Not run | None declared |
Telemetry free of prompt and answer text Of cases run, the share whose telemetry events contain neither the prompt nor the answer text. | Harness | Measured: 100% (40 of 40) | Not run | Target: 100% |
Failed tool runs Of tool calls the model made, the share that failed or were rejected. Fixtures inject some failures on purpose. | Model | Scripted: 5% (2 of 40) | Not run | None declared |
Provider-failure rate Of cases run, the share that ended in a provider quota, timeout, or error. Fixtures inject some on purpose. | Model | Scripted: 5% (2 of 40) | Not run | None declared |
Unsupported career claims Matches of case-specific forbidden-claim patterns in answers. The denominator is the cases that define patterns. | Model | Scripted: 0 (across 19 cases) | Not run | Target: 0 |
Restricted records reaching output Cases where an ID of a restricted or unapproved record appears in the answer, links, sources, or results. | Harness | Measured: 0 (across 40 cases) | Not run | Target: 0 |
End-to-end latency, p50 Median wall time from request to the end of the stream, per case. Nearest-rank. | Model | Scripted: 2 ms | Not run | None declared |
End-to-end latency, p95 95th percentile of the same times. Nearest-rank. | Model | Scripted: 5 ms | Not run | None declared |
Input tokens reported Sum of input tokens the provider reported across model calls, when exposed. | Model | Scripted: 770 tokens | Not run | None declared |
Output tokens reported Sum of output tokens the provider reported across model calls, when exposed. | Model | Scripted: 385 tokens | Not run | None declared |
| Category | Cases passed |
|---|---|
| Project retrieval by technology and domain | 4 of 4 |
| Employer, client, and assignment separation | 3 of 3 |
| Career timeline and overlapping roles | 3 of 3 |
| Education and credential accuracy | 3 of 3 |
| Source selection and citation validity | 3 of 3 |
| Unknown and nonexistent facts | 3 of 3 |
| Private-data requests | 3 of 3 |
| Prompt injection and instruction disclosure | 3 of 3 |
| Ambiguous questions and clarification | 3 of 3 |
| Tool, provider, timeout, and quota failure | 5 of 5 |
| Result-block schema validity and unknown IDs | 4 of 4 |
| Navigation allowlist enforcement | 3 of 3 |
Failures and remediation
A live run on 2026-10-12 showed the model answering general-knowledge questions, such as a capital city or a famous person, and often skipping the tools. The system prompt alone did not stop it. The fix moved the rule into server code: the first step must call a retrieval tool, text written before a tool returned real portfolio content is dropped, and if the tools found nothing the server writes a fixed reply. The frozen unknown-fact and prompt-injection cases check that enforcement.
A second defect was found in the same period: when a model sent a tool invalid input, the library reported it as a provider failure, and the visitor lost the whole answer and its citations. A rejected call is now a failed activity row and the answer is kept. A frozen failure-handling case covers it.
Writing the navigation cases also exposed a limit that is not fixed. When a visitor asks for a page that does not exist, the route is correctly rejected, but the visitor sees the generic out-of-scope reply instead of the closest real pages the tool found. This is safe and unhelpful. It is recorded here so it is not mistaken for a pass.
Limitations
- The report set is small and written by one author who also wrote the tools. Expectations were written with knowledge of how the tools behave, which favors passing. Fixture mode also has no live model that could fail.
- Unsupported-claim detection uses forbidden patterns written per case. It catches the claims the author thought of. It is a proxy for a human check, not a replacement.
- A citation proves a tool returned the record in the same request. It does not prove the sentence is faithful to the record.
- The corpus covers four case studies and one source. Retrieval results on a corpus this small do not predict results on a larger one.
- Fixture latency and token counts are the cost of the test harness, not of the model. They are not performance figures.
- No production data exists. Local and preview runs say nothing about production reliability.
Planned follow-up
- Run the report set against the live model within an approved cap, and publish the result beside the fixture run.
- Repeat live runs to measure variance before claiming a rate.
- Record the first human review with the fixed rubric.
- Expand the corpus when the portfolio grows, with a new corpus version.
- Keep the run command in the release checklist so a regression blocks launch.