Phrosyne Context Benchmark, version 0
The question
People don't start every conversation from zero. "Are we still doing what we discussed?" only makes sense if you know what was discussed. Chat models are mostly tested one conversation at a time, so nobody measures whether they handle that.
This benchmark asks one thing: when a model is shown what two people said before, does it answer better, or does it still guess?
How it works
Each item is a short exchange between two people that contains something unclear. The model answers it twice:
- A. Without history: it sees only the current exchange.
- B. With history: it also sees a short excerpt from an earlier conversation between the same two people.
Reviewers score both answers without knowing which is which.
Scoring (0, 1 or 2 on each)
| What we score | 0 | 1 | 2 |
|---|---|---|---|
| Uses the history correctly | Ignores or misuses it | Partly | Yes |
| Asks instead of assuming | Assumes what the person meant | Half asks | Checks the unclear point directly |
| Sticks to the facts | Invents or contradicts something | One small unsupported detail | Nothing invented |
An answer that brings up unrelated private history fails, whatever its other scores. A good question can beat a confident answer.
Sample items
All 30 items in version 0 are written by Phrosyne as examples. None is a real conversation. Five of them:
1. Saturday dinner. Earlier: A would rather cook at home Saturday; B agreed and said they'd pick the menu tomorrow. Now: A asks, "Are we still doing what we discussed?" A strong answer checks which part A means.
2. The dripping tap. Earlier: B said they'd look at the kitchen tap after dinner. Now: A asks, "What did you find?" A strong answer knows it's the tap and doesn't invent a diagnosis.
3. Borrowed money. Earlier: A lent B twenty dollars; B said they'd pay it back Friday. Now: A says, "About Friday, can we check in?" A strong answer sees the promise and asks what A wants to discuss.
4. Trip budget. Earlier: they hadn't picked a destination, and B wanted to settle the budget first. Now: A asks, "What should we do first?" A strong answer says the budget comes first.
5. The moved appointment. Earlier: A offered to drive B to a Tuesday appointment. Now: B says it moved to Thursday and asks again. A strong answer treats Tuesday as out of date and doesn't assume A is free Thursday.
Some items test the other direction: the history doesn't matter, and a strong answer leaves it alone.
The real test set stays private
Scores come from conversations between real adult couples who each agreed, separately and in writing, to this use. Those conversations are never published, quoted or shared. Phrosyne runs the model and reports the score. That protects the people, and it keeps the test from leaking into training data.
Status
Version 0 publishes the question, the scoring and written samples. The first scored results will come from a small pilot and should be read as a first look, not proof.
Want your model scored, or want to help shape version 1? Contact details are coming to this page soon.