# Bench run: qwen3_4b-instruct-2507-q4_K_M · pipeline v4 · OCR vision · heldout split

26 screenshots, 9 questions. Machine: macOS-27.0.1-arm64-arm-64bit-Mach-O.

## Results by question type

| System | Type | n | Top-1 | Top-3 | Answer correct | Confident wrong | Correct abstain |
|---|---|---|---|---|---|---|---|
| B1_ocr_keyword | keyword | 2 | 100% | 100% | – | – | – |
| B1_ocr_keyword | paraphrase | 3 | 100% | 100% | – | – | – |
| B1_ocr_keyword | field | 2 | 50% | 50% | – | – | – |
| B1_ocr_keyword | recency | 1 | 100% | 100% | – | – | – |
| B1_ocr_keyword | unanswerable | 1 | – | – | – | – | – |
| B1_ocr_keyword | ALL | 9 | 88% | 88% | – | – | – |
| B2_plus_cards | keyword | 2 | 100% | 100% | – | – | – |
| B2_plus_cards | paraphrase | 3 | 100% | 100% | – | – | – |
| B2_plus_cards | field | 2 | 50% | 100% | – | – | – |
| B2_plus_cards | recency | 1 | 0% | 100% | – | – | – |
| B2_plus_cards | unanswerable | 1 | – | – | – | – | – |
| B2_plus_cards | ALL | 9 | 75% | 100% | – | – | – |
| B3_full_ai | keyword | 2 | 100% | 100% | 100% | 0% | – |
| B3_full_ai | paraphrase | 3 | 100% | 100% | 100% | 0% | – |
| B3_full_ai | field | 2 | 100% | 100% | 100% | 0% | – |
| B3_full_ai | recency | 1 | 100% | 100% | 0% | 100% | – |
| B3_full_ai | unanswerable | 1 | – | – | – | 0% | 100% |
| B3_full_ai | ALL | 9 | 100% | 100% | 88% | 11% | 100% |

## Latency per question (this machine)

| System | p50 | p95 |
|---|---|---|
| B1_ocr_keyword | 0 ms | 0 ms |
| B2_plus_cards | 0 ms | 0 ms |
| B3_full_ai | 975 ms | 2808 ms |

## Failures

Questions where the system missed the target at top-1, or gave a wrong answer.

- **B1_ocr_keyword** · q23 (field) “car registration number” → ranked nothing; answer “” 
- **B2_plus_cards** · q23 (field) “car registration number” → ranked s07 s06 s16 s14 s08; answer “” 
- **B2_plus_cards** · q27 (recency) “latest delivery update” → ranked s14 s15 s05 s16 s04; answer “” 
- **B3_full_ai** · q27 (recency) “latest delivery update” → ranked s15 s14 s16 s05 s04; answer “Expected by: 6 PM” · newer s15 replaced s14 (“Arriving: Thu 8 Oct”)

Decision rule (provisional): B3 must beat B1 top-1 by ≥20 points on paraphrase + field questions, lose ≤5 points on keyword questions, and be confidently wrong on ≤2% of held-out questions.