# Bench run: qwen3_4b-instruct-2507-q4_K_M · pipeline v4 · OCR vision · dev split

26 screenshots, 23 questions. Machine: macOS-27.0.1-arm64-arm-64bit-Mach-O.

## Results by question type

| System | Type | n | Top-1 | Top-3 | Answer correct | Confident wrong | Correct abstain |
|---|---|---|---|---|---|---|---|
| B1_ocr_keyword | keyword | 4 | 100% | 100% | – | – | – |
| B1_ocr_keyword | paraphrase | 6 | 50% | 67% | – | – | – |
| B1_ocr_keyword | field | 6 | 100% | 100% | – | – | – |
| B1_ocr_keyword | recency | 3 | 33% | 67% | – | – | – |
| B1_ocr_keyword | unanswerable | 4 | – | – | – | – | – |
| B1_ocr_keyword | ALL | 23 | 74% | 84% | – | – | – |
| B2_plus_cards | keyword | 4 | 100% | 100% | – | – | – |
| B2_plus_cards | paraphrase | 6 | 50% | 83% | – | – | – |
| B2_plus_cards | field | 6 | 100% | 100% | – | – | – |
| B2_plus_cards | recency | 3 | 33% | 100% | – | – | – |
| B2_plus_cards | unanswerable | 4 | – | – | – | – | – |
| B2_plus_cards | ALL | 23 | 74% | 95% | – | – | – |
| B3_full_ai | keyword | 4 | 100% | 100% | 75% | 0% | – |
| B3_full_ai | paraphrase | 6 | 83% | 83% | 83% | 0% | – |
| B3_full_ai | field | 6 | 100% | 100% | 100% | 0% | – |
| B3_full_ai | recency | 3 | 100% | 100% | 100% | 0% | – |
| B3_full_ai | unanswerable | 4 | – | – | – | 25% | 75% |
| B3_full_ai | ALL | 23 | 95% | 95% | 89% | 4% | 75% |

## Latency per question (this machine)

| System | p50 | p95 |
|---|---|---|
| B1_ocr_keyword | 0 ms | 0 ms |
| B2_plus_cards | 0 ms | 0 ms |
| B3_full_ai | 1012 ms | 1708 ms |

## Failures

Questions where the system missed the target at top-1, or gave a wrong answer.

- **B1_ocr_keyword** · q07 (paraphrase) “Ravi's address” → ranked s05 s24 s02 s03 s01; answer “” 
- **B1_ocr_keyword** · q10 (paraphrase) “which restaurant did Ravi suggest” → ranked s02 s03 s05 s01 s24; answer “” 
- **B1_ocr_keyword** · q12 (paraphrase) “that comedy show” → ranked s24 s02; answer “” 
- **B1_ocr_keyword** · q24 (recency) “when is my car service” → ranked s06 s08; answer “” 
- **B1_ocr_keyword** · q25 (recency) “when is the cooktop arriving” → ranked s14 s15; answer “” 
- **B2_plus_cards** · q07 (paraphrase) “Ravi's address” → ranked s24 s05 s01 s02 s03; answer “” 
- **B2_plus_cards** · q10 (paraphrase) “which restaurant did Ravi suggest” → ranked s02 s03 s01 s05 s24; answer “” 
- **B2_plus_cards** · q12 (paraphrase) “that comedy show” → ranked s24 s02; answer “” 
- **B2_plus_cards** · q24 (recency) “when is my car service” → ranked s06 s07 s08; answer “” 
- **B2_plus_cards** · q25 (recency) “when is the cooktop arriving” → ranked s14 s15; answer “” 
- **B3_full_ai** · q10 (paraphrase) “which restaurant did Ravi suggest” → ranked s02 s25 s18 s03 s01; answer “” · no answer in top 3
- **B3_full_ai** · q30 (unanswerable) “how much did the September car service cost” → ranked s08 s06 s07 s17; answer “₹5,950” 

Decision rule (provisional): B3 must beat B1 top-1 by ≥20 points on paraphrase + field questions, lose ≤5 points on keyword questions, and be confidently wrong on ≤2% of held-out questions.