🩺 Storyboard · Medical Imaging · Model-vs-Model Audit

Two AIs, Three Scans,
One Missed EmergencyThe same 3 real cases, read by the production Gemini path and by Claude — then scored against the diagnosis on file

Storyboard's medical-imaging domain reads an uploaded scan with a multimodal model via the radiologist skill. We ran three real, publicly-documented cases through the actual production path — gemini-2.5-flash (the deployed /api/agent/gemini endpoint) — and, for comparison, through Claude Opus 4.8. Both were graded against the published ground truth. The headline isn't "AI is great." It's that the two models disagreed materially, and the production model missed a life-threatening finding. That's the honest case for why this is a second opinion, never a primary read.

cases
3 · CXR + MRI
production reader
gemini-2.5-flash
comparison
Claude Opus 4.8
scored vs
published ground truth

Case 1 · Chest X-ray

Frontal CXR · 67 M
frontal chest radiograph
frontal chest radiograph
Gemini 2.5 Flash · productionpartial
Ill-defined opacity in the right LOWER lung field; right hemidiaphragm partially obscured (silhouette sign) → "right LOWER lobe involvement." Impression: right lower lobe pneumonia; differential atelectasis / aspiration / mass.
Claude Opus 4.8 · comparisonmatch
Subtle opacity right mid-to-lower zone, effacing the right heart border → right MIDDLE lobe favored, cannot exclude RLL. Impression: RML consolidation / pneumonia.
Ground truthLobar pneumonia, right MIDDLE lobe (CC0, Mikael Häggström MD). → Both caught right-sided pneumonia; Gemini localized to the wrong lobe (RLL), Claude to the correct lobe (RML). Lobe localization rests on which border is effaced — they read it differently.

Case 2 · Chest X-ray

AP CXR · acute setting
AP chest radiograph with ECG leads
AP chest radiograph · ECG leads visible
Gemini 2.5 Flash · productioncritical miss
Called RIGHT lung "patchy opacities, consolidation or contusion"; reported a "metallic projectile-like foreign body" in the right hemithorax; "mediastinum midline." Did not identify the pneumothorax — listed it only as something a CT should "assess for."
Claude Opus 4.8 · comparisonmatch
LEFT hyperlucency with a collapsed-lung edge → large left pneumothorax; mediastinum shifted right (away from the side) → tension features, an emergency. Missed the small left effusion.
Ground truthLeft tension pneumothorax (tracheal/mediastinal shift away from the side), small left effusion (CC BY-SA 2.5, Clinical Cases). → Gemini missed the diagnosis entirely: wrong side, no pneumothorax called, and it hallucinated a foreign body (most likely a misread ECG-lead marker). Claude caught the emergency but missed the small effusion. This is the case that matters most — and the production model failed it.

Case 3 · Brain MRI

Axial T1 + contrast
axial post-contrast brain MRI
axial T1 post-gadolinium · level of orbits
Gemini 2.5 Flash · productionmatch + over-call
Ring-enhancing mass with central necrosis (RIGHT temporal), edema + mass effect → high-grade glioma / GBM. But also reported a 2nd and 3rd lesion ("multifocal") and a midline shift not in the reference.
Claude Opus 4.8 · comparisonmatch
Single ring-enhancing mass, thick irregular rim + central necrosis (left temporal by convention) → glioblastoma leading; differential metastasis / abscess.
Ground truthGlioblastoma multiforme — a ring of enhancement around central necrosis (public domain, AFIP). → Both landed GBM as the leading read. Gemini over-called multifocality (extra "lesions" most consistent with normal vessels / choroid plexus) and used opposite laterality; Claude described the single lesion. Over-reading is its own failure mode.

The scorecard — vs ground truth

CaseGround truthGemini 2.5 Flash (production)Claude Opus 4.8
CXR 1RML pneumoniaright LOWER lobe — ✓ side, ✗ lobeRML — ✓
CXR 2Left tension pneumothoraxMISSED — wrong side, hallucinated foreign bodyleft tension PTX — ✓ (missed effusion)
MRI 3Glioblastoma (single)GBM ✓ + over-called multifocalsingle GBM — ✓

Same three images. Two capable models. One clean match each on the MRI's headline diagnosis, a lobe-level disagreement on the pneumonia, and — critically — the production model completely missed the tension pneumothorax while inventing a foreign body. If you had trusted either model alone as a primary read, you'd have had a different (and on Case 2, dangerous) outcome.

What this actually shows

AI radiology reads are not interchangeable, and not reliable enough to trust alone. Two strong multimodal models, given identical inputs, produced materially different reports — including a missed emergency and a hallucinated foreign body from the production model. This is the empirical case for the disclaimer: the output is a second opinion / teaching aid that a licensed radiologist must confirm. It is never the primary read, and "the AI agreed" means nothing if you didn't check which AI, on which image, against the truth.

This is NOT a validation study. Three cases, hand-picked because they are real, documented, and textbook-clear. n=3 proves nothing about population accuracy — a real claim needs hundreds of consecutive unselected studies, blinded reads, and a radiologist reference standard. What it does show, honestly, is the shape of the failure modes: wrong-lobe localization, laterality flips, over-reading normal structures as lesions, distraction by markers, and missing a finding that's plainly there.

Method & provenance. The Gemini reads are the literal production path — gemini-2.5-flash via the deployed /api/agent/gemini endpoint, prompted with the radiologist skill's report structure. The Claude reads are claude-opus-4-8 for comparison. Neither is a medical-trained model. The right tool is a vision-language model (not yolo/supervision, which are trained on everyday objects and find nothing on a radiograph); sam3 + topaz-upscale assist; medical-trained classifiers are a future phase.

Chest X-ray playbook → MRI readback playbook All showcases