Deepfake literacy tool
Multimodal deepfake detector (image + caption) with plain-language explanations of what the model flagged.
Problem
AI-manipulated images are getting harder for regular readers to spot, and most detection tools respond with a single confidence number - "73% likely fake" - that tells you the model has an opinion but not what it actually noticed, or why a non-expert should trust it.
System design & approach
The detector runs multimodal checks - a vision pass over the image, GPT-4 reasoning over the caption - and aligns the two, rather than scoring the image alone. Because this was an HCI project as much as a modeling one, the explanation layer was as much the point as the detection itself. The result scores an uploaded image, plus an optional caption or claim, across three plain-language axes - misleadingness, image/caption misalignment, and how likely the image is AI-generated - instead of one opaque confidence number.
Challenges
Getting a vision model and a language model to agree on a shared, plain-language vocabulary for something as fuzzy as "misleadingness" is a harder design problem than getting either model to output a single number. The real work was in the explanation layer: deciding what "the model noticed X" should even mean to someone with no background in media forensics, and keeping that fast enough to feel usable, not academic.
My role
I designed the multimodal pipeline pairing vision with GPT-4 text reasoning and built the full-stack prototype end to end, including the plain-language explanation layer that sits on top of the raw model output.
Retrospective
94% model accuracy at 25ms inference latency, with three explicit, human-readable axes in place of a single confidence score.
Working at the intersection of what a model outputs and how people actually read and trust that output - an HCI lens on something I could have treated as a pure classification task. Explanation quality turned out to matter as much as raw accuracy for a tool meant for non-experts.
- GPT-4