Deepfake detection accuracy: what the percentages mean

One number, four hidden parameters

Every "X percent accurate" claim silently fixes four things a buyer must unfix.

Detection of what: presented deepfakes photographed off screens behave differently from injected streams; face swaps differ from full synthesis, lip-sync puppets, and AI-generated documents; and a detector tuned on one family can be blind to the next. A score without an attack taxonomy is a score on an undisclosed exam.

Against which generators: detection models learn generator fingerprints, so accuracy against the generators in the training set says little about the one released last month. The methodological gold standard is the holdout: scores reported against generator families the model never saw, which approximates the production reality that the adversary ships updates on their schedule, not the lab's.

At what false-positive rate: catch rate and false-alarm rate trade against each other on a curve, and quoting one without the other is choosing the flattering point. A 96 percent catch at a 1 percent false-positive rate is a very different machine from the same catch at 0.01 percent, and the difference is the entire operations budget.

On what media quality: lab corpora are clean; production arrives compressed, re-encoded, low-light, and screen-captured, conditions that erode synthesis artifacts and detector confidence together. Scores on pristine media bound the best case, not the expected one.

The base-rate arithmetic nobody quotes

The parameter that most decides a detector's operational value appears in no vendor deck: the attack base rate. Suppose one session in ten thousand is a deepfake, a plausible onboarding mix, and a detector catches 96 percent of fakes at a 1 percent false-positive rate. Across a million sessions: 100 attacks, 96 caught, and 10,000 genuine customers flagged, meaning each true catch arrives buried under roughly a hundred false alarms, and the review queue, not the model, becomes the system. The same detector at a 0.01 percent false-positive rate produces 100 false alarms per 96 catches, a workable one-to-one, and that difference, two orders of magnitude in operational cost, lives entirely in the half of the claim that usually goes unstated.

Three design consequences follow. Detectors in low-base-rate flows must be tuned and reported at stringent false-positive points, where their catch rates are always lower than the headline. Detection output should route to cheap corroborating checks, liveness, provenance, step-up verification, rather than straight to human review, so false positives cost seconds instead of analysts. And the honest comparison metric between vendors is catch rate at your false-positive budget on your media mix, which no leaderboard publishes and any serious proof of concept can measure.

Reading a reported result

Apply the decoder to the quarter's genuinely encouraging sandbox results. A 96-percent-plus detection rate on manipulated and synthetic selfies in identity verification is a strong result against the dominant attack class, and the sandbox venue, shared infrastructure, regulator oversight, raises the testing bar. The unstated parameters are the standard ones: attack composition, generator holdout, false-positive rate, and media conditions, and twenty-of-twenty repelled attempts is a promising anecdote whose evidential weight depends on who authored the twenty and how adversarially. None of this is criticism of the trials, first numbers are what sandboxes are for, it is the reading discipline that keeps first numbers from becoming procurement folklore, the same discipline the human-guessing fallacy made necessary when organizations graded detection by whether staff could spot fakes.

Detection's place in the stack

The final reading rule is architectural: detection accuracy, however honest, measures one layer, and the layers compose. A detector's misses are survivable where capture integrity independently constrains what media can arrive, liveness physics and provenance checks that a synthetic stream must also defeat; its false positives are survivable where corroboration is cheap; and its generator lag is survivable where reuse analytics catch the artifacts that recur across attacks whatever the generator. Conversely, the costliest fraud of the year rode channels no detector monitors, a messaging app and a phone call, which is why ceremonies that no convincing media can waive sit beside detection in every durable program. The right ambition for a detection layer is honest and bounded: measured catch rates at budgeted false-positive points against current attack families, inside an architecture that does not need the detector to be perfect. Vendors who present their numbers that way are describing a system; vendors who present a lone percentage are describing a demo.

Running your own honest evaluation

Buyers do not need a laboratory to apply the decoder; a disciplined proof of concept covers most of it. Assemble the attack set from your threat model: current open and commercial generators across the families you face, swaps, full synthesis, document generation, plus presented and injected delivery of each, holding out at least one generator family the vendor cannot have tuned for. Mirror your media reality: your capture SDKs, your compression, your device mix, your lighting, because detectors degrade with the artifacts your pipeline adds. Fix the operating point before measuring: declare your false-positive budget from review capacity, then read catch rates there, not at the vendor's chosen point. Score the stack, not the component: run the same attacks against the full flow, detection plus liveness plus provenance, because composition is what you are actually buying, and a stack catch rate at a stack false-positive rate is the number that predicts production. And re-run quarterly: the attack set that graded the purchase becomes the drill set that maintains it, which converts evaluation from a procurement event into the standing measurement the testing era expects.

The claims glossary, for fast meetings

Five phrases decode most detection decks. "Accuracy" without qualifiers: assume the flattering operating point on clean media against trained generators, and ask for the curve. "Benchmark-leading": ask which benchmark, which date, and whether the benchmark includes injected delivery and holdout generators. "Real-time": ask the latency budget at your volume and what happens to sessions awaiting verdicts. "Continuously updated": ask the measured lag between a major generator release and the model update that covers it, with history. "Human-in-the-loop": ask the queue math at your base rate, because the loop's humans are the cost the percentage hides. None of the phrases is disqualifying; all of them are invitations to the one request that sorts the field: show the methodology, on our media mix, at our false-positive budget.