FinTechCryptoGenerator

Red Team Prompt to Benchmark Deepfake Detection at Your Real False-Positive Budget

This Arbiter red-team prompt runs a methodology-first evaluation of your deepfake and synthetic-media defenses, producing the numbers you would defend to a regulator, an insurer, or your own board. Arbiter, the deepidv autonomous red agent, starts with an attack taxonomy declaration covering face swaps, full synthesis, lip-sync puppets, generated documents, and cloned voice where in scope, each delivered both presented to the sensor and injected through virtual cameras, emulators, or API substitution. A generator holdout reserves at least one family per modality that your models and vendors cannot have tuned for, reported separately as the generalization score, and a production media mirror runs every attack through your real capture SDKs, compression, device mix, and lighting. Operating-point discipline fixes your false-positive budget from measured review capacity first, then reports catch rates per family at that point, with the stack (detection plus liveness plus provenance plus reuse analytics) as the headline and component scores as diagnostics, and base-rate translation converts the results into expected weekly true catches and false alarms at your actual traffic. It delivers the internal methodology report, a quarterly re-run calendar with refreshed generators, and the three cheapest fixes ranked by stack catch rate gained per false positive added. Built for fraud, risk, and identity leads who treat any score quoted without its false-positive rate as marketing, including their own.

How to use this prompt

1
Open Arbiter in the deepidv dashboard and paste the full prompt, or run it in Claude, ChatGPT, or Gemini to design the evaluation protocol before running it live.
2
Add an INPUT section below the prompt with your detection, liveness, provenance, and reuse-analytics layers, your capture SDKs and device mix, your weekly session volume, and your measured manual review capacity.
3
Fix the false-positive budget from review capacity before reading any catch rate, then read the stack-level result first and use component scores only as diagnostics.
4
Take the base-rate translation to operations so the expected weekly false alarms are staffed, and hand the three ranked fixes to the owning teams.
5
Follow the quarterly re-run calendar with refreshed generators, so the holdout keeps measuring generalization rather than familiarity.

The prompt

Arbiter, run a methodology-first evaluation of our deepfake and synthetic-media defenses, producing the numbers we would defend to a regulator, an insurer, or our own board.

Protocol:
1. Attack taxonomy declaration: enumerate the families tested, face swaps, full synthesis, lip-sync puppets, generated documents, cloned voice where in scope, each delivered both presented (photographed/replayed to sensor) and injected (virtual camera, emulator, API substitution).
2. Generator holdout: build the attack set across current open and commercial generators, reserving at least one family per modality that our models and vendors cannot have tuned for, reported separately as the generalization score.
3. Production media mirror: run all attacks through our real capture SDKs, compression, device mix, and lighting conditions, since artifacts our pipeline adds are artifacts detectors lean on.
4. Operating-point discipline: fix our false-positive budget from measured review capacity first, then report catch rates per family at that point, stack-level (detection plus liveness plus provenance plus reuse analytics) as the headline, component scores as diagnostics.
5. Base-rate translation: convert results into expected weekly true catches and false alarms at our actual traffic, the operational forecast the percentage hides.

Deliver the internal methodology report, the quarterly re-run calendar with refreshed generators, and the three cheapest fixes ranked by stack catch-rate gained per false-positive added. Treat any component score quoted without its false-positive rate as marketing, including our own.

Test it in Claude or another LLM

This prompt is built for Arbiter inside deepidv, where Arbiter runs the attack set against your deployed detection stack in a controlled environment. You can design the protocol in any general LLM first with a synthetic stack and traffic profile before running real attacks.

1
Paste the full prompt into Claude, ChatGPT, or Gemini, replacing the opening 'Arbiter,' with a role instruction such as 'Act as a deepfake-defense evaluation lead designing a methodology-first benchmark.' Keep the protocol and the deliverables as written.
2
Add an INPUT section under the prompt and paste the synthetic sample block below so the model has layers, media conditions, traffic, and review capacity to work with.
3
Add one framing line: 'This is synthetic test data. Set the false-positive budget from the stated review capacity before reporting any catch rate.'
4
Check the output shape: a declared attack taxonomy, a holdout generalization score reported separately, stack-level catch rates at the fixed false-positive budget, expected weekly catches and false alarms, a quarterly re-run calendar, and three fixes ranked by catch rate gained per false positive added. If a catch rate appears without its false-positive rate, tighten the framing and re-run.
5
Once the shape is right, run it live in the deepidv dashboard where Arbiter executes the attack set against your real detection stack.

Synthetic sample data to paste alongside the prompt

Fake test data, safe to share with any LLM. Swap in your own once the output looks right.

DETECTION STACK (synthetic, fake):
- deepfake detector: vendor score of 98% quoted with no false-positive rate
- liveness: passive check on selfie capture, injection detection on mobile only
- provenance and reuse analytics: not wired into the decision
MEDIA MIX (fake): 70% mid-range Android with heavy upload compression, 30% web capture with virtual-camera exposure
TRAFFIC (fake): 250,000 verification sessions per week
REVIEW CAPACITY (fake): 2 analysts, about 400 manual reviews per week
OPEN ITEM (fake): which generator families the vendor trained on

Pairs with on deepidv

Sources & further reading

FAQ

Why is a single deepfake detection accuracy number misleading?

Because one percentage silently fixes four parameters: which attack types were tested, whether unseen generators were included, the false-positive rate at that operating point, and the media quality. A 96 percent catch rate at a 1 percent false-positive rate and the same catch rate at 0.01 percent are very different machines, and the difference is the entire operations budget.

What is a generator holdout?

It is a set of generator families deliberately kept out of anything your models and vendors could have tuned on, scored separately as a generalization score. New generators arrive on the adversary's schedule, so the holdout shows whether a detector learned the artifacts of synthesis or memorized the fingerprints of the generators it was trained against.

Why fix the false-positive budget before measuring catch rates?

Attacks are rare relative to genuine traffic, so at realistic base rates even a strong detector can bury each true catch under many false alarms unless it runs at a stringent false-positive point. Setting the budget from measured review capacity first, then reading catch rates there, is what turns a benchmark into an operational forecast of weekly catches and false alarms.

Run it with live verification data

These prompts work in any LLM. Inside the deepidv dashboard, Luna, Arbiter, and Arc run them against your real sessions, screening lists, and audit trails.

Book a Demo