Research

The checks come from a final-year research study, P-001. Its tests were written down before any model ran, and it reports the ones that failed.

The question

Audio-visual deepfake detectors score close to perfect on public benchmarks. How much of that is the detector seeing the fake, and how much is it reading the file? And once the file stops giving the answer away, what still works on video the detector has never seen?

What was tested, and what happened

H1Missed

Cleaning removes the file shortcut

Goal: a file-only classifier at or under 0.60 AUC after cleaning. FakeAVCeleb fell from 0.994 to 0.632, close but over the bar. Video bitrate still carries a little signal, because it follows the image content.

SyncWorks, then fades

A lip-sync check trained on real clips only

AUC 0.780 to 0.810 on FakeAVCeleb speakers it never saw. On new datasets: LAV-DF 0.634 to 0.657, DFDC 0.553 to 0.575, HiFi-AVDF 0.559 to 0.572. That is above chance on every dataset, but well below its score on familiar data.

H4Mostly met

False alarms set from real clips

A threshold set from 200 real clips of the new dataset flags 5 to 6% of real HiFi-AVDF clips, against 40 to 48% with the training threshold. Met on HiFi-AVDF and on LAV-DF by file label; two of three seeds miss it on LAV-DF by window label. The price: at that threshold only 7 to 8% of HiFi-AVDF fakes are caught.

H2Holds, but weak

Combining the checks beats the face check alone

It passes the preregistered rule: +0.088 on FakeAVCeleb and +0.170 on LAV-DF; +0.003 on DFDC (not significant). But without the cloned-voice clips, which the face check was never trained to flag, the FakeAVCeleb gain is 0.001 to 0.004. On LAV-DF the face check scores even fakes with untouched video below the real clips (0.175 to 0.243), so the gain is over a broken baseline. A combination fitted on other datasets fell to 0.477 on DFDC. That is why this site shows the three checks side by side instead of merging them.

A11 to A14Exploratory

Four checks added after the main results

Stronger frozen models (CLIP for faces, XLS-R for voices) did not transfer better once the file clues were gone: on HiFi-AVDF, 0.508 and 0.693. No fixed way of combining the checks carried over between datasets. Six simple frame measures do not explain what the face check reads on LAV-DF (together 0.564). And the voice check does not lean on loudness or clip edges: with both removed it stays at 0.933 or above.

What the research found

Results from the study behind the site.

File details alone
0.994

AUC from file details alone on FakeAVCeleb: codec, length and silence, without looking at a face or hearing a voice. Chance is 0.5.

FakeAVCeleb0.994
HiFi-AVDF0.999
LAV-DF0.936
DFDC0.588
After cleaning
0.629

HiFi-AVDF's file-clue ceiling, down from 0.999. One codec, a fixed length, one loudness for every clip.

False alarms
5 to 6%

Of real HiFi-AVDF clips flagged, with a threshold set from 200 real clips of that source. The threshold carried over from training flagged 40 to 48%.

Lip sync, familiar data
0.78 to 0.81

AUC on FakeAVCeleb speakers it never trained on.

Lip sync, new data
0.56 to 0.57

AUC on HiFi-AVDF, from newer generators.

Sora 2 clips
0.42

Below chance (seed 17). Sora 2 makes the lips and voice together, so they already match.

Limits

  • One training dataset. Every model learned from FakeAVCeleb: about 500 real videos from 2021 and older fakes. Newer generators, like Sora 2, are often missed.
  • A small audio clue remains. After cleaning, file details still predict FakeAVCeleb's fake audio at 0.793 AUC, through loudness and the clip edges.
  • Some fixes came after a result was read. Two changes to the protocol (amendments A9 and A10) were made after early results; both versions are reported.
  • No published detectors compared yet. AVH-Align and HAVIC have not been run on this protocol.
  • The false-alarm rate is measured on research data. On your video it is an estimate.

Prior work

The leading-silence shortcut was found by Smeu et al. (CVPR 2025) and confirmed by DeepFake Doctor (2025), which also proposed splitting by identity. Training a sync model on real clips only follows AVH-Align. HiFi-AVDF comes from HAVIC (CVPR 2026). This study adds a codec census, a classifier that uses file details only, and cleaning applied to training and test data together.