Cleaning removes the file shortcut
Goal: a file-only classifier at or under 0.60 AUC after cleaning. FakeAVCeleb fell from 0.994 to 0.632, close but over the bar. Video bitrate still carries a little signal, because it follows the image content.
The checks come from a final-year research study, P-001. Its tests were written down before any model ran, and it reports the ones that failed.
The preregistration Every result, with receipts Paper and report Code
Audio-visual deepfake detectors score close to perfect on public benchmarks. How much of that is the detector seeing the fake, and how much is it reading the file? And once the file stops giving the answer away, what still works on video the detector has never seen?
Goal: a file-only classifier at or under 0.60 AUC after cleaning. FakeAVCeleb fell from 0.994 to 0.632, close but over the bar. Video bitrate still carries a little signal, because it follows the image content.
AUC 0.780 to 0.810 on FakeAVCeleb speakers it never saw. On new datasets: LAV-DF 0.634 to 0.657, DFDC 0.553 to 0.575, HiFi-AVDF 0.559 to 0.572. That is above chance on every dataset, but well below its score on familiar data.
A threshold set from 200 real clips of the new dataset flags 5 to 6% of real HiFi-AVDF clips, against 40 to 48% with the training threshold. Met on HiFi-AVDF and on LAV-DF by file label; two of three seeds miss it on LAV-DF by window label. The price: at that threshold only 7 to 8% of HiFi-AVDF fakes are caught.
It passes the preregistered rule: +0.088 on FakeAVCeleb and +0.170 on LAV-DF; +0.003 on DFDC (not significant). But without the cloned-voice clips, which the face check was never trained to flag, the FakeAVCeleb gain is 0.001 to 0.004. On LAV-DF the face check scores even fakes with untouched video below the real clips (0.175 to 0.243), so the gain is over a broken baseline. A combination fitted on other datasets fell to 0.477 on DFDC. That is why this site shows the three checks side by side instead of merging them.
Stronger frozen models (CLIP for faces, XLS-R for voices) did not transfer better once the file clues were gone: on HiFi-AVDF, 0.508 and 0.693. No fixed way of combining the checks carried over between datasets. Six simple frame measures do not explain what the face check reads on LAV-DF (together 0.564). And the voice check does not lean on loudness or clip edges: with both removed it stays at 0.933 or above.
Results from the study behind the site.
AUC from file details alone on FakeAVCeleb: codec, length and silence, without looking at a face or hearing a voice. Chance is 0.5.
HiFi-AVDF's file-clue ceiling, down from 0.999. One codec, a fixed length, one loudness for every clip.
Of real HiFi-AVDF clips flagged, with a threshold set from 200 real clips of that source. The threshold carried over from training flagged 40 to 48%.
AUC on FakeAVCeleb speakers it never trained on.
AUC on HiFi-AVDF, from newer generators.
Below chance (seed 17). Sora 2 makes the lips and voice together, so they already match.
The leading-silence shortcut was found by Smeu et al. (CVPR 2025) and confirmed by DeepFake Doctor (2025), which also proposed splitting by identity. Training a sync model on real clips only follows AVH-Align. HiFi-AVDF comes from HAVIC (CVPR 2026). This study adds a codec census, a classifier that uses file details only, and cleaning applied to training and test data together.