Alarmed at the growing forged or deepfake videos on its platform, Facebook organised a Deepfake Detection Challenge and the results were not very encouraging, as the highest-performing model achieved an average precision of 65.18 per cent against real world examples.
This, however, establishes a new shared baseline as the artificial intelligence (AI) community continues to work on this difficult and important task, said Facebook.
The Deepfake Detection Challenge drew more than 2,000 participants, who trained and tested more than 35,000 models using a unique new data set created for the challenge.
Tested against a black box data set with challenging real world examples that were not shared with entrants, the top model achieved an average precision of 65.18 per cent.
Facebook partnered with other industry leaders and academic experts last year to create the Deepfake Detection Challenge.
“By creating and sharing a unique new data set of more than 100,000 videos, the DFDC has enabled experts from around the world to come together, benchmark their deepfake detection models, try new approaches, and learn from each others’ work,” the social network said in a statement.
Facebook in January this year has announced tough policies against the spread of manipulated media on its platform.
The company said it will remove misleading manipulated media if it has been edited or synthesized beyond adjustments for clarity or quality.
Deepfakes are video forgeries that make people appear to be saying things they never did, like the popular forged videos of Facebook CEO Zuckerberg and that of US House Speaker Nancy Pelosi that went viral last year.
Videos that don’t meet standards for removal are still eligible for review by one of Facebook’s independent third-party fact-checkers, which include over 50 partners worldwide fact-checking in over 40 languages.
The compaby said it would also share details on its plans to open-source the raw data set used to construct the Deepfake Detection Challenge, featuring more than 3,500 actors and 38.5 days’ worth of data.
“This will help AI researchers develop new generation and detection methods to advance the state of the art in this field. Moreover, this data set will be opened for use for other research work in AI domains as well as work on deepfakes,” Facebook elaborated.