.png)
The Deepfakes Analysis Unit (DAU) analysed a video in which Indian television actor and anchor Anup Soni is apparently talking about a match-fixer involved with the ongoing matches of the Indian Premier League (IPL). Even as the voice in the video appears to warn against the fixer’s criminal record, it seems to encourage people to follow his Telegram channel. The actor has spoken to several Indian publications clarifying that it is not his voice in that video, instead synthetic voice has been used to doctor an old video featuring him, our assessment confirmed the same.
Mr. Soni is the latest celebrity targeted by A.I. manipulated videos that promote betting and gaming apps, other prominent faces such as Virat Kohli, Sachin Tendulkar, and Anant Ambani have been similarly targeted in the recent past.
The 25-second video featuring Soni has garnered more than 35 million views on Facebook since it was first posted on April 26, 2024. The video is in Hindi and the voice behind it sounds very similar to Soni’s. The name on the Facebook account from which the video was uploaded and the name of the fixer mentioned in the doctored video are the same.
Soni has been associated with the popular Hindi show “Crime Patrol” in which he presents dramatised versions of real criminal cases to create social awareness about crimes and criminals in India. The viral, doctored video of his features a clip from that show but without the original audio.
We discovered the original video by running a reverse image search on screenshots from the manipulated video and landed on a Youtube channel featuring popular Hindi crime shows including “Crime Patrol”. A good part of the 30-second segment following the three-minute and 18-second mark in the original video, uploaded on July 22, 2022 from that channel, features in the doctored video.
The lip movements and audio do not match in the manipulated video, which also features visuals of Virat Kohli and other cricketers playing IPL matches, none of those visuals are part of the original video.
The actor, in his interviews to Indian publications, did suggest the use of A.I. in the audio, however, we wanted to track if the audio was fully A.I.-generated or bits of his original voice was also patched together with the fake audio. To check that we ran the video through A.I. detection tools.
Hive AI’s deepfake video detection tool indicated A.I. manipulation in different parts of the video clip and their audio tool specifically caught bits of A.I. in the audio.
We also used TrueMedia’s deepfake detector, which overall categorised the video as “highly suspicious”, suggesting that there was a very high probability of manipulation in the video. It gave a 100 percent confidence score to the sub-categories “audio analysis” and “A.I. generated audio detection”, the former indicates that the audio was produced using an A.I. audio generator and latter is an indicator of synthetic speech. The tool also gave a 99 percent confidence score to the “deepfake face detection” sub-category.
To get further clarity on the A.I. elements in the video we escalated it to our partner IdentifAI, a San Francisco-based deepfake security startup, they used their audio detection software to examine the authenticity of audio in the clip.
First, they extracted Soni’s voice samples from the original video and the manipulated video. Then, a real voice sample was run through the ElevenLabs voice cloning software to create a voice clone of Soni’s. Following which, the generated voice sample and the voice sample from the manipulated video were compared with his original voice through a heat-map analysis.
The two images above that draw a comparison between the voice samples look identical. This suggests that the voice clone created by our partner generated a very similar audio pattern to the voice from the manipulated video. Such a close resemblance can only be explained by the possibility of both having been created by generative A.I., if not necessarily by the same voice cloning software.
On the basis of our findings, expert analysis and the actor’s statement on this issue, we can conclude that Anup Soni’s voice in the video was produced using generative A.I.
(Written by Debraj Sarkar with inputs from Debopriya Bhattacharya and edited by Pamposh Raina)
Kindly Note: The manipulated audio/video files that we receive on our tipline are not embedded in our assessment reports because we do not intend to contribute to their virality.
