Meta allows deepfake "election cancelled" content to spread in #Aras25
This video is worrying; it shows a combination of both intent and skill in building out very deceptive content, in a way that resonates with Irish audiences.
Update Oct 22nd at 5pm: the post previously linked to that contained the fake video has been removed; a watermarked version is embedded in this RTE report on the incident, which includes commentary from me and responses from regulators and Meta.
Last night (October 21st) a Facebook account called "RTE News AI", which had all of the branding and insignia of the Irish state broadcaster, pushed out a very realistic deepfake video of a news report declaring Friday's presidential election cancelled, as Catherine Connolly had announced her withdrawal from the race (this is not true, just to clarify).
It wasn't that account's first AI created electoral disinformation post, and I suspect it won't be the last major piece we see this week.
The video: a sophisticated deepfake
If you haven't seen the video yet, I suggest you go watch it to get a sense of just how real it seems; I don't want to proliferate more copies of it by uploading it here, but it is embedded in this RTE report about the incident.
This is no Sora cheap fake; the video is sophisticated. It has 3 distinct scenes. The faces and voices are ones that are so familiar to us and really well synced; the spelling is all correct; the background are spot on - the familiar RTE newsroom branding to Leinster House behind the reporter.
I have never seen this quality of fake content in an Irish context before.
And the subject is not just disinformation about a candidate - it is electoral process disinformation. This means that it content that is trying to mislead people about voting itself. This is something that is treated very differently both in platform policy and often in laws governing elections (though our laws governing this have not yet been enacted).
The Facebook account was deleted a few hours after this video came out, only after it was escalated within Meta by organisations monitoring the campaign. Meta's own tools failed to pick this up; despite the content being explicitly about the election, and the account mimicking the branding of the national broadcaster.
What is more, when I looked at the Facebook account last night, before it was deleted, I found other videos going back to Sept 9th by this account that also posted election disinformation.
One, posted earlier in the week, was a similar deepfake video clip of a news report, this one declaring that a majority of votes in the campaign had been spoilt (again, this is not true, counting will happen on Saturday and no votes have been cast yet).
That video again had all of the RTE branding and insignia, and it even had a ticker at the bottom with results from different constituencies.
Again - this was allowed to run for 4 days, on Facebook, in the lead up to a major election, had garnered over a thousand likes, with no action taken by Meta.
Another shows that the page had been active for some time. The page had a fake RTE report, this time posted on September 9th, 6 weeks ago, claiming that the National Women's Council (a favourite target of far right agitation) had made disparaging remarks about Catherine Connolly.
The emergence of this video is worrying; it shows a combination of both intent and skill in building out very deceptive content, in a way that resonates with Irish audiences.
And it looks like neither our legal structure nor the platforms who govern our information system are set up for it; according to the Institute for Strategic Dialogue, the same people behind this account had an X account, and were running this same content on YouTube until a few hours ago (Wednesday morning).
**Seen anything else weird? Email me!**
ICYMI:
Liz Carolan
Liz Carolan
Liz Carolan
