An artificial intelligence system developed to automatically detect and block Islamic State propaganda on any online platform could be bypassed by a change in the group’s media strategy, one expert has warned.
The tool, announced by the Home Office today, can reportedly detect 94 per cent of official Isis propaganda with 99.995 per cent accuracy. But, as the tool only focusses on videos produced by Isis’ central and provincial media organisations, it would likely be possible for the group to dodge it.
“Presumably all it would take it to put us back to square one would be if the Islamic State changes the format that it uses, changes logos and is more careful about what soundtrack it uses,” says Charlie Winter a senior research fellow at the International Centre for the Study of Radicalisation.
The branding of Islamic State videos, he continues, is monolithic: it uses the same structures, the same soundtracks, the same icons and the same logos. While this makes it possible for an algorithm to detect and block videos that match this signature, it also raises questions about how effective such a system can be against changes to Islamic State propaganda.
But ASI, the London-based machine learning startup behind the system, claims it will be onerous for Isis to make such changes. “We've been very thoughtful about trying to identify characteristics of the propaganda that are very difficult for Isis to change,” says John Gibson, head of data science consulting at ASI, and technical lead on the project. “It's something we've thought about a great deal and clearly for this thing to work well it needs to be adaptive and it needs to be able to keep up to date as the threat evolves.”
The system works by sitting on a website’s back-end and automatically scanning all video uploads for signs of Isis propaganda. “Every time a new video is released, there's a very good chance the model will capture it and then a human reviewer would confirm that it looks like a [Isis] propaganda video,” says Gibson. Any videos that were new to the system would be added to the training data, helping the model to adapt and learn. The Home Office says the system has been trained on more than 1,000 Isis propaganda videos and will be made available to smaller websites used by the terrorist organisation to stem the spread of radicalisation.
While Google and Facebook have developed advanced artificial intelligence systems to broadly rid their platforms of Isis propaganda, the Home Office’s intention with this new system is to make the same technology available to all platforms. Isis reportedly used 400 different platforms to host propaganda it uploaded in 2017, with 145 new ones used from July until the end of the year alone. By making a sophisticated detection tool available free of charge, the government will hope to stop its spread at source.
According to ASI, the system has been fine-tuned to be manageable even on the largest platforms – with a false positive rate of 0.005 per cent adjusted accordingly. “We did some back of the envelope maths for what that would mean for the very largest platforms – along the lines of YouTube – and 0.005 per cent that false positive rate means you would flag about 250 videos a day if you were having about five million videos a day uploaded. It feels about the limit for a human reviewer,” says Gibson.
But that compromise presents a problem. While such a low false positive rate is good, a detection rate of just 94 per cent means that six per cent of Isis propaganda videos slip through. “It's a boring job but it would be one person's job to flick through those and make sure that true false positives are passed onto the website,” says Gibson. “We optimised the model for false positive performance and we think those numbers are pretty good.”
Creating the system “involved an awful lot of machine learning underneath the surface,” he adds. “We've created are a large amount of different models all running in parallel and all interacting with one other.” Specific details of how the system works cannot not be disclosed for security reasons. The system, which uses ASI’s SherlockML platform, will be made freely available by the Home Office to any partners the government considers responsible, with ASI maintaining the service for the foreseeable future.
Read more: Inside the collapse of Islamic State’s propaganda machine
ASI co-founder and CEO Marc Warner says the machine learning algorithm was trained on what the Home Office considered to be the most dangerous videos produced by Isis. “These are the videos that the Home Office considers to be the most important to take down, the most poisonous as far as their radicalising ability goes,” he says. “For them, it's less about volume and more about how impactful they think it is to tackle particular sets of videos.” Work started in September 2017, with the Home Office approaching ASI looking to develop a system that could automatically flag “a particular set of videos”, Warner adds. Winter argues that Isis’ obsession with “a very precise set of repeating structures and motifs” would be the sort of thing “an algorithm would lap up”.
But he also warns that the scale of Isis propaganda is far greater than any such system could handle. “The majority of its propaganda is in the form of photographs, magazines, radio bulletins and audio statements,” Winter explains. “This isn't a solution to any problem, there is no solution to this problem. At best it's going to mitigate it somewhat, but it is very far from a solution.”
The government’s announcement comes during home secretary Amber Rudd’s two day visit to San Francisco, where she will discuss how best to tackle terrorist content online with Kirstjen Nielsen, US secretary of homeland security. While the system is ready to be implemented, the announcement made no reference to any sites that would be implementing it.
Winter points to the Internet Archive, a major platform used by Islamic State to host propaganda, as one major hurdle. “Is the Internet Archive going to start using this software? Judging by how its operated over the last few years, it seems reluctant to take stuff down.” Such a system can only work, he adds, if it is “widely implemented”, effectively leaving Islamic State with very few outlets for its propaganda.
While Winter agrees that targeting propaganda produced by Isis’ central and provincial media teams was the best way to stymie online radicalisation, he warned that huge quantities of videos and other content would not be caught by ASI’s system. “They will also be missing a whole lot of other content that isn't necessarily as immediately dangerous.”
As well as assisting with the roll-out of the Isis system, ASI is also now working to create a similar algorithm for spotting Al Qaeda propaganda. “The propaganda will be different in some subtle and some obvious ways,” says Gibson. “But the overall set of techniques and the general approach is something that we will most likely be repurpose to Al Qaeda in the near future.” He estimates that the new algorithm will be ready “in the next few months”.
Winter argues that while training machine learning algorithms on Isis propaganda is technically possible, Al Qaeda’s media operation is more complex. “Al Qaeda propaganda is far more varied and subtle as well,” he says, adding that the group was far less obsessed with branding, making it more difficult to detect its propaganda.
Updated 13.02.18: This story originally stated that ASI’s system wasn’t trained on material from the Amaq News Agency. This is not the case.
This article was originally published by WIRED UK
