A data ?black hole?: Europol ordered to delete vast store of personal data

The EU's police agency, Europol, will be forced to delete much of a vast store of personal data that it has been found to have amassed unlawfully by the bloc's data protection watchdog. The unprecedented finding from the European Data Protection Supervisor (EDPS) targets what privacy experts are calling a "big data ark" containing billions of points of information. Sensitive data in the ark has been drawn from crime reports, hacked from encrypted phone services and sampled from asylum seekers never involved in any crime.

According to internal documents seen by the Guardian, Europol's cache contains at least 4 petabytes - equivalent to 3m CD-Roms or a fifth of the entire contents of the US Library of Congress. Data protection advocates say the volume of information held on Europol's systems amounts to mass surveillance and is a step on its road to becoming a European counterpart to the US National Security Agency (NSA), the organisation whose clandestine online spying was revealed by whistleblower Edward Snowden.

Among the quadrillions of bytes held are sensitive data on at least a quarter of a million current or former terror and serious crime suspects and a multitude of other people with whom they came into contact. It has been accumulated from national police authorities over the last six years, in a series of data dumps from an unknown number of criminal investigations.

The watchdog ordered Europol to erase data held for more than six months and gave it a year to sort out what could be lawfully kept.

The confrontation pits the EU data protection watchdog against a powerful security agency being primed to become the centre of machine learning and AI in policing.

The ruling also exposes deep political divisions among Europe's decision-makerson the trade-offs between security and privacy. The eventual outcome of their face-off has implications for the future of privacy in Europe and beyond.

The EU home affairs commissioner, Ylva Johansson appeared to defend Europol. "Law enforcement authorities need the tools, resources and the time to analyse data that is lawfully transmitted to them," she said. "In Europe, Europol is the platform that supports national police authorities with this herculean task."