European Commission white paper on artificial intelligence: our response

Keeping of records and data

The accountability principle is an important element of the GDPR. Organisations are expected to show how they comply with the legislation, for example, by keeping records of their processing activities and making them available to regulators on request. We believe that this requirement on AI systems could therefore be seen as an extension of those provisions. 

To be effective, specific prescriptions to enable audit and accountability will need to take into account the complexities of machine learning systems. For instance and in broad terms, in AI applications based on neural networks, the system's decisions are based on the pattern and weights of connections between the neurons in the network, which in turn result from training the network on a typically large dataset. The decisions made by the system thus reflect the underlying training data in indirect and complex ways. As a result, requiring the developer to retain training data and make it "available upon request, in particular for testing or inspection by competent authorities" might not always be effective (in terms of how much it will help audit/forensic analysis) or proportionate (particularly for "deep" neural network AI applications, which require particularly large training datasets). In addition, as pointed out in the White Paper, machine learning systems often learn continuously, making training data a moving target. Finally, requiring data to be made available to authorities might raise intellectual property and trade issues, related to those that would arise from requiring source code to be made available, as well as data protection issues. The White Paper acknowledges such issues and proposes that "where necessary, arrangements should be made to ensure that confidential information, such as trade secrets, is protected", but this might be challenging to achieve in practice.