How do you approach regulatory issues in artificial intelligence (AI)?
Regulatory issues need to align with the technical reality. At Télécom Paris we adopt an interdisciplinary approach through the Operational AI Ethics programme which brings together six disciplines: applied mathematics, statistics, computer science, economics, law, and sociology. Interdisciplinarity is exciting, but it takes work! We each speak different languages and must build bridges between our different scientific disciplines.
What is the status of AI regulation in Europe?
AI is not an unregulated field. Existing regulations already apply to AI, be it the RGPD for personal data, or sector-specific regulations in the field of health (medical devices), finance (trading models, solvency), or automotive for example.
So why does the European AI Act propose to add specific limitations?
AI software, and in particular Machine Learning (ML) software, poses new problems. Traditional software – symbolic AI, sometimes called “good old-fashioned AI” – is developed from precise specifications, with certain and provable output data. They are deterministic algorithms: input “a” plus input “b” will always lead to output “c”. If this is not the case, there is a bug.
In ML, the algorithms create themselves by learning from large volumes of data and operate probabilistically. Their results are accurate most of the time. As such, they can base their predictions on irrelevant correlations that they have learned from the training data. The risk of error is an unavoidable feature of probabilistic ML models, which raises new regulatory issues, especially for high-risk AI systems. Is it possible to use a probabilistic algorithm in a critical system like image recognition in an autonomous car? Moreover, ML algorithms are relatively unintelligible.
The 2018 crash involving the autonomous Uber car in Arizona is a perfect illustration of the problem. The image recognition system learned that a human is usually crossing the road near a crosswalk. A pedestrian was crossing the road with his bike away from the crosswalk, and the system classified the image as a vehicle, not a pedestrian, right up until the last second before the collision. Hence, the car did not brake in time and the pedestrian was killed. In addition, the driver who was supposed to supervise the system was inattentive (inattention is a common phenomenon called “automation complacency”). The challenge for the future will be to surround these probabilistic systems – which are very efficient for tasks like image recognition – with safeguards. Hybrid systems, which combine ML and symbolic AI, are a promising way forward.
How can we regulate to address this issue?
The draft EU AI Regulation will require compliance testing and CE marking for any high-risk AI system placed on the market in Europe. The first challenge is to define what is meant by a high-risk AI system! At present, this would include software used by the police, for credit scoring, for reviewing university or job applicants, software in cars, etc. The list will continue to grow. Real-time facial recognition used by the police for identification purposes will be subject to specific constraints, including independent testing, and the involvement of at least two human operators before confirming a ‘match’.
For other high-risk systems, the draft regulation envisages compliance testing by the company itself. Each system will have to be subject to a risk assessment and be accompanied by documentation explaining the risks. The systems will have to ensure effective human control. The operator of the system should generate event logs allowing for auditability of the system. For AI systems integrated into systems already covered by regulation (e.g. medical devices), the testing and compliance regime will be governed by the sectoral regulation. This avoids duplication in the regulation.
Why is there so much distrust of ML algorithms when risks are accepted in other areas?
This mistrust is not new. The Tricot report of 1975 – the report that led to the adoption of the French Data Protection Act in 1978 – already mentioned the distrust of computer systems that reduce human beings to a series of statistical probabilities. By reducing us to numbers, such systems deny our individuality and humanity. We are used to statistical profiling when it comes to receiving an advertisement on Facebook or a music recommendation on Deezer. But for more serious decisions – a hiring decision, admission to a university, triggering a tax audit, or getting a loan – being judged solely on a statistical profile is problematic, especially when the algorithm that creates the profile is unintelligible!
The algorithm should therefore provide statistical insight into the issue, but never replace the discernment and nuance of a human decision-maker. But beware, human shortcomings should not be minimised either – in the US it has been shown that judges make heavier prison decisions before lunch when they are hungry. Algorithms can help compensate for these human biases.