Privacy and AI

When is a hot dog not a hot dog? Why, when the Bayesian classifier finds enough points of differentiation to cross a probabilistic threshold, of course!

I have significant concerns that as we approach the regulation of AI and algorithms with the proposed EU AI Act and other forthcoming legislation, we may fail in our attempts to create the kind of transparency required for 1) ethical AI, 2) effective regulation, and 3) consumer trust-building.

This concern centers on the black box of Machine Learning. By some accounts, the more accurate the prediction, the more unexplainable the result. This black box problem means that explainability in AI systems will not be readily apparent to the system operators. It is unlikely to be apparent even to the developers. How can it be apparent to regulators?

As a result, the only way to know if most AI systems are creating disparate outcomes is to do extensive testing; Not simply testing in the development of the system, but likely testing as an ongoing cost of operation. Even in these cases, it is unlikely that an advanced AI system in Hiring and Consumer Credit, for instance, will ever really be able to point to a factor or two and say: “This is why you did not qualify.” ML output doesn’t currently come with this kind of factor analysis; The black box is mysterious, but in the longer term, it doesn’t necessarily need to be.

Regulations like the EU AI propose that high-risk systems have at least two humans in the loop to review machine output. Human in the loop is an excellent precaution and will help in many ways. But can it help with explainability? I don’t think so. Humans looking at inscrutable results will not add much insight.

TL;DR: “Houston, we have a problem with explainability.”