How (and how not) to fix AI

While artificial intelligence was once heralded as the key to unlocking a new era of economic prosperity, policymakers today face a wave of calls to ensure AI is fair, ethical and safe. New York City Mayor de Blasio recently announced the formation of the nation’s first task force to monitor and assess the use of algorithms. Days later, the European Union enacted sweeping new data protection rules that require companies be able to explain to consumers any automated decisions. And high-profile critics, like Elon Musk, have called on policymakers to do more to regulate AI.

Unfortunately, the two most popular ideas — requiring companies to disclose the source code to their algorithms and explain how they make decisions — would cause more harm than good by regulating the business models and the inner workings of the algorithms of companies using AI, rather than holding these companies accountable for outcomes.

The first idea — “algorithmic transparency” — would require companies to disclose the source code and data used in their AI systems. Beyond its simplicity, this idea lacks any real merits as a wide-scale solution. Many AI systems are too complex to fully understand by looking at source code alone. Some AI systems rely on millions of data points and thousands of lines of code, and decision models can change over time as they encounter new data. It is unrealistic to expect even the most motivated, resource-flush regulators or concerned citizens to be able to spot all potential malfeasance when that system’s developers may be unable to do so either.

Additionally, not all companies have an open-source business model. Requiring them to disclose their source code reduces their incentive to invest in developing new algorithms, because it invites competitors to copy them. Bad actors in China, which is fiercely competing with the United States for AI dominance but routinely flouts intellectual property rights, would likely use transparency requirements to steal source code.

The other idea — “algorithmic explainability” — would require companies to explain to consumers how their algorithms make decisions. The problem with this proposal is that there is often an inescapable trade-off between explainability and accuracy in AI systems. An algorithm’s accuracy typically scales with its complexity, so the more complex an algorithm is, the more difficult it is to explain. While this could change in the future as research into explainable AI matures — DARPA devoted $75 million in 2017 to this problem — for now, requirements for explainability would come at the cost of accuracy. This is enormously dangerous. With autonomous vehicles, for example, is it more important to be able to explain an accident or avoid one? The cases where explanations are more important than accuracy are rare.

The debate about how to make AI safe has ignored the need for a nuanced, targeted approach to regulation.

Rather than demanding companies reveal their source code or limiting the types of algorithms they can use, policymakers should instead insist on algorithmic accountability — the principle that an algorithmic system should employ a variety of controls to ensure the operator (i.e. the party responsible for deploying the algorithm) can verify it acts as intended, and identify and rectify harmful outcomes should they occur.

A policy framework built around algorithmic accountability would have several important benefits. First, it would make operators responsible for any harms their algorithms might cause, not developers. Not only do operators have the most influence over how algorithms impact society, but they already have to comply with a variety of laws designed to make sure their decisions don’t cause harm. For example, employers must comply with anti-discrimination laws in hiring, regardless of whether they use algorithms to make those decisions.

Second, holding operators accountable for outcomes rather than the inner workings of algorithms would free them to focus on the best methods to ensure their algorithms do not cause harm, such as confidence measures, impact assessments or procedural regularity, where appropriate. For example, a university could conduct an impact assessment before deploying an AI system designed to predict which students are likely to drop out to ensure it is effective and equitable. Unlike transparency or explainability requirements, this would enable the university to effectively identify any potential flaws without prohibiting the use of complex, proprietary algorithms.

This is not to say that transparency and explanations do not have their place. Transparency requirements, for example, make sense for risk-assessment algorithms in the criminal justice system. After all, there is a long-standing public interest in requiring the judicial system be exposed to the highest degree of scrutiny possible, even if this transparency may not shed much light on how advanced machine-learning systems work.

Similarly, laws like the Equal Credit Opportunity Act require companies to provide consumers an adequate explanation for denying them credit. Consumers will still have a right to these explanations regardless of whether a company uses AI to make its decisions.

The debate about how to make AI safe has ignored the need for a nuanced, targeted approach to regulation, treating algorithmic transparency and explainability like silver bullets without considering their many downsides. There is nothing wrong with wanting to mitigate the potential harms AI poses, but the oversimplified, overbroad solutions put forth so far would be largely ineffective and likely do more harm than good. Algorithmic accountability offers a better path toward ensuring organizations use AI responsibly so that it can truly be a boon to society.

In the public sector, algorithms need a conscience

In a recent MIT Technology Review article, author Virginia Eubanks discusses her book Automating Inequality. In it, she argues that the poor are the testing ground for new technology that increases inequality— highlighting that when algorithms are used in the process of determining eligibility for/allocation of social services, it creates difficulty for people to get services, while forcing them to deal with an invasive process of personal data collection.

I’ve spoken a lot about the dangers associated with government use of face recognition in law enforcement, yet, this article opened my eyes to the unfair and potentially life threatening practice of refusing or reducing support services to citizens who may really need them — through determinations based on algorithmic data.

To some extent, we’re used to companies making arbitrary decisions about our lives — mortgages, credit card applications, car loans, etc. Yet, these decisions are based almost entirely on straightforward factors of determination — like credit score, employment and income. In the case of algorithmic determination in social services, there is bias in the form of outright surveillance in combination with forced PII share imposed upon recipients.

Eubanks gives as an example the Pittsburgh County Office of Children, Youth and Families using the Allegheny Family Screening Tool (AFST) to assess the risk of child abuse and neglect through statistical modeling. The use of the tool leads to disproportionate targeting of poor families because the data fed to the algorithms in the tool often comes from public schools, the local housing authority, unemployment services, juvenile probation services and the county police, to name just a few — basically, the data of low-income citizens who typically use these services/interact with them regularly. Conversely, data from private services such as private schools, nannies and private mental health and drug treatment services isn’t available.

Determination tools like AFST equate poverty with signs of risk of abuse, which is blatant classism — and a consequence of the dehumanization of data. Irresponsible use of AI in this capacity, like that of its use in law enforcement and government surveillance, has the real potential to ruin lives.

Taylor Owen, in his 2015 article titled The Violence of Algorithms, described a demonstration he witnessed by intelligence analytics software company Palantir, and made two major points in response — the first being that oftentimes these systems are written by humans, based on data tagged and entered by humans, and as a result are “chock full of human bias and errors.” He then suggests that these systems are increasingly being used for violence.

“What we are in the process of building is a vast real-time, 3-D representation of the world. A permanent record of us…but where does the meaning in all this data come from?” he asked, establishing an inherent issue in AI and data sets.

Historical data is useful only when it is given meaningful context, which many of these data sets are not given. When we are dealing with financial data like loans and credit cards, determinations, as I mentioned earlier, are based on numbers. While there are surely errors and mistakes made during these processes, being deemed unworthy of credit will likely not lead the police to their door.

However, a system built to predict deviancy, which uses arrest data as a main factor in determination, is not only likely to lead to police involvement — it is intended to do so.

Image courtesy of Getty Images

When we recall modern historical policies that were perfectly legal in their intention to target minority groups, Jim Crow certainly comes to mind. And let’s also not forget that these laws were not declared unconstitutional until 1967, despite the Civil Rights Act of 1965.

In this context you can clearly see that according to the Constitution, Blacks have only been considered full Americans for 51 years. Current algorithmic biases, whether intentional or inherent, are creating a system whereby the poor and minorities are being further criminalized, and marginalized.

Clearly, there is the ethical issue around the responsibility we have as a society to do everything in our power to avoid helping governments get better at killing people, yet the lion’s share of this responsibility lies in the lap of those of us who are actually training the algorithms — and clearly, we should not be putting systems that are incapable of nuance and conscience in the position of informing authority.

In her work, Eubanks has suggested something close to a Hippocratic oath for those of us working with algorithms — an intent to do no harm, to stave off bias, to make sure that systems did not become cold, hard oppressors.

To this end, Joy Buolamwini of MIT,  the founder and leader of the Algorithmic Justice League, has created a pledge to use facial analysis technology responsibly.

The pledge includes commitments like showing value for human life and dignity, which includes refusing to engage in the development of lethal autonomous weapons, and not equipping law enforcement with facial analysis products and services for unwarranted individual targeting.

This pledge is an important first step in the direction of self-regulation, which I see as the beginning of a larger grass-roots regulatory process around the use of facial recognition.