Gendered Reality Traps: How "Objective" Machine Learning Contributes to Intersectional Exploitation

In 2012, the multinational supermarket chain Target sent targeted advertisements for maternity products to women, none of whom had disclosed to the supermarket that they were pregnant. This targeting was based on the volunteered and behavorial data of its customers with regard to 25 products, which enabled Target to assign its customers an automated “pregnancy prediction” algorithmic score. Shortly after, an angry man stormed into one of Target’s stores to complain that his daughter, who was still in high school had received coupons from the store in the mail for baby clothes and cribs. The manager admitted a mistake and called the man again two days later to apologise. Upon which the father confessed that he had had a talk with his daughter since and that she was indeed pregnant.

Let’s consider another instance: Founded in 2011, Lenddo is a Singapore-based startup that uses entire digital footprints of potential loan applicants in emerging markets- many of the global south where people often lack traditional histories or even bank accounts- to assess their creditworthiness by having individuals download their app onto their smart phones. Lenddo claims that 5 million people have received loans via their partners in this way. This obviously also indicates that some people have also been denied loans in this way.

These examples illustrate that the trend in today’s world is not just widespread data collection but the processing of this data for profiling. In both these cases, it is not the mere collection of data, which was indeed obtained with the rightful legal requirements of consent, but the collation and processing of such data together to find correlative patterns between several random variables in order to gauge new information about the person concerned. On the basis of this information, the person is either targeted for maternity marketing or assessed for a loan.

In both these cases, it is not the mere collection of data, which was indeed obtained with the rightful legal requirements of consent, but the collation and processing of such data together to find correlative patterns between several random variables in order to gauge new information about the person concerned.

Though both examples above leave one with a sense of unease, it is difficult to articulate what exactly is wrong about such big data based targeted advertising or credit score calculations. Especially when one considers that both instances represent “truths” about the data subject - the Target customer indeed shopped said products which she voluntarily shared with Target, and then was indeed pregnant and the receipt and denial of loans for Lenddo users was based on actual empirical data which was acquired with the app users’ consent. This is the exact argument that many Big Data enthusiasts make: that Big Data allows us to get more precise information about an individual and act upon the basis of such knowledge according to real conditions. Entrepreneur, journalist and TED-owner Chris Anderson even heralds Big Data as the end of theory because the sheer amount of data available in real-time allows us to track and measure what people do with unprecedented fidelity. The argument goes that “numbers speak for themselves.”

But how far are such claims true? Does decision-making by such algorithms really correspond to reality? To understand this we need to understand how exactly the algorithms which make decisions about targeted advertising or credit lending work.

Does decision-making by such algorithms really correspond to reality?

How Machine Learning Works or, Coding is Political

Many algorithms which are employed for targeted advertising or credit score calculation are based on predictive machine learning, which develops upon statistical methods of prediction, but ignores much of the baseline wisdom of statistical theory like the sample size is not the population or that correlation is not causation. Machine learning refers to the process by which an algorithm is ‘trained’ on large sets of data, viz. made familiar with such sets whereby the algorithm ‘learns’ from such data sets. Such learning is actualized through establishment of correlative patterns between different parameters in the database. In this sense, machine learning algorithms construct knowledge which was not explicitly there in the database. This new knowledge is then used to make predictions about real world scenarios outside of the training data set.

Machine learning refers to the process by which an algorithm is ‘trained’ on large sets of data, viz. made familiar with such sets whereby the algorithm ‘learns’ from such data sets. 

For example, consider a credit score machine learning algorithm. The algorithm may be trained on a database which contains hundreds of thousands of parameters of information about an individual. This could range from everything from age, gender, previous loan application history, success of getting loans in the past, successful settlement of debts, race, income group, hair colour, eye colour to the movies which a person watched in the last month, the search terms the person used in the last week, the restaurants they ate at, the location they visited, or the colour of their car. This is a truly random set of data, and it is an extremely large and diverse data set – say from all people accessing the internet- which allows for the (flawed) assumption that this sample of this data set, or Big Data, is actually the population.

In traditional programming, the programmer would have decided which parameters or inputs are relevant for the decision on the credit score. In which case, the programmer would make certain assumptions about which parameters have a causal relationship to the ability to recover the loan back from the applicant in the future. Accordingly, parameters like previous credit history would then perhaps go into the algorithm, and parameters like gender or race would not. However in machine learning algorithms, the assumption of there being a causal relationship between the creditworthiness of a person and any other parameter on which data about her exists is altogether abandoned and the “numbers are left to speak for themselves.” What this means is that rather than being programmed to look only for specific parameters of information for determining creditworthiness, the algorithm analyses large sets of data (assumed to be representative of the population, given their largeness) to learn if there is any correlation between different random parameters and the creditworthiness of people. In this process the algorithm might ‘learn,’ for example, that people who are fans of boy bands from the 2000s have often defaulted on loans. It might also ‘learn’ that people who identify as ‘transgender’ or ‘female’ have been often denied loans when they applied for it in the past.  This knowledge can then lead to the machine learning algorithm assign low credit rating scores to such populations. Which is obviously unfair.

The algorithm analyses large sets of data (assumed to be representative of the population, given their largeness) to learn if there is any correlation between different random parameters and the creditworthiness of people. In this process the algorithm might ‘learn,’ for example, that people who are fans of boy bands from the 2000s have often defaulted on loans.

Are Algorithms Biased? Or is Reality Biased? Can Algorithms Escape Reality?

But of course programmers are not that stupid and they do not use a database with obviously discriminatory parameters like gender, race, sexuality or differently-abled bodies. To use such parameters would also be illegal under the fundamental rights, human rights and anti-discrimination laws of many jurisdictions across the world and algorithm developers and programming firms are aware of this. But what we understand as unfair or discriminatory programming nevertheless creeps in through other ways.

Mathematician and data scientist Cathy O’Neill explains how. “From time to time,” she writes, “people ask me how to teach ethics to a class of data scientists. I usually begin with a discussion of how to build an e-score model and ask them whether it makes sense to use “race” as an input in the model. They inevitably respond that such a question would be unfair and probably illegal. The next question is whether to use “zip code.” This seems fair enough, at first. But it doesn’t take long for the students to see that they are codifying past injustices into their model. When they include an attribute such as “zip code,” they are expressing the opinion that the history of human behavior in that patch of real estate should determine, at least in part, what kind of loan a person who lives there should get.”O'neill, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown/Archetype. p 122

What this illustrates is that even though the parameters for machine learning algorithms like credit rating score might not be built upon directly discriminatory grounds like race, gender or caste, they can still be indirectly discriminatory through the use of random parameters that encode these grounds of discrimination. For example, a person’s zip code could be an indicator of their caste status since caste-based segregated living is a historical reality. 

What this illustrates is that even though the parameters for machine learning algorithms like credit rating score might not be built upon directly discriminatory grounds like race, gender or caste, they can still be indirectly discriminatory through the use of random parameters that encode these grounds of discrimination.

So coming back to the question of does decision-making by such algorithms really correspond to reality? Maybe yes. But like seen, ‘reality’ is rigged. ‘Reality’ is unfair, biased and discriminatory. So this question itself is not very helpful in assessing the situation because deeper entrenchments of power are at stake.

Big data machine learning algorithms might be reflective of ‘reality’ without even being directly discriminatory, but a reality built on assessing Dalits, women, trans-persons or people of colour (or anyone really!) upon code based correlations is not a reality which we want to perpetuate. Decisions made on these versions of reality are realities which need to be resisted. The hard-to-articulate unease we feel in both the examples of the pregnancy test scores and credit rating scores is then then unease we feel with the reality where capitalism exploits both child-bearing bodies and participation of caste-ised, racialized and gendered bodies in the marketplace on empowered terms. In the surveillance economy of the 21st century it is with this precise nuance that abhorrent forms of power perpetuate themselves.

In the surveillance economy of the 21st century it is with this precise nuance that abhorrent forms of power perpetuate themselves.

However, most practices of algorithmic programming do not realize this. Which is why we have enthusiasts applauding this data deluge as a neutral rendition of reality upon which free and fair decisions are possible. But this is missing the forest for the trees since such data is not a mere representation of reality but rather creates and perpetuates a reality. In this sense, algorithms cannot escape the biases of historic realities and cannot be ‘objective’ simply because data itself is never ‘objective’. The assessment of whether an algorithmic decision can be constituted as free and fair needs more contextual and historical understanding than what machine learning correlations provide. The Marxist revolutionary Rosa Luxemburg once said, ‘To be apolitical is political, without realizing it.” The exuberance big data driven algorithms and their possibilities for better decision-making seems fueled by exactly this myopia which misreads the low visibility of power codes as a mark of neutral objectivity.