This is a blog post based on the chapter ’Contract and Power in the Age of AI’, by Aurelia
Colombi Ciacchi, Greta Berardi and Alma-Cristina Onaie, as published in ‘Ideologies,
Inequalities, and Marginalisation in European Contract Law’, Larcier, 2026

Introduction

Artificial intelligence is quietly reshaping modern life. Algorithms decide who gets hired, who qualifies for a loan, what advertisements we see, and even how much we pay for insurance. These systems promise efficiency, speed, and objectivity. Yet behind that promise lies a growing legal and ethical concern: algorithmic discrimination.
Across Europe, courts, regulators, and academics are increasingly confronting a difficult question: what happens when automated systems reproduce or intensify social inequalities?
This issue is no longer theoretical. Real-world examples demonstrate how algorithms can disadvantage vulnerable groups, often in subtle and indirect ways. In response, the European Union is developing new legal tools designed to force organisations to think about fundamental rights before deploying high-risk AI systems.

Employment and Platform Work: When Algorithms Manage Workers

One of the clearest examples of algorithmic discrimination can be seen in platform work.
On-demand companies, such as Deliveroo and Uber, increasingly rely on algorithms to allocate work, evaluate performance, and rank workers. These systems often appear neutral on their face, yet they may disproportionately disadvantage certain groups.
A major example emerged in Italy in 2020 involving the food delivery platform Deliveroo. The company used a reputational ranking algorithm to prioritise riders when assigning work shifts. The algorithm measured “reliability” and “participation” without accounting for legitimate absences such as illness, childcare responsibilities, disability, or strike action.
In practice, this disproportionately harmed female workers, who are statistically more likely to undertake caregiving responsibilities. The Italian court held that the algorithm indirectly discriminated against women because it treated all cancellations identically, regardless of the reason.
This case reveals a key problem with algorithmic systems: even formally neutral criteria can reinforce existing social inequalities.
The EU has responded through the Platform Work Directive 2024/2831, which introduces stronger protections for platform workers. The Directive creates safeguards against harmful algorithmic management, including:

• Human oversight of automated decisions
• Transparency obligations for digital platforms
• Restrictions on certain forms of automated monitoring
• Rights to explanation and review of algorithmic decisions

Still, critics argue that the law does not go far enough, particularly in addressing gendered and intersectional harms.

Banking and Insurance: Credit Scores, Risk, and Invisible Bias

Banking and insurance are increasingly dependent on automated decision-making. AI systems can assess loan applications, calculate creditworthiness, predict financial risk, and determine insurance pricing far faster than human employees. However, efficiency can come at the expense of fairness.
One landmark case in this area is the SCHUFA decision before the Court of Justice of the European Union (CJEU).
SCHUFA, a German credit-scoring company, used automated systems to generate scores predicting whether individuals were likely to repay loans. When a woman was denied a loan based on her score, she requested detailed information about how the score had been calculated. SCHUFA refused to disclose the methodology.
The CJEU ultimately ruled that the scoring system constituted “automated individual decision-making” under Article 22 GDPR because the automated score significantly affected the applicant’s legal and economic position.
Yet transparency alone does not eliminate discrimination.
Many credit-scoring systems rely on variables such as income, employment history, or geographic location. Although these factors may appear neutral, they can indirectly disadvantage women, ethnic minorities, disabled persons, or economically marginalised communities.
This creates a difficult legal challenge because indirect discrimination is often harder to prove and easier to justify than direct discrimination.

Targeted Advertising and Online Retail: The Hidden World of Personalised Discrimination

Modern online platforms collect enormous quantities of personal data. This data allows companies to personalise advertisements, prices, and product recommendations with extraordinary precision. However, personalisation can quickly become discriminatory.
For example, shopping preferences may be used as proxies for gender, race, religion, or age. Certain consumers may see different prices, products, or opportunities based on inferred characteristics rather than explicit choices.
EU anti-discrimination law only partially addresses these risks.
While EU law prohibits discrimination in access to goods and services on grounds such as gender or racial origin, protection remains fragmented. Some forms of discrimination, particularly those based on age or religion in private commercial settings, may fall outside the scope of existing secondary legislation.
Data protection law offers some indirect safeguards.
Meta Platforms Inc and Others v Bundeskartellamt illustrated how GDPR protections can restrict the processing of sensitive personal data used for targeted advertising. The CJEU held that merely visiting websites or interacting with online content does not automatically mean users consent to their sensitive personal data being made public.
Still, GDPR protections have limits.
Users frequently consent to extensive data processing through terms and conditions they never fully read. Moreover, some protected characteristics are not comprehensively covered under GDPR’s rules on sensitive data.
As a result, discriminatory profiling can still occur despite formal consent mechanisms.

The Netherlands and the Rise of AI Accountability

Several major scandals in the Netherlands dramatically accelerated European discussions around algorithmic accountability.

The Childcare Benefits Scandal

Perhaps the most infamous example involved the Dutch childcare benefits system.
Dutch tax authorities used algorithms to detect welfare fraud among recipients of childcare benefits. The system disproportionately targeted individuals with immigrant backgrounds, leading to devastating financial and social consequences for thousands of families, leading to the resignation of the ‘Rutte III’ government

The SyRI Case

Another major controversy involved SyRI (System Risk Indication), a government tool designed to identify potential welfare fraud.
SyRI combined large datasets to flag individuals considered “high risk.” In practice, the system disproportionately targeted low-income and immigrant communities.
The Hague District Court ruled that the system violated the right to private life under Article 8 ECHR, criticising its lack of transparency and disproportionate effects.

The DUO Student Fraud System

A third scandal emerged from the Dutch student finance agency DUO.
The agency used risk-scoring algorithms to identify suspected student benefit fraud. Although the system did not directly use race or ethnicity, it relied heavily on geographic indicators that functioned as proxies for ethnicity.
Dutch authorities later concluded that the system was discriminatory and unlawful.
Together, these scandals demonstrated a recurring pattern:

• Algorithms can replicate structural inequalities
• Seemingly neutral criteria can operate as discriminatory proxies
• Lack of transparency makes accountability difficult
• Vulnerable groups often bear the greatest risks

The FRAIA: A New Model for Preventing Harm

In response to these scandals, the Netherlands developed the Fundamental Rights and Algorithms Impact Assessment (FRAIA).
Rather than focusing only on punishment after harm occurs, the FRAIA is designed as a preventative tool to help organisations identify risks to fundamental rights before deploying algorithmic systems.
The FRAIA asks organisations to evaluate four major areas:

1. Why is the algorithm being used?

This stage examines:

• The problem the algorithm aims to solve
• Whether non-digital alternatives exist
• The public values involved
• The legal basis for using the system
• Responsibility and accountability structures

2. What does the algorithm do?

This stage focuses on:

• Data quality
• Embedded biases
• Security risks
• Transparency and explainability
• Ownership and control of the system

3. How will the algorithm be implemented?

This section considers:

• Human oversight
• Context-specific risks
• Communication with affected individuals
• Auditing and review mechanisms
• Risks of stigmatisation and discrimination

4. Which fundamental rights are affected?

The FRAIA requires organisations to assess how algorithms may interfere with rights such as:

• Equality and non-discrimination
• Privacy and data protection
• Freedom of expression
• Procedural fairness

Importantly, the FRAIA openly acknowledges that many algorithms inherently produce unequal treatment because they rely on categorisation and profiling.
The framework therefore requires organisations to weigh the seriousness of rights infringements against the objectives pursued.

The EU AI Act and the FRIA

The European Union has now taken a major step toward institutionalising this approach through the EU AI Act. Article 27 of the AI Act introduces the Fundamental Rights Impact Assessment (FRIA) for certain high-risk AI systems.
This obligation applies not only to public bodies but also to several categories of private actors, including:

• Providers of public services
• Credit institutions using AI for credit scoring
• Insurance providers using AI for risk assessment and pricing

The FRIA represents an important shift in European law.
For years, debates about fundamental rights focused primarily on state action. The AI Act recognises that private corporations deploying AI systems can exercise enormous power over individuals’ lives.
The FRIA therefore reflects a broader trend toward extending fundamental rights obligations into private contractual relationships.

Fundamental Rights Beyond the State

One of the most significant legal developments underlying these changes is the growing recognition that fundamental rights can apply horizontally between private parties.
Historically, human rights protections primarily regulated relationships between individuals and the state. However, modern private corporations, especially digital platforms and AI developers, now wield immense influence over employment, access to credit, insurance, housing, and information.
The CJEU’s decisions in cases such as Egenberger and Bauer confirmed that certain provisions of the EU Charter can create direct obligations for private actors.
This matters enormously in the AI context. If private companies deploy AI systems that discriminate against individuals in areas regulated by EU law, fundamental rights protections may apply directly to those contractual relationships.
The AI Act reinforces this development by requiring certain private actors to proactively assess the impact of AI systems on fundamental rights.

Can Impact Assessments Really Prevent Discrimination?

Despite these legal developments, important concerns remain.
Impact assessments such as the FRAIA and FRIA are ultimately risk-management tools. They do not automatically eliminate discrimination.
Much depends on:

• How honestly organisations conduct assessments
• The quality of oversight and enforcement
• The transparency of AI systems
• The willingness of courts and regulators to intervene

There is also a structural problem within anti-discrimination law itself.
Direct discrimination is generally prohibited outright. Indirect discrimination, however, can often be justified if organisations can show a legitimate aim and proportionate means. Because many algorithmic harms take the form of indirect discrimination, companies may still defend systems that produce unequal outcomes.
Most victims of discrimination never bring legal claims. This makes prevention especially important.

Final Thoughts

Artificial intelligence is transforming contractual relationships across employment, finance, insurance, and digital commerce. The benefits are undeniable, but so are the risks.
The rise of algorithmic discrimination demonstrates that technology is never truly neutral. Algorithms reflect the data, assumptions, and institutional priorities embedded within them. The Dutch FRAIA and the EU’s FRIA under the AI Act represent important attempts to shift the focus from reacting to discrimination after harm occurs toward identifying risks before systems are deployed. Whether these frameworks succeed will depend on meaningful enforcement, transparency, and judicial oversight.
As AI systems become more deeply integrated into everyday life, the central legal challenge will not simply be whether algorithms are efficient. It will be the extent to which they remain compatible with equality, accountability, and fundamental rights in a democratic society.