“People receiving social benefits reserved for disabled people [the Allocation Adulte Handicapé, or AAH] are directly targeted by a variable in the algorithm,” says Bastien Le Querrec, legal expert at La Quadrature du Net. “There is an increased risk for people receiving AAH and working.”
The groups say it indirectly discriminates against single mothers, who are statistically more likely to have sole custody, because it also ranks single-parent families higher than single-parent families. “According to the criteria of the 2014 version of the algorithm, the score is higher for beneficiaries who have been divorced for less than 18 months,” says Le Querrec.
Changer de Cap says that following its investigation, it was approached by both single mothers and disabled people seeking assist.
CNAF, which distributes financial assistance including housing, pensions and child benefits, did not immediately respond to a request for comment or to WIRED’s inquiry whether the algorithm currently used has changed significantly since the 2014 version.
As in France, human rights groups in other European countries say they are subjecting members of society with the lowest incomes to intense surveillance – often with sedate consequences.
When tens of thousands of people in the Netherlands – many of them come from this country Ghanaian communities — were falsely accused of defrauding the child benefit system, in addition to being ordered to pay back the money the algorithm says they allegedly stole. Many of them say they have fallen into a spiral of debt and damaged credit ratings.
The problem is not the way the algorithm was designed, but its application in the social welfare system, says Soizic Pénicaud, lecturer in AI policy at Sciences Po Paris, who previously worked for the French government on the transparency of public sector algorithms. “The use of algorithms in the context of social policy carries much greater risks than benefits,” he says. “I have not seen any example in Europe or the world where these systems have been used with positive results.”
The case has ramifications beyond France. Social care algorithms are expected to be an early test of enforcement of the EU’s modern AI rules when they come into force in February 2025. From there, “social scoring” – the exploit of AI systems to assess people’s behavior and then subject some of it to evaluation for harmful treatment – will be prohibited all over the block.
“In my opinion, many of these welfare systems that detect fraud may actually be using social scoring,” says Matthias Spielkamp, co-founder of the nonprofit Algorithm Watch. However, public sector representatives are unlikely to agree with this definition, and disputes over how to define these systems will likely end up in court. “I think that’s a very difficult question,” Spielkamp says.