The mixed assessments of one of the Internet’s main resources, Wikipedia, are reflected in the somewhat dystopian article “List of Wikipedia Scandals”. However, billions of users regularly visit the online, anonymously editable, encyclopedic knowledge bank to get just about anything. It is complex to reliably trace how this non-authoritative source influences our discourse and decisions. But a novel study tries to measure how knowledge gleaned from Wikipedia can be used in one specific field: the courts.
A team of researchers led by Neil Thompson, a scientist at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), recently conducted a cordial experiment: creating novel legal articles on Wikipedia to see how they influence judges’ legal decisions. They started by compiling over 150 novel Wikipedia articles on Irish Supreme Court decisions written by law students. Half of them were randomly selected and placed on the Internet, where they could be used by judges, clerks, lawyers, etc. – the “treatment” group. The other half were kept offline, and this second group of cases provided an alternative basis for what would have happened if there had been no Wikipedia article about it (the “control”). They then looked at two measures: whether cases were more likely to be cited as precedents in subsequent court decisions, and whether the arguments in court decisions reflected the linguistic content of the novel Wikipedia pages.
It turned out that published articles tipped the scales: getting a public article on Wikipedia increased the number of case citations by more than 20 percent. The escalate was statistically significant, and the effect was particularly forceful in cases that supported the reasoning of the referee in his decision (but not vice versa). Unsurprisingly, the escalate was greater for appointments by the lower courts – the High Court – and largely absent for appointments by the appellate courts – the Supreme Court and the Court of Appeal. The researchers suspect that this shows that Wikipedia is used more often by judges or officials who have heavier workloads and for whom the convenience of Wikipedia is more attractive.
“To our knowledge, this is the first randomized field experiment to examine the influence of legal sources on judicial behavior. And because randomized experiments are the gold standard in this type of research, we know that the effect we observe is causal, not just correlation,” says Thompson, the study’s lead author. “The fact that we wrote up all these cases, but only the ones that ended up on Wikipedia were the ones that won the proverbial coin toss, allows us to show that Wikipedia influences both what judges cite and how how they record their decisions.”
“Our results also highlight an important public policy issue,” adds Thompson. “Having such a widely used source as Wikipedia, we want to make sure that we are building institutions that provide the highest quality of information. “A much greater concern is that judges or their staff are found to be using Wikipedia if the information they find there is not reliable.”
AND paper a description of the study was published in “The Cambridge Handbook of Experimental Jurisprudence” (Cambridge University Press, 2022). Thompson was joined in the article by Brian Flannigan, Edana Richardson and Brian McKenzie of Maynooth University in Ireland, and Xueyun Luo of Cornell University.
The researchers’ statistical model essentially compared how much citation behavior changed in the treatment group (first difference: before and after) and compared to the change that occurred in the control group (second difference: treatment vs. control group).
In 2018, Thompson first became interested in the idea of proving the causal role that Wikipedia plays in shaping knowledge and behavior by looking at how it shapes academic learning. It turns out that adding scientific articles, in this case about chemistry, changed the way the topic was discussed in the scientific literature, and scientific articles added as references to Wikipedia also received more academic citations.
This prompted Brian McKenzie, an associate professor at Maynooth University, to make a call. “I was working with students on adding articles to Wikipedia when I read Neil’s research on Wikipedia’s impact on scientific research,” McKenzie explains. “There were only a few Irish Supreme Court cases on Wikipedia, so I contacted Neil to ask if he would like to design another iteration of his experiment using court cases.”
The Irish legal system has proven to be an ideal testing ground because it has fundamental similarities with other national legal systems such as the United Kingdom and the United States – it operates within a hierarchical court structure in which decisions of higher courts are then binding on lower courts. Additionally, Wikipedia has relatively few articles on Irish Supreme Court decisions compared to US Supreme Court decisions – researchers increased the number of such articles tenfold over the course of the project.
In addition to looking at the case references in the decisions, the team also analyzed the language used in the written decision using natural language processing. They discovered the linguistic fingerprints of the Wikipedia articles they created.
So what might this impact look like? Suppose A sues B in federal district court. A claims that B is liable for breach of contract; B acknowledges the facts presented by A, but maintains that they did not give rise to any contract between them. The appointed judge, aware of the enormous amount of work already entrusted to her officials, decides to conduct her own research. Considering the parties’ statements, the judge formulates a preliminary opinion that the contract was not actually concluded and that the judge should pronounce a judgment for the accused. To write his official opinion, the judge googles some previous judgments cited in file B that seem similar to the case between A and B. Confirming their similarity by reading the relevant case summaries on Wikipedia, the judge paraphrases some of the text of the Wikipedia entries in the draft opinion in to complete the analysis. The judge then issues a ruling and publishes his opinion.
“The text of the court judgment itself will determine the direction of the law, because it will become a source of precedent for the subsequent process of judicial decision-making. “Future lawyers and judges will look back at this written judgment and use it to decide what its implications are so that they can treat ‘like’ cases in a similar way,” says co-author Brian Flanagan. “If, as this experiment shows, the text itself is influenced by anonymous online content, that is a problem. Given the many potential cracks that have opened in our “information superhighway”: the Internet, it is conceivable that this vulnerability could potentially lead to information manipulation by hostile actors. “If it is already based on readily available analysis of legal issues, it behooves the legal community to accelerate its efforts to ensure that such analysis is both comprehensive and expert.”