Tests
Our AI system surpasses state-of-the-art approaches to geometric problems by advancing artificial intelligence reasoning in mathematics
Reflecting the Olympic spirit of archaic Greece, International Mathematical Olympiad is a contemporary arena for the most talented high school mathematicians in the world. The competition not only showcases youthful talents, but has also become a testing ground for advanced artificial intelligence systems in mathematics and reasoning.
In an article published today in Nature, we present AlphaGeometry, an artificial intelligence system that solves sophisticated geometric problems at a level similar to a human Olympic gold medalist – a breakthrough in artificial intelligence performance. In a benchmark of 30 Olympiad geometry problems, AlphaGeometry solved 25 within the standard Olympiad time limit. By comparison, the previous state-of-the-art system solved 10 of these geometry problems, and the average gold medalist solved 25.9 problems.
In our benchmark set of 30 Olympic geometry problems (IMO-AG-30), based on the 2000–2022 Olympics, AlphaGeometry solved 25 problems within the competition time limits. This result is similar to the average result of gold medalists in the same problems. The previous state-of-the-art approach, known as the “Wu method”, solved 10.
AI systems often struggle with sophisticated geometry and math problems due to a lack of reasoning skills and training data. The AlphaGeometry system combines the predictive power of a neural language model with a rule-based deduction engine that works together to find solutions. By developing a method to generate a huge pool of synthetic training data – 100 million unique examples – we can train AlphaGeometry without any human demonstrations, bypassing the data bottleneck.
With AlphaGeometry, we demonstrate artificial intelligence’s growing ability to reason logically and discover and verify novel knowledge. Solving geometry problems at the Olympic level is an significant milestone in developing deep mathematical reasoning on the path to more advanced and general artificial intelligence systems. We are open-sourcing AlphaGeometry code and modeland we hope that, together with other tools and approaches to generating and training synthetic data, it will lend a hand open up novel possibilities in mathematics, science and artificial intelligence.
Now it makes complete sense to me that AI researchers first try their hand at IMO geometry problems, because finding solutions to them works a bit like chess in the sense that at each step we have a rather tiny number of reasonable moves. But I still think it’s amazing that they managed to do it. This is an impressive achievement.
Ngo Bao Chau, Fields medalist and IMO gold medalist
AlphaGeometry takes a neurosymbolic approach
AlphaGeometry is a neurosymbolic system consisting of a neural language model and a symbolic deduction engine that work together to find evidence for sophisticated geometry theorems. Close to the idea of ”thinking, fast and slow”, one system provides quick, “intuitive” ideas, and the other, more thoughtful, rational decision-making.
Because language models excel at identifying general patterns and relationships in data, they can quickly predict potentially useful constructs, but they often lack the ability to rigorously reason about or explain their decisions. In turn, symbolic deduction engines are based on formal logic and use clear rules to draw conclusions. They are rational and easy to explain, but can be “leisurely” and inflexible – especially when we solve large, complex problems on our own.
The AlphaGeometry language model guides the symbolic deduction engine towards plausible solutions to geometric problems. Olympic geometry problems are based on diagrams that require the addition of novel geometric structures before they can be solved, such as points, lines, or circles. The AlphaGeometry language model predicts which novel constructs will be most useful to add, based on an infinite number of possibilities. These clues lend a hand fill in the gaps and allow the symbolic engine to make further inferences about the diagram and get closer to a solution.
AlphaGeometry solves a basic problem: Given a problem diagram and its theorem assumptions (left), AlphaGeometry (center) first uses its symbolic engine to derive novel statements about the diagram until a solution is found or novel statements are exhausted. If no solution is found, the AlphaGeometry language model adds one potentially useful construct (blue), opening novel deduction paths for the symbolic engine. This loop continues until a solution is found (right). In this example, only one structure is required.
AlphaGeometry solves the Olympic problem: problem 3 of the 2015 International Mathematical Olympiad (left) and a shortened version of the AlphaGeometry solution (right). Blue elements are added structures. The AlphaGeometry solution consists of 109 logical steps.
Generate 100 million synthetic data examples
Geometry is based on the understanding of space, distance, shape and relative positions and is fundamental to art, architecture, engineering and many other fields. People can learn geometry with pen and paper by examining diagrams and using existing knowledge to discover novel, more sophisticated geometric properties and relationships. Our synthetic data generation approach emulates this knowledge building process at scale, allowing us to train AlphaGeometry from scratch, without any human demonstrations.
Using highly parallel computing, the system started by generating a billion random diagrams of geometric objects and exhaustively derived all the relationships between the points and lines in each diagram. AlphaGeometry found all the evidence contained in each diagram and then worked backward to find out what additional constructions were needed to obtain that evidence. We call this process “symbolic deduction and tracing.”
Visual representations of synthetic data generated by AlphaGeometry
This huge pool of data was filtered to exclude similar examples, resulting in a final training dataset consisting of 100 million unique examples of varying difficulty, nine million of which contained the added constructs. With so many examples showing how these constructions led to proofs, the AlphaGeometry language model is able to make good suggestions for novel constructions for Olympiad geometry problems.
Pioneering mathematical reasoning with artificial intelligence
The solution to each Olympic problem provided by AlphaGeometry has been computer-checked and verified. We also compared its results with previous artificial intelligence methods and with human performances in the Olympics. Additionally, Evan Chen, math coach and former Olympic gold medalist, evaluated selected AlphaGeometry solutions for us.
Chen said: “AlphaGeometry’s results are impressive because they are both verifiable and transparent. Previous evidence-based AI solutions to competitive problems have sometimes been hit or miss (results are only sometimes correct and require human review). AlphaGeometry does not have this weakness: its solutions have a machine-verifiable structure. Despite this, its result is still human readable. You can imagine a computer program solving geometric problems using brute-force coordinate systems: pages of thought and pages of tedious algebra calculations. AlphaGeometry is not that. It uses classic geometry principles with angles and similar triangles, just like students do.”
AlphaGeometry’s results are impressive because they are verifiable and see-through… It uses classic geometry principles with angles and similar triangles, just as students do.
Evan Chen, math coach and Olympic gold medalist
Because each Olympiad contains six problems, only two of which typically focus on geometry, AlphaGeometry can only be applied to one-third of the problems in a given Olympiad. Nevertheless, its geometry capabilities alone make it the first AI model in the world to exceed the IMO bronze medal threshold in 2000 and 2015.
In geometry, our system is approaching IMO gold medalist standards, but we have our eyes on an even bigger prize: advances in the reasoning of next-generation artificial intelligence systems. Given the broader potential of training AI systems from scratch using large-scale synthetic data, this approach could shape the way future AI systems discover novel knowledge, both in mathematics and beyond.
AlphaGeometry builds on the work of Google DeepMind and Google Research to pioneer mathematical reasoning with artificial intelligence – from discovering the beauty of pure mathematics to solving mathematical and scientific problems using language models. And most recently, we introduced FunSearch, which made the first discoveries on open problems in mathematical sciences using large-language models.
Our long-term goal remains to build AI systems that can generalize across mathematical domains, developing the sophisticated problem-solving and reasoning methods on which general AI systems will depend, while at the same time pushing the boundaries of human knowledge.