Last month Google GameNGen’s AI model showed that generalized image diffusion techniques can be used for generate an acceptable, playable version MooseNow, scientists are using similar techniques with a model called MarioVGG to see if AI can generate reliable video. Super Mario Bros. in response to information entered by the user.
Results MarioVGG Model-available as pre-printing paper published by a cryptocurrency-related AI company Virtual Protocol—still displays many noticeable bugs and is too tardy to compare to real-time gameplay. However, the results show that even a narrow model can infer impressive gameplay physics and dynamics by simply studying a bit of video and input data.
The researchers hope this is the first step toward “producing and demonstrating a reliable and controllable video game generator” or perhaps even “completely replacing game development and game engines with video generation models” in the future.
Watching 737,000 frames of Mario
To train their model, researchers MarioVGG (GitHub users) erniechew AND Brian Lim are listed as co-authors) started with public dataset With Super Mario Bros. gameplay with 280 “levels” of input and image data organized for machine learning (level 1-1 was removed from the training data so that images from it could be used in evaluation). The 737,000+ individual frames in this dataset were “preprocessed” into 35-frame chunks so the model could start learning what the overall immediate results of different inputs look like.
To “simplify the gameplay situation,” the researchers decided to focus on just two potential inputs in the dataset: “run right” and “run right and jump.” Even this narrow set of movements posed some challenges for the machine-learning system, because the preprocessor had to look back several frames before the jump to determine whether and when the “run” had begun. Any jumps that involved midair changes (i.e., the “left” button) also had to be discarded, because “this would introduce noise into the training dataset,” the researchers write.
After pre-processing (and about 48 hours of training on a single RTX 4090 graphics card), the researchers applied the standard turn AND denoising the process of generating recent video frames from a unchanging image of the game’s opening and input text (in this narrow case, “run” or “jump”). While these generated sequences only last a few frames, the last frame of one sequence can be used as the first of a recent sequence, allowing gameplay videos of any length to be created that still show “coherent and consistent gameplay,” according to the researchers.
Super Mario 0.5
Even with all this setup, MarioVGG doesn’t exactly generate silky-smooth video that’s indistinguishable from a real NES game. For performance, the researchers reduce the output frame resolution from the NES’s 256×240 to a much blurrier 64×48. They also condense 35 frames of video into just seven generated frames, which are spaced “equally,” creating a “gameplay” video that looks significantly worse than the real game output.
Despite these limitations, the MarioVGG model still struggles to come close to generating real-time video at this stage. The single RTX 4090 used by the researchers took six whole seconds to generate a six-frame video sequence, which is just over half a second of video, even with the extremely narrow frame rate. The researchers admit that this is “not practical or friendly for interactive video games,” but they hope that future optimizations to the quantization of weights (and perhaps the operate of more computational resources) can improve this speed.
With these limitations in mind, however, MarioVGG can create a fairly believable video of Mario running and jumping from a unchanging opening image similar to Game creator Genie by Google. The model was even able to “learn the physics of the game solely from video frames in the training data without any explicit, hard-coded rules,” the researchers write. This includes inferring behaviors like Mario falling when he runs off a cliff edge (in plausible gravity) and (usually) stopping Mario’s forward motion when he’s next to an obstacle, the researchers write.
While MarioVGG focused on simulating Mario’s movements, the researchers found that the system could effectively hallucinate recent obstacles for Mario as the video scrolled through an imagined level. These obstacles “are consistent with the game’s graphical language,” the researchers write, but they currently can’t be influenced by user prompts (e.g., place a pit in front of Mario and make him jump over it).
Just figure it out
Like all probabilistic AI models, MarioVGG has a frustrating tendency to sometimes produce completely useless results. Sometimes that means simply ignoring user prompts (“we observe that the input action text is not followed all the time,” the researchers write). Other times, it means hallucinating Obvious visual bugs: Mario sometimes lands inside obstacles, runs through obstacles and enemies, flashes different colors, shrinks/grows from frame to frame, or completely disappears for a few frames and then reappears.
One particularly absurd video shared by researchers shows Mario falling through a bridge, transforming into Cheep-Cheep, then flying back through the bridges and transforming back into Mario. This is something we would expect from Wonder Flower, not the original AI movie Super Mario Bros.
The researchers speculate that more training on “more diverse gameplay data” could aid address these significant issues and aid their model simulate more than just running and jumping relentlessly to the right. Still, MarioVGG provides a fun proof of concept that even narrow training data and algorithms can create decent initial models of basic games.
This story originally appeared on Ars Technica.