Microsoft has unveiled Muse, a unique generative AI model that's described as the first "World and Human Action Model (WHAM for short)." In essence, the model will be able to generate game visuals and input controller actions.
The technology was developed by Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Xbox Games Studios’ Ninja Theory. For the research, the 4v4 game Bleeding Edge was utilized.
In Microsoft's blog post, multiple gameplay clips using Muse were shown. Katja Hofmann, Senior Principal Research Manager and lead of the Microsoft Research Game Intelligence team explained that each sequence was created by prompting the model with only 10 frames of action (human gameplay) and controller actions of the entire sequence.
In short, the idea behind this continuously learning AI is to give developers a better look at how players could interact with their games without the need to build entire prototypes.
Microsoft Gaming CEO also mentioned in a YouTube video that this could open up the possibility of helping game preservation efforts in the distant future.
"One of the things we care a lot about at Xbox is game preservation. I think about an opportunity to have models learn about older games, games that were maybe tied to unique pieces of hardware (...) and really make them portable to any platform where these models can run."
Before you raise your pitchforks, Ninja Theory Studio Head Dom Matthews did mention that this technology is aimed at helping teams with their development, not directly "using AI to generate content."
"It's actually about creating workflows and approaches that allow our team of a hundred creative experts to do more." Matthews then added that Ninja Theory has not used Muse in their game development so far, reaffirming that if they do so in the future they don't intend to use it to create content.
For those interested, Microsoft has open-sourced the weights, sample data, and the WHAM Demonstrator executable via the Azure AI Foundry.