GAMESLASHAI built your game—but the spikes do nothing? SWE-Game tests whether it really works
By GameSlash
SWE-Game tested AI agents on 247 tasks from 41 Godot games. Learn why opening a project is not enough, and how creators can test gameplay rules.
AI can deliver a game project that opens successfully, but that does not guarantee it plays like the game you asked for. A platformer can look correct while its spikes do no damage, or a level can load without the intended win condition. SWE-Game is a research benchmark built to measure exactly that gap.
The study was first submitted September 27, 2026 and its third revision appeared October 7. Researchers assembled 247 development tasks from 41 runnable Godot games spanning 13 types. It evaluates complete game creation from a short prompt or design document, filling in unfinished games, fixing injected bugs and porting Godot projects to Unity.
How the researchers checked an AI-built game
Instead of grading screenshots alone, the benchmark runs the game, checks internal state, replays player inputs and tests whether the requested behavior happens. This matters because a visually convincing character can still fail to collide, lose health or complete a level properly.
In paper v3, the highest Brief-to-Game score is 50.38 out of 100, and the best scores across the three creation/completion categories remain under 60. On 1,200 human-labeled behavior assertions from 100 agent-built games, runtime checks achieve 92.59% balanced accuracy versus 78.41% for a video-based judge. The accuracy measures how well the tests judge behavior, not the percentage of games an agent successfully builds.
A practical checklist for Vibe Coding projects
Ask the coding agent to build a tiny playable loop first, then have it demonstrate concrete test cases: walk into a hazard and lose HP, collect an item and update inventory, reach an exit and finish the stage. Run and observe the game yourself. If something fails, give the agent a clear reproduction sequence and ask it to fix the behavior rather than regenerate the whole project.
What to watch out for
The benchmark uses controlled reference projects, scripts and evaluation tools; it is not a one-click game maker. Real projects have different art, inputs and integrations, so do not treat its model scores as a direct buyer's guide. The transferable lesson is to verify actions and game state, not only a good-looking screenshot.
Original SWE-Game paper · Figures and methodology · Figure license
For reproduction, note that the public v1.0.0 release contains 246 tasks, while paper v3 reports 247. Keep the release inventory and paper results distinct. Release documentation