P
Pac-Bench
Open Source🔄 Alt to SWE-benchTest LLM spatial reasoning by one-shot building a Pac-Man game
🐳 Self-Hostable🔓 No Sign-up⚡ Traction Score: 71/100★17 Stars
⚡
Quick Installation / Run
git clone https://github.com/jonclegg/pacman-bakeoff.git💡 What Problem Does Pac-Bench Solve?
Pac-Bench is an interactive evaluation framework designed to test frontier models on complete game development and spatial reasoning tasks. It challenges LLMs to one-shot build a fully working Pac-Man game, providing a standardized benchmark for complex code generation.
CategoryAI & Machine Learning
Commercial AlternativeSWE-bench
Self-HostableYes (Docker/Bare-metal)
Sign-up BarrierNo (Instant Access)
License ModelOpen Source
Discovery Sourcehackernews
⚖️ Pros & Cons Analysis
🟢 Key Advantages
- Tests holistic application design rather than isolated algorithmic snippets
- Visually engaging and immediately intuitive evaluation metric (playable game)
- Bridges the gap between raw text generation and tangible software execution
🟡 Things to Consider
- Narrow domain focus limited specifically to arcade game clones
- Relies on deterministic interpretation of code correctness which can be tricky for generative outputs
⚡ Core Architecture & Key Capabilities
01One-Shot Game Generation
Challenges models to generate an entire playable Pac-Man game in a single prompt response.
02Spatial Reasoning Assessment
Evaluates how well models understand coordinate geometry, collision detection, and grid mechanics.
03Interactive Evaluation
Provides instant feedback loops to verify if the generated code results in an executable, playable game.
🎯 Practical Applications & High-Value Use Cases
Scenario 01
Benchmarking the spatial reasoning and code synthesis limits of newly released frontier LLMs
Scenario 02
Comparing coding agent capabilities across complex, multi-component application development
Scenario 03
Researching failure modes of LLMs when handling game loop architecture and state management
🔄 Why Choose Pac-Bench Over SWE-bench?
Unlike SWE-bench which focuses on real-world GitHub issues, Pac-Bench specifically targets spatial reasoning and end-to-end creative code generation in a controlled sandbox environment.
🎯 Target Audience & Who is this for?
AI researchers, ML engineers, and developer tool creators evaluating frontier code generation models.
Top Related Alternatives in AI & Machine Learning
Compare other trending developer tools and open-source projects in this space.