Kradle · prepared for the SIMA team
A demo Minecraft eval suite for SIMA
- 30 tasks across 10 empirical difficulty tiers.
- Every task is a natural-language instruction paired with a machine-checkable end state, so we can automatically judge evals at scale.
- The same worlds are playable through a first-person screen feed with keyboard and mouse (SIMA), or through observations in JSON and skill calls in JavaScript with our harness (Gemini/LLMs).
The videos below are elevated replay views of real runs, rendered for human viewing — they are not agent POV. A pixel agent sees the same world first-person.
30 challenges
Baseline scores below are structured state + skill calls, not pixels.
| Challenge | Description | Gemini 3.7 Flashstate + skills | gemma-3-27b-itstate + skills |
|---|---|---|---|
| Tier 13 challenges | |||
| 01 Blue or Red▸ | Capability floor: read one color word, stand on the matching pad. | 3/3 | 3/3 |
| 13 Up the Stairs▸ | Find and climb the single staircase onto a raised platform. | 3/3 | 3/3 |
| 28 Craft Sticks▸ | Craft sticks from planks in the 2x2 inventory grid. | 3/3 | 3/3 |
| Tier 23 challenges | |||
| 40 Smelt It▸ | Smelt raw iron into an ingot with chest, coal, and furnace. | 3/3 | 3/3 |
| 56 Map Reading▸ | Translate a textual mini-map quadrant into world coordinates. | 3/3 | 3/3 |
| 85 Trade Up▸ | Three-hop trade chain: wheat → bread → emerald → delivery. | 3/3 | 3/3 |
| Tier 33 challenges | |||
| 06 Sign Says▸ | The target color lives on an in-world sign, not in the prompt. | 3/3 | 3/3 |
| 42 Look Behind the Wall▸ | The answer is only visible from a lookout room away from the pads. | 3/3 | 3/3 |
| 45 Scout Then Choose▸ | Four rooms, one beacon — entering any room is final. | 3/3 | 3/3 |
| Tier 43 challenges | |||
| 18 Simple Maze▸ | A small walled maze with three turns. | 3/3 | 2/3 |
| 62 Full Craft Chain▸ | Full chain: logs → planks → table → sticks → wooden pickaxe. | 2/3 | 3/3 |
| 97 Minimal Instructions▸ | The task says only 'Escape.' Discover the mechanism yourself. | 3/3 | 2/3 |
| Tier 53 challenges | |||
| 26 Lever Logic▸ | Two levers, one door — only the announced lever works. | 3/3 | 2/3 |
| 35 Three Buttons in Order▸ | Press three buttons in the announced order; wrong presses reset. | 3/3 | 2/3 |
| 68 Lava Crossing▸ | Bridge a lava strip with a small cobblestone budget. | 3/3 | 2/3 |
| Tier 63 challenges | |||
| 09 Count to Choose▸ | Count torches (3 vs 2) to distinguish two identical pads. | 2/3 | 2/3 |
| 41 Vanishing Sign▸ | The clue is shown once, then vanishes — pure memory. | 3/3 | 1/3 |
| 71 Pillar Up▸ | Tower 5 blocks above the ground with 8 dirt and stay there. | 3/3 | 1/3 |
| Tier 73 challenges | |||
| 04 Odd One Out▸ | Spot the pad that differs — the color is never named. | 3/3 | 1/3 |
| 33 Craft a Pickaxe▸ | Craft a wooden pickaxe, then mine through a wall. | 3/3 | 0/3 |
| 93 The Observatory▸ | Climb to observe an encoded answer, then commit below. | 2/3 | 2/3 |
| Tier 83 challenges | |||
| 72 Copy the Shape▸ | Reproduce a model structure exactly on a build plot. | 3/3 | 0/3 |
| 77 Staircase to Heaven▸ | Build any climbable route to a floating platform 8 blocks up. | 3/3 | 0/3 |
| 89 Efficient Courier▸ | Six deliveries, tight clock — plan an efficient route. | 3/3 | 0/3 |
| Tier 93 challenges | |||
| 52 Arithmetic Pad▸ | Compute 7+5 and stand on the matching numbered pad. | 0/3 | 1/3 |
| 64 Farm Hand▸ | Harvest mature wheat only — leave immature crops. | 2/3 | 0/3 |
| 100 The Exam▸ | Capstone: five sequential puzzle chambers behind iron doors. | 2/3 | 0/3 |
| Tier 103 challenges | |||
| 07 Farthest Pad▸ | Relational comparison: choose the farther of two identical pads. | 0/3 | 0/3 |
| 70 Iron Age | Bare hands to iron pickaxe: the full early tech chain. | 0/3 | 1/3 |
| 78 Enclosure▸ | Herd three free-roaming chickens into a fenced pen. | 0/3 | 0/3 |