Language

Lumine: Building Generalist Agents in 3D Open Worlds

We introduce Lumine, a generalist AI agent trained within Genshin Impact that can perceive, reason, and act in real time, completing hours-long missions within complex 3D open-world environments.

Mondstadt Region: Lumine successfully completes a full five-hour, three-act main storyline playthrough of Mondstadt. Even though its reasoning ability is trained only on the content of the first act, Lumine autonomously carries out all story progressions.

Liyue Region: In the completely out-of-distribution region of Liyue, Lumine demonstrates impressive generalization, successfully reaching and completing the main storyline in Liyue Harbor, and visiting the Adeptus dwelling deep within the mountains.

Beyond Genshin: Lumine exhibits exceptional cross-game generalization ability. Without any additional fine-tuning and modification, Lumine directly completes the entire chapter of Herta Space Station in Honkai: Star Rail within 7 hours; similarly, it successfully finishes approximately 100 minutes of main storyline content in Wuthering Waves.

A Scalable Recipe for Building General-Purpose Agents

Lumine adopts a human-like interaction paradigm that unifies perception, reasoning, and action in an end-to-end manner, powered by a vision-language model. It processes raw pixels at 5 Hz to produce precise 30 Hz keyboard–mouse actions and adaptively invokes reasoning only when necessary.

Lumine

Lumine makes full use of the collected raw gameplay data, following a curriculum that builds skills incrementally:

  • 1731 hours of human gameplay for pre-training to master action primitives;
  • 200 hours of instruction following data to ground control in language;
  • 15 hours of reasoning data to enable adaptive thinking.

The resulting model can not only complete hours-long missions but also follow diverse instructions to accomplish a broad spectrum of tasks in both 3D open-world exploration and 2D GUI manipulation across collection, combat, puzzle-solving, and NPC interaction. Genshin's diverse gameplay offers the perfect setting to showcase these abilities across different types of activities.

Lumine

Combat

Benefiting from large-scale pretraining, Lumine has mastered the essential combat skills, dynamically tracking enemies, accurately striking distant targets with a bow, seamlessly switching characters to perform combo attacks, and efficiently locating and opening unlocked treasure chests.

Defeat the enemies ahead and open the chest

Clear the Domain

Complete the Daily Commission: Defeat all enemies

Boss Fight

Besides regular combat, Lumine also shows a strong understanding of boss mechanics and the ability to respond effectively. It can skillfully evade powerful attacks and employ appropriate strategies to defeat them.

Defeat the Electro Hypostasis

Defeat the Electro Hypostasis

Defeat the Electro Hypostasis

Defeat Stormterror

Defeat Stormterror

Defeat the Anemo Hypostasis

Puzzle

Lumine can handle various challenges and puzzles in the game, which typically require a thorough understanding of game mechanics, strong spatial reasoning skills, and precise low-level control.

Fly along the Wind Current to collect the Anemoculus

After defeating the floating Anemo Slime, open the chest

Hit the Iron Chunk and collect the dropped Iron Chunk

Open the chest wrapped in thorns ahead

Activate the Elemental Monument using the corresponding element

Complete the Time Trial Challenge ahead: Open the chest within the time limit

NPC Interaction

Lumine exhibits reliable instruction-following ability, consistently interacting with designated NPCs within crowds, laying a solid foundation for accomplishing long-term missions.

Talk to NPC Grace

Talk to NPC Monroe

Talk to NPC Sayid

GUI Manipulation

Beyond open-world exploration, Lumine can also perform efficient GUI operations through human-like relative mouse movements, achieving a unified interaction between 2D interfaces and the 3D world, a capability that is crucial for generalist agents.

Cook Sweet Madame

Teleport using a Teleport Waypoint

Change the character's weapon

Complex Instruction Following

Meanwhile, Lumine has demonstrated strong in-context learning abilities. When provided with prior task information or more detailed decomposition steps within the instruction, Lumine can successfully complete a range of tasks that it was previously unable to perform.

Climb the stone pillar on the right and, once you reach the top, collect the blue Anemoculus floating in the air on the left

Switch to Kaeya, continuously use his Elemental Skill (E Skill) to freeze the water surface, and collect the Anemoculus floating ahead

Collect the Wind Anemograna to activate a Wind Current, then enter the Wind Barrier to open the chest