What is it?
Unity × Scope is two things at once: an immersive cyberpunk experience where the game world is generated in real time by AI, and a reusable toolkit that makes it easier to build that kind of experience in Unity.
The system captures live gameplay footage enriched with depth data and character information, streams it via WebRTC to a Scope pipeline running on RunPod, and feeds the AI-transformed output back into the engine — live, with no recording or post-processing step. What you see on screen is not the game as rendered. It is the game as imagined by a video diffusion model, re-rendered every frame.
The Idea
Scope is an open-source engine by Daydream for running and customising real-time, interactive AI video pipelines. It supports models like LongLive, Krea Realtime, Stream Diffusion V2, and Memflow — each transforming a live video input into something that looks radically different from the source.
The question this project explores: what if a game world was not designed visually, but prompted visually? Instead of placing assets and painting textures, you describe what you want the world to look like — and the AI renders it around you in real time.
Development: Four Checkpoints
The project was built across four milestones during the Daydream Scope Workshop 25.
Checkpoint 1 established the core pipeline: Unity 6 with Universal Render Pipeline, depth extraction, WebRTC streaming to RunPod, and a basic AI pass-through. The earliest screenshots show a simple environment with the first AI transformation applied.
Checkpoint 2 introduced the prompt system. Player movement states (standing, walking, sprinting, jumping) now dynamically alter the prompt sent to the AI model — the world literally shifts in response to how you move through it.
Checkpoints 3 & 4 completed the toolkit with location-based prompt zones, vision-based object recognition, AI NPC conversations via Google Gemini, and a full cyberpunk demo prototype. The final submission was the complete experience.
Key Features
The toolkit ships as a collection of Unity components:
- Real-Time AI Video Streaming — gameplay footage streamed via WebRTC to Scope; AI output displayed live as a full-screen or picture-in-picture overlay
- Depth Extraction with Character Masking — dual-camera depth system isolates the player character; OpenPose skeleton overlay provides pose data to the AI model
- Dynamic Prompt System — prompts update automatically based on the player's current movement state
- Location-Based Prompt Zones — trigger prefix/suffix changes or full prompt replacements when the player enters a zone; cache reset zones prevent the AI from holding onto old context
- Vision-Based Prompt Injection — tagged objects in the scene are described and injected into the live prompt when the player looks at them
- Interactive NPC Chat — Gemini-powered NPCs with configurable personalities; full dialogue UI with dedicated chat and exit states
- Minimap Mode — top-down depth-based perspective toggle
- Multi-State UI System — setup screen, start screen, gameplay, and chat modes
First Place — Scope Workshop 25
The project was submitted to the Daydream Scope Workshop 25, a showcase and competition for creators building with the Scope real-time AI engine. It received the first place award — recognised for the depth of its Unity integration, the quality of the interactive demo, and the practical value of the reusable toolkit it produces.
Technical Requirements
The full experience runs on:
- Unity 6 with Universal Render Pipeline (URP Depth Texture enabled)
- Scope on RunPod — LongLive model with VACE disabled (RTX 5090 or 6000 Pro recommended for real-time performance)
- Google Gemini API for NPC text generation
The toolkit and source code are available on GitHub (work in progress).