🐉 Project: Infinite Realm

The Voice-Controlled AI Dungeon Master

Infinite Realm is a full-stack, multimodal web application. It allows a user to play an endless Tabletop Role-Playing Game (like Dungeons & Dragons) using their voice. The system transcribes the player's voice locally, generates the story and artwork using the Gemini API, and displays the living world on a React dashboard.

---

🏗️ System Architecture

The project is split into three main layers: The Client, The Server, and The AI Core.

1. The Interface (Frontend: React)

• Role: The player's viewport into the fantasy world.
• Features:
• The "Push to Talk" Button: Captures the user's microphone audio using the browser's MediaRecorder API and sends the audio file to the backend.
• The Scene Viewer: A large central image container that displays the current location or monster (generated by Gemini).
• The Adventure Log: A scrolling text window showing the history of the conversation (User vs. Dungeon Master).
• Audio Player: Automatically plays the voice of the Dungeon Master when a new response arrives.

2. The Game Engine (Backend: Python / Flask)

• Role: The brain that connects the user to the AI and saves the game.
• Features:
• Audio Receiver: An endpoint (POST /api/action) that receives the .wav or .webm audio file from React.
• Local Transcriber: Uses a local AI model to turn that audio into text.
• The Prompt Engineer: Wraps the user's text in a hidden prompt (e.g., "The player says: '[User Text]'. You are the Dungeon Master. What happens next? Keep it under 3 sentences.")
• API Manager: Sends the prompt to Gemini to get the story text, the scene image, and the spoken audio.
• Game Saver: Saves the entire interaction to a database so the player can close the browser and resume their game tomorrow.

3. The AI Core (Local + Gemini APIs)

• Speech-to-Text (Local): Uses OpenAI's Whisper (the local Python package, not the paid API) to instantly and privately transcribe the user's voice.
• Text Generation: Gemini Pro API handles the storytelling and game logic.
• Image Generation: Gemini API (Imagen model) generates a visual representation of the current scene.
• Voice Generation: Gemini's audio capabilities read the Dungeon Master's response out loud.

---

💾 Database Schema (MongoDB / PostgreSQL)

To keep track of an infinite game, your Python backend needs to save the "Game State." If you use a database, your Campaign table/collection would look like this:

Field	Type	Description
id	String	Unique ID for the game session.
player_name	String	e.g., "Thorin the Brave"
health_points	Integer	Starts at 100. (Gemini can tell Python to lower this!)
inventory	Array	e.g., ["Rusty Sword", "Health Potion"]
chat_history	Array	Every message sent and received, so Gemini remembers the past.

---

📝 Step-by-Step Execution Plan

Phase 1: The Local Ears (Python)

• Set up your Python environment.
• Install openai-whisper and sounddevice (or SpeechRecognition).
• Write a script that records your microphone for 5 seconds and prints exactly what you said to the terminal. Do not move to Phase 2 until this works perfectly offline.

Phase 2: The Storyteller (Python + Gemini)

• Import the google-generativeai SDK.
• Connect your Phase 1 script to Gemini. When Whisper prints your text, send that text to the Gemini API.
• Print Gemini's response to the terminal.

Phase 3: The Senses (Adding Images & Voice)

• Update your Python script so that along with the text story, it asks Gemini to generate an image of the scene and the audio voiceover.
• Save the output image as current_scene.jpg and the audio as dm_voice.mp3.

Phase 4: The Dashboard (React & Web Connection)

• Build the React UI.
• Set up a Python Flask server.
• Make React send the recorded microphone audio to Flask via a POST request.
• Flask runs the logic (Whisper -> Gemini) and returns the JSON containing the Text, Image URL, and Audio URL back to React to display and play.

---

🌟 Bonus Challenges for a Pro

• Dynamic Inventory: Instruct Gemini to output its responses in JSON format. Have it return the story text, AND a list of items the player just found. Python can then update your React inventory UI in real-time.
• Sound Effects: If Gemini mentions a "sword," have Python trigger a metallic clash.mp3 sound effect in the background before the Dungeon Master speaks.