Agents and Building Blocks

• Understand how Agents coordinate runtime data flow and events in AI Building Blocks.
• Learn how Providers handle model inference across Cloud, Local, and On-Device backends.
• Connect multiple Agents to build modular, event-driven, and multimodal AI pipelines.
• Combine vision, speech, and language Agents to create interactive XR experiences.
• Extend or customize existing Agents using inheritance and UnityEvents without modifying Providers.

• ObjectDetectionAgent: Runs inference and returns structured detection data.
• ObjectDetectionVisualizer: Renders bounding boxes or 3D meshes in the scene.

1. The Agent captures a frame from PassthroughCamera.
2. It sends the texture to the assigned Provider (for example, UnityInferenceEngineProvider or HuggingFaceProvider).
3. The Provider returns detection results (bounding boxes, class labels, confidence scores).
4. The Visualizer draws the results in world space.

1. EnvironmentDepthManager: Retrieves per-eye environment depth textures from the Meta Quest system and makes them available globally for other components.
2. DepthTextureAccess: Reads these depth textures from the GPU (via EnvironmentDepthManager), copies them into a CPU-accessible array, and exposes per-frame depth data.
3. ObjectDetectionAgent: Uses the 2D bounding boxes from its model output and maps them to world-space using the latest depth information from DepthTextureAccess.
4. ObjectDetectionVisualizer: Renders 3D bounding boxes or markers in the scene at the world-space positions provided by ObjectDetectionAgent.

agent.OnDetectionsUpdated += (detections) =>
{
    foreach (var d in detections)
        Debug.Log($"Detected {d.label} at {d.box}");
};

• LlmAgent: Manages text or multimodal (image and text) conversation flow.
• LlmAgentHelper: Connects LLMs/VLMs with speech agents (STT/TTS) or other custom logic.

1. SendPrompt() forwards user input to the Provider (Llama API, OpenAI, Ollama, and so on).
2. The Provider streams text tokens asynchronously.
3. The Agent emits OnResponseReceived and OnStreamUpdate events.
4. Other Agents (like Text to Speech) can subscribe to stream updates for immediate feedback.

speechToTextAgent.OnTranscript += llmAgentHelper.SendPrompt;
llmAgent.OnResponseReceived += textToSpeechAgent.SpeakText;

• SpeechToTextAgent: Captures microphone or audio clip input.
• Works with Providers implementing ISpeechToTextTask.

1. The Agent records audio or receives an AudioClip.
2. Audio is encoded and sent to the Provider (OpenAI, ElevenLabs).
3. The Provider transcribes it to text and triggers OnTranscript.

speechToTextAgent.OnTranscript += (string transcript) =>
{
    Debug.Log($"User said: {transcript}");
};

• Supports real-time or clip-based transcription.
• Some Providers offer streaming recognition for live captions.
• Optional language code field for multilingual setups.

• TextToSpeechAgent: Converts text into audio clips using ITextToSpeechTask Providers.

1. The Agent receives text via SpeakText().
2. It sends the request to the Provider (for example, ElevenLabs, OpenAI TTS).
3. The Provider returns or streams an AudioClip.
4. The Agent plays it automatically or exposes it for custom playback.

llmAgent.OnResponseReceived += textToSpeechAgent.SpeakText;

• MRUK-only events (like OnBoxesUpdated) are wrapped in compile guards and only available if MRUK is installed and the Passthrough Camera is available.
• LlmAgentHelper connects existing events (onPromptSent, onResponseReceived) for simplified wiring but doesn’t introduce new ones.

[Microphone Input] → SpeechToTextAgent → LlmAgent → TextToSpeechAgent → [Audio Output]

[Camera Frame] → ObjectDetectionAgent → LlmAgent (context injection) → TextToSpeechAgent

public class MyCustomAgent : ObjectDetectionAgent
{
    // Override HandleResults to inject custom logic after detection processing
    // Use this pattern to add haptic feedback, audio cues, or custom UI updates
    protected override void HandleResults(List<Detection> results)
    {
        // Call base implementation first to ensure proper detection handling
        base.HandleResults(results);

        // Add custom behavior that responds to detection results
        foreach (var detection in results)
        {
            if (detection.confidence > 0.8f)
            {
                // Trigger haptic feedback, play sound, or update custom UI
                Debug.Log($"High confidence detection: {detection.label}");
            }
        }
    }
}