Unity for Human-Robot Interaction Environments

Accessibility Statement

This chapter follows accessibility standards for educational materials, including sufficient color contrast, semantic headings, and alternative text for images.

Introduction

This section explores how Unity can be used to create photorealistic environments for simulating human-robot interaction scenarios.

Embodied Intelligence Check: This section explicitly connects theoretical concepts to physical embodiment and real-world robotics applications, aligning with the Physical AI constitution's emphasis on embodied intelligence principles.

Unity provides a powerful platform for creating photorealistic simulation environments that are particularly valuable for modeling human-robot interaction scenarios. Its advanced rendering capabilities, extensive asset library, and physics engine make it an ideal choice for simulating complex environments where robots interact with humans and manipulate objects in realistic settings. Unity's capabilities complement traditional robotics simulators like Gazebo by providing more visually realistic environments that can better prepare AI systems for real-world deployment.

Unity's strength lies in creating immersive environments where human-robot interaction can be studied, tested, and refined before real-world deployment. The platform allows for detailed modeling of indoor and outdoor environments, realistic lighting conditions, and complex object interactions, which are essential for developing embodied intelligence applications that will operate in human-populated spaces.

This chapter will explore how Unity enables the Physical AI principle of simulation-to-reality progressive learning by providing photorealistic environments where AI systems can learn to interact with humans and manipulate objects in realistic settings.

Core Concepts

Key Definitions

Unity: A cross-platform game engine that can be used for creating 3D simulation environments for robotics applications.
Photorealistic Rendering: The process of generating synthetic images that closely resemble real-world photographs, used to create more realistic simulation environments.
Human-Robot Interaction (HRI): The study of interactions between humans and robots, including communication, collaboration, and social behaviors.
Asset Packages: Pre-built 3D models, materials, animations, and other resources for populating Unity environments.
Physics Engine: A software system that simulates physical interactions like collisions, forces, and motion within the Unity environment.
Environment Mapping: The process of creating detailed virtual environments that match real-world locations.
Real-time Simulation: Simulation that runs at or near actual speed, allowing for interactive development and testing.
Mixed Reality: Environments that blend physical and virtual worlds for training and testing robot behaviors.
Interaction Design: The process of designing how humans and robots communicate and collaborate within the simulation.

Architecture & Components

Technical Standards Check: All architecture diagrams and component descriptions include references to ROS 2, Gazebo, Isaac Sim, VLA, and Nav2 as required by the Physical AI constitution's Multi-Platform Technical Standards principle.

Unity for HRI environments includes:

Rendering Engine: High-fidelity graphics rendering for photorealistic visuals
Physics Simulation: Integration with physics engines for realistic object interactions
ROS Bridge: Middleware for connecting Unity to ROS/ROS 2 systems
Asset Management: System for importing and managing 3D models and materials
Animation System: Toolset for simulating human movements and object dynamics
Lighting System: Advanced lighting to simulate real-world illumination conditions
Audio System: Spatial audio for realistic sound simulation
UI System: Interfaces for human operators and data visualization

This architecture enables realistic simulation of human-robot interaction scenarios.

Technical Deep Dive

Click here for detailed technical information

Architecture considerations: Real-time rendering with physics simulation and ROS integration
Framework implementation: Unity-Rosbridge integration and communication protocols
API specifications: Standard interfaces for Unity-ROS communication
Pipeline details: Rendering, physics simulation, and message handling
Mathematical foundations: 3D graphics, physics simulation, and spatial transformations
ROS 2/Gazebo/Isaac/VLA structures: Integration points with AI frameworks
Code examples: Implementation details for Unity simulation systems

Unity's architecture for robotics simulation typically involves integration with ROS through the Unity-Rosbridge package, which provides communication between Unity's native C# environment and the ROS messaging system. This integration enables:

Unity-Rosbridge Communication: WebSocket-based communication between Unity and ROS nodes
Message Translation: Converting between Unity data types and ROS message formats
Real-time Control: Sending commands from ROS to Unity and receiving sensor data back
Synchronization: Maintaining timing consistency between Unity simulation and ROS control loops

Example Unity C# script that interfaces with ROS:

UnityRobotController.cs
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using RosSharp.RosBridgeClient;
using RosSharp.Messages.Geometry;

public class UnityRobotController : MonoBehaviour
{
    private RosSocket rosSocket;
    
    // Robot parameters
    public float linearVelocity = 0.0f;
    public float angularVelocity = 0.0f;
    
    // Current pose
    private Vector3 currentPosition;
    private Quaternion currentRotation;
    
    // ROS topic names
    private string cmdVelTopic = "/cmd_vel";
    private string odomTopic = "/odom";
    
    void Start()
    {
        // Initialize ROS connection
        // This assumes Unity-Rosbridge is running
        RosBridgeClient.RosBridgeWebSocketConnection.RosBridgeClientWebSocketSettings settings =
            new RosBridgeClient.RosBridgeWebSocketConnection.RosBridgeClientWebSocketSettings(
                "ws://127.0.0.1:9090");
                
        rosSocket = new RosSocket(settings);
        
        // Subscribe to command topic
        rosSocket.Subscribe<Twist>(cmdVelTopic, ReceiveCmdVel);
        
        // Initialize robot pose
        currentPosition = transform.position;
        currentRotation = transform.rotation;
    }
    
    void ReceiveCmdVel(Twist cmd)
    {
        // Process velocity commands from ROS
        linearVelocity = cmd.linear.x;   // Forward/backward
        angularVelocity = cmd.angular.z; // Turn left/right
    }
    
    void Update()
    {
        // Apply motion in Unity based on ROS commands
        // Convert ROS velocities to Unity movement
        float deltaTime = Time.deltaTime;
        
        // Move forward/backward
        Vector3 forwardMovement = transform.forward * linearVelocity * deltaTime;
        
        // Apply turning
        float turnAmount = angularVelocity * deltaTime;
        transform.Rotate(Vector3.up, turnAmount * Mathf.Rad2Deg);
        
        // Update position
        transform.position += forwardMovement;
        
        // Publish odometry back to ROS
        PublishOdometry();
    }
    
    void PublishOdometry()
    {
        // Create odometry message and publish to ROS
        RosSharp.Messages.Nav.Odometry odom = new RosSharp.Messages.Nav.Odometry();
        
        odom.header.stamp = new RosSharp.Messages.Std.Time();
        odom.header.frame_id = "odom";
        odom.child_frame_id = "base_link";
        
        // Position
        odom.pose.pose.position = new RosSharp.Messages.Geometry.Point();
        odom.pose.pose.position.x = transform.position.x;
        odom.pose.pose.position.y = transform.position.z; // Unity Y is up, ROS Y is sideways
        odom.pose.pose.position.z = 0; // 2D navigation
        
        // Convert Unity rotation to ROS quaternion
        odom.pose.pose.orientation = new RosSharp.Messages.Geometry.Quaternion();
        odom.pose.pose.orientation.x = transform.rotation.x;
        odom.pose.pose.orientation.y = transform.rotation.y;
        odom.pose.pose.orientation.z = transform.rotation.z;
        odom.pose.pose.orientation.w = transform.rotation.w;
        
        // Velocity (simplified)
        odom.twist.twist.linear = new RosSharp.Messages.Geometry.Vector3();
        odom.twist.twist.linear.x = linearVelocity;
        odom.twist.twist.angular = new RosSharp.Messages.Geometry.Vector3();
        odom.twist.twist.angular.z = angularVelocity;
        
        if (rosSocket != null)
        {
            rosSocket.Publish(odomTopic, odom);
        }
    }
    
    void OnDestroy()
    {
        // Clean up ROS connection
        if (rosSocket != null)
        {
            rosSocket.Close();
        }
    }
}

Example Unity environment setup for HRI:

HRIEnvironment.cs
using UnityEngine;
using UnityEngine.AI;

public class HRIEnvironment : MonoBehaviour
{
    public GameObject robot;  // Robot GameObject
    public GameObject[] humans;  // Human GameObjects in the scene
    public GameObject[] objects; // Objects to manipulate
    
    // Environment parameters
    public float interactionDistance = 2.0f;
    public float minSafetyDistance = 0.5f;
    
    // For Physical AI simulation
    private void Start()
    {
        SetupEnvironment();
    }
    
    private void SetupEnvironment()
    {
        // Configure physics properties for realistic interactions
        foreach (var obj in objects)
        {
            var rb = obj.GetComponent<Rigidbody>();
            if (rb != null)
            {
                rb.interpolation = RigidbodyInterpolation.Interpolate;
            }
        }
        
        // Set up navigation for humans
        foreach (var human in humans)
        {
            var agent = human.GetComponent<NavMeshAgent>();
            if (agent != null)
            {
                agent.radius = 0.3f;
                agent.speed = 1.0f;
            }
        }
    }
    
    private void Update()
    {
        // Monitor human-robot interactions
        CheckInteractions();
    }
    
    private void CheckInteractions()
    {
        foreach (var human in humans)
        {
            float distance = Vector3.Distance(robot.transform.position, human.transform.position);
            
            if (distance < interactionDistance && distance > minSafetyDistance)
            {
                // Potential interaction - implement HRI logic
                Debug.Log($"Potential human-robot interaction at distance: {distance:F2}");
            }
            else if (distance <= minSafetyDistance)
            {
                // Safety distance violated - trigger avoidance
                Debug.LogError($"Safety distance violated: {distance:F2}");
                TriggerSafetyProtocol(human, robot);
            }
        }
    }
    
    private void TriggerSafetyProtocol(GameObject human, GameObject robot)
    {
        // Implement safety behavior when human comes too close
        // For Physical AI & Humanoid Robotics, this might involve:
        // - Stopping robot movement
        // - Sending warning to control system
        // - Changing robot pose to be less threatening
        var rb = robot.GetComponent<Rigidbody>();
        if (rb != null)
        {
            rb.velocity = Vector3.zero;
            rb.angularVelocity = Vector3.zero;
        }
    }
}

Hands-On Example

In this hands-on example, we'll create a Unity environment for simulating human-robot interaction:

Setup Unity Environment: Install Unity and ROS bridge packages
Create 3D Scene: Build an environment with humans and objects
Implement Robot Controller: Create a Unity-based robot controller
Connect to ROS: Integrate Unity simulation with ROS
Test Interaction: Verify human-robot interaction scenarios

Step 1: Basic Unity-ROS setup (Python bridge script)

# unity_ros_bridge.py
#!/usr/bin/env python3

"""
ROS bridge node that forwards messages between Unity and other ROS nodes,
following Physical AI principles for embodied intelligence through realistic
simulation of human-robot interactions.
"""

import rclpy
from rclpy.node import Node
from geometry_msgs.msg import Twist, Pose
from nav_msgs.msg import Odometry
from sensor_msgs.msg import LaserScan
import asyncio
import websockets
import json
import threading

class UnityROSBridge(Node):
    def __init__(self):
        super().__init__('unity_ros_bridge')
        
        # Publishers for sensor data to Unity
        self.unity_cmd_publisher = self.create_publisher(Twist, '/unity/cmd_vel', 10)
        
        # Subscribers for ROS data that Unity needs to simulate
        self.cmd_vel_subscriber = self.create_subscription(
            Twist,
            '/cmd_vel',
            self.cmd_vel_callback,
            10
        )
        
        self.odom_subscriber = self.create_subscription(
            Odometry,
            '/odom',
            self.odom_callback,
            10
        )
        
        # Initialize WebSocket server for Unity communication
        self.websocket_server = None
        self.unity_connection = None
        
        # Start WebSocket server in a separate thread
        ws_thread = threading.Thread(target=self.start_websocket_server)
        ws_thread.daemon = True
        ws_thread.start()
        
        self.get_logger().info('Unity ROS Bridge initialized')
    
    def start_websocket_server(self):
        """Start WebSocket server for Unity communication"""
        start_server = websockets.serve(self.handle_unity_connection, "localhost", 8765)
        asyncio.get_event_loop().run_until_complete(start_server)
        asyncio.get_event_loop().run_forever()
    
    async def handle_unity_connection(self, websocket, path):
        """Handle connection from Unity"""
        self.get_logger().info('Unity connected')
        self.unity_connection = websocket
        
        try:
            async for message in websocket:
                # Process message from Unity
                data = json.loads(message)
                self.process_unity_message(data)
        except websockets.exceptions.ConnectionClosed:
            self.get_logger().info('Unity disconnected')
            self.unity_connection = None
    
    def cmd_vel_callback(self, msg):
        """Forward ROS cmd_vel to Unity"""
        if self.unity_connection:
            cmd_data = {
                'type': 'cmd_vel',
                'linear_x': msg.linear.x,
                'angular_z': msg.angular.z
            }
            asyncio.get_event_loop().create_task(
                self.unity_connection.send(json.dumps(cmd_data))
            )
    
    def odom_callback(self, msg):
        """Forward ROS odometry to Unity"""
        if self.unity_connection:
            odom_data = {
                'type': 'odom',
                'x': msg.pose.pose.position.x,
                'y': msg.pose.pose.position.y,
                'theta': msg.pose.pose.orientation.z  # Simplified
            }
            asyncio.get_event_loop().create_task(
                self.unity_connection.send(json.dumps(odom_data))
            )
    
    def process_unity_message(self, data):
        """Process messages received from Unity"""
        msg_type = data.get('type', '')
        if msg_type == 'sensor_data':
            # Publish sensor data received from Unity to ROS
            scan_msg = LaserScan()
            # Fill in scan data based on Unity simulation
            # This would include raycast results, etc.
            pass

def main(args=None):
    rclpy.init(args=args)
    bridge = UnityROSBridge()
    
    try:
        rclpy.spin(bridge)
    except KeyboardInterrupt:
        pass
    finally:
        bridge.destroy_node()
        rclpy.shutdown()

if __name__ == '__main__':
    main()

Each step connects to the simulation-to-reality learning pathway.

Real-World Application

Simulation-to-Reality Check: This section clearly demonstrates the progressive learning pathway from simulation to real-world implementation, following the Physical AI constitution's requirement for simulation-to-reality progressive learning approach.

In real-world robotics applications, Unity's photorealistic environments are valuable for:

Training robots to interact safely with humans in realistic settings
Testing navigation and manipulation in complex indoor environments
Developing social robotics capabilities in human-populated spaces
Validating perception systems under various lighting conditions

When transitioning from Unity simulation to reality, the system must account for:

Differences in visual processing between Unity and real cameras
Timing differences in simulation vs. real-time execution
Physical properties that are simplified in simulation
The need for domain randomization to improve real-world transfer

Unity's photorealistic capabilities enable the Physical AI principle of simulation-to-reality progressive learning by providing environments that are visually indistinguishable from reality, helping AI systems develop robust perception and interaction capabilities.

Summary

This chapter covered the fundamentals of using Unity for human-robot interaction environments:

How Unity creates photorealistic simulation environments
Core components of Unity-based HRI simulation
Technical implementation of Unity-ROS integration
Practical example of Unity bridge implementation
Real-world considerations for deploying on physical hardware

Unity provides a powerful platform for simulating human-robot interaction scenarios, enabling effective AI training and robot behavior validation in realistic environments, supporting the Physical AI principle of simulation-to-reality progressive learning.

Key Terms

Unity: A cross-platform game engine that can be used for creating 3D simulation environments for robotics applications in the Physical AI context.
Photorealistic Rendering: The process of generating synthetic images that closely resemble real-world photographs, used to create more realistic simulation environments.
Human-Robot Interaction (HRI): The study of interactions between humans and robots, including communication, collaboration, and social behaviors.
Unity-Rosbridge: Middle ware for connecting Unity to ROS/ROS 2 systems, enabling communication between Unity and ROS nodes.

Compliance Check

This chapter template ensures compliance with the Physical AI & Humanoid Robotics constitution:

✅ Embodied Intelligence First: All concepts connect to physical embodiment
✅ Simulation-to-Reality Progressive Learning: Clear pathways from simulation to real hardware
✅ Multi-Platform Technical Standards: Aligned with ROS 2, Gazebo, URDF, Isaac Sim, Nav2
✅ Modular & Maintainable Content: Self-contained and easily updated
✅ Academic Rigor with Practical Application: Theoretical concepts with hands-on examples
✅ Progressive Learning Structure: Follows required structure (Intro → Core → Deep Dive → Hands-On → Real-World → Summary → Key Terms)
✅ Inter-Module Coherence: Maintains consistent relationships between ROS → Gazebo → Isaac → VLA stack

Inter-Module Coherence

Inter-Module Coherence Check: This chapter maintains consistent terminology, concepts, and implementation approaches with other modules in the Physical AI & Humanoid Robotics textbook, particularly regarding the ROS → Gazebo → Isaac → VLA stack relationships.

This chapter establishes the Unity simulation environment that connects to other modules:

The HRI simulation complements Gazebo physics in Module 2
Unity environments support Isaac Sim integration in Module 3
Photorealistic environments enhance VLA perception in Module 4

Accessibility Statement

Introduction​

Core Concepts​

Key Definitions​

Architecture & Components​

Technical Deep Dive​

Hands-On Example​

Step 1: Basic Unity-ROS setup (Python bridge script)​

Real-World Application​

Summary​

Key Terms​

Compliance Check​

Inter-Module Coherence​

Introduction

Core Concepts

Key Definitions

Architecture & Components

Technical Deep Dive

Hands-On Example

Step 1: Basic Unity-ROS setup (Python bridge script)

Real-World Application

Summary

Key Terms

Compliance Check

Inter-Module Coherence