HML — HRI Modeling Language

Overview

This overview locates HML within the overall SOCIAL framework. The SOCIAL principles define the architectural commitments, HML provides the modeling language, and HRI Design Patterns reuse HML primitives to describe concrete socially intelligent robot architectures.

HML (HRI Modeling Language) is a lightweight architectural modeling language for designing, documenting, and reasoning about socially intelligent Human-Robot Interaction (HRI) systems.

The purpose of HML is to provide a shared visual and semantic vocabulary for describing how embodied AI systems perceive, interpret, reason, validate, and act within human environments.

HML is not intended to replace implementation frameworks such as ROS, behavior trees, state machines, or agent orchestration platforms. Instead, it operates at a higher architectural level: it describes the semantic flow of interaction.

It answers questions such as:

What kind of interaction information is flowing through the system?
Which semantic transformations are applied?
Where is context synchronized?
Where is reasoning escalated?
Where is social or safety validation performed?
Which parts of the architecture are deterministic, learned, cached, or generative?
Where does semantic reasoning become embodied action?

HML provides the modeling foundation for the SOCIAL principles and the HRI Design Patterns.

Motivation

Modern HRI systems increasingly combine many heterogeneous components:

Cameras
Microphones
LiDAR
Robot sensors
Object detectors
Speech recognition
Gesture recognition
Tracking systems
Knowledge bases
LLMs
Multimodal models
Planning systems
Navigation stacks
Social policies
Safety monitors
Actuators

Without a modeling language, these systems are often described using implementation diagrams that expose software nodes, APIs, or message topics, but hide the semantic logic of interaction.

This creates several problems:

The architecture becomes difficult to explain.
Social reasoning is hidden inside implementation details.
Multimodal synchronization is unclear.
Escalation between simple and complex reasoning is not explicit.
Safety and social validation are difficult to locate.
Design patterns cannot be compared systematically.
Academic communication becomes overly implementation-specific.

HML addresses these limitations by modeling the semantic structure of HRI systems directly.

Core Idea

HML models HRI systems as flows of semantic interaction frames processed by explicit semantic cells.

At the center of HML is the HIF:

HIF — HRI Interaction Frame

A HIF is a semantic carrier that wraps raw content, interpreted properties, metadata, and processing history.

HIFs flow through semantic cells such as:

HIF Creators
Semantic Transformers
Sync Gates
Escalation Switches
HIF Executors

Each cell applies a semantic operation, policy, expert, or transformation to the HIF.

In simplified form:

Source
  ↓
HIF Creator
  ↓
HIF
  ↓
Semantic Cells
  ↓
Enriched / validated / executable HIF
  ↓
HIF Executor

This allows an HRI architecture to be described as an interpretable semantic pipeline rather than as a collection of disconnected software modules.

The Three Modeling Layers

HML can be understood through three modeling layers.

1. Semantic Data

Semantic data elements represent what flows through the system.

Examples include:

HIF
HIF Stream
Semantic Object
HRI_DB
Semantic Cache
Queue
Buffer
Time-series Window
Pending Task Queue

These elements describe interaction information, memory, state, and temporal storage.

2. Semantic Operators

Semantic operators describe how meaning is computed, transformed, validated, or selected.

The main symbolic abstraction is:

λ — Semantic Operator

A λ may represent:

Deterministic code
A heuristic
A classifier
A neural network
An LLM
A multimodal model
A rule engine
A database query
A policy function
A social law
A safety rule

From the modeling perspective, λ represents an encapsulated expert or policy.

3. Semantic Cells

Semantic cells are architectural primitives that apply semantic operators to HIFs.

Core cells include:

Cell	Name	Purpose
HC	HIF Creator	Converts a source into a HIF
ST	Semantic Transformer	Adds or modifies semantic properties
SG	Sync Gate	Synchronizes multiple HIFs
ES	Escalation Switch	Selects among experts or reasoning paths
HE	HIF Executor	Converts a HIF into an embodied or external effect

These cells are the basic building blocks used by the HRI Design Patterns.

Why Not Use UML Alone?

HML is not a general-purpose software modeling language.

UML can describe classes, components, sequences, and state machines, but it does not directly capture the central concerns of social HRI:

Semantic enrichment
Multimodal synchronization
Temporal grounding
Contextual reasoning
Social validation
Human-facing explainability
Escalation between heterogeneous experts
Late-bound behavior generation
Embodied execution

HML complements UML by focusing on the semantic and interactional structure of the system.

Why Not Use ROS Graphs Alone?

ROS graphs are excellent for representing runtime communication between nodes.

However, ROS-level diagrams often describe transport and computation rather than meaning.

A ROS graph may show that a camera topic feeds a perception node, but it does not necessarily show:

Which social properties were extracted
Whether gaze and speech were synchronized
Whether confidence was sufficient
Whether a task was delayed due to missing context
Whether a social convention blocked execution
Whether an LLM was used only after simpler reasoning failed

HML abstracts above the implementation layer to expose these semantic decisions.

HML and SOCIAL

The SOCIAL principles define what a socially intelligent architecture should preserve.

HML defines how such an architecture can be modeled.

SOCIAL Principle	HML Support
S-Separated Contexts	Contexts are modeled as separated HIF streams
O-Open Declarative	Semantic state and policies are explicitly represented
C-Clear Cognition	Cognitive stages are visible as HIF transformations
I-Interpretable Gates	Gates are modeled as explicit semantic cells or roles
A-Adaptive Autonomy	Autonomy is represented as explicit state and policy
L-Layered Validation	Validation is distributed across visible semantic layers

In this sense, HML is the notation layer that makes SOCIAL architectures visible.

HML and HRI Design Patterns

The HRI Design Patterns are reusable architectural structures built from HML primitives.

For example:

Pattern	HML Interpretation
Synchronous Multi-Extractor	Multiple STs synchronized by an SG
Elastic Attention Governor	Priority SG plus resource-aware ES
Tiered Semantic Cache	ES over code, cache, and LLM experts
Adaptive Signature Learner	Time-series window plus SG plus learning ST
Context Novelty Extractor	Context unification followed by delta extraction
HRI_DB Handler	ES-mediated query/update logic over declarative memory
Task Prerequisite Resolver	Queue, missing-info ST, QSH, and pending queue
Social Convention Validator	Consistency validation plus social acceptance gate
System Integrity & Agency Handler	Self-HIF generation plus integrity ES
Social Action Stylist	Parallel styling STs synchronized into action
Late-Binding Behavioral Choreographer	Social opportunity timing followed by behavioral styling

This allows the patterns to be documented, compared, and reused systematically.

Modeling Philosophy

HML follows several design commitments.

Semantic First

The language models meaning before implementation.

Explicit Boundaries

Transitions between perception, reasoning, validation, and action should be visible.

Traceability

A HIF should carry enough history to explain how it was produced.

Modularity

Experts, policies, and gates should be replaceable without changing the overall pattern.

Hybrid Intelligence

HML supports symbolic, neural, deterministic, cached, and generative components within the same diagram.

The language is designed specifically for human-facing embodied systems.

Reading an HML Diagram

An HML diagram should be read as a semantic interaction flow.

In general:

HIFs flow between cells.
Cells apply semantic operations.
λ symbols describe the expert or policy inside a cell.
Gates regulate synchronization, escalation, validation, or execution.
Queues and buffers represent temporal or unresolved state.
HRI_DB and caches provide memory or reusable semantic knowledge.
Executors create actions or effects outside the reasoning pipeline.

A typical diagram therefore answers:

What is known? How was it derived? What changed? What is uncertain? What was validated? What is ready for action? What must wait? What requires escalation? What must be explained?

Scope

HML is designed to model:

Multimodal perception pipelines
Human context interpretation
Scene understanding
Robot self-state and integrity
Social reasoning
Task readiness
Validation gates
Autonomy regulation
Behavioral synthesis
Interaction timing
Embodied execution

HML is not intended to specify:

Low-level control algorithms
Exact neural network architectures
Middleware APIs
Memory layouts
Full formal verification semantics
UI implementation details

Those may be documented separately at lower levels of abstraction.

Conclusion

HML provides the semantic modeling layer for SOCIAL-based HRI architectures.

It defines a small vocabulary of interaction frames, semantic operators, memory structures, gates, and processing cells that can be composed into reusable HRI Design Patterns.

By making social cognition architecturally visible, HML supports:

Better documentation
Clearer academic communication
More reusable design patterns
Explainable autonomy
Safer embodied AI
More trustworthy human-robot interaction

HML is therefore the bridge between high-level SOCIAL principles and concrete HRI Design Patterns.

Overview​

Motivation

Core Idea

The Three Modeling Layers

1. Semantic Data​

2. Semantic Operators​

3. Semantic Cells​