HIF — HRI Interaction Frame

Overview

HIF (HRI Interaction Frame) is the central semantic carrier of HML.

A HIF is a structured envelope that represents a meaningful unit of interaction information. It may originate from a sensor, a user utterance, a perception module, a reasoning process, a database query, an internal robot monitor, or an action request.

A HIF is not merely a message, event, or video frame. It is a semantic interaction object that carries raw content, interpreted properties, metadata, confidence, and traceability information through the HRI architecture.

In HML, most architectural patterns can be understood as operations over HIFs:

Creating HIFs
Transforming HIFs
Enriching HIFs
Synchronizing HIFs
Validating HIFs
Delaying HIFs
Escalating HIFs
Executing HIFs

The HIF is therefore the atomic modeling unit that connects perception, cognition, memory, validation, autonomy, and embodied action.

Why HIF is Needed

Human-Robot Interaction systems combine many heterogeneous streams:

Camera frames
Audio snippets
Speech transcripts
LiDAR scans
Skeleton estimates
Gaze vectors
Facial expressions
Object detections
User instructions
Robot resource states
Database facts
LLM outputs
Navigation requests
Social policies
Safety constraints

Without a shared semantic carrier, each subsystem tends to exchange information in its own local representation. This makes the architecture difficult to inspect, synchronize, validate, and reuse.

HIF provides a common conceptual structure:

raw content + semantic properties + metadata + confidence + history

The same modeling unit can therefore represent both a raw camera frame and a high-level instruction such as:

Bring this bottle to Bob

This enables design patterns to be expressed independently of a specific robot platform, middleware, or implementation language.

Conceptual Structure

A HIF is a conceptual schema, not necessarily a strict serialization format.

It may be implemented as JSON, a Python dataclass, a TypeScript interface, a ROS message, protobuf, or any other suitable representation.

A typical HIF contains:

HIF {
  id
  type
  timestamp
  content
  properties
  confidence
  source_history
  processing_history
}

Fields

Field	Meaning
`id`	Unique identifier for the interaction frame
`type`	Semantic type of the frame
`timestamp`	Time point or time interval represented by the HIF
`content`	Raw or primary content carried by the HIF
`properties`	Dynamic semantic dictionary added by experts and transformers
`confidence`	Confidence value or structured confidence map
`source_history`	Origin of the data, such as sensor, DB, user, or module
`processing_history`	Record of semantic cells, policies, or experts applied to the HIF

Not every field must be present in every implementation. However, the conceptual distinction between raw content, semantic properties, and traceability should remain clear.

HIF as a Semantic Envelope

The HIF acts as an envelope around interaction information.

This envelope allows the system to preserve both the original information and the semantic interpretations that are gradually attached to it.

HIF Types

A HIF may represent many different kinds of interaction information.

Type	Meaning
`VideoFrame`	A camera frame or image-based input
`AudioSnippet`	A bounded audio segment
`Text`	Typed, transcribed, or generated text
`Gesture`	Interpreted human movement or gesture
`PersonState`	Human identity, pose, gaze, affect, or availability
`SceneState`	Objects, spaces, obstacles, and environmental relations
`RobotState`	Internal robot state such as battery, CPU, localization, or actuator status
`Fact`	Declarative statement about the world
`Query`	Request for information from memory, a human, or an expert
`Instruction`	User command or goal specification
`Task`	Partially or fully resolved goal
`PendingTask`	Task delayed due to missing information or timing constraints
`ActionRequest`	Request for speech, motion, navigation, manipulation, or another effect
`ValidationResult`	Result of a consistency, safety, social, or autonomy check

The exact type system is domain-specific. HML requires only that the type communicate the semantic role of the frame.

Properties as Dynamic Semantic Memory

The properties field is the main place where meaning accumulates.

Semantic Transformers, Sync Gates, Escalation Switches, and validation layers may all add or modify properties.

Example:

{
  "person": {
    "id": "p12",
    "name": "Bob",
    "location": "kitchen"
  },
  "gesture": "pointing",
  "intent": "bring_object",
  "target_object": {
    "type": "bottle",
    "id": "obj7"
  }
}

Properties may be produced by:

Object detectors
Person trackers
Skeleton extractors
Speech-to-text modules
Natural language interpreters
Gesture classifiers
Spatial reasoners
Social policy evaluators
Safety monitors
LLMs
Database lookups

A HIF therefore becomes richer as it moves through the system.

HIF Lifecycle

A HIF usually evolves through several architectural stages.

optionally executed

This lifecycle is not mandatory or linear in every architecture. Some HIFs may be synchronized, cached, delayed, split, merged, or routed back into memory.

However, the general idea remains:

source information becomes semantic information,
semantic information becomes validated information,
validated information may become executable action.

Source History

source_history records where a HIF came from.

Examples include:

{
  "source_history": [
    {
      "source": "camera_front",
      "type": "sensor",
      "timestamp": "2026-05-11T12:34:01.120Z"
    }
  ]
}

or:

{
  "source_history": [
    {
      "source": "HRI_DB",
      "type": "memory",
      "query": "person_location(Bob)"
    }
  ]
}

Source history is important because socially intelligent systems often need to reason about trust, provenance, and uncertainty.

For example:

Was this fact stated by the user?
Was it inferred by a model?
Was it retrieved from memory?
Was it detected by a noisy sensor?
Is the information fresh or stale?

Processing History

processing_history records how a HIF was transformed.

Example:

{
  "processing_history": [
    {
      "cell": "SpeechToTextST",
      "operator": "local_stt_model",
      "output": "text",
      "confidence": 0.91
    },
    {
      "cell": "IntentInterpreterST",
      "operator": "semantic_parser",
      "output": "instruction",
      "confidence": 0.84
    },
    {
      "cell": "SpatialResolverES",
      "operator": "pointing_vector_resolver",
      "output": "object_id",
      "confidence": 0.79
    }
  ]
}

Processing history supports:

Explainability
Debugging
Auditing
Failure analysis
Safety review
Reproducibility
Human-facing explanations

It helps the system answer questions such as:

Why did the robot think I meant that bottle?
Why did it ask for clarification?
Which model produced this interpretation?
Was an LLM involved?
Was the action blocked by a safety policy?

Confidence

Confidence may be represented in different ways.

A simple HIF may contain one global confidence value:

{
  "confidence": 0.87
}

A more detailed HIF may contain confidence per property:

{
  "confidence": {
    "speech_to_text": 0.91,
    "intent": 0.84,
    "object_reference": 0.72,
    "target_person": 0.95
  }
}

Property-level confidence is often preferable in HRI because different semantic components may have different reliability levels.

For example, the system may confidently know that the user said Bob, but be uncertain about which bottle was referenced.

Example: From Speech to Pending Task

This example shows how a HIF may evolve from raw text into a pending task.

User Input

Bring this bottle to Bob

Initial Text HIF

{
  "id": "hif_001",
  "type": "Text",
  "timestamp": "2026-05-11T12:34:01.120Z",
  "content": "Bring this bottle to Bob",
  "properties": {},
  "source_history": [
    {
      "source": "microphone_front",
      "type": "sensor"
    }
  ],
  "processing_history": []
}

After Intent Interpretation

{
  "id": "hif_001",
  "type": "Instruction",
  "content": "Bring this bottle to Bob",
  "properties": {
    "intent": "pick_and_place",
    "object_reference": "this bottle",
    "target_person": "Bob"
  },
  "confidence": {
    "intent": 0.88,
    "target_person": 0.96,
    "object_reference": 0.74
  },
  "processing_history": [
    {
      "cell": "IntentInterpreterST",
      "operator": "semantic_parser",
      "result": "instruction"
    }
  ]
}

After Spatial Resolution

{
  "id": "hif_001",
  "type": "Instruction",
  "properties": {
    "intent": "pick_and_place",
    "object_id": "bottle_5",
    "object_type": "bottle",
    "target_person": "Bob",
    "target_location": "unknown"
  },
  "confidence": {
    "object_id": 0.82,
    "target_person": 0.96,
    "target_location": 0.0
  },
  "processing_history": [
    {
      "cell": "SpatialResolverES",
      "operator": "pointing_vector_resolver",
      "result": "object_id: bottle_5"
    }
  ]
}

After Task Prerequisite Resolution

{
  "id": "hif_001",
  "type": "PendingTask",
  "properties": {
    "task": "pick_and_place",
    "object_id": "bottle_5",
    "target_person": "Bob",
    "missing_information": [
      "target_location"
    ],
    "clarification_request": "Where is Bob?"
  },
  "processing_history": [
    {
      "cell": "TaskPrerequisiteResolver",
      "operator": "missing_info_detector",
      "result": "pending_task"
    }
  ]
}

The important point is that the HIF does not merely move through the system. It changes semantic status as the system understands more about the interaction.

HIF Versus Message, Event, or Object

A HIF is related to common software concepts, but it is not identical to them.

Concept	Similarity	Difference
Message	Carries information between components	HIF also carries semantics, provenance, and processing history
Event	Represents something that happened	HIF may represent perception, memory, query, task, or action
Data Object	Contains structured fields	HIF is designed for semantic transformation and traceability
ROS Message	Can be used as an implementation format	HIF is a modeling abstraction above middleware
LLM Context Object	Can carry semantic state	HIF is grounded in embodied interaction and temporal synchronization

A HIF should therefore be understood as an interaction-level semantic object.

HIF and SOCIAL Principles

The HIF directly supports the SOCIAL principles.

Principle	HIF Contribution
S-Separated Contexts	HIFs allow separate context streams to remain explicit
O-Open Declarative	HIF properties expose structured semantic state
C-Clear Cognition	HIF history makes reasoning stages traceable
I-Interpretable Gates	Gates can explain how and why a HIF changed state
A-Adaptive Autonomy	HIFs can carry autonomy state, confidence, and constraints
L-Layered Validation	Validation results can accumulate as HIF properties

The HIF is therefore not only a data structure. It is the semantic substrate that makes SOCIAL architectures inspectable.

Design Guidelines

When designing with HIFs:

Keep HIFs Semantic

A HIF should represent interaction meaning, not just transport data.

Separate Content from Properties

Raw or primary data should remain in content.

Interpreted meaning should be placed in properties.

Preserve Traceability

Important transformations should be recorded in processing_history.

Do Not Hide Logic Inside the HIF

A HIF carries information. Semantic cells perform computation.

Use Confidence Explicitly

Probabilistic outputs should expose confidence, especially when used for action.

Avoid Silent Overwrites

If a property is modified, the system should preserve provenance or record the change.

Keep Properties Human-Readable

Prefer JSON-like structures that can be inspected, logged, and explained.

Treat HIF Types as Semantic Roles

The same underlying object may change type as it moves from raw input to interpreted instruction to pending task or executable action.

Implementation Notes

HML does not require a specific implementation format.

A HIF may be implemented using:

JSON
YAML
Python dataclass
TypeScript interface
ROS message
Protobuf
Pydantic model
Database record
Event object

The implementation should preserve the conceptual separation between:

content
properties
metadata
confidence
history

This separation is more important than the exact serialization format.

Conclusion

The HIF is the core semantic carrier of HML.

It allows HRI systems to represent interaction information in a way that is:

Multimodal
Traceable
Enrichable
Synchronizable
Validatable
Explainable
Executable when appropriate

By using HIFs as the common modeling unit, HML enables reusable HRI Design Patterns that remain independent of specific robots, middleware frameworks, or AI implementations.

The HIF is therefore the bridge between raw interaction data and socially meaningful robotic behavior.

Overview​

Why HIF is Needed

Conceptual Structure

Fields​

HIF as a Semantic Envelope

HIF Types

Properties as Dynamic Semantic Memory

HIF Lifecycle

Source History

Processing History

Confidence

Example: From Speech to Pending Task

User Input​

Initial Text HIF​

After Intent Interpretation​

After Spatial Resolution​

After Task Prerequisite Resolution​