Synchronous Multi-Extractor (SME)

Intent

The Synchronous Multi-Extractor (SME) coordinates several Semantic Transformers that process the same input frame, source moment, or interaction slice, and synchronizes their outputs into one coherent enriched HIF.

In short:

SME = parallel semantic extraction + synchronization into one enriched HIF

The pattern is useful when several experts can extract complementary semantic properties from the same interaction moment, but downstream reasoning should receive a single temporally coherent HIF.

A useful intuition is map-reduce:

map:    several STs extract different semantic properties
reduce: one SG synchronizes and aggregates them into a coherent HIF

However, SME is not merely a data-processing pattern.

In HRI, the synchronization step is semantic and temporal.
It determines whether extracted properties belong to the same interaction moment.

---

Problem

Human-Robot Interaction often depends on multiple cues that occur close together in time:

• a person enters the frame
• the person turns toward the robot
• the person smiles
• the person points
• the person speaks
• the person shifts attention
• the person begins a gesture

These cues may be processed by different experts with different latencies.

For example:

person detection may be fast
skeleton extraction may be slower
gaze estimation may be noisy
facial expression classification may arrive a few frames later

If the system simply merges the latest available result from each expert, it may create semantic ghosting.

Semantic ghosting occurs when the robot combines cues that do not belong to the same interaction moment.

Example:

smile from frame X
+
gaze from frame Y
+
gesture from frame Z
+
speech from a different moment
=
incorrect interpretation of the human state

SME prevents this by making synchronization explicit.

The SG does not merely concatenate outputs.
It applies a synchronization policy and aggregation logic before publishing an enriched HIF.

---

Context

Use SME when:

• several experts process the same source frame, source segment, or interaction slice
• each expert extracts a different semantic property
• downstream reasoning requires a coherent enriched HIF
• the extracted properties must be temporally aligned
• experts may complete asynchronously
• partial results may need timeout, buffering, or confidence policies
• the architecture should allow adding or removing experts without changing the whole pipeline

SME is especially useful for:

• video-frame analysis
• audio analysis
• multimodal perception
• human-state estimation
• scene analysis
• action styling
• any situation where multiple semantic properties are extracted in parallel and then synchronized

SME may be unnecessary when a single expert already produces all required properties with acceptable timing, explainability, and modularity.

---

HML Structure

Abstract Pattern

The abstract SME structure contains:

Input HIF
  → parallel STs
  → SG
  → Enriched HIF

The STs perform the extraction.
The SG performs synchronization, alignment, and aggregation.

Each ST is autonomous; the SG synchronizes their outputs into one coherent enriched HIF.

---

Participants

Participant	HML Role	Responsibility
Input HIF	HIF	Carries the shared input frame, segment, or interaction slice
ST₁..STₙ	Semantic Transformers	Extract complementary semantic properties from the same input
λ₁..λₙ	Semantic Operators	Implement the extraction expertise
SG	Sync Gate	Waits, aligns, validates, and aggregates the extracted outputs
Synchronization policy	Policy artifact	Defines time window, required/optional extractors, timeout, confidence, and merge behavior
λ aggregator	Semantic Operator	Combines accepted properties into one enriched HIF
Output HIF	HIF	Carries the synchronized enriched semantic frame

---

Flow

A typical SME flow is:

1. An input HIF enters the pattern.
2. The input HIF is distributed to several ST cells.
3. Each ST applies its own λ expert and produces an enriched HIF or extracted property set.
4. The SG receives the extraction outputs, possibly asynchronously.
5. The SG applies a synchronization policy.
6. Accepted properties are aggregated into a single enriched HIF.
7. The enriched HIF is sent downstream.

The SG may also:

• wait for missing extractors
• emit a partial HIF after timeout
• reject inconsistent evidence
• preserve disagreement
• mark confidence per property
• route low-confidence cases to later validation or escalation

---

Example Composition: Human Video Analyzer

One possible Human Context instantiation begins with a camera stream.

A camera source is converted into a VideoFrame HIF by an HC.
A Person Detector ST first checks whether a human is present.
Only if a person is detected does the architecture enter the SME pattern.

Inside the SME, several image-based experts process the same VideoFrame HIF:

skeleton extractor
facial expression classifier
gaze detector

The Aggregator SG then synchronizes the extracted properties into one HumanState HIF.

This example illustrates one possible composition.
It is not a required perception stack.

image-based detectors over the same frame

SkeletonExtractor STFacial ExpressionClassifier STGazeDetector STλ[]λ[]λ[]SGsync policyλ[]aggregateHumanStateHIF

The important point is not the exact set of detectors.

The important point is that several semantic properties are extracted from a shared frame or interaction slice and then synchronized before being treated as one human-context state.

---

Temporal Synchronization

The SG in SME defines when extracted properties are allowed to belong together.

For example:

frame t:     person detected
frame t+1:   skeleton result arrives
frame t+2:   facial expression result arrives
frame t+3:   gaze result arrives

The SG may decide that these still belong to the same interaction window.

In another case, a delayed smile may not belong to the same event:

frame t:     user speaks
frame t+20:  smile detected

Whether the smile belongs to the spoken event depends on the synchronization policy.

The policy may define:

• maximum frame offset
• maximum time window
• required extractors
• optional extractors
• confidence thresholds
• partial-output rules
• timeout behavior
• whether to hold, emit, reject, or mark uncertain

This is why SME is a semantic pattern, not merely a parallel-processing pattern.

---

Example HIF Output

A synchronized HumanState HIF may contain properties such as:

{
  "type": "HumanState",
  "source": "video_analyzer",
  "timestamp_window": {
    "start": "t",
    "end": "t+3"
  },
  "properties": {
    "person_detected": true,
    "person_id": "track_17",
    "skeleton_2d": "...",
    "gaze_direction": "toward_robot",
    "facial_expression": "smile"
  },
  "confidence": {
    "person": 0.94,
    "skeleton": 0.88,
    "gaze": 0.76,
    "expression": 0.81,
    "synchronization": 0.86
  },
  "processing_history": [
    "PersonDetectorST",
    "SkeletonExtractorST",
    "GazeDetectorST",
    "FacialExpressionClassifierST",
    "AggregatorSG"
  ]
}

The schema is illustrative.

A concrete implementation may use different fields, types, or confidence structures.

The important requirement is that the resulting HIF preserves:

• extracted properties
• confidence
• source history
• timing
• processing history
• synchronization assumptions

---

Reuse

SME is introduced in Human Context because human-state estimation is an intuitive first example.

However, SME is not limited to Human Context.

Reuse Location	How SME Reappears
Human Context	Person, skeleton, gaze, expression, speech, sentiment extraction
Scene Context	Object, door, stair, obstacle, slope, affordance extraction
Robot Context	Sensor health, battery, CPU, memory, actuator state synchronization
Social Action Stylist	Several stylists generate candidate modalities that must be synchronized
Actuation	Multiple execution feedback streams may be synchronized into one ExecutionFeedback HIF

This is why SME is a foundational pattern.

It captures a general HRI need:

several semantic views of the same interaction moment must become one coherent HIF

---

SOCIAL Principles Supported

Within the SOCIAL framework, SME supports:

• S-Separated Contexts by allowing each extractor to operate as a separate semantic path before controlled synchronization.
• O-Open Declarative by making each extracted property explicit in the resulting HIF rather than hidden inside a monolithic model output.
• C-Clear Cognition by decomposing perception into named semantic stages such as person detection, skeleton extraction, gaze estimation, and expression classification.
• I-Interpretable Gates by making the SG responsible for explicit synchronization decisions such as wait, emit, partial emit, reject, or mark uncertain.
• A-Adaptive Autonomy by enabling the robot to reduce confidence, wait for better evidence, or avoid action when synchronized human context is incomplete.
• L-Layered Validation by preserving per-extractor confidence and synchronization confidence for downstream validation before planning or execution.

---

Tradeoffs

Tradeoff	Explanation
Latency	The SG may wait for slower experts before emitting an enriched HIF
Buffer complexity	The system must manage timing windows, partial results, and stale outputs
Resource cost	Running several experts in parallel may require CPU/GPU resources
Policy tuning	Synchronization windows and timeout policies must be tuned carefully
Partial evidence	The system must decide whether incomplete extraction is usable
Error propagation	Bad extractor outputs may still enter the synchronized HIF if validation is weak
Modularity vs overhead	Adding experts is easy architecturally, but each expert increases operational complexity

The main design tension is:

semantic consistency vs. latency

SME improves data integrity but may slow down response if synchronization is too strict.

---

Failure Modes

Failure Mode	Possible Mitigation
One extractor is slow	Use timeout policy and emit partial HIF with missing-property markers
One extractor fails	Preserve failure in processing history and continue if policy allows
Extractors disagree	Preserve disagreement and route to validation or escalation
Results come from different frames	Use timestamp windows and synchronization confidence
SG waits too long	Use bounded buffers and maximum latency policy
SG emits too early	Require minimum evidence or confidence thresholds
Stale result is reused accidentally	Attach timestamps and source history to each property
Too many experts overload compute	Combine SME with EAG to allocate resources adaptively
Downstream layer overtrusts output	Preserve per-property confidence and validation status

---

Implementation Notes

An SME implementation should usually define:

• input HIF type
• extractor list
• required vs optional extractors
• per-extractor timeout
• synchronization window
• per-property confidence representation
• buffer size
• stale-result policy
• partial-output policy
• aggregation strategy
• processing-history format

A possible configuration may look like:

pattern: SME
input_hif: VideoFrame
extractors:
  - name: skeleton_extractor
    required: true
    timeout_ms: 80
  - name: facial_expression_classifier
    required: false
    timeout_ms: 120
  - name: gaze_detector
    required: false
    timeout_ms: 100
sync_gate:
  max_window_ms: 150
  partial_output: true
  min_required_confidence: 0.70

This is only a reference style.

The pattern does not require YAML, ROS, Python, or any specific middleware.

The key is that the synchronization contract remains explicit.

---

Related Patterns

SME is closely related to:

• Elastic Attention Governor (EAG) — EAG can decide which SME extractors run, at what fidelity, or for which human target.
• Adaptive Signature Learner (ASL) — ASL may consume synchronized skeleton or gesture HIFs produced by SME.
• Context Novelty Extractor (CNE) — CNE may detect meaningful changes in the synchronized HumanState HIF.
• Social Action Stylist (SAS) — SAS can reuse the same multi-output-and-synchronize structure for candidate behavior styles.
• Layered Validation — downstream validation can use SME's per-extractor confidence and synchronization history.

---

Minimal Summary

SME runs multiple semantic extractors over a shared interaction slice.
Each extractor produces a partial semantic view.
An SG synchronizes and aggregates those views into one enriched HIF.
The pattern prevents semantic ghosting and temporal hallucination.

SME is the first foundational HRI Design Pattern because it establishes a central idea of the framework:

social meaning should be assembled through explicit, inspectable, synchronized semantic structure
rather than hidden inside one opaque perception result.