Adaptive Signature Learner (ASL)

Intent

Adaptive Signature Learner (ASL) is an HRI design pattern for learning and classifying temporal semantic signatures from HIF streams.

ASL is useful when the meaning of an interaction is not contained in a single frame, utterance, or event, but in a time-series pattern.

Examples include:

• a hand gesture
• a body-motion pattern
• a repeated social cue
• a short melody
• a whistle
• a rhythmic sound
• a multimodal ritual
• a user-specific interaction pattern

The core idea is:

synchronized HIF stream
  → time-series window
  → learn or classify signature
  → output semantic label

ASL allows a robot to learn new social or interactional meanings on the fly, during the interaction itself.

---

Problem

Many socially meaningful signals unfold over time.

A single skeleton frame does not define a gesture.
A single audio frame does not define a melody.
A single gaze sample does not necessarily define attention.

The system must observe a bounded temporal window and interpret the pattern inside that window.

A common solution is to use a pre-trained classifier, such as a neural gesture-recognition model. That may work well for a fixed set of known classes, but it creates several limitations:

• the recognized gesture set is closed
• some gestures may be irrelevant to the robot
• important local gestures may be missing
• the model may have been trained from a different camera viewpoint
• the model may not match the robot's embodiment
• the model is not naturally personalized to a specific user or family
• adding a new gesture may require retraining or fine-tuning

ASL addresses this by making signature learning explicit, local, and controllable.

Instead of assuming all social signals are known in advance, the robot can learn:

"This pattern means X"

and later classify similar patterns as X.

---

Context

Use ASL when the architecture needs to:

• classify temporal patterns
• learn new labels during interaction
• personalize interaction to a user
• adapt to the robot's own sensor viewpoint
• support session-level or persistent learned behaviors
• keep learned symbolic associations inspectable
• avoid relying only on fixed pre-trained recognition classes

ASL is especially useful after the basic Human Context layer has already produced HIF streams such as:

tracked skeletons
gaze estimates
facial expressions
spoken text
speaker identity
audio features
position relative to map

These HIF streams provide the semantic material from which time-series signatures can be learned.

---

HML Structure

The ASL pattern has two modes:

learn
classify

In learn mode, the ST stores a signature extracted from a recent time-series window.

In classify mode, the ST compares a new time-series window against stored signatures and emits the best matching label if confidence is high enough.

The SG forms a time-series window from incoming HIFs.
The ST then either learns a new signature or classifies the current window against stored signatures.

The policy determines:

• the length of the time window
• whether the ST is in learn mode or classify mode
• which signature dictionary is used
• confidence thresholds
• persistence rules
• user or session scope
• whether a new signature requires confirmation

---

Participants

Participant	HML Role	Responsibility
Input HIF stream	HIF stream	Sequence of semantically meaningful observations
SG	Sync Gate	Builds a coherent time-series window
Time-Series Window HIF	HIF	Carries a bounded temporal sequence
ST	Semantic Transformer	Learns or classifies the window
Policy	Policy artifact	Controls mode, thresholds, persistence, and scope
λ signature expert	Semantic Operator	Extracts, stores, compares, or classifies signatures
Signature dictionary	Semantic memory	Maps learned signatures to labels
Output HIF	HIF	Carries learned or classified semantic information

---

Flow

ASL has two main flows.

Learning mode

1. HIFs arrive over time.
2. SG groups them into a time-series window.
3. A learning cue activates learn mode.
4. ST receives the window and the target label.
5. λ extracts or stores a signature.
6. The signature is inserted into the selected dictionary.
7. Output HIF records what was learned.

Classification mode

1. HIFs arrive over time.
2. SG groups them into a time-series window.
3. ST receives the window in classify mode.
4. λ compares the window to stored signatures.
5. The closest match is selected.
6. If confidence passes threshold, the label is emitted.
7. Output HIF is enriched with the classification.

---

Concrete Example: Teaching a Gesture

Suppose a person waves with the right hand and says:

this gesture is "come here"

The verbal instruction acts as a learning cue.

The ASL ST looks back over a recent skeleton time window, for example the previous 1.5 seconds, and stores the motion signature under the label:

come here

Later, when a similar skeleton motion appears, the ST classifies the time-series window and enriches the output HIF:

{
  "gesture": "come_here"
}

---

Signature Dictionary

The ST maintains or accesses a dynamic dictionary.

A simplified dictionary may look like:

{
  "global": {
    "sig_17": {
      "label": "come_here",
      "modality": "skeleton_2d",
      "features": "...",
      "created_by": "user_teaching",
      "confidence": 0.91
    }
  },
  "users": {
    "alice": {
      "sig_42": {
        "label": "come_here",
        "modality": "skeleton_3d",
        "features": "...",
        "persistence": "session",
        "examples": 3
      }
    }
  }
}

The dictionary can be:

global
per-user
per-session
temporary
persistent
task-specific
culture-specific

This makes ASL suitable for both shared social knowledge and personalized interaction.

---

Matching and Classification

The pattern does not require a specific algorithm.

The signature expert may use:

• nearest-neighbor matching
• dynamic time warping
• learned embeddings
• feature extraction
• classical signal-processing features
• small local classifiers
• heuristic distance functions
• multimodal similarity metrics

For example:

new time-series window
  → extract feature signature
  → compare to stored signatures
  → choose nearest match
  → accept only if confidence exceeds threshold

A simple conceptual rule may be:

if min_distance(window_signature, stored_signature) < threshold:
    emit label
else:
    emit no_match

The key point is not the specific distance function.

The key point is that the system treats temporal interaction patterns as learnable and inspectable semantic signatures.

---

Why Not Just Use a Pre-Trained Neural Gesture Recognizer?

A pre-trained neural classifier can be useful, but ASL solves a different problem.

1. Fixed labels vs. online learning

A neural classifier usually recognizes a closed set of gestures.

ASL lets the robot learn exactly what is needed:

this gesture means come here
this whistle means call me
this melody means dinner time
this motion pattern means stop following

The learned signature may be kept permanently or only for the current interaction.

2. Generic training data vs. robot embodiment

A pre-trained model may have learned gestures from ideal viewpoints.

A real robot may be:

• shorter than a human
• looking from below
• using a wide-angle camera
• mounted on a moving base
• seeing partial skeletons
• operating in a specific room layout

ASL can learn from the robot's own sensor perspective.

The signature may be based on:

2D skeleton from the robot camera
3D skeleton relative to the map
audio time series
multimodal HIF stream

3. Static classifier vs. personalized interaction

ASL can maintain a different dictionary per user.

This allows the robot to learn how a specific person gestures, whistles, or signals.

A generic classifier may detect a standard waving gesture, but it will not naturally learn:

Alice's way of calling the robot
Bob's family-specific hand signal
a session-only training gesture
a temporary rehearsal cue

4. Opaque model vs. inspectable memory

ASL stores explicit learned associations.

A developer or authorized operator can inspect, approve, remove, or scope them.

This is much easier than inspecting a gesture embedded only inside a model's weights.

---

Beyond Gestures

The gesture example is only one use case.

ASL applies whenever a time-series pattern can be labeled and later recognized.

Domain	Example
Gesture	“this gesture means come here”
Audio	“remember this whistle as call me”
Music	“this melody means dinner time”
Motion	“when I move like this, follow me”
Social rhythm	repeated turn-taking or approach pattern
Multimodal ritual	gesture + phrase + gaze pattern

For example:

remember this whistle as "call me"

may create:

{
  "signature_type": "audio_melody",
  "label": "call_me",
  "scope": "user:alice"
}

---

Output HIFs

ASL may emit either a learning result or a classification result.

Learning result

{
  "type": "SignatureLearningHIF",
  "properties": {
    "label": "come_here",
    "stored_signature": "sig_42",
    "modality": "skeleton_3d",
    "dictionary_scope": "user:alice",
    "persistence": "session"
  },
  "processing_history": [
    "GestureWindowSG",
    "AdaptiveSignatureLearnerST"
  ]
}

Classification result

{
  "type": "GestureHIF",
  "properties": {
    "gesture": "come_here",
    "source_window": "recent_skeleton_window",
    "matched_signature": "sig_42",
    "dictionary_scope": "user:alice"
  },
  "confidence": {
    "match_confidence": 0.88
  },
  "processing_history": [
    "GestureWindowSG",
    "AdaptiveSignatureLearnerST"
  ]
}

---

SOCIAL Principles Supported

Within the SOCIAL framework, ASL supports:

S — Separated Contexts

ASL keeps the raw time-series, learned signature, classification label, user scope, and persistence policy separate.

A motion pattern is not automatically a command.
It becomes meaningful only after a learned or validated semantic association exists.

O — Open Declarative

The learned association can be represented declaratively:

{
  "signature": "sig_42",
  "label": "come_here",
  "scope": "user:alice"
}

This makes the learned meaning inspectable rather than hidden only inside a model.

C — Clear Cognition

ASL exposes whether the system learned or classified.

It can record:

which window was used
which signature was matched
which label was emitted
which confidence threshold was applied
which dictionary scope was used

I — Interpretable Gates

The pattern contains explicit gates:

learn vs classify
store vs reject
match vs no match
session vs persistent
global vs user-specific

These gates can be governed by policies.

A — Adaptive Autonomy

The robot adapts its interaction capabilities during the interaction.

It can learn a new signal, personalize it, or restrict it to the current session depending on policy.

L — Layered Validation

Learning may require validation.

For example:

only an authorized user may teach persistent gestures
low-confidence matches require confirmation
unsafe labels cannot become executable commands

---

Tradeoffs

Tradeoff	Explanation
Online learning vs. noise	Learning from live interaction may capture noisy examples
Personalization vs. fragmentation	Per-user dictionaries improve fit but may create many variants
Flexibility vs. governance	Letting users teach the robot is powerful but must be controlled
Session learning vs. persistent memory	Temporary signatures avoid pollution but disappear later
Simple signatures vs. robust recognition	Simple distance metrics are transparent but may be less robust
Viewpoint adaptation vs. generalization	Robot-specific learning may not transfer to other robots

---

Failure Modes

Failure Mode	Possible Mitigation
Wrong learning cue	Require explicit confirmation before storing
Bad time window	Use pre/post buffers and configurable window length
Noisy skeleton or audio	Store confidence and require repeated examples
Overgeneralized match	Use thresholds and negative examples
User-specific signal applied globally	Store scope explicitly
Dictionary grows without control	Add expiration, pruning, and review
Gesture resembles unsafe command	Validate label before linking to behavior
Classification confidence too low	Emit no-match or ask for clarification
Conflicting labels	Keep provenance and require resolution
Cross-user mismatch	Prefer per-user dictionary when identity is known

---

Implementation Notes

A practical ASL implementation should define:

• input HIF types
• time-window length
• buffer behavior
• learning cues
• classification thresholds
• feature extraction method
• signature representation
• dictionary scope
• persistence rules
• validation requirements
• deletion or forgetting policy
• user authorization rules
• processing-history fields

Examples of persistence policies:

session_only
user_persistent
global_persistent
task_local
expires_after_idle_time
requires_operator_approval

Examples of dictionary scopes:

global
robot_specific
user:<id>
household:<id>
session:<id>
task:<id>

---

Relationship to Previous Human Context Patterns

ASL depends naturally on the previous Human Context patterns.

SME can produce synchronized visual or audio HIFs.
EAG can decide how much perception fidelity to allocate.
The Basic Input Sublayer can produce tracked skeletons, speech, gaze, and sentiment.
TSC/TSP can interpret the verbal cue that names the pattern.

ASL then adds the missing capability:

learn and classify temporal signatures during interaction

This moves the Human Context layer from static interpretation toward interactive adaptation.

---

Minimal Summary

ASL learns and classifies time-series signatures from HIF streams.

It uses SG to form a temporal window.
It uses ST to learn or classify.
It stores signatures in a scoped dictionary.
It emits HIFs enriched with learned labels.

The pattern turns interaction into a teaching channel:

"this gesture is come here"
  → store signature
  → recognize similar future gestures