Elastic Attention Governor (EAG)

Intent

The Elastic Attention Governor (EAG) manages limited computational resources by deciding:

what should be processed first
+
how deeply or expensively it should be processed

EAG connects semantic priority with resource-aware processing.

In short:

EAG = priority-aware ordering + resource-aware expert selection

The pattern usually combines two HML roles:

Priority SG
  decides which HIFs deserve attention first

ES
  decides which processing expert should handle each HIF under current resource constraints

This is not merely a scheduler.

In social HRI, attention is a semantic decision.

The robot must remain responsive to the most socially relevant people, events, or risks while still adapting to CPU, GPU, memory, latency, battery, and other runtime constraints.

Problem

HRI systems often receive more input than they can process at full fidelity.

For example:

several people are visible
several skeletons are detected
several objects require spatial grounding
several gestures may be unfolding
several microphones or cameras produce input
multiple models compete for the same GPU
CPU, memory, or latency budget changes dynamically

A naive architecture may try to process everything at maximum quality.

This can cause the robot to freeze, lag, or become socially unresponsive.

A different naive architecture may process everything cheaply.

This preserves responsiveness, but may lose the accuracy needed for socially important interactions.

EAG addresses this tension by separating two questions:

Which HIF should be processed first?

Which expert should process it under current resources?

The first question is handled by a Priority SG.
The second question is handled by an Escalation Switch (ES).

Why Not Hide This Inside One Model?

A black-box end-to-end model might appear to solve attention and resource management implicitly.

However, this creates several problems.

The system may not know whether the model learned a cheaper processing path at all.

Even if the model did learn a cheaper internal path, it may use that path all the time because that was the easiest way to optimize the training objective.

Likewise, if the model learned a high-fidelity expensive path, it may use that path all the time unless resource awareness was explicitly part of the objective, training distribution, and evaluation protocol.

In practice, it is difficult to verify that a monolithic model switches between computational routes according to a meaningful runtime resource policy.

EAG makes this switch explicit.

The architecture exposes:

the resource state
the priority policy
the prioritized queue
the expert-selection policy
the chosen expert
the reason for degraded or high-fidelity processing
the confidence and provenance of the output

This makes EAG easier to inspect, tune, test, replace, and govern.

It also allows future changes without retraining a monolithic model:

replace the priority policy
replace the neural expert
replace the heuristic expert
change the GPU threshold
add a new intermediate expert
roll back to a previous policy

EAG therefore supports transparent adaptation rather than hidden computational behavior.

Context

Use EAG when:

multiple HIFs compete for attention
not all HIFs can be processed equally deeply
resource availability changes over time
the system has multiple processing experts with different cost and fidelity
low-priority HIFs may be delayed, downgraded, or skipped
high-priority HIFs require deeper processing
the system should degrade gracefully rather than fail
resource decisions should be inspectable and policy-driven

EAG is especially useful for:

multi-person perception
2D-to-3D skeleton lifting
gaze or gesture processing
object grounding
scene understanding
social opportunity detection
edge robotics
any robot that must remain responsive under resource constraints

EAG may be unnecessary when the input rate is low, resources are abundant, or there is only one processing method.

HML Structure

Abstract Pattern

The abstract EAG structure contains:

Input HIFs
  → Priority SG
  → Prioritized Queue
  → ES
  → Processed HIFs

ResourceState HIFs influence both parts of the pattern:

ResourceState HIF → Priority SG policy and priority expert
ResourceState HIF → ES policy and expert selection

The Priority SG decides ordering.
The ES decides processing mode.

Participants

Participant	HML Role	Responsibility
Input HIFs	HIF stream	Candidate items competing for attention
ResourceState HIF	HIF / context input	Exposes CPU, GPU, memory, latency, battery, or other resource conditions
Priority SG	Sync Gate / priority gate	Scores and orders HIFs according to policy
Priority policy	Policy artifact	Defines priority criteria such as proximity, gaze, speaking, task relevance, or safety
λ priority expert	Semantic operator	Computes score or ordering
Prioritized Queue	Flow-control structure	Carries the ordered HIF stream
ES	Escalation Switch	Chooses processing expert for each HIF
ES resource policy	Policy artifact	Defines when to use high-fidelity, fallback, or degraded processing
λ high-fidelity expert	Semantic operator	Expensive but accurate expert, often neural or GPU-dependent
λ fallback expert	Semantic operator	Cheaper expert, often deterministic code or heuristic
Output HIF	HIF	Processed result with confidence, mode, and provenance

Flow

A typical EAG flow is:

Several candidate HIFs enter the EAG.
ResourceState HIFs expose current CPU, GPU, memory, and latency state.
Priority SG scores the candidate HIFs according to policy.
Priority SG emits a prioritized queue.
ES consumes the queue in priority order.
ES selects the appropriate expert according to resource policy.
The selected expert processes the HIF.
The output HIF records processing mode, confidence, resource assumptions, and provenance.

Priority and processing mode are related, but distinct.

A HIF may be high priority yet still processed with a degraded expert if resources are constrained.

A HIF may be low priority but processed with a high-fidelity expert later if resources become available.

Example Composition: 2D Skeleton to 3D Skeleton

One concrete Human Context example is 2D-to-3D skeleton enrichment.

Assume that an earlier SME has already produced several 2D skeleton HIFs.

The architecture now needs to estimate 3D joint positions.

There are at least two possible processing routes:

λ[NN]
  dedicated neural 2D-to-3D skeleton lifting model
  higher fidelity
  usually requires GPU

λ[code]
  heuristic depth-based joint estimator
  lower fidelity
  works under constrained resources

The heuristic may work by sampling depth values around each 2D joint:

for each 2D joint:
  sample depth points in a small radius around the joint
  reject invalid or noisy depth samples
  compute average or robust depth
  project the joint into 3D using camera geometry

This is a degraded mode, but it is still useful.

The robot continues to function instead of freezing when GPU is unavailable.

The Priority SG may rank people according to a policy such as:

closer to the robot first

A more social policy may combine several signals:

priority =
  proximity
+ gaze_toward_robot
+ speaking_probability
+ pointing_probability
+ task_relevance
+ safety_relevance
- estimated_processing_cost

The ES then selects the processing route according to resource policy.

Example:

if GPU is available:
    use neural 2D-to-3D skeleton lifting
else:
    use depth-based heuristic joint estimation

The output HIF should record which path was used.

ResourceState as Explicit Input

Resource state should not be hidden inside implementation code.

It should be modeled as explicit semantic input.

A ResourceState HIF may contain:

{
  "type": "ResourceState",
  "properties": {
    "cpu_load": 0.47,
    "gpu_load": 0.94,
    "memory_available_mb": 2800,
    "gpu_memory_available_mb": 300,
    "latency_budget_ms": 80,
    "battery_level": 0.62
  },
  "timestamp": "t",
  "source_history": ["robot_resource_monitor"]
}

This resource state affects both:

Priority SG:
  Which HIFs deserve attention first under current constraints?

ES:
  Which expert should process the next HIF?

The same information may be used as:

input to priority scoring
input to expert selection
criteria inside declarative policy
metadata in the output HIF
evidence for later explanation

Example Output HIF

A 3D Skeleton HIF produced under resource constraints may include:

{
  "type": "Skeleton3D",
  "source": "EAG_2D_to_3D_skeleton",
  "properties": {
    "person_id": "track_17",
    "skeleton_3d": "...",
    "processing_mode": "heuristic_depth_joint_lifting",
    "reason": "gpu_unavailable",
    "priority_rank": 1
  },
  "confidence": {
    "skeleton_3d": 0.71,
    "resource_decision": 0.95
  },
  "processing_history": [
    "PrioritySG",
    "PrioritizedQueue",
    "Skeleton3DES",
    "DepthHeuristicExpert"
  ],
  "resource_state_used": {
    "gpu_load": 0.94,
    "gpu_memory_available_mb": 300,
    "cpu_load": 0.47
  }
}

If the neural expert was used, the same HIF may instead include:

{
  "processing_mode": "neural_3d_skeleton_lifting",
  "reason": "gpu_available",
  "confidence": {
    "skeleton_3d": 0.89,
    "resource_decision": 0.96
  }
}

The key point is that degraded processing is not silent.

It remains visible in the HIF.

SOCIAL Principles Supported

Within the SOCIAL framework, EAG supports:

S-Separated Contexts

EAG keeps resource context explicit rather than hiding it inside perception code.

Human Context HIFs, ResourceState HIFs, and processing decisions remain distinguishable.

This allows the architecture to reason about human attention and robot capability without collapsing them into a single opaque state.

O-Open Declarative

Priority rules and resource policies are represented as explicit artifacts.

The system can expose statements such as:

Person track_17 was processed first because it was closest and looking toward the robot.
The heuristic expert was used because GPU load was above threshold.

C-Clear Cognition

EAG separates two cognitive responsibilities:

priority decision
processing-depth decision

This makes it easier to debug whether a failure came from bad prioritization or bad expert selection.

I-Interpretable Gates

The Priority SG and ES are explicit decision points.

They can emit decisions such as:

prioritize
delay
downgrade
skip
escalate
use_nn
use_heuristic

These decisions can be traced and audited.

A-Adaptive Autonomy

EAG allows the robot to preserve useful behavior under resource constraints.

Instead of freezing or failing, the robot may reduce fidelity, delay low-priority items, or process only the most socially relevant HIFs.

L-Layered Validation

Downstream layers can validate results according to their processing mode.

For example, an action planner may treat a neural 3D skeleton and a heuristic 3D skeleton differently because the HIF records confidence, method, and resource conditions.

Tradeoffs

Tradeoff	Explanation
Accuracy vs latency	High-fidelity experts may improve output but increase delay
Fidelity vs availability	Degraded experts preserve behavior when resources are low
Social priority vs fairness	A proximity-based policy may starve low-priority people
Responsiveness vs completeness	Processing fewer HIFs may keep the robot responsive but less informed
Policy clarity vs tuning burden	Explicit policies are inspectable but require careful calibration
GPU quality vs CPU fallback	Neural processing may be more accurate, but heuristics may be more reliable under load
Modularity vs orchestration complexity	The pattern is flexible, but introduces queueing and resource-state management

The main design tension is:

social presence over maximum fidelity

A robot that freezes while trying to process everything perfectly is socially failing.

EAG favors useful responsiveness under constraints.

Failure Modes

Failure Mode	Possible Mitigation
GPU unavailable	Use heuristic fallback and lower confidence
Too many HIFs	Use prioritized queue, bounded buffer, or drop/decay policy
Wrong priority policy	Expose priority score and allow policy tuning
Low-priority starvation	Add fairness or maximum-wait policy
Resource state stale	Timestamp ResourceState HIF and expire old values
Heuristic output noisy	Mark lower confidence and route to later validation
ES overuses expensive expert	Add budget policy and rate limits
ES overuses degraded expert	Add minimum quality thresholds
Socially relevant person ranked low	Combine proximity with gaze, speech, task relevance, and uncertainty
Hidden degraded mode	Require output HIF to record processing mode and reason
Policy conflicts	Route to validation or configuration review

Implementation Notes

An EAG implementation should usually define:

candidate HIF type
resource state schema
priority policy
priority scoring function
queue behavior
maximum queue length
starvation policy
ES expert stack
expert cost model
expert resource requirements
fallback behavior
output confidence structure
processing-history fields

A possible configuration may look like:

pattern: EAG
input_hif: Skeleton2D
resource_hif: ResourceState
priority_sg:
  policy:
    proximity_weight: 0.45
    gaze_weight: 0.25
    speaking_weight: 0.15
    task_relevance_weight: 0.15
    max_wait_ms: 300
  output: prioritized_queue
es:
  policy:
    use_neural_if:
      gpu_load_below: 0.70
      gpu_memory_available_mb_above: 1200
      latency_budget_ms_above: 60
    fallback: depth_heuristic
  experts:
    - name: neural_3d_lifter
      type: nn
      requires_gpu: true
    - name: depth_joint_heuristic
      type: code
      requires_gpu: false

The pattern does not require YAML, ROS, Python, or any specific middleware.

The key is that both priority and expert selection remain explicit and inspectable.

Related Patterns

EAG is closely related to:

Synchronous Multi-Extractor (SME) — SME may generate many candidate HIFs that EAG later prioritizes.
System Integrity and Agency Handler (SIAH) — SIAH may produce ResourceState or integrity HIFs that inform EAG.
Task Prerequisite Resolver (TPR) — TPR may depend on EAG outputs when deciding whether enough information exists for a task.
Social Opportunity TPR — social opportunity may depend on where attention is allocated.
Layered Validation — downstream validation may treat high-fidelity and degraded outputs differently.
Adaptive Autonomy — EAG contributes to resource-aware autonomy adjustment.

Minimal Summary

EAG receives multiple candidate HIFs.
Priority SG orders them according to social, task, and resource policy.
ES processes them in that order using the best available expert.
When resources are constrained, ES may choose a cheaper fallback expert.
The output remains useful, explainable, and marked with its processing mode.

EAG is a foundational pattern for embodied AI on edge hardware:

It does not merely reduce computation.
It preserves socially useful behavior under changing resource conditions.

Intent​