Back to Projects
Back to Projects
MULTIMODAL ACCESSIBILITY ENGINE

AuraLink AI

A fully offline, on-device, real-time sensory transcoder for iOS — it re-projects meaning across senses a user can't access: sign language to text and speech, ambient speech to live captions and haptic prosody, environmental sound to directional cues.

github.com/Thet9354/AuraLink-AI
AuraLink AI — offline multimodal accessibility engine

The Thesis & The Problem

Accessibility tools too often stop at “it runs a model.” AuraLink is a sensory transcoder: it ingests camera and microphone streams and re-projects their meaning across a modality the user can access. American Sign Language becomes text and speech; ambient speech becomes live captions plus a haptic sense of prosody; environmental sound becomes directional alerts. Four working modalities, all on device.

The engineering claim isn't the model — it's the pipeline around it: a provably data-race-free, back-pressured stream graph that holds a hard latency budget under thermal and battery stress, with zero network dependency.

Zero egress, provable: no networking entitlement, no URLSession in the target — verified with the Network Instrument. No data leaves the phone because there is no code path that could send it.

How It Holds Real-Time Latency

  • Compile-time race safety: Swift 6 language mode, strict concurrency complete, a strict DAG of actors with exactly one @MainActor boundary at the UI sink.
  • Bounded latency via a single-slot channel: a LatestSlot<T> gives implicit back-pressure — the pipeline always processes the freshest frame and drops stale ones, capping staleness at one frame interval instead of growing an unbounded queue.
  • Allocation-free hot loop: pre-allocated pixel-buffer and tensor pools plus ring buffers give zero steady-state heap growth, verified by a flat Instruments allocations graph.
  • Quality ladder A14 → A17: a device-capability tier and a thermal / battery / memory governor visibly and gracefully degrade model quality under stress rather than hitching.

Recognition Approach

ASL v1 is a ~200-sign “Everyday Needs” vocabulary via a pose-only, rule / language-model gloss pipeline: Vision hand and body pose → normalised feature vectors → motion-energy segmentation → DTW template match → a tiny on-device language model → grammar rules → fluent text. Dynamic time warping handles signing-speed variation for free, needs few exemplars, and yields calibrated confidence. Out-of-vocabulary signs are shown honestly as “…”, never fabricated.

Tech

Swift 6 concurrency · SwiftUI + Observation · AVFoundation · Vision · Metal / MPS · Core ML on the Neural Engine · Core Haptics · CryptoKit + Secure Enclave · OSSignposter. Six development phases, each gated by a hard verification test, are complete — the build is submission-ready ahead of an August 2026 launch.