SignBridge: mapping spoken video to a sequence of Indian Sign Language concepts

A weekend MVP, then a separate avatar research track

I wanted to explore how spoken-video information could be made more accessible to people who rely on sign language, starting with Indian Sign Language (ISL).

SignBridge takes a spoken video, transcribes it, maps the meaning to a sequence of sign-language concepts, and plays those signs back in sync with the video.

It began as a weekend-scale MVP. A research track followed, testing whether motion from real Indian Sign Language (ISL) clips could drive a rigged 3D avatar. That research is isolated and not wired into the app.

My role: built the SignBridge prototype hands-on, using Claude Code as an AI coding assistant. The seven commits are dated 8–9 August 2026. The later FRIEND and collision research exists only as local, uncommitted work.

Captions are not the same thing as sign language

The project spec states the problem in two lines: spoken video content is not automatically available in sign-language form, and captions help many people but are not the same thing as sign language.

The spec lists where that matters: YouTube videos, online courses, news, government announcements, educational content, tutorials, company training and public information.

What the spec leaves open, and this prototype does not answer, is whether automatically generated signing is accurate and useful to the people it is meant for.

Built for sign-language accessibility, not tested with Deaf users

The spec names one user group: people who benefit from sign-language accessibility.

No user research or testing with Deaf users is recorded.

No Deaf reviewer or sign-language interpreter has checked the translations or the avatar motion. The project notes treat review by a qualified ISL interpreter as a requirement, not an option.

Prove feasibility first, and write down what it doesn’t prove

The project was scoped as “a weekend-scale MVP/research prototype, not a finished product,” with one aim: a pipeline proving that speech → to meaning → to sign sequence → to visual is technically feasible.

The README sets that aim next to a plain statement of what the prototype is not.

The project spec, a master prompt for the AI coding assistant committed with the first version, sets the principle: work with meaning, not word by word. Mapping each English word to an image “does not scale and does not accurately represent how sign languages work,” so the pipeline goes from speech to meaning, then to a sign-language representation, a sign sequence and a visual.

Indian Sign Language (ISL) was the initial focus. The spec warns against treating ASL, ISL, BSL and other sign languages as the same language, and the weekend MVP targeted short videos of about 30 seconds to 2 minutes.

The avatar research ran as a sequence of questions. Can motion be extracted from a real ISL clip, frame by frame, with tools free to run locally? Can it be retargeted onto a rigged avatar? Does the same code hold on a second sign? Where does it break when the hands touch?

Each answer was measured against the source clip. The project’s own evaluation scripts compare human and avatar wrist trajectories and report error as a fraction of arm length.

A weekend scope, a small vocabulary, and licences that limit what can be shown

The spec lists the MVP’s non-goals: authentication, payments, subscriptions, social features, a mobile app, real-time live translation, a 3D avatar, perfect sign-language translation and large-scale cloud infrastructure.

The MVP focused on pre-recorded video; real-time live translation was explicitly listed as a non-goal. Reason not recorded.

• The backend processes one video at a time, with no authentication and no database; records are stored as JSON files.
• ISL only, with a small curated vocabulary. Words that are not mapped fall back to fingerspelling.
• The README rules out use for “medical, legal, emergency, or other high-stakes content without human review.”
• The two real sign clips come from the INCLUDE dataset and are used under CC BY 4.0, with attribution.
• The avatar model (VRoid AvatarSample_A) has an unresolved redistribution question, so this page shows no avatar renders.

A six-step pipeline from spoken video to synced signs

1. Audio extractionFFmpeg pulls the audio track from the input video.
2. Transcriptionfaster-whisper transcribes the speech.
3. Concept extractionGemini, with a rule-based fallback, turns the transcript into sign concepts. Local test runs used the fallback.
4. Sign mappingEach concept is mapped to an ISL sign entry.
5. Asset resolutionEach sign resolves to a visual asset; unmapped words fall back to fingerspelling.
6. Synced playbackA React player shows the current sign in step with the video’s playback time.

The backend runs on FastAPI. The frontend is React 19 with Vite and Tailwind.

Fall back, stay small, and name failures precisely

Each of these is documented in the project spec or the research notes.

• Fall back instead of failing. Unmapped words become fingerspelling, and concept extraction falls back from Gemini to a rule-based method.
• Keep the MVP small. One video at a time, no authentication, JSON files instead of a database.
• Separate what a sign means from how it is shown. A dedicated mapping layer, and a renderer that picks a view per asset type: image, GIF, video or avatar.
• Let visual review overrule a metric that can’t see clipping. A fix that made the arms look right was kept even though the metric worsened, and, in the local, uncommitted collision research, a correction that resolved more frames but looked less natural was switched off by default.
• Name failures precisely. The FRIEND result, from local, uncommitted research, is recorded as a tracking failure and kept distinct from retargeting.

FastAPI and React for the app; MediaPipe and three.js for the research

• BackendFastAPI, FFmpeg and faster-whisper, with background processing and real stage and progress reporting.
• FrontendReact 19, Vite and Tailwind. The player syncs to video time; the renderer picks a view per asset type.
• Avatar researchMediaPipe Holistic landmark extraction from real ISL clips, retargeted onto a rigged VRoid avatar in three.js, respecting its bind pose.
• Evaluation and collisionEvaluation scripts plot trajectories and per-frame error; capsule-based hand-collision detection and a finger-curl solver.

The retargeting code has no sign-specific hacks. For THANK YOU it was generalised to two hands within the same code; FRIEND then ran through the retargeting math and UI unchanged.

Nine stages, labelled by what the evidence showed

Error figures are mean normalised position error, as a fraction of arm length (lower is better), measured on one clip per sign.

About the figures: generated by the project’s own evaluation scripts from INCLUDE dataset clips. INCLUDE dataset (Sridhar, Ganesan, Kumar, Khapra — AI4Bharat/IIT Madras), CC BY 4.0, INCLUDE dataset on Zenodo; the HELLO and THANK YOU clips were performed by a student at St. Louis School for the Deaf, Adyar, Chennai. Modified: landmark extraction and retargeting. The figures contain no people, video frames or avatar renders.

In the committed prototype (7 commits, 8–9 August 2026)

1. MVP pipeline

   Worked

   The first version ran the whole chain, from audio extraction to synced playback, showing the idea was technically feasible. Local test runs used the rule-based concept extractor rather than Gemini.

2. Async processing

   Worked

   Processing moved to the background, with real stage and progress reporting. It still handles one video at a time.

3. Sign mapping architecture

   Worked

   A dedicated mapping layer now sits between concepts and visuals, so the renderer can choose an image, GIF, video or avatar view for each asset.

4. Real ISL assets: HELLO and THANK YOU

   Worked, with caveats

   HELLO and THANK YOU became real sign clips from the INCLUDE dataset. They remain the only real signs among the 209 manifest entries.

5. Avatar renderer, as isolated research

   Worked, with caveats

   three.js test pages load a rigged VRoid avatar, and MediaPipe Holistic extracts landmarks from real ISL clips to drive it.

   None of this is wired into the app, and the avatar’s licence question is unresolved.

   The spec had listed a 3D avatar among the MVP’s non-goals. This research track came afterwards; the reason for expanding into it is not recorded.

6. HELLO motion retargeting

   Worked, with caveats

   Landmarks from one HELLO clip were retargeted onto the avatar, respecting its bind pose.

   The first, clamp-based approach left a mean wrist error of 0.2405 and a mean elbow error of 0.1954.

   

7. Analytical IK

   Worked

   A two-bone IK reach correction cut mean wrist error on the same HELLO clip from 0.2405 to 0.0354, roughly 85%. Mean elbow error fell from 0.1954 to 0.0696.

   

8. Bimanual retargeting: THANK YOU

   Worked, with caveats

   THANK YOU turned out to be two-handed, discovered mid-phase rather than assumed. The retargeter was generalised to two hands with the same code and no sign-specific hacks.

   Hand detection: right 45/53 frames (85%), left 40/53 (75%). Mean wrist error: 0.1238 right, 0.1330 left.

   It exposed a new failure mode, the avatar’s garment hiding the hands, and two defects that were fixed: the hands converged to 16–22% of shoulder width against the signer’s ~about 54% (52–59% after the fix), and the resting arm clipped into the hip.

   

1. FRIEND occlusion stress test

   Exposed a limit Local, uncommitted

   FRIEND was chosen for genuine hand-to-hand contact, after inspecting real footage. The retargeting math and UI generalised unchanged, but source hand-tracking degraded: the right hand was detected in 20 of 59 frames (34%), against 36/47 (77%) for HELLO and 45/53 (85%) for THANK YOU.

   A dropout of about 20 frames after contact remains unexplained and unfixed. It is recorded as a tracking failure, kept separate from retargeting.

   Capsule-based collision detection found 16 frames where the hands penetrate each other. A finger-only correction resolved 8 of them but looked less natural than the uncorrected pose, so it was switched off by default. A redesigned, coordinated finger-curl solver resolved 6 of 16 (to 2 mm or less). With it, the worst remaining frame-to-frame rotation is 23° beyond the source motion’s own rate of change; earlier, before the temporal fixes, one finger joint had jumped 163° between two adjacent frames. Palm-against-palm penetration is still out of reach.

   

Feasibility shown for the pipeline and two signs; a limit found on the third

• The core pipeline is feasible: a spoken video goes in, and a sign sequence plays back in sync with it, running locally.
• HELLO and THANK YOU were retargeted from real ISL clips onto a rigged 3D avatar, including a two-handed sign, with the same code.
• In the local, uncommitted research, FRIEND showed where the approach breaks: source hand-tracking during hand-to-hand contact, which is recorded separately from retargeting. Palm-against-palm penetration on the avatar also remains out of reach.
• Not established: linguistic correctness, usefulness to Deaf viewers, or coverage beyond two real signs. The prototype is not deployed.
• Facial and head interaction was explicitly left untested.

Retargeting generalised; hand-tracking was the limit

The same retargeting code handled a one-handed sign (HELLO), a two-handed sign (THANK YOU) and, in the local, uncommitted research, a contact sign (FRIEND) without sign-specific changes.

The weak point was upstream, in source hand-tracking. Right-hand detection fell to 34% on FRIEND, where the hands touch, from 77% on HELLO and 85% on THANK YOU.

The error metric alone was not a reliable judge: it measures trajectory shape, so a fix that removed visible clipping scored worse.

Collision correction, also in the uncommitted research, helped only in part. The version that resolved the most frames looked less natural, and palm-against-palm penetration is unsolved.

Error values are after the hip-clearance fix. “Hand detected” counts frames where the source hand-tracker found that hand. The FRIEND rows come from local, uncommitted research.

What this project taught me

1. A metric is only as good as what it can see. The trajectory metric said the hip-clearance fix made things worse; the avatar on screen said the opposite. The fix was kept and the cost written down, because a number that can’t see clipping can’t be the only judge.
2. Test the case you expect to break. HELLO and THANK YOU made the retargeter look general. FRIEND, picked because the hands actually touch, showed that the limit was upstream, in hand-tracking, rather than in the retargeting.
3. Fixing more frames is not the same as looking right. The finger-only correction resolved 8 of 16 penetrating frames but looked less natural than doing nothing, so it was switched off by default and the approach redesigned.
4. Write the limits down first. The project brief listed what SignBridge is not before any code was written, and the case study keeps those limits: 207 of 209 signs are placeholders, and no Deaf user or interpreter has reviewed the output. The brief’s own stated purpose is a foundation “that can later be validated and improved with Deaf/sign-language communities and qualified interpreters.”