🤟 sign-language-bridge

Continuous American Sign Language → English translation.

This Space demos mamounyosef/sign-language-bridge, a multi-tier LoRA / RSLoRA fine-tune of Qwen3-VL-2B-Instruct trained on How2Sign + OpenASL by Ma'moun Yosef. The model and the research are theirs; this Space is just a demo wrapper.

What the app does

You upload or record a 1–15 second ASL clip and it returns an English translation — alongside the preprocessed clip the model actually saw, so the pipeline is visible rather than hidden.

Why the preprocessing matters

The adapter was trained with three preprocessing stages applied to every clip in every split. Feeding it a raw video would put it badly out of distribution, so the app reproduces all three from the original training code:

StageWhat it does
Pose-guided crop MediaPipe PoseLandmarker: upper-body + hand landmarks unioned across the clip, padded 25 %, snapped to a multiple of 32 — one stable box around the signing space.
CLAHE Contrast equalisation on the L channel in LAB (clip 2.0, 8×8 tiles).
Landmark overlay RTMPose Wholebody draws a 1 px skeleton straight onto the pixels: 6 upper-body joints, 21 left-hand keypoints, 21 right-hand keypoints.

Sampling and decoding match the evaluated step-4,610 checkpoint exactly: 20 fps, pixel budget 4,096 / 184,320 / 20,971,520, beam size 5, length penalty 0.6, no-repeat n-gram 4, repetition penalty 1.1, 32 max new tokens.

Turning it on

  1. Get a ZeroGPU slot — subscribe to PRO, wait out the 30-day account age, or be awarded a community GPU grant.
  2. Replace the README frontmatter with: 
    sdk: gradio
sdk_version: 6.24.0
app_file: app.py
python_version: "3.12"
startup_duration_timeout: 45m
  3. Attach zero-a10g hardware in the Space settings.

Limitations

This is a research preview. On the author's How2Sign test partition (944 clips, a custom 90/5/5 split — not the official split, so these are not comparable to published numbers): BLEU-1 19.76, BLEU-4 1.64, chrF 17.42, ROUGE-L 10.43, METEOR 9.71, WER 112.5 %.

Output is fluent English and often topically right, but frequently disagrees with the reference word for word — and it can be confidently wrong. Don't use it anywhere a mistranslation could cause harm: medical, legal, safety-critical, or emergency settings. The model has seen almost only How2Sign / OpenASL signers, framings, and lighting.