unfortunately the rendering of the hands in the large window jumps all over the place on firefox, ubuntu, razer laptop. :-(
Not quite high enough fidelity to handle ASL though.
Some issues I ran into testing it, if it's something that interests you:
- Cannot distinguish closed vs. open fingers (always adds gaps between fingers, even if they're touching) (B) - Can't handle crossed fingers (R) - Doesn't seem to like extended vs. curled fingers in some cases (H) - Other failed letters: (Q), (E?), (F?), (G), (Q), (S), (U/V)
But, when signing naturally, it seems to get enough of the shapes and orientation correct enough to understand what I'm seeing. I'm sure there's things it'd trip up on because of some of the above weaknesses in detecting hand shapes but it does seem to get movement, orientation, and position "good enough".
An "air-piano" seems well within the realms of possibility now.
Glad to see the conversation about hand tracking in the browser over here.
This demos is done under the context of a series of creative experiments on how to use real time hand tracking in the browser for creative interactions. Will be posting more experiments soon.
Tech background: I am using MediaPipe to control the hand rig in threejs. MediaPipe provides landmarks that are used to control a threejs Skeleton (hierarchy of bones with rotations).
Feel free to ask, I will answer any questions!