Page 85 - 2024F

78 UEC Int’l Mini-Conference No.53

53rd UEC International mini-Conference on Informatics, Sciences and Engineering

Deep Learning Based Japanese Sign Language Translation

Rattapoom KEDTIWERASAK and Hiroki TAKAHASHI
Department of Informatics
The University of Electro-Communications

Introduction Methodology

Focus on:
Japanese Sign Language (JSL) Translation
Convert sign language video to spoken language
text
Serve better accessibility and communication
Knowledge Transferring
Transfer American Sign Language (ASL)
knowledge to JSL
Challenges:
A Scarcity of JSL dataset

Fig. 4 Detail Architecture of the Knowledge Transferring
and Fine-Tuned Translation
Fig. 1 Parallel SL dataset [1] Mask Data
Differences in grammar, vocabulary, and cultural Support autoregressive translation generation
context Self-supervised Video Pretraining
Collect ASL gestures
Supervised Sign Language Translation
Transformer Encoder
Fig. 2 “Weather” Sign Language Symbols [4] Receive knowledge
Proposed Method: Fine-tune
Self-Supervised Video Pretraining for Sign Language Transformer Decoder
Translation (SSVP-SLT) [2] Learn for translation
For Sign Language (SL) knowledge transferring
Transformer-based model
For fine-tuned JSL translation [3] Conclusion
Our proposed SSVP-SLT will be beneficial for the JSL
translation. We will be able to transfer the ASL knowledge
by fine-tuning technique. The proposed model will
overcomes data limitations and demonstrates the potential
Fig. 3 Overall Structure of the Proposed Method of cross-lingual transfer learning in JSL translation. The
proposed method will be useful for better accessibility and
Expected Contributions communication.

We will develop the novel approach to overcome the
scarcity of JSL dataset by transferring knowledge from References
the pre-trained ASL model.
[1] Tanzer, G., & Zhang, B. “YouTube-SL-25: A large-scale, open-domain
multilingual sign language parallel corpus”. (2024).
We will propose the SSVP-SLT to learn ASL dataset, [2] Rust, P., Shi, B., Wang, S., Camgoz, N.C., & Maillard, J. “Towards privacy-
which are then fine-tuned by JSL dataset. aware sign language translation at scale”. In ACL, pp. 8624–8641 (2024).
[3] Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. “Sign language
transformers: Joint end-to-end sign language recognition and
The SSVP-SLT will improve the JSL translation to spoken translation”. In CVPR, pp. 10023–10033 (2020).
language with limited data. [4] “Spreadthesign”, https://www.spreadthesign.com/, Accessed: 2025-02-23