Available online ,
doi: 10.11999/JEIT260268
Abstract:
Objective Sign language is the primary language form ensuring normal communication between hearing-impaired individuals and hearing individuals. According to World Health Organization statistics, the global population with hearing loss is large and continues to grow, which makes a bidirectional sign language–spoken language translation system highly necessary. Sign Language Recognition (SLR) and Sign Language Production (SLP) are the two core technologies of such a system. SLP aims to convert spoken language information in text or speech form into grammatically well-formed, natural, and semantically equivalent sign language action sequences, and then to generate readable sign language videos through stitching, rendering, or avatar-driven techniques. Although SLP has attracted increasing attention, current technologies still face critical bottlenecks, including difficulties in multi-channel feature co-modeling, challenges in maintaining temporal consistency, scarcity of high-quality sign language data, discrepancies between subjective and objective evaluation metrics, and constraints on real-time generation and edge deployment. Existing reviews have summarized representative methods and Transformer-based progress, but they have not sufficiently analyzed multilingual adaptation, avatar diversity, linguistic coverage, and cross-method fusion. Therefore, this review systematically examines datasets, evaluation metrics, core architectures, presentation-based generation methods, and development trends of SLP, with particular attention to the presentation forms of generated sign language content and the “uncanny valley” effect in avatar-based generation. The review aims to provide a structured reference for improving the naturalness, intelligibility, controllability, and deployability of SLP systems. Methods This review is organized around the complete technical pipeline of SLP, including datasets, evaluation metrics, model architectures, presentation-based generation methods, linguistic coverage analysis, and cross-method fusion. First, continuous sign language datasets applicable to SLP are reviewed, including CLS-Daily, RWTH-PHOENIX-2014T, How2Sign, Prompt2Sign, SignAvatars, LSE-Sign, and GSL (Table 1). Their language coverage, sample scale, data modalities, sample types, number of signers, and construction years are compared. The development directions of sign language datasets are further summarized from four perspectives: continuous expansion of data scale, more comprehensive language coverage, transition from single-modal to multi-modal fusion, and increasing realism and naturalness of scenarios and expressions. Second, objective and subjective evaluation metrics are summarized. Objective metrics include Mean Per Joint Position Error (MPJPE), Dynamic Time Warping (DTW), back-translation-based semantic metrics such as BLEU and ROUGE, and Fréchet Inception Distance (FID). Subjective metrics include Mean Opinion Score (MOS), the classical four-dimensional scoring system covering semantic expression, structure and spatial usage, fluency, and Non-Manual Marker (NMM) accuracy, and the seven-dimensional mixed evaluation framework that further introduces expressiveness, mental load, and acceptance. Third, representative deep learning architectures are reviewed, including Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Generative Adversarial Network (GAN), Variational Autoencoder (VAE), and Transformer, as well as their variants and hybrid models. On this basis, SLP methods are categorized according to the presentation form of the final sign language video or image into motion graph-based generation, conditional video/image-based generation, and digital avatar-based generation. Representative models, datasets, reported results, and major limitations are compared (Table 2, Table 3). Moreover, a linguistic-dimension analysis is conducted to evaluate the coverage of the three method categories over handshape, location, movement, orientation, and NMM (Table 5). Finally, cross-method fusion paths and future development directions are discussed. Results and Discussions The review shows that the three presentation-based SLP paradigms have distinct advantages and limitations (Table 4). Motion graph-based methods construct directed graphs from motion capture data and generate new motion sequences by graph traversal, interpolation, and transition modeling. They have natural advantages in motion continuity and physical plausibility, and can be combined with Neural Machine Translation (NMT) and image translation models for continuous text-to-pose translation. However, their performance is highly dependent on the scale, diversity, and annotation quality of motion capture data. The graph structure also suffers from limited scalability and high search complexity, which restricts motion richness, cross-language transfer, and large-scale application. Conditional video/image-based methods adopt encoder–decoder architectures and generative models such as CNN, RNN, VAE, GAN, Transformer, and diffusion models to generate sign language videos or images conditioned on text, speech, pose, or visual guidance. They can produce highly realistic videos and support end-to-end generation and multi-modal input. Representative studies have reported competitive results on datasets such as RWTH-PHOENIX-2014T, How2Sign, CLS-Daily, and Turkish Sign Language (Table 2). Nevertheless, temporal consistency and long-sequence stability remain insufficient, and problems such as action drift, error accumulation, occlusion sensitivity, and fine-grained finger motion distortion are still prominent. Digital avatar-based methods use parameterized human models, especially the SMPL family, to control body, hand, and facial movements in a unified parameter space. SMPL, SMPL+H, SMPL-X, STAR, and SMPLest-X provide different trade-offs among modeling capability, computational efficiency, and deployment cost (Table 3). Avatar-based methods offer precise pose control, standardized baselines, and strong reusability, but cross-modal temporal synchronization between facial expressions and hand movements is still inadequate. In addition, unrealistic skin texture, facial micro-expressions, eye dynamics, and motion synchronization may induce the “uncanny valley” effect, reducing the trust and acceptance of hearing-impaired users. Recent progress in neural rendering, 3D Gaussian splatting, and diffusion models, such as GaussianAvatars and DiffPoseTalk, has improved facial detail and expression controllability, but challenges remain in real-time performance, cross-identity generalization, and semantic alignment with sign language NMMs.From the linguistic perspective, the three method categories show significant differences in covering the five core parameters of sign language (Table 5). Motion graph-based methods are limited by database coverage in handshape generation and have weak absolute position control and orientation controllability. Conditional video/image-based methods can generate realistic hand textures and large-scale body movements, but they lack explicit 3D spatial modeling and are weak in fine-grained finger motion, palm orientation, and temporal alignment of facial expressions with manual signs. Avatar-based methods achieve relatively high handshape precision and precise spatial control through parameterized models, but spatial grammar modeling, orientation control, joint face–hand optimization, and NMM semantic modeling remain insufficient. Overall, no single technical route can simultaneously satisfy the requirements of temporal consistency, visual realism, structural controllability, linguistic completeness, and real-time deployment. Cross-method fusion has therefore become an important development direction. Three fusion paths are identified: the retrieval–generation path, which introduces motion graph retrieval as a prior or initialization for deep generative models; the parametric structure–neural rendering path, which combines SMPL-X-based geometric constraints with diffusion models or 3D Gaussian splatting; and the unified discrete representation–large model path, which maps continuous motions into a shared discrete codebook and uses large language models for end-to-end semantic-to-motion modeling. These paths can integrate complementary advantages, but they also introduce new problems such as inter-module error propagation, inconsistent training objectives, and increased system complexity. Therefore, unified joint optimization objectives and end-to-end training mechanisms are essential for achieving genuine fusion gains rather than simple stacking. Conclusions and Prospects SLP is a key technology for building an accessible communication bridge between hearing-impaired and hearing populations. Although rapid progress has been made in datasets, generative models, avatar modeling, and evaluation methods, large-scale practical deployment still faces multiple challenges. Future research should advance in five dimensions. First, latent-space modeling and generation paradigm innovation should be strengthened. The combination of Vector Quantized Variational Autoencoder (VQ-VAE) and Transformer can construct compact motion codebooks and reduce the complexity of long-sequence modeling, while diffusion models and flow models are expected to improve training stability, sample diversity, and generation quality. Hybrid architectures that combine the real-time capability and interpretability of VAEs with the high-quality generation of diffusion models deserve further exploration. Second, sign language dataset construction should shift from simply pursuing sample scale to coordinated development of scale, quality, scenario coverage, and annotation. Fine-grained annotations covering frame-level 3D poses, expression parameters, handshape categories, and temporal alignment should be established, and real-world corpora covering daily conversation, public services, education, and healthcare should be supplemented. Cross-lingual data sharing and pre-training mechanisms should also be explored to support low-resource sign languages and dialects. Third, neural rendering and personalized avatars should be further developed. Neural Radiance Fields and 3D Gaussian splatting can improve skin texture, dynamic lighting, and hair details, while text-driven diffusion models can support personalized appearance generation. Hybrid representations that combine parameterized models with neural rendering are expected to preserve physical plausibility and driving controllability while improving visual realism and reducing the uncanny valley effect. Fourth, a multi-dimensional evaluation system combining objective and subjective metrics should be established. In addition to geometric and temporal metrics such as MPJPE and DTW, structured linguistic evaluation dimensions should be introduced, including handshape classification accuracy, temporal alignment accuracy of NMMs, and spatial location consistency. Standardized subjective evaluation protocols and multi-dimensional questionnaires should be developed, and comprehension accuracy should be treated as a core metric. Fifth, end-to-end system integration and scenario-based deployment should be promoted. SLP should be deeply integrated with speech recognition, machine translation, and large language models to build real-time “speech/text-to-sign language” translation systems. Cloud–edge collaborative computing, knowledge distillation, neural architecture search, dynamic inference, and quantization compression can support low-latency inference on mobile and embedded devices. Scenario adaptation in education, medical consultation, judicial services, public transportation, and public service halls will be an important breakthrough for moving SLP from laboratory prototypes to large-scale public services and daily assistance.