Cross-lingual NLP Solutions

Alibaba Released Babel: An Open Multilingual Large Language Model LLM Serving Over 90% of Global Speakers Researchers from DAMO Academy at Alibaba Group introduced Babel, a multilingual LLM designed to support over 90% of global speakers by covering the top 25 most spoken languages to bridge this gap. Babel employs a unique layer extension technique to expand its model capacity without compromising performance. The research team introduced two model variants: Babel-9B, optimized for efficiency in inference and fine-tuning, and Babel-83B, which establishes a new benchmark in multilingual NLP. Unlike previous models, Babel includes widely spoken but often overlooked languages such as Bengali, Urdu, Swahili, and Javanese. The researchers focused on optimizing data quality by implementing a rigorous pipeline that curates high-quality training datasets from multiple sources. Babel’s architecture differs from conventional multilingual LLMs by employing a structured layer extension approach. Rather than relying on continuous pretraining, which requires extensive computational resources, the research team increased the model’s parameter count through controlled expansion. Additional layers were integrated strategically to maximize performance while preserving computational efficiency. For instance, Babel-9B was designed to balance speed and multilingual comprehension, making it suitable for research and localized deployment, whereas Babel-83B extends its capabilities to match commercial models. The model’s training process incorporated extensive data-cleaning techniques, using an LLM-based quality classifier to filter and refine training content. The dataset was sourced from diverse origins, including Wikipedia, news articles, textbooks, and structured multilingual corpora such as MADLAD-400 and CulturaX..... Read full article: https://lnkd.in/g2m9-UiA Paper: https://lnkd.in/gu8FG7dq Model on Hugging Face: https://lnkd.in/gwze_gm3 GitHub Page: https://lnkd.in/gyd2UNFg Project Page: https://lnkd.in/gNYhmtct Alibaba.com

You’re in an AI Engineer interview. Interviewer asks: How do you handle multi language prompting effectively? Most people jump to translation APIs. Strong answer goes deeper. 1. Detect language first Never assume. Identify the user’s language and script before prompting. 2. Preserve intent, not just words Literal translation often breaks tone, context, and business meaning. 3. Prompt in the user’s language when possible Models usually respond better when instructions and output language align. 4. Use English for complex reasoning, then localize output For harder logic tasks, reasoning in English + final response in target language often works better. 5. Handle mixed language inputs Real users switch languages mid sentence. Your system should too. 6. Keep terminology consistent Especially for healthcare, finance, legal, and product names. 7. Test by language, not globally Kannada, Hindi, Tamil, Japanese, Arabic, Spanish all fail differently. 8. Build fallback layers If confidence is low, ask clarifying questions instead of hallucinating. What interviewers want to hear: You understand that multilingual AI is a product problem, not just a translation problem. #AI #GenerativeAI #PromptEngineering #LLM #AIEngineer #MachineLearning #NLP #AIEngineering Follow Sneha Vijaykumar for more... 😊

❓ If we ask a multilingual language model a factual question written on different languages, do the answers always refer to the same entity? well..not quite. 🤔 I'm happy to report that our '24 Summer Research Intern Mahardika Krisna Ihsani from @MBZUAI collaboration came to fruition in joint work with Barid Xi Ai! We study crossling consistency across LLMs 🌎🌍🌏. See the ❇️EMNLP Findings🎇preprint https://t.co/zyo37zV9r6 & thread 🧵 for details! In our work, we did the evaluation on code-switched sentence and we expect that by this setting, the model aligns the knowledge in more language-agnostic fashion. We limited scope to only consider English as the pivot language and we examined the top-5 answers rather than top-1. We discovered that query whose language is distinct from the pivot language could elicit model to answer in different entity. This finding is substantially pronounced when the writing script is different than the pivot language. Additionally, we could see that larger model doesnt give substantial consistency improved and we explored why this happened. So we examined the cross-lingual consistency across layer and we discovered that there is no monotonic improvement and this could possibly explain why. Lastly, we also tried several methods to alleviate the inconsistency bottleneck. Among the other methods, we found that training objective that promotes cross-lingual alignment shows the best improvement and alleviates bottleneck as shown by the result of xlm-align and xlm-r-cs. If you're keen to know more about the details, please check out the preprint: https://lnkd.in/gv2gb6zh. Huge thanks to the co first authors Mahardika Krisna Ihsani and Barid Xi Ai.

🌟 Excited to share our latest research on enhancing multilingual capabilities in large language models! 🌟 Introducing SPHINX, a novel multilingual synthetic instruction tuning dataset created to address the performance gap in non-English languages. By translating instruction-response pairs from English into 50 languages, we achieved impressive results. In our study, fine-tuning models PHI-3-SMALL and MISTRAL-7B using SPHINX led to significant performance improvements, surpassing other multilingual datasets in benchmarks. Incorporating N-shot examples further boosted performance, showcasing the effectiveness and efficiency of SPHINX. This advancement marks a significant step forward in making large language models more inclusive and effective across diverse languages. Our research highlights the importance of sample efficiency and diversity while minimizing dataset creation costs. Excited for further discussions and collaborations in the realm of NLP, Multilingual AI, Machine Learning, and Artificial Intelligence! 🚀 Link to the paper : https://lnkd.in/g5CP9EZc Sanchit Ahuja Kumar Tanmay Hardik Chauhan Barun Patra Vishrav Chaudhary Monojit Choudhury Arindam Mitra Luciano Del Corro Tejas Indulal Dhamecha Ahmed Awadallah Sunayana Sitaram #NLP #MultilingualAI #MachineLearning #ArtificialIntelligence #Research #Innovation

💸 Don't speak English? You must pay more! Even though modern NLP systems are elegantly end-to-end, tokenization has stubbornly remained unchanged. While recent work explores alternatives like different representations and tokenization-free models, tokenization remains critical for LLMs. The problem? Current tokenization creates massive disparities across languages. This means higher API costs, slower processing, and reduced context windows for non-English speakers. We've developed a simple fix to Byte Pair Encoding (BPE) that dramatically improves cross-lingual fairness with minimal overhead. Our parity-based approach delivers better compression and vocabulary utilization, especially for low- and medium-resource languages that need it most. The elegance is in the simplicity: existing BPE implementations need only minor modifications. If you're using BPE in multilingual settings (and you probably are), consider switching to the parity-based version. The implementation change is minimal, but the fairness gains are substantial. More information on this project in this post: https://lnkd.in/d-k9c_b3