Zalo AI and JAIST Institute join hands with the community to develop advanced LLM

Artificial Intelligence (AI) is developing explosively and opening up unprecedented opportunities thanks to important advances in AI model research, creating the premise to promote applications as well as develop products to serve practical needs. In Vietnam, right after the strong development of ChatGPT, which led to the birth of a series of similar AI models globally, domestic research groups with different scales and potentials have joined the race by building Vietnamese large language models (LLM).

The proliferation of Vietnamese LLM models requires a set of general evaluation criteria to help developers measure model quality in order to have appropriate training strategies.

Dr. Nguyen Truong Son - Scientific Director at Zalo AI, the platform developer - evaluated VMLU Vietnamese LLM: "The Vietnamese market is lacking quality assessment standards when compared to the world."

The VMLU LLM assessment platform was developed by Zalo AI and the Japan Advanced Institute of Science and Technology (JAIST).

According to Dr. Nguyen Truong Son, this reality requires the Vietnamese AI community to join hands to create common standards to help properly and adequately evaluate Vietnamese AI models, creating a foundation for the development of increasingly better quality models.

Promote the development of new quality standards

In November 2023, Zalo AI and the Japan Advanced Institute of Science and Technology (JAIST) will cooperate to build and provide free to the community a set of standards for evaluating the quality of Vietnamese LLM models called VMLU (Vietnamese Multitask Language Understanding Benchmark Suite for Large Language Models). This is the first set of "Make in Vietnam" standards researched and launched to the community by a team of leading Vietnamese experts.

Instead of having to build their own assessment tools with their own standards, Vietnamese LLM research groups have been able to access a comprehensive and general assessment dataset.

The VMLU standards focus on 4 areas including STEM, social sciences, humanities and extension with increasing difficulty levels: Primary, Secondary, High School and Professional (undergraduate & postgraduate). With 10,880 multiple choice questions, covering 58 topics, divided into many levels, the 2023 version has helped to effectively assess the basic knowledge of LLM.

By the end of 2024, VMLU had published 45 LLMs on the rankings, received evaluation requests from more than 155 organizations and individuals, and summarized 691 downloads of the evaluation criteria and 3,729 LLM evaluations from the platform. Many domestic and foreign organizations use the VMLU standards such as VinBigData, VNPT AI, Viettel Solutions, Ho Chi Minh City University of Technology - VNU, UONLP x Ontocord - Oregon University (USA), DAMO Academy - Alibaba Group, SDSRV teams - Samsung...

In the new phase, LLM models are strongly upgraded, requiring benchmarks to more deeply assess complex competencies.

“LLM models are becoming smarter, almost fully capable of understanding and answering questions correctly. Therefore, developers are focusing more on equipping LLMs with diverse capabilities such as reading comprehension, planning, dialogue and reasoning similar to humans,” said Professor Nguyen Le Minh, Japan Advanced Institute of Science and Technology (JAIST), a partner of Zalo AI in developing the VMLU assessment set.

Responding to the increasingly diverse needs of developers, VMLU has recently launched a new set of standards to assess 3 skills including (1) Reading Comprehension (ViSQuAD), (2) Reasoning (ViDrop) and (3) Interaction (ViDialog).

VMLU 2025 standards.

The new set of standards has been launched on the VMLU website https://vmlu.ai/ for individuals and research groups to evaluate their models.

Efforts to accompany the AI mastery community

VMLU experts said they will continue to research and build more diverse evaluation sets in different domains with different levels of difficulty to evaluate large language models more comprehensively and accurately reflect users' usage patterns.

In addition, VMLU also aims to develop a set of assessment standards for the safety and integrity of the LLM model to ensure that Vietnamese LLMs are developed responsibly.

To promote the capacity and spirit of mastering new technology of Vietnamese people, VMLU's assessment standards will continue to be provided free of charge to the Vietnamese LLM research community.

The VMLU 2025 standards have been updated on the VMLU website.

As a pioneer in the field of Artificial Intelligence in Vietnam, Zalo AI also always accompanies the community in researching and developing AI solutions for Vietnamese users.

In addition to the platform for evaluating and ranking the capabilities of major Vietnamese language models, since 2017, Zalo AI has also organized the Zalo AI Challenge and the annual Zalo AI Summit forum. These events not only connect the Vietnamese AI community, but also contribute to inspiring and promoting the creation of AI technology products by Vietnamese people to serve Vietnamese people.

Source: https://znews.vn/zalo-ai-vien-jaist-dong-hanh-cung-cong-dong-phat-trien-llm-bac-cao-post1589913.html