Fewer parameters, more data

PaLM 2, Google's latest large language model (LLM), announced last week, uses nearly five times the amount of training data compared to its 2022 predecessor, enabling it to program, solve problems, and create more advanced content.

At the Google I/O developer conference, the search giant introduced PaLM 2 – a language training model based on 3.6 trillion tokens. These tokens are strings of words – the building blocks used in LLM training to predict the next word that will appear.

The previous version of PaLM was released in 2022 and trained using 780 billion tokens.

At last week's Google I/O event, Google CEO Sundar Pichai introduced the company's latest big language model, PaLM 2.

Although Google constantly showcases the power of AI in search, email, word processing, and spreadsheet tasks, the company is reluctant to disclose the size or details of its training datasets. OpenAI also keeps the details of its latest LLM training parameter, GPT-4, confidential.

Technology companies explain this by citing the competitive nature of their business. Both Google and OpenAI are racing to attract users with chatbot products instead of traditional search engines.

Compact, powerful, and cost-effective.

Google stated that PaLM 2 is more compact than its predecessors, having been trained with 340 billion parameters compared to 540 billion parameters in the previous version. This demonstrates that the company's technology is becoming more efficient in performing complex tasks.

To achieve this, PaLM 2 uses a new technique called “extended computing optimization,” which delivers “better overall performance, including faster inference with fewer parameters, thus reducing operating costs.”

Google's latest AI language model, trained in over 100 languages, is performing various tasks for 25 features and products, including the experimental chatbot Bard. PaLM 2 comes in four versions based on size, from smallest to largest: Gecko, Otter, Bison, and Unicorn.

According to publicly available documents, PaLM 2 is more powerful than any existing model. Facebook's LlaMA, launched in February, was trained on 1.4 trillion tokens. Meanwhile, the last time OpenAI publicly disclosed the training data size for ChatGPT was the GPT-3 version with 300 billion tokens.

The explosion of AI applications has generated controversy surrounding the technology. Earlier this year, El Mahdi, a senior scientist at Google Research, resigned in protest against the search giant's lack of transparency.

This week, OpenAI CEO Sam Altman also testified before the US Senate Judiciary Subcommittee on privacy and technology in the context of AI's increasing prevalence. There, the creator of ChatGPT agreed with lawmakers that new regulations are needed to govern AI.

(According to CNBC)

Source