The dedicated artificial intelligence (AI) chip LPU developed by Groq is becoming an overnight social media sensation, as its public benchmark tests went viral on social media platform X.

Groq claims to offer ' the world's fastest natural language processing models' and extensive independent testing suggests that claim may be accurate.

735783 1szzxeq groq.jpg
Dedicated AI LPU chip developed by Groq delivers superior performance for large language models in experiments.

Groq makes a dedicated AI chip called an LPU (Language Processing Unit) that is said to be faster than Nvidia's GPUs. Nvidia's GPUs are often considered the industry standard for running AI models, but early results suggest that LPUs will outperform them, at least in tasks that require high speed.

In essence, Groq’s LPU chip is a ‘compute engine’ that enables chatbots like ChatGPT and Gemini to operate at lightning-fast reception and response speeds. In tests conducted, the LPU-powered chatbot outperformed eight chatbots using other AI chips on several key performance metrics, including latency versus throughput and total response time. The LPU’s superior performance was most evident when tested against Meta’s Llama 2-70b model.

According to independent tests by Artificial Analysis, chatbots using LPU chips can achieve a throughput of 247 tokens/second. For comparison, OpenAI's ChatGPT currently only achieves a maximum throughput of 18 tokens/second. This means that ChatGPT could be 13 times faster if using Groq's chips, although this depends on many other related factors.

According to Artificial Analysis, this level of performance could open up new possibilities for large language models (LLMs) across a wide range of domains. Chatbots using Groq’s LPU chips can generate hundreds of words in a fraction of a second, allowing users to perform tasks in real time. In one test, Groq founder and CEO Jonathon Ross guided a CNN anchor through a live, verbal conversation with an AI chatbot on the other side of the world.

AI chatbots like ChatGPT, Gemini, Llama… could be significantly more useful if they responded faster. One of the significant limitations currently is that these chatbots cannot keep up with human speech in real time. The large delay makes the conversation robotic and unsatisfactory in many fields.

Despite the hype surrounding Groq’s LPU, the immediate question is how it compares to similar products in real-world applications. Unlike Nvidia’s GPU or Google’s TPU, Groq’s LPU is a specialized AI chip, designed to be optimized for specific tasks, rather than a ‘general-purpose’ AI chip.

In addition, this is also the time when major AI developers are looking to develop their own chips to avoid dependence on Nvidia products. OpenAI is said to be looking for trillions of dollars in funding to develop its own chips. OpenAI CEO Sam Altman is even considering rebuilding an entirely new AI chip architecture from scratch.

(according to Cryptoslate)