Sina science and technology news Beijing time on May 17 morning news, it is reported that Google released the latest large language model last week to use training data, is almost 2022 of the previous generation of model 5 times. As a result, the model performs better in programming, math, and creative writing.
Google announced PaLM2, its new universal large language model, at its I/O developer conference. The model was trained on 3.6 trillion tokens, according to internal documents. Tokens refer to strings of words, which are an important basis for training large language models because they teach the model how to predict the next word that might appear in a string.
The previous version of PaLM, released in 2022, used 780 billion tokens.
While Google is keen to demonstrate the strength of its AI technology and how well it can be embedded in search, email, word processing and spreadsheets, the company is reluctant to release the size and other details of its training data. Microsoft-backed OpenAI is also keeping details of its latest GPT-4 language model under wraps.
The companies say they do not disclose the information because of commercial competition. Both Google and OpenAI are trying to attract users who want chatbots instead of traditional search engines to get answers directly.
But as the AI arms race heats up, researchers are calling for more transparency.
Since the launch of PaLM2, Google has said that the new model is smaller than the previous large language model, which means that the company's technology is more efficient but can do more complex tasks. Internal documents show that PaLM2 is trained on 340 billion parameters -- an indicator of how complex the model is. The original PaLM was trained on 540 billion parameters.
Google has yet to comment.
In a blog post about PaLM2, Google said the model uses a new technique called "computer optimization expansion." This makes large languages "more efficient and better overall performance, including faster reasoning, fewer parameter calls, and lower service costs."
In announcing PaLM2, Google confirmed previous media reports that the model could be trained for 100 languages and perform a wider range of tasks. It has been used in 25 features and products, including the company's experimental chatbot Bard. In terms of scale from small to large, there are four kinds of models, namely Gecko, Otter, Bison and Unicorn.
According to public disclosures, PaLM2 is more powerful than any existing model. Facebook's LLaMA large language model, announced in February, uses 1.4 trillion tokens. The last time OpenAI disclosed the training size of GPT-3, it said it was based on 300 billion tokens. When OpenAI released GPT-4 in March, it said it had demonstrated "human-like performance" in many professional tests.
LaMDA, a large conversational language model launched by Google two years ago, was unveiled with Bard in February. The model is based on 1.5 trillion token training.
As new AI applications rapidly enter the mainstream, so does the controversy surrounding the underlying technology.
El Mahdi El Mhamdi, a senior research scientist at Google, resigned in February, citing the lack of transparency in AI technology in large part. At a congressional hearing on privacy and technology on Tuesday, OpenAI CEO Sam Altman agreed that a new regime is needed to deal with the potential problems of AI.
"For a completely new technology, we need a completely new framework." "Certainly, a company like ours has a lot of responsibility for the tools we put out there," Mr. Altman said.
Responsible Editor: Zheng Zhuo
User comments