Google partner admits to 'guessing' when assessing AI chatbot response accuracy

Google released the Bard chatbot in limited form last March, in an attempt to respond to OpenAI's ChatGPT .

To evaluate the quality of the AI chatbot’s responses, the company hired a large workforce of third-party workers. However, the partners admitted that they often did not have enough time to evaluate the accuracy of these query responses.

Appen is a contractor helping improve Google’s chatbot. Employees at the company were not told that their assignment involved Bard, but internal discussions about the new assignment began on February 7, around the time the search giant first announced its AI chatbot.

Google partner admits to 'guessing' when assessing AI chatbot response accuracy - 1 — The accuracy of the responses Bard AI provides is being questioned as Google partners admit they have to "guess" the answers in some cases.

The partners, known as “raters,” typically review Google’s search algorithms and the relevance of ads placed in search results, as well as flag malicious websites so they don’t appear on the results page.

Since January, most of the reviewers’ work has shifted to reviewing AI prompts, Insider sources said. They said they don’t have enough time to rate the bot’s responses for accuracy, so they sometimes just make a “best guess.”

Bard received criticism after people discovered the chatbot gave incorrect answers during its launch event. Google said the chatbot would get better over time and should not be viewed as a replacement for its search engine.

Before its launch in February, Google also required employees to spend two to four hours testing the bot, asking it questions and flagging answers that didn't meet the company's standards.

Contractors said they had a set amount of time to complete each task, ranging from as little as 60 seconds to more than several minutes. However, evaluators said it was difficult to evaluate responses when they did not understand the topic the chatbot was talking about, which included technical topics such as blockchain.

Each assigned task represents billable time, so employees will find ways to complete the task even if they can't accurately assess the responses the chatbot gives.

These employees say they want to get the facts right and provide the best quality chatbot experience possible, but simply don't have enough time to research the issue before giving a rating.

“You need 3 hours of research to complete a 60-second task, that's the problem we're facing right now,” one of the reviewers shared.

(Source: Vietnamnet/Insider)

Useful
Emotion
Creative
Unique
Wrath