Galileo lowers a Fortune 50 CPG's risk associated with monitoring prompts

A Fortune 50 consumer products company, uses Galileo to accelerate their GenAI evaluations and experimentation for key customer service and customer satisfaction use cases. The organization's Research & Development focuses on innovative programs, including their GenAI initiatives.
One year ago, they had their own chatbot advisor which would nudge conversations, and generate specific clarifying questions to get at root issues. They went looking for a tool to test out prompts, chains, and questions, and to look at all the complex layers therein. And, they knew that they'd need a monitoring tool down the road.

CHALLENGE

Prior to Galileo, the global leader's GenAI team was using Jupyter Notebooks, docs, and text to run and log tests and experiments. They spent excessive time manually evaluating systems and chatbot outputs for hallucinations. As prompts and systems became more complicated, it was no longer simple RAG. Supervised learning became a limitation where you can only identify known knowns. They needed a scalable method to evaluate customer sentiment across all online reviews for their products.

SOLUTION

Galileo connected with the organization's AI and innovation teams to address these needs. Prompt engineering and experimentation were important, but they also wanted to minimize risk by showing the results of hundreds of thousands of experiments, which brings more credibility. They leveraged Galileo's metrics plus their own custom metrics for more targeted use cases, such as knowledge graphs. Galileo was more nimble for their small pool of users, and allowed for monitoring of many thousands of experiments.

RESULTS

As a result of Galileo’s systems, the organization has reduced their time spent conducting prompt and output evaluations from weeks to days. Galileo Guardrail Metrics offered the team a consistent evaluation framework across the development lifecycle. These results and insights enabled the GenAI division to provide data-backed recommendations to leadership and key business units. Ultimately, using Galileo adds credibility to their proposals, and now reduces the risk they were targeting.