Galileo helps a leading FinTech solution reduce mean time to detect from days to minutes

COMPANY OVERVIEW

A world leading investment and accounting solution, trusted by insurers, asset managers, corporations, and governments to manage over $6.4 trillion in assets, automates the entire investment lifecycle—offering tools for portfolio management, performance reporting, data aggregation, reconciliation, compliance, and risk management. Their comprehensive, automated platform ensures efficient and scalable investment operations across traditional and alternative asset types.

CHALLENGE

Recently, the company launched an integrated generative AI system, a first of its kind generative AI solution for investment management. The system provides customers with AI-generated content and insights for sales, service, marketing, IT, and client interactions. Users can interact with their investment data through secure, tailored prompts, and unlock investment lifecycle insights, performance comparisons, and daily portfolio overviews. The system facilitates complex data queries directly, simplifying workflows that typically require manual data aggregation and calculation.

As an investment management and accounting leader, evaluating and monitoring outputs generated by their generative AI system was critical to ensuring application accuracy, security, and explainability. Prior to launch, the team would manually review complex traffic logs and identify anomalies and unexpected behavior. While this workflow worked in pre-production, it would not scale for production traffic. They needed to streamline evaluation operations without the added overhead of building evaluation tooling themselves.

SOLUTION

The company turned to Galileo to streamline evaluation and production-monitoring operations and efficiently monitor and optimize AI performance. Using Galileo Observe’s continuous monitoring and evaluation intelligence capabilities, their AI team was able to automatically monitor all agent traffic and instantly identify anomalies and hallucinations.

Using Galileo’s granular traces and evaluation metrics like Context Adherence, Chunk Attribution, and Chunk Utilization, the team can collaboratively work with subject matter experts to quickly pinpoint and troubleshoot the issue. They are working on further automating this process with Galileo Protect, which allows the team to proactively identify and block erroneous AI responses without needing a human-in-the-loop.

RESULTS

With Galileo, their AI developers saw 30% efficiency gains in their AI monitoring workflows. The team dramatically reduced their mean-time-to-detect (MTTD) and mean-time-to-remediate (MTTR) from days to minutes. Lastly, the real-time interception offered by Galileo Protect is helping the team move towards more proactive safeguarding practices.

"Before Galileo, we could go three days before knowing if something bad is happening. With Galileo, we can know within minutes. This is key for us to bring down our mean time to detect and respond. Galileo's instrumentation gives us visibility so we can very quickly respond to anything, whether that's downtime or hallucinations.

Galileo just fills in the gap we had in instrumentation and observability."

– Distinguished Engineer