S&P Global (NYSE: SPGI) is proud to announce the launch of S&P AI Benchmarks by Kensho. This innovative benchmarking solution assesses and ranks the quantitative reasoning abilities and expertise of large language models (LLMs) across the business and finance industry. The results of these assessments will be displayed on a scoreboard, providing a transparent and comprehensive view of the performance of different LLMs.
S&P AI Benchmarks by Kensho is set to overhaul the way financial professionals evaluate which LLM to leverage for their financial workflows. The solution measures and ranks how well LLMs can solve complex quantitative reasoning questions, understand business fundamentals, and extract relevant financial information from documents. By rigorously validating the model’s outputs, S&P AI Benchmarks by Kensho offers accurate and reliable assessments of LLM applications for financial professionals.
“We’re delivering what started as an in-house tool to help enhance our technology into our customers’ hands,” said Bhavesh Dayalji, Chief AI Officer for S&P Global and CEO of Kensho. “It serves as further evidence of S&P Global’s longstanding commitment to harnessing AI in ways that consistently enhance our data and experiences for our customers.”
S&P AI Benchmarks by Kensho stands out due to its rigorous validation process, developed in partnership with a diverse team of specialists. This includes esteemed academics, seasoned researchers, subject matter experts across various domains, and financial professionals from across S&P Global’s divisions, all contributing to the tool’s precision and efficacy.
“As GenAI and LLMs become pervasive across products and services, it is critical that we keep accuracy and reliability at the forefront of what we do,” added Dayalji. “This solution is a huge step forward in ensuring that we are building solutions our customers can rely on.”
One of the key advantages of S&P AI Benchmarks by Kensho is its secure submission process. Financial professionals can benchmark their LLMs without sharing any part of their model, only its outputs. We use the submitted outputs to create the user’s score, which will be populated on the leaderboard. Any user who wishes to have their scores removed from the leaderboard can request so at any time.
S&P Global has been at the forefront of AI for the financial services industry since acquiring Kensho in 2018. Since then, the teams have worked closely together to deliver solutions for customers using Kensho’s core capabilities of machine learning, natural language processing, and data discovery. S&P Global aims to redefine the intelligence that powers global markets with AI, embracing the transformative potential of GenAI across the enterprise to drive innovation, speed up time to market, and build sophisticated, intuitive experiences for customers.