Arthur, an artificial intelligence (AI) startup based in New York City, has unveiled a new tool called Arthur Bench. This open-source tool is designed to evaluate and compare the performance of large language models (LLMs) like OpenAI’s GPT-3.5 Turbo and Meta’s LLaMA 2.
Adam Wenchel, the CEO and co-founder of Arthur, stated that Arthur Bench was developed to provide teams with insights into the distinctions between LLM providers, different prompting and augmentation methods, and customized training approaches.
The Functionality of Arthur Bench:
Arthur Bench empowers companies to assess the performance of diverse language models according to their specific use cases. The tool offers metrics that enable comparisons based on accuracy, readability, hedging, and other relevant criteria.
For those experienced with LLMs, the issue of “hedging” often arises. This refers to instances where an LLM includes superfluous language that outlines its terms of service or programming limitations, such as phrases like “as an AI language model.” These statements are generally irrelevant to the desired user response.
Adam Wenchel further elaborated that these nuanced differences in behavior can be crucial for specific applications.
Customizable Criteria and Practical Implementation:
Although Arthur has included initial criteria for comparing LLM performance, the tool’s open-source nature allows enterprises to incorporate their own criteria to suit their requirements.
For example, by taking the most recent 100 user questions and testing them against all available models, Arthur Bench highlights instances where responses significantly diverge, prompting manual review. The ultimate objective is to empower businesses to make well-informed decisions when adopting AI technologies.
Arthur Bench expedites benchmarking efforts and translates academic measures into real-world business relevance. The tool utilizes a blend of statistical measures, scores, and evaluations from other LLMs to grade the responses of desired models side by side.
Real-World Applications:
It has already found utility in various industries. Financial-services firms are utilizing the tool to expedite the creation of investment theses and analyses. Vehicle manufacturers are transforming complex technical manuals into LLMs capable of providing accurate and rapid customer support while reducing inaccuracies.
Axios HQ, an enterprise media and publishing platform, has integrated Arthur Bench into its product-development processes. The tool assists in creating a standardized evaluation framework and conveying performance metrics to the Product team effectively.
Open-Source and Collaborative Efforts:
Arthur has chosen to open-source Bench, making it available for free use and contributions from the community. The company believes that an open-source approach fosters the development of superior products and creates avenues for monetization through team dashboards.
Arthur has also announced a hackathon in partnership with Amazon Web Services (AWS) and Cohere. This collaborative event aims to encourage developers to devise new metrics for Arthur Bench. Adam Wenchel highlighted the alignment between AWS’s Bedrock environment and Arthur Bench, as both platforms facilitate informed decision-making regarding the selection and deployment of LLMs.
In conclusion, Arthur’s introduction of Arthur Bench addresses the need for comprehensive LLM evaluation tools. Its open-source nature and collaborations with industry giants like AWS position it as a valuable asset for AI development and decision-making processes. Additionally, the company’s commitment to refining LLM performance through tools like Arthur Shield demonstrates its dedication to advancing the field of artificial intelligence.
