New Relic launched New Relic AI monitoring (AIM), the industry’s first APM solution for AI-powered applications. New Relic is pioneering AI observability with AIM to provide engineers unprecedented visibility and insights across the AI application stack, making it easier to troubleshoot and optimise their AI applications for performance, quality, cost, responsible use of AI, and forthcoming AI regulations and standards. With over 50 integrations and features like LLM response tracing and model comparison, AIM helps teams build and run LLM-based applications with confidence.
AI-powered applications introduce new complexity to organisations. AI components like large language models (LLMs) and vector databases are often a black box for engineers, with the potential to provide inaccurate and biased results, introduce security issues, and generate large volumes of telemetry data that need to be tracked and analysed. AIM solves this by providing end-to-end observability to any AI-powered applications. Engineers can access a single view to troubleshoot, compare, and optimise different LLM prompts and responses for performance, cost, security, and quality issues including hallucinations, bias, toxicity, and fairness. It provides engineers with full visibility on all components of the AI stack alongside services and infrastructure so that they have the data they need for the responsible use of AI and to prove their compliance with forthcoming AI regulations.
“Just about every organisation is integrating AI applications into their tech stacks to provide better customer experiences and improve efficiency, with the hope of improving their bottom line. The trade-off is that AI creates complexity in their tech stack and must be adopted responsibly with security, quality, compliance, and cost top of mind,” said IDC Group Vice President Stephen Elliot. “Applying observability to AI applications is a smart and efficient way to address these complexities so that companies can scale and drive innovation. Any company that provides these solutions is ultimately enabling organisations to deliver better products and customer experiences.”
“With every organisation integrating AI into their products and processes, AI workloads are now part of modern organisations’ application architectures,” New Relic Chief Product Officer Manav Khurana. “With AI monitoring, we have applied our deep expertise from inventing cloud APM to providing end-to-end visibility into AI-powered applications to help businesses manage performance and costs, while complying with emerging AI regulations and standards.”
Key features and use cases include:
– Auto instrumentation: New Relic agents come equipped with all AIM capabilities, including full AI stack visibility, response tracing, model comparison and more for quick and easy setup.
– Full AI Stack Visibility: Holistic view across the application, infrastructure, and the AI layer, including AI metrics like response quality and tokens alongside APM golden signals.
– Deep trace insights for every LLM response: Trace the lifecycle of complex LLM responses built with tools like LangChain to fix performance issues and quality problems such as bias, toxicity, and hallucination.
– Compare performance and costs: Track usage, performance, quality, and cost across all models in a single view; optimise use with insights on frequently asked prompts, chain of thought, and prompt templates and caches.
– Enable responsible use of AI: Ensure safe and responsible AI use by verifying that responses are appropriately tagged to indicate AI-generated and are free from bias, toxicity, and hallucinations using response trace insights.
– Instantly monitor your AI ecosystem: The most comprehensive solution for monitoring the entire stack of any AI ecosystem with 50+ integrations and quickstarts including:
* Orchestration framework: LangChain
* LLM: OpenAI, PaLM2, HuggingFace
* Machine learning libraries: Pytorch, TensorFlow
* Model serving: Amazon SageMaker, AzureML
* Vector databases: Pinecone, Weaviate, Milvus, FAISS
* AI infrastructure: Azure, AWS, GCP
“By seamlessly integrating the Pinecone vector database with New Relic AI monitoring, we are providing transparency and actionable insights to help developers build better search and Generative AI applications in the enterprise,” said Pinecone Founder and CEO Edo Liberty. “We have seen incredible demand for vector databases as companies build and deploy AI applications since it is a core part of the AI stack. Now developers can build better search and Generative AI solutions by ensuring relevant and fast responses alongside their AI observability practice.”
