LastMile AI provides a developer platform for taking generative AI prototypes to production. The platform facilitates debugging and evaluating RAG pipelines, versioning and optimizing prompts, and managing models.

LastMile AI

We believe that AI application development extends beyond ML researchers, and that it will be software engineers applying AI across verticals and industries. Our mission is to empower software developers and product teams, not just ML practitioners, to build with generative AI.

Explainability of ML-powered recommendations is emerging as a promising application of LLMs for enterprises. 

Indeed's recent collaboration with OpenAI on their "Invite to Apply" feature showcases how effective LLMs are for explainability tasks. 

Indeed’s "Invite to Apply" feature invites users to apply to a list of recommended jobs. Typically, these job recommendations lack explanations, offering users no insight into why specific positions were suggested to them. This absence of context has been shown to reduce user trust and ultimately their pursuit of the recommended jobs. 

OpenAI and Indeed worked together to fine-tune a small GPT model that gave relevant "why" explanations for the user given their profile and the job description. Read more here: https://lnkd.in/eR84xpMm

The results were outstanding: 
➤ +20% increase in initiated job applications
➤ +13% uplift in downstream success indicating that not only were more candidates likely to apply, but that employers were finding those applicants to be a good fit, ultimately leading to more hires

Recommendation systems are the industry standard for personalizing user experiences across various domains. However, they often lack transparency, leaving users wondering about the rationale behind suggestions.
LLMs offer an effective solution to this challenge, helping drive user trust and engagement.

LLMs can add significant value to recommendation systems by solving key problems: 
1. Mitigation of data sparsity - LLMs have extensive world knowledge that basic recommender systems lack since they are trained on smaller sets of data. 
2. Natural language understanding and generation - LLMs excel at generating human-like text which makes it easier for tasks like explainability. 

If you’re interested in how to leverage LLMs for your business needs, visit us at: https://lastmileai.dev/

Human evaluation methods are a critical component in building reliable, safe AI applications.

While technical metrics and benchmarks help scale evaluation, enterprises still rely on human feedback to asses real-world performance, quality, and impact. 

It’s essential for researchers, developers, and business leaders to understand how humans are “looped” in the journey of AI development and the potential risks and biases of doing so. 

We’ve compiled a brief overview of the most common human evaluation methods that enterprises are leveraging today. 

If you’re interested in how you can best leverage human evaluation for your AI application, please reach out to us: https://lastmileai.dev/

BlackRock and NVIDIA's recent collaboration on HybridRAG highlights how domain-specific needs, particularly in finance, are shaping the design and implementation of enterprise AI systems.

HybridRAG is an innovative approach that merges VectorRAG and GraphRAG techniques to achieve more efficient information extraction particularly for processing complex financial documents. Read the full study: https://lnkd.in/ed6MU48J

1 - Why This Matters
Financial documents pose unique challenges for Large Language Models (LLMs) due to their domain-specific language, multiple data formats, and complex contextual relationships. These challenges can lead to inaccurate analyses and unreliable decision-making in financial contexts. HybridRAG addresses these issues by enhancing the accuracy and consistency of information extraction, potentially revolutionizing how we process and analyze financial data.

2 - HybridRAG
HybridRAG integrates VectorRAG and GraphRAG techniques, leveraging their strengths while mitigating limitations. VectorRAG uses dense vector representations for broad context retrieval, while GraphRAG constructs knowledge graphs for structured, relationship-rich representations.

HybridRAG combines these approaches by searching both vector space and knowledge graph simultaneously. It extracts and merges relevant text chunks and subgraphs, providing comprehensive information that integrates broad context with precise entity relationships. This unified input for the language model results in more accurate and contextually rich responses.

HybridRAG outperformed both VectorRAG and GraphRAG in:
➤ Faithfulness: Measures the extent to which the generated answer can be inferred from the provided context
➤ Answer relevancy: Measures how well the generated answer addresses the original question
➤ Context recall: Measures how completely the retrieved information covers all aspects of the correct answer. Higher recall indicates the system found more of the relevant information needed to fully answer a question.

3 - Our Perspective
We're excited to see teams exploring customized implementations for information retrieval, moving beyond one-size-fits-all approaches. We envision a future where engineering teams can seamlessly experiment with various system designs, effortlessly transitioning between approaches like GraphRAG and VectorRAG, or crafting bespoke combinations to suit their needs.

If you're interested in optimizing information retrieval, let's connect: https://lastmileai.dev/

Today OpenAI released an updated version of SWE-bench - a benchmark for evaluating LLMs' abilities to solve real-world software issues sourced from GitHub. 

SWE-bench (Verified) by OpenAI is a human-validated subset of SWEbench aimed to address issues with the original benchmark dataset to ensure more accurate evaluations. Read more here: https://lnkd.in/equ6ZuS7

The update highlights critical principles of the evaluation story: 

1. Reliable Benchmarks are Essential
As AI systems tackle increasingly complex real-world problems, we need dependable datasets and metrics metrics to assess their capabilities accurately. This ensures that AI applications can be trusted to perform as expected in diverse scenarios.

2. Continuous Improvement is Non-Negotiable
The SWEbench update revealed that even carefully designed benchmarks can underestimate model capabilities. As AI rapidly evolves, so must our evaluation methods to keep pace with these advancements.

3. Recognize Inherent Limitations
Static dataset evaluations, while valuable, have constraints such as potential data contamination and narrow risk coverage. Comprehensive evaluation strategies that go beyond single benchmarks provide a more holistic view of AI system performance.

4. Preparing for Advanced AI Requires Expertise
As we move closer to AGI, the complexity of our evaluation methods must increase accordingly. This demands deeper expertise in benchmark design and validation to ensure our tests remain relevant and challenging.

These insights emphasize the need for sophisticated, adaptable evaluation tools in the AI industry. 

At LastMile AI, we're dedicated to addressing these challenges. Our platform is designed to empower organizations to build AI applications that are not just functional, but truly reliable and safe for end-users. Learn more at: https://lastmileai.dev/

Generative AI is changing how enterprises use data -- especially unstructured data. This shift brings new challenges. 

➤ Data Access and Infrastructure 
LLMs now leverage a wider range of data sources, including unstructured text, images, and even audio. Enterprises can extract insights from previously untapped resources like internal documents, customer interactions, and social media. As LLMs become more versatile, the focus has shifted to developing data infrastructure to support seamless integration of diverse data types.

➤ LLM Strategy 
Enterprises often align their model selection with their data strategy. Open-source models allow for customization through fine-tuning but require in-house ML expertise, while closed-source models provide out-of-the-box performance but offer less control over data usage. The choice between on-premises and cloud deployment also affects data handling, with on-premises prioritizing data control and cloud focusing on scalability and ease of use. The chosen approach should balance governance, privacy, and compliance needs.

➤ Governance
Enterprises typically implement guardrails preventing exposure of sensitive data and inappropriate content. Independent audits identify biases and risks, while feedback systems enable quick issue resolution. These measures, along with proactive adaptation to evolving AI capabilities and regulations, ensure alignment with ethics and business objectives. 

If you're interested in learning how to build with AI for your business, come chat with us! https://lastmileai.dev/

The current state of the art for LLM app evaluation is to use an LLM-as-a-judge. This approach is fraught with issues because LLMs weren’t designed to be evaluators, and can also be cost-prohibitive to use as a testing harness – evals should be run regularly, potentially thousands of times a day.

LastMile’s AI research team is working on a new class of small evaluator models that improve upon the LLM-as-a-judge paradigm (check out our blog). We are rolling this out with a select-few customers, including several Fortune 500’s, and focusing on retrieval (RAG) application evaluation to mitigate hallucinations and identify performance bottlenecks in the pipeline.

If you’d like to learn more or use AutoEval for your eval needs, please reach out to us: https://lnkd.in/eBZBn2Uu

Financial institutions are actively exploring opportunities to integrate AI beyond back-office operations. This growing interest reflects a broader shift towards leveraging AI to enhance customer experiences and streamline front-end operations. 

However, this shift comes with its own set of challenges. Here's what we're seeing:
➤ Regulatory scrutiny is much higher for customer-facing AI
➤ Explainability and traceability of AI decisions are crucial
➤ Preventing AI "hallucinations" is critical for maintaining accuracy and trust

We've learned a ton about these challenges and nuances through Fintech Innovation Lab and are tackling these challenges head-on with our financial institution customers. 

📺 Interested in what we're building? Check out our demo from Fintech Innovation Lab: https://lnkd.in/ebwfXpyJ

🔍 Distributed Systems Observability with RAG Workbench

Enterprise applications leveraging Large Language Models (LLMs) are revolutionizing business processes, but they come with unique challenges. These apps often span multiple services, creating complex distributed systems that are difficult to optimize and debug.

Enter RAG Workbench – your solution for comprehensive observability in LLM-powered enterprise systems.

For instance, consider a major airline's LLM-powered customer support system. A single query triggers a complex chain: web server → authentication service → document retrieval system → LLM integration → response generation. Each step introduces potential bottlenecks and failure points.

How do they ensure consistent performance and quickly identify issues?

RAG Workbench provides: 
✅ End-to-End Tracing
Track requests across your entire system, from query input to document retrieval to LLM generation, identifying performance bottlenecks with precision.

📈 RAG-Specific Metrics
Capture crucial RAG metrics like retrieval quality and faithfulness, enabling data-driven optimization of your AI system.

🔥 Rapid Iteration
Quickly test and compare different RAG configurations, fine-tuning your system for optimal performance in production environments.

By leveraging RAG Workbench, enterprises can ensure their AI systems not only function but excel in production, leading to enhanced user experiences!

➤ Sign up for early access to RAG Workbench: https://lnkd.in/e-vDGaVM 
➤ Find the tutorial here: https://lnkd.in/eduCQVT2

Exciting news! 🚀 We've partnered with IBM watsonX to bring you Granite models on RAG Workbench! 

Granite, IBM's flagship LLM series, offers:
• Decoder-only transformer architecture
• Training on trusted enterprise data (internet, academic, code, legal, finance)
• Four specialized models

RAG Workbench helps you evaluate, debug, and optimize enterprise RAG applications with ease.

➤ See it in action! Tutorial: https://lnkd.in/dfSgbjXu
➤ Chat with our team: https://lnkd.in/e-vDGaVM

"How should I evaluate my RAG / LLM application?" repeatedly comes up in my conversations with developers. My response has gotten more consistent over time. Wrote out my thoughts and my typical response in a blog post. Check it out!

DEV:LASTMILEAI

LastMile AI