Neural Magic provides AI model deployment solutions, focusing on streamlining AI model deployment and maximizing computational efficiency for enterprise inference server solutions.

Neural Magic

Together with our community, we engineer sparse LLM, CV, and NLP models that are more efficient and performant in production. Why does this matter? Sparse models are more flexible and can achieve unrivaled latency and throughput performance on your private CPU and GPU infrastructure. Check us out on GitHub and join the Neural Magic Slack Community to get started with software-delivered AI.

New Enterprise Associates

Amdocs

Comcast Ventures

Andreessen Horowitz

Ridgeline

Pillar VC

GuideLLM Released by Neural Magic: A Powerful Tool for Evaluating and Optimizing the Deployment of Large Language Models (LLMs)

GuideLLM is a comprehensive solution that helps users gauge the performance, resource needs, and cost implications of deploying large language models on various hardware configurations. By simulating real-world inference workloads, GuideLLM enables users to ensure that their LLM deployments are efficient and scalable without compromising service quality. This tool is particularly valuable for organizations looking to deploy LLMs in production environments where performance and cost are critical factors.

Key Features of GuideLLM

GuideLLM offers several key features that make it an indispensable tool for optimizing LLM deployments:

➡️ Performance Evaluation: GuideLLM allows users to analyze the performance of their LLMs under different load scenarios. This feature ensures the deployed models meet the desired service level objectives (SLOs), even under high demand.

➡️  Resource Optimization: By evaluating different hardware configurations, GuideLLM helps users determine the most suitable setup for running their models effectively. This leads to optimized resource utilization and potentially significant cost savings.

➡️  Cost Estimation: Understanding the financial impact of various deployment strategies is crucial for making informed decisions. GuideLLM gives users insights into the cost implications of different configurations, enabling them to minimize expenses while maintaining high performance.

➡️  Scalability Testing: GuideLLM can simulate scaling scenarios to handle large numbers of concurrent users. This feature is essential for ensuring the deployment can scale without performance degradation, which is critical for applications that experience variable traffic loads.....

Read our full take on this: https://lnkd.in/gsGBGjpj

GitHub: https://lnkd.in/gfQeCA-x

Neural Magic Mark Kurtz Saša Zelenović Alex Matveev Nicole Kim Robert Shaw Brian Stevens #opensource #ai #llms

🌟 Join the Future of Efficient LLM Deployments with GuideLLM! 🌟

Exciting news! With Neural Magic, we're thrilled to introduce GuideLLM—an open-source tool that simplifies evaluating and optimizing LLM deployments for everyone. Whether you're developing chatbots, summarization tools, or any other LLM application, GuideLLM offers crucial insights to streamline your development process.

Key features and benefits:
- Performance Evaluation: Ensure your LLMs perform under real-world conditions.
- Cost Efficiency: Optimize deployments to save resources and cut expenses.
- Scalability Testing: Get your models ready to handle high user loads.

We're actively enhancing GuideLLM with an intuitive UI and more features like accuracy evaluation. Your feedback and contributions are vital as we push the boundaries and ensure its benefits reach every user and company. Explore the GitHub repository, test it out, and join us in this journey of innovation together!

Check out the project here: https://lnkd.in/ef8MSJWQ

#opensource #efficientai #llms #community #llama31

Our #vLLM Office Hours continue on Thursday, September 5th, at 2PM ET / 11AM PT! Learn more and sign up: https://lnkd.in/euF8m73q

A new Mixture of Experts (MoE) model was added to the latest @MLPerf Inference v4.1 benchmark with results out today! The addition of MOE to the benchmark suite adds a more comprehensive evaluation of LLM capabilities, reflecting the latest advancements in AI technology.   Curious about the selection of this new MoE benchmark?  Read more: https://lnkd.in/g7piFmYE

#MLCommons #MLPerf #MoE #Benchmarking #MachineLearning #AI #ML #Inference

🌟 New Featured Community Model: FP8 Llama-3.1-Storm-8B! 🌟

A big shoutout to Ashvini Jindal, Pawan Rajpoot, Ankur Parikh, and Akshita Sukhlecha for their incredible model contribution to the open-source community. This model outshines the existing Llama 3.1 8B in multiple benchmarks, making it an excellent choice for applications requiring both speed and accuracy. Best of all, it was compressed by utilizing Neural Magic's LLM Compressor, enabling up to 3X faster inference than the base model with the same accuracy! 

If you're creating or optimizing LLMs, check out LLM Compressor and share your compressed models with us at Neural Magic so we can feature your work!

Community collection: https://lnkd.in/epcCKAbM
Model: https://lnkd.in/evpSHb5T
LLM Compressor: https://lnkd.in/eTFvDyqF

#llama31 #opensource #generativeai #community

Thank you to everyone who joined our recent vLLM office hours with Woosuk Kwon, co-creator of vLLM! We explored vLLM on AMD GPUs and Google TPUs, covered performance benchmarks, and discussed what's next. Missed it? Catch the recording here: https://lnkd.in/eYmSccN4

When you quantize down an LLM model, can you maintain its accuracy?

I used to think no, but the data says otherwise. For example, Llama 3.1 405B FP16 has an average Accuracy score of 86.63. If you quantize down this model from FP16 to INT4, its Accuracy score falls to 86.47. That’s a recovery ratio of 99.81%. 🔥 🔥 

In return, you get 4x more VRAM and higher performance. As people struggle to fit their models onto 8x H100s, a tiny loss in accuracy for 4x more VRAM sounds like a great bargain.

Secret sauce seems to be the GPTQ algorithm, combined with neural networks being robust to quantization.

Source: Neural Magic's Llama-3.1 70B and 405B INT4 models on Hugging Face. The link's in the comments.

#opensource #llama3

📢 Introducing LLM Compressor: SOTA open-sourced framework for compressing LLMs (including Llamas)! 📢

Starting as an internal research project at Neural Magic, I could never imagine that nearly six years later, what I called neuralmagicML would evolve into the incredibly powerful and capable LLM Compressor framework. Thanks to the hard work of our incredible engineering team, this tool can compress LLMs of any size to remarkable levels while recovering full accuracy. (such as our recent Llama 3.1 405B results!)

We are thrilled to announce that we have donated this library and its innovative techniques to the vLLM community. By offering efficient, performant, and accurate solutions for large language models, we aim to empower researchers, hackers, and enterprises.

Why This Matters:
- Cutting-Edge Algorithms: Implement the latest techniques and best practices for top-tier model performance without extensive research.
- Superior Flexibility: Enjoy various compression techniques, quantization schemes, and sparsity options for a solution that fits your use cases.
- Community-Driven: This tool is open-sourced and seamlessly integrated with vLLM and Hugging Face to ensure compatibility and encourage future contributions.

We are excited to witness how the community will leverage this incredible tool! Dive deeper into LLM Compressor by exploring our blog and the repo:
- Blog: https://lnkd.in/eHwW5snX
- Repo: https://lnkd.in/eTFvDyqF

#llms #opensource #ai #generativeAI #llama31

Exciting times ahead for LLMs and the open-source community! 💪 #AI #LLMs #opensource

Neural Magic Releases LLM Compressor: A Novel Library to Compress LLMs for Faster Inference with vLLM

Neural Magic has released the LLM Compressor, a state-of-the-art tool for large language model optimization that enables far quicker inference through much more advanced model compression. Hence, the tool is an important building block in Neural Magic’s pursuit of making high-performance open-source solutions available to the deep learning community, especially inside the vLLM framework.

LLM Compressor reduces the difficulties that arise from the previously fragmented landscape of model compression tools, wherein users had to develop multiple bespoke libraries similar to AutoGPTQ, AutoAWQ, and AutoFP8 to apply certain quantization and compression algorithms. Such fragmented tools are folded into one library by LLM Compressor to easily apply state-of-the-art compression algorithms like GPTQ, SmoothQuant, and SparseGPT. These algorithms are implemented to create compressed models that offer reduced inference latency and maintain high levels of accuracy, which is critical for the model to be in production environments....

Read our full take on LLM Compressor here: https://lnkd.in/g-smSZQF

GitHub: https://lnkd.in/gf3Hh3fd

Robert Shaw Mark Kurtz Sara Adkins Benjamin Fineran Neural Magic

NEUR

COM:NEURALMAGIC

Neural Magic