Member Board of Directors

Board Director, Member of Audit Committee

Board Director, Chair of Security and Privacy Committee, Member of Audit Committee

Advisor

Executive Chairman

CEO at Neural Magic. Ex CTO & VP Google Cloud, CTO & EVP Red Hat.

Brian Stevens

Neural Magic

COO / SVP Engineering

COO / SVP Engineering / Operations / Co Founder

President and CEO

EVP Engineering

COO / SVP Engineering at Neural Magic

John O'Hara

Chief Technology Officer

Director of Machine Learning

Engineering Manager - Machine Learning

Lead Software / ML Engineer

Senior Software / ML Engineer

Software Engineer

Data Analyst / Software Engineer

Chief Technology Officer @ Neural Magic | Engineering Leader and ML Researcher

Mark Kurtz

Chief Scientist & Co-founder

Research Scientist

Postdoctoral Associate

PhD student

Chief Scientist & Co-founder at Neural Magic

Alex Matveev

Head Of Marketing

Director, Engagement Marketing

Sr. Customer Marketing Manager

Customer Marketing Manager

Legislative Assistant at the Office of Senator Stephen Brewer

Head of Marketing at Neural Magic

Sasa Zelenovic

Head of Product Marketing

Director of Strategic Communications

Director of Creative Strategy, Event Programs

Strategic Communications Leader

Executive Communications Leader

Principal, Product Marketing Manager

Product Line Marketing Manager, vCloud Air

Sr. Product Marketing Manager

Nicole Kim

Head of Customer Success

Jeannie Finks

Senior Director of Engineering

Product Manager

Private Equity Associate

Private Equity Analyst

Senior Director of Eng | Committer @vllm-project

Robert Shaw

Creative Director

Graphic Designer

Creative Consultant, Multidisciplinary Artist, Illustrator and Animator

Creative Specialist

Design Contractor

Developer

Designer, Animator and Motion Graphics Designer

Graphics, Flash and Web Designer - Contractor

Graphic, Interface and Presentation Designer

Creative Director at Neural Magic

Bill Schiffbauer

Technical Director

Director, Software Engineering

Software Engineering Manager

Postdoc

Graduate Research Assistant

HPC Systems for LLM Inference

Tyler Smith

Neural Magic provides AI model deployment solutions, focusing on streamlining AI model deployment and maximizing computational efficiency for enterprise inference server solutions.

Together with our community, we engineer sparse LLM, CV, and NLP models that are more efficient and performant in production. Why does this matter? Sparse models are more flexible and can achieve unrivaled latency and throughput performance on your private CPU and GPU infrastructure. Check us out on GitHub and join the Neural Magic Slack Community to get started with software-delivered AI.

New Enterprise Associates

Amdocs

Comcast Ventures

Andreessen Horowitz

Ridgeline

Pillar VC

GuideLLM Released by Neural Magic: A Powerful Tool for Evaluating and Optimizing the Deployment of Large Language Models (LLMs)

GuideLLM is a comprehensive solution that helps users gauge the performance, resource needs, and cost implications of deploying large language models on various hardware configurations. By simulating real-world inference workloads, GuideLLM enables users to ensure that their LLM deployments are efficient and scalable without compromising service quality. This tool is particularly valuable for organizations looking to deploy LLMs in production environments where performance and cost are critical factors.

Key Features of GuideLLM

GuideLLM offers several key features that make it an indispensable tool for optimizing LLM deployments:

➡️ Performance Evaluation: GuideLLM allows users to analyze the performance of their LLMs under different load scenarios. This feature ensures the deployed models meet the desired service level objectives (SLOs), even under high demand.

➡️  Resource Optimization: By evaluating different hardware configurations, GuideLLM helps users determine the most suitable setup for running their models effectively. This leads to optimized resource utilization and potentially significant cost savings.

➡️  Cost Estimation: Understanding the financial impact of various deployment strategies is crucial for making informed decisions. GuideLLM gives users insights into the cost implications of different configurations, enabling them to minimize expenses while maintaining high performance.

➡️  Scalability Testing: GuideLLM can simulate scaling scenarios to handle large numbers of concurrent users. This feature is essential for ensuring the deployment can scale without performance degradation, which is critical for applications that experience variable traffic loads.....

Read our full take on this: https://lnkd.in/gsGBGjpj

GitHub: https://lnkd.in/gfQeCA-x

Neural Magic Mark Kurtz Saša Zelenović Alex Matveev Nicole Kim Robert Shaw Brian Stevens #opensource #ai #llms

🌟 Join the Future of Efficient LLM Deployments with GuideLLM! 🌟

Exciting news! With Neural Magic, we're thrilled to introduce GuideLLM—an open-source tool that simplifies evaluating and optimizing LLM deployments for everyone. Whether you're developing chatbots, summarization tools, or any other LLM application, GuideLLM offers crucial insights to streamline your development process.

Key features and benefits:
- Performance Evaluation: Ensure your LLMs perform under real-world conditions.
- Cost Efficiency: Optimize deployments to save resources and cut expenses.
- Scalability Testing: Get your models ready to handle high user loads.

We're actively enhancing GuideLLM with an intuitive UI and more features like accuracy evaluation. Your feedback and contributions are vital as we push the boundaries and ensure its benefits reach every user and company. Explore the GitHub repository, test it out, and join us in this journey of innovation together!

Check out the project here: https://lnkd.in/ef8MSJWQ

#opensource #efficientai #llms #community #llama31

Our #vLLM Office Hours continue on Thursday, September 5th, at 2PM ET / 11AM PT! Learn more and sign up: https://lnkd.in/euF8m73q

A new Mixture of Experts (MoE) model was added to the latest @MLPerf Inference v4.1 benchmark with results out today! The addition of MOE to the benchmark suite adds a more comprehensive evaluation of LLM capabilities, reflecting the latest advancements in AI technology.   Curious about the selection of this new MoE benchmark?  Read more: https://lnkd.in/g7piFmYE

#MLCommons #MLPerf #MoE #Benchmarking #MachineLearning #AI #ML #Inference

🌟 New Featured Community Model: FP8 Llama-3.1-Storm-8B! 🌟

A big shoutout to Ashvini Jindal, Pawan Rajpoot, Ankur Parikh, and Akshita Sukhlecha for their incredible model contribution to the open-source community. This model outshines the existing Llama 3.1 8B in multiple benchmarks, making it an excellent choice for applications requiring both speed and accuracy. Best of all, it was compressed by utilizing Neural Magic's LLM Compressor, enabling up to 3X faster inference than the base model with the same accuracy! 

If you're creating or optimizing LLMs, check out LLM Compressor and share your compressed models with us at Neural Magic so we can feature your work!

Community collection: https://lnkd.in/epcCKAbM
Model: https://lnkd.in/evpSHb5T
LLM Compressor: https://lnkd.in/eTFvDyqF

#llama31 #opensource #generativeai #community

Thank you to everyone who joined our recent vLLM office hours with Woosuk Kwon, co-creator of vLLM! We explored vLLM on AMD GPUs and Google TPUs, covered performance benchmarks, and discussed what's next. Missed it? Catch the recording here: https://lnkd.in/eYmSccN4

When you quantize down an LLM model, can you maintain its accuracy?

I used to think no, but the data says otherwise. For example, Llama 3.1 405B FP16 has an average Accuracy score of 86.63. If you quantize down this model from FP16 to INT4, its Accuracy score falls to 86.47. That’s a recovery ratio of 99.81%. 🔥 🔥 

In return, you get 4x more VRAM and higher performance. As people struggle to fit their models onto 8x H100s, a tiny loss in accuracy for 4x more VRAM sounds like a great bargain.

Secret sauce seems to be the GPTQ algorithm, combined with neural networks being robust to quantization.

Source: Neural Magic's Llama-3.1 70B and 405B INT4 models on Hugging Face. The link's in the comments.

#opensource #llama3

📢 Introducing LLM Compressor: SOTA open-sourced framework for compressing LLMs (including Llamas)! 📢

Starting as an internal research project at Neural Magic, I could never imagine that nearly six years later, what I called neuralmagicML would evolve into the incredibly powerful and capable LLM Compressor framework. Thanks to the hard work of our incredible engineering team, this tool can compress LLMs of any size to remarkable levels while recovering full accuracy. (such as our recent Llama 3.1 405B results!)

We are thrilled to announce that we have donated this library and its innovative techniques to the vLLM community. By offering efficient, performant, and accurate solutions for large language models, we aim to empower researchers, hackers, and enterprises.

Why This Matters:
- Cutting-Edge Algorithms: Implement the latest techniques and best practices for top-tier model performance without extensive research.
- Superior Flexibility: Enjoy various compression techniques, quantization schemes, and sparsity options for a solution that fits your use cases.
- Community-Driven: This tool is open-sourced and seamlessly integrated with vLLM and Hugging Face to ensure compatibility and encourage future contributions.

We are excited to witness how the community will leverage this incredible tool! Dive deeper into LLM Compressor by exploring our blog and the repo:
- Blog: https://lnkd.in/eHwW5snX
- Repo: https://lnkd.in/eTFvDyqF

#llms #opensource #ai #generativeAI #llama31

Exciting times ahead for LLMs and the open-source community! 💪 #AI #LLMs #opensource

Neural Magic Releases LLM Compressor: A Novel Library to Compress LLMs for Faster Inference with vLLM

Neural Magic has released the LLM Compressor, a state-of-the-art tool for large language model optimization that enables far quicker inference through much more advanced model compression. Hence, the tool is an important building block in Neural Magic’s pursuit of making high-performance open-source solutions available to the deep learning community, especially inside the vLLM framework.

LLM Compressor reduces the difficulties that arise from the previously fragmented landscape of model compression tools, wherein users had to develop multiple bespoke libraries similar to AutoGPTQ, AutoAWQ, and AutoFP8 to apply certain quantization and compression algorithms. Such fragmented tools are folded into one library by LLM Compressor to easily apply state-of-the-art compression algorithms like GPTQ, SmoothQuant, and SparseGPT. These algorithms are implemented to create compressed models that offer reduced inference latency and maintain high levels of accuracy, which is critical for the model to be in production environments....

Read our full take on LLM Compressor here: https://lnkd.in/g-smSZQF

GitHub: https://lnkd.in/gf3Hh3fd

Robert Shaw Mark Kurtz Sara Adkins Benjamin Fineran Neural Magic

Photo	Title	Links
	Brian Stevens CEO
	John O'Hara COO / SVP Engineering
	Mark Kurtz Chief Technology Officer
	Alex Matveev Chief Scientist & Co-founder
	Sasa Zelenovic Head Of Marketing
	Nicole Kim Head of Product Marketing
	Jeannie Finks Head of Customer Success
	Robert Shaw Senior Director of Engineering
	Bill Schiffbauer Creative Director
	Tyler Smith Technical Director

NEUR

COM:NEURALMAGIC