Neural Magic provides AI model deployment solutions, focusing on streamlining AI model deployment and maximizing computational efficiency for enterprise inference server solutions.

Neural Magic

Together with our community, we engineer sparse LLM, CV, and NLP models that are more efficient and performant in production. Why does this matter? Sparse models are more flexible and can achieve unrivaled latency and throughput performance on your private CPU and GPU infrastructure. Check us out on GitHub and join the Neural Magic Slack Community to get started with software-delivered AI.

New Enterprise Associates

Amdocs

Comcast Ventures

Andreessen Horowitz

Ridgeline

Pillar VC

GuideLLM Released by Neural Magic: A Powerful Tool for Evaluating and Optimizing the Deployment of Large Language Models (LLMs)

GuideLLM is a comprehensive solution that helps users gauge the performance, resource needs, and cost implications of deploying large language models on various hardware configurations. By simulating real-world inference workloads, GuideLLM enables users to ensure that their LLM deployments are efficient and scalable without compromising service quality. This tool is particularly valuable for organizations looking to deploy LLMs in production environments where performance and cost are critical factors.

Key Features of GuideLLM

GuideLLM offers several key features that make it an indispensable tool for optimizing LLM deployments:

➡️ Performance Evaluation: GuideLLM allows users to analyze the performance of their LLMs under different load scenarios. This feature ensures the deployed models meet the desired service level objectives (SLOs), even under high demand.

➡️  Resource Optimization: By evaluating different hardware configurations, GuideLLM helps users determine the most suitable setup for running their models effectively. This leads to optimized resource utilization and potentially significant cost savings.

➡️  Cost Estimation: Understanding the financial impact of various deployment strategies is crucial for making informed decisions. GuideLLM gives users insights into the cost implications of different configurations, enabling them to minimize expenses while maintaining high performance.

➡️  Scalability Testing: GuideLLM can simulate scaling scenarios to handle large numbers of concurrent users. This feature is essential for ensuring the deployment can scale without performance degradation, which is critical for applications that experience variable traffic loads.....

Read our full take on this: https://lnkd.in/gsGBGjpj

GitHub: https://lnkd.in/gfQeCA-x

Neural Magic Mark Kurtz Saša Zelenović Alex Matveev Nicole Kim Robert Shaw Brian Stevens #opensource #ai #llms

🌟 Join the Future of Efficient LLM Deployments with GuideLLM! 🌟

Exciting news! With Neural Magic, we're thrilled to introduce GuideLLM—an open-source tool that simplifies evaluating and optimizing LLM deployments for everyone. Whether you're developing chatbots, summarization tools, or any other LLM application, GuideLLM offers crucial insights to streamline your development process.

Key features and benefits:
- Performance Evaluation: Ensure your LLMs perform under real-world conditions.
- Cost Efficiency: Optimize deployments to save resources and cut expenses.
- Scalability Testing: Get your models ready to handle high user loads.

We're actively enhancing GuideLLM with an intuitive UI and more features like accuracy evaluation. Your feedback and contributions are vital as we push the boundaries and ensure its benefits reach every user and company. Explore the GitHub repository, test it out, and join us in this journey of innovation together!

Check out the project here: https://lnkd.in/ef8MSJWQ

#opensource #efficientai #llms #community #llama31

Our #vLLM Office Hours continue on Thursday, September 5th, at 2PM ET / 11AM PT! Learn more and sign up: https://lnkd.in/euF8m73q

A new Mixture of Experts (MoE) model was added to the latest @MLPerf Inference v4.1 benchmark with results out today! The addition of MOE to the benchmark suite adds a more comprehensive evaluation of LLM capabilities, reflecting the latest advancements in AI technology.   Curious about the selection of this new MoE benchmark?  Read more: https://lnkd.in/g7piFmYE

#MLCommons #MLPerf #MoE #Benchmarking #MachineLearning #AI #ML #Inference

🌟 New Featured Community Model: FP8 Llama-3.1-Storm-8B! 🌟

A big shoutout to Ashvini Jindal, Pawan Rajpoot, Ankur Parikh, and Akshita Sukhlecha for their incredible model contribution to the open-source community. This model outshines the existing Llama 3.1 8B in multiple benchmarks, making it an excellent choice for applications requiring both speed and accuracy. Best of all, it was compressed by utilizing Neural Magic's LLM Compressor, enabling up to 3X faster inference than the base model with the same accuracy! 

If you're creating or optimizing LLMs, check out LLM Compressor and share your compressed models with us at Neural Magic so we can feature your work!

Community collection: https://lnkd.in/epcCKAbM
Model: https://lnkd.in/evpSHb5T
LLM Compressor: https://lnkd.in/eTFvDyqF

#llama31 #opensource #generativeai #community

Thank you to everyone who joined our recent vLLM office hours with Woosuk Kwon, co-creator of vLLM! We explored vLLM on AMD GPUs and Google TPUs, covered performance benchmarks, and discussed what's next. Missed it? Catch the recording here: https://lnkd.in/eYmSccN4

When you quantize down an LLM model, can you maintain its accuracy?

I used to think no, but the data says otherwise. For example, Llama 3.1 405B FP16 has an average Accuracy score of 86.63. If you quantize down this model from FP16 to INT4, its Accuracy score falls to 86.47. That’s a recovery ratio of 99.81%. 🔥 🔥 

In return, you get 4x more VRAM and higher performance. As people struggle to fit their models onto 8x H100s, a tiny loss in accuracy for 4x more VRAM sounds like a great bargain.

Secret sauce seems to be the GPTQ algorithm, combined with neural networks being robust to quantization.

Source: Neural Magic's Llama-3.1 70B and 405B INT4 models on Hugging Face. The link's in the comments.

#opensource #llama3

📢 Introducing LLM Compressor: SOTA open-sourced framework for compressing LLMs (including Llamas)! 📢

Starting as an internal research project at Neural Magic, I could never imagine that nearly six years later, what I called neuralmagicML would evolve into the incredibly powerful and capable LLM Compressor framework. Thanks to the hard work of our incredible engineering team, this tool can compress LLMs of any size to remarkable levels while recovering full accuracy. (such as our recent Llama 3.1 405B results!)

We are thrilled to announce that we have donated this library and its innovative techniques to the vLLM community. By offering efficient, performant, and accurate solutions for large language models, we aim to empower researchers, hackers, and enterprises.

Why This Matters:
- Cutting-Edge Algorithms: Implement the latest techniques and best practices for top-tier model performance without extensive research.
- Superior Flexibility: Enjoy various compression techniques, quantization schemes, and sparsity options for a solution that fits your use cases.
- Community-Driven: This tool is open-sourced and seamlessly integrated with vLLM and Hugging Face to ensure compatibility and encourage future contributions.

We are excited to witness how the community will leverage this incredible tool! Dive deeper into LLM Compressor by exploring our blog and the repo:
- Blog: https://lnkd.in/eHwW5snX
- Repo: https://lnkd.in/eTFvDyqF

#llms #opensource #ai #generativeAI #llama31

Exciting times ahead for LLMs and the open-source community! 💪 #AI #LLMs #opensource

Neural Magic Releases LLM Compressor: A Novel Library to Compress LLMs for Faster Inference with vLLM

Neural Magic has released the LLM Compressor, a state-of-the-art tool for large language model optimization that enables far quicker inference through much more advanced model compression. Hence, the tool is an important building block in Neural Magic’s pursuit of making high-performance open-source solutions available to the deep learning community, especially inside the vLLM framework.

LLM Compressor reduces the difficulties that arise from the previously fragmented landscape of model compression tools, wherein users had to develop multiple bespoke libraries similar to AutoGPTQ, AutoAWQ, and AutoFP8 to apply certain quantization and compression algorithms. Such fragmented tools are folded into one library by LLM Compressor to easily apply state-of-the-art compression algorithms like GPTQ, SmoothQuant, and SparseGPT. These algorithms are implemented to create compressed models that offer reduced inference latency and maintain high levels of accuracy, which is critical for the model to be in production environments....

Read our full take on LLM Compressor here: https://lnkd.in/g-smSZQF

GitHub: https://lnkd.in/gf3Hh3fd

Robert Shaw Mark Kurtz Sara Adkins Benjamin Fineran Neural Magic

Domain Not Resolving

HSTS

DNSSEC

Amazon

nginx

reCAPTCHA

Cloudflare Hosting

Person Schema

Google

Google Cloud

CTO & Co-Founder

Alex Matveev

Chief Executive Officer

Brian Stevens

Director of Machine Learning

Mark Kurtz

Machine Learning Research Team Lead

Alexandre Marques

Engineering Manager, Machine Learning

Benjamin Fineran

Engineering

Dan Huang

Full Stack Tech Lead and Manager

Danny Guinther

Business Development Lead

Kierstin Mallett

Marketing and Product

Nicole Kim

Head of Machine Learning

Ricky ҈̿҈̿҈̿҈̿҈̿҈̿Costa̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈̿҈

Gil Beyda

Greg Papadopoulos

Jamie Goldstein

Chief Scientist and Co-Founder

John O'Hara

COO / SVP Engineering

Dan Alistarh

ML Research Lead

Nir Shavit

Co-Founder

Sasa Zelenovic

Head of Marketing

United States Acquired Companies With Fewer Than 100 Employees (Top 10K)

New England Companies With More Than 10 Employees (Top 10K)

Software Companies that Exited (Top 10K)

East Coast Acquired Companies With Fewer Than 1000 Employees (Top 10K)

Neural Magic specializes in enterprise inference solutions that optimize the performance of open-source large language models (LLMs) across various infrastructures. The company focuses on maximizing computational efficiency and hardware utilization, enabling organizations to deploy AI models securely and cost-effectively.

Series A - Neural Magic

Seed Round - Neural Magic

Computer and Software Products and Electrical and Scientific Products

Somerville

Massachusetts

United States

North America

Red Hat Is Purchasing Neural Magic

Red Hat to Acquire Neural Magic to Drive Gen AI Across Hybrid Cloud Environments

Red Hat acquires tech to lower the cost of machine learning

Red Hat Announces Definitive Agreement to Acquire Neural Magic

Red Hat to acquire Neural Magic

Red Hat to acquire AI optimization startup Neural Magic

Edge Ai Software Market Size to Surpass USD 13.12 Billion by 2032 | Straits Research

Cerebras and Neural Magic Unlock the Power of Sparse LLMs for Faster, More Power Efficient, Lower Cost AI Model ...

Cerebras and Neural Magic Unlock the Power of Sparse LLMs for Faster, More Power Efficient, Lower Cost AI Model Training and Deployment

A sparsity-aware inference server for CPUs designed to run computer vision, natural language processing, and large language models efficiently.

An enterprise inference server for deploying open-source large language models on GPU infrastructure.

A sparsity-aware inference server for CPUs, designed to run computer vision, natural language processing, and large language models efficiently.

An enterprise inference server for deploying open-source large language models on GPU infrastructure, offering control over performance, privacy, and security.

An open-source optimization library for computer vision and language models, enabling sparsity to enhance neural network inference performance.

Open-source libraries for optimizing computer vision and language models through sparsity to enhance inference performance.

Italy

Europe

Japan

Asia

India

France

HTML5

Comodo PositiveSSL Certificates

Cloudflare CDN

jQuery

Wordpress

Google Tag Manager

HubSpot Forms

Google Drive

HubSpot Marketing Hub

Google Analytics

Senior Director of Engineering

Product Manager

Private Equity Associate

Private Equity Analyst

Senior Director of Eng | Committer @vllm-project

Robert Shaw

Technical Director

Director, Software Engineering

Software Engineering Manager

Postdoc

Graduate Research Assistant

HPC Systems for LLM Inference

Tyler Smith

Chief Scientist & Co-founder

Research Scientist

Postdoctoral Associate

PhD student

Chief Scientist & Co-founder at Neural Magic

COO / SVP Engineering / Operations / Co Founder

President and CEO

EVP Engineering

COO / SVP Engineering at Neural Magic

Member Board of Directors

Board Director, Member of Audit Committee

Board Director, Chair of Security and Privacy Committee, Member of Audit Committee

news impact assignment basic	general
public peers	general	Sep 3, 2024 3:11 PM	5ff81eb2
news daily refresh	general
news processing daily refresh	general
news impact assignment daily	general
news processing basic	general	Sep 3, 2024 3:37 PM	b00b29b0
company summary	general
signals	general
crunchbase trigger	general	Nov 22, 2024 7:44 AM	93452b0e
crunchbase profile	general	Nov 22, 2024 9:34 AM	016ef362
li profile	general	Sep 3, 2024 3:11 PM	74c8fac0
translate crunchbase fundingrounds	general	Nov 22, 2024 2:08 PM	0894517b
peers	general	Sep 3, 2024 3:11 PM	f9faa5d2
news basic	general	Sep 3, 2024 3:11 PM	03e23d5e
bundesanzeiger	filings
merger acquisitions	company_analysis
financials download nd	financials
private financials	financials

NEUR

COM:NEURALMAGIC

Neural Magic