Hopsworks provides an Enterprise AI Lakehouse platform for developing and deploying reliable AI systems. The platform allows users to orchestrate, monitor, validate, and version features and models in a single operational environment.

Hopsworks

We are Hopsworks - the Platform for Data-Intensive Artificial Intelligence with the industry's Python-centric Enterprise Feature Store. 

Launched in 2018 by the members of the Distributed Computing Group at KTH –Royal Institute of Technology and RISE SICS AB, aiming to provide the market’s go to Enterprise Data & AI platform. Our mission is to build the world’s most scalable, secure, reliable, efficient and human-friendly data-intensive machine learning platform to enable businesses to easily develop data and AI products that help their businesses thrive. Our vision is to enable the discovery of trustworthy, actionable insights in data and easily design and apply AI solutions to complex problems.

AI Seed

Frontline Ventures

Inventure

Industrifonden

Vendep Capital

We are proud to announce our biggest and most innovative release yet! 🎉

Hopsworks 4.0 introduces game-changing innovations for building AI systems, whether for batch, real-time or LLM applications, all powered by our AI Lakehouse infrastructure.

Release Highlights:

🔹 Kubernetes Deployments - Fully containerized services deployable on any Kubernetes cluster, cloud, or on-premises.

🔹 Hopsworks Query Service - Provides Python clients with up to 45 times higher throughput when reading data.

🔹 Vector Search / Embeddings - Added support for vector & similarity search to easily build GenAI applications.

The product will be generally available soon! Stay tuned.

https://lnkd.in/d2vR-aZ7

⚠️ Data Transformation; When AI Systems Fail ⚠️ 
One of the major reasons AI systems fail is not due to a lack of performance of the models. But mostly a missunderstanding of how data transformation works. Let's resolve that. 

🛒 Take the e-commerce scenario of my diagram: 
From the Data side; 
→ Current cart value (125) 
→→ This is only available at request time 
→→→ Transformation = On-demand
------
→ Historical Data (count: 5, total spend: 1000) 
→→ Required for multiple models 
→→→ Transformation = Independent to the model

From the Model side;
🔮 For inference: 
Current Cart + Historical Data →  Model-Dependent.
🤼 For Training: 
Historical Data for a specific Model →  Model-Dependent. 

This Taxonomy for Data Transformations in AI systems (link below) is critical, we have;
1️⃣ Model-Independent Transformations 
• Reusable across models 
• Think aggregations, windowed counts, joins 
• Stored in feature stores

2️⃣ Model-Dependent Transformations 
• Specific to one model 
• For feature encoding, normalization
• Applied in both training and inference

3️⃣ On-Demand Transformations 
• Require real-time data 
• Used in online inference 
• Can be backfilled for training

Why this matters:
- Prevents training-inference skew
- Enables data reuse across multiple models
- Optimizes storage and compute

#AIEngineering #MachineLearning #DataScience #FeatureStore #AiLakehouse

News from Hopsworks 🗞️

Trouble getting started on your Hopsworks journey? This week, we share a few resources to help you become an ML and AI Lakehouse pro!

In our 5-minute interview series we’ve met with several MLOps and ML advocates telling us about their experiences and contributions to the field. In one episode Paul Iusztin told us about his idea of how to master the fundamentals of MLOps in a fast paced field like AI. 

Watch all interviews here 👉 https://bit.ly/3X37gPq

This is a visual blueprint to create an LLM, End-to-End System using the FTI Framework, based on one of the many content and talks by Jim Dowling during this year. 

And this might be a bit daunting, so, what is in the diagram here?

1️⃣ Data Collection: Gathering Raw Data From Different Sources
📂 /Data Sources
│ ├──── PDFs 📄
│ │ └──── External storage (e.g., S3, Drive)
│ ├──── ClickLog 🖱️
│ │ ├──── Last Clicks
│ │ └──── Chat History
│ └──── Orders 🛒
│ ├──── Name
│ └──── Last Buy
──────
2️⃣ Data Transformation: Turning Data into Intelligence
🔧 /ML Pipelines
│ ├──── Feature Pipeline 🛠️
│ │ ├──── Extract Text
│ │ ├──── Feature Engineering
│ │ └──── Compute Embeddings
│ └──── Training Pipeline 🧠
│ ├──── Instruction Dataset
│ └──── Fine-Tuned Model
──────
3️⃣ Infrastructure: Supporting the System
🏗️ /Data Infrastructure (Hopsworks)
│ ├──── Feature Store 📊
│ ├──── Model Registry 🗂️
│ └──── ANN Index 🔍
──────
4️⃣ User Interaction: Delivering 
👥 /Users & Application
│ ├──── Inference Pipeline 💡
│ │ ├──────────── RAG (Retrieval-Augmented Generation)
│ │ ├──────────── Enriched Prompt
│ │ └──────────── Fine-Tuned Model
│ └──── Response Logging 📈
──────

If you are curious on how to build such systems; feel free to go the github repo and build it yourself ; https://lnkd.in/gHvAUtwU

Keep your ML models accurate with feature monitoring in Hopsworks 4.0! Track data changes over time and compare with training data for proactive retraining 🤓

https://bit.ly/3W7BCQm

Our research paper, "The Hopsworks Feature Store for Machine Learning", is the first feature store to appear at a top-tier database or systems conference (SIGMOD 2024). Over the summer we’ve been breaking down the results and concepts from the research into digestible parts. The blog series is now complete! 

Read the full series here:

Part 1: Modularity & Composability for AI Systems ➡️ https://bit.ly/3SFTTlv

Part 2: The Taxonomy for Data Transformations in AI Systems ➡️ https://bit.ly/4fEC8Nm

Part 3: Use all features: Snowflake Schema ➡️ https://bit.ly/4dAPN6s

Part 4: The Feature Store Makes Your Data Warehouse Easy to Use for AI ➡️ https://bit.ly/4dcVcAV

Part 5: From Lakehouse to AI Lakehouse with a Python-Native Query Engine ➡️ https://bit.ly/4dzlJYL

Part 6: RonDB: A Real-Time Database for Real-Time AI Systems ➡️ https://bit.ly/3Ag42iF

Part 7: Reproducible Data for the AI Lakehouse ➡️ https://bit.ly/4dcp7cs

In the news 📣

Over the summer we’ve been breaking down the results and concepts from our SIGMOD 2024 research paper into digestible parts. The blog series is now complete!

It’s inspiring to see what users out there do with Hopsworks! 🌟

Here is a an example that explores ML integration into an IoT application. Sean Yap used the FTI (feature, inference, training) pipeline architecture with Hopsworks feature store and model registry to integrate ML into an IoT application for predictive maintenance. Read more about the project background, view the demos and learn about the system design in Sean’s article. And don’t forget to give him a shoutout! 

Article ➡️ https://bit.ly/3Aeuqtf

AI:HOPSWORKS

Hopsworks

Analysis Not Available