SDF is a technology company that specializes in developing innovative solutions for data engineering challenges. The company is founded by a team of PhD engineers and startup product experts with experience at major tech firms such as Meta/Facebook, Google, and Microsoft. They focus on creating tools and products that streamline data engineering processes, emphasizing moral integrity, agile development, customer-centricity, simplicity, and vision-driven innovation.

SQL development made simple. Local execution, Intelligent Workflows, and Data Governance packed into 60MB of Rust based magic.

Want to try it out? Let us know at info@sdf.com

SDF Labs

Today, SDF Labs is excited to announce the Semantic Data Fabric.

The seeds of SDF began at Meta, where co-founders Wolfram Schulte and Michael Y. Levin architected a revolutionary system to tackle Meta’s unique data privacy, quality, and authoring challenges. Returning to first principles, SDF Labs grew their vision beyond the challenges of a single company, to a fundamental reimagining of what a data warehouse can be. SDF is the natural culmination of a team with over 12,000 citations, several dozen patents, and a combined 75+ years of programming language and compiler expertise.

At its core, SDF is a groundbreaking multi-dialect SQL compiler and build system that leverages static analysis to comprehensively examine SQL code at warehouse scale. It’s a SQL engine unlike any other - SDF considers all queries in any dialect simultaneously to build an incredibly rich dependency graph and provide a holistic, end-to-end view of data assets.

SDF’s utility comes from its built in Type System and Code Checks for SQL which enable data engineers to uncover problems org-wide and address them proactively. Data privacy, data quality, and data governance become dramatically easier to manage as a byproduct.

SDF was designed to provide safety and automation at scale. Label <1% of your columns (the sources) and these data classifiers will propagate through the warehouse automatically. Column level lineage, code checks, and automated classifiers work together to ensure data quality is enforced actively during development, not passively in production. The engine is built in Rust, and is highly performant even with millions of queries.

SDF is in private beta. Learn more at https://www.sdf.com/ or read the full announcement on our blog.

https://lnkd.in/eP86aJ5e

Extending a warm welcome to our Product Management Intern, Shelly Matskel. Before starting her MBA at Columbia, she was Senior Data Engineer at Instagram working on the internal data platform team. We’re excited to have you on the team - it’s going to be a great summer! ☀

Traditionally, data classification methods rely on manual tagging, and, as a result, are tedious and untenable at the scale of modern warehouses. AI-based data classification addresses some of the scale concerns at the cost of being inherently imprecise: due to hallucination, its guesses are often wrong.

SDF Labs  has developed a novel solution to address the above challenges.

They use the familiar techniques from the field of programming languages — static analysis and information flow analysis — in the context of data, queries, and pipelines. The resulting approach achieves precise column-level and table-level data classification automatically for the vast majority of tables, requiring manual tagging or AI-based classification only for root tables — a small percentage of the overall warehouse.

More detail here: https://lnkd.in/edxRPFaz.

I had the privilege of having a demo of SDF Labs yesterday with Elias DeFaria and Grace Hawkins. I can say it looks great! What I liked the most is how they improve the developer experience:

	•	Static analysis of your models → You can detect errors at (very fast) compile time. No need to build your whole pipeline only to have it fail at the last model.
	•	Column-level lineage… game changer!
	•	Metadata propagation, DRY principle at its best.
	•	Possibility to cache a subset of your data for fast development cycles.
	•	And much more...

For all dbt fans out there, check it out! This is a serious contender....

The SDF Labs team is headed to the Snowflake Data Cloud Summit 2024! 

Come and hang with Lukas Schulte and Elias DeFaria in San Francisco as we search for innovations and best practices in data management.

Reach out if you'd like to meet up - we always love talking about:
- Bringing Software Development best practices to Data Engineering
- Analytics Engineering, Static Analysis, and Data Transformation
- Building a SQL Compiler and Engine in Rust

Register here:
https://lnkd.in/gS--cV4

Our team has been growing in the past few weeks - we want to wish a warm welcome Ani Venkateshwaran and Vladimir Gapeyev! 👨‍💻

COM:SDF

SDF

People Not Available