Streamkap provides a streaming ETL (Extract, Transform, Load) platform built on Apache Kafka and Flink. The platform is designed for easy use and requires no maintenance.

Streamkap

Replace batch ETL with streaming for lower latency and cost. 

Begin Capital

Haatch

I'm excited to announce we now have DynamoDB CDC support live on Streamkap! 

This has been an urgent request from folks migrating from Rockset (acquired by OpenAI) so we're excited to help migrate to other real-time databases. 




#rockset #clickhouse #singlestore #startree #starrocks #realtimeanalytics #realtimedata #streamingCDC #dynamodb #cdc #changedatacapture #streamingETL

The data community can be a beautiful thing. 

I'm excited to announce that Joe Gannon is joining Streamkap to lead our sales efforts (Joe is an avid lacrosse player and coach and his dog Callie (pictured) is an occasional bonus on zoom calls 🐶 :)) 

It all started a few weeks back when Joe Reis 🤓 had Nicholas Freund on his podcast. 

Nick shared the challenges of startup life when it doesn't all come together, it was real, it was raw, and a lot of folks found it really valuable.  

Nick also shared that he had a great team now looking for roles. I had the pleasure of meeting with some of those folks as we're starting to build out GTM here at Streamkap. A few phone calls and a quick trip to Denver and Joe is now a part of the Streamkap team. 

Joe G, welcome aboard, we're excited to have you and to see you already making an impact. 

Nick, thank you for your transparency in the most challenging of times and for looking out for your employees. 

Joe Reis, thanks for continuing to have real and impactful conversations with folks in the data community.

Join us for a live demo of Streamkap. We'll show in real-time how fast you can stream data to Snowflake with Snowpipe Streaming.

In the wake of OpenAI's acquisition of Rockset, which BTW is a a great outcome for the Rockset team and a bullish signal for the importance of real-time data for AI in the enterprise, we're finding out that Rockset was pretty darn unique. 

Rockset's customers have been scrambling to find alternative solutions before the service is sunsetted (for month-to-month customers) on Sept 30. 

The challenges we've seen folks face fall into 2 buckets, solving for CDC/ETL from NoSQL DBs and solving for some of the unique workloads that Rockset could support, we're mostly seeing ClickHouse as the choice to replace but the overlap is not 100%. I'll address that in another post. 

Rockset was very good at ingesting from NoSQL databases directly and had a lot of customers using them for MongoDB and DynamoDB data. We're helping folks with MongoDB data already and while there are complications with nested data for many Rockset alternatives, we've already solved this for our ClickHouse destination so it's not been too tricky. 

CDC DynamoDB is a bit of a different animal, it's not as broadly adopted given it's specific to AWS so there just isn't as much support in the open-source world. We're working on a solution for this and will have one soon but would love to talk to folks about what they are doing. 

There are 2 basic approaches for streaming updates: 

DynamoDB Streams - The preferred approach, offers exactly once delivery but is constrained by a 24-hour limit on retention of change events which means careful management of consumption to ensure you're not losing the end of the log and having to re-snapshot/backfill data again 

Kinesis Data Streams - Fully managed and serverless option if your destination supports Kinesis, if not you will have to connect with Kafka or other solution and deal with additional complexity of multiple streaming platforms. Kinesis data streams only supports at-least-once delivery so you will have to manage duplication. In addition to paying for Kinesis you will have additional fees for DynamoDB CDC with Kinesis. My sense is that this will be limited to businesses that are planning on just using AWS services but I'm interested to hear what folks are doing. 

Separate from streaming data, you also have to solve for backfilling historical data if you're setting up a new pipeline. For this, you'll need to either use the DynamoDB API to query the DB (best for tables under 5GB) or use the bulk export to S3. 

If you're working on solving these problems drop a comment or DM, would love to chat. 


#cdc #changedatacapture #rockset #dynamodb #mongodb #openai #streamingdata #kafka #kinesis #realtimeanalytics #realtimedata #clickhouse #olap #nosql #realtimedatabase #realtimedb

Join us for a webinar on Tuesday, June 25th at 12 pm ET, where we’ll explore how Snowpipe Streaming, Dynamic Tables, and Streamkap combine to simplify data engineering by low-latency AND lower cost data transformation pipelines in Snowflake.

Are Snowflake  Dynamic tables, a better way to model? 

If you're running incremental models on batch ETL pipelines you could face:

❌ Higher costs
❌ Stale data
❌ High latency

Join us for our upcoming webinar tomorrow, June 25th at 12 ET, where Marcin Migda, Staff Data Engineer at leading POS company SpotOn, and Streamkap CTO Ricky Thomas will dive deep into the benefits of using Dynamic Tables — the new way to model your data on Snowflake.

Register for “Dynamic Tables and Snowpipe Streaming in Snowflake: How Streamkap Helps Unlock Faster & Cheaper Data Pipelines”: [insert link]

We’ll cover: 

⏩ When to use Dynamic Tables for the fastest and most efficient way to model data streamed into Snowflake

🔄 How to refresh data and track dependencies without writing code

🗓️ How to ensure your data is always fresh without the need for fine-grained refresh schedule control

Don't miss this opportunity to learn how Dynamic Tables can simplify your data modeling process and unlock the full potential of your Snowflake data.

Can’t wait for you to join us!

COM:STREAMKAP

Streamkap