Apache Kafka is a powerful tool for processing data streams - but have you ever wondered what's really happening under the hood? 🔍
You can check it with our Kafka Visualization 👉 https://softwaremill.com/kafka-visualisation/
With this tool, you can simulate how data flows through a replicated Kafka topic, to gain a better understanding of the message processing model. Have fun exploring Kafka!
Radio for DuckDB – DuckDB Now Talks to WebSockets and Redis Pub/Sub
https://query.farm/duckdb_extension_radio.html
#HackerNews #DuckDB #WebSockets #Redis #PubSub #DataStreaming #TechNews
#DataStreaming is tough! Despite 10+ years of attempting to simplify it, teams often spend up to 80% of their time wrangling bad data at the lake and optimizing real-time pipelines.
Discover the basic challenges of Data Streaming and a few design & architecture patterns used to tackle these challenges.
This is about pragmatic solutions to build fast, scalable & manageable Data Streaming Pipelines! Watch the #InfoQ video now: https://bit.ly/3Zmjrrd
Our Client needed a robust solution to process vast amounts of data in real-time. We chose Apache Kafka for this project 🚀 See how it helped overcome all the challenges: https://softwaremill.com/case-study/data-streaming-using-kafka-for-a-retail-company/
Kafka rocks, but has the romance faded? 😢 Is produce/consume feeling… stale?
Spice things up with 3 new ways to work with Kafka:
🔥 S3-compatible file system
⚡ JDBC database access
❄️ Apache Iceberg
@streambased makes it easy.
#ApacheKafka #DataStreaming #DataArchitecture
Troubleshooting Debezium Kafka Blob Converter for DataVault Database
Troubleshooting your Debezium Kafka Blob Converter for efficient DataVault DB2 data streaming? This post explores common issues & solutions for smooth pipeline operation. Learn to handle BLOB data effectively! #DataVault #Debezium #Kafka #BlobConverter #DataStreaming #DB2
https://tech-champion.com/database/db2luw/__trashed-22/
Resolve issues with Debezium Kafka Blob Converter when integrating with DataVault databases. Lea...
Amazon debuts data streaming service for advertising metrics optimization: New service enables advertisers to access hourly performance data and campaign changes in near real-time through AWS integration. https://ppc.land/amazon-debuts-data-streaming-service-for-advertising-metrics-optimization/?utm_source=dlvr.it&utm_medium=mastodon #Amazon #DataStreaming #Advertising #Marketing #Metrics
Over the past three years in #devRel, sharing my knowledge around #dataStreaming technologies was enough to make me love everything about my job. But, we all know, it was the supportive #openSource community that truly shaped my experience and made it the very best...
It's lunch time at the datastream summit!
Leave the data, take the cannoli
#datastreaming #apacheflink #apachepulsar
#Berlin and #FlinkForward was amazing!,
But I'm in #SanFrancisco again for the #datastreaming Summit next week so I'll take it!
The 5am #Nespresso is an admission of jetlag, nothing else....
@ververicadata @streamnativeio
It's the formal opening of @FlinkFoward!
If you're here, the opening talks are starting!! #apacheflink #datastreaming #realtime!
The solution to long-running spans in is more creative than “don’t have long-running spans.” Learn how to address this common problem.
https://thenewstack.io/opentelemetry-challenges-handling-long-running-spans/
Developing stateful applications can get pretty chaotic as they move upstream and downstream. 2 open source tools that aim to tame that chaos are gaining popularity.
#Data #DataStreaming #OpenSource
https://thenewstack.io/how-apache-iceberg-and-flink-can-ease-developer-pain/
It was nice to see that ClickHouse has a direct integration with @warpstream , so I took the opportunity to make a video that shows you just how it is done.
Can you consume from a #ApacheKafka topic and write #ApacheParquet easily? It turns out that you can, using the #Bento open-source pipeline tool. I made a short video that shows how to consume from a @warpstream topic and map and write into #Parquet files. I wrap it up by querying with DuckDB and then persisting it as a DuckDB database. It's fun for the whole family.
Is there a simple way to consume an #ApacheKafka compatible @warpstream Topic into #SQLite? While there isn't a direct way to do it, I accomplished it using the open-source #Bento pipeline project (link in the comments). With just a handful of YAML, I mapped out the inbound JSON and wrote it to a table in SQLite. Check out the goodness here:
@warpstream provides powerful and easy-to-use features for managing your #ApacheKafka compatible topics in a WarpStream cluster. I made this 2-minute video to show you how it all works.
I had a great chat with 🐣 Simon Chaffetz at @warpstream about #ApacheKafka compatible BYOC clusters. We did a practical walkthrough using #AWS and #Kubernetes. If you want to learn more about WarpStream and Bring Your Own Compute, then this is the video for you.
If you're going to be in Austin, TX on Sept 19, and want to learn about event data streaming in Rust and WASM in a workshop setting, I have a number of discount seats to share.
DM me if you want to learn more, or are interested in a discount code.
ShadowTraffic is a great tool for creating realistic sample data for just about anything. Recently, I've been using it in my content at @warpstream to test out our software with other software in the ecosystem. It then occurred to me that I should get Michael Drogalis, the creator/founder, and create some content with him to show how it works and generate a couple of topics to illustrate it.
#apachekafka #kafka #dataengineering #datastreaming #streamingdata #testdata
