Confused about streams vs databases? I break it down in this video! Understanding the difference is crucial for effective software architecture. Check it out: https://www.youtube.com/watch?v=m4S56pL_n2w #softwarearchitecture #streamprocessing #databases
https://www.youtube.com/watch?v=m4S56pL_n2w
#streamprocessing
Just kicked off Module 6 of the #DataEngineering Zoomcamp by @DataTalksClub! ๐ It's all about #StreamProcessing with #Kafka, #Flink, and #PyFlink. Can't wait to get hands-on with real-time data streaming! ๐ฅ๏ธ๐
๐ฅด Behold, a #GitHub page promising to solve "real-world problems" with stream processing demosโbecause nothing screams practicality like endless code snippets that send developers into a #debugging abyss. ๐๐ง Meanwhile, Copilot will 'help' by auto-generating more bugs, proving that #AI is the future... of creating jobs in tech support. ๐ค๐ ๏ธ
https://github.com/risingwavelabs/awesome-stream-processing #StreamProcessing #TechSupport #CodeSnippets #HackerNews #ngated
Demos of Stream Processing Solving Real-World Problems โ https://github.com/risingwavelabs/awesome-stream-processing
#HackerNews #StreamProcessing #RealWorldProblems #Demos #TechInnovation #RisingWave
System design basics: Real-time data processing
realtimedataprocessing #streamprocessing #dataprocessing #systemdesigntips #systemdesign #computerscience ... source
https://quadexcel.com/wp/system-design-basics-real-time-data-processing/
Do you know SQL? Exactly!
Most databases and data processing tools support SQL for exactly that reason. And we see a strong movement for all of them to get closer to the standard, day by day.
In this weeks episode of the Cloud Commute podcast, our host @noctarius2k talks with @gunnarmorling from #Decodable about the benefits of #SQL, how #CDC (change data capture) works and why Decodable uses #ApacheFlink as the underlying technology for its #StreamProcessing offering.
FOMO'd on #KafkaSummit? Want to catch up on what's happening in the data and streaming world?
Here is the ๐ผ March ๐ธ edition of #CheckpointChronicle with tons of great content for you.
Excellent #apachekafka performance analysis blog from Maciej Moลcicki and Piotr Rลผysko.
Nice methodical use of tcpdump, flame charts and more to pinpoint the issue and work out remedial steps.
https://blog.allegro.tech/2024/03/kafka-performance-analysis.html
#streamProcessing #performanceAnalysis #performanceTuning #tcpdump #openSource
๐คซShhh don't tell anyoneโฆ actually tell everyone!
๐ #**ApacheFlink 1.19 RC2 **has just dropped. This is a release candidate, prior to the official release once it's signed off by the committers.
โผ๏ธDownload it! Read the release notes! Try it out (just not in Production)!
๐ Details and links: https://lists.apache.org/thread/x61673q3ro52jvwj137hct2bqowzcy2v
The February edition of **#CheckpointChronicle** is out ๐
It's got tons of content with links out to excellent stuff from folk across the data and streaming landscape
๐ https://dcbl.link/checkpoint-chronicle-6f6a0a
**#dataEngineering** **#streamProcessing** **#apacheFlink** **#apacheKafka** **#RDBMS** **#datadon**
๐ฉโ๐ป Hands-On with Catalogs in Flink SQL
๐ง In this second post in the series, @rmoff shows how to use Flink SQL with catalogs including #apacheHive, #JDBC, & #apacheIceberg. It also includes a closer look at the data structures within the Hive Metastore.
This is a really useful write up of how Grab evolved their #apacheKafka-based ad-attribution reporting from a Kappa to Lambda architecture, including pros and cons of each.
๐ https://engineering.grab.com/attribution-platform
I always had in mind that the Kappa architecture (https://www.oreilly.com/radar/questioning-the-lambda-architecture/) was something that one evolved *to* from Lambdaโand perhaps it's just a terminology differenceโso it's interesting to see it the other way around.
Guten Morgen! Heute starten wir mit der Java User Group Switzerland in unser neues Vortragsjahr! Den Anfang macht Fawaz Ghali mit einem Triple in Zรผrich (16.01.), St. Gallen (17.01.) und Bern (18.01.) mit Vortrรคgen รผber #StreamProcessing.
Ebenfalls am Donnerstag (18.01.) wird uns @maeddes in Luzern etwas รผber #Java Entwicklung mit Cloud-basierten IDEs erzรคhlen.
Bitte meldet euch an, damit wir fรผr ausreichend Snacks und Drinks sorgen kรถnnen: https://www.jug.ch/
Last month @gunnarmorling and I launched #CheckpointChronicle, a roundup of interesting things in the streaming & data space.
The next edition is out soonโstay tuned to the @decodable blog, or read last month's and subscribe to receive it by email at https://www.decodable.co/blog/checkpoint-chronicle-november-2023
(scroll down to the bottom and leave your email address. we'll only use it for sending you the newsletter each month, promise ๐ค)
#streamProcessing #dataEngineering #dataStreaming #openSource
My learning #apacheflink blog series continues, with a poke around options for connecting to Flink with JDBC: https://rmoff.net/2023/11/16/learning-apache-flink-s01e06-the-flink-jdbc-driver/
It might be 45 sleeps until Christmas, but it's only FOUR DAYS until @gunnarmorling and I launch our monthly roundup of interesting things happening in the streaming and data space. Stay tuned to the Decodable blog next week :)
๐ ๐๏ธ https://decodable.co/blog ๐
#openSource #dataEngineering #streamProcessing #databases #changeDataCapture
๐A nice collection of real-world production uses of @ApacheKafka and #ApacheFlink, collected and curated by Thanh Tung Dao ๐:
๐ Flink: https://github.com/dttung2905/flink-at-scale
๐ Kafka: https://github.com/dttung2905/kafka-in-production
๐ #ApacheFlink 1.18 release notes: https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/ with tons of new goodies, including โฆ
๐ Time Travel! ๐คฏhttps://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/time-travel/
๐ JDBC driver for the SQL gateway https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/#introduce-flink-jdbc-driver-for-sql-gateway
๐ Stored Procedure support for catalogs https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/procedures/ (already implemented in #ApachePaimon)
๐ Flink SQL performance improvements https://flink.apache.org/2023/10/24/announcing-the-release-of-apache-flink-1.18/#performance-improvements--tpc-ds-benchmark
๐ SQL Client "Quality of Life" improvements (DX) https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements
#dataEngineering #openSource #streamingSQL #streamProcessing
๐พ I've been having some fun (and not so much fun) poking around with Flink SQL for the first time. Lots to get my head around quite a surface area of APIs and concepts to grok.
๐ฅ๏ธ The client has different ways to display query results, configured by the result-mode option. These include an interactive table, and something much more akin to other SQL CLIs like psql etc.
#opensource #datadon #apacheflink #dataengineering #streamprocessing
Client Info
Server: https://mastodon.social
Version: 2025.04
Repository: https://github.com/cyevgeniy/lmst