π Big Data Pipeline Cheatsheet for AWS, Azure & GCP π©οΈ
This one visual explains it all: from Ingestion β‘οΈ Data Lake β‘οΈ Computation β‘οΈ Data Warehouse β‘οΈ Presentation.
Perfect for:
π§ Data Engineers
βοΈ Cloud Architects
π€ ML Engineers
π Boost this if you're building in the cloud!
#DataEngineering #BigData #CloudComputing #AWS #Azure #GCP #MachineLearning #ETL #DataPipeline #MastodonTech #OpenSource #Fediverse
#Coroot is excited to partner with #Glassflow to share how you can optimize data streaming, storage, and observability using a fully-#FOSS stack (including tools like GlassFlow, #Kafka, #Clickhouse and Coroot!): https://t.ly/oVAOL
#SRE #DevOps #tech #datapipeline #Python #Apache #linux #observability
#TBT... to an entire week ago at #RSAC where Seth Goldhammer had the chance to demo Graylog's data telemetry pipeline management! π₯οΈ β
Join Seth as he talks about data lakes, data lake previews, getting your data back when you need it, and more.
Wanna learn more about this topic? Here you go: https://graylog.org/post/security-data-lake-strategy/ #RSA #RSAC2025 #datalake #datamanagement #datapipeline
We are Hiring!
Python Data Engineer
π Location: Dubai
π© Submit your CV: resumes@overturerede.zohorecruitmail.in
π Contact: +917428694900
π Apply Now: .
https://zurl.co/IeK73
#overturerede #corporate #trainingsolutions #PythonDataEngineer #DataEngineering #DataScience #PythonProgramming #BigData #MachineLearning #DataPipeline #ETL #SQL #DataAnalytics #DataVisualization #TechCareers #ProgrammingLife
Why Data Engineering Skills Are Crucial for the Success of Data Scientists
#DataEngineering #DataScience #DataAnalytics #BigData #MachineLearning #DataManagement #DataQuality #ETL #DataPipeline #Analytics #DataInfrastructure
In 2018, our SQL queries on a 700k-row dataset were crawling and crashing. Switched to Python with Pandas, processed in chunks, and reduced the time from 5 hours to 40 minutes. Python handled it like a charm! #python #datapipeline
One More feather to the my developer hat !
love for creating high-performance Secured applications is always my quest and this is full stack this time .
Julia with Dash: To Supercharge data analysis with Juliaβs lightning-fast computations, outpacing Python in speed.
Rust Leptos: To broaden Rust in to full stack, a framework create lightning-fast and smooth web applications.
My love for learning by Doing resulted 2 project π, are live π’ for you to check out π.
Sharang (live at : https://sharang.s9lab.dev/ ): A Julia Dash-powered productivity app, Extensible by APIs and data (data pipelines) for Kubernetes admins. Simplify admin and security admin, and best part is AI can be integrated. Check it out and let me know what features you want in this App, I will code it π.
Note: App is in hobby, Ideation and Showcase phase.
What Next on this App?
- I will extend to various widgets with other tools, and more slim π on user PC.
- I am ready to take up any suggestion π‘ .
- More AI to make it more productive less Fancy π
- optimize for mobile view, Clean the code π
Capita7.com (live at: https://capita7.com/): This a beautiful website and outcome of my Rust hobby exploration. Built with Rust, leptos and TailwindCSS.
The Journey has just started many to witness . Please visit https://ls-lrt.com/ for the future updates π
#rust #julia #dashjl #leptos #mistral #ai #generativeai #learning #opensource #coding #container #kubernetes #digitalocean #container #datapipeline #podman #vscode #githubaction #automation
π‘ Clean, curated data in an enterprise data warehouse is essential for successful #GenAI projects. Stay ahead by investing in advanced ETL / #datapipeline tools.
@techreview findings in @BigDATAwireNews: https://loom.ly/kOHAYco
How @fluentbit handles data loss and gives you options for data back pressure in #cloudnative #datapipeline solutions. Level up your #observability today! #chronosphereio https://calyptia.com/blog/avoiding-data-loss-and-backpressure-problems-with-fluent-bit
Scoop alert! Amid an enterprise #observability cost crunch, #Dynatrace expands its data ingest pricing options, third-party data support and introduces a #datapipeline. #o11y #Perform2024
Data patterns for community survey data integrated into a data harmonization workflow description, O'Brien etal 2021 tackles the issue of diverse data harmonization for ecological community surveys. https://www.sciencedirect.com/science/article/pii/S1574954121001655 #DataScience #DataPipeline #DataCuration #SciLit
Update: Since this post is getting attention again, I should probably let everyone know that I got #FediHired by a nice little AI company about two months ago π©π»βπ€ thank you all for helping make that possible, and I'm sorry for blowing up your timelines with this update π€ͺ
Well, the company I work(ed) for just folded. Poof! π©βοΈ.
No notice, no severance, no nothing. π€¬
So I'm asking for boosts and leads. If anyone needs someone who knows data, please reach out.
I'm located in Seattle, WA. Remote preferred.
Last few titles include: CDO, Head of Data, and Sr. Operations Analyst.
I've worked in gaming, fintech, and B2C/C2C marketplaces most recently.
I'm proficient in Python, SQL, statistics, team management, (almost) all things data-related, and a host of other stuff.
I'm opinionated and anti-capitalist, but I also routinely bring in multiple times my department's cost in profits for the companies I work for.
I'm also hella nice (despite my RBF) and easy to get along with.
God, pitching myself is so awkward.
That's the toot. Thanks.
πππ
Edit: I wanted to say thank you to everyone who is sharing this and a huge thank you to everyone who has reached out with encouragement and leads. This community is amazing.
Edit 2: I'm going through all the replies I got, and will reach out to a lot of you soon. I've just finished updating my resume and LinkedIn.
I realize that I'm going to have to give up some of my anonymity when I respond to people, so I'd really appreciate it if anyone I send my resume or LinkedIn profile to could keep my work and personal life separate. Thank you.
πππ
---
**Gif is of parent company explaining our off-boarding process**
#PleaseBoost #Boost #FediHire #LWF #Layoffs #LookingForJob #LookingForWork #Data #DataScience #Python #SQL #Snowflake #ELT #DataPipeline #Analytics #BusinessIntelligence #CDO #HeadOfData #PleaseHelp #Gaming #Fintech #ChiefDataOfficer #HireAlice #ThankYou
Monday blues hit hardest after a wonderful weekend. On the other hand, nothing makes Monday fly faster than a technical go live!
#dataintegration #datapipeline #automation #bi #datastudies
β―οΈ #zen3geo v0.6.1. Quick patch release with full support for π Python 3.11! Made a couple of bug fixes π to DatashaderRasterizer (for turning points/lines/polygons to pixel masks based on a template raster). Also sneaked in a tiny enhancement β¨ to XpySTACAssetReader (which wraps around `xarray.open_dataset`) to work without xpystac π. Find out more in the release notes at https://zen3geo.readthedocs.io/en/v0.6.1/changelog.html.
@pytorch #TorchData #Python #DataShader #xarray #gischat #geospatial #datapipeline #computervision
1/ ποΈ When building a data pipeline, focus on scalability, reliability, and maintainability to ensure long-term success. π
A thread π§΅ππ»
From 0 to a complete #streaming #datapipeline in ~12 minutes with @aiven_io for #ApacheFlink!
Check it out!
Resonance Calendar for Week 50, 2022
There were a few posts I read this week which resulted in my sitting back and saying to myself: βHmmmmβ.
Yes, all the readings are nominally about Data Engineering, but they tend to point out the both technical issues and the pragmatics of Data Engineering, particularly the complexity of constructing a βmodern data stackβ.
#DataEngineering #DataContract #DataPipeline #DataHusbandry
https://medium.com/@rhm2k/resonance-calendar-for-week-50-2022-80c7c5a8c601
I just pocketed: We need new data books - so we started one
I just pocketed: We need new data books - so we started one
