The Astral team published a guide on moving from a pip and pip‑tools workflow based on requirements files to uv’s project workflow using pyproject.toml and uv.lock.
https://docs.astral.sh/uv/guides/migration/pip-to-project/
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
Discover how CocoIndex transforms data orchestration with a pure Data Flow Programming model — ensuring traceable, immutable, and declarative pipelines for know https://hackernoon.com/redefining-data-operations-with-data-flow-programming-in-cocoindex-u486ao8 #dataengineering
@hynek released another great video on uv, where he explained how he uses the just tool to store commands in a cross‑platform, portable way for everyday tasks like installing or refreshing virtual environments, running tests and code checks and even development tasks like sending requests.
https://www.youtube.com/watch?v=TiBIjouDGuI
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
Ever wonder about the mind behind Pandas & Apache Arrow? 🤔 Ep. 2 of #TheTestSet (Part 1!) unpacks Wes McKinney's journey – including his speedrunning past! What makes good tools good?
🎧 Listen at https://thetestset.co, on Spotify, or Apple Podcasts
#DataStack #DataEngineering #Pandas #OpenSource #PodcastLaunch #Python
#dataengineering If you needed to use a data lake with Redshift, would you use Iceberg, given some native support, over Delta Lake, which is arguably a better format?
Asking for a friend who is me
🔍 Excited about AXLearn for modular ML training, Pinterest's Moka for massive data processing, and PromiseTune for causal configuration tuning! #MachineLearning #DataEngineering
Ah, yes, the riveting saga of cramming "user-defined indexes" into Apache Parquet files. 😴 Because who doesn’t love a story about exploiting footer metadata to do something nobody asked for? Next time, tell us how to alphabetize your sock drawer using ForestDB. 🧦📚
https://datafusion.apache.org/blog/2025/07/14/user-defined-parquet-indexes/ #userdefinedindexes #apacheparquet #footermetadata #techhumor #dataengineering #HackerNews #ngated
Excited to sponsor the inaugural Data Debug SF happy hour! 🍻
Data Debug SF is building a data engineering community for people who get the "why doesn't this data add up" struggle
Tues July 29 • 5:30-7:30pm • details: http://lu.ma/g92ckftj
Data dose of the Day: Day 12
🔹 Tip: Use data profiling tools like Deequ, Great Expectations, or Spark describe() before writing any transformation logic.
🔸 Why?: Avoid incorrect assumptions (nulls, cardinality, outliers). A bad assumption = a bad model or broken report.
This week's pip Trends newsletter is out. Interesting stuff by Vivis Dev, Jacob Padilla, Adrien, dash0 team & Simon Willison are covered this week
https://newsletter.piptrends.com/p/making-a-simple-http-server-with
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
Logs are crucial for understanding what's happening inside your application, especially in production. This article by the Dash0 team is a must-read for mastering logging in Python. It covers structured JSON logs, centralised logging config and enriching logs with contextvars and other modern observability practices.
https://www.dash0.com/guides/logging-in-python
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
In this article, Jacob Padilla walked through building a simple HTTP server from scratch using asyncio Protocols - no external libraries. It's a great way to understand how an HTTP server works under the hood.
https://jacobpadilla.com/articles/asyncio-protocols
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
How do you review data changes in PRs? 📊
A) Auto-diff everything
B) Explore impact then validate
C) Manual spot checks
D) No review
We're seeing a big shift from A → B, see https://datarecce.io/blog/recce-vs-datafold/
But curious what's working or not
AI-Ready Data: как дообучить LLM без боли и с максимальной отдачей
В последние месяцы я всё чаще сталкиваюсь с одним и тем же выводом: внедрение LLM-систем (особенно с использованием RAG-подхода) тормозится не из-за самой модели, а из-за отсутствия качественных данных. Самое дорогое в процессе — это не запуск пайплайна, не подбор архитектуры, а подготовка структурированных, очищенных и корректных данных, пригодных для обучения или дообучения моделей. Всё чаще этот подход называют AI-Ready Data.
https://habr.com/ru/companies/naumen/articles/925520/
##AIReadyData ##LLM ##DataEngineering ##RubyOnRails ##RAG ##LowCode
My weekly newsletter is out! 🚀
This week's agenda:
🔹 Open Source of the Week - The dagster project
🔹 New learning resources - Forecasting with linear regression, multi-model LLM, multiprocessing with Python
🔹 Book of the week - Visualization for Social Data Science by Roger Beecham
📌 Join 29k subscribers and subscribe to get weekly updates 🗞️👇🏼
https://ramikrispin.substack.com/p/the-dagster-project-visualization
#DataScience #DataEngineering #Python #RStats #AI #OpenSource
This week's newsletter will be out tomorrow. Interesting stuff by Tim Kamanin, Iván Palomares Carrascosa, @rgoswami, Eric Chung ( @towardsdatascience ), @koaning & Johnny covered
https://newsletter.piptrends.com/p/how-can-you-structure-your-python
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
In this video, @koaning and Johnny demoed some of the powerful features of Marimo Notebook and walked through the amazing things you can build with it. It’s a great showcase of how Marimo stands out from traditional notebooks.
https://www.youtube.com/watch?v=V77fXADveo0
#python #Programming #PythonProgramming #ai #ml #MachineLearning #SoftwareDevelopment #WebDevelopment #TechNews #OpenSource #DataEngineering
📢 Join our Azure Data Engineer with Data Factory training!
🗓️ 10th July | ⏰ 6:30 PM IST
👨🏫 Led by Mr. Gareth – Get real-world exposure in Azure Pipelines, ETL tools & Synapse
🔗 https://tr.ee/AZD10JU
#AzureTraining #DataEngineering #CloudSkills #NareshIT #MicrosoftAzure
🎧 New episode in the AWS Developers podcast
Join Poonam Pratik for a practical discussion on serverless ETL, AWS Glue, and data pipeline monitoring. Real examples, tested solutions.
