#MLOps
Spark on Kubernetes: наш путь к автоматизации через кастомный оператор Airflow
Всем привет! Меня зовут Дмитрий Третьяков, я ML Engineer в компании «Лента». Мы регулярно запускаем PySpark-приложения в Kubernetes-кластере, используя Airflow. Этот процесс важен для нашей ежедневной работы с данными, но в какой-то момент мы столкнулись с тем, что стандартный подход через SparkKubernetesOperator стал сдерживать развитие: не хватало гибкости, возникали сложности в сопровождении и процесс настройки был излишне сложным для разработчиков.
https://habr.com/ru/companies/lentatech/articles/914884/
#airflow #kubernetes #python3 #pyspark #dags #operator #spark #mlops #big_data #orchestrator
The surprising truth: choosing a simpler MLOps platform might be slowing down AI innovation! 🤯
Reddit shows a ClearML preference over Kubeflow – but is ease of use worth sacrificing the potential for truly groundbreaking AI?
What are your thoughts on this complex trade-off?
🧠 5,000 AI agent failures.
🔁 42% now predicted before they happen.
📉 Downtime slashed by 57%.
Discover how agentic AI systems are evolving to forecast their own breakdowns—before executing a single action. A system-aware cognitive loop could be the future of intelligent decision-making.
#AI #AgenticAI #LLM #PredictiveAnalytics #FailureForecasting #CognitiveSystems #AIEngineering #MLOps #AIAgents
https://medium.com/@rogt.x1997/5-000-failures-later-what-we-learned-about-forecasting-ai-agent-crashes-in-real-time-9219a8e60881
🚀 Predict Before You Train!
Low-code LLM workflows just got smarter. Discover how forecasting GPU cost, precision, and token throughput can help you build intelligent pipelines from the start. 💻📊
#LowCode #AIArchitecture #GPUForecasting #MLOps #LLM
🔗
https://medium.com/@rogt.x1997/architecting-the-future-of-ai-training-forecasting-gpu-decisions-with-predictive-intelligence-a554cf6c9f41
Want to learn TPUs? Check this open-source Python simulator - perfect for getting hands-on with ML hardware! #MLOps #Python #OpenSource https://github.com/UCSBarchlab/OpenTPU
🧠⚠️ What if your AI could sense failure before it even runs?
We built a system that predicts agent execution outcomes with 88.6% accuracy using interpretable ML, prompt entropy, and chain depth indicators.
This is not sci-fi—it's agentic foresight in action.
#AI #PredictiveML #GenAI #MLOps #ManusAI #ExplainableAI
https://medium.com/@rogt.x1997/why-40-of-ai-agent-failures-could-be-prevented-inside-the-predictive-engine-of-manus-ai-cf8194fa1f56
Scaling AI takes a team.
Bring your architects, leaders, & engineers to #QConAI (Dec 16–17, NYC) to align on what works now in #GenAI, #MLOps & production AI systems.
📩 Bringing 3+? Email info@qcon.ai for group discounts.
🚀 From hallucinations to hardware overload, GenAI is chaos without structure.
🧠 MLOps isn't optional anymore — it's the OS of modern AI.
💡 Learn how companies like Ocado scale LLMs with tools like MLflow, DVC, and Vertex AI.
#MLOps #GenAI #AIEngineering #LLMOps #AI2025
👉
https://medium.com/@rogt.x1997/mlops-vs-genai-chaos-inside-the-systems-keeping-ai-from-going-rogue-1dacbbb59fbc
🎯 "Prediction isn't the endgame—cognition is."
Explore how a real-world AI stack achieved 91.3% accuracy by listening to data, not just crunching it.
📡 From Kafka to SHAP to CatBoost, this is the blueprint for cognitive infrastructure in 2025.
🔗 https://medium.com/@rogt.x1997/why-the-smartest-ai-doesnt-just-predict-it-listens-a-journey-into-cognitive-infrastructure-f8b5ab0a3c80
#AIArchitecture #CognitiveAI #RealTimeML #MLops #TechStrategy 🤖🔍
https://medium.com/@rogt.x1997/why-the-smartest-ai-doesnt-just-predict-it-listens-a-journey-into-cognitive-infrastructure-f8b5ab0a3c80
The MLOps skillset is evolving! 🤯
Reddit shows a clear trend: DataOps is emerging as a distinct specialization, separate from traditional DevOps.
Is your organization prepared for this shift?
Full blog post: https://dougortiz.blogspot.com/2025/05/the-quiet-revolution-in-mlops-dataops.html
What training programs are you implementing to address this emerging DataOps talent gap?
Surprising MLOps twist! 🤯
Recent trends point to a potential convergence of MLE and MLOps roles.
Could this mean the end of dedicated MLOps roles in smaller teams? 🤔
Full blog post: https://dougortiz.blogspot.com/2025/05/the-blurring-lines-in-ml-engineering.html
What if your CI/CD wasn’t just for deployments… but part of the product?
Argo Workflows turns Kubernetes into a pipeline engine — DAGs, pods, full visibility inside the cluster.
🚕💡 The model is up and running! It predicts ride durations for NY Yellow Taxi trips, and I’m loving the MLOps journey. Now focusing on deploying the model and automating the process. #DataScience #AI #MachineLearning #MLOps #ZoomCamp #DataTalksClub
📊💻 Just completed the linear regression model to predict ride durations based on data from Jan-Feb 2023. Now on to tuning and integrating the model into a Docker container. Next steps ahead! #MachineLearning #DataScience #MLOps #ZoomCamp #DataTalksClub
🗽🚖 Starting with the NY Yellow Taxi dataset from Jan-Feb 2023! Preparing to build a regression model to predict ride durations. Time to dive into the data and start exploring! #MLOps #ZoomCamp #DataTalksClub #MachineLearning
🚀 $3.83 Billion by 2025? That’s the MLOps market powering GenAI at scale.
Discover the 5-stage blueprint keeping LLMs from going rogue – from hallucination drift to real-time prompt correction. 🧠⚙️
#GenAI #MLOps #AIEngineering #ModelOps #LLM #AIStack
https://medium.com/@rogt.x1997/3-83-billion-by-2025-why-mlops-is-becoming-the-backbone-of-scalable-ai-c9571db4e66b
[Перевод] Теневая сторона AutoML: когда no-code инструменты вредят больше, чем помогают
AutoML обещает лёгкий вход в машинное обучение: не нужно разбираться с пайплайнами, настраивать гиперпараметры и писать код — просто загрузи данные и получи модель. Но за удобством скрываются архитектурные риски, которые не всегда видны на первом этапе. От непрозрачных решений до «тихого дрейфа» и невозможности отката — автоматизация может не упростить, а усложнить жизнь, особенно в проде. В этой статье — разбор того, что на самом деле строит AutoML, где он действительно полезен, и почему его нельзя воспринимать как волшебную кнопку.
https://habr.com/ru/companies/otus/articles/911456/
#mlops #data_science #машинное_обучение #Автоматизация_ML #Версионирование_данных
To Docker or not to Docker? Вот в чём JupyterLab
Локальная работа в Jupyter-ноутбуках – неотъемлемая часть исследований и экспериментов нашего ML-отдела. Но из какой среды эти ноутбуки лучше запускать? Мы пользуемся двумя вариантами: запуском из Docker-контейнера и запуском в изолированном локальном poetry-окружении. В статье соберем минимальный сетап для работы с Jupyter-ноутбуками и ссылки на полезные ресурсы для ознакомления.
https://habr.com/ru/articles/911298/
#poetry #docker #dockercompose #mlops #ml #jupyterlab #jupyter #jupyter_notebook #infrastructure #mlops_tools