Почему multiprocessing.Queue() тормозит и как обойти это с помощью shared_memory

Привет, Хабр! Вы запускаете многопроцессную задачу, кидаете данные в multiprocessing.Queue() , а потом вдруг замечаете... что всё тормозит. Муторно. Медленно. Местами прям отвратительно. Вы смотрите в монитор, на top, на htop, на код — и не понимаете: ну ведь должно же летать! А не летит.

https://habr.com/ru/companies/otus/articles/913200/

#python #multiprocessing #очередь #производительность #shared_memory #межпроцессное_взаимодействие #кольцевой_буфер

https://kitfucoda.medium.com/telegram-chatbots-evolution-decoupling-parallel-python-s-shared-state-for-clarity-e76880ce9b1f

My recent refactoring journey began with a take-home assignment's hidden pitfall: a hard-coded queue in settings.py. This seemingly small detail quickly grew into a global state headache, making testing and modularity a real challenge.

To tackle this, I focused on explicitly managing inter-process communication. The solution involved leveraging multiprocessing.Manager to centralize shared synchronization primitives, allowing for much cleaner dependency injection across parallel processes.

This refactoring delivered significant benefits. It drastically improved testability by eliminating the need for global patching. It also brought consistency and enhanced modularity, laying a robust foundation for future scalability.

It's been a valuable lesson, directly applicable to my own operational chatbot. While not a performance optimization, the structural clarity gained is immensely rewarding. I'm looking forward to diving deeper into the testing aspects of this setup soon!

#Python #Refactoring #Multiprocessing #CleanCode #SoftwareArchitecture #getfedihired #fedihire #OpenToWork

I've been working on a #space #visualization tool for our operators. It basically needs to always know, and be ready to plot, where every single one of 60k+ objects is down to millidegree/meter/second resolution just in case the sensor suddenly slews there

My own constraint is that it has to be 1) a single 2) #python executable because otherwise I'm not interested

Earlier this year, I found a great 30x faster technique for determining which #satellites are above the horizon. (In fact, it's far more general than that, but that's all the help it gives me to this problem.)

I also realized I could spawn a #multiprocessing child to do lookahead on data and then pass a huge #numpy array to my graphing process. (Investigated ~9 different ways, chose the best)

But there I was stuck.

At any given moment, there are ~4500 space objects above the horizon (at our latitude). Putting 4500 points with little persistence trails and labels and then updating all that at 1Hz let alone the 10Hz I'd like was taking too long, even using the amazing #pyqtgraph

So there I was stuck. Until this week.

https://kitfucoda.medium.com/concurrency-vs-parallelism-achieving-scalability-with-processpoolexecutor-c366bfcc6207

Concurrency and parallelism are often confused in async programming discussions. Go's goroutines highlighted the difference: concurrency is doing many things at once, while parallelism is doing many things at the same time.

AsyncIO handles concurrency well for I/O, but CPU-bound tasks need parallelism. Python uses AsyncIO for concurrency, and ProcessPoolExecutor for parallelism, distributing work across CPU cores.

Process communication is harder than thread communication. AsyncIO's task cancellation differs from ProcessPoolExecutor's, requiring workarounds like event objects for reliable cancellation and shutdown.

Essentially, ProcessPoolExecutor enables parallelism for CPU-bound tasks, scaling them across multiple cores, while AsyncIO handles I/O concurrently.

#python #asyncio #concurrency #parallelism #multiprocessing #opentowork #getfedihired #fedihire #opentowork

#question for #Python developers who do #multiprocessing

- What tips / tricks do you have for minimising the pickling / serialisation overhead?

- Are there any tools for #profiling what is being pickled?

- Bonus: what's your favourite Python profiler and why?

Параллельные вычисления, конкурентность и асинхронное программирование в Python: обзор для начинающих

Однопоточные приложения на Python ограничены в производительности: они выполняют задачи последовательно и не используют преимущества многоядерных процессоров. Кроме того, такие программы не справляются с обработкой множества операций одновременно, особенно если речь идет о задачах, связанных с вводом-выводом, например сетевыми запросами или чтением файлов. Производительность можно значительно улучшить, внедрив в код параллельные вычисления, конкурентность или асинхронное программирование. Для этого Python предлагает такие инструменты, как multiprocessing, threading и asyncio.

https://habr.com/ru/companies/skillfactory/articles/896160/

#python #multiprocessing #asyncio #threading

История эволюции веб-сервиса: от примера из доки до космолета

5k RPS, 5ms Latency и 100 экспериментов одновременно. История о том, как наша команда перестраивала веб-сервис для сплитования трафика в высокопроизводительную систему. С какими ограничениями Cpython и Gil столкнулись на пути, как обходили "узкие места" и оптимизировали сервис до микросекунд. В общем, всё о том, как мы построили "космолет" на Python и взлетели! Ну и, конечно же, ответ на вопрос: "Почему не Go? ".

https://habr.com/ru/companies/okko/articles/894462/

#python #multiprocessing #performance #okko #ab_testing

The LOCK# signal is like “stop trying to make fetch happen.”

#programming #multiprocessing

Finally done with day 6 of #adventofcode2024
Here is my solution in #python .
I'm using Python's multiprocessing module for part 2, which then takes 2-3 seconds on my 8-core CPU, whereas it takes about 4 times as long if i compute it only with a single process.
So, this exercise helped me learning a bit about #multiprocessing in Python.

https://github.com/BugDevelopments/AdventOfCode/blob/main/2024/python/day6.py

#adventofcode

Как мы используем разделяемую память в Aqueduct

Привет. Меня зовут Денис Лисовик, я Backend-инженер в команде Data Science SWAT Авито. В этой статье рассказываю, как использовать разделяемую память в Aqueduct. Вместе мы шаг за шагом пройдем от сервиса, который едва держит один RPS, до сервиса, который может держать сотни запросов в секунду. В процессе вы узнаете, как использовать разделяемую память и как сделать так, чтобы она не утекала, а приложение не падало с Segmentation fault.

https://habr.com/ru/companies/avito/articles/866338/

#python #shared_memory #multiprocessing

@kentpitman @sigrid @mdhughes @awkravchuk

Featuring #unix_surrealism ! @prahou

@pkw ncurses #lisp #asdf #multiprocessing https://codeberg.org/pkw/open-borders

telnet lambda.moo.mud.org 8888
co guest
@join screwtape

#lambdaMOO !

I think @me bumped eir head on the resource limit in-MOO finally? Meet in paradise sushi!

You know it's time to stop when you have to implement __setstate__ and __getstate__ methods to exclude the boto3 s3 client from being serialized/deserialized by Pickle in multiprocessing.

#python #s3 #multiprocessing #pickle

I used to love #Python, but dealing with the honestly kind of scary #multiprocessing library was a reminder of the #GIL #threading situation really hurts working with #parallelism. 😰🫠

I've started learning #Kotlin after honestly really enjoying the threading library it provides. It's so's easy to work with when you understand it! 😀 (I would prefer to stay away from some of the #Java conventions after working with them for so long. 👍) #JVM #programming
♲ f.kawa-kun.com/display/881761a…

Never, ever, use the fork context in multiprocessing. :blobfoxsweating: POSIX OSes (aside from macOS) have that as the default. :blobfoxangrylaugh: Always use the spawn context! :blobfoxdead:

Of course, unless you have an extreme edge case. :blobfoxgoogly: (Yes, the names are funny. :blobfoxgooglytrash: )

#Python #multiprocessing

Have you ever programmed a human computer? Having 30 people walking around the room to exchange information between RAM addresses and CPU registers, and human CPUs execute operations on the clock is a very special experience*.

This week I learned more than in a ~year of self-study, thanks to the 16th Advanced Scientific Programming in #Python https://aspp.school
We covered version control, packaging, testing, debugging, computer architecture, some #numpy and #pandas -fu, programming patterns aka what goes into a class and what doesn't, big-O to understand the scaling of various operations and how to find the fastest one for the given data type and size, and an intro to #multithreading and #multiprocessing 🍭

A personal highlight for me was pair programming. I never thought writing code in with a buddy would be so much fun, but I learned a lot from my buddies and now I don't want to go back to writing code alone 😅

Very indebted to the teachers and organizers; https://aspp.school/wiki/faculty if you ever meet one of those people, please buy them a drink for what they have done for a better code karma state in the universe

*our human computer didn't manage to execute the simplest sorting algorithm and the CPUs started to sweat; we experienced what happens when the code is ambiguous and imprecise 😱🫨

Just added support for multiprocessing. Using six std::thread for the whole image results into a massive speedup (no benchmark yet). Mostly high-iteration consuming locations are much faster now.

The following image is a typical 'dense' location:

Coordinates:
x: -0.47495141886912982127993494293396986175716518

y: -0.53679576259407916556417805843833696556268279

size: 1.137e-13

#fractal #mandelbrot #mandelbrotset #fractalzoom #magnification #multiprocessing

Ускорение Python в 2 раза с помощью multiprocessing, async и MapReduce

Python действительно может считаться относительно медленным языком программирования по сравнению с некоторыми другими языками, такими как C++ или Java. Однако, существуют различные библиотеки и инструменты, которые позволяют ускорить выполнение счетных задач в Python. Рассмотрим как можно ускорить анализ данных в 2 раза!

https://habr.com/ru/articles/825206/

#python3 #python #asyncio #async/await #multiprocessing #mapreduce

Article: 10 times faster, running cases in parallel

In this article, we explore running optimization model cases in parallel. Specifically, we use the Python multiprocessing and mpi4py libraries to fully use the many CPU cores/threads in modern computers.

Our goals are to:
- Illustrate how to apply the multiprocessing and mpi4py libraries to running optimization model cases in parallel.
- Measure the performance of running cases in parallel compared with serially.
- Compare the performance of an old 4 core / 4 thread CPU with a new 20 core / 28 thread CPU, using the HiGHS solver.

https://www.solvermax.com/blog/10-times-faster-running-cases-in-parallel
#Python #pyomo #orms #optimization #modelling #HiGHS #multiprocessing #mpi4py

Разница между pool.map и pool.map_async в Python

Еще одна статья-шпаргалка о модуле multiprocessing в Python, без воды, от новичка для новичков многопроцессорного программирования. pool.map и pool.map_async являются методами модуля multiprocessing.Pool в Python, которые позволяют параллельно выполнять функции на нескольких процессах.

https://habr.com/ru/articles/804917/

#multiprocessing #многопроцессорность

Модули multiprocessing и threading в Python

Данная статья написана новичком для новичков, т.е. для тех, кто только начинает изучать возможности многопроцессорного и многопоточного программирования в Python. Статья намеренно пишется без воды и со скомканной теорией, в стиле шпаргалки.

https://habr.com/ru/articles/804799/

#multiprocessing #multithreading