問題描述
我一直在使用 Python 的多處理模塊分析一些代碼('job' 函數(shù)只是將數(shù)字平方).
I've been profiling some code using Python's multiprocessing module (the 'job' function just squares the number).
data = range(100000000)
n=4
time1 = time.time()
processes = multiprocessing.Pool(processes=n)
results_list = processes.map(func=job, iterable=data, chunksize=10000)
processes.close()
time2 = time.time()
print(time2-time1)
print(results_list[0:10])
我發(fā)現(xiàn)奇怪的一件事是，最佳塊大小似乎是 10k 元素左右 - 這在我的計(jì)算機(jī)上花了 16 秒.如果我將塊大小增加到 100k 或 200k，那么它會(huì)減慢到 20 秒.
One thing I found odd is that the optimal chunksize appears to be around 10k elements - this took 16 seconds on my computer. If I increase the chunksize to 100k or 200k, then it slows to 20 seconds.
這種差異可能是由于更長的列表需要更長的酸洗時(shí)間嗎?100 個(gè)元素的塊大小需要 62 秒，我假設(shè)這是由于在不同進(jìn)程之間來回傳遞塊所需的額外時(shí)間.
Could this difference be due to the amount of time required for pickling being longer for longer lists? A chunksize of 100 elements takes 62 seconds which I'm assuming is due to the extra time required to pass the chunks back and forth between different processes.
推薦答案
關(guān)于最優(yōu)chunksize:
About optimal chunksize:
- 擁有大量的小塊將允許 4 個(gè)不同的工作人員更有效地分配負(fù)載，因此更小的塊將是可取的.
- 另一方面，每次必須處理新塊時(shí)，與進(jìn)程相關(guān)的上下文更改都會(huì)增加開銷，因此需要更少的上下文更改，因此需要更少的塊.
由于兩個(gè)規(guī)則都需要不同的方法，所以中間的點(diǎn)是要走的路，類似于供需圖.
As both rules want different aproaches, a point in the middle is the way to go, similar to a supply-demand chart.
這篇關(guān)于Python多處理:為什么大塊更慢?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！