問題描述
關于使用 Multiprocessing.Pool() 在多個進程之間共享變量的問題.
Have a quick question about a shared variable between multiple processes using Multiprocessing.Pool().
如果我從多個進程中更新全局列表,我會遇到任何問題嗎?IE.如果兩個進程同時嘗試更新列表.
Will I run in to any issues if I am updating a global list from within multiple processes? I.e. if two of the processes were to try to update the list at the same time.
我看過關于在類似事情上使用鎖的文檔,但我想知道是否有必要.
I have seen documentation about using a Lock for similar things but I was wondering if it was necessary.
我共享這個變量的方式是在我的回調函數中使用一個全局變量,'success' 在目標函數完成后,我將所有成功的操作附加到:
The way I am sharing this variable is by using a global variable in my callback function, 'successes' in which i append all of the successful actions to after the target function has completed:
TOTAL_SUCCESSES = []
def func(inputs):
successes = []
for input in inputs:
result = #something with return code
if result == 0:
successes.append(input)
return successes
def callback(successes):
global TOTAL_SUCCESSES
for entry in successes:
TOTAL_SUCCESSES.append(entry)
def main():
pool = mp.Pool()
for entry in myInputs:
pool.apply_async(func, args=(entry,),callback=callback)
為任何語法錯誤道歉,很快就寫出來了,但是程序正在運行,只是想知道如果我有問題我是否添加了共享變量.
Apologize for any syntax errors, wrote this up quickly but the program is working just wondering if I add the shared variable if I will have issues.
提前致謝!
推薦答案
使用您當前的代碼,您實際上并沒有在進程之間共享 CURRENT_SUCCESSES
.callback
在主進程中的結果處理線程中執行.只有一個結果處理線程,因此每個 callback
將一次運行一個,而不是同時運行.所以你寫的代碼是進程/線程安全的.
With your current code, you're not actually sharing CURRENT_SUCCESSES
between processes. callback
is executed in the main process, in a result handling thread. There is only one result handling thread, so each callback
will be run one at a time, not concurrently. So your code as written is process/thread safe.
但是,您忘記從 func
中返回 success
,這是您想要修復的.
However, you are forgetting to return successes
from func
, which you'll want to fix.
此外,使用 map
可以更簡潔地編寫:
Also, this could be much more succinctly written using map
:
def func(inputs):
successes = []
for input in inputs:
result = #something with return code
if result == 0:
successes.append(input)
return successes
def main():
pool = mp.Pool()
total_successes = pool.map(func, myInputs) # Returns a list of lists
# Flatten the list of lists
total_successes = [ent for sublist in total_successes for ent in sublist]
這篇關于Python多處理附加列表的文章就介紹到這了,希望我們推薦的答案對大家有所幫助,也希望大家多多支持html5模板網!