問題描述
我正在嘗試使用多進程池對象.我希望每個進程在啟動時打開一個數(shù)據(jù)庫連接，然后使用該連接來處理傳入的數(shù)據(jù).(而不是為每一位數(shù)據(jù)打開和關閉連接.)這似乎是初始化程序對于，但我無法理解工作人員和初始化程序是如何通信的.所以我有這樣的事情:
I'm trying to use the multiprocess Pool object. I'd like each process to open a database connection when it starts, then use that connection to process the data that is passed in. (Rather than opening and closing the connection for each bit of data.) This seems like what the initializer is for, but I can't wrap my head around how the worker and the initializer communicate. So I have something like this:
def get_cursor():
return psycopg2.connect(...).cursor()
def process_data(data):
# here I'd like to have the cursor so that I can do things with the data
if __name__ == "__main__":
pool = Pool(initializer=get_cursor, initargs=())
pool.map(process_data, get_some_data_iterator())
我如何(或我如何)將光標從 get_cursor() 取回 process_data()?
how do I (or do I) get the cursor back from get_cursor() into the process_data()?
推薦答案
初始化函數(shù)是這樣調用的:
The initialize function is called thus:
def worker(...):
...
if initializer is not None:
initializer(*args)
所以在任何地方都沒有保存返回值.你可能認為這注定了你的命運，但不是！每個工人都在一個單獨的進程中.因此，您可以使用普通的 global 變量.
so there is no return value saved anywhere. You might think this dooms you, but no! Each worker is in a separate process. Thus, you can use an ordinary global variable.
這不是很漂亮，但它確實有效:
This is not exactly pretty, but it works:
cursor = None
def set_global_cursor(...):
global cursor
cursor = ...
現(xiàn)在您可以在 process_data 函數(shù)中使用 cursor.每個獨立進程內的cursor變量與所有其他進程是分開的，所以它們不會互相踩踏.
Now you can just use cursor in your process_data function. The cursor variable inside each separate process is separate from all the other processes, so they do not step on each other.
(我不知道 psycopg2 是否有不同的方法來處理這個問題，首先不涉及使用 multiprocessing；這是一個一般性的答案multiprocessing 模塊的一般問題.)
(I have no idea whether psycopg2 has a different way to deal with this that does not involve using multiprocessing in the first place; this is meant as a general answer to a general problem with the multiprocessing module.)
這篇關于如何使用初始化程序來設置我的多進程池?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！