問題描述
我正在嘗試在多個文件上并行運行一些 python 代碼.構造基本上是:
I'm trying to run some python code on several files in parallel. The construct is basically:
def process_file(filename, foo, bar, baz=biz):
# do stuff that may fail and cause exception
if __name__ == '__main__':
# setup code setting parameters foo, bar, and biz
psize = multiprocessing.cpu_count()*2
pool = multiprocessing.Pool(processes=psize)
map(lambda x: pool.apply_async(process_file, (x, foo, bar), dict(baz=biz)), sys.argv[1:])
pool.close()
pool.join()
我以前使用 pool.map 來做類似的事情并且效果很好，但我似乎不能在這里使用它，因為 pool.map 不允許(似乎)允許我傳遞額外的參數(并且使用 lambda 來做這件事是行不通的，因為 lambda 不能被編組).
I've previously used pool.map to do something similar and it worked great, but I can't seem to use that here because pool.map doesn't (appear to) allow me to pass in extra arguments (and using lambda to do it won't work because lambda can't be marshalled).
所以現在我正在嘗試直接使用 apply_async() 來讓事情正常工作.我的問題是代碼似乎掛起并且永遠不會退出.一些文件因異常而失敗，但我不明白為什么會導致連接失敗/掛起?有趣的是，如果沒有一個文件因異常而失敗，它確實會干凈地退出.
So now I'm trying to get things to work using apply_async() directly. My issue is that the code seems to hang and never exit. A few of the files fail with an exception, but i don't see why what would cause join to fail/hang? Interestingly if none of the files fail with an exception, it does exit cleanly.
我錯過了什么?
當函數(以及工作程序)失敗時，我看到了這個異常:
When the function (and thus a worker) fails, I see this exception:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 505, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 376, in _handle_results
task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
如果我看到其中之一，進程父進程將永遠掛起，永遠不會收獲子進程并退出.
If i see even one of these, the process parent process hangs forever, never reaping the children and exiting.
推薦答案
很抱歉回答我自己的問題，但我至少找到了一種解決方法，所以如果其他人有類似的問題，我想在這里發布.我會接受任何更好的答案.
Sorry to answer my own question, but I've found at least a workaround so in case anyone else has a similar issue I want to post it here. I'll accept any better answers out there.
我認為問題的根源是 http://bugs.python.org/issue9400 .這告訴我兩件事:
I believe the root of the issue is http://bugs.python.org/issue9400 . This tells me two things:
- 我沒瘋，我正在努力做的事情真的應該奏效
- 至少在 python2 中，如果不是不可能的話，將異常"返回到父進程是非常困難的.簡單的方法有效，但許多其他方法無效.
就我而言，我的工作函數正在啟動一個存在段錯誤的子進程.這返回了 CalledProcessError 異常，這是不可腌制的.出于某種原因，這使得父對象中的池對象出去吃午飯，而不是從對 join() 的調用中返回.
In my case, my worker function was launching a subprocess that was segfaulting. This returned CalledProcessError exception, which is not pickleable. For some reason, this makes the pool object in the parent go out to lunch and not return from the call to join().
在我的特殊情況下，我不在乎異常是什么.最多我想記錄它并繼續前進.為此，我只需將我的頂級工作函數包裝在 try/except 子句中.如果工作進程拋出任何異常，它會在嘗試返回父進程之前被捕獲，記錄，然后工作進程正常退出，因為它不再嘗試發送異常.見下文:
In my particular case, I don't care what the exception was. At most I want to log it and keep going. To do this, I simply wrap my top worker function in a try/except clause. If the worker throws any exception, it is caught before trying to return to the parent process, logged, and then the worker process exits normally since it's no longer trying to send the exception through. See below:
def process_file_wrapped(filenamen, foo, bar, baz=biz):
try:
process_file(filename, foo, bar, baz=biz)
except:
print('%s: %s' % (filename, traceback.format_exc()))
然后，我有我的初始映射函數調用 process_file_wrapped() 而不是原來的.現在我的代碼按預期工作.
Then, I have my initial map function call process_file_wrapped() instead of the original one. Now my code works as intended.
這篇關于Python多處理池在加入時掛起?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！