問(wèn)題描述
我正在使用多處理的進(jìn)程和隊(duì)列.我并行啟動(dòng)了幾個(gè)函數(shù),并且大多數(shù)函數(shù)都表現(xiàn)良好:它們完成,它們的輸出進(jìn)入它們的隊(duì)列,它們顯示為 .is_alive() == False.但是由于某種原因,一些函數(shù)沒(méi)有運(yùn)行.它們總是顯示 .is_alive() == True,即使在函數(shù)的最后一行(打印語(yǔ)句說(shuō)完成")完成之后也是如此.無(wú)論我啟動(dòng)了哪些功能,都會(huì)發(fā)生這種情況,即使它只有一個(gè).如果不并行運(yùn)行,則函數(shù)運(yùn)行良好并正常返回.什么種類可能是問(wèn)題?
I am using multiprocessing's Process and Queue. I start several functions in parallel and most behave nicely: they finish, their output goes to their Queue, and they show up as .is_alive() == False. But for some reason a couple of functions are not behaving. They always show .is_alive() == True, even after the last line in the function (a print statement saying "Finished") is complete. This happens regardless of the set of functions I launch, even it there's only one. If not run in parallel, the functions behave fine and return normally. What kind of thing might be the problem?
這是我用來(lái)管理作業(yè)的通用函數(shù).我沒(méi)有展示的只是我傳遞給它的函數(shù).它們很長(zhǎng),經(jīng)常使用 matplotlib,有時(shí)會(huì)啟動(dòng)一些 shell 命令,但我不知道失敗的命令有什么共同點(diǎn).
Here's the generic function I'm using to manage the jobs. All I'm not showing is the functions I'm passing to it. They're long, often use matplotlib, sometimes launch some shell commands, but I cannot figure out what the failing ones have in common.
推薦答案
好吧,好像函數(shù)的輸出太大時(shí),用來(lái)填充Queue的管道被堵塞了(我粗略的理解?這是一個(gè)未解決的/關(guān)閉的錯(cuò)誤?http://bugs.python.org/issue8237).我已經(jīng)修改了我的問(wèn)題中的代碼,以便有一些緩沖(在進(jìn)程運(yùn)行時(shí)定期清空隊(duì)列),這解決了我所有的問(wèn)題.所以現(xiàn)在這需要一組任務(wù)(函數(shù)及其參數(shù)),啟動(dòng)它們,并收集輸出.我希望它看起來(lái)更簡(jiǎn)單/更干凈.
Alright, it seems that the pipe used to fill the Queue gets plugged when the output of a function is too big (my crude understanding? This is an unresolved/closed bug? http://bugs.python.org/issue8237). I have modified the code in my question so that there is some buffering (queues are regularly emptied while processes are running), which solves all my problems. So now this takes a collection of tasks (functions and their arguments), launches them, and collects the outputs. I wish it were simpler /cleaner looking.
編輯(2014 年 9 月;2017 年 11 月更新:重寫以提高可讀性):我正在使用我此后所做的增強(qiáng)來(lái)更新代碼.新代碼(功能相同,但功能更好)在這里:https://gitlab.com/cpbl/cpblUtilities/blob/master/parallel.py
Edit (2014 Sep; update 2017 Nov: rewritten for readability): I'm updating the code with the enhancements I've made since. The new code (same function, but better features) is here: https://gitlab.com/cpbl/cpblUtilities/blob/master/parallel.py
調(diào)用說(shuō)明也在下方.
這篇關(guān)于python多處理:某些函數(shù)完成后不返回(隊(duì)列材料太大)的文章就介紹到這了,希望我們推薦的答案對(duì)大家有所幫助,也希望大家多多支持html5模板網(wǎng)!