問題描述

我正在使用 multiprocessing，特別是一個 Pool 來分拆幾個線程"來完成我擁有的一堆慢速工作.但是，由于某種原因，我無法讓主線程重新加入，即使所有的孩子似乎都已經死了.

I'm using multiprocessing, and specifically a Pool to spin off a couple of 'threads' to do a bunch of slow jobs that I have. However, for some reason, I can't get the main thread to rejoin, even though all of the children appear to have died.

已解決:看來這個問題的答案是只啟動多個 Process 對象，而不是使用 Pool.目前尚不清楚為什么，但我懷疑剩余的進程是池的管理器，并且當進程完成時它并沒有死亡.如果其他人有這個問題，這就是答案.

Resolved: It appears the answer to this question is to just launch multiple Process objects, rather than using a Pool. It's not abundantly clear why, but I suspect the remaining process is a manager for the pool and it's not dying when the processes finish. If anyone else has this problem, this is the answer.

主線程

pool = Pool(processes=12,initializer=thread_init)
for x in xrange(0,13):
    pool.apply_async(thread_dowork)
pool.close()
sys.stderr.write("Waiting for jobs to terminate
")
pool.join()

xrange(0,13) 比進程數多一，因為我認為我有一個關閉，并且一個進程沒有得到工作，所以是'不會死，我想強迫它去工作.我也試過 12 次.

The xrange(0,13) is one more than the number of processes because I thought I had an off by one, and one process wasn't getting a job, so wasn't dying and I wanted to force it to take a job. I have tried it with 12 as well.

多處理函數

def thread_init():
    global log_out
    log_out = open('pool_%s.log'%os.getpid(),'w')
    sys.stderr = log_out
    sys.stdout = log_out
    log_out.write("Spawned")
    log_out.flush()
    log_out.write(" Complete
")
    log_out.flush()


def thread_dowork():
    log_out.write("Entered function
")
    log_out.flush()
    #Do Work
    log_out.write("Exiting ")
    log_out.flush()
    log_out.close()
    sys.exit(0)

所有 12 個孩子的日志文件的輸出是:

The output of the logfiles for all 12 children is:

Spawned
Complete
Entered function
Exiting

主線程打印等待作業終止"，然后就坐在那里.

The main thread prints 'Waiting for jobs to terminate', and then just sits there.

top 僅顯示腳本的一份副本(我相信是主要的).htop 顯示兩個副本，一個是從頂部開始的副本，另一個是其他副本.根據它的 PID，它也不是孩子.

top shows only one copy of the script (the main one I believe). htop shows two copies, one of which is the one from top, and the other one of which is something else. Based on its PID, it's none of the children either.

有人知道我不知道的事情嗎?

Does anyone know something I don't?

推薦答案

我真的沒有答案，但我閱讀了 Apply_async 的文檔，這似乎與您所說的問題背道而馳......

I don't really have an answer but I read the docs for Apply_async and it seems counter to your stated problem...

回調應該立即完成，否則線程處理結果將被阻止.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

我不熟悉池，但在我看來，您的用例可以通過 本周 Python 模塊

I'm not familiar with the Pool but it seems to me that your use-case could easily be handled by this recipe on Python Module of the Week

這篇關于Python 多處理從不加入的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！