問題描述

我有這樣一個簡單的任務:

I have a simple task like that:

def worker(queue):
    while True:
        try:
            _ = queue.get_nowait()
        except Queue.Empty:
            break

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    # queue = multiprocessing.Queue()
    queue = manager.Queue()

    for i in range(5):
        queue.put(i)

    processes = []

    for i in range(2):
        proc = multiprocessing.Process(target=worker, args=(queue,))
        processes.append(proc)
        proc.start()

    for proc in processes:
        proc.join()

似乎 multiprocessing.Queue 可以完成我需要的所有工作，但另一方面，我看到許多 manager().Queue() 的示例并且無法理解我真正需要什么.看起來 Manager().Queue() 使用某種代理對象，但我不明白這些目的，因為 multiprocessing.Queue() 在沒有任何代理對象的情況下做同樣的工作.

It seems that multiprocessing.Queue can do all work that i needed, but on the other hand I see many examples of manager().Queue() and can't understand what I really need. Looks like Manager().Queue() use some sort of proxy objects, but I doesn't understand those purpose, because multiprocessing.Queue() do the same work without any proxy objects.

所以，我的問題是:

1) multiprocessing.Queue 和 multiprocessing.manager().Queue() 返回的對象有什么區別?

1) What really difference between multiprocessing.Queue and object returned by multiprocessing.manager().Queue()?

2) 我需要使用什么?

2) What do I need to use?

推薦答案

雖然我對這個主題的理解有限，但從我所做的我可以看出 multiprocessing.Queue() 和 multiprocessing.Manager() 之間的一個主要區別.Queue():

Though my understanding is limited about this subject, from what I did I can tell there is one main difference between multiprocessing.Queue() and multiprocessing.Manager().Queue():

• multiprocessing.Queue() 是一個對象，而 multiprocessing.Manager().Queue() 是一個地址(代理)，指向由 multiprocessing.Manager() 對象管理的共享隊列.
• 因此您不能將普通的 multiprocessing.Queue() 對象傳遞給 Pool 方法，因為它不能被腌制.
• 此外，python 文檔告訴我們在使用多處理時要特別注意.Queue() 因為它可能會產生不良影響

• multiprocessing.Queue() is an object whereas multiprocessing.Manager().Queue() is an address (proxy) pointing to shared queue managed by the multiprocessing.Manager() object.
• therefore you can't pass normal multiprocessing.Queue() objects to Pool methods, because it can't be pickled.
• Moreover the python doc tells us to pay particular attention when using multiprocessing.Queue() because it can have undesired effects

注意當一個對象被放入隊列時，該對象被腌制并且后臺線程稍后將腌制數據刷新到底層管道.這會產生一些令人驚訝的后果，但不會造成任何實際困難——如果它們真的困擾您，那么您可以改用由經理創建的隊列.將對象放入空隊列后，隊列的 empty() 方法返回 False 和 get_nowait() 可以在不引發 Queue.Empty 的情況下返回之前可能會有一個無限小的延遲.如果多個進程正在對對象進行排隊，則對象可能會在另一端亂序接收.但是，由同一進程排隊的對象將始終按預期順序排列.

Note When an object is put on a queue, the object is pickled and a background thread later flushes the pickled data to an underlying pipe. This has some consequences which are a little surprising, but should not cause any practical difficulties – if they really bother you then you can instead use a queue created with a manager. After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False and get_nowait() can return without raising Queue.Empty. If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order. However, objects enqueued by the same process will always be in the expected order with respect to each other.

警告 如上所述，如果子進程已將項目放入隊列(并且它沒有使用 JoinableQueue.cancel_join_thread)，那么該進程將不會終止，直到所有緩沖的項目都被刷新到管道.這意味著如果您嘗試加入該進程，您可能會遇到死鎖，除非您確定已放入隊列的所有項目都已被消耗.類似地，如果子進程是非守護進程，則父進程在嘗試加入其所有非守護子進程時可能會掛起退出.請注意，使用管理器創建的隊列不存在此問題.

Warning As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe. This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children. Note that a queue created using a manager does not have this issue.

通過將隊列設置為全局變量并在初始化時為所有進程設置它，可以將 multiprocessing.Queue() 與 Pool 一起使用:

There is a workaround to use multiprocessing.Queue() with Pool by setting the queue as a global variable and setting it for all processes at initialization :

queue = multiprocessing.Queue()
def initialize_shared(q):
    global queue
    queue=q

pool= Pool(nb_process,initializer=initialize_shared, initargs(queue,))

將創建具有正確共享隊列的池進程，但我們可以爭辯說 multiprocessing.Queue() 對象不是為此用途而創建的.

will create pool processes with correctly shared queues but we can argue that the multiprocessing.Queue() objects were not created for this use.

另一方面，manager.Queue() 可以在池子進程之間共享，方法是將它作為函數的普通參數傳遞.

On the other hand the manager.Queue() can be shared between pool subprocesses by passing it as normal argument of a function.

在我看來，使用 multiprocessing.Manager().Queue() 在任何情況下都很好，而且麻煩更少.使用經理可能會有一些缺點，但我不知道.

In my opinion, using multiprocessing.Manager().Queue() is fine in every case and less troublesome. There might be some drawbacks using a manager but I'm not aware of it.

這篇關于Python multiprocessing.Queue 與 multiprocessing.manager().Queue()的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！