問題描述

我正在嘗試模擬使用 twisted 運(yùn)行的應(yīng)用程序網(wǎng)絡(luò).作為我模擬的一部分，我想同步某些事件并能夠?yàn)槊總€(gè)進(jìn)程提供大量數(shù)據(jù).我決定使用多處理事件和隊(duì)列.但是，我的進(jìn)程正在掛起.

I am trying to simulate a network of applications that run using twisted. As part of my simulation I would like to synchronize certain events and be able to feed each process large amounts of data. I decided to use multiprocessing Events and Queues. However, my processes are getting hung.

我編寫了下面的示例代碼來說明問題.具體來說，(大約 95% 的時(shí)間在我的沙橋機(jī)器上)，'run_in_thread' 函數(shù)完成，但是直到我按下 Ctrl-C 后才會(huì)調(diào)用 'print_done' 回調(diào).

I wrote the example code below to illustrate the problem. Specifically, (about 95% of the time on my sandy bridge machine), the 'run_in_thread' function finishes, however the 'print_done' callback is not called until after I press Ctrl-C.

此外，我可以更改示例代碼中的一些內(nèi)容以使這項(xiàng)工作更可靠，例如:減少衍生進(jìn)程的數(shù)量、從 reactor_ready 調(diào)用 self.ready.set 或更改 deferLater 的延遲.

Additionally, I can change several things in the example code to make this work more reliably such as: reducing the number of spawned processes, calling self.ready.set from reactor_ready, or changing the delay of deferLater.

我猜在扭曲反應(yīng)器和阻塞多處理調(diào)用(例如 Queue.get() 或 Event.wait())之間存在競爭條件?

I am guessing there is a race condition somewhere between the twisted reactor and blocking multiprocessing calls such as Queue.get() or Event.wait()?

我遇到的具體問題是什么?我的代碼中是否有我遺漏的錯(cuò)誤?我可以解決這個(gè)問題還是扭曲與多處理事件/隊(duì)列不兼容?

What exactly is the problem I am running into? Is there a bug in my code that I am missing? Can I fix this or is twisted incompatible with multiprocessing events/queues?

其次，像 spawnProcess 或 Ampoule 這樣的東西會(huì)是推薦的替代品嗎?(如 Mix Python Twisted 與多處理? 中的建議)

Secondly, would something like spawnProcess or Ampoule be the recommended alternative? (as suggested in Mix Python Twisted with multiprocessing?)

編輯(按要求):

我嘗試過 glib2reactor selectreactor、pollreactor 和 epollreactor 的所有反應(yīng)器都遇到了問題.epollreactor 似乎給出了最好的結(jié)果，并且對于下面給出的示例似乎工作正常，但在我的應(yīng)用程序中仍然給我同樣(或類似)的問題.我會(huì)繼續(xù)調(diào)查.

I've run into problems with all the reactors I've tried glib2reactor selectreactor, pollreactor, and epollreactor. The epollreactor seems to give the best results and seems to work fine for the example given below but still gives me the same (or a similar) problem in my application. I will continue investigating.

我正在運(yùn)行 Gentoo Linux 內(nèi)核 3.3 和 3.4、python 2.7，并且我嘗試過 Twisted 10.2.0、11.0.0、11.1.0、12.0.0 和 12.1.0.

I'm running Gentoo Linux kernel 3.3 and 3.4, python 2.7, and I've tried Twisted 10.2.0, 11.0.0, 11.1.0, 12.0.0, and 12.1.0.

除了我的沙橋機(jī)器，我在我的雙核 amd 機(jī)器上也看到了同樣的問題.

In addition to my sandy bridge machine, I see the same issue on my dual core amd machine.

#!/usr/bin/python
# -*- coding: utf-8 *-*

from twisted.internet import reactor
from twisted.internet import threads
from twisted.internet import task

from multiprocessing import Process
from multiprocessing import Event

class TestA(Process):
    def __init__(self):
        super(TestA, self).__init__()
        self.ready = Event()
        self.ready.clear()
        self.start()

    def run(self):
        reactor.callWhenRunning(self.reactor_ready)
        reactor.run()

    def reactor_ready(self, *args):
        task.deferLater(reactor, 1, self.node_ready)
        return args

    def node_ready(self, *args):
        print 'node_ready'
        self.ready.set()
        return args

def reactor_running():
    print 'reactor_running'
    df = threads.deferToThread(run_in_thread)
    df.addCallback(print_done)

def run_in_thread():
    print 'run_in_thread'
    for n in processes:
        n.ready.wait()

def print_done(dfResult=None):
    print 'print_done'
    reactor.stop()

if __name__ == '__main__':
    processes = [TestA() for i in range(8)]
    reactor.callWhenRunning(reactor_running)
    reactor.run()

推薦答案

簡短的回答是肯定的，Twisted 和 multiprocessing 彼此不兼容，你不能像你嘗試的那樣可靠地使用它們.

The short answer is yes, Twisted and multiprocessing are not compatible with each other, and you cannot reliably use them as you are attempting to.

在所有 POSIX 平臺(tái)上，子進(jìn)程管理與 SIGCHLD 處理密切相關(guān).POSIX 信號(hào)處理程序是進(jìn)程全局的，每種信號(hào)類型只能有一個(gè).

On all POSIX platforms, child process management is closely tied to SIGCHLD handling. POSIX signal handlers are process-global, and there can be only one per signal type.

Twisted 和 stdlib multiprocessing 不能同時(shí)安裝 SIGCHLD 處理程序.只有其中一個(gè)可以.這意味著其中只有一個(gè)可以可靠地管理子進(jìn)程.您的示例應(yīng)用程序無法控制它們中的哪一個(gè)將贏得該能力，因此我預(yù)計(jì)它的行為會(huì)因該事實(shí)而產(chǎn)生一些不確定性.

Twisted and stdlib multiprocessing cannot both have a SIGCHLD handler installed. Only one of them can. That means only one of them can reliably manage child processes. Your example application doesn't control which of them will win that ability, so I would expect there to be some non-determinism in its behavior arising from that fact.

但是，您的示例更直接的問題是您在父進(jìn)程中加載?? Twisted，然后使用 multiprocessing 派生 而不是 exec 所有子進(jìn)程.Twisted 不支持這樣使用.如果你 fork 然后 exec，沒有問題.但是，缺少新進(jìn)程(可能是使用 Twisted 的 Python 進(jìn)程)的 exec 會(huì)導(dǎo)致 Twisted 無法解釋的各種額外共享狀態(tài).在您的特定情況下，導(dǎo)致此問題的共享狀態(tài)是用于實(shí)現(xiàn) deferToThread 的內(nèi)部waker fd".由于 fd 在父級(jí)和所有子級(jí)之間共享，當(dāng)父級(jí)試圖喚醒主線程以傳遞 deferToThread 調(diào)用的結(jié)果時(shí)，它很可能會(huì)喚醒 其中一個(gè)子級(jí)而是處理.子進(jìn)程沒有什么有用的事情可做，所以這只是浪費(fèi)時(shí)間.同時(shí)，父線程中的主線程永遠(yuǎn)不會(huì)醒來，也不會(huì)注意到您的線程任務(wù)已完成.

However, the more immediate problem with your example is that you load Twisted in the parent process and then use multiprocessing to fork and not exec all of the child processes. Twisted does not support being used like this. If you fork and then exec, there's no problem. However, the lack of an exec of a new process (perhaps a Python process using Twisted) leads to all kinds of extra shared state which Twisted does not account for. In your particular case, the shared state that causes this problem is the internal "waker fd" which is used to implement deferToThread. With the fd shared between the parent and all the children, when the parent tries to wake up the main thread to deliver the result of the deferToThread call, it most likely wakes up one of the child processes instead. The child process has nothing useful to do, so that's just a waste of time. Meanwhile the main thread in the parent never wakes up and never notices your threaded task is done.

您可以通過在創(chuàng)建子進(jìn)程之前不加載任何 Twisted 來避免此問題.就 Twisted 而言，這會(huì)將您的使用轉(zhuǎn)變?yōu)閱芜M(jìn)程用例(在每個(gè)進(jìn)程中，它最初會(huì)被加載，然后該進(jìn)程將不會(huì)繼續(xù)分叉，所以毫無疑問 fork 和 Twisted 是如何交互的).這意味著在創(chuàng)建子進(jìn)程之前甚至不導(dǎo)入 Twisted.

It's possible you can avoid this issue by not loading any of Twisted until you've already created the child processes. This would turn your usage into a single-process use case as far as Twisted is concerned (in each process, it would be initially loaded, and then that process would not go on to fork at all, so there's no question of how fork and Twisted interact anymore). This means not even importing Twisted until after you've created the child processes.

當(dāng)然，這只對 Twisted 有幫助.您使用的任何其他庫都可能遇到類似的問題(您提到了 glib2，這是另一個(gè)庫的一個(gè)很好的例子，如果您嘗試像這樣使用它會(huì)完全窒息).

Of course, this only helps you out as far as Twisted goes. Any other libraries you use could run into similar trouble (you mentioned glib2, that's a great example of another library that will totally choke if you try to use it like this).

我強(qiáng)烈建議不要使用 multiprocessing 模塊.相反，使用任何涉及 fork 和 exec 的多進(jìn)程方法，而不是單獨(dú)使用 fork.安瓿屬于這一類.

I highly recommend not using the multiprocessing module at all. Instead, use any multi-process approach that involves fork and exec, not fork alone. Ampoule falls into that category.

這篇關(guān)于扭曲與多處理事件和隊(duì)列不兼容?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！