問題描述
我正在嘗試使用多處理的池來運行一組進程，每個進程都將運行一個 gevent 的 greenlets 池.這樣做的原因是有很多網絡活動，但也有很多 CPU 活動，所以為了最大化我的帶寬和我所有的 CPU 內核，我需要多個進程和 gevent 的異步猴子補丁.我正在使用多處理的管理器創建一個隊列，進程將訪問該隊列以獲取要處理的數據.
I am attempting to use multiprocessing's pool to run a group of processes, each of which will run a gevent pool of greenlets. The reason for this is that there is a lot of network activity, but also a lot of CPU activity, so to maximise my bandwidth and all of my CPU cores, I need multiple processes AND gevent's async monkey patching. I am using multiprocessing's manager to create a queue which the processes will access to get data to process.
這是代碼的簡化片段:
import multiprocessing
from gevent import monkey
monkey.patch_all(thread=False)
manager = multiprocessing.Manager()
q = manager.Queue()
這是它產生的異常:
Traceback (most recent call last):
File "multimonkeytest.py", line 7, in <module>
q = manager.Queue()
File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 667, in temp
token, exp = self._create(typeid, *args, **kwds)
File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 565, in _create
conn = self._Client(self._address, authkey=self._authkey)
File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 175, in Client
answer_challenge(c, authkey)
File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 409, in answer_challenge
message = connection.recv_bytes(256) # reject large message
IOError: [Errno 35] Resource temporarily unavailable
我相信這一定是由于普通 socket 模塊和 gevent 的 socket 模塊的行為之間存在一些差異.
I believe this must be due to some difference between the behaviour of the normal socket module and gevent's socket module.
如果我在子進程中進行monkeypatch，則隊列創建成功，但是當子進程嘗試從隊列中獲取()時，會發生非常相似的異常.由于在子進程中執行大量網絡請求，因此確實需要對套接字進行修補.
If I monkeypatch within the subprocess, The queue is created successfully, but when the subprocess tries to get() from the queue, a very similar exception occurs. The socket does need to be monkeypatched due to doing large numbers of network requests in the subprocesses.
我的 gevent 版本，我認為是最新的:
My version of gevent, which I believe is the latest:
>>> gevent.version_info
(1, 0, 0, 'alpha', 3)
有什么想法嗎?
推薦答案
使用 monkey.patch_all(thread=False, socket=False)
我在類似的情況下遇到了同樣的問題，并在 patch_socket() 函數下的 gevent/monkey.py 中將其追蹤到第 115 行:_socket.socket = socket.socket.將此行注釋掉可以防止損壞.
I have run into the same issue in a similar situation and tracked this down to line 115 in gevent/monkey.py under the patch_socket() function: _socket.socket = socket.socket. Commenting this line out prevents the breakage.
這是 gevent 用它自己的替換 stdlib socket 庫的地方.multiprocessing.connection 非常廣泛地使用 socket 庫，顯然不能容忍這種變化.
This is where gevent replaces the stdlib socket library with its own. multiprocessing.connection uses the socket library quite extensively, and is apparently not tolerant to this change.
具體來說，在您導入的模塊執行 gevent.monkey.patch_all() 調用而不設置 socket=False 的任何情況下，您都會看到這一點.在我的例子中，是 grequests 做到了這一點，我必須重寫套接字模塊的補丁來修復這個錯誤.
Specifically, you will see this in any scenario where a module you import performs a gevent.monkey.patch_all() call without setting socket=False. In my case it was grequests that did this, and I had to override the patching of the socket module to fix this error.
這篇關于Gevent 猴子補丁破壞多處理的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！