問題描述

我需要用 Python 編寫一個類似代理的程序，工作流程與 Web 代理非常相似.該程序位于客戶端和服務(wù)器之間，接收客戶端向服務(wù)器發(fā)送的請求，處理請求，然后將其發(fā)送到原始服務(wù)器.當(dāng)然使用的協(xié)議是私有協(xié)議，使用TCP.

I need to write a proxy like program in Python, the work flow is very similar to a web proxy. The program sits in between the client and the server, incept requests sent by the client to the server, process the request, then send it to the original server. Of course the protocol used is a private protocol uses TCP.

為了盡量減少工作量，我想使用 Python Twisted 來處理請求接收(部分充當(dāng)服務(wù)器)和重新發(fā)送(部分充當(dāng)客戶端).

To minimize the effort, I want to use Python Twisted to handle the request receiving (the part acts as a server) and resending (the part acts as a client).

為了最大化性能，我想使用 python 多處理(線程有 GIL 限制)將程序分成三個部分(進(jìn)程).第一個進(jìn)程運(yùn)行 Twisted 接收請求，將請求放入隊列，并立即向原始客戶端返回成功.第二個進(jìn)程從隊列中獲取請求，進(jìn)一步處理請求并將其放入另一個隊列.第三個進(jìn)程從第二個隊列中獲取請求并將其發(fā)送到原始服務(wù)器.

To maximum the performance, I want to use python multiprocessing (threading has the GIL limit) to separate the program into three parts (processes). The first process runs Twisted to receive requests, put the request in a queue, and return success immediately to the original client. The second process take request from the queue, process the request further and put it to another queue. The 3rd process take request from the 2nd queue and send it to the original server.

我是 Python Twisted 的新手，我知道它是事件驅(qū)動的，我還聽說最好不要將 Twisted 與線程或多處理混合使用.所以我不知道這種方式是否合適或者僅使用Twisted是否有更優(yōu)雅的方式?

I was a new comer to Python Twisted, I know it is event driven, I also heard it's better to not mix Twisted with threading or multiprocessing. So I don't know whether this way is appropriate or is there a more elegant way by just using Twisted?

推薦答案

Twisted 有自己的事件驅(qū)動運(yùn)行子進(jìn)程的方式(在我的謙虛但正確的觀點中)比 multiprocessing 模塊.核心 API 是 spawnProcess，但是像 ampoule 提供更高級別的包裝器.

Twisted has its own event-driven way of running subprocesses which is (in my humble, but correct, opinion) better than the multiprocessing module. The core API is spawnProcess, but tools like ampoule provide higher-level wrappers over it.

如果您使用 spawnProcess，您將能夠像處理 Twisted 中的任何其他事件一樣處理子流程的輸出；如果你使用multiprocessing，你需要開發(fā)你自己的基于隊列的方法，以某種方式從子進(jìn)程獲取輸出到Twisted mainloop，因為正常的callFromThread 線程可能使用的 API 在另一個進(jìn)程中不起作用.根據(jù)你如何稱呼它，它要么嘗試腌制反應(yīng)器，要么只是在子進(jìn)程中使用不同的非工作反應(yīng)器；無論哪種方式，它都會永遠(yuǎn)失去你的電話.

If you use spawnProcess, you will be able to handle output from subprocesses in the same way you'd handle any other event in Twisted; if you use multiprocessing, you'll need to develop your own queue-based way of getting output from a subprocess into the Twisted mainloop somehow, since the normal callFromThread API that a thread might use won't work from another process. Depending on how you call it, it will either try to pickle the reactor, or just use a different non-working reactor in the subprocess; either way it will lose your call forever.

這篇關(guān)于將 Python Twisted 與多處理混合使用?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！