問題描述

我正在尋找一個 python 包，它不僅可以在單臺計算機內的不同內核上進行多處理，而且還可以在分布在多臺機器上的集群中進行多處理.有很多不同的用于分布式計算的 Python 包，但大多數似乎都需要更改代碼才能運行(例如，表示對象位于遠程計算機上的前綴).具體來說，我想要盡可能接近多處理 pool.map 函數的東西.因此，例如，如果在一臺機器上，腳本是:

I am looking for a python package that can do multiprocessing not just across different cores within a single computer, but also with a cluster distributed across multiple machines. There are a lot of different python packages for distributed computing, but most seem to require a change in code to run (for example a prefix indicating that the object is on a remote machine). Specifically, I would like something as close as possible to the multiprocessing pool.map function. So, for example, if on a single machine the script is:

from multiprocessing import Pool
pool = Pool(processes = 8)
resultlist = pool.map(function, arglist)

那么分布式集群的偽代碼將是:

Then the pseudocode for a distributed cluster would be:

from distprocess import Connect, Pool, Cluster

pool1 = Pool(processes = 8)
c = Connect(ipaddress)
pool2 = c.Pool(processes = 4)
cluster = Cluster([pool1, pool2])
resultlist = cluster.map(function, arglist)

推薦答案

如果你想要一個非常簡單的解決方案，沒有.

If you want a very easy solution, there isn't one.

但是，有一個解決方案具有 multiprocessing 接口 -- pathos -- 它能夠通過并行映射建立與遠程服務器的連接，并做多處理.

However, there is a solution that has the multiprocessing interface -- pathos -- which has the ability to establish connections to remote servers through a parallel map, and to do multiprocessing.

如果您想建立 ssh 隧道連接，您可以這樣做……或者如果您可以使用不太安全的方法，您也可以這樣做.

If you want to have a ssh-tunneled connection, you can do that… or if you are ok with a less secure method, you can do that too.

>>> # establish a ssh tunnel
>>> from pathos.core import connect
>>> tunnel = connect('remote.computer.com', port=1234)
>>> tunnel       
Tunnel('-q -N -L55774:remote.computer.com:1234 remote.computer.com')
>>> tunnel._lport
55774
>>> tunnel._rport
1234
>>> 
>>> # define some function to run in parallel
>>> def sleepy_squared(x):
...   from time import sleep
...   sleep(1.0)
...   return x**2
... 
>>> # build a pool of servers and execute the parallel map
>>> from pathos.pp import ParallelPythonPool as Pool
>>> p = Pool(8, servers=('localhost:55774',))
>>> p.servers
('localhost:55774',)
>>> y = p.map(sleepy_squared, x)
>>> y
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

或者，您可以配置為直接連接(無 ssh)

Or, instead you could configure for a direct connection (no ssh)

>>> p = Pool(8, servers=('remote.computer.com:5678',))
# use an asynchronous parallel map
>>> res = p.amap(sleepy_squared, x)
>>> res.get()
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

這有點挑剔，要讓遠程服務器工作，你必須事先在指定端口啟動一個運行在 remote.computer.com 上的服務器——你必須確保本地主機和遠程主機上的設置都將允許直接連接或 ssh 隧道連接.另外，您需要在每個主機上運行相同版本的 pathos 和 pp 的 pathos 分支.此外，對于 ssh，您需要運行 ssh-agent 以允許使用 ssh 進行無密碼登錄.

It's all a bit finicky, for the remote server to work, you have to start a server running on remote.computer.com at the specified port beforehand -- and you have to make sure that both the settings on your localhost and the remote host are going to allow either the direct connection or the ssh-tunneled connection. Plus, you need to have the same version of pathos and of the pathos fork of pp running on each host. Also, for ssh, you need to have ssh-agent running to allow password-less login with ssh.

但是，希望一切正常……如果您的功能代碼可以使用 dill.source.importable 傳輸到遠程主機.

But then, hopefully it all works… if your function code can be transported over to the remote host with dill.source.importable.

僅供參考，pathos 早就應該發布了，基本上，在新的穩定版本被刪除之前，有一些錯誤和界面更改需要解決.

FYI, pathos is long overdue a release, and basically, there are a few bugs and interface changes that need to be resolved before a new stable release is cut.

這篇關于具有分布式集群的 Python 多處理的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！