問題描述

我正在嘗試使用 Python 的 pathos 將計算指定到單獨的進程中，以便使用多核處理器加速它.我的代碼組織如下:

I am trying to use Python's pathos to designate computations into separate processes in order to accelerate it with multicore processor. My code is organized like:

class:
   def foo(self,name):
    ...
    setattr(self,name,something)
    ...
   def boo(self):
      for name in list:
         self.foo(name)

由于我在使用 multiprocessing.Pool 時遇到了酸洗問題，所以我決定嘗試一下 pathos.我嘗試過，如先前主題中所建議的那樣:

As I had pickling problems with multiprocessing.Pool, I decided to try pathos. I tried, as suggested in previous topics:

import pathos.multiprocessing

但它導致錯誤:沒有模塊多處理 - 我在最新的 pathos 版本中找不到.

but it resulted in error: No module multiprocessing - which I can't find in latest pathos version.

然后我嘗試修改boo方法:

Then I tried modify boo method:

def boo(self):
 import pathos
 pathos.pp_map.pp_map(self.foo,list)

現在沒有拋出錯誤，但 foo 不起作用 - 我的類的實例沒有新屬性.請幫助我，因為在花了一天時間之后，我不知道下一步該去哪里.

Now there is no error thrown, but foo does not work - instance of my class has no new attributes. Please help me, because I have no idea where to move next, after a day spent on that.

推薦答案

我是pathos的作者.我不確定您想從上面的代碼中做什么.但是，我也許可以闡明一些觀點.下面是一些類似的代碼:

I'm the pathos author. I'm not sure what you want to do from your code above. However, I can maybe shed some light. Here's some similar code:

>>> from pathos.multiprocessing import ProcessingPool
>>> class Bar:
...   def foo(self, name):
...     return len(str(name))
...   def boo(self, things):
...     for thing in things:
...       self.sum += self.foo(thing)
...     return self.sum
...   sum = 0
... 
>>> b = Bar()
>>> results = ProcessingPool().map(b.boo, [[12,3,456],[8,9,10],['a','b','cde']])
>>> results
[6, 4, 5]
>>> b.sum
0

那么上面發生的事情是 Bar 實例 b 的 boo 方法在 b.boo 被傳遞給一個新的 python 進程，然后對每個嵌套列表進行評估.可以看到結果是正確的……len("12")+len("3")+len("456")是6，以此類推.

So what happens above, is that the boo method of the Bar instance b is called where b.boo is passed to a new python process, and then evaluated for each of the nested lists. You can see that the results are correct… len("12")+len("3")+len("456") is 6, and so on.

但是，您也可以看到，當您查看 b.sum 時，它神秘地仍然是 0.為什么 b.sum 仍然為零?好吧，multiprocessing(以及pathos.multiprocessing)所做的，就是將您通過地圖傳遞給其他python的任何內容進行COPY過程……然后(并行)調用復制的實例并返回被調用的方法調用的任何結果.請注意，您必須返回結果，或打印它們，或記錄它們，或將它們發送到文件，或以其他方式.它們無法像您預期的那樣返回到原始實例，因為它不是發送到其他處理器的原始實例.實例的副本被創建，然后被丟棄——它們每個都增加了它們的 sum 屬性，但原始的 `b.sum' 沒有改變.

However, you can also see that when you look at b.sum, it's mysteriously still 0. Why is b.sum still zero? Well, what multiprocessing (and thus also pathos.multiprocessing) does, is make a COPY of whatever you pass through the map to the other python process… and then the copied instance is then called (in parallel) and return whatever results are called by the method invoked. Note you have to RETURN results, or print them, or log them, or send them to a file, or otherwise. They can't go back to the original instance as you might expect, because it's not the original instance that's sent over to the other processors. The copies of the instance are created, then disposed of -- each of them had their sum attribute increased, but the original `b.sum' is untouched.

然而，pathos 內計劃使上述工作如您所料 - 原始對象 IS 已更新，但它不起作用還是那樣.

There is however, plans within pathos to make something like the above work as you might expect -- where the original object IS updated, but it doesn't work like that yet.

如果您使用 pip 進行安裝，請注意最新發布的 pathos 版本已有幾年歷史，可能無法安裝正確，或者可能無法安裝所有子模塊.一個新的 pathos 版本正在等待中，但在那之前，最好從 github 獲取最新版本的代碼，然后從那里安裝.主干大部分穩定在開發中.我認為您的問題可能是由于安裝中的新"pip --舊"pathos 不兼容，并非所有軟件包都已安裝.如果 pathos.multiprocessing 缺失，這很可能是罪魁禍首.

If you are installing with pip, note that the latest released version of pathos is several years old, and may not install correctly, or may not install all of the submodules. A new pathos release is pending, but until then, it's better to get the latest version of the code from github, and install from there. The trunk is for the most part stable under development. I think your issue may have been that not all packages were installed, due to a "new" pip -- "old" pathos incompatibility in the install. If pathos.multiprocessing is missing, this is the most likely culprit.

在此處從 github 獲取 pathos:https://github.com/uqfoundation/pathos

Get pathos from github here: https://github.com/uqfoundation/pathos

這篇關于帶有悲情的 Python 多處理的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！