問題描述

我正在使用 Python 2.6 和多處理模塊進行多線程處理.現在我想要一個同步的字典(我真正需要的唯一原子操作是一個值上的 += 運算符).

I am using Python 2.6 and the multiprocessing module for multi-threading. Now I would like to have a synchronized dict (where the only atomic operation I really need is the += operator on a value).

我應該用 multiprocessing.sharedctypes.synchronized() 調用來包裝字典嗎?或者是另一種方式?

Should I wrap the dict with a multiprocessing.sharedctypes.synchronized() call? Or is another way the way to go?

推薦答案

簡介

似乎有很多扶手椅建議，但沒有工作示例.這里列出的答案都沒有建議使用多處理，這有點令人失望和不安.作為 python 愛好者，我們應該支持我們的內置庫，雖然并行處理和同步從來都不是一件小事，但我相信通過適當的設計可以讓它變得微不足道.這在現代多核架構中變得極為重要，怎么強調都不為過！也就是說，我對多處理庫還很不滿意，因為它仍處于起步階段，有很多陷阱、錯誤，并且面向函數式編程(我討厭).目前，由于多處理在在服務器運行時無法共享新創建的對象.管理器對象的注冊"類方法只會在管理器(或其服務器)啟動之前實際注冊一個對象.廢話不多說，更多代碼:

Intro

There seems to be a lot of arm-chair suggestions and no working examples. None of the answers listed here even suggest using multiprocessing and this is quite a bit disappointing and disturbing. As python lovers we should support our built-in libraries, and while parallel processing and synchronization is never a trivial matter, I believe it can be made trivial with proper design. This is becoming extremely important in modern multi-core architectures and cannot be stressed enough! That said, I am far from satisfied with the multiprocessing library, as it is still in its infancy stages with quite a few pitfalls, bugs, and being geared towards functional programming (which I detest). Currently I still prefer the Pyro module (which is way ahead of its time) over multiprocessing due to multiprocessing's severe limitation in being unable to share newly created objects while the server is running. The "register" class-method of the manager objects will only actually register an object BEFORE the manager (or its server) is started. Enough chatter, more code:

from multiprocessing.managers import SyncManager


class MyManager(SyncManager):
    pass


syncdict = {}
def get_dict():
    return syncdict

if __name__ == "__main__":
    MyManager.register("syncdict", get_dict)
    manager = MyManager(("127.0.0.1", 5000), authkey="password")
    manager.start()
    raw_input("Press any key to kill server".center(50, "-"))
    manager.shutdown()

在上面的代碼示例中，Server.py 使用了 multiprocessing 的 SyncManager，它可以提供同步的共享對象.此代碼無法在解釋器中運行，因為多處理庫對于如何找到每個注冊對象的可調用"非常敏感.運行 Server.py 將啟動一個自定義的 SyncManager，該 SyncManager 共享同步字典以供多個進程使用，并且可以連接到同一臺機器上的客戶端，或者如果在環回以外的 IP 地址上運行，則可以連接到其他機器.在這種情況下，服務器在端口 5000 上的環回 (127.0.0.1) 上運行.使用 authkey 參數在操作 syncdict 時使用安全連接.按下任意鍵時，管理器將關閉.

In the above code example, Server.py makes use of multiprocessing's SyncManager which can supply synchronized shared objects. This code will not work running in the interpreter because the multiprocessing library is quite touchy on how to find the "callable" for each registered object. Running Server.py will start a customized SyncManager that shares the syncdict dictionary for use of multiple processes and can be connected to clients either on the same machine, or if run on an IP address other than loopback, other machines. In this case the server is run on loopback (127.0.0.1) on port 5000. Using the authkey parameter uses secure connections when manipulating syncdict. When any key is pressed the manager is shutdown.

from multiprocessing.managers import SyncManager
import sys, time

class MyManager(SyncManager):
    pass

MyManager.register("syncdict")

if __name__ == "__main__":
    manager = MyManager(("127.0.0.1", 5000), authkey="password")
    manager.connect()
    syncdict = manager.syncdict()

    print "dict = %s" % (dir(syncdict))
    key = raw_input("Enter key to update: ")
    inc = float(raw_input("Enter increment: "))
    sleep = float(raw_input("Enter sleep time (sec): "))

    try:
         #if the key doesn't exist create it
         if not syncdict.has_key(key):
             syncdict.update([(key, 0)])
         #increment key value every sleep seconds
         #then print syncdict
         while True:
              syncdict.update([(key, syncdict.get(key) + inc)])
              time.sleep(sleep)
              print "%s" % (syncdict)
    except KeyboardInterrupt:
         print "Killed client"

客戶端還必須創建一個自定義的 SyncManager，注冊syncdict"，這一次不傳入可調用來檢索共享字典.然后，它使用自定義的 SycnManager 使用端口 5000 上的環回 IP 地址 (127.0.0.1) 和在 Server.py 中啟動的與管理器建立安全連接的 authkey 進行連接.它通過調用管理器上注冊的可調用對象來檢索共享字典同步字典.它會提示用戶以下內容:

The client must also create a customized SyncManager, registering "syncdict", this time without passing in a callable to retrieve the shared dict. It then uses the customized SycnManager to connect using the loopback IP address (127.0.0.1) on port 5000 and an authkey establishing a secure connection to the manager started in Server.py. It retrieves the shared dict syncdict by calling the registered callable on the manager. It prompts the user for the following:

1. syncdict 中要操作的鍵
2. 每次循環增加鍵訪問的值的數量
3. 每個周期的睡眠時間(以秒為單位)

客戶端然后檢查密鑰是否存在.如果不是，它會在同步字典上創建密鑰.客戶端然后進入一個無限"循環，在該循環中它通過增量更新鍵的值，休眠指定的數量，并打印同步字典，僅重復此過程，直到發生 KeyboardInterrupt (Ctrl+C).

The client then checks to see if the key exists. If it doesn't it creates the key on the syncdict. The client then enters an "endless" loop where it updates the key's value by the increment, sleeps the amount specified, and prints the syncdict only to repeat this process until a KeyboardInterrupt occurs (Ctrl+C).

1. 必須在管理器啟動之前調用管理器的注冊方法，否則即使對管理器的 dir 調用會顯示它確實具有已注冊的方法，也會出現異常.
2. 必須使用方法而不是 dict 分配來完成對 dict 的所有操作(由于多處理共享自定義對象的方式，syncdict["blast"] = 2 會慘遭失敗)
3. 使用 SyncManager 的 dict 方法可以緩解煩人的問題 #2，但煩人的問題 #1 會阻止 SyncManager.dict() 返回的代理被注冊和共享.(SyncManager.dict() 只能在管理器啟動后調用，并且注冊只會在管理器啟動之前起作用，因此 SyncManager.dict() 僅在進行函數式編程并將代理作為參數傳遞給進程時有用，如文檔示例)
4. 服務器和客戶端都必須注冊，盡管從直覺上看，客戶端似乎只能在連接到管理器后才能弄清楚(請將此添加到您的愿望清單多處理開發人員中)

結束

我希望你和我一樣喜歡這個相當徹底但稍微費時的答案.我很難弄清楚為什么我在多處理模塊上苦苦掙扎，Pyro 讓它變得輕而易舉，現在多虧了這個答案，我已經一針見血了.我希望這對 python 社區如何改進多處理模塊有用，因為我相信它有很大的希望，但在它的起步階段還沒有實現.盡管描述了令人討厭的問題，但我認為這仍然是一個非常可行的選擇并且非常簡單.您也可以使用 SyncManager.dict() 并將其作為參數傳遞給 Processes，就像文檔顯示的那樣，根據您的要求，這可能是一個更簡單的解決方案，這對我來說是不自然的.

Closing

I hope you enjoyed this quite thorough and slightly time-consuming answer as much as I have. I was having a great deal of trouble getting straight in my mind why I was struggling so much with the multiprocessing module where Pyro makes it a breeze and now thanks to this answer I have hit the nail on the head. I hope this is useful to the python community on how to improve the multiprocessing module as I do believe it has a great deal of promise but in its infancy falls short of what is possible. Despite the annoying problems described I think this is still quite a viable alternative and is pretty simple. You could also use SyncManager.dict() and pass it to Processes as an argument the way the docs show and it would probably be an even simpler solution depending on your requirements it just feels unnatural to me.

這篇關于如何將python dict與多處理同步的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！