問(wèn)題描述

我想在隨后將傳遞給多處理池的函數(shù)上使用裝飾器.但是，代碼因PicklingError: Can't pickle : attribute lookup __builtin__.function failed"而失敗.我不太明白為什么它在這里失敗.我確信這很簡(jiǎn)單，但我找不到.下面是一個(gè)最小的工作"示例.我認(rèn)為使用 functools 函數(shù)就足以讓它工作.

I would like to use a decorator on a function that I will subsequently pass to a multiprocessing pool. However, the code fails with "PicklingError: Can't pickle : attribute lookup __builtin__.function failed". I don't quite see why it fails here. I feel certain that it's something simple, but I can't find it. Below is a minimal "working" example. I thought that using the functools function would be enough to let this work.

如果我注釋掉函數(shù)裝飾，它可以正常工作.我在這里誤解的 multiprocessing 是什么?有什么辦法可以做到嗎?

If I comment out the function decoration, it works without an issue. What is it about multiprocessing that I'm misunderstanding here? Is there any way to make this work?

編輯:在添加了一個(gè)可調(diào)用的類裝飾器和一個(gè)函數(shù)裝飾器之后，函數(shù)裝飾器按預(yù)期工作.可調(diào)用的類裝飾器繼續(xù)失敗.防止它被腌制的可調(diào)用類版本是什么?

Edit: After adding both a callable class decorator and a function decorator, it turns out that the function decorator works as expected. The callable class decorator continues to fail. What is it about the callable class version that keeps it from being pickled?

import random
import multiprocessing
import functools

class my_decorator_class(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, elements):
        f = []
        for element in elements:
            f.append(self.target([element])[0])
        return f

def my_decorator_function(target):
    @functools.wraps(target)
    def inner(elements):
        f = []
        for element in elements:
            f.append(target([element])[0])
        return f
    return inner

@my_decorator_function
def my_func(elements):
    f = []
    for element in elements:
        f.append(sum(element))
    return f

if __name__ == '__main__':
    elements = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([e],)) for e in elements]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

推薦答案

問(wèn)題是 pickle 需要有一些方法來(lái)重新組裝你 pickle 的所有東西.請(qǐng)參閱此處了解可以腌制的內(nèi)容:

The problem is that pickle needs to have some way to reassemble everything that you pickle. See here for a list of what can be pickled:

http://docs.python.org/library/pickle.html#what-c??an-be-pickled-and-unpickled

酸洗my_func時(shí)，需要酸洗以下組件:

When pickling my_func, the following components need to be pickled:

• my_decorator_class 的一個(gè)實(shí)例，稱為 my_func.

• An instance of my_decorator_class, called my_func.

這很好.Pickle 將存儲(chǔ)類的名稱并腌制其 __dict__ 內(nèi)容.unpickling 時(shí)，它使用名稱查找類，然后創(chuàng)建一個(gè)實(shí)例并填寫(xiě) __dict__ 內(nèi)容.但是，__dict__ 內(nèi)容存在問(wèn)題...

This is fine. Pickle will store the name of the class and pickle its __dict__ contents. When unpickling, it uses the name to find the class, then creates an instance and fills in the __dict__ contents. However, the __dict__ contents present a problem...

存儲(chǔ)在 my_func.target 中的原始 my_func 的實(shí)例.

The instance of the original my_func that's stored in my_func.target.

這不太好.它是頂層的函數(shù)，通常可以腌制.Pickle 將存儲(chǔ)函數(shù)的名稱.然而，問(wèn)題在于名稱my_func"不再綁定到未修飾的函數(shù)，而是綁定到修飾的函數(shù).這意味著 pickle 將無(wú)法查找未修飾的函數(shù)來(lái)重新創(chuàng)建對(duì)象.遺憾的是，pickle 無(wú)法知道它試圖腌制的對(duì)象總是可以在名稱 __main__.my_func 下找到.

This isn't so good. It's a function at the top-level, and normally these can be pickled. Pickle will store the name of the function. The problem, however, is that the name "my_func" is no longer bound to the undecorated function, it's bound to the decorated function. This means that pickle won't be able to look up the undecorated function to recreate the object. Sadly, pickle doesn't have any way to know that object it's trying to pickle can always be found under the name __main__.my_func.

你可以這樣改變它，它會(huì)起作用:

You can change it like this and it will work:

import random
import multiprocessing
import functools

class my_decorator(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, candidates, args):
        f = []
        for candidate in candidates:
            f.append(self.target([candidate], args)[0])
        return f

def old_my_func(candidates, args):
    f = []
    for c in candidates:
        f.append(sum(c))
    return f

my_func = my_decorator(old_my_func)

if __name__ == '__main__':
    candidates = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([c], {})) for c in candidates]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

您已經(jīng)觀察到裝飾器功能在類不起作用時(shí)起作用.我相信這是因?yàn)?functools.wraps 修改了裝飾函數(shù)，使其具有它包裝的函數(shù)的名稱和其他屬性.就 pickle 模塊而言，它與普通的頂級(jí)函數(shù)沒(méi)有區(qū)別，因此它通過(guò)存儲(chǔ)其名稱來(lái)腌制它.解壓后，名稱將綁定到裝飾函數(shù)，因此一切正常.

You have observed that the decorator function works when the class does not. I believe this is because functools.wraps modifies the decorated function so that it has the name and other properties of the function it wraps. As far as the pickle module can tell, it is indistinguishable from a normal top-level function, so it pickles it by storing its name. Upon unpickling, the name is bound to the decorated function so everything works out.

這篇關(guān)于具有多處理功能的 Python 裝飾器失敗的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，也希望大家多多支持html5模板網(wǎng)！