問題描述

我已經定義了這個函數

def writeonfiles(a,seed):
    random.seed(seed)

    f = open(a, "w+")
    for i in range(0,10):
        j = random.randint(0,10)
        #print j
        f.write(j)
    f.close()

其中 a 是包含文件路徑的字符串，seed 是整數種子.我想以這樣一種方式并行化一個簡單的程序，即每個內核采用我提供的可用路徑之一，為其隨機生成器播種并在該文件上寫入一些隨機數，例如，如果我通過向量

Where a is a string containing the path of the file and seed is an integer seed. I want to parallelize a simple program in such a way that each core takes one of the available paths that I give in, seeds its random generator and write some random numbers on that files, so, for example, if I pass the vector

vector = [Test/file1.txt, Test/file2.txt] 

和種子

seeds = (123412, 989898), 

它為第一個可用的核心提供功能

it gives to the first available core the function

writeonfiles(Test/file1.txt, 123412) 

對于第二個具有不同參數的相同函數:

and to the second one the same function with different arguments:

writeonfiles(Test/file2.txt, 989898)

我在 Stackoverflow 上查看了很多類似的問題，但我無法做出任何解決方案.我嘗試的是:

I have looked through a lot of similar questions here on Stackoverflow, but I cannot make any solution work. What I tried is:

def writeonfiles_unpack(args):
    return writeonfiles(*args)
if __name__ == "__main__":
     folder = ["Test/%d.csv" %i for i in range(0,4)]
     seed = [234124, 663123, 12345 ,123833]
     p = multiprocessing.Pool()
     p.map(writeonfiles, (folder,seed))

并給我 TypeError: writeonfiles() 正好需要 2 個參數(1 個給定).

and gives me TypeError: writeonfiles() takes exactly 2 arguments (1 given).

我也試過了

if __name__ == "__main__":
    folder = ["Test/%d.csv" %i for i in range(0,4)]
    seed = [234124, 663123, 12345 ,123833]
    p = multiprocessing.Process(target=writeonfiles, args= [folder,seed])
    p.start()

但它給了我
種子文件/usr/lib/python2.7/random.py"，第 120 行超級(隨機，自我).seed(一)TypeError: unhashable type: 'list'

But it gives me
File "/usr/lib/python2.7/random.py", line 120, in seed super(Random, self).seed(a) TypeError: unhashable type: 'list'

最后，我嘗試了上下文管理器

Finally, I tried the contextmanager

 @contextmanager
 def poolcontext(*args, **kwargs):
     pool = multiprocessing.Pool(*args, **kwargs)
     yield pool
     pool.terminate()

if __name__ == "__main__":
    folder = ["Test/%d" %i for i in range(0,4)]
    seed = [234124, 663123, 12345 ,123833]
    a = zip(folder, seed)
    with poolcontext(processes = 3) as pool:
    results = pool.map(writeonfiles_unpack,a )

它會導致文件/usr/lib/python2.7/multiprocessing/pool.py"，第 572 行，在 get提高self._value

and it results in File "/usr/lib/python2.7/multiprocessing/pool.py", line 572, in get raise self._value

TypeError: 'module' 對象不可調用

TypeError: 'module' object is not callable

推薦答案

Python 2.7 缺少 Python 3.3+ 中的 starmap 池方法.您可以通過使用包裝器來裝飾您的目標函數來克服這個問題，該包裝器將參數元組解包并調用目標函數:

Python 2.7 lacks the starmap pool-method from Python 3.3+ . You can overcome this by decorating your target function with a wrapper, which unpacks the argument-tuple and calls the target function:

import os
from multiprocessing import Pool
import random
from functools import wraps


def unpack(func):
    @wraps(func)
    def wrapper(arg_tuple):
        return func(*arg_tuple)
    return wrapper

@unpack
def write_on_files(a, seed):
    random.seed(seed)
    print("%d opening file %s" % (os.getpid(), a))  # simulate
    for _ in range(10):
        j = random.randint(0, 10)
       print("%d writing %d to file %s" % (os.getpid(), j, a))  # simulate


if __name__ == '__main__':

    folder = ["Test/%d.csv" % i for i in range(0, 4)]
    seed = [234124, 663123, 12345, 123833]

    arguments = zip(folder, seed)

    pool = Pool(4)
    pool.map(write_on_files, iterable=arguments)
    pool.close()
    pool.join()

這篇關于Python 2.7:如何彌補缺少的 pool.starmap?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！