久久久久久久av_日韩在线中文_看一级毛片视频_日本精品二区_成人深夜福利视频_武道仙尊动漫在线观看

    <bdo id='qW1am'></bdo><ul id='qW1am'></ul>
    <tfoot id='qW1am'></tfoot>
    1. <legend id='qW1am'><style id='qW1am'><dir id='qW1am'><q id='qW1am'></q></dir></style></legend>

      1. <i id='qW1am'><tr id='qW1am'><dt id='qW1am'><q id='qW1am'><span id='qW1am'><b id='qW1am'><form id='qW1am'><ins id='qW1am'></ins><ul id='qW1am'></ul><sub id='qW1am'></sub></form><legend id='qW1am'></legend><bdo id='qW1am'><pre id='qW1am'><center id='qW1am'></center></pre></bdo></b><th id='qW1am'></th></span></q></dt></tr></i><div class="qwawimqqmiuu" id='qW1am'><tfoot id='qW1am'></tfoot><dl id='qW1am'><fieldset id='qW1am'></fieldset></dl></div>

        <small id='qW1am'></small><noframes id='qW1am'>

      2. 帶有生成器的 Python 多處理

        Python multiprocessing with generator(帶有生成器的 Python 多處理)
            <bdo id='iOsQw'></bdo><ul id='iOsQw'></ul>
              <legend id='iOsQw'><style id='iOsQw'><dir id='iOsQw'><q id='iOsQw'></q></dir></style></legend>

                  <tbody id='iOsQw'></tbody>

                <tfoot id='iOsQw'></tfoot>

                <i id='iOsQw'><tr id='iOsQw'><dt id='iOsQw'><q id='iOsQw'><span id='iOsQw'><b id='iOsQw'><form id='iOsQw'><ins id='iOsQw'></ins><ul id='iOsQw'></ul><sub id='iOsQw'></sub></form><legend id='iOsQw'></legend><bdo id='iOsQw'><pre id='iOsQw'><center id='iOsQw'></center></pre></bdo></b><th id='iOsQw'></th></span></q></dt></tr></i><div class="qwawimqqmiuu" id='iOsQw'><tfoot id='iOsQw'></tfoot><dl id='iOsQw'><fieldset id='iOsQw'></fieldset></dl></div>
                • <small id='iOsQw'></small><noframes id='iOsQw'>

                  本文介紹了帶有生成器的 Python 多處理的處理方法,對大家解決問題具有一定的參考價值,需要的朋友們下面隨著小編來一起學習吧!

                  問題描述

                  限時送ChatGPT賬號..

                  我正在嘗試處理一個文件(每一行都是一個 json 文檔).文件的大小可以達到 100 mbs 到 gb 的.所以我寫了一個生成器代碼來逐行從文件中獲取每個文檔.

                  I'm trying to process a file(every line is a json document). The size of the file can go up to 100's of mbs to gb's. So I wrote a generator code to fetch each document line by line from file.

                  def jl_file_iterator(file):
                      with codecs.open(file, 'r', 'utf-8') as f:
                          for line in f:
                              document = json.loads(line)
                              yield document
                  

                  我的系統有 4 個核心,所以我想并行處理 4 行文件.目前我有這段代碼,一次需要 4 行,并調用代碼進行并行處理

                  My system has 4 cores, So I would like to process 4 lines of the file in parallel. Currently I have this code which takes 4 lines at a time and calls the code for parallel processing

                  threads = 4
                  files, i = [], 1
                  for jl in jl_file_iterator(input_path):
                      files.append(jl)
                      if i % (threads) == 0:
                          # pool.map(processFile, files)
                          parallelProcess(files, o)
                          files = []
                      i += 1
                  
                  if files:
                      parallelProcess(files, o)
                      files = []
                  

                  這是我進行實際處理的代碼

                  This is my code where actual processing happens

                  def parallelProcess(files, outfile):
                      processes = []
                      for i in range(len(files)):
                          p = Process(target=processFile, args=(files[i],))
                          processes.append(p)
                          p.start()
                      for i in range(len(files)):
                          processes[i].join()
                  
                  def processFile(doc):
                      extractors = {}
                      ... do some processing on doc
                      o.write(json.dumps(doc) + '
                  ')
                  

                  如您所見,我等待所有 4 行完成處理,然后再發送接下來的 4 個文件進行處理.但是我想做的是,一旦一個進程完成處理文件,我就想開始下一行以分配給已發布的處理器.我怎么做?

                  As you can see I wait for all the 4 lines to finish processing before I send the next 4 files to process. But what I would like to do is as soon as one process finish processing file I want to start the next line to be assigned to realeased processor. How do I do that?

                  PS:問題是因為它是一個生成器,所以我無法加載所有文件并使用 map 之類的東西來運行進程.

                  PS: The problem is since its an generator I cannot load all the files and use something like map to run the processes.

                  感謝您的幫助

                  推薦答案

                  正如@pvg 在評論中所說,(有界)隊列是在不同速度的生產者和消費者之間進行調解的自然方式,確保他們都保持不變盡可能忙,但不讓制作人領先.

                  As @pvg said in a comment, a (bounded) queue is the natural way to mediate among a producer and consumers with different speeds, ensuring they all stay as busy as possible but without letting the producer get way ahead.

                  這是一個獨立的可執行示例.隊列被限制為等于工作進程數的最大大小.如果消費者的運行速度比生產者快得多,那么讓隊列變得更大是很有意義的.

                  Here's a self-contained, executable example. The queue is restricted to a maximum size equal to the number of worker processes. If the consumers run much faster than the producer, it could make good sense to let the queue get bigger than that.

                  在您的特定情況下,將行傳遞給消費者并讓他們并行執行 document = json.loads(line) 部分可能是有意義的.

                  In your specific case, it would probably make sense to pass lines to the consumers and let them do the document = json.loads(line) part in parallel.

                  import multiprocessing as mp
                  
                  NCORE = 4
                  
                  def process(q, iolock):
                      from time import sleep
                      while True:
                          stuff = q.get()
                          if stuff is None:
                              break
                          with iolock:
                              print("processing", stuff)
                          sleep(stuff)
                  
                  if __name__ == '__main__':
                      q = mp.Queue(maxsize=NCORE)
                      iolock = mp.Lock()
                      pool = mp.Pool(NCORE, initializer=process, initargs=(q, iolock))
                      for stuff in range(20):
                          q.put(stuff)  # blocks until q below its max size
                          with iolock:
                              print("queued", stuff)
                      for _ in range(NCORE):  # tell workers we're done
                          q.put(None)
                      pool.close()
                      pool.join()
                  

                  這篇關于帶有生成器的 Python 多處理的文章就介紹到這了,希望我們推薦的答案對大家有所幫助,也希望大家多多支持html5模板網!

                  【網站聲明】本站部分內容來源于互聯網,旨在幫助大家更快的解決問題,如果有圖片或者內容侵犯了您的權益,請聯系我們刪除處理,感謝您的支持!

                  相關文檔推薦

                  What exactly is Python multiprocessing Module#39;s .join() Method Doing?(Python 多處理模塊的 .join() 方法到底在做什么?)
                  Passing multiple parameters to pool.map() function in Python(在 Python 中將多個參數傳遞給 pool.map() 函數)
                  multiprocessing.pool.MaybeEncodingError: #39;TypeError(quot;cannot serialize #39;_io.BufferedReader#39; objectquot;,)#39;(multiprocessing.pool.MaybeEncodingError: TypeError(cannot serialize _io.BufferedReader object,)) - IT屋-程序員軟件開
                  Python Multiprocess Pool. How to exit the script when one of the worker process determines no more work needs to be done?(Python 多進程池.當其中一個工作進程確定不再需要完成工作時,如何退出腳本?) - IT屋-程序員
                  How do you pass a Queue reference to a function managed by pool.map_async()?(如何將隊列引用傳遞給 pool.map_async() 管理的函數?)
                  yet another confusion with multiprocessing error, #39;module#39; object has no attribute #39;f#39;(與多處理錯誤的另一個混淆,“模塊對象沒有屬性“f)

                    • <bdo id='gZOCb'></bdo><ul id='gZOCb'></ul>

                        <tbody id='gZOCb'></tbody>

                        1. <tfoot id='gZOCb'></tfoot>
                          • <legend id='gZOCb'><style id='gZOCb'><dir id='gZOCb'><q id='gZOCb'></q></dir></style></legend>
                            <i id='gZOCb'><tr id='gZOCb'><dt id='gZOCb'><q id='gZOCb'><span id='gZOCb'><b id='gZOCb'><form id='gZOCb'><ins id='gZOCb'></ins><ul id='gZOCb'></ul><sub id='gZOCb'></sub></form><legend id='gZOCb'></legend><bdo id='gZOCb'><pre id='gZOCb'><center id='gZOCb'></center></pre></bdo></b><th id='gZOCb'></th></span></q></dt></tr></i><div class="qwawimqqmiuu" id='gZOCb'><tfoot id='gZOCb'></tfoot><dl id='gZOCb'><fieldset id='gZOCb'></fieldset></dl></div>
                          • <small id='gZOCb'></small><noframes id='gZOCb'>

                            主站蜘蛛池模板: 成人一区二区三区在线观看 | 国产一级视频在线观看 | 在线观看黄色 | 国产精品久久久久久久久久免费看 | 国产精品精品久久久 | 精品久久电影 | 成人网在线 | 国产激情毛片 | 日韩不卡在线 | 久久综合久 | 91素人| 欧美综合一区 | 成在线人视频免费视频 | 国产精品爱久久久久久久 | 在线免费看毛片 | 亚洲欧美久久 | 国产精品一区二区三区四区五区 | 免费观看一级特黄欧美大片 | www.操.com | 亚洲五码久久 | 日韩精品一区二区三区视频播放 | 成人在线一级片 | 国产做爰 | 国产精品色哟哟网站 | 精品久久久精品 | 成人久久 | 久久久久国产一区二区三区不卡 | jdav视频在线观看免费 | 亚洲美女在线一区 | 视频国产一区 | 性色的免费视频 | 精品国产一区二区在线 | 99久久精品国产一区二区三区 | 亚洲精品观看 | 日本不卡一区二区三区在线观看 | 日韩欧美中文字幕在线视频 | 99国内精品 | 在线播放亚洲 | 日韩中文字幕视频在线观看 | 99热碰| 伊人精品在线视频 |