問題描述
我需要能夠在 Python 中通過 FTP 和 SFTP 上傳文件，但有一些不太常見的限制.
I need to be able to upload a file through FTP and SFTP in Python but with some not so usual constraints.
文件不得寫入磁盤.
File MUST NOT be written in disk.
文件的生成方式是調用 API 并將 JSON 格式的響應寫入文件.
The file how it is generated is by calling an API and writing the response which is in JSON to the file.
API 有多次調用.無法通過一次 API 調用來檢索整個結果.
There are multiple calls to the API. It is not possible to retrieve the whole result in one single call of the API.
我無法將完整的結果存儲在字符串變量中，方法是執行所需的多次調用并附加到每個調用中，直到我將整個文件存儲在內存中.文件可能很大并且存在內存資源限制.應該發送每個塊并釋放內存.
I can not store in a string variable the full result by doing the multiple calls needed and appending in each call until I have the whole file in memory. File could be huge and there is a memory resource constraint. Each chunk should be sent and memory deallocated.
所以這里有一些我想要的示例代碼:
So here some sample code of what I would like to:
def chunks_generator():
range_list = range(0, 4000, 100)
for i in range_list:
data_chunk = requests.get(url=someurl, url_parameters={'offset':i, 'limit':100})
yield str(data_chunk)
def upload_file():
chunks_generator = chunks_generator()
for chunk in chunks_generator:
data_chunk= chunk
chunk_io = io.BytesIO(data_chunk)
ftp = FTP(self.host)
ftp.login(user=self.username, passwd=self.password)
ftp.cwd(self.remote_path)
ftp.storbinary("STOR " + "myfilename.json", chunk_io)
我只想要一個附加了所有塊的文件.我已經擁有并且正在工作的是，如果我將整個文件保存在內存中并像這樣立即發送它:
I want only one file with all the chunks appended. What I have already and works is if I have the whole file in memory and send it at once like this:
string_io = io.BytesIO(all_chunks_together_in_one_string)
ftp = FTP(self.host)
ftp.login(user=self.username, passwd=self.password)
ftp.cwd(self.remote_path)
ftp.storbinary("STOR " + "myfilename.json", string_io )
獎金
我在 ftplib 中需要它，但在 Paramiko 中也需要它用于 SFTP.如果有任何其他庫可以更好地工作，我是開放的.
I need this in ftplib but will need it in Paramiko as well for SFTP. If there are any other libraries that this would work better I am open.
如果我需要壓縮文件怎么辦?我可以壓縮每個塊并一次發送 zip-chunked 塊嗎?
How about if I need to zip the file? Can I zip each chunk and send the zip-chunked chunk at a time?
推薦答案
可以實現類文件類，調用.read(blocksize)方法從requests 對象.
You can implement file-like class that upon calling .read(blocksize) method retrieves data from requests object.
類似這樣的東西(未經測試):
Something like this (untested):
class ChunksGenerator:
i = 0
requests = None
def __init__(self, requests)
self.requests = requests
def read(self, blocksize):
# TODO: somehow detect end-of-file and return false in that case
buf = requests.get(
url=someurl, url_parameters={'offset':self.i, 'limit':blocksize})
self.i += blocksize
return buf
generator = ChunksGenerator(requests)
ftp.storbinary("STOR " + "myfilename.json", generator)
<小時>
使用 Paramiko，您可以使用與 SFTPClient.putfo 方法.
這篇關于Python - 在 FTP 中按塊上傳內存文件(由 API 調用生成)的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！