問題描述

我正在使用 urllib2 從 ftp 和 http 服務(wù)器加載文件.

某些服務(wù)器僅支持每個(gè) IP 一個(gè)連接.問題是，urllib2 不會(huì)立即關(guān)閉連接.查看示例程序.

從 urllib2 導(dǎo)入 urlopen從時(shí)間導(dǎo)入睡眠url = 'ftp://user:pass@host/big_file.ext'定義加載文件(網(wǎng)址):f = urlopen(url)加載 = 0而真:數(shù)據(jù) = f.read(1024)如果數(shù)據(jù) == '':休息已加載 += len(數(shù)據(jù))f.close()#睡眠(1)print('已加載 {0}'.format(已加載))加載文件(網(wǎng)址)加載文件(網(wǎng)址)

代碼從僅支持 1 個(gè)連接的 ftp 服務(wù)器加載兩個(gè)文件(此處兩個(gè)文件相同).這將打印以下日志:

已加載 463675266回溯(最近一次通話最后):文件conection_test.py"，第 20 行，在 <module>加載文件(網(wǎng)址)文件connection_test.py"，第 7 行，在 load_file 中f = urlopen(url)文件/usr/lib/python2.6/urllib2.py"，第 126 行，在 urlopenreturn _opener.open(網(wǎng)址，數(shù)據(jù)，超時(shí))文件/usr/lib/python2.6/urllib2.py"，第 391 行，打開響應(yīng) = self._open(請求，數(shù)據(jù))_open 中的文件/usr/lib/python2.6/urllib2.py"，第 409 行'_open'，請求)_call_chain 中的文件/usr/lib/python2.6/urllib2.py"，第 369 行結(jié)果 = 函數(shù)(*args)文件/usr/lib/python2.6/urllib2.py"，第 1331 行，在 ftp_openfw = self.connect_ftp(用戶，密碼，主機(jī)，端口，目錄，req.timeout)文件/usr/lib/python2.6/urllib2.py"，第 1352 行，在 connect_ftpfw = ftpwrapper(用戶、密碼、主機(jī)、端口、目錄、超時(shí))__init__ 中的文件/usr/lib/python2.6/urllib.py"，第 854 行self.init()文件/usr/lib/python2.6/urllib.py"，第 860 行，在 initself.ftp.connect(self.host，self.port，self.timeout)文件/usr/lib/python2.6/ftplib.py"，第 134 行，在連接中self.welcome = self.getresp()文件/usr/lib/python2.6/ftplib.py"，第 216 行，在 getresp 中提高error_temp，respurllib2.URLError: <urlopen 錯(cuò)誤 ftp 錯(cuò)誤: 421 來自您的 Internet 地址的連接太多.>

所以第一個(gè)文件被加載，第二個(gè)文件失敗，因?yàn)榈谝粋€(gè)連接沒有關(guān)閉.

但是當(dāng)我在 f.close() 之后使用 sleep(1) 時(shí)不會(huì)發(fā)生錯(cuò)誤:

已加載 463675266已加載 463675266

有什么辦法可以強(qiáng)制關(guān)閉連接，以免第二次下載失敗?

原因確實(shí)是文件描述符泄漏.我們還發(fā)現(xiàn)，使用 jython 時(shí)，問題比使用 cpython 時(shí)要明顯得多.一位同事提出了這個(gè)解決方案:

修復(fù)很丑陋，但確實(shí)有效，不再有打開的連接太多".

I'm using urllib2 to load files from ftp- and http-servers.

Some of the servers support only one connection per IP. The problem is, that urllib2 does not close the connection instantly. Look at the example-program.

from urllib2 import urlopen
from time import sleep

url = 'ftp://user:pass@host/big_file.ext'

def load_file(url):
    f = urlopen(url)
    loaded = 0
    while True:
        data = f.read(1024)
        if data == '':
            break
        loaded += len(data)
    f.close()
    #sleep(1)
    print('loaded {0}'.format(loaded))

load_file(url)
load_file(url)

The code loads two files (here the two files are the same) from an ftp-server which supports only 1 connection. This will print the following log:

loaded 463675266
Traceback (most recent call last):
  File "conection_test.py", line 20, in <module>
    load_file(url)
  File "conection_test.py", line 7, in load_file
    f = urlopen(url)
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1331, in ftp_open
    fw = self.connect_ftp(user, passwd, host, port, dirs, req.timeout)
  File "/usr/lib/python2.6/urllib2.py", line 1352, in connect_ftp
    fw = ftpwrapper(user, passwd, host, port, dirs, timeout)
  File "/usr/lib/python2.6/urllib.py", line 854, in __init__
    self.init()
  File "/usr/lib/python2.6/urllib.py", line 860, in init
    self.ftp.connect(self.host, self.port, self.timeout)
  File "/usr/lib/python2.6/ftplib.py", line 134, in connect
    self.welcome = self.getresp()
  File "/usr/lib/python2.6/ftplib.py", line 216, in getresp
    raise error_temp, resp
urllib2.URLError: <urlopen error ftp error: 421 There are too many connections from your internet address.>

So the first file is loaded and the second fails because the first connection was not closed.

But when i use sleep(1) after f.close() the error does not occurr:

loaded 463675266
loaded 463675266

Is there any way to force close the connection so that the second download would not fail?

The cause is indeed a file descriptor leak. We found also that with jython, the problem is much more obvious than with cpython. A colleague proposed this sollution:

 

    fdurl = urllib2.urlopen(req,timeout=self.timeout)
    realsock = fdurl.fp._sock.fp._sock** # we want to close the "real" socket later 
    req = urllib2.Request(url, header)
    try:
             fdurl = urllib2.urlopen(req,timeout=self.timeout)
    except urllib2.URLError,e:
              print "urlopen exception", e
    realsock.close() 
    fdurl.close()

The fix is ugly, but does the job, no more "too many open connections".

這篇關(guān)于關(guān)閉 urllib2 連接的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！