問題描述

我有一個要求，我必須使用大型 XML(4 GB 文件)來查找其中的值.基本上我必須編寫大約 30 個不同的 Xpath 并將值存儲在一個列表中.當我嘗試解析 XML 時，它會引發內存錯誤.我已經嘗試使用 lxml 和 ElementTree 來處理開始和結束事件，但處理時間太長還是不行，而且我的 Pycharm/Jupyter 筆記本會引發內存錯誤.

I have a requirement where I have to use an large XML (4 GB file) for finding values in it. Basically I have to write around 30 different Xpath and store the values in a list. When I try to parse an XML, it throws memory error. I have tried using lxml and ElementTree with start and end events, still no luck the processing time is too high and my Pycharm/Jupyter notebook throws me memory error.

有沒有更好的方法呢?盡管此實現不限于任何編程語言，但我更喜歡 Python，因為它是我的右手.提前致謝.

Is there a better way to do it? Even though this implementation is not restricted to any programming language, I prefer Python because its my right hand. Thanks in advance.

例如搜索:如果我想要類別正在烹飪的年份的值.然后我使用 ./bookstore[@category=cooking]/book/year.所以值為 2005

Eg of a search: If I want the value of year where category is cooking. Then I use ./bookstore[@category=cooking]/book/year. So the value is 2005

同樣，我必須根據我的業務需求找到我的標簽的值.在我的要求中，XML 并不像下面的示例那樣簡單.

Similarly I have to find the values of my tags based on my business requirements. In my requirement the XML is not simple as the below example.

<?xml version="1.0" encoding="UTF-8"?>

<bookstore>

<book category="cooking">
  <title lang="en">Everyday Italian</title>
  <author>Giada De Laurentiis</author>
  <year>2005</year>
  <price>30.00</price>
</book>

<book category="children">
  <title lang="en">Harry Potter</title>
  <author>J K. Rowling</author>
  <year>2005</year>
  <price>29.99</price>
</book>

<book category="web">
  <title lang="en">XQuery Kick Start</title>
  <author>James McGovern</author>
  <author>Per Bothner</author>
  <author>Kurt Cagle</author>
  <author>James Linn</author>
  <author>Vaidyanathan Nagarajan</author>
  <year>2003</year>
  <price>49.99</price>
</book>

</bookstore>

推薦答案

不知道這樣能不能滿足你的需求.先試試看.如果有問題，我們可以繼續溝通.

I don't know if this will meet your needs. Try it first. If there is a problem, we can continue to communicate.

from simplified_scrapy import SimplifiedDoc,req,utils
def getBook(lines,category,year):
  html = "".join(lines)
  book = SimplifiedDoc(html).book
  if book.category==category and book.year.text==year:
    return book.category,book.title.text,book.authors.text,book.year.text,book.price.text
lst = []
with open('bookstore.xml', 'r') as file: # bookstore.xml is your xml file path
  lines = []
  flag = False
  for line in file:
    if flag or line.find('<book ')>=0:
      flag = True
      lines.append(line)
    if line.find('</book>')>=0:
      b = getBook(lines,'web','2003')
      if b: 
        lst.append(b)
        break # If you only want the first one, add a break
      flag = False
      lines = []
print (lst)

結果:

[('web', 'XQuery Kick Start', ['James McGovern', 'Per Bothner', 'Kurt Cagle', 'James Linn', 'Vaidyanathan Nagarajan'], '2003', '49.99')]

這篇關于使用 Xpath 處理較大的 XML 文件的最佳方法是什么?的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網！