問題描述

我需要在 CSV 格式文件中搜索超過 PB 的數(shù)據(jù).使用 LUCENE 建立索引后，索引文件的大??小是原始文件的兩倍.是否可以減少索引文件的大??小???HADOOP中如何分發(fā)LUCENE索引文件以及如何在搜索環(huán)境中使用?還是有必要，我應(yīng)該使用 solr 來分發(fā) LUCENE 索引嗎???我的要求是對(duì) PB 的文件進(jìn)行即時(shí)搜索....

I need to search over petabyte of data in CSV formate files. After indexing using LUCENE, the size of the indexing file is doubler than the original file. Is it possible to reduce the indexed file size??? How to distribute LUCENE index files in HADOOP and how to use in searching environment? or is it necessary, should i use solr to distribute the LUCENE index??? My requirement is doing instant search over petabyte of files....

推薦答案

任何體面的現(xiàn)成搜索引擎(如 Lucene)都應(yīng)該能夠提供超過您擁有的數(shù)據(jù)大小的搜索功能.您可能需要預(yù)先做一些工作來設(shè)計(jì)索引并配置搜索的工作方式，但這只是配置.

Any decent off the shelf search engine (like Lucene) should be able to provide search functionality over the size of data you have. You may have to do a bit of work up front to design the indexes and configure how the search works, but this is just config.

您不會(huì)立即獲得結(jié)果，但您也許能夠很快獲得結(jié)果.速度可能取決于您的設(shè)置方式以及運(yùn)行的硬件類型.

You won't get instant results but you might be able to get very quick results. The speed will probably depend on how you set it up and what kind of hardware you run on.

您提到索引大于原始數(shù)據(jù).這是可以預(yù)料的.索引通常包括某種形式的非規(guī)范化.索引的大小通常是與速度的權(quán)衡；提前對(duì)數(shù)據(jù)進(jìn)行切片和切塊的方式越多，找到參考的速度就越快.

You mention that the indexes are larger than the original data. This is to be expected. Indexing usually includes some form of denormalisation. The size of the indexes is often a trade off with speed; the more ways you slice and dice the data in advance, the quicker it is to find references.

最后你提到了分發(fā)索引，這幾乎肯定不是你想做的事情.分發(fā)許多 PB 數(shù)據(jù)的實(shí)用性非常令人生畏.您可能想要的是將索引放在某處的大型計(jì)算機(jī)上并提供數(shù)據(jù)搜索服務(wù)(將查詢帶到數(shù)據(jù)中，不要將數(shù)據(jù)帶到查詢中).

Lastly you mention distributing the indexes, this is almost certainly not something you want to do. The practicalities of distributing many petabytes of data are pretty daunting. What you probably want is to have the indexes sat on a big fat computer somewhere and provide search services on the data (bring the query to the data, don't take the data to the query).

這篇關(guān)于即時(shí)搜索 PB 級(jí)數(shù)據(jù)的文章就介紹到這了，希望我們推薦的答案對(duì)大家有所幫助，也希望大家多多支持html5模板網(wǎng)！