RAPTOR: RECURSIVE ABSTRACTIVE PROCESSING FOR TREE-ORGANIZED RETRIEVAL
Raptor:用于树组织检索的递归抽象处理
https://github.com/parthsarthi03/raptorABSTRACT
Retrieval-augmented language models can better adapt to changes in world state and incorporate long-tail knowledge. However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic understanding of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing a tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over traditional retrieval-augmented LMs on several tasks. On question-answering tasks that involve complex, multi-step reasoning, we show state-of-the-art results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy.
检索增强的语言模型能够更好地适应世界状态的变化，并融入长尾知识。然而，大多数现有方法仅从检索语料库中检索短的连续块，限制了对整个文档上下文的整体理解。我们引入了递归嵌入、聚类和总结文本块的新方法，从下向上构建了一个具有不同总结级别的树。在推理时，我们的RAPTOR模型从这棵树中检索，在不同抽象级别上集成跨冗长文档的信息。对照实验表明，与传统的检索增强LMs相比，递归摘要检索在若干任务上有显著的改进。在涉及复杂、多步骤推理的问答任务上，我们展示了最先进的结果;例如，通过将RAPTOR检索与GPT-4结合使用，我们可以将质量基准上的最佳性能提高20%的绝对准确性。
1 INTRODUCTION
Large Language Models (LLMs) have emerged as transformative tools showing impressive performance on many tasks. With the growing size of LLMs, they can serve standalone as very effective knowledge stores, with facts encoded within their parameters (Petroni et al, 2019; Jiang et al, 2020; Talmor et al, 2020; Rae et al, 2021; Hoffmann et al, 2022; Chowdhery et al, 2022; Bubeck et al, 2023; Kandpal et al, 2023) and models can be further improved with fine-tuning on downstream tasks (Roberts et al, 2020). Nevertheless, even a large model does not contain sufficient domainspecific knowledge for particular tasks and the world continues to change, invalidating facts in the LLM. Updating the knowledge of these models through additional fine-tuning or editing is difficult, particularly when dealing with vast text corpora (Lewis et al, 2020; Mitchell et al, 2022). An alternative approach, pioneered in open domain question answering systems (Chen et al, 2017; Yu et al, 2018), is to index large quantities of text, after splitting it into chunks (paragraphs), in a separate information retrieval system. Retrieved information is then presented to the LLM along with the question as context (“retrieval augmentation”, Lewis et al, 2020; Izacard et al, 2022; Min et al, 2023; Ram et al, 2023), making it easy to provide a system with current knowledge particular to some domain and enabling easy interpretability and provenance tracking, whereas the parametric knowledge of LLMs is opaque and difficult to trace back to its source (Akyurek et al, 2022).
Nevertheless, existing retrieval-augmented approaches also have flaws. The one we tackle is that most existing methods retrieve only a few short, contiguous text chunks, which limits their ability to represent and leverage large-scale discourse structure. This is particularly relevant for thematic questions that require integrating knowledge from multiple parts of a text, such as understanding an entire book, as in the NarrativeQA dataset (Kocisk ˇ y et al ` , 2018). Consider the fairy tale of Cinderella, and the question “How did Cinderella reach her happy ending?”. The top-k retrieved short contiguous texts will not contain enough context to answer the question.
To address this, we design an indexing and retrieval system that uses a tree structure to capture both high-level and low-level details about a text. As shown in Figure 1, our system, RAPTOR, clusters chunks of text, generates text summaries of those clusters, and then repeats, generating a tree from the bottom up. This structure enables RAPTOR to load into an LLM’s context chunks representing the text at different levels so that it can effectively and efficiently answer questions at different levels.
Our main contribution is the idea of using text summarization to allow retrieval augmentation of
context at different scales, and to show its effectiveness in experiments on collections of long documents. Controlled experiments with three language models (UnifiedQA (Khashabi et al, 2020),
GPT-3 (Brown et al, 2020) and GPT-4 (OpenAI, 2023)) show that RAPTOR outperforms current
retrieval augmentation. Moreover, RAPTOR coupled with GPT-4, and sometimes even with UnifiedQA, gives new state-of-the-art results on three QA tasks: free text response questions on books
and movies (NarrativeQA, Kocisk ˇ y et al ` 2018), full-text NLP papers (QASPER, Dasigi et al 2021),
and multiple-choice questions based on medium-length passages (QuALITY, Pang et al 2022).
大型语言模型(llm)已经成为变革性的工具，在许多任务中表现出令人印象深刻的性能。随着法学硕士规模的不断扩大，它们可以独立作为非常有效的知识存储，并在其参数中编码事实(Petroni等人，2019;Jiang等，2020;Talmor et al, 2020;Rae等，2021;Hoffmann et al, 2022;Chowdhery等人，2022;Bubeck等，2023;Kandpal等人，2023)，并且可以通过对下游任务进行微调来进一步改进模型(Roberts等人，2020)。然而，即使是一个大型模型也不包含足够的特定领域知识来完成特定的任务，而且世界在不断变化，使法学硕士中的事实失效。通过额外的微调或编辑来更新这些模型的知识是困难的，特别是在处理大量文本语料库时(Lewis等人，2020;Mitchell et al, 2022)。在开放领域问答系统中开创的另一种方法(Chen等人，2017;Yu等人，2018)，是将大量文本分成块(段落)后，在单独的信息检索系统中对其进行索引。然后将检索到的信息与问题一起作为上下文呈现给法学硕士(“检索增强”，Lewis等人，2020;Izacard等人，2022;Min等，2023;Ram等人，2023)，这使得为系统提供特定于某个领域的当前知识变得容易，并且易于可解释性和来源跟踪，而法学硕士的参数化知识是不透明的，难以追溯到其来源(Akyurek等人，2022)。
然而，现有的检索增强方法也存在缺陷。我们解决的一个问题是，大多数现有的方法只检索几个简短的、连续的文本块，这限制了它们表示和利用大规模话语结构的能力。这对于需要整合文本多个部分的知识的主题问题尤其相关，例如理解整本书，如在NarrativeQA数据集中(Kocisk + y等人，2018)。想想灰姑娘的童话故事，以及“灰姑娘是如何得到幸福结局的?”这个问题。检索到的前k个连续短文本将不包含足够的上下文来回答问题。
为了解决这个问题，我们设计了一个索引和检索系统，该系统使用树形结构来捕获关于文本的高级和低级细节。如图1所示，我们的系统RAPTOR对文本块进行聚类，生成这些聚类的文本摘要，然后重复，从下向上生成树。这种结构使RAPTOR能够将表示不同层次文本的上下文块加载到法学硕士的上下文块中，以便它能够有效地回答不同层次的问题。
我们的主要贡献是使用文本摘要的想法，以允许检索增强
上下文在不同尺度下，并在长文档集合实验中显示其有效性。三种语言模型的对照实验(UnifiedQA (Khashabi et al, 2020)，
GPT-3 (Brown et al, 2020)和GPT-4 (OpenAI, 2023))表明RAPTOR优于当前检索增大。此外，RAPTOR与GPT-4结合，有时甚至与UnifiedQA结合，在三个QA任务上提供了新的最先进的结果:书籍上的自由文本回答问题
电影(nartiveqa, Kocisk等人2018年)，全文NLP论文(QASPER, Dasigi等人2021年)，
以及基于中等长度文章的多项选择题(QuALITY, Pang et al . 2022)。
Figure 1: Tree construction process: RAPTOR recursively clusters chunks of text based on their vector embeddings and generates text summaries of those clusters, constructing a tree from the bottom up. Nodes clustered together are siblings; a parent node contains the text summary of that cluster.
图1:树构建过程:RAPTOR基于它们的向量嵌入递归地对文本块进行聚类，并生成这些聚类的文本摘要，从下向上构建树。聚集在一起的节点是兄弟姐妹;父节点包含该集群的文本摘要。
2 RELATED WORK
Why Retrieval?
Recent advances in hardware and algorithms have indeed expanded the context lengths that models can handle, leading to questions about the need for retrieval systems (Dai et al, 2019; Dao et al, 2022; Liu et al, 2023). However, as Liu et al (2023) and Sun et al (2021) have noted, models tend to underutilize long-range context and see diminishing performance as context length increases, especially when pertinent information is embedded within a lengthy context.
Moreover, practically, use of long contexts is expensive and slow. This suggests that selecting the most relevant information for knowledge-intensive tasks is still crucial
为什么检索?
硬件和算法的最新进展确实扩大了模型可以处理的上下文长度，从而引发了关于检索系统需求的问题(Dai等人，2019;Dao等，2022;Liu et al, 2023)。然而，正如Liu等人(2023)和Sun等人(2021)所指出的那样，模型往往没有充分利用远程上下文，并且随着上下文长度的增加，特别是当相关信息嵌入到冗长的上下文中时，模型的性能会下降。
此外，实际上，使用长上下文既昂贵又缓慢。这表明，为知识密集型任务选择最相关的信息仍然至关重要
Retrieval Methods
Retrieval-augmented language models (RALMs) have seen improvements in various components: the retriever, the reader, and end-to-end system training. Retrieval methods have transitioned from traditional term-based techniques like TF-IDF (Sparck Jones ¨ , 1972) and BM25 (Robertson et al, 1995; Roberts et al, 2020) to deep learning–based strategies (Karpukhin et al, 2020; Khattab & Zaharia, 2020; Sachan et al, 2023). Some recent work proposes using large language models as retrievers due to their ability to memorize extensive knowledge (Yu et al, 2022; Sun et al, 2022). Research on the reader component includes Fusion-in-Decoder (FiD) (Izacard & Grave, 2022), which employs both DPR and BM25 for retrieval and processes passages independently in the encoder and RETRO (Borgeaud et al, 2022; Wang et al, 2023), which utilizes cross-chunked attention and chunkwise retrieval to generate text grounded on retrieved context.
End-to-end system training work includes Atlas (Izacard et al, 2022), which fine-tunes an encoderdecoder model in conjunction with the retriever; REALM (Guu et al, 2020), a bidirectional, masked LM fine-tuned for open-domain question answering; and RAG (Retrieval-Augmented Generation) (Lewis et al, 2020), which integrates pre-trained sequence-to-sequence models with a neural retriever. Min et al (2021) introduced Joint Passage Retrieval (JPR) model which uses a treedecoding algorithm to handle passage diversity and relevance in multi-answer retrieval. Dense Hierarchical Retrieval (DHR) and Hybrid Hierarchical Retrieval (HHR) represent advancements in retrieval accuracy by combining document and passage level retrievals and integrating sparse and dense retrieval methods, respectively (Liu et al, 2021; Arivazhagan et al, 2023).
检索方法
检索增强语言模型(ralm)在各个组件中都得到了改进:检索器、阅读器和端到端系统训练。检索方法已经从传统的基于术语的技术，如TF-IDF (Sparck Jones¨，1972)和BM25 (Robertson et al, 1995;Roberts等人，2020)到基于深度学习的策略(Karpukhin等人，2020;哈塔卜和扎哈里亚，2020;Sachan et al, 2023)。最近的一些研究建议使用大型语言模型作为检索器，因为它们具有记忆广泛知识的能力(Yu et al .， 2022;Sun et al, 2022)。对读取器组件的研究包括融合解码器(FiD) (Izacard & Grave, 2022)，它使用DPR和BM25进行检索，并在编码器和RETRO中独立处理段落(Borgeaud et al .， 2022;Wang et al .， 2023)，它利用交叉分块注意和分块检索来生成基于检索上下文的文本。
端到端系统训练工作包括Atlas (Izacard等人，2022)，它与检索器一起微调编码器-解码器模型;REALM (Guu et al .， 2020)，一种双向、掩膜的LM，用于开放域问答;以及RAG(检索增强生成)(Lewis et al, 2020)，它将预先训练的序列到序列模型与神经检索器集成在一起。Min等人(2021)引入了联合通道检索(JPR)模型，该模型使用树解码算法来处理多答案检索中的通道多样性和相关性。密集层次检索(DHR)和混合层次检索(HHR)分别通过结合文档级和通道级检索以及整合稀疏和密集检索方法来提高检索精度(Liu et al .， 2021;Arivazhagan等人，2023)。
Despite a diversity in methods, the retrieving components of models predominantly rely on standard approaches, i.e., chunking corpora and encoding with BERT-based retrievers. Although this approach is widely adopted, Nair et al (2023) highlights a potential shortcoming: contiguous segmentation might not capture the complete semantic depth of the text. Reading extracted snippets from technical or scientific documents may lack important context making them difficult to read or even misleading. (Cohan & Goharian, 2017; Newman et al, 2023; Zhang et al, 2023).
尽管方法多种多样，但模型的检索组件主要依赖于标准方法，即语料库分块和基于bert的检索器编码。尽管这种方法被广泛采用，但Nair等人(2023)强调了一个潜在的缺点:连续分割可能无法捕获文本的完整语义深度。阅读从技术或科学文件中摘录的片段可能缺乏重要的上下文，使其难以阅读，甚至具有误导性。(Cohan & Goharian, 2017;Newman等人，2023;Zhang et al, 2023)。
Recursive summarization as Context
Summarization techniques provide a condensed view of documents, enabling more focused engagement with the content (Angelidis & Lapata, 2018). The summarization/snippet model by Gao et al (2023) uses summarizations and snippets of passages, which improves correctness on most datasets but can sometimes be a lossy means of compression.
The recursive-abstractive summarization model by Wu et al (2021) employs task decomposition to summarize smaller text chunks, which are later integrated to form summaries of larger sections.
While this method is effective for capturing broader themes, it can miss granular details. LlamaIndex (Liu, 2022) mitigates this issue by similarly summarizing adjacent text chunks but also retaining intermediate nodes thus storing varying levels of detail, keeping granular details. However, both methods, due to their reliance on adjacency for grouping or summarizing adjacent nodes, may still overlook distant interdependencies within the text, which we can find and group with RAPTOR.
作为上下文的递归汇总
摘要技术提供了文档的浓缩视图，使人们能够更集中地参与内容(Angelidis & Lapata, 2018)。Gao等人(2023)的摘要/片段模型使用段落的摘要和片段，这提高了大多数数据集的正确性，但有时可能是一种有损的压缩方式。
Wu等人(2021)的递归-抽象摘要模型采用任务分解来总结较小的文本块，然后将其集成为较大部分的摘要。
虽然这种方法对于捕获更广泛的主题是有效的，但它可能会错过颗粒状的细节。LlamaIndex (Liu, 2022)通过类似地总结相邻的文本块来缓解这个问题，但也保留中间节点，从而存储不同级别的细节，保持颗粒细节。然而，这两种方法，由于它们依赖邻接性来分组或总结相邻节点，可能仍然忽略了文本中遥远的相互依赖关系，我们可以找到并使用RAPTOR进行分组。
3 METHODS
Overview of RAPTOR
Building on the idea that long texts often present subtopics and hierarchical structures (Cao & Wang, 2022; Dong et al, 2023b), RAPTOR addresses the issue of semantic depth and connection in reading by building a recursive tree structure that balances broader thematic comprehension with granular details and which allows nodes to be grouped based on semantic similarity not just order in the text.
Construction of the RAPTOR tree begins with segmenting the retrieval corpus into short, contiguous texts of length 100, similar to traditional retrieval augmentation techniques. If a sentence exceeds the 100-token limit, we move the entire sentence to the next chunk, rather than cutting it mid-sentence.
This preserves the contextual and semantic coherence of the text within each chunk. These texts are then embedded using SBERT, a BERT-based encoder (multi-qa-mpnet-base-cos-v1) (Reimers & Gurevych, 2019). The chunks and their corresponding SBERT embeddings form the leaf nodes of our tree structure.
To group similar text chunks, we employ a clustering algorithm. Once clustered, a Language Model is used to summarize the grouped texts.
扫一扫
1946
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
