|
| 1 | +import re |
| 2 | + |
| 3 | +import json |
| 4 | +import traceback |
| 5 | + |
| 6 | +from llm_api import GPT, parse_json, llm_request |
| 7 | +from langchain.document_loaders.text import TextLoader |
| 8 | +from langchain.text_splitter import LatexTextSplitter |
| 9 | +from util import multiprocess, get_cpu_count |
| 10 | + |
| 11 | + |
| 12 | +def replace_at_sentences(text): |
| 13 | + # 定义正则表达式模式，匹配@句子@(句子)格式 |
| 14 | + pattern = re.compile(r'@([^@]+)@\(([^)]+)\)') |
| 15 | + |
| 16 | + # 替换函数 |
| 17 | + def replace(match): |
| 18 | + sentence1 = match.group(1) # 获取第一个句子 |
| 19 | + sentence2 = match.group(2) # 获取第二个句子 |
| 20 | + # 返回替换后格式的字符串 |
| 21 | + return f'<span style="color:red;">{sentence1}</span>(<span style="color:orange;">{sentence2}</span>)' |
| 22 | + |
| 23 | + # 使用正则表达式的sub()方法进行查找和替换 |
| 24 | + new_text = pattern.sub(replace, text) |
| 25 | + |
| 26 | + return new_text |
| 27 | + |
| 28 | + |
| 29 | +_content_analysis_prompt = """ |
| 30 | +请你完成一个英文论文片段（latex格式）的语言表达和行文逻辑的检查任务，具体要求如下： |
| 31 | +1. 我会提供给你一篇论文的一个section，需要你帮忙检查这个section的语言表达和行文逻辑 |
| 32 | +<1> 语言表达方面：检查内容是否有语法错误、口语化表达、不够好的表达等问题 |
| 33 | +<2> 行文逻辑方面：检查内容的行文逻辑是否合理，上下文衔接是否顺畅，是否存在逻辑不通顺等问题 |
| 34 | +2. 无论是整篇文章的行文逻辑，还是单个section的行文逻辑，都是使用JSON-encoded directed acyclic graph (DAG)的结构表示 |
| 35 | +3. 整篇文章的逻辑结构为：{paper_structure} |
| 36 | +4. 这个section的逻辑结构为：{section_structure} |
| 37 | +5. 这个section的内容为：{section_content} |
| 38 | +6. 输出格式要求： |
| 39 | +<1> 输出的内容应该是markdown格式，请将latex格式的section内容转成markdown格式，删去图像，表格等不包含长文本的内容。内容总共分"section内容和标注"，以及"行文逻辑点评"两个部分。 |
| 40 | +<2> "section内容和标注"：对于你认为语言表达问题的句子，请用@符号包裹，同时在后面插入一个括号，括号中注明该句子的语言表达问题是什么 |
| 41 | +<3> "行文逻辑点评"：在输出整个标注过的section后，在最后另起一段（这段用#符号包括），用1句话简单的评价一下该段落的行文逻辑，写明是否存在问题，如果存在，问题是什么。 |
| 42 | +注意，这里我们只关注文章语言表达方面的行文逻辑问题，不要关注文章讲述的内容是否正确。 |
| 43 | +7. 下面是一个供你参考输出格式的例子： |
| 44 | +{content_analysis_example} |
| 45 | +""" |
| 46 | + |
| 47 | +content_analysis_prompt = """ |
| 48 | +Your task is to review a section of an academic paper (in LaTeX format) and check it for language proficiency and logical flow. Follow these steps: |
| 49 | +
|
| 50 | +I will provide you with a section of the paper. You need to check the section for: |
| 51 | +
|
| 52 | +Language: Look for grammatical errors, colloquial expressions, or any subpar phrases and suggest improvements. |
| 53 | +Logical Flow: Ascertain whether the content follows a reasonable sequence, whether transitions between ideas are smooth, and pinpoint any incoherence in the logic. |
| 54 | +Both the whole paper and individual sections' logical flow are depicted using JSON-encoded directed acyclic graph (DAG) structures. |
| 55 | +
|
| 56 | +The logical structure of the entire paper is: {paper_structure} |
| 57 | +
|
| 58 | +The logical structure of the specific section is: {section_structure} |
| 59 | +
|
| 60 | +The content of the section is: {section_content} |
| 61 | +
|
| 62 | +Formatting Requirements: |
| 63 | +
|
| 64 | +Present your output in Markdown format. Convert LaTeX section content to Markdown, omitting images, tables, and other non-textual elements. Divide your output into "Section Content with Annotations" and "Logical Flow Commentary." |
| 65 | +"Section Content with Annotations": Wrap any sentences with language issues in "@" symbols and immediately follow them with a parenthesis stating what the issue is. |
| 66 | +"Logical Flow Commentary": After presenting the annotated section, add a separate part summarizing the logical flow of the paragraph. State whether it's logical or if any issues are present, and describe them briefly. |
| 67 | +Please focus solely on the language-related logical flow issues for this task, without considering the factual accuracy of the content. |
| 68 | +
|
| 69 | +Here's an example for reference: |
| 70 | +
|
| 71 | +{content_analysis_example} |
| 72 | +
|
| 73 | +Ensure your response is clear and it enables gpt to understand and fulfill the task effectively. |
| 74 | +""" |
| 75 | + |
| 76 | +content_analysis_example = """ |
| 77 | +### Section Content with Annotations |
| 78 | +Planning a trip to the moon differs in no essential respect from planning a trip to the beach. \ |
| 79 | +@You hve to decide what to take along, what to leave behind.@(There is a typographical error: "hve" should be "have.". It is generally more common to see "take with you" instead of "take along.") \ |
| 80 | +Should the thermos jug go? The child's rubber horse? The dill pickles? \ |
| 81 | +@These are the sometimes fateful decisions on which the success or failure of the whole outing turns.@\ |
| 82 | +(The first issue with this sentence is the subject-verb disagreement. "These" is plural, but "is" is singular. \ |
| 83 | +The correct verb form should be "are" to match the plural subject.The last part of the sentence, "on which the success or failure of the whole outing turns," \ |
| 84 | +is a bit awkward. A smoother way to write this might be: "These decisions can sometimes be crucial, determining the success or failure of the entire outing.") |
| 85 | +Among the items they sent along, of course, was the little jointed flagpoles and the flag that could be stiffened to the breeze that did not blow. \ |
| 86 | +(It is traditional among explorers to plant the flag.) Yet the two men who stepped out on the surface of the moon were in a class by themselves and should \ |
| 87 | +have been equipped accordingly: they were of the new breed of men, those who had seen the earth whole. |
| 88 | +### Logical Flow Commentary |
| 89 | +The passage is mostly coherent and logically consistent, but the phrase "should have been equipped accordingly" is a bit abrupt without further explanation \ |
| 90 | +of what specific equipment would have been appropriate for their unique status. This leaves the reader with a question of what the author means by \ |
| 91 | +"equipped accordingly," which slightly disrupts the flow. |
| 92 | +""" |
| 93 | + |
| 94 | +if __name__ == '__main__': |
| 95 | + paper_structure = {'nodes': [{'name': 'Title', 'type': 'title', 'parents': []}, {'name': 'Abstract', 'type': 'abstract', 'parents': ['Title']}, |
| 96 | + {'name': 'Introduction', 'type': 'section', 'parents': ['Abstract']}, |
| 97 | + {'name': 'Methodology', 'type': 'section', 'parents': ['Introduction']}, |
| 98 | + {'name': 'Data Analysis', 'type': 'subsection', 'parents': ['Methodology']}, |
| 99 | + {'name': 'Image Processing', 'type': 'subsection', 'parents': ['Methodology']}, |
| 100 | + {'name': 'Gap Filling', 'type': 'subsection', 'parents': ['Methodology']}, |
| 101 | + {'name': 'Experiments', 'type': 'section', 'parents': ['Methodology']}, |
| 102 | + {'name': 'Experimental Approach', 'type': 'subsection', 'parents': ['Experiments']}, |
| 103 | + {'name': 'Results', 'type': 'subsection', 'parents': ['Experiments']}, |
| 104 | + {'name': 'Conclusions and Lessons Learned', 'type': 'section', 'parents': ['Results']}, |
| 105 | + {'name': 'Acknowledgements', 'type': 'section', 'parents': ['Conclusions and Lessons Learned']}], |
| 106 | + 'edges': [{'from': 'Title', 'to': 'Abstract'}, {'from': 'Abstract', 'to': 'Introduction'}, |
| 107 | + {'from': 'Introduction', 'to': 'Methodology'}, |
| 108 | + {'from': 'Methodology', 'to': 'Data Analysis'}, {'from': 'Methodology', 'to': 'Image Processing'}, |
| 109 | + {'from': 'Methodology', 'to': 'Gap Filling'}, {'from': 'Methodology', 'to': 'Experiments'}, |
| 110 | + {'from': 'Experiments', 'to': 'Experimental Approach'}, {'from': 'Experiments', 'to': 'Results'}, |
| 111 | + {'from': 'Results', 'to': 'Conclusions and Lessons Learned'}, |
| 112 | + {'from': 'Conclusions and Lessons Learned', 'to': 'Acknowledgements'}]} |
| 113 | + |
| 114 | + section_structure = {'nodes': [ |
| 115 | + {'name': 'Challenge Overview', |
| 116 | + 'content': 'The Urban Life and Air Pollution task at MediaEval 2022 is introduced, which required participants to predict the air quality index (AQI) value at future intervals using a variety of data sources.', |
| 117 | + 'parents': []}, {'name': 'Data Gaps Issue', |
| 118 | + 'content': 'The paper acknowledges the common issue of gaps in air quality datasets, which is particularly problematic in poorer or developing countries.', |
| 119 | + 'parents': ['Challenge Overview']}, {'name': 'Research Contribution', |
| 120 | + 'content': 'The paper outlines its contribution by describing the approach taken to address the large gaps in the air quality data encountered.', |
| 121 | + 'parents': ['Data Gaps Issue']}], 'edges': [ |
| 122 | + {'from': 'Challenge Overview', 'to': 'Data Gaps Issue'}, {'from': 'Data Gaps Issue', 'to': 'Research Contribution'}]} |
| 123 | + |
| 124 | + section_content = '\\label{sec:intro}\n\n\\begin{comment}\n% Just in case we want some form of Intro\nAccording to the World Health Organisation (WHO), 91\\% of the world\'s population reside in conditions where WHO\'s air quality guidelines levels were not met \\cite{organizacion2021global}. This report on 2016 also showed that ambient (outdoor) air pollution in both cities and rural areas was estimated to cause 4.2 million premature deaths worldwide. The research concluded that policies and investments supporting cleaner transport, energy-efficient homes, power generation, industry and better municipal waste management would would be crucial to the reduction of outdoor air pollution. In a separate report, it is estimated that air pollution globally accounts for roughly seven million premature deaths a year \\cite{Gar21}, where it was again stated that the majority of those deaths are caused by outdoor air pollution with the rest generally attributed to poor air quality from indoor cooking. While the majority of these deaths occur in developing countries, with China and India accounting for roughly 50\\%, developed countries also have a problem with deaths resulting from air pollution.\nIn this research, the focus will mostly be on the modelling of concentrations in particulate matter - tiny particles in the air generated both by natural processes and human activity. These particles are generally \n12 categorised (in the public health domain) by their diameter; fine particles with diameter less than 2.5 $\\mu$m are referred to as "PM2.5" and coarse particles with diameter between 2.5 and 10 $\\mu$m are referred to as "PM10".\\\\\n\\end{comment}\n\nThe Urban Life and Air Pollution task at MediaEval 2022 required participants to predict the air quality index (AQI) value at +1, +5 and +7 days using an archive of air quality, weather and images from 16 CCTV cameras, one image taken every 60 seconds \\cite{UA22}. Participating groups were required to download the data from online sources for local processing.\nGaps in air quality datasets are common with the problem exacerbated for data gathered in poorer or developing countries \\cite{PINDER2019116794, Falge2001, Hui2004, Moffat2007, Kim2020}. In this paper we describe how we addressed the very large gaps in data that we encountered in the data we downloaded.\n\n\n%\\section{Related Work}\n%\\label{sec:rr}\n%.. a snapshot of how our graph? technique has been used for solving other problems.\n\n% Mark: I restructured this to stop duplication of text' |
| 125 | + |
| 126 | + request = content_analysis_prompt.format(paper_structure=paper_structure, section_structure=section_structure, section_content=section_content, |
| 127 | + content_analysis_example=content_analysis_example) |
| 128 | + reply = llm_request(request) |
| 129 | + print(reply) |
| 130 | + reply = replace_at_sentences(reply) |
| 131 | + print(reply) |
| 132 | + |
| 133 | + |
0 commit comments