<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Philipp Schreiber on Medium]]></title>
        <description><![CDATA[Stories by Philipp Schreiber on Medium]]></description>
        <link>https://medium.com/@pcschreiber1?source=rss-6cbfe67cb12c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MQcH3AXkTzhUmPFV-VLsgA.jpeg</url>
            <title>Stories by Philipp Schreiber on Medium</title>
            <link>https://medium.com/@pcschreiber1?source=rss-6cbfe67cb12c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 11:52:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pcschreiber1/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ML for Macroeconomics #1]]></title>
            <link>https://medium.com/data-science/ml-for-macroeconomics-1-cb7fccf93ccf?source=rss-6cbfe67cb12c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cb7fccf93ccf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[economics]]></category>
            <category><![CDATA[lasso-regression]]></category>
            <category><![CDATA[variable-selection]]></category>
            <dc:creator><![CDATA[Philipp Schreiber]]></dc:creator>
            <pubDate>Sun, 03 Apr 2022 15:16:16 GMT</pubDate>
            <atom:updated>2022-04-06T17:37:04.131Z</atom:updated>
            <content:encoded><![CDATA[<h4>Variable Selection with Random Forests in low-quality data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CzgFVGVJmV2bQEsn" /><figcaption>Photo by <a href="https://unsplash.com/@carlosaranda?utm_source=medium&amp;utm_medium=referral">carlos aranda</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>While <em>Machine Learning</em> (ML) has taken many applied sciences by storm, economics’ emphasis on causality has rendered it difficult to find relevant applications. Over the past years, Susan Athey has successfully pioneered various applications in Microeconomics, however, students of Macroeconomics have yet to find similar role models to emulate. That is why with this article, I want to use the issue of <em>variable selection</em> to illustrate a smaller area where ML can easily supplement classical techniques.</p><p>Given the aggregate nature of Macro analyses, such as cross-country studies, high-dimensional (aka “Big”) data forms the most common scenario researchers face. But while the <em>Least Absolute Shrinkage and Selection Operator</em> (<strong>Lasso</strong>) has long been popular among economists, non-parametric alternatives like <em>the Random Forest</em> (<strong>RF</strong>) have not enjoyed similar attention. Therefore, together we’re going to <strong><em>i)</em></strong> conduct a simulation study to compare the behaviour of Lasso and RF under low-quality data and <strong><em>ii)</em></strong> apply them to Sala-I-Martin’s famous “millions” data set of economic growth.</p><p><strong><em>Disclaimer</em></strong><em>: This article is based on a course project which is available on </em><a href="https://github.com/pcschreiber1/LASSO_RF_for_Macro"><em>GitHub</em></a><em>. All the material, further explanations and additional references can be found there.</em></p><h3>Refresher on Variable Selection</h3><p>Variable selection — known as “feature selection” in the ML literature — is one of the most common approaches to dealing with high-dimensional data, i.e. data where the number of predictors is high relative to the number of observations. The idea is that, if the researcher knows that only a subset of all covariates influences the dependent variable, correctly identifying these can alleviate some of the “curses of dimensionality”, such as prediction accuracy (Fan and Lv, 2009). In practice, this is called the <em>sparsity hypothesis</em>.</p><p>The Lasso and applications of random forests are then two examples of the variety of variable selection techniques that have been developed over the years. Importantly, like always in statistics, there exists no single optimal procedure. As Hastie et al. (2017) explain:</p><blockquote>Different procedures have different operating characteristics, i.e., give rise to different bias-variance tradeoffs as we vary their respective tuning parameters. In fact, depending on the problem setting, the bias-variance tradeoff provided by best-subset selection may be more or less useful than the tradeoff provided by the lasso. (p.582)</blockquote><p>Now, what Hastie et al. are referring to is that “best-subset selection” naturally has a very low bias, but, in noisy data, methods such as the Lasso can often deliver better results because they trade-in some bias for a substantial reduction in variance. Indeed, this type of tradeoff is the central motivation of this article, as we compare the Lasso to another technique entailing efficient variance reduction: RFs. Before proceeding to the simulation study, let me also offer a very brief description of Lasso and RF.</p><p><strong>The Lasso</strong> belongs to the class of Shrinkage Methods with the special property that, for an appropriate choice of the penalty term, coefficients are shrunken exactly to zero. Since the penalty term shrinks the size of all coefficients, a <strong>relaxed Lasso</strong> has been developed, where control of non-zero coefficients is separated from overall shrinkage.</p><p>The defining feature of <strong>RFs</strong> is the aggregation of decorrelated decision trees, which is particularly effective for the reduction in variance. While RF, unlike the Lasso, does not naturally perform variable selection, Genuer et al. (2010) developed a stepwise ascending variable introduction strategy for this purpose and implemented it in the very good <a href="https://cran.r-project.org/web/packages/VSURF/VSURF.pdf"><strong>VSURF</strong></a> package, available on the R-CRAN library.</p><h3>Simulation Study</h3><p>Given the dependence of the bias-variance trade-off on the problem structure, it is key for our simulation study to emulate our real-world application, i.e. a cross-country study of economic growth. Besides <em>a (very) small sample size </em>(there exist roughly 100 countries in the world), this critically relates to the <strong>quality of data</strong> we have for any given year: censuses are only conducted infrequently, are not globally standardized, strongly rely on data imputations — issues which are exacerbated for less prosperous countries.</p><p>We make data quality the focal point of our comparison and instrumentalise it through the <em>Signal-to-Noise Ratio </em>(<strong>SNR</strong>). Intuitively, if there is a lot more noise (error) than signal (our explanatory variables), it becomes difficult to identify the effect of the signal. We follow Hasties et al. (2017) and consider ten values of SNR ranging from 0.05 to 6 on a log scale — for comparison the corresponding values of the “Proportion of Variance Explained” are reported, as well. Importantly, as the authors emphasise, studies are often too optimistic about signal clarity and argue that, in the real world, observational data tends to have <strong>log(SNR) &lt; 1</strong>!</p><figure><img alt="Log Scale of Signal-to-Noise Ratio" src="https://cdn-images-1.medium.com/max/546/1*LAbyvJ0HUQsarCYZK_WOUw.png" /><figcaption>Image by author.</figcaption></figure><p>As a result of this SNR scale, a simulation study calculates 10 x #sim synthetic data sets and applies the Lasso, relaxed Lasso and RF to each of them. The data sets are sparse so that only 5 out of 50 covariates influence the outcome. A Toeplitz matrix is used for the variance-covariance Matrix, so that variable’s relative position determines the correlation structure. Note that we choose a linear DGP, which needs to be taken into account when comparing the results of parametric to non-parametric methods. Finally, the number of simulations is set to 100.</p><p>Now that we are familiar with the conceptual setup of the simulation studies, we can have a look at the results. If you’re interested in the code or more detailed explanations, including on the control of SNR, I cordially invite you to check out the original project on <a href="https://github.com/pcschreiber1/LASSO_RF_for_Macro">GitHub</a>.</p><h4>Simulation 1 — The Baseline</h4><figure><img alt="Results Dashboard of Simulation 1" src="https://cdn-images-1.medium.com/max/1024/1*u3lUXp0KICchab0yH3AgKA.png" /><figcaption>Image by author</figcaption></figure><ul><li><strong>RF consistently closer to true sparsity, with very low variance at all levels of SNR</strong></li><li><strong>Good retention frequency of Lasso methods a by-product of overly dense estimates and hides the number of null-models</strong></li></ul><p>In the baseline simulation, the general degree of correlation is moderate (0.5 to direct neighbours) without special correlation between the five true predictors. The graph on the top left plots the retention frequency, i.e. how many true predictors were identified. Not surprisingly, all three models become better at identifying the significant variables when the signal becomes clearer. However, the relative underperformance of RF between SNR 0.25 and 1.22 is misleading: looking at the graph at the top right illustrates the number of nonzero coefficients predicted by the model, in other words, its estimated density. Here we can see that the high retention frequency of the Lasso methods is a product of their strong over-estimation of density.</p><p>Indeed, while the relaxed Lasso, thanks to its ability to separate shrinkage and control of sparsity, eventually moves closer to the true sparsity, for the realistic levels of SNR it still selects twice as many covariates, on average. Looking at the graph at the bottom right, which zooms in on this “real-world window”, we see that the RF density is firmly centred around the true number, while at the lower end of SNR even the relaxed Lasso selects many null models (i.e. where the model identifies no covariates as significant).</p><p>Finally, as is customary, the MSE is also reported (bottom left). We can see that for low levels of SNR, the MSE is astronomical and rapidly decreases with higher data quality. Importantly, we might wonder why a non-parametric method apparently outperforms linear methods despite the fact that these are correctly specified. However, for RF we only report the VSURF Out-of-Bag MSE, which is biased downwards because ranking and selection are based on the same observations (Genuer et al., 2015). The main takeaway from this graph is then that RF and Lasso are both similarly sloped in their MSE reduction.</p><h4>Simulation 2— High Covariance</h4><figure><img alt="Results Dashboard of Simulation 2" src="https://cdn-images-1.medium.com/max/1024/1*yzi-bhdR_nrwlXznjVvziw.png" /><figcaption>Image by author</figcaption></figure><ul><li><strong>RF very robust to high collinearity, unlike Lasso</strong></li></ul><p>Since high collinearity is a common issue in macro-economic analyses, better understanding the methods’ relative performance will be important for interpreting the results of the application. That is why, in the second simulation study, we make all true predictors neighbours, thereby giving them a relatively high-correlation structure. Importantly, given the nature of our Toeplitz matrix, this also means that there now also exists a set of insignificant variables which is strongly correlated to <em>all</em> true predictors. This set-up targets the well-known weakness of the Lasso of choosing the correct variables if correlation is high (see, e.g. James et al., 2013). In contrast, for the RF, Genuer et al. (2010) find that the variable importance ordering is very robust to the presence of collinearity.</p><p>Indeed, the performance of RF is nearly unchanged from the first study, where the violin plot actually indicates a reduction in variance at the low level of the realistic SNR window. On the other hand, we see that the standard Lasso generated more null models and has a lower increase in retention frequency. Similarly, the relaxed Lasso also generates many null models but also demonstrates cases where it inflates models with up to 20 variables. As a result, even though both its retention frequency as well as average density closely match the RF, their performance only converges for high levels of SNR.</p><p><em>The </em><a href="https://github.com/pcschreiber1/LASSO_RF_for_Macro"><em>original project</em></a><em> contains several additional studies illustrating further aspects of the techniques’ bias-variance trade-offs.</em></p><h3>Application</h3><p>Armed with these insights, we now bring the three techniques to the task with the famous “millions” data set from Sala-I-Martin (1997) [<a href="http://www.columbia.edu/~xs23/data/millions.htm">available here</a>]. It consists of 63 macro variables for a cross-section of 134 countries in the 1970s, matching the observations-to-variables ratio of our simulation studies. After standardizing and replacing missing values with the sample mean, we are left with 119 countries and 62 predictor variables.</p><p>With the iconic title “<em>I just ran two million regressions</em>”, Sala-I-Martin (1997) set out to develop a faster and more flexible method for identifying relevant macro-economic growth variables than the “extreme bounds test”, which was very popular at the time. He himself proposed to estimate and select variables based on the cumulative distribution function, which we also include in the results table below.</p><figure><img alt="Millions Data Results Table" src="https://cdn-images-1.medium.com/max/608/1*wqUIq7ZKWx1YUDvDiQHZyA.png" /><figcaption>Millions Data Results Table (Image by author).</figcaption></figure><p>Between the four methods, 20 different variables are selected as predictors of 1970s GDP growth rates. While, at first glance, we might find this substantial disagreement surprising, closer inspection of the estimation results points to the presence of high collinearity (discussed in detail in the <a href="https://github.com/pcschreiber1/LASSO_RF_for_Macro">original project</a>). Remembering the results of the second simulation study, while the rate of recovery will be relatively low for all four methods, <em>we should be comparatively confident in the density estimated by the RF</em>. As such, while the parsimony of the CDF and relaxed Lasso might seem appealing, this suggests that their estimates are too sparse.</p><p>For the sake of comparison, the results from the Lasso methods are ranked by the absolute value of the estimated coefficient. We can see that four variables are not only shared between all or Lasso&amp;RF (marked in green) but are also commonly assigned a high degree of importance: although the random forest approach only assigns a minor role to religion, it also counts “Equipment Investment” (i.e. investment into mechanization) among the group of very important drivers of economic growth. The same is true for the measure of the openness of the economy, which is also given very high importance by all four methods. In contrast, Sala-I-Martin’s CDF method fails to select primary school enrollment, which is assigned primacy by the Lasso methods and ranks third for the RF. However, the apparent instability of the results cautions against placing too much faith in these ranks: for the Lasso, the variables are so difficult to select that randomness in the cross-validation process caused the inclusion of “Life Expectancy” in the relaxed Lasso but not the standard Lasso.</p><p>Taking a step back, we see that there exists a small set of (relatively) strong predictors, but many weak ones. From a macroeconomic perspective, what stands out when comparing the four subset selections is the lack of political variables in favour of social and economic ones: while the CDF, standard Lasso and random forest each select one political variable, there is no congruence. Interestingly, even “Revolution and Coups”, which relates to violent conflict, is only selected by the standard Lasso and still only assigned the lowest rank — running contrary to the very successful literature on institutional economics.</p><h3>Conclusion</h3><p>Thank you for staying to the end. I hope this article gave you a hands-on example of a simple application of ML to Macroeconomics. While, of course, our simple example with the “Millions” data set does not warrant any causal interpretations — an obvious next step would be time-series data — we were able to see that variable selection by random forest can be more robust to some of the problems of empirical Macroeconomics. If you can think of more macroeconomic applications or have even worked on some, I’d love to hear them, so please feel free to share them below.</p><p><em>Stay safe &amp; stay in touch!</em></p><h4>References</h4><ul><li><strong>Fan, J. and Lv, J. (2009)</strong>. A Selective Overview of Variable Selection in High Dimensional Feature Space (Invited Review Article). <em>Statistica Sinica, 20 (1), </em>pp.101–148.</li><li><strong>Genuer, R., Poggi, J.-M. and Tuleau-Malot, C. (2010)</strong>. Variable selection using random forests. <em>Pattern Recognition Letters, 31 (14)</em>, pp.2225–2236. [Online]. Available at: <a href="https://doi:10.1016/j.patrec.2010.03.014">https://doi:10.1016/j.patrec.2010.03.014</a>.</li><li><strong>Genuer, R., Poggi, J.-M. and Tuleau-Malot, C. (2015)</strong>. VSURF: An R Package for Variable Selection Using Random Forests. <em>The R Journal, 7 (2)</em>, pp.19–33. [Online]. Available at: <a href="https://doi:10.32614/RJ-2015-018">https://doi:10.32614/RJ-2015-018</a>.</li><li><strong>Hastie, T., Tibshirani, R. and Tibshirani, R. (2017)</strong>. Best Subset, Forward Stepwise or Lasso? Analysis and Recommendations Based on Extensive Comparisons. <em>Statistical Science, 35 (4)</em>, pp.579–592. [Online]. Available at: <a href="https://doi:10.1214/19-STS733">https://doi:10.1214/19-STS733</a>.</li><li><strong>James, G. et al. (2013)</strong>. <em>An introduction to statistical learning</em>. Springer.</li><li><strong>Sala-I-Martin, X. X. (1997)</strong>. I Just Ran Two Million Regressions. <em>The American Economic Review, 87 (2)</em>, pp.178–183.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb7fccf93ccf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/ml-for-macroeconomics-1-cb7fccf93ccf">ML for Macroeconomics #1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Translate long PDF-Reports in Python]]></title>
            <link>https://medium.com/data-science/translate-long-pdf-reports-in-python-eab3be08ceb4?source=rss-6cbfe67cb12c------2</link>
            <guid isPermaLink="false">https://medium.com/p/eab3be08ceb4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pdf-extraction]]></category>
            <category><![CDATA[pdf-translation]]></category>
            <category><![CDATA[pdfminer]]></category>
            <category><![CDATA[official-document]]></category>
            <dc:creator><![CDATA[Philipp Schreiber]]></dc:creator>
            <pubDate>Fri, 04 Mar 2022 15:58:50 GMT</pubDate>
            <atom:updated>2023-01-30T18:05:50.480Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Automatically extract and translate a complete German Central Bank report for free</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cHe8bDeb0GWnFOPR" /><figcaption>Photo by <a href="https://unsplash.com/@mbaumi?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For work, I recently had to translate many old Central Bank reports from various OECD countries. While, luckily, gibberish online translation is a thing of the past, common manual solutions are often not viable when working with many long documents. A multitude of useful Python packages exist to help with this task and are introduced in a variety of excellent existing <a href="https://towardsdatascience.com/pdf-text-extraction-in-python-5b6ab9e92dd">articles</a>. However, when faced with this task I found that the commonly used examples are often too <em>stylised</em> and many of the established tools are <em>no longer being maintained</em> in favour of community-built follow-up projects.</p><p>That is why, with this article I want to <strong>1)</strong> provide a real-world example of PDF translation and <strong>2)</strong> give an update on the best packages to use.</p><h3><strong>2 + 1 Tasks</strong></h3><p>So, together we’re going to translate a Central Bank Report, which — just like the code — you can find on my <a href="https://github.com/pcschreiber1/PDF_Extraction-Translation">Git repository</a>. To get started, we need a clear idea of what it is that we want to do. In our case, we need to somehow extract the content of a pdf, translate it and, then, (potentially) bring it into a format easily readable by humans: <strong>Extract -&gt; Translate -&gt; Write. </strong>We deal with each task separately and tie them together in the end.</p><h4>Extract</h4><p>As you might already have experienced yourself, retrieving the text from a PDF can be quite tricky. The reason for this is that PDFs only store the <em>location</em> of characters and do <em>not record</em> what constitutes <em>words or lines</em>. Our library of choice is the new <strong>pdfplumber </strong>project, which is built on the very good <strong>pdfminer.six</strong> library (itself replacing <strong>PDFMiner</strong>) but sports a better documentation and exciting new features. One feature we’ll make use of here is the filtering of tables. For completeness, note that the popular <strong>PyPDF2</strong> package serves better for PDF merging, rather than text extraction.</p><pre>import pdfplumber</pre><pre>pdf = pdfplumber.open(“src/examples/1978-geschaeftsbericht-data.pdf”)</pre><p>We import the library and open the desired document. The central object of <strong>pdfplumber</strong> is the <strong>Page Class</strong>, which allows us to access each page and its content individually. Note that, while we could simply extract all text at once, reducing the pdf to one large string causes us to lose a lot of useful information.</p><p>Below how indices can be used to access individual pages and easily access their text by appling the <strong>extract_text()</strong> method.</p><pre>page11 = pdf.pages[11]<br>page11.extract_text()<br>&gt;&gt;&gt; 2  schließlich diese Wende in ihrer Politik durch die Heraufsetzung des Diskont- und Lom \nbardsatzes.</pre><p>While this already looks great (for comparison, check the 12th page of the PDF), we see that sentences are disrupted by end-of-line breaks, which we can predict will create problems for translation. Since paragraphs naturally have a line-break after a full-stop, we’ll exploit this to keep only desired line breaks.</p><pre>def extract(page):<br>  &quot;&quot;&quot;Extract PDF text and Delete in-paragraph line breaks.&quot;&quot;&quot;</pre><pre>  # Get text</pre><pre>  extracted = page.extract_text()</pre><pre>  # Delete in-paragraph line breaks</pre><pre>  extracted = extracted.replace(&quot;.\n&quot;, &quot;**/m&quot; # keep par breaks<br>                      ).replace(&quot;. \n&quot;, &quot;**/m&quot; # keep par breaks<br>                      ).replace(&quot;\n&quot;, &quot;&quot; # delete in-par breaks     <br>                      ).replace(&quot;**/m&quot;, &quot;.\n\n&quot;) # restore par break<br> return extracted</pre><pre>print(extract(page11)[:500])<br>&gt;&gt;&gt; 2  schließlich diese Wende in ihrer Politik durch die Heraufsetzung des Diskont- und Lom bardsatzes.</pre><p>Much better! But looking at the next page, we see that we have problems with the tables in the document.</p><pre>page12 = pdf.pages[12]</pre><pre>print(extract(page12)[:500])<br>&gt;&gt;&gt; 1  3 Zur Entwicklung des Wirtschaftswachstums Jährliche Veränderung in o;o Zum Vergleich: I  Bruttoin-Brutto- I ...</pre><p><strong>Filtering-out tables</strong></p><p>A highlight of the <strong>pdfplumber</strong> package is the <strong>filter</strong> method. The library comes with built-in functionality for finding tables but combining it with <strong>filter</strong> requires some <a href="https://github.com/jsvine/pdfplumber/issues/242#issuecomment-668448246">ingenuity</a>. Essentially, <strong>pdfplumber</strong> allocates each character to so-called “boxes”, the coordinates of which <strong>filter</strong> takes as input. For the sake of brevity, I won’t explain the <strong>not_within_bboxes</strong> function but point towards the original <a href="https://github.com/jsvine/pdfplumber/blob/stable/pdfplumber/table.py#L404">Git issue</a>. We pass the identified characters belonging to tables and combine them with the <strong>not_within_bboxes</strong> function. Importantly, since the filter method only accepts a function without arguments, we freeze the boxes argument using <strong>partial</strong>. This is added as a prior step to the <strong>extract</strong> function we created above.</p><pre>from functools import partial</pre><pre>def not_within_bboxes(obj, bboxes):<br>&quot;&quot;&quot;Check if the object is in any of the table&#39;s bbox.&quot;&quot;&quot;<br>  def obj_in_bbox(_bbox):<br>    &quot;&quot;&quot;Find Bboxes of objexts.&quot;&quot;&quot;<br>    v_mid = (obj[&quot;top&quot;] + obj[&quot;bottom&quot;]) / 2<br>    h_mid = (obj[&quot;x0&quot;] + obj[&quot;x1&quot;]) / 2<br>    x0, top, x1, bottom = _bbox</pre><pre>    return (h_mid &gt;= x0) and (h_mid &lt; x1) and (v_mid &gt;= top) and (v_mid &lt; bottom)</pre><pre>  return not any(obj_in_bbox(__bbox) for __bbox in bboxes)</pre><pre>def extract(page):<br>  &quot;&quot;&quot;Extract PDF text, Filter tables and delete in-par breaks.&quot;&quot;&quot;<br>  # Filter-out tables<br>  if page.find_tables() != []:</pre><pre>  # Get the bounding boxes of the tables on the page.<br>  bboxes = [table.bbox for table in page.find_tables()]<br>  bbox_not_within_bboxes = partial(not_within_bboxes, bboxes=bboxes)</pre><pre>  # Filter-out tables from page<br>  page = page.filter(bbox_not_within_bboxes)</pre><pre>  # Extract Text<br>  extracted = page.extract_text()</pre><pre>  # Delete in-paragraph line breaks<br>  extracted = extracted.replace(&quot;.\n&quot;, &quot;**/m&quot; # keep par breaks<br>                      ).replace(&quot;. \n&quot;, &quot;**/m&quot; # keep par breaks<br>                      ).replace(&quot;\n&quot;, &quot;&quot; # delete in-par breaks<br>                      ).replace(&quot;**/m&quot;, &quot;.\n\n&quot;) # restore par break<br> return extracted</pre><pre>print(extract(page12)[:500])<br>&gt;&gt;&gt; 3 des Produktionspotentials anzusehen. Die statistischen Möglichkeiten lassen es nur an näherungsweise zu, ...</pre><p>Fantastic! The table was successfully filtered out and we can now see that the page starts with a sentence cut in half by the page break. We leave it with extraction, but I encourage you to play around with more features, such as extracting page numbers, improving paragraph separation and fixing frequent mistakes such as recognising “0/o” for “%”.</p><h4>Translation</h4><p>AWS and DeepL offer two prominent APIs for high quality translation of text, but the character-based pricing schemes can turn-out extremely costly if we want to translate several long reports. To translate free of charge, we use the Google Api with a key workaround, enabling the translation of long texts.</p><pre>from deep_translator import GoogleTranslator</pre><p>Since the GoogleTranslate API is not maintained by Google, the community has repeatedly faced issues in translation. That is why we here use the <strong>deep_translator</strong> package, which acts as a useful wrapper for the API and enables us to seamlessly switch between translation engines, should we wish to. Importantly, GoogleTranslator can automatically identify the source language (in our example German), so we only need to specify our target language: English.</p><pre>translate = GoogleTranslator(source=’auto’, target=’en’).translate</pre><p>With this wrapper translation is extremely simple, as the following example demonstrates.</p><pre>translate(&quot;Ich liebe Python programmieren.&quot;)<br>&gt;&gt;&gt; &#39;I love Python programming.&#39;</pre><p>However, the key issue is that most translation engines have a 5000-byte upload limit. If a job should exceed it, the connection is simply terminated — which would, for example, prevent the translation of <strong>page11</strong>. Of course, we could translate every word/sentence individually, however, this undermines the translation quality. That is why we’re collecting chunks of sentences just below the upload the limit and translate them together.</p><p>Originally, I found this the workaround <a href="https://aws.amazon.com/de/blogs/media/how-to-translate-large-text-documents-with-amazon-translate/">here</a>. It uses the popular natural language processing tool <strong>nltk</strong> for identifying sentences. The package’s documentation is great, and I recommend anyone interested to try it out. Here, we’re limiting our attention to the package’s <strong>tokenizer</strong>. Importantly, tt cannot be stressed enough that only high-quality input will lead to high-quality translation output, so going the extra mile in these preparation steps will easily pay-off!</p><p>Because this can be daunting for first time-users, I present here the shell-script to install the relevant <strong>nltk</strong> fuctionality (on Windows OS). The “popular” subset includes the <strong>nltk.tokenize</strong> package, which will use now.</p><pre># Shell script</pre><pre>pip install nltk<br>python -m nltk.downloader popular</pre><p>As you can see below, the <strong>sent_tokenize</strong> function creates a list of sentences. The language argument defaults to English, which works just fine for most European languages. Please check-out the <strong>nltk</strong> documentation to see if the language you need is supported.</p><pre>from nltk.tokenize import sent_tokenize</pre><pre>text = &quot;I love Python. &quot; * 2<br>sent_tokenize(text, language = &quot;english&quot;)<br>&gt;&gt;&gt; [&#39;I love Python.&#39;, &#39;I love Python.&#39;]</pre><p>Now, the second ingredient we need is an algorithm for collecting chunks of sentences below the upload limit. Once we find that adding another sentence would exceed 5k bytes, we translate the collection and start a new chunk with the current sentence. Importantly, if a sentence itself should be longer than a 5k bytes (which, remember, corresponds to roughly a page), we simply discard it and provide an in-text note, instead. Combining the i) set-up of the translation client, ii) sentence tokenization, and iii) chunk wise translation, we end up with the following translation function.</p><pre>def translate_extracted(Extracted):<br>  &quot;&quot;&quot;Wrapper for Google Translate with upload workaround.&quot;&quot;&quot;<br>  # Set-up and wrap translation client<br>  translate = GoogleTranslator(source=&#39;auto&#39;, target=&#39;en&#39;).translate</pre><pre>  # Split input text into a list of sentences<br>  sentences = sent_tokenize(Extracted)</pre><pre>  # Initialize containers<br>  translated_text = &#39;&#39;<br>  source_text_chunk = &#39;&#39;</pre><pre>  # collect chuncks of sentences, translate individually<br>  for sentence in sentences:<br>    # if chunck + current sentence &lt; limit, add the sentence<br>    if ((len(sentence.encode(&#39;utf-8&#39;)) +  len(source_text_chunk.encode(&#39;utf-8&#39;)) &lt; 5000)):<br>      source_text_chunk += &#39; &#39; + sentence</pre><pre>    # else translate chunck and start new one with current sentence<br>    else:<br>      translated_text += &#39; &#39; + translate(source_text_chunk)</pre><pre>     # if current sentence smaller than 5000 chars, start new chunck<br>     if (len(sentence.encode(&#39;utf-8&#39;)) &lt; 5000):<br>       source_text_chunk = sentence</pre><pre>     # else, replace sentence with notification message<br>     else:<br>       message = &quot;&lt;&lt;Omitted Word longer than 5000bytes&gt;&gt;&quot;<br>       translated_text += &#39; &#39; + translate(message)</pre><pre>       # Re-set text container to empty<br>       source_text_chunk = &#39;&#39;</pre><pre>  # Translate the final chunk of input text, if there is any valid   text left to translate<br>  if translate(source_text_chunk) != None:<br>    translated_text += &#39; &#39; + translate(source_text_chunk)</pre><pre>  return translated_text</pre><p>To see if it works, we apply our translation function to a page we already worked with earlier. As is now also evident for the non-German speakers, apparently the hourly productivity rate increased by roughly 4% in 1978.</p><pre>extracted = extract(pdf.pages[12])<br>translated = translate_extracted(extracted)[:500]</pre><pre>print(translated)<br>&gt;&gt;&gt;3 of the production potential. The statistical possibilities allow only an approximation of the closures that still occur physically due to long-term shrinkage ...</pre><h4>Writing</h4><p>We almost have everything we need. Like me, you might need to bring your extracted text back into a format <em>easily</em> readable by humans. While it is easy to save strings to <strong>.txt</strong> in Python, the lack of line-breaks makes them a poor choice for long reports. Instead, we will here write them back to PDF using the <strong>fpdf2</strong> <a href="https://pyfpdf.github.io/fpdf2/index.html">library</a>, which apparently succeeds the no longer maintained <strong>pyfpdf </strong>package.</p><pre>from fpdf import FPDF</pre><p>After initializing an FPDF object, we can add a page object for every page we translated and write them on there. This will help us maintain the structure of the original document. Two things to note: firstly, in <strong>multi_cell</strong> we set width to zero to have full width and choose a hight of <em>5</em> to have slim line spacing. Secondly, since the pre-installed fonts are not Unicode compatible, we change the encoding to “<em>latin-1</em>”. For instructions on download and the use of Unicode compatible fonts, see the instructions on the <strong>fpdf2</strong> <a href="https://pyfpdf.github.io/fpdf2/Unicode.html">website</a>.</p><pre>fpdf = FPDF()<br>fpdf.set_font(&quot;Helvetica&quot;, size = 7)</pre><pre>fpdf.add_page()<br>fpdf.multi_cell(w=0,h=5,<br>               txt= translated.encode(&quot;latin-1&quot;,errors = &quot;replace&quot;<br>                             ).decode(&quot;latin-1&quot;)<br>                )<br>fpdf.output(&quot;output/page12.pdf&quot;)</pre><p>Now, just like in extraction, there is obviously a lot more you could do with the <strong>fpdf2</strong>, such as the adding of page numbers, title layout, etc. However, for the purpose of this article, this minimal set-up will suffice.</p><h3><strong>Tying everthing together</strong></h3><p>We’ll now bring everything together in one pipeline. Remember that, to avoid losing too much information, we operate on each page individually. Importantly, we make two adaptations to the translation: since some pages are empty, but empty strings are not valid input for <strong>GoogleTranslator</strong>, we place an if condition before the translation. Secondly, because <strong>nltk</strong> allocates our paragraph breaks (i.e., “<em>\n\n</em>”) to the beginning of the <em>following</em> sentence, <strong>GoogleTranslate</strong> ignores these. That is why we translate each paragraph individually using a list comprehension. <em>Be patient, translating 150 pages can take up to 7 minutes!</em></p><pre># Open PDF<br>with pdfplumber.open(“src/examples/1978-geschaeftsbericht-data.pdf”) as pdf:<br>  # Initialize FPDF file to write on<br>  fpdf = FPDF()<br>  fpdf.set_font(“Helvetica”, size = 7)</pre><pre>  # Treat each page individually<br>  for page in pdf.pages:<br>    # Extract Page<br>    extracted = extract(page)</pre><pre>    # Translate Page<br>    if extracted != “”:<br>      # Translate paragraphs individually to keep breaks<br>      paragraphs = extracted.split(“\n\n”)<br>      translated = “\n\n”.join(<br>        [translate_extracted(paragraph) for paragraph in paragraphs]<br>        )<br>    else:<br>      translated = extracted</pre><pre>    # Write Page<br>    fpdf.add_page()<br>    fpdf.multi_cell(w=0, h=5,<br>                   txt= translated.encode(“latin-1”,<br>                                          errors = “replace”<br>                                 ).decode(“latin-1”))</pre><pre> # Save all FPDF pages<br> fpdf.output(“output/trans_1978-geschaeftsbericht-data.pdf.pdf”)</pre><h3>Conclusion</h3><p>Thank you for staying to the end. I hope this article gave you a hands-on example of how to translate PDFs and what the state-of-the-art packages are. Throughout the article I pointed out various potential extensions to this rudimentary example (i.e., adding page numbers, layout, etc.), so please share your approaches for these — I’d love to hear them. And, of course, I am also always eager to hear suggestions on how to improve on the code.</p><p><em>Stay safe and stay in touch!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eab3be08ceb4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/translate-long-pdf-reports-in-python-eab3be08ceb4">Translate long PDF-Reports in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>