<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ellen Schellekens on Medium]]></title>
        <description><![CDATA[Stories by Ellen Schellekens on Medium]]></description>
        <link>https://medium.com/@ellen.schellekens?source=rss-cbd0f5ba7697------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*WtB0APlaG1hu0wYWak5_tQ.jpeg</url>
            <title>Stories by Ellen Schellekens on Medium</title>
            <link>https://medium.com/@ellen.schellekens?source=rss-cbd0f5ba7697------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:41:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ellen.schellekens/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Speed up Python Code with CPU Profiling]]></title>
            <link>https://medium.com/ixor/how-to-speed-up-python-code-with-cpu-profiling-39d6788962cf?source=rss-cbd0f5ba7697------2</link>
            <guid isPermaLink="false">https://medium.com/p/39d6788962cf</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[speed]]></category>
            <category><![CDATA[profiling]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ellen Schellekens]]></dc:creator>
            <pubDate>Wed, 20 Jul 2022 14:08:50 GMT</pubDate>
            <atom:updated>2022-07-20T14:08:50.373Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/283/1*GXRNi5IjPMLQh-DHKBQpAw.jpeg" /><figcaption><a href="https://playgrounds.ai/models/dalle-mega">Dall-e mega</a> assisted impression of fast python code</figcaption></figure><p>Optimising your code’s running time has only advantages. For instance if your code runs in the cloud on a pay-for-use solution, such as AWS Lambda, you will see an immediate cost reduction if you decrease the running time. And if your code runs in a time sensitive environment, or on a device with limited resources, optimising the running time is crucial.</p><p>But in order to do so, you must first analyse which parts of the code use the most time to start with. You can spend as much time as you want to perfect the running time of a specific piece of code, if that code only took a small part of the total running time to begin with, you will barely notice improvements. This is also known as <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s law</a>.</p><p>But how to make this analysis? That is where profilers come in! In this article, we’ll discuss a few different type of python profilers, namely deterministic profilers, probabilistic profilers and line profilers.</p><h3>Deterministic Profiling</h3><p>Deterministic profiling is a way to get a general overview that keeps track of all the calls that happen in the run, and how long they take. This way, they can provide a detailed overview of everything that happens. However, the downside to this is that there is considerable overhead, and the result is so detailed it might be hard to make head or tail of it.</p><p>You can do deterministic profiling with a build-in library of python: <a href="https://docs.python.org/3/library/profile.html#module-cProfile">cProfile</a>. The library can be used in a number of ways: you can directly run a piece of code, you can use it as context manager or as a class. Another approach, and the approach that we will use, is to use the command line interface. Since the direct result is very cluttered, we will format the results in a <em>pstats</em> file. The following command does this:</p><pre>python -m cProfile -o output_filename.pstats path/to/script arg1 arg2</pre><p>There are various libraries that convert the <em>pstats </em>file to a graph, we use a library called <a href="https://github.com/jrfonseca/gprof2dot">gprof2dot</a>:</p><pre>gprof2dot -f pstats myProfileFile | dot -Tpng -o image_output.png</pre><p>The result looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VAG6xpLuITJNqhJXGcfBsA.png" /></figure><p>This might seem quite overwhelming, but big parts of the graph are initialisation or low level functions. By focusing on the root node (one of the red ones), you can follow the graph to find the biggest shares of the time distribution. In this case, we see that the running the AI model takes up 83% of the time, which is to be expected. Furthermore, the postprocessing of our model takes almost 14% of the time, while preprocessing only takes 1,3%! It’s clear we should take a closer look at the postprocessing for speed optimisations.</p><h3>Probabilistic Profiling</h3><p>The deterministic profiling is all good and fine, but what if your code is too big, and the profiling requires too much overhead to perform? Fear not, my friend! This is where probabilistic profiling comes in. Instead of keeping track of every single call, this type of profiling checks the call stack at regular intervals to see which function is performing. This way, it creates an overview of the total time distribution. This kind of ‘checking at regular intervals’ is also known as Monte Carlo sampling.</p><p>The build-in profiling package cProfile does not offer probabilistic profiling at the time of writing, but there are numerous other packages that offer this. We used the package <a href="https://pyinstrument.readthedocs.io/en/latest/guide.html#">PyInstrument</a>. It allows to export the results to different formats, among which an interactive html page. In terms of user-friendliness, this is definitely a big improvement compared to the previous approach! If you use the html output format, the result will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRJMV0WdO-Pc8gZplkWKDA.png" /></figure><p>The main conclusions are the same as with the deterministic profiling, but the exact time estimates do vary quite a bit. For example, the probabilistic profiler estimates the time spent in the model itself as 62%, compared to the 83% estimate of the deterministic model. But for the purpose of identifying slow parts of the code, the exact percentages don’t matter.</p><h3><strong>Line Profiling</strong></h3><p>In some cases, the above measures already give enough pointers to where the slow parts of the code are. But in other cases, you might have narrowed it down to a specific function that is quite long or complex. In this case, you can use line profilers to profile how much time is spent in each line of the function.</p><p>To do this, we use the line_profiler package available at <a href="https://github.com/pyutils/line_profiler">https://github.com/pyutils/line_profiler</a>. To use it, just add the decorator @profile above the function(s) you want to profile. The package will print out the number of hits (times the function is executed), the total time, the average time per hit, the time percentages and the content of the line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*X1JS1RmMPi_l39jZu3liAw.png" /></figure><p>In this case, it’s quite clear that the compiling of the regexes took way too much time, and be combining multiple regexes in 1 more complex regex, we could save a considerable amount of time.</p><h3>Conclusion</h3><p>With the tools discussed in this article, you can identify the slow parts of your Python code in order to accelerate them. You can follow these steps:</p><ol><li>Analyse the whole module to identify bottleneck functions. You can either use deterministic profiling (cProfiler) or probabilistic profiler (Pyinstrument)</li><li>Use line profilers when necessary to identify specific problematic lines</li><li>Now you know which parts to speed up!</li></ol><p>The advantage of deterministic profilers is that they are precise and count all function calls. But compared to probabilistic profilers, they have way more overhead and are often less user-friendly to work with.</p><p>At <a href="http://ixorthink.com/">IxorThink</a>, the machine learning practice of <a href="http://ixor.be/">Ixor</a>, we are constantly trying to improve our methods to create state-of-the-art solutions. As a software company, we can provide stable products from proof-of-concept to deployment. Feel free to contact us for more information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39d6788962cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ixor/how-to-speed-up-python-code-with-cpu-profiling-39d6788962cf">How to Speed up Python Code with CPU Profiling</a> was originally published in <a href="https://medium.com/ixor">Ixor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing Tesseract OCR with Google Vision OCR for Text Recognition in Invoices]]></title>
            <link>https://medium.com/ixor/comparing-tesseract-ocr-with-google-vision-ocr-for-text-recognition-in-invoices-bddf98f3f3bd?source=rss-cbd0f5ba7697------2</link>
            <guid isPermaLink="false">https://medium.com/p/bddf98f3f3bd</guid>
            <category><![CDATA[ocr]]></category>
            <category><![CDATA[invoice]]></category>
            <category><![CDATA[document-processing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ellen Schellekens]]></dc:creator>
            <pubDate>Thu, 21 Apr 2022 13:04:31 GMT</pubDate>
            <atom:updated>2022-04-25T14:54:40.700Z</atom:updated>
            <content:encoded><![CDATA[<p>At IxorThink, we develop document processing algorithms specialised in invoices and receipts. We automatically extract key information from the documents, such as the supplier and the total cost. The first step of this process is to recognise and extract the text from the documents using Optical Character Recognition (OCR). Improving the quality of the extracted text is important to improve the whole document processing pipeline. With this use case in mind, we compare two OCR engines in this blog: Tesseract and Google Vision API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*Xvjab71osFVjWPR2G95tZg.png" /></figure><h3>Tesseract OCR</h3><p>Tesseract is the most prominent opensource OCR engine. Originally developed by Hewlett-Packard, it is now sponsored by Google. The active community continually works to improve it and there are regular new updates. Since it is open source and thus runs locally, the only cost of using Tesseract are the resources the machine uses, and there is no need to communicate the document and the results over the internet. The LSTM used in the model can also be trained on additional languages or fonts.</p><p>Tesseract can be executed from command-line interface, and also has a C++ api. There are numerous wrappers that allow the api to be used in other programming languages too, for example in Python. In this comparison, we will use the command-line interface.</p><h3>Google Vision OCR</h3><p>The Google Vision API is part of the Google Cloud and includes among many interesting services also the option for text detection. In contrast to Tesseract, there is a service cost of $1.50 per 1,000 units for the first 5 million documents per month. For documents on top of those 5 million per month, the price is reduced to $0.60 per 1,000 units. It is possible to use the service as an API, providing the image or pdf encoded as base64 string, or to store the file on Google Cloud Storage and provide the bucket information.</p><p>There are two annotation features: TEXT_DETECTION is specialised to extract text from any image, while DOCUMENT_TEXT_DETECTION is optimised for dense text and documents. We found that while DOCUMENT_TEXT_DETECTION is a little better at capturing clean text lines and columns in the document, it also splits words based on punctuation marks such as . , - and : . In our case, there is structured information such as invoice numbers and VAT numbers that often contain punctuation marks, but that we do prefer to detect as one word. This is why we use the TEXT_DETECTION feature, since it does not have the excessive splitting issue.</p><h3>Experiments</h3><p><strong>Methodology</strong><br>To compare the two OCR engines, we first have to create a testset of invoices. Some hard cases that are added to the testset: skewed scanned documents, documents with very small text,… On each document, we first compare all found word-level bounding boxes to find matches. When there is at least a 50% overlap, we consider two bounding boxes to correspond to the same word. For the matched bounding boxes, we compute the Hamming distance between the recognised words. Next to the quality, we also evaluate the time needed for each engine.</p><p><strong>Results</strong><br>One clear result is that Tesseract is faster than Google Vision by 0.689 seconds on average. However, do the recognised words differ a lot? The figure below shows a histogram of the Hamming distances between the matched words, where the ‘-1’ and ‘-2’ categories are unmatched words found by respectively Tesseract and Google Vision. We see that for around 60% of words, the words are recognised exactly the same. For another 11%, there are small differences of one or two characters in the word. However, as much as 18% of the recognised words are only found by Google and are not matched to words found by Tesseract.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*AKyWu3TI-SAAtPKp3ssmOQ.png" /></figure><p>When we look closer at this category, we find that the problem lies with the small text that often resides at the bottom of invoices. The figure below shows an example of this. Google Vision is better capable of recognising these small letters, even when additional preprocessing is added to Tesseract to improve the contrast of the letters. Since this small text sometimes contains useful information, for example about the supplier of the invoice, this is a dealbreaker to use Tesseract in this application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5_D-4TDfqBC1kr8BgSt-Q.png" /></figure><h3>Conclusion</h3><p>Tesseract OCR has many strengths, such as the low cost and high speed. Being in full control of the model and having the ability to further train or finetune are additional advantages. However, the quality of the Google Vision OCR is still better, especially on difficult cases such as very small text. Since the quality is most important to us, the Google Vision OCR wins the comparison in our use case.</p><p>At <a href="http://ixorthink.com/">IxorThink</a>, the machine learning practice of <a href="http://ixor.be/">Ixor</a>, we are constantly trying to improve our methods to create state-of-the-art solutions. As a software company, we can provide stable products from proof-of-concept to deployment. Feel free to contact us for more information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bddf98f3f3bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ixor/comparing-tesseract-ocr-with-google-vision-ocr-for-text-recognition-in-invoices-bddf98f3f3bd">Comparing Tesseract OCR with Google Vision OCR for Text Recognition in Invoices</a> was originally published in <a href="https://medium.com/ixor">Ixor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Designing a Document Segmentation Architecture for Invoice Recognition]]></title>
            <link>https://medium.com/ixor/designing-a-document-segmentation-architecture-for-invoice-recognition-2d61ceeed9cd?source=rss-cbd0f5ba7697------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d61ceeed9cd</guid>
            <category><![CDATA[resnet]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[document-processing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ellen Schellekens]]></dc:creator>
            <pubDate>Tue, 25 Jan 2022 13:19:13 GMT</pubDate>
            <atom:updated>2022-01-25T13:19:13.586Z</atom:updated>
            <content:encoded><![CDATA[<p>Choosing the right architecture for your deep learning task is difficult. To address this problem in an efficient way, it is important to understand why an architecture fails to provide good results. In this article I’ll demonstrate my process of finding a suitable architecture to perform <a href="https://en.wikipedia.org/wiki/Named-entity_recognition">Named Entity Recognition</a> for invoices.</p><h3>First Attempt</h3><p>A good first step is to look at the segmentation algorithms, that are already implemented in <a href="https://pytorch.org/vision/stable/models.html">torchvision</a>. These architectures are proven to work in general settings, and there is no need to reinvent the wheel! The models that are implemented are constructed by two components: the <em>backbone</em>, a convolutional network that will extract useful features from the input document, followed by a <em>classification head </em>that will use these extracted features to predict the labels.</p><h4>DeeplabV3 ResNet50</h4><p>The first implemented architecture I experimented with was a ResNet50 backbone, followed by a DeeplabV3 classifier head. The <a href="https://arxiv.org/pdf/1706.05587v3.pdf">DeeplabV3 classifier</a> head utilizes <em>Atrous Spatial Pyramid Pooling </em>(ASPP)<em> </em>to capture multi-scale context. Additionally, image-level features are also encoded in the image pooling to achieve the best performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*upqRDf_UwEML9upv1hsuoA.png" /><figcaption>[The architecture of the DeeplabV3 classifier [1].</figcaption></figure><h4>FCN ResNet50</h4><p>The second architecture that was tested was the FCN ResNet50 model. Here, the ResNet backbone is followed by a simple Fully Convolutional Network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*IitDkVhvHgVYqMU1.png" /><figcaption>The FCN ResNet50 architecture [2].</figcaption></figure><h4>Experimental Results</h4><p>The experimental results of these two models were quite disappointing. But why was that? The ResNet50 backbone is designed to extract high-level features from images to do image classification. In order to do this in a time and memory efficient manner, the generated feature map has a much lower dimension compared to the original input image. Afterwards, the classifier heads make sure the final segmentation map has the same dimensions as the input again, by interpolating the feature map.</p><p><em>Interpolation</em> is used to go from one dimension size to another, by calculating the unknown values based on the proximate known values. In this case, the image is interpolated from a lower dimension to a higher dimension, which causes some blurring to happen. This is no problem for <em>image</em> segmentation, since the desired output is often the general region of the different segments. But in our use case of <em>document</em> segmentation, each word resides in exactly 1 pixel and our desired output is to label the individual words, and thus the individual pixels. The blurring effect of interpolation prevents the prediction to be precise for each individual pixel. This is why these models don’t work well for us, and why we want to avoid interpolation. To make sure that there is no need for interpolation in the classifier heads, we need to find a backbone network that has the same output dimensions as input dimensions.</p><h3>Let’s Try This Again</h3><p>The previous bad results do not mean that the classifier heads aren’t useful, we just need a backbone that returns a feature set of the same size as the input dimensions. For this purpose, UNet-like architectures are the perfect choice. In this architecture, max pooling is performed to make sure the output segmentation map have the same dimensions as the input image. In principle, UNet can already be used as standalone method, but adding an extra classifier head on top of it, like Deeplab and LR-ASPP will further improve the performance by being able to look at the complete picture. Since UNet is used as backbone, there is also no need for interpolation anymore in the classifier head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tlb1pu0IAJEVZQy7.png" /><figcaption>The UNet architecture [3].</figcaption></figure><h4>Experimental Results</h4><p>Using this model, the results improved greatly. This shows that understanding your use case, and using that to understand why some architectures don’t provide good results, is a crucial part to the design of a deep learning architecture.</p><h3>References</h3><p>[1] Chen, L. C., Papandreou, G., Schroff, F., &amp; Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. <em>arXiv preprint arXiv:1706.05587</em>.</p><p>[2] Labao, Alfonso &amp; Naval, Prospero. (2017). Weakly-Labelled Semantic Segmentation of Fish Objects in Underwater Videos Using a Deep Residual Network. 255–265. 10.1007/978–3–319–54430–4_25.</p><p>[3] <a href="https://towardsdatascience.com/unet-line-by-line-explanation-9b191c76baf5">https://towardsdatascience.com/unet-line-by-line-explanation-9b191c76baf5</a></p><p>At <a href="http://ixorthink.com/">IxorThink</a>, the machine learning practice of <a href="http://ixor.be/">Ixor</a>, we are constantly trying to improve our methods to create state-of-the-art solutions. As a software company, we can provide stable products from proof-of-concept to deployment. Feel free to contact us for more information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d61ceeed9cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ixor/designing-a-document-segmentation-architecture-for-invoice-recognition-2d61ceeed9cd">Designing a Document Segmentation Architecture for Invoice Recognition</a> was originally published in <a href="https://medium.com/ixor">Ixor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Used RoBERTa to Learn Common Sense Knowledge for Spatial Reasoning]]></title>
            <link>https://medium.com/ixor/how-i-used-roberta-to-learn-common-sense-knowledge-for-spatial-reasoning-711073790e01?source=rss-cbd0f5ba7697------2</link>
            <guid isPermaLink="false">https://medium.com/p/711073790e01</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ellen Schellekens]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 07:51:19 GMT</pubDate>
            <atom:updated>2021-09-13T17:27:23.923Z</atom:updated>
            <content:encoded><![CDATA[<p>Humans have a natural profound understanding about the physical world we are born in. We know how objects move, and where they are located in relation to us. This kind of common sense knowledge is learned implicitly and is difficult to capture by machine systems.</p><p>With the improving quality of language models, the common sense aspect stays a challenging and fascinating component. <strong>Language models</strong> like BERT have enabled breakthroughs in many applications, such as dialogue systems and generative language modelling. These models are pre-trained on huge datasets, in an unsupervised manner. Recently, there has been a switch in literature to see these models as <strong>knowledge bases</strong> instead of simply generating syntactically correct language. After all, they encounter a lot of information during the pre-training. Research shows that the pre-training is sufficient for the model to learn factual and even some common sense knowledge [1, 2]. But do they also capture common sense knowledge about the <em>physical structure </em>of our world?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*G4RRnJflLuDAkzfEpumETw.jpeg" /></figure><p>This is the subject of my thesis: exploring whether neural language models are capable of performing qualitative spatial reasoning. Spatial reasoning itself is still a very broad topic, and so it is restricted to a specific domain: <strong>relative positional reasoning</strong> (left, right, above, below, in front of and behind). If the model is told that the cat is to the left of the tree, and the tree is to the left of the cow; can it deduce that the cat is also to the left of the cow?</p><h3>Constructing a Dataset of Relative Positions</h3><p>The data we use to train and test is <strong>textual</strong> and consists of two parts:</p><ul><li>The <strong>context</strong> explains the relative locations of a number of objects.</li><li>The <strong>question</strong> contains two objects that appeared in the context, with a masked positional predicate. This predicate can only be determined by using spatial reasoning on the information given in the context.</li></ul><p>To make the data as realistic as possible, it will contain positional relations in three dimensions, and will be based on real-life scenario’s. The two existing datasets on textual relative positions did not satisfy these conditions, so we created a new dataset.</p><p>We extracted our scenario’s from the COCO image dataset [3]. This ensures that the data is both based on real-life scenario’s and has relative positions in three dimensions. However, it is not straightforward to extract the 3D spatial information from the images as they are only 2D representations of a 3D world. To do this, we used the FCRN depth estimator [4]. The depth estimator takes as input the image, and outputs a heatmap of the depth coordinations. An example of such a heatmap can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlozBZ6-VBvn3ErHELMfZg.png" /><figcaption>An example output of the FCRN depth estimator [4] on an image of the COCO dataset</figcaption></figure><p>With the images and their respective depth estimations, we can extract our scenario’s. The COCO dataset provides segmentations for the objects in an image. In the first step, these segmentations and the depth estimations are used to <strong>extract the spatial relations</strong> between each pair of objects. In the next step, a number of these relations are chosen to <strong>create the context</strong>. Based on the chosen relations, the <strong>target relation is determined</strong>. This relation can only be reasoned from the relations mentioned in the context using either <strong>transitive or reverse reasoning</strong>. And lastly, the context and question are <strong>formulated in natural language</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gniBytEJUJhtx45V10W3DQ.png" /></figure><p>We use the language model <strong>RoBERTa </strong>[5], which is pre-trained using masked-language modelling, so the question is formulated as a <strong>masked sentence</strong>. This means that the positional relation is replaced with a mask. The model will then predict the most probable word to fill in the mask. Masked sentences are a promising probing approach in zero-shot setting, as the research of Schick and Schtze shows [6, 7]. When using masked sentences, the specific formulation can influence the performance of the model. So to get a more robust evaluation, multiple patterns of masked sentences are used, each with a different formulation.</p><h3>Results</h3><p>The goal of the research is two-fold: evaluate whether the model captures such common sense knowledge during the pre-training, and evaluate whether the model is able to learn it during finetuning. The first situation is also known as <em>zero-shot learning</em>. We also perform some ablation studies on the number of objects mentioned in the context, on the complexity of the sentences and on providing some additional information in the context.</p><h4>Zero-shot Evaluation</h4><p>The figure below shows the confusion matrix of the results in a zero-shot setting. In this masked sentence pattern, the model had to fill in the mask with either ‘yes’ or ‘no’. It is clear that the model mostly picks the same answer, regardless of the given information in the context. It cannot outperform a random guessing model. This result is observed in all masked sentence patterns. When changing factors such as the number of objects or sentence complexity, the behaviour of the model does not change. This suggests that the model does not learn spatial common sense information during pre-training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*xaS4JhquQqQx7Mc3Cb27Aw.png" /><figcaption>Confusion matrix of one of the masked sentence patterns in a zero-shot setting: the model picks the same word to fill in the blank 66.44% of the time.</figcaption></figure><h4>Fine-tuned Evaluation</h4><p>After training the model on a training set, the performance of the model became nearly perfect on the test set. This is further confirmed when we look at how many patterns the model gets right for each scenario. As you can see in the figure above, the model gets all patterns correct for around 80% of the scenario’s, which is quite good. When we add extra information (the caption of the original image) to the context information, the performance improves even more. Since the data only contains six relative locations, the model is most likely overfitted on this limited number of patterns. Future research could study whether the reasoning ability of the finetuned model improves on unknown patterns as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zDv_fESKzXT4rtTqmTu9g.png" /><figcaption>Visualisation of the number of patterns the model is able to predict correctly for each scenario. In nearly 80% of the scenario’s, the model is able to predict all the patterns correctly. This shows the robustness of the finetuned model.</figcaption></figure><h3>Conclusion</h3><p>Although research shows that language models can be used as knowledge bases through the information they learn during pre-training, this research shows that they do not learn the common sense needed to reason with relative locations. This knowledge can be learned, as the finetuned experiments clearly show. Future research can extend the number of relative positions, and explore different kinds of common sense knowledge.</p><h3>References</h3><p>[1]<em> F. Petroni, T. Rocktschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases?, 2019<br></em>[2] <em>A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2018.<br></em>[3]<em> T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollr. Microsoft coco: Common objects in context, 2015.<br></em>[4]<em> I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239–248. IEEE, 2016.<br></em>[5]<em> Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.<br></em>[6] <em>T. Schick and H. Schtze. Exploiting cloze questions for few shot text classification and natural language inference, 2021.</em><br>[7]<em> T. Schick and H. Schtze. It’s not just size that matters: Small language models are also few-shot learners, 2021.</em></p><p>At <a href="http://ixorthink.com/">IxorThink</a>, the machine learning practice of <a href="http://ixor.be/">Ixor</a>, we are constantly trying to improve our methods to create state-of-the-art solutions. As a software company, we can provide stable products from proof-of-concept to deployment. Feel free to contact us for more information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=711073790e01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ixor/how-i-used-roberta-to-learn-common-sense-knowledge-for-spatial-reasoning-711073790e01">How I Used RoBERTa to Learn Common Sense Knowledge for Spatial Reasoning</a> was originally published in <a href="https://medium.com/ixor">Ixor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>