<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Devshree Patel on Medium]]></title>
        <description><![CDATA[Stories by Devshree Patel on Medium]]></description>
        <link>https://medium.com/@devshree.p?source=rss-af78b7628cac------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RTJUmk7R38QSW61DinWQpg.jpeg</url>
            <title>Stories by Devshree Patel on Medium</title>
            <link>https://medium.com/@devshree.p?source=rss-af78b7628cac------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:25:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@devshree.p/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[LISA: An end-to-end deep framework for Instance Shadow Detection]]></title>
            <link>https://medium.com/visionwizard/lisa-an-end-to-end-deep-framework-for-instance-shadow-detection-48dc9c9ba450?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/48dc9c9ba450</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Thu, 13 Aug 2020 06:22:27 GMT</pubDate>
            <atom:updated>2020-08-13T06:22:27.759Z</atom:updated>
            <content:encoded><![CDATA[<h3>LISA: An End-to-End Deep Framework for Instance Shadow Detection</h3><h4>Understanding an unconventional approach for detecting object-shadow pairs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-GloeN1476Wi_md" /><figcaption>Photo by <a href="https://unsplash.com/@jakobowens1?utm_source=medium&amp;utm_medium=referral">Jakob Owens</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote>“When you light a candle, you also cast a shadow” — Ursula K. Le Guin written in A Wizard of Earthsea.</blockquote><blockquote>Shadows are light-deficient regions in a scene, due to light occlusion, but they carry the shape of the light-occluding objects, as they are projections of these objects onto the physical world -[1]</blockquote><blockquote>Instance shadow detection is aimed at finding shadow instances with corresponding object instances.</blockquote><h3>Table Of Contents</h3><p>1. Introduction</p><p>2. How is the LISA different than the previous frameworks?</p><p>3. Proposed technical contributions</p><p>4. Methodology and Architecture</p><p>5. Comparative results and Evaluation</p><p>6. Applications</p><p>7. Conclusion and Limitations</p><p>8. References</p><h3>1. Introduction</h3><ul><li>Computer vision and image processing fields have mainly focused on shadow-detection and shadow removal problems.</li><li>In [1], authors propose an <strong><em>end to end deep learning framework — Light guided Instance Shadow-object Association (LISA)</em></strong> to leverage the computation capability of DNNs, for instance, shadow detection.</li><li>Instance shadow detection refers to the discovery of shadow instances in the images along with the corresponding object that casts the shadow.</li><li>Detection of shadow-object pairs finds their use in a varied number of applications such as privacy protection and photo-editing tasks.</li><li>Association between shadow and object gives a clue about the light direction in the scene, bolstering applications such as relighting.</li><li>To facilitate instance shadow detection, authors prepare a dataset <strong><em>Shadow Object Association (SOBA) </em></strong>consisting of 3623 shadow-object pairs, each with three masks.</li><li>For evaluating the framework, a new metric called <strong><em>Shadow-Object Average Precision</em></strong> <strong><em>(SOAP)</em></strong> is introduced.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*mLZcy9uq1ebZKLCFKAKGag.png" /><figcaption>Figure 1. (a) Input Image (b) Output Image (c,d,e) Masks used for training (Source: [1])</figcaption></figure><h3>2. How is LISA different than previous frameworks?</h3><ul><li>Earlier works made use of different image processing techniques like physical illumination and color models for shadow detection.</li><li>In later works, convolutional neural networks (CNNs) were introduced to learn high-level features for shadow detection.</li><li>Recently, shadow detection and shadow removal tasks were performed using end-to-end networks.</li><li>Instance shadow detection is different from general shadow detection, as it finds associated objects with shadows altogether instead of finding a single mask for all shadows.</li></ul><h3>3. Proposed technical contributions</h3><ul><li>To approach the problem of instance shadow detection, authors present three technical contributions in [1].</li><li>Firstly, the authors create a novel dataset — SOBA consisting of 1000 images and 3623 shadow-object association pairs, where each input image is accompanied by three instance masks.</li><li>Secondly, the authors develop an end-to-end framework — LISA for predicting masks and boxes of object and shadow instances.</li><li>The framework also builds boxes around shadow-object associations along with light directions.</li><li>Later, ground truth instances of object and shadow are paired to match with the predicted shadow-object association and light-directions for producing output.</li><li>Thirdly, the authors formulate an evaluation metric — SOAP for calculating the efficiency of the proposed model with existing baseline models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*GxLPzJQ3QIVVHJwg_iHa_w.png" /><figcaption>Figure 2: Example predictions from the LISA framework (Source: [2])</figcaption></figure><h3>4. Methodology and Architecture</h3><ul><li>In comparison to primitive methods of shadow detection, LISA predicts shadow instances rather than just a single mask for all shadows in the image.</li><li>LISA takes the input of a single image and outputs: box and mask for each shadow/object instance, box for each shadow-object pair, and light direction for each couple.</li><li>LISA makes use of light direction as it is usually consistent with the object-shadow associations.</li><li>The architecture of LISA includes feature pyramid network as ConvNet for extracting important semantic features from the input image.</li><li>The architecture is bifurcated into two branches: the top one predicts the mask and box for each shadow/object, and the bottom one predicts the box for shadow-object pair with possible light direction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuLM-pyTN25kxMj0a2L05Q.png" /><figcaption>Figure 3: The schematic illustration of the LISA framework. (Source: [1])</figcaption></figure><ul><li>For finding regions with a high probability of containing the shadow-object pairs, region proposal network (RPN) is used in the top branch of the architecture.</li><li>Next, RoIAlign is used to extract features for each proposal and leverage the box and mask heads to make predictions for the shadow and object instances.</li><li>This, in turn, is done by minimizing the loss between the supervision signals from the training data and prediction results.</li><li>The bottom branch uses RPN for generating region proposals for the shadow-object pairs and then uses ROIAlign for extracting features for each proposal.</li><li>After discovering shadow-object associations, masks for each object-shadow associations are obtained by combining the shadow and objects masks for prediction.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*Cd2_yYJD-zWgAcnN8Y3NbQ.png" /><figcaption>Figure 4: The pair-and-match procedure for pairing the predicted shadow and object instances and efficiently matching them with the predicted shadow-object associations. (Source: [1])</figcaption></figure><h3>5. Comparative results and evaluation</h3><ul><li>Existing metrics for evaluating the efficiency of shadow detection and shadow removal models look at object instances separately.</li><li>Whereas, in case of instance shadow detection, multiple types of instances such as objects, shadows, and associations are involved.</li><li>Thus, the authors propose a new metric by considering traditional average precision and intersection over union technique.</li><li>SOAP — Shadow Object Average Precision works on three conditions based on IoU between predicted shadow/object/association and ground truth shadow/object/association.</li><li>To evaluate the efficiency of the model, the authors test it with various baseline frameworks.</li><li>The model proves to be efficient in terms of finding the light direction and in terms of performance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z71pge8TSiirPywqDHpNHw.png" /><figcaption>Figure 5: Visual comparison of instance shadow detection results produced by full pipeline and two other baseline frameworks. [(a) Input image, (b) Baseline 1 © Baseline 2 (d) Proposed Model] (Source: [1])</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*IREGNXd5-6EbayGV2PbaWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*yu3e7bcVIq3ccagwNxpKAw.png" /><figcaption>Figure 6,7: Evaluation Results of the proposed model (Source: [1])</figcaption></figure><h3>6. Applications</h3><ul><li>In [1], authors present versatile applications of their proposed model — LISA.</li><li>Some of the application scenarios include Light direction estimation, Photo editing, and privacy protection.</li><li>Instance shadow detection helps extensively to estimate the direction of light in images. Estimated light trends are shown as a virtual red post, which is consistent with the real shadows cast by objects.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*KyzDqZboVHc_qRNtRJAerw.png" /><figcaption>Figure 8: Examples of light direction estimation (Source: [1])</figcaption></figure><ul><li>Another useful application of instance shadow detection can be viewed in the photo-editing task. That is because LISA removes not only the object instances but also their associated shadows.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*g5_-lFouSJgt97fq_yii7g.png" /><figcaption>Figure 9: Examples in photo editing task (Source: [1])</figcaption></figure><ul><li>For instance, shadow detection works on shadow-object pairs. It makes cropping and editing of images efficient in comparison to naive cut and paste. That is because the light direction is maintained.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*m04HshaQlOFPMSvBpF5PGA.png" /><figcaption>Figure 10: Examples in photo editing task (Source: [1])</figcaption></figure><h3>7. Conclusion and Limitations</h3><ul><li>In [1], authors create a dataset — SOBA, propose a new end-to-end framework — LISA and formulate a metric SOAP to leverage the task of instance shadow detection.</li><li>Apart from several advantages, proposed technical contributions also have some limitations.</li><li>One of the limitations is that [1] fails to consider the overlapping area between shadows of different objects.</li><li>Also, shadow-object detection can be improved as there are many unsolved problems remaining to be addressed.</li></ul><blockquote>Code is accessible from the link: [<a href="https://github.com/stevewongv/InstanceShadowDetection?utm_source=catalyzex.com">github</a>]</blockquote><h3>8. References</h3><p>[1] Wang, Tianyu, et al. “Instance shadow detection.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>. 2020.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48dc9c9ba450" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/lisa-an-end-to-end-deep-framework-for-instance-shadow-detection-48dc9c9ba450">LISA: An end-to-end deep framework for Instance Shadow Detection</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text2SQL — Part 5: Final Insights]]></title>
            <link>https://medium.com/visionwizard/text2sql-part-5-final-insights-d00f85093d16?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/d00f85093d16</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Thu, 06 Aug 2020 13:05:25 GMT</pubDate>
            <atom:updated>2020-08-06T13:05:25.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Text2SQL — Part 5: Final Insights</h3><h4>Understanding AllenNLP library and getting final insights in Text2SQL tasks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5csxuizIDA1z_KqnbYQ-Q.png" /></figure><p>Welcome to the last part of the Text2SQL series! In this article, we will discuss some perks of using the AllenNLP library. Along with that, we will cast a light on some final insights in Text2SQL tasks.</p><blockquote>1. Text2SQL — Part 1: Introduction</blockquote><blockquote>2. Text2SQL — Part 2: Datasets</blockquote><blockquote>3. Text2SQL — Part 3: Baseline Models</blockquote><blockquote>4. Text2SQL — Part 4: State-of-the-art Models</blockquote><blockquote><strong>5.Text2SQL — Part 5: Final Insights</strong></blockquote><h3>Getting started with AllenNLP Library</h3><p>AllenNLP is an Apache NLP research library that is built on PyTorch for developing various state-of-the-art(SOTA) models used for a variety of NLP tasks.</p><p>Some of the relevant documentation on AllenNLP can be found <a href="https://docs.allennlp.org/master/">here</a>.</p><h3>AllenNLP Package Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*MayYCsadvqPwrPjYHwjFCA.png" /><figcaption>Figure 1: List of packages of AllenNLP Library (Source: Web)</figcaption></figure><h3><strong>Installing AllenNLP</strong></h3><ul><li>AllenNLP library can be installed using the virtual environment and pip command.</li><li>PIP command: <strong><em>pip install allennlp</em></strong></li><li>For installing various models supported by AllenNLP, use <strong><em>pip install allennlp-models</em></strong></li></ul><blockquote>Various different models like reading comprehension and semantic parser are available for <a href="https://demo.allennlp.org/reading-comprehension">demo</a>.</blockquote><h3>Modules available in AllenNLP</h3><ul><li>AllenNLP covers a wide range of modules listing from attention-based, transformer-based and encoder-decoder based modules.</li><li>These modules consist of classes that take various input parameters as arguments.</li><li>On the basis of the task to be performed, various class parameters can be exploited.</li><li>The following image is an example of a class of Bilinear Attention in the AllenNLP module.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*A5wU-U3gg2g2K3sn38VE8g.png" /><figcaption>Figure 2: Bilinear Attention class in AllenNLP module (Source: Web)</figcaption></figure><ul><li>AllenNLP also covers various modules such as learning rate scheduler and optimizer to ease the training process.</li><li>At the time of the release of AllenNLP, reference implementations of SRL, MC, and TE models were included.</li></ul><blockquote><strong>Semantic Role Labeling (SRL)</strong> models predict the verbal predicate argument structure of a sentence.</blockquote><blockquote><strong>Machine Comprehension (MC) </strong>systems take an evidence text and a question as input, and predict a span within the evidence that answers the question.</blockquote><blockquote><strong>Textual Entailment (TE)</strong> systems take a pair of sentences and predict whether the facts in one necessarily imply that the other is true.<br>-[1]</blockquote><h3>Final Insights</h3><ul><li>Release of libraries like AllenNLP will help to focus on core semantic problems including efforts to generalize semantic role labeling to all words and not just verbs.</li><li>Building semantic parsers which are capable of mapping language to code relatively easily.</li><li>Approaches such as semi-supervised learning can be made efficient with improved word representations.</li><li>Building models that can work well on texts which are out of domain such as scientific texts.</li><li>In this last article, we explored the AllenNLP library along with some final insights in the Text2SQL task.</li><li>I hope this series was helpful in imparting some knowledge in the domain of Text2SQL.</li></ul><h3>References</h3><p>[1] Gardner, Matt, et al. “Allennlp: A deep semantic natural language processing platform.” <em>arXiv preprint arXiv:1803.07640</em> (2018).</p><p>[2] AllenNLP documentation: <a href="https://docs.allennlp.org/master/">https://docs.allennlp.org/master/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d00f85093d16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/text2sql-part-5-final-insights-d00f85093d16">Text2SQL — Part 5: Final Insights</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text2SQL — Part 4: State-Of-The-Art Models]]></title>
            <link>https://medium.com/visionwizard/text2sql-part-4-state-of-the-art-models-cf81a377d4d2?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf81a377d4d2</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Sun, 02 Aug 2020 06:42:05 GMT</pubDate>
            <atom:updated>2020-08-02T06:42:05.699Z</atom:updated>
            <content:encoded><![CDATA[<h3>Text2SQL — Part 4: State-Of-The-Art Models</h3><h4>Understanding State-Of-The-Art models in Text2SQL domain</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lbAoBjjuTbhX-ISmQhmZdg.png" /></figure><p>In the last blog, we explored a few baseline models used in Text2SQL domain. These models with relatively less architectural complexities are unable to capture the larger the picture in Text2SQL task. However, these models prove to be beneficent for constructing and designing of new complex models.</p><p>Several different complex models like RAT-SQL make use of different concepts in order to solve the problems in Text2SQL task.</p><blockquote>1. Text2SQL — Part 1: Introduction</blockquote><blockquote>2. Text2SQL — Part 2: Datasets</blockquote><blockquote>3. Text2SQL — Part 3: Baseline Models</blockquote><blockquote><strong>4. Text2SQL — Part 4: State-of-the-art Models</strong></blockquote><blockquote>5.Text2SQL — Part 5: Final Insights</blockquote><p>Recent models challenge the natural language to SQL query task using SPIDER dataset. In order to learn more about SPIDER, refer my <a href="https://medium.com/visionwizard/text2sql-part-2-datasets-4ef61359e90e">2nd article</a> on datasets.</p><h4><strong>Some state of the art models to be covered in this blog:</strong></h4><ul><li><em>IRNet</em></li><li><em>EditSQL</em></li><li><em>RAT-SQL</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xhO56fCdhoGmsIRGI-4E-A.png" /><figcaption>Figure 1: A challenging text-to-SQL task from the Spider dataset from [3]</figcaption></figure><h3>1. IRNet</h3><ul><li>A neural approach called Intermediate Representation(IRNet) for complex and cross-domain Text-to-SQL aims to address 2 main challenges in Text2SQL tasks.</li><li>The challenges include the mismatch between expressed intents in natural language and the challenge in predicting columns caused by a greater number of out-of-domain words.</li><li>IRNet decomposes natural language into 3 phases instead of end-to-end synthesizing of SQL Query.</li><li>During the first phase, the schema linking process over a database schema and a question is performed.</li><li>IRNet employs SemQL domain-specific language, which serves as an intermediate representation between SQL and natural language.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*g3Dv0g28pGSpYv8lQmbgOg.png" /><figcaption>Figure 2: An illustrative example of SemSQL from [1]</figcaption></figure><ul><li>The model includes Natural Language (NL) encoder, Schema Encoder and Decoder.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qVIaSRPn3OdblO1_xOqrVQ.png" /><figcaption>Figure 3: An overview of the neural model proposed in [1]</figcaption></figure><ul><li>Every component of the model provides a different function for accomplishing Text2SQL task.</li><li>NL encoder takes the input of natural language and encodes it into an embedding vector. These embedding vectors are there used to construct hidden states using a bi-directional LSTM.</li><li>Schema encoder takes database schema as input and outputs representations for columns and tables.</li><li>Finally, the decoder is used to synthesize SemQL queries using a context-free grammar.</li><li>IRNet yields a significant improvement of 19.5% over previous benchmark models and achieves 46.7% accuracy on SPIDER dataset.</li><li>Incorporating IRNet with BERT shows a substantial improvement in performance and yields 54.7% accuracy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*HL3iKT-1J6kbekn6j_kT3A.png" /><figcaption>Figure 4: F1 scores of component matching of SyntaxSQLNet, SyntaxSQLNet(BERT), IRNet and IRNet(BERT) on the test set from [1]</figcaption></figure><h3>2. EditSQL</h3><ul><li>EditSQL focusses on the cross-domain context-dependent text-to-SQL task.</li><li>The authors exploit the fact that adjacent natural language questions are dependent and corresponding SQL queries overlap.</li><li>They utilise this dependency by editing the previously predicted query to improve the generation quality.</li><li>The editing mechanism takes the input of SQL as sequences and reuses generation results at the token level.</li><li>In order to deal with complex tables in different domains, an utterance-table encoder and a table-aware decoder are used to incorporate the context of the natural language and the schema.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*BDOOlC7OMnWTmG8KKoealA.png" /><figcaption>Figure 5: The model architecture of EditSQL from [2]</figcaption></figure><ul><li>The utterance-table encoder encodes user utterance and table schema. A bi-LSTM is used to encode tokens of utterances.</li><li>For each token, attention weighed an average of column header embedding is used to obtain the most relevant columns.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*A727oc3DIPz3Kc_TR8gpeQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*BApP8mIb_V5H3GIlDJiv9Q.png" /><figcaption>Figure 6,7: An example of user utterance and column headers and Utterance Encoder from [2].</figcaption></figure><ul><li>An attention layer is incorporated to capture the relationship between the table schema and utterance.</li><li>For capturing information across different utterances, an interaction-level decoder is built on the top of the utterance-level encoder.</li><li>The last step of the conversion includes generation of SQL queries by incorporating interaction history, table schema and user utterance using an LSTM decoder.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjsmTAMiNMzeiWAf8yaeQw.png" /><figcaption>Figure 8: Table Encoder from [2]</figcaption></figure><ul><li>The model is evaluated on SParC dataset, a large scale cross-domain context-dependent semantic parsing dataset developed from SPIDER.</li><li>The model performs well on both SPIDER and SParC but fails to surpass the previous state of the art model- IRNet.</li><li>The model achieves 32.9% accuracy in cross-domain text2SQL generation. Additionally, utilising BERT embedding yields significant improvement in accuracy, achieving 53.4%.</li></ul><h3>3. RAT-SQL</h3><ul><li>One of the main challenges in translating natural language questions into SQL queries is to generalize to unseen database schemas.</li><li>The generalisation depends on encoding the database relations in an accessible way and modelling alignment between relevant database columns in the query.</li><li>The proposed framework is based on the relation-aware self-attention mechanism which enables address schema encoding, feature representation and schema linking within a text2SQL encoder.</li><li>To get a quick overview of the encoder-decoder structure of RAT-SQL, go through the flow chart below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41A7FEpZAzDGnKpqCZLTSQ.png" /><figcaption>Figure 9: A flow chart of RAT-SQL model (Source: Author)</figcaption></figure><ul><li>RAT-SQL yields a significant improvement of 8.7% over previous benchmark models and achieves 57.2% accuracy on SPIDER dataset.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*Mbejv2i3Pnf0kXrC-RUQeg.png" /><figcaption>Figure 10: Accuracy on the Spider development and test sets, compared to the other approaches at the top of the dataset leaderboard as of May 1st, 2020 from [3]</figcaption></figure><ul><li>Incorporating RAT-SQL with BERT shows a substantial improvement in performance and yields 65.6% accuracy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*xYgi2dLoqYBYfy1KI-hrWA.png" /><figcaption>Figure 11: Accuracy on the Spider development and test sets, by difficulty from [3]</figcaption></figure><ul><li><strong><em>Currently, RAT-SQL+BERT tops the leaderboard of the benchmark dataset-SPIDER!</em></strong></li><li>In this article, we explored a few state-of-the-art models in the field of Text2SQL.</li><li>In the next part of the series, we will learn the famous AllenNLP library along with some final insights.</li></ul><h3>References</h3><p>[1] Guo, Jiaqi, et al. “Towards complex text-to-SQL in the cross-domain database with intermediate representation.” <em>arXiv preprint arXiv:1905.08205</em> (2019).</p><p>[2] Zhang, Rui, et al. “Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions.” <em>arXiv preprint arXiv:1909.00786</em> (2019).</p><p>[3] Wang, Bailin, et al. “Rat-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers.” <em>arXiv preprint arXiv:1911.04942</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf81a377d4d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/text2sql-part-4-state-of-the-art-models-cf81a377d4d2">Text2SQL — Part 4: State-Of-The-Art Models</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text2SQL — Part 3: Baseline Models]]></title>
            <link>https://medium.com/visionwizard/text2sql-part-3-baseline-models-85f14fba4122?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/85f14fba4122</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Fri, 24 Jul 2020 03:31:00 GMT</pubDate>
            <atom:updated>2020-07-24T03:31:00.906Z</atom:updated>
            <content:encoded><![CDATA[<h3>Text2SQL — Part 3: Baseline Models</h3><h4>Exploring a few baseline models in Text2SQL domain.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WYlL2taHkkxJS-Mq.png" /></figure><ul><li>In this blog, we will explore some of the primitive(baseline) models for performing Text2SQL task. At the time when SPIDER was released, SQLNet was the state-of-the-art model. SQLNet yielded 12.4% accuracy. <em>Pretty low right?</em></li><li>We will go through the transition of different models adopting different structures and algorithms. <em>Currently, the accuracy on SPIDER is around 66% on the test set!</em></li></ul><blockquote><em>1. Text2SQL — Part 1: Introduction</em></blockquote><blockquote><em>2. Text2SQL — Part 2: Datasets</em></blockquote><blockquote><strong><em>3. Text2SQL — Part 3: Baseline Models</em></strong></blockquote><blockquote><em>4. Text2SQL — Part 4: State-of-the-art Models</em></blockquote><blockquote><em>5.Text2SQL — Part 5: Final Insights</em></blockquote><ul><li>Several models were released after SQLNet for improving accuracy on SPIDER. A total of 44 models have been released until now!</li><li>The main aim of this blog is to help get familiarise with the common structures of baseline models that will make understanding of some complex models simpler.</li></ul><h3>Models to be covered in this blog:</h3><ol><li><em>SQLNet</em></li><li><em>SyntaxSQLNet</em></li><li><em>GrammarSQL</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*q1NzM05qsVnWVv6k.png" /><figcaption>Figure 1: Structure of the predictor model for aggregate clauses(max, min,etc.) (Source: Web)</figcaption></figure><h3>1. SQLNet</h3><ul><li>The model was developed with an aim to show that the use of reinforcement learning in Text2SQL tasks should be limited.</li><li>Before SQLNet, all the models in the literature employed reinforcement learning to improve the decoder results when it generated any of the appropriate serializations.</li><li>SQLNet overcomes the limitation by avoiding the seq2seq structure in cases when order is irrelevant.</li><li>The model uses a sketch-based approach in which a sketch consists of a dependency graph so that previous predictions can be taken into consideration for making the predictions.</li><li>The model also incorporates column attention <em>(attention refers to weights assigned to important words and phrases in sentences) </em>mechanism for improving the results.</li><li>Below flowchart explains the 3 phases for the generation of SQL queries for WikiSQL task used by SQLNet.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/0*HShBJlwqmoZCcNmx.png" /><figcaption>Figure 2: SQL query generation for WikiSQL task</figcaption></figure><h3>Sketch-based query synthesis</h3><ul><li>Bold tokens represent SQL keywords such as SELECT, WHERE, etc.</li><li>Tokens beginning with a $ sign represent that the slot is yet to be filled. And the token name represents the type of prediction.</li></ul><blockquote>For example, the $AGG slot can be filled with either an empty token or one of the aggregation operators, such as SUM and MAX. The $COLUMN and the $VALUE slots need be filled with a column name and a sub-string of the question respectively. The $OP slot can take a value from {=, }. The notion (…)∗ employ a regular expression to indicate zero or more AND clauses.</blockquote><blockquote><em>- </em><strong>Example for sketch syntax in [1]</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/0*dnHaSnqNq74wgQ76.png" /><figcaption>Figure 3: Sketch Syntax and dependency in a sketch (Source:[1])</figcaption></figure><h3>Column attention for sequence-to-set prediction</h3><ul><li>The idea of sequence-to-set prediction includes the prediction of column names of interest instead of generating a sequence of column names.</li><li>Column attention is a part of generic attention mechanism for computing the feature attention map on a question based on the column names.</li></ul><h3>Predicting WHERE and SELECT clause</h3><ul><li>Prediction of WHERE clause in Text2SQL task is believed to be the most difficult one.</li><li>SQLNet finds the columns that appear in WHERE clause and for each chosen column it finds the constraints by predictions the slots of OP and value as shown in SQL sketch.</li><li>OP slot is filled by one of the 3 classes:{&lt;,&gt;,=} and VALUE slot is predicted based on the substring from the natural language question</li><li>SELECT clause consists of a column name and an aggregator function like count, sum, max, etc. The only difference between SELECT and WHERE is the column name selection. In SELECT, only one column is chosen.</li></ul><blockquote>Note: For more training details and evaluation, refer to section 3.3.3 and 4 of [1].</blockquote><p><strong><em>Accuracy of SQLNet on WikiSQL test set is 64.4% and on SPIDER test set is around 12.4%.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*gUOpGgvm6LJiBVUkPBAiFg.png" /><figcaption>Figure 4: An example of a query executed by SQLNet on WikiSQL (Source: [1])</figcaption></figure><h3>2. SyntaxSQLNet</h3><ul><li>The main aim of developing SyntaxSQLNet model was to generate complex SQL queries with multiple clauses and generalisation to new databases.</li><li>The model is based on a syntax tree network in order to address complex and cross-domain queries. It uses table aware encoders along with decoders having SQL generation path history.</li><li>SyntaxSQLNet was evaluated on SPIDER dataset, which outperformed previous models such as SQLNet with a blooping increase in accuracy by 7.3%.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*LbxVdOJalP7LyHyDI3KSuw.png" /><figcaption>Figure 5: Tree-based SQL generator in SyntaxSQLNet (Source: [2])</figcaption></figure><ul><li>The accuracy is further improved by using a cross-domain data augmentation technique in order to generate more variance during the training.</li><li>The below chart consisting of various modules and their functionalities.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*2l_Gq5QrjyXeqXINeA9k6w.png" /><figcaption>Figure 6: Modules defined in SyntaxSQLNet model</figcaption></figure><h3>SQL Grammar and Attention Mechanism</h3><ul><li>For enabling the decoder to handle complex queries, SQL grammar is used. It is used to determine the module to be invoked at each recursive decoding step.</li><li>SQL path generation history and present SQL tokens are also used to predict the next SQL token.</li><li>For encoding the question representation, attention mechanism is also employed. Attention is also applied on SQL path history encoding.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*p15Iq7PvMwvdcdflYjX_mA.png" /><figcaption>Figure 7: Modules and SQL Grammar used in the decoding process. (Source:[2])</figcaption></figure><h3>Data Augmentation</h3><ul><li>Even though SPIDER dataset is large, it lacks complex queries.</li><li>Cross-domain datasets are used for data augmentation, in order to enable model for proper generalisation.</li><li>A list of patterns for natural language questions and corresponding SQL queries is prepared from different training databases of SPIDER dataset.</li></ul><blockquote>Note: For more training details and evaluation, refer to section 5 of [2].</blockquote><p><strong><em>SyntaxSQLNet model with decoding history and data augmentation reaches 27.2% accuracy on SPIDER.</em></strong></p><p><strong><em>The accuracy increase is 15% compared to former models like SQLNet.</em></strong></p><h3>3. GrammarSQL</h3><blockquote><em>The sequence-to-sequence paradigm employed by neural text-to-SQL models typically performs token-level decoding and does not consider generating SQL hierarchically from a grammar. -[3]</em></blockquote><ul><li>In [3], authors propose a grammar-based model for reducing the complexity of incorporating hierarchical grammar in text2SQL task.</li><li>They introduce a schema-dependent grammar with minimal over-generation technique.</li><li>The grammar developed in [3] covers 98% of instances present in ATIS and SPIDER datasets.</li></ul><h3>SQL Grammar</h3><ul><li>Shallow parsing expression grammar is designed in order to capture the least amount of SQL necessary to cover most of the instances given in the dataset.</li><li>For context-sensitive components of SQL, authors added additionally non-terminals to context-free grammar to ensure consistency of table, column and value references.</li><li>For ensuring proper joining of different tables together using a foreign key, they use runtime constraints during decoding to make sure that only valid programs are allowed.</li></ul><h3>Few details on the proposed model</h3><ul><li>The proposed model takes an utterance of natural language as an input, a database and grammar pertaining to that utterance.</li><li>After taking the input, string matching heuristics are applied to link words in the input to the identifiers or the tokens in the database.</li><li>After that, a concatenated string of the learned word and the link embedding for each token is sent as an input to the bidirectional LSTM.</li><li>The decoder then iteratively builds up the SQL query using attention mechanism on the input sequence.</li><li>Identifier anonymization is also applied for replacing the database identifiers that appear in both the natural language question and SQL query.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*qjX9Aci4LEWyCehI2JuFoA.png" /><figcaption>Figure 8: Structure of the proposed model (Source:[3])</figcaption></figure><blockquote><em>Note: For more experimentation details refer to Section 4 of [3].</em></blockquote><p><strong><em>GrammarSQL model was evaluated on ATIS and SPIDER datasets. It surpassed the accuracy of previous SOTA model SyntaxSQLNet by 14%.</em></strong></p><h3>References</h3><p>[1] Lin, Kevin, et al. “Grammar-based neural text-to-SQL generation.” <em>arXiv preprint arXiv:1905.13326</em> (2019).</p><p>[2] Yu, Tao, et al. “Syntaxsqlnet: Syntax tree networks for the complex and cross-domain text-to-SQL task.” <em>arXiv preprint arXiv:1810.05237</em> (2018).</p><p>[3] Xu, Xiaojun, Chang Liu, and Dawn Song. “Sqlnet: Generating structured queries from natural language without reinforcement learning.” <em>arXiv preprint arXiv:1711.04436</em> (2017).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85f14fba4122" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/text2sql-part-3-baseline-models-85f14fba4122">Text2SQL — Part 3: Baseline Models</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text2SQL — Part 2: Datasets]]></title>
            <link>https://medium.com/visionwizard/text2sql-part-2-datasets-4ef61359e90e?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ef61359e90e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 03:31:00 GMT</pubDate>
            <atom:updated>2020-07-16T03:31:00.918Z</atom:updated>
            <content:encoded><![CDATA[<h3>Text2SQL — Part 2: Datasets</h3><h4>Investigating datasets for Text2SQL tasks.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZUoBxpVdOjqfzuyINSX0g.png" /></figure><blockquote>You can have data without information, but you cannot have information without data. -Daniel Moran</blockquote><p>Along with other natural language processing tasks, Text2SQL highly relies on the type of dataset used. Different datasets with different structures, lengths, and queries have been created. There are a total of 9 datasets in the field of semantic parsing out of which SPIDER is the current benchmark dataset.</p><blockquote>1. Text2SQL — Part 1: Introduction</blockquote><blockquote><strong>2. Text2SQL — Part 2: Datasets</strong></blockquote><blockquote>3. Text2SQL — Part 3: Baseline Models</blockquote><blockquote>4. Text2SQL — Part 4: State-of-the-art Models</blockquote><blockquote>5.Text2SQL — Part 5: Final Insights</blockquote><h3><strong>Datasets to be covered in this blog:</strong></h3><ol><li><a href="https://github.com/howl-anderson/ATIS_dataset">ATIS</a></li><li><a href="http://www.cs.utexas.edu/users/ml/nldata/geoquery.html">GeoQuery</a></li><li><a href="http://ai.stanford.edu/~amaas/data/sentiment/index.html">IMDb</a></li><li><a href="https://github.com/jkkummerfeld/text2sql-data">Advising</a></li><li><a href="https://github.com/salesforce/WikiSQL">WikiSQL</a></li><li><a href="https://yale-lily.github.io/spider">Spider</a></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kjC7utuAX4AYxNwom3LdjA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKzJqeouHyPJlNE_oKcucw.png" /><figcaption>Figure 1: IMDb and SPIDER dataset logos (Source: Web)</figcaption></figure><p>The creation of every dataset was based on different tasks. For example, the ATIS dataset was designed to measure progress in spoken language systems that include both a speech and a natural language component.</p><p>Let’s discover more about all of them one after another….</p><h4>1. ATIS (Air Travel Information System) Dataset</h4><ul><li>ATIS corpus includes data collected from the official airline guide, organized under a relational schema.</li><li>It consists of 25 tables which contain information about fares, airlines, flights, cities, airports, and ground services. The questions associated with this dataset can be answered using a single relational query.</li><li>The relational database corresponding to this dataset is designed to answer queries in an intuitive manner, i.e. by using shorter tables for answers.</li><li>Example query from ATIS dataset: Input is in natural language form whereas the output is in λ-calculus form.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*5Yx4dToFVS_K4mP4GvuA-w.png" /><figcaption>Figure 2: Example from ATIS dataset for semantic parsing (Source: [3])</figcaption></figure><h4>2. GeoQuery Dataset</h4><ul><li>Geoquery dataset consists of information about United States geography. It has about 800 facts which are expressed in Prolog.</li><li>The database consists of information regarding States, cities, rivers, and mountains.</li><li>The attributes mainly compose of geographical and topographical attributes like capitals, population densities, etc.</li></ul><h4>3. IMDb Dataset</h4><ul><li>IMDb dataset is a huge collection of 50K reviews from IMDb. Reviews for each movie is restricted to 30.</li><li>Dataset consists of an equal number of positive and negative reviews.</li><li>The dataset creators have considered highly polarised reviews with a negative review with a score ≤ 4 out of 10 and a positive review with a score ≥ 7 out of 10.</li><li>Neural reviews are not taken into consideration while creating the dataset.</li><li>The dataset is equally divided for training and testing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*yI3uvCQvEAF8OoFo5MlQWA.png" /><figcaption>Figure 3: Database Structure of IMDb dataset (Source: Web)</figcaption></figure><h4>4. Advising Dataset</h4><ul><li>Advising dataset was created in order to propose improvements in text2SQL systems.</li><li>The creators of the dataset compare human-generated and automatically generated questions, citing properties of queries that relate to real-world applications.</li><li>Dataset consists of questions from university students about courses that lead to particularly complex queries. The records of students in the database are fictional.</li><li>The dataset includes student profile information such as the recommended courses, grades, and student’s previous courses.</li><li>Questions were formulated by students with knowledge of the database and were asked to frame questions they might ask in an academic advising meeting.</li><li>The dataset had many queries in common with ATIS, GeoQuery, and Scholar datasets.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*ghoeiO_ovPQ_3n4jNj_sJQ.png" /><figcaption>Figure 4:Example from Advising Dataset (Source: [1])</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*ZfxXXfCDaq0LctfDbzOakg.png" /><figcaption>Figure 5: Common Queries stats (Source: [4])</figcaption></figure><h4>5. WikiSQL dataset</h4><ul><li>WikiSQL is a massive semantic parsing dataset consisting of 80K+ natural language questions and respective SQL based queries on 24K+ tables extracted from Wikipedia.</li><li>The database in the test set does not appear in the train or development sets.</li><li>The dataset creators have made simplified assumptions about the SQL queries and databases. Thus, the dataset consists of SQL labels that only cover a single SELECT column and aggregation, and WHERE conditions. Moreover, all the databases only contain single tables.</li><li>Complex queries consisting of advanced operations like JOIN, GROUP BY, and ORDER BY, etc. are not included.</li><li>This dataset was considered as the benchmark dataset before the release of the SPIDER dataset. A lot of research has been done using WikiSQL.</li><li>Some of the previous states of the art models like SQLNet and SyntaxSQL prove to be very efficient on WikiSQL queries consisting of the “WHERE” clause, which is known to be the most difficult clause for prediction in semantic parsing tasks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*7rLs2_4uS8BaUdTJrYc5_w.png" /><figcaption>Figure 6: Example from WikiSQL dataset (Source: [6])</figcaption></figure><h4>6. SPIDER</h4><p>Finally, its time to explore the current benchmark dataset SPIDER. The dataset was created by a group of students from Yale University.</p><ul><li>SPIDER consists of 10K questions and 5K+ unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains.</li><li>It is distinct from previous datasets as it incorporates multiple datasets and the former ones(most of them) use only a single database.</li><li>The main motive of the creators was to prepare a corpus in order to tackle complex queries and the problem of generalizing across databases without multi-turn interactions required.</li><li>The dataset was created after 1000 man-hours! <em>Isn’t that huge?</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*eA_mW5UDLc9cpunBMnS8KQ.png" /><figcaption>Figure 7: The annotation process of our Spider corpus (Source: [5])</figcaption></figure><ul><li>The dataset consists of a few databases from the WikiSQL dataset. It consists of a complex structure as it links various tables using several foreign keys.</li><li>3 main aspects of the dataset creation include SQL pattern coverage, SQL consistency, and Question Clarity.</li><li>SQL queries in SPIDER consist of: <strong><em>SELECT with multiple columns and aggregations, WHERE, GROUP BY, HAVING, ORDER BY, LIMIT, JOIN, INTERSECT, EXCEPT, UNION, NOT IN, OR, AND, EXISTS, LIKE</em></strong> as well as nested queries.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*DtaMu5_TOmqkettbylr_Jw.png" /><figcaption>Example 8: Example of Question-Query set from SPIDER (Source: [5])</figcaption></figure><ul><li>The existing state of the art models when SPIDER was released gave an exact matching accuracy of 12.4%. This low accuracy shows that SPIDER presents a strong challenge in research.</li><li>The current best accuracy on SPIDER is around 66% with an exact set match without values(<em>refers to values in WHERE clause</em>) and around 63% with values.</li></ul><blockquote>More information on results from different models on SPIDER can be found <a href="https://yale-lily.github.io/spider">here</a>.</blockquote><p>So that’s all for the datasets. In part 3, we will be exploring some of the efficient models built on these datasets in the Text2SQL domain.</p><p>Stay tuned for more!</p><h3>References</h3><p>[1] Vig, Jesse, and Kalai Ramea. “Comparison of transfer-learning approaches for response selection in multi-turn conversations.” <em>Workshop on DSTC7</em>. 2019.</p><p>[2] Maas, Andrew, et al. “Learning word vectors for sentiment analysis.” <em>Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies</em>. 2011.</p><p>[3] Sun, Zeyu, et al. “A grammar-based structural CNN decoder for code generation.” <em>Proceedings of the AAAI Conference on Artificial Intelligence</em>. Vol. 33. 2019.</p><p>[4] Finegan-Dollak, Catherine, et al. “Improving text-to-SQL evaluation methodology.” <em>arXiv preprint arXiv:1806.09029</em> (2018).</p><p>[5] Yu, Tao, et al. “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task.” <em>arXiv preprint arXiv:1809.08887</em> (2018).</p><p>[6] Hwang, Wonseok, et al. “A comprehensive exploration on wikisql with table-aware word contextualization.” <em>arXiv preprint arXiv:1902.01069</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ef61359e90e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/text2sql-part-2-datasets-4ef61359e90e">Text2SQL — Part 2: Datasets</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text2SQL — Part 1: Introduction]]></title>
            <link>https://medium.com/visionwizard/text2sql-part-1-introduction-1327cf7f5f51?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/1327cf7f5f51</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 03:31:01 GMT</pubDate>
            <atom:updated>2020-07-14T03:31:01.135Z</atom:updated>
            <content:encoded><![CDATA[<h3>Text2SQL — Part 1: Introduction</h3><h4>Introduction to the conversion of Natural Language to SQL</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CaNCI-d3tt8HDXs-GX6Iw.png" /></figure><blockquote>Text to SQL is a system that converts natural language statements to SQL queries. This can help in retrieving information stored in a database by expressing commands in natural language.</blockquote><ul><li>Text2SQL offers applications in various domains like healthcare, customer support and search engines, which requires elaborating of structured data having information on the text.</li><li>This proves to be useful as it helps to bridge the gap between non-technical users and database systems, as users do not require to understand the database schemas and query language syntax.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*rZWyLHxoYCFnh9nAayfP2Q.png" /><figcaption>Figure 1: Example of Text2SQL from WikiSQL dataset. (Source: WikiSQL)</figcaption></figure><p>We will be covering the following topics in Text2SQL series:</p><blockquote><strong>1. Text2SQL — Part 1: Introduction</strong></blockquote><blockquote>2. Text2SQL — Part 2: Datasets</blockquote><blockquote>3. Text2SQL — Part 3: Baseline Models</blockquote><blockquote>4. Text2SQL — Part 4: State-of-the-art Models</blockquote><blockquote>5.Text2SQL — Part 5: Final Insights</blockquote><ul><li>SQL(Structured query language) is one of the most used languages for querying the database of the retrieval of data. With the help of machine learning and knowledge-based resources, text language to SQL conversion is facilitated.</li><li>Various other methods like rule-based syntax analysis, semantic matching, and pattern matching are included in the NoSQL category.</li><li><strong><em>However, we limit our study in this series on the conversion of natural language into SQL only.</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*Nla6tx6G6mJZKycCKaoOWQ.png" /><figcaption>Figure 2: Classification of natural language to database querying frameworks. (Source: [2])</figcaption></figure><h3>Natural Language Processing technologies</h3><ul><li>In this section, we will go through some of the most commonly used NLP technologies which are relevant for the conversion of text language into Structured Query Language (SQL).</li><li>This section includes topics like handling of stop words and synonyms, tokenization, part of speech tagging, stemming, lemmatization and parsing.</li></ul><h4>1. Handling stop words and synonyms</h4><ul><li>Stop words refer to the most common words present in a language. Depending on the task, any group of words can be chosen as stop words. NLTK library provides various functions for handling and removal of stop words.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5JKRHVLX1TXWsD1wHyYt_Q.png" /><figcaption>Figure 3: Examples of stop words (Source: geeks for geeks)</figcaption></figure><ul><li>The difficulty associated with the usage of the synonyms is that a simple lookup or matching is not enough.</li></ul><blockquote>For example, the question ‘All movies starring Brad Pitt from 2000 until 2010.’ (Q3) could also be phrased as ‘All movies playing Brad Pitt from 2000 until 2010.’</blockquote><blockquote>The answer should be the same, but because in the sample world no element is named ‘playing,’ a lookup would not find an answer</blockquote><blockquote>-[1]</blockquote><ul><li>Thus, it is necessary to take account of all the possible synonyms. An easy way to incorporate them is by using dictionaries based on WordNet, etc.</li></ul><h4>2. Tokenization</h4><ul><li>Tokenization is used to split an input question into a varied list of tokens. It can be as simple as a separator on space but in most cases, it is based on multiple rules such as regex(regular expressions).</li><li>It is important to perform tokenization during the preprocessing state as it will reduce the mismatches during the conversion.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*GJ8R-FZT2V103JHVGgV53g.jpeg" /><figcaption>Figure 4: Example of Tokenization (Source: KDNuggets)</figcaption></figure><h4><strong>3. Part of Speech Tagging</strong></h4><ul><li>A part of speech is a category of words with similar grammatical properties. POS tagging is the process of annotating each token in a text with the corresponding POS tag.</li><li>This tagging is based on tokens and their contexts. Stanford POS tagger is a good example of a tool that can be used for the POS tagging task.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*a6FkMJ0A7aiK55DhZsnUrQ.png" /><figcaption>Figure 5: Parts of Speech (Source: Web)</figcaption></figure><h4>4. Stemming/Lemmatization</h4><ul><li>Stemming reduces correlated words to the same token by removing different endings of the words. Whereas, Lemmatization removes inflectional endings and gives a word corresponding to a base form of the word.</li><li>Stemming and Lemmatisation can be easily performed using the NLTK library.</li><li>One of the foundational algorithms for the stemming task is Porter’s algorithm. It is based on simple heuristics that are applied to the longest suffix.</li><li>However, there is a disadvantage of this technique. It is because it chops of words like “director” to “direct” which is erroneous.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*zX9Lr6pCY53rmZsVgL6mnw.png" /><figcaption>Figure 6: Example of Stemming and Lemmatization (Source: Web)</figcaption></figure><h4>5. Parsing</h4><ul><li>Parsing involves analyzing the grammar of a syntax. There are two main directions involved in parsing: dependency syntax direction (focuses on relations between words) and constituency syntax direction(pays attention to complex entities).</li><li>The information retrieved after the parsing stage is then represented as a syntax tree.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*bzRVEAKvoqMauo2vNQaLqQ.png" /><figcaption>Figure 7: Example of Parsing (Source: Web)</figcaption></figure><ul><li>The above listed preprocessing techniques are very useful in any NLP task especially in Text2SQL where the processing is important in order to retrieve context for SQL query generation.</li><li>SQL consists of different clauses like SELECT, WHERE, JOIN, and other aggregate functions. For every different clause, the text is handled in a different way.</li><li>For example, WHERE clause is always associated with a value. These values are mapped to output in different ways( to be discussed in Part 3 on different frameworks and state of the art models in Text2SQL) using a knowledge graph or by simply mapping with the noun corresponding in the question.</li></ul><p><strong><em>Question: What are the movies with the actor Brad Pitt?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*uAyrK3p8chP2LeRMCMy2kQ.png" /><figcaption>Example 8: Example of a query with different SQL clauses (Source: [1])</figcaption></figure><ul><li>Before diving deep into the frameworks used, it is important to learn the structures of different datasets employed for training these huge frameworks for such a challenging task.</li><li>In part 2, different datasets like ATIS, WIKISQL, and SPIDER will be covered.</li></ul><h3>References</h3><p>[1] Affolter, Katrin, Kurt Stockinger, and Abraham Bernstein. “A Comparative Survey of Recent Natural Language Interfaces for Databases.” The VLDB Journal 28.5 (2019): 793–819. Crossref. Web.</p><p>[2] Dar, Hafsa Shareef, et al. “Frameworks for Querying Databases Using Natural Language: A Literature Review.” <em>arXiv preprint arXiv:1909.01822</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1327cf7f5f51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/text2sql-part-1-introduction-1327cf7f5f51">Text2SQL — Part 1: Introduction</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SNIPER: Efficient Multi-Scale Training]]></title>
            <link>https://medium.com/visionwizard/sniper-efficient-multi-scale-training-826e90573b20?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/826e90573b20</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Sat, 04 Jul 2020 06:59:01 GMT</pubDate>
            <atom:updated>2020-07-05T08:46:28.855Z</atom:updated>
            <content:encoded><![CDATA[<h4>An understanding of a new algorithm for performing efficient multi-scale training in instance-level visual recognition tasks.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r1EWse50qdlkzZwLexbrbA.jpeg" /><figcaption>Source: Web</figcaption></figure><blockquote>Humans tend to pay attention to objects at a fixed distance and size. For instance, when we focus on nearby objects, far away objects get blurred. We only process a small field of view at any given point of time and adaptively ignore the remaining visual content in the image. However, computer algorithms which are designed for instance-level visual recognition tasks like object detection depart from this natural way of processing visual information.</blockquote><h3>Table Of Contents</h3><ol><li>Introduction</li><li>Background</li><li>Why adapt to SNIPER while training?</li><li>Major components of SNIPER</li><li>Comparison with State-Of-The-Art</li><li>Conclusion</li><li>References</li></ol><h3>1. Introduction</h3><ul><li>Large scale variation across object instances, and especially, the challenge of detecting very small objects stands out as one of the factors behind the difference in performance.</li><li>This variation in scale which a detector needs to handle is enormous and presents an extreme challenge to the scale invariance properties of convolutional neural networks.</li><li>In [1], authors present SNIPER, a novel training algorithm for performing efficient multi-scale training in instance-level visual recognition tasks.</li><li>SNIPER brings training of instance-level recognition tasks like object detection closer to the protocol for image classification and suggests that the commonly accepted guideline that it is important to train on high-resolution images for instance-level visual recognition tasks might not be correct.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*F3du8ok9K6rIW2q5UClsnQ.png" /><figcaption>Figure 1: Large Scale variations in visual recognition tasks (Source:slideshare.net)</figcaption></figure><h3>2. Background</h3><ul><li>Deep learning-based object detection algorithms have primarily evolved from the R-CNN detector, which resized object proposals to a canonical 224x224 size image and classified using a CNN.</li><li>Inference and detections from multiple scales are combined by selecting features from a pair of adjacent scales closer to the resolution of the pre-trained network in RCNN.</li><li>The Fast-RCNN model has since become <em>de-facto</em> approach for classifying region proposals as it is faster, captures more context and up/downsamples every proposal in the image.</li><li>SNIP is also trained on almost all the pixels of the image pyramid(like Fast-RCNN), although gradients from extreme resolutions are neglected.</li><li>SNIPER takes this cropping procedure to an extreme level by sampling 512 pixels of crops from 3 scales of an image pyramid.<em> </em><strong><em>At extreme scales (like 3x), SNIPER observes less than one-tenth of the original content present in the image!</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*utycBFMhsa0Cyum63agRSg.png" /><figcaption>Figure 2: The same layer convolutional features at different scales of the image are different and map to different semantic regions in the image at different scales. (Source:[2])</figcaption></figure><h3>3. Why adapt to SNIPER while training?</h3><ul><li>Since every pixel is processed at each scale, a multi-scale training approach to process visual information increases the training time significantly.</li></ul><blockquote>If this is the case, an intuitive question which arises is, do we need to process the entire image at a high resolution? Wouldn’t it suffice to sample a much smaller RoI (chip) around small objects at this resolution?</blockquote><blockquote>On the other hand, if the image is already high resolution, and objects in it are also large in size, is there any benefit in upsampling that image?</blockquote><ul><li>While ignoring significant portions of the image would save computation, a smaller chip would also lack the context required for recognition. A significant portion of the background would also be ignored at a higher resolution.</li><li>So, there is a trade-off between computation, context, and negative mining while accelerating multi-scale training.</li><li><strong><em>Scale Normalization for Image Pyramids with Efficient Resampling </em>(SNIPER)</strong>, which adaptively samples chips from multiple scales of an image pyramid, conditioned on the image content. It samples positive chips conditioned on the ground-truth instances and negative chips based on proposals generated by a region proposal network.</li></ul><h3>4. Major Components of SNIPER</h3><h4>4.1 Chip Generation</h4><ul><li>SNIPER generates chips Ci at multiple scales in the image. For each scale, the image is first re-sized to width (Wi) and height (Hi).</li><li>On this canvas, KxK pixel chips are placed at equal intervals of d pixels. This leads to a two-dimensional array of chips at each scale.</li></ul><h4>4.2 Positive Chip Selection</h4><ul><li>For each scale, there is a desired range that determines which ground-truth boxes participate in training for each scale.</li><li>Then, chips are greedily selected so that the maximum number of valid ground truth boxes are covered. A ground truth box is said to be covered if it is completely enclosed inside a chip.</li><li>Ground-truth instances that have partial overlap with a chip are cropped. All the cropped ground-truth boxes are retained in the chip and used in label assignment(Section 4.3).</li><li>Since the crop-size is much smaller than the resolution of the image, SNIPER does not process most of the background at high-resolutions. This leads to significant savings in computation and memory requirements while processing high-resolution images.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*sIfvSahCqpu2rCiVWSPfRA.png" /><figcaption>Figure 3: SNIPER Positive chip selection. SNIPER adaptively samples context regions (aka chips) based on the presence of the objects inside the image. (Source:[1])</figcaption></figure><h4>4.3 Negative Chip Selection</h4><ul><li>Although positive chips cover all the positive instances, a significant portion of the background is not covered by them. Incorrectly classifying background increases the false positive rate.</li><li>In [1], the authors posit that a significant amount of the background is easy to classify, and hence, performing computation can be avoided in those regions.</li><li>For negative chip selection, for each scale, they first remove all the proposals which have been covered in positive chip selection.</li><li>Then, for each scale, they greedily select all the chips which cover at least M proposals in range. This generates a set of negative chips for each scale per image.</li><li>As illustrated, SNIPER only processes regions that likely contain false positives, leading to faster processing time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y-R8ELYClRh7_gbkbqhn7A.png" /><figcaption>Figure 4: SNIPER negative chip selection. First row: the image and the ground-truth boxes. Bottom row: negative proposals not covered in positive chips (represented by red circles) and the generated negative chips based on the proposals (represented by orange rectangles). (Source:[1])</figcaption></figure><h4>4.4 Label assignment</h4><ul><li>While training, generated proposals are assigned labels and bounding box targets based on all the ground-truth boxes which are present inside the chip.</li><li>Proposals that do not fall in the range are ignored during training. So, a large ground-truth box that is cropped could generate a valid proposal that is small.</li><li>Like Fast-RCNN, authors mark any proposal which has an overlap greater than 0.5 with a ground-truth box as positive and assigns bounding-box targets for the proposal.</li></ul><blockquote>For detailed Experimentation setup and Recall analysis refer to Section 4 Experimental Details in [1].</blockquote><h3>5. Comparison with State-Of-The-Art</h3><ul><li>It is difficult to fairly compare different detectors like ResNet, ResNext, and Xception as they differ in backbone architectures.</li><li>Therefore, the authors compare their results with SNIP, which is a recent method for training object detectors on an image pyramid.</li><li>While SNIP processes almost all the image pyramid, SNIPER, on the other hand, reduces the computational cost by skipping easy regions.</li><li>Moreover, since SNIPER operates on a lower resolution input, it reduces the memory footprint. This allows us to increase the batch size and unlike SNIP, we can benefit from batch normalization during training.</li><li>With batch normalization, SNIPER significantly outperforms SNIP in all metrics.</li></ul><h3>6. Conclusion</h3><ul><li>In [1], the authors presented an algorithm for efficient multi-scale training that sampled low-resolution chips from a multi-scale image pyramid to accelerate multi-scale training by a factor of 3 times.</li><li>In doing so, SNIPER did not compromise on the performance of the detector due to effective sampling techniques for positive and negative chips.</li><li>As SNIPER operates on re-sampled low-resolution chips, it can be trained with a large batch size on a single GPU which brings it closer to the protocol for training image classification.</li><li>This is in contrast with the common practice of training on high-resolution images for instance-level recognition tasks.</li></ul><h3>7. References</h3><p>[1] Singh, Bharat, Mahyar Najibi, and Larry S. Davis. “Sniper: Efficient multi-scale training.” <em>Advances in neural information processing systems</em>. 2018.</p><p>[2] Singh, Bharat, and Larry S. Davis. “An analysis of scale invariance in object detection snip.” <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2018.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826e90573b20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/sniper-efficient-multi-scale-training-826e90573b20">SNIPER: Efficient Multi-Scale Training</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TransCoder from Facebook AI]]></title>
            <link>https://medium.com/visionwizard/transcoder-from-facebook-ai-902042baed?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/902042baed</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-translation]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 03:31:01 GMT</pubDate>
            <atom:updated>2020-06-30T03:31:01.076Z</atom:updated>
            <content:encoded><![CDATA[<h3>Facebook AI TransCoder</h3><p><em>A highly efficient model for translating various programming languages from Facebook AI.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SRcoCQn1s15iQqg7os2QFA.jpeg" /><figcaption>Source: Web</figcaption></figure><blockquote>The Commonwealth Bank of Australia spent around $750 million and 5 years of work to convert its platform from COBOL to Java. -<strong>[1]</strong></blockquote><blockquote>In natural language, recent advances in neural machine translation have been widely accepted, even among professional translators, who rely more and more on automated machine translation systems. A similar phenomenon could be observed in programming language translation[1].</blockquote><h3>Table Of Contents</h3><ol><li>Introduction</li><li>Background</li><li>Model</li><li>Preprocessing</li><li>Evaluation</li><li>Results</li><li>Examples of Translation</li><li>References</li></ol><h3>1. Introduction</h3><ul><li>A Transcompiler is used as a source to source translation system, which is used to convert source code from one language to another. They rely on handwritten rules applied to the source code <a href="https://www.sciencedirect.com/topics/computer-science/abstract-syntax-tree">abstract syntax tree</a>.</li><li>The translation process is time-consuming as well as cumbersome. It requires extensive knowledge of both the source and target languages.</li><li>Although neural machine translation systems can prove to be successful, their applications have been limited due to the shortage of parallel data available in the domain.</li></ul><h3>2. Background</h3><ul><li>The main aim is to translate an existing codebase written in an obsolete or deprecated language to a recent one or to integrate code written in a different language to an existing codebase.</li><li>In <strong>[1]</strong>, they present a model that efficiently translates functions between 3 languages(c++, Python, JAVA) based on monolingual source code. They assert that generalization to different languages is also possible.</li><li>They make use of open-source codes from Github Projects. The methodology used in [1] is an unsupervised machine translation. This is because it is difficult to create a parallel corpus for training purposes in case of a supervised model.</li><li>For evaluating the model, they prepare and release a test set of 852 parallel functions.</li></ul><h3>3. Model</h3><ul><li>For TransCoder, a sequence-to-sequence model with attention, composed of encoder and decoder with a <strong>transformer</strong> architecture, is used.</li><li>A single shared model is used for all languages. The training is done using three principles: initialization, language modeling, and back-translation.</li></ul><h4>3.1 Cross programming language model pretraining</h4><ul><li>Pre-training is a crucial component of unsupervised machine translation that ensures the sequences with a similar meaning are mapped to the same latent representation, regardless of their languages.</li><li>Cross-lingual word embeddings can be obtained by training monolingual word embeddings(common keywords such as <em>for, while, if, try,etc.</em>) and aligning them in an unsupervised manner.</li></ul><p>The pretraining strategy used here includes masked language modeling in which input stream of source code sequence is randomly masked and fed into a TransCoder to predict the tokens that have been masked out based on their contexts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*V5SyVZD6-RWW6jIC2iZqvA.png" /><figcaption>Figure 1: Illustration of the three principles of unsupervised machine translation used in [1]</figcaption></figure><h4>3.2 Denoising Auto Encoding (DAE)</h4><blockquote><strong>Cross-Lingual Model (XLM)</strong> <br>XLM uses a known pre-processing technique Byte-Pair Encoding (BPE) (that splits the input into the most common sub-words across all languages, thereby increasing the shared vocabulary between languages) and a dual-language training mechanism with BERT in order to learn relations between words in different languages. A detailed explanation can be found <a href="https://towardsdatascience.com/xlm-enhancing-bert-for-cross-lingual-language-model-5aeed9e6f14b">here</a>.</blockquote><ul><li>XLM pre-training allows the seq2seq model to generate high-quality representations of the input sequence. However, the decoder cannot translate as it has never been trained to decode a sequence based on a source representation.</li><li>DAE objective operates like a supervised machine translation algorithm, where the model is trained to predict a sequence of tokens given a corrupted version of the sequence(by randomly masking, removing and shuffling input tokens).</li><li>It also trains the language modeling aspect of the model, i.e., decoder is always trained to generate a valid function even when the encoder output is noisy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NpqF72CKhqHEU8WUprV2CQ.png" /><figcaption>Figure 2: XLM Training Model from [2]</figcaption></figure><h4>3.3 Back Translation</h4><p>In practice, XLM pre-training and denoising auto-encoding alone is enough to generate translations. However, the quality of these translations tends to be weak, as the model is never trained to do what it is expected to do at test time. To address, this issue, back translation is used to leverage the scenario.</p><blockquote>Note: Detailed experimentation setup can be referred from Section 4: Experiments of <strong>[1]</strong>.</blockquote><h3>4. Preprocessing</h3><ul><li>Preprocessing involves using a common tokenizer and a shared vocabulary for all languages, which will reduce the overall vocab size and maximizes the token overlap between languages, improving the cross linguality of the model.</li><li>A universal tokenizer would be suboptimal, as different languages use different patterns and keywords. The logical operators such as &amp;&amp; and || exist in C++, where they should be tokenized as a single token, but not in Python.</li><li>The indentations are critical in Python as they define the code structure but have no meaning in languages like C++ or Java. The tokenizers used are javalang5 tokenizer for Java, the tokenizer of the standard library for Python, and the clang7 tokenizer for C++.</li><li>These tokenizers ensure that meaningless modifications in the code (e.g., adding extra new lines or spaces) do not have any impact on the tokenized sequence.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*MuFyKkKVm6PndTKKOatkPQ.png" /><figcaption>Figure 3: Example of tokenization performed during Preprocessing in [1]</figcaption></figure><h3>5. Evaluation</h3><ul><li>Various coding problem solutions from GeeksForGeeks are used to extract a set of parallel functions in C++, JAVA, and Python for creating test and validation sets. These functions not only return the same output but also compute the result with similar algorithms.</li><li>The transCoder is evaluated with different metrics such as reference match(percentage of translations that perfectly match the ground truth) and BLeU score(based on the relative overlap between the tokens in the translation and reference).</li><li>A limitation of these metrics is that they do not take into account the syntactic correctness of the equation.</li><li>Thus, a new metric, computational accuracy, is used to evaluate whether the hypothesis function generates the same outputs as the reference when given the same input.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*SrawpvXfVp-Wk2QtTNYXfQ.png" /><figcaption>Figure 4: Evaluation Metrics from [1]</figcaption></figure><h3>6. Results</h3><ul><li>Many failures come from compilation errors when the target language is Java or C++. It suggests that the model could be improved by constraining the decoder to generate compilable code.</li><li>Runtime errors mainly occur when translating from Java or C++ into Python. Since Python code is interpreted and not compiled, this category also includes syntax errors in Python.</li><li>The majority of remaining errors are due to the program returning the wrong output on one or several of the unit tests.</li><li>Infinite loops generally cause timeout errors and mainly occur in the Java-Python pair.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*0eBgRLbjQxb3nBRKVGfQ8Q.png" /><figcaption>Figure 5: Detailed results for all possible translation pairs from [1]</figcaption></figure><h3>7. Examples of translation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCYdu784J0rErERc3RTZAg.png" /><figcaption>Figure 6: Example of a correct translation from [1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LcJ0OgA7thvtONkzq_uJ_Q.png" /><figcaption>Figure 7: Example of an incorrect interpretation from [1]</figcaption></figure><h3>8. References</h3><p>[1] Lachaux, Marie-Anne, et al. “Unsupervised Translation of Programming Languages.” <em>arXiv preprint arXiv:2006.03511</em> (2020).</p><p>[2] Lample, Guillaume, and Alexis Conneau. “Cross-lingual language model pretraining.” <em>arXiv preprint arXiv:1901.07291</em> (2019).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=902042baed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/visionwizard/transcoder-from-facebook-ai-902042baed">TransCoder from Facebook AI</a> was originally published in <a href="https://medium.com/visionwizard">VisionWizard</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Video Question Answering using BERT Representations]]></title>
            <link>https://medium.com/clique-org/video-question-answering-using-bert-representations-29b74d05877d?source=rss-af78b7628cac------2</link>
            <guid isPermaLink="false">https://medium.com/p/29b74d05877d</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[visual-question-answering]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Devshree Patel]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 13:21:02 GMT</pubDate>
            <atom:updated>2020-06-27T13:23:26.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*tMBjM0fJwPtrnMbz8sk76w.png" /><figcaption>Figure 1: Proposed Model for Video QA based on BERT from [1]</figcaption></figure><h3><strong>1. Introduction</strong></h3><p>As of today, the field of Natural Language Processing has leveraged some of its important tasks using a sequential modelling technique based on Transformers. One of the domains which connect natural language processing with computer vision is Visual Question Answering(VQA). The main objective of VQA is to answer the questions based on an image or a video. Most of the work in VQA is targetted on image-based question answering, and less focus has been paid to video-based question answering. VQA in video presents numerous challenges as it not only requires a sequence of visual features to model over time but it also needs to take corresponding subtitles into consideration.</p><h3>2. Datasets</h3><p>In [1] the authors exhaustively study the performance for their <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a> based model by taking different input arrangements. They show a significant improvement on 2 well-known video VQA datasets such as TVQA and Pororo. Links to the datasets can be found here :</p><ol><li><a href="http://tvqa.cs.unc.edu/">TVQA</a></li><li><a href="https://github.com/Kyung-Min/PororoQA">Pororo</a></li></ol><h3>3. Proposed Framework</h3><p>The framework used in [1] aims at answering various multiple-choice questions based on the videos. Visual and language information is processed in two independent flows, which are then fused to obtain the answer prediction. In visual flow, they represent the visual semantic information from each video frame as the set of objects and features such as <em>grey pants, blonde hair,</em> etc. that appear in the scene. They call it a visual concept. In language flow, the language semantic information is fetched from the corresponding subtitles. For each flow, visual concept attributes, subtitles, question and candidate answer are processed together with BERT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*Fpo-vsa_eswK9y-ek9ZJFA.png" /><figcaption>Figure 2: The framework of the proposed model from [1]</figcaption></figure><h4><strong>3.1</strong> Feature Representations and Predictions</h4><p>They use 2 different BERT networks to predict the answer to each question based on the extracted information from the visual concept attributes and subtitles.</p><p><strong>3.1.1 Visual Representations</strong></p><p>They extract visual concept features using <a href="https://arxiv.org/pdf/1506.01497.pdf">Faster R-CNN</a> fine-tuned on Visual Genome. In every extracted frame, the visual concept features are represented by corresponding words or noun phrases. The unique visual concept features from a whole scene are then obtained by aggregating the visual concepts from all the frames and removing duplicates. Then, the question, the unique visual concept features and each candidate answer are concatenated and rearranged into a single string. The obtained string is then tokenised to obtain a sequence which is fed into the BERT network, which outputs a matrix containing the vector representation of each word in the input sentence. The output vector is then fed to Fully Connected Layer to obtain visual flow prediction.</p><p><strong>3.1.2 Language Representations</strong></p><p>Similar to the visual representations, subtitles are concatenated with question and answer to form a string in the language flow. The rearranged string is tokenised to form the sequence of tokens which is fed into BERT to obtain language flow prediction.</p><p><strong>3.1.3 Combining predictions from both the flows</strong></p><p>The predictions of visual and language flow for each answer are summed together and softmax is used to convert the summed vector into answer scores. The answer with the maximum score is selected as the final answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*5HgAef9wAl_CpQ70ihErAA.png" /><figcaption>Figure 3: An example of input representations using BERT from [1]</figcaption></figure><blockquote>Note: For detailed experimentation setup refer to Section 4: Experiments from [1].</blockquote><h3>4. BERT vs LSTM</h3><p>The proposed model is able to solve both visual and language-related questions that cannot be solved by LSTM based model. However, when the answer is not explicit in the video frames or the subtitles and needs outside knowledge sources, the model gives a bad prediction. The reason may be that BERT uses a self-attention bidirectional structure, making every word to attend its context on both sides, while in LSTM, the follow-up words in the long sentence may have a weak attendance to its long previous words. However, the attention mechanism in TVQA model helps to pinpoint relevant words for correctly predicting questions.</p><h3>5. Results</h3><p>The evaluation was done with different techniques like full-length elements(visual concepts and subtitles without timestamp), different sequence length elements(different length of input tokens) and with minor embedding changes(by removing stop words and other indicator tokens).</p><p>Results show that the model gave correct predictions from the language and visual representations on TVQA and Pororo datasets. It improved the performance by 5.09% compared to the previous work based on LSTM, and 3.34% compared with the STAGE network. However, the model presents some limitations when using full-length subtitles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*vWQBChgdR-kgokFXaONBcw.png" /><figcaption>Figure 4: An example of a successful prediction of the proposed framework from [1]</figcaption></figure><h3>6. Conclusion</h3><p>In this paper, the authors proposed an improved framework for video question answering tasks. They use language representations for visual concept features and subtitles based on BERT to capture the semantics in both the video scenes and subtitles more accurately. They aim to explore the use of attention mechanism for identifying the relevant part of the long full-length subtitles in future.</p><h3>7. Bibliography</h3><p>[1] Yang, Zekun, et al. “BERT Representations for Video Question Answering.” <em>The IEEE Winter Conference on Applications of Computer Vision</em>. 2020.</p><p>[2] Lei, Jie, et al. “Tvqa: Localized, compositional video question answering.” <em>arXiv preprint arXiv:1809.01696</em> (2018).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29b74d05877d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/clique-org/video-question-answering-using-bert-representations-29b74d05877d">Video Question Answering using BERT Representations</a> was originally published in <a href="https://medium.com/clique-org">Clique Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>