<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aastha Singh on Medium]]></title>
        <description><![CDATA[Stories by Aastha Singh on Medium]]></description>
        <link>https://medium.com/@aastha-eng?source=rss-5f308360d2e5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*svMc_wUu7yYLhXIUR_JMQA.jpeg</url>
            <title>Stories by Aastha Singh on Medium</title>
            <link>https://medium.com/@aastha-eng?source=rss-5f308360d2e5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 22:40:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aastha-eng/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[English to Hindi Neural Machine Translation with NVIDIA NeMo toolkit]]></title>
            <link>https://aastha-eng.medium.com/english-to-hindi-neural-machine-translation-with-nvidia-nemo-toolkit-df11ed854fa1?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/df11ed854fa1</guid>
            <category><![CDATA[language-translation]]></category>
            <category><![CDATA[translation]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[nvidia]]></category>
            <category><![CDATA[machine-translation]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Sat, 09 Apr 2022 11:04:27 GMT</pubDate>
            <atom:updated>2022-04-09T11:04:27.397Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This blog aims to walk through the process of building and understanding a Neural Machine Translation model for English to Hindi language. In what follows, I shall bring upon the most advance techniques to sketch the importance of NVIDIA NeMo, a toolkit for building AI applications using Neural Modules.</p><h3>Machine Translation : Connexion between Computational Linguistics &amp; NLP</h3><p>Natural Language Processing combines computational linguistics, rule-based modeling of human language with statistics, machine learning, and deep learning to enable computers to understand the human language which can be in the form of text or voice data.</p><blockquote>Machine Translation (MT) is the task of automatically converting one natural language into another, preserving the meaning of the input text, and producing fluent text in the output language.</blockquote><h4>Impact of Machine Translation advancements in cultural context</h4><p>Culture represents the totality of a society’s distinctive beliefs, knowledge and values. It exhibits the ways humans interpret their environments. Hence, it is important to analyze the effectiveness of translating languages in cultural context to improve research in Machine Translation systems.</p><p>The current model is trained by mapping the <strong>source texts in English</strong>, as the input texts, into the <strong>target language in Hindi</strong>, as the output texts. Thus aiming to make it widely accessible to over <strong>1 Billion</strong> speakers.</p><h3>Progress in Machine Translation</h3><p>Below are the early developments and the challenges involved leading to the current advancements in Machine Translation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*gGgcI7JTNvT-lJBs-i4z7g.png" /><figcaption>Image : Progress in Machine Translation</figcaption></figure><h3>Configuring NMT with NVIDIA NeMo</h3><h4>NMT Model</h4><p>This blog uses an Encoder Decoder NMT model available in NeMo to train a 6x6 transformer for translation, named as AAYN (Attention is All You Need). It is using <strong>YouTokenToMe</strong> tokenizer</p><p>It relies on transformers for sequence transduction, that is transforming an input sequence to an output sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*6JU8_XWoVX32NjBa1ajzBA.jpeg" /></figure><h4>NVIDIA NeMo —</h4><p><a href="https://github.com/NVIDIA/NeMo">NVIDIA NeMo</a> (<strong>Ne</strong>ural <strong>Mo</strong>dules) is an open-source toolkit based on PyTorch that allows us to quickly build, train, and fine-tune conversational AI models.</p><p>The structure of NeMo consists of —</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*dipTj_6MpQVa31ePXIWZow.png" /><figcaption>Fig : Framework agnostic structure of NeMo. <a href="https://arxiv.org/pdf/1909.09577.pdf">Source</a></figcaption></figure><p><strong>NeMo Core</strong><em> </em>: It facilitates a neural type system, capable of semantic checks. <strong>NeMo Collections:</strong> Supports pre-built modules for conversational AI — <em>nemo_asr</em> and <em>nemo_nlp</em> to make building and re-using deep neural networks eaiser.</p><p><strong>Unique features of NeMo</strong> —</p><ul><li>You can <strong>define models directly using NeMo API </strong>as NeMo does not impose any particular structure and does not require configuration files to be in any particular format.</li><li><strong>NeMo is Python framework-agnostic </strong>even though current implementations rely on PyTorch. You can create your own Neural Models (NMs) by combining existing NMs or providing an implementation in a particular frame-work.</li><li><strong>High performance training</strong> — It automatically supports <em>mixed precision </em>training, with <em>fp16 </em>using NVIDIA Apex for computationally intensive operations.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwBgpMf_KQVw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwBgpMf_KQVw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FwBgpMf_KQVw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d033a654aae0703785bb577c686757f0/href">https://medium.com/media/d033a654aae0703785bb577c686757f0/href</a></iframe><h3>Dataset</h3><h4>About Dataset</h4><p>NVIDIA NeMo’s NMT model is trained on <a href="https://arxiv.org/abs/2104.05596v3">samanantar dataset</a>. <em>Samanantar</em> is the largest publicly available parallel corpora collection for Indic languages.</p><blockquote>The collection contains a total of <strong>49.7 million sentence pairs</strong> between English and <strong>11 Indic languages</strong> (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu). Further, it contains 83.4 million sentence pairs between all <strong>55 Indic language pairs from the English-centric parallel corpus</strong> using English as the pivot language.</blockquote><h4>Download Dataset</h4><p>For English to Hindi translation, download and unzip the english-hindi corpus for training using following command :</p><pre>wget https://storage.googleapis.com/samanantar-public/V0.2/data/en2indic/en-hi.zip<br>unzip en-hi.zip</pre><p>Download benchmarks for validation and testing using following command :</p><pre>wget https://storage.googleapis.com/samanantar-public/benchmarks.zip<br>unzip benchamrks.zip</pre><h3>Prerequisites</h3><p>Before you begin using NeMo, it’s assumed you meet the following prerequisites.</p><ul><li>You have Python version 3.6, 3.7 or 3.8.</li><li>You have Pytorch version 1.8.1.</li><li>You have access to an NVIDIA GPU for training.</li></ul><h3>Dependency installation</h3><h4>1. Python dev and pip</h4><pre>sudo apt install python3-dev python3-pip</pre><h4>2. Cython</h4><pre>pip3 install cython</pre><h4>3. NeMo</h4><p>Clone NeMo repository and setup its dependencies</p><pre>git clone https://github.com/NVIDIA/NeMo.git<br>cd NeMo<br>./reinstall.sh</pre><h4>4. Apex</h4><p>Apex is used for mixed precision and distribued training.</p><pre>git clone https://github.com/NVIDIA/apex<br>cd apex<br>pip install -v --disable-pip-version-check --no-cache-dir --global-option=&quot;--cpp_ext&quot; --global-option=&quot;--cuda_ext&quot; ./</pre><h3>Data Preprocessing</h3><h4>1. Length and Ratio Filtering</h4><p>Transformers are focused towards learning sequences. Keeping this in mind, the sequence length of source and target languages should be comparable for optimal training.</p><p>The source and target sentences should be in a ratio lesser than or equal to 1.3</p><p>The rules for length and ratio filtering are as follows -</p><ul><li>src_len / tgt_len or tgt_len / src_len should be &lt; 1.3</li><li>Source and Target sequence lengths should be above 1 and less than 250</li></ul><p><strong>Moses Decoder</strong> is used for cleaning the corpus. To proceed with filtering :</p><ul><li><strong>Clone the repository</strong></li></ul><pre>git clone https://github.com/moses-smt/mosesdecoder data/mosesdecoder</pre><ul><li><strong>Run the perl script </strong>clean-corpus-n.perl</li></ul><pre>perl data/mosesdecoder/scripts/training/clean-corpus-n.perl -ratio 1.3 \<br>    en-hi/train \<br>    en hi \<br>    en-hi/train.lengthratio \<br>    1 250</pre><p>This will filter and create two files — train.lengthratio.en and train.lengthratio.hi</p><h4>2. Max character length normalisation</h4><p>The transformer6x6 is configured to be trained with max characher length of 512. Lines with total characters more than 512 need to removed.</p><p>The following python snippet cleans the dataset -</p><pre>wget https://gist.githubusercontent.com/Aasthaengg/ef05605e3227555215e489022e072ad5/raw/ce7fad611d8024ac48536fb0af72a9436fc42d95/max_len_filter.py</pre><pre>python3 max_len_filter.py</pre><p>At the end of this step, two filtered files called train.lengthratio.norm.en and train.lengthratio.norm.hi are created.</p><h4>Print the filtered dataset</h4><pre>head -5 en-hi/train.lengthratio.norm.en<br>head -5 en-hi/train.lengthratio.norm.hi</pre><p>The output should look something like this</p><pre>In reply, Pakistan got off to a solid start.<br>The Congress leader represents Sivaganga Lok Sabha segment from Tamil Nadu.<br>Police is present on the spot.<br>Prime Minister Narendra Modi is slated to campaign for BJP in second phase of Assembly elections in Udhampur and Poonch district of Jammu and Kashmir tomorrow<br>It has a battery backup of 3050mAh.</pre><pre>जिसके जवाब में पाक ने अच्छी शुरुआत की थी.<br>कांग्रेस नेता तमिलनाडु से शिवगंगा लोकसभा क्षेत्र का प्रतिनिधित्व करते हैं.<br>पुलिस मौके पर मौजूद है।<br>जम्मू एवं कश्मीर विधानसभा के लिए दूसरे चरण के तहत होने वाले मतदान के लिए प्रधानमंत्री नरेंद्र मोदी भारतीय जनता पार्टी (भाजपा) के पक्ष में शुक्रवार को यहां रैलियां करेंगे।<br>और 3050एमएएच की दमदार बैटरी से लैस है।</pre><h4>3. BPE Tokenizer and Tarred Dataset</h4><p>create_tarred_parallel_dataset.py is used to train a BPE Tokenizer and created tarred files out of the filtered corpora.</p><h4><em>Understanding BPE Tokenizer</em></h4><p>Byte-Pair Encoding Tokenizer is a sub-word tokenizer that performs data compression by finding sequences of bytes common across the input and replaces that sequence with a byte that does not exist in the dataset.</p><p>BPE ensures that the most common words are represented in the vocabulary as a single token while the rare words are broken down into two or more subword tokens and this is in agreement with what a subword-based tokenization algorithm does. It is a greedy algorithm that tries to compress the data and create sub-word tokens with each iteration.</p><p>The variant of BPE used in NeMo for NMT is <a href="https://github.com/VKCOM/YouTokenToMe"><strong>YTTM</strong></a><strong> (YouTokenToMe)</strong>. It is an unsupervised sub-word text tokenizer which has good computational efficieny. We prefer using this as the corpus size of Indic Languages from Samananthar is large and YTTM can create a tokenizer in acceptable time with great accuracy.</p><p>We train two BPE tokenizers — the first one is an encoder, which is for the input language English and the other is a decoder which is for the target indic language.</p><h4><em>English Tokenization example</em></h4><p>During tokenization, the most common sequences of characters are learnt. Common words are represented as a subword and an id is assigned to them. The more complex words are broken down into multiple subwords and a combination of ids can be used to represent them.</p><h4><em>Hindi Tokenization example</em></h4><p>Just like with English, common words are represented as individual sub-words, with ids assigned to them and the more complex words are represented as a combination of ids.</p><h4><em>Tarred files</em></h4><p>Once the tokenizer is created, the filtered data is tokenized and stored in the form of .npy files, which are further compressed into .tar files. This ensures faster training as tokenization need not be performed per epoch for the entire dataset.</p><p>Copy the script create_tarred_parallel_dataset.py to the current directory</p><pre>cp examples/nlp/machine_translation/create_tarred_parallel_dataset.py .</pre><p>Run the following command to begin the creation of BPE tokenizer and subsequently, creation of tarred files</p><pre>python3 create_tarred_parallel_dataset.py \<br>    --src_fname en-hi/train.lengthratio.norm.en \<br>    --tgt_fname en-hi/train.lengthratio.norm.hi \<br>    --out_dir tarred_dataset_en_hi_8k_tokens \<br>    --clean \<br>    --encoder_tokenizer_name yttm \<br>    --encoder_tokenizer_vocab_size 32000 \<br>    --encoder_tokenizer_coverage 0.999 \<br>    --encoder_tokenizer_bpe_dropout 0.1 \<br>    --decoder_tokenizer_name yttm \<br>    --decoder_tokenizer_vocab_size 32000 \<br>    --decoder_tokenizer_coverage 0.999 \<br>    --decoder_tokenizer_bpe_dropout 0.1 \<br>    --max_seq_length 512 \<br>    --min_seq_length 1 \<br>    --tokens_in_batch 8000 \<br>    --lines_per_dataset_fragment 100000 \<br>    --num_batches_per_tarfile 20</pre><ul><li>--src_fname - Source language, i.e English.</li><li>--tgt_fname - Target language, i.e Hindi.</li><li>--encoder_tokenizer_name - Tokenizer to use. yttm is used for our training.</li><li>--decoder_tokenizer_name - Tokenizer to use. yttm is used for our training.</li><li>--encoder_tokenizer_vocab_size - Vocabulary size for the encoder tokenizer.</li><li>--decoder_tokenizer_vocab_size - Vocabulary size for the decoder tokenizer.</li><li>--max_seq_length - Maximum length of a sentence.</li><li>--min_seq_length - Minimum length of a sentence.</li></ul><h3>Training</h3><p>Now that the dataset is preared, we can start training. Here’s a snippet from the <em>config file</em> tha shows the transformer setup.</p><pre>optim:<br>    name: adam<br>    lr: 0.001<br>    betas:<br>      - 0.9<br>      - 0.98<br>    weight_decay: 0.0<br>    sched:<br>      name: InverseSquareRootAnnealing<br>      min_lr: 0.0<br>      last_epoch: -1<br>      warmup_ratio: 0.1<br><br>  encoder:<br>    library: nemo<br>    model_name: null<br>    pretrained: false<br>    max_sequence_length: 512<br>    num_token_types: 0<br>    embedding_dropout: 0.1<br>    learn_positional_encodings: false<br>    hidden_size: 512<br>    num_layers: 6<br>    inner_size: 2048<br>    num_attention_heads: 8<br>    ffn_dropout: 0.1<br>    attn_score_dropout: 0.1<br>    attn_layer_dropout: 0.1<br>    hidden_act: relu<br>    mask_future: false<br>    pre_ln: false<br>    pre_ln_final_layer_norm: true<br><br>  decoder:<br>    library: nemo<br>    model_name: null<br>    pretrained: false<br>    max_sequence_length: 512<br>    num_token_types: 0<br>    embedding_dropout: 0.1<br>    learn_positional_encodings: false<br>    hidden_size: 512<br>    inner_size: 2048<br>    num_layers: 6<br>    num_attention_heads: 8<br>    ffn_dropout: 0.1<br>    attn_score_dropout: 0.1<br>    attn_layer_dropout: 0.1<br>    hidden_act: relu<br>    pre_ln: falsez<br>    pre_ln_final_layer_norm: true</pre><ul><li>6 encoders and 6 decoders are used, i.e <strong>transformer 6x6.</strong></li><li>the activation function used is <strong>Rectified Linear Unit</strong> or ReLU</li><li>The number of self-attention heads is set to 8.</li><li>The optimiser used is <strong>Adaptive Momentum Estimation</strong> or adam. It combines Momentum and Root Mean Square Propgation allowing for a faster gradient descent with less memory.</li><li>For the learning rate scheduler, InverseSquareRootAnnealing. This keeps the learning rate high for the initial few epochs then exponentially decays it.</li></ul><h4><strong>Copy the config file </strong><strong>aayn_base.yaml into </strong><strong>conf directory</strong></h4><pre>mkdir -p conf<br>cp examples/nlp/machine_translation/conf/aayn_base.yaml conf/</pre><h4>Run the following command to start training -</h4><pre>python3 enc_dec_nmt.py --config-path=conf \<br>--config-name=aayn_base trainer.gpus=[0,1,2,3,4,5,6,7] \<br>~trainer.max_epochs +trainer.max_steps=100000 model.beam_size=4 \<br>model.max_generation_delta=5 model.label_smoothing=0.1 \<br>model.encoder_tokenizer.tokenizer_model=tarred_dataset_en_hi_8k_tokens/tokenizer.encoder.32000.BPE.model \<br>model.decoder_tokenizer.tokenizer_model=tarred_dataset_en_hi_8k_tokens/tokenizer.decoder.32000.BPE.model \<br>model.encoder.inner_size=2048 model.encoder.ffn_dropout=0.1 model.decoder.num_layers=6 \<br>model.decoder.hidden_size=512 model.decoder.inner_size=2048 model.decoder.num_attention_heads=8 \<br>model.decoder.ffn_dropout=0.1 \<br>model.train_ds.use_tarred_dataset=true \<br>model.train_ds.tar_files=tarred_dataset_en_hi_8k_tokens/parallel.batches.tokens.8000.OP_0..10300_CL.tar \<br>model.train_ds.metadata_file=tarred_dataset_en_hi_8k_tokens/metadata.tokens.8000.json \<br>model.train_ds.shard_strategy=scatter \<br>model.train_ds.tokens_in_batch=8000 \<br>model.validation_ds.src_file_name=benchmarks/wat2021-devtest/test.en \<br>model.validation_ds.tgt_file_name=benchmarks/wat2021-devtest/test.hi \<br>model.validation_ds.tokens_in_batch=8000 \<br>model.test_ds.src_file_name=benchmarks/wat2021-devtest/test.en \<br>model.test_ds.tgt_file_name=benchmarks/wat2021-devtest/test.hi \<br>model.optim.lr=0.001 \<br>+exp_manager.create_wandb_logger=True +exp_manager.wandb_logger_kwargs.name=TEST-nmt-base \<br>+exp_manager.wandb_logger_kwargs.project=bpe-en-hi-100 +exp_manager.create_checkpoint_callback=True \+exp_manager.checkpoint_callback_params.monitor=val_sacreBLEU +exp_manager.exp_dir=nmt_base \<br>+exp_manager.checkpoint_callback_params.mode=max</pre><ul><li>trainer.gpus - Set the index of GPUs to use for training.</li><li>model.encoder_tokenizer.tokenizer_model - Path to encoder tokenizer.</li><li>model.decoder_tokenizer.tokenizer_model - Path to decoder tokenizer.</li><li>model.train_ds.use_tarred_dataset - Enable use of tarred files.</li><li>model.train_ds.tar_files - Path to tar files.</li><li>model.validation_ds.src_file_name - Path to validation file for source language.</li><li>model.validation_ds.tgt_file_name - Path to validation file for target language.</li><li>model.test_ds.src_file_name - Path to test file for source language.</li><li>model.test_ds.tgt_file_name - Path to test file for target language.</li></ul><h4><em>Note for tar file paths :</em></h4><blockquote>The tar files are in the format parallel.batches.tokens.8000.XXX.tar. To mention a continous path, pass the model.train_ds.use_tarred_dataset argument with a value in the following format - parallel.batches.tokens.8000.OP_XX..YY_CL.tar where XX is the starting index and YY is the ending index.</blockquote><h3>Validation</h3><p>The validation data used is from WAT. Workshop on Asian Translation is a machine translation evaluation campaign that aims at creating a standardized validation data that can be used to evaluate different Machine Translation models and report accuracy differences over time.</p><p>The validation data is present in benchmarks/wat2021-devtest/ directory -</p><pre>➜ ls benchmarks/wat2021-devtest</pre><pre>dev.bn  dev.gu  dev.kn  dev.mr  dev.pa  dev.te   test.en  test.hi  test.ml  test.or  test.ta<br>dev.en  dev.hi  dev.ml  dev.or  dev.ta  test.bn  test.gu  test.kn  test.mr  test.pa  test.te</pre><p>Since we are training an NMT model for English to Hindi, only dev.en, dev.hi, test.en and test.hi are relevant.</p><h3>Conclusion</h3><p>In the second part of this blog, I will discuss evaluation with<strong> BLEU score</strong> and show a visual representation of our progress.<br>Thank you for reading all the way to the end of this article. Please let me know in the comments what worked best for you. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df11ed854fa1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speed up the development with advanced pair programming: GitHub Copilot]]></title>
            <link>https://medium.com/analytics-vidhya/speed-up-the-development-with-advanced-pair-programming-github-copilot-68eb6d30a993?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/68eb6d30a993</guid>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 14:55:09 GMT</pubDate>
            <atom:updated>2021-07-12T06:24:31.052Z</atom:updated>
            <content:encoded><![CDATA[<h4>Something Microsoft couldn’t get their hands off of, and you probably won’t be able to either!</h4><h4>Hello technophiles, Welcome to my blog! Read along to know political, technical, and public understandings and viewpoints on the great magical AI algorithm launched this week.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qiZ7WGg-YNJ2uueefhnfoA.jpeg" /></figure><h3>“Anyone who puts in money at OpenAI can only expect returns 100 times their investment”</h3><h4>Let’s talk about Microsoft investment of $1 billion in OpenAI</h4><p>OpenAI, a San Francisco-based research lab founded by Silicon Valley luminaries, including Elon Musk and Sam Altman, that’s dedicated to creating artificial general intelligence (AGI).</p><p>In order to restrain the greed of investors, OpenAI operates as a <strong>capped-profit entity.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/0*RbwyXyHp2vGzI63v" /><figcaption>The CEO moment : <a href="https://medium.com/u/22acd7905c72">Sam Altman</a> and <a href="https://medium.com/u/d989edf5a0ea">Satya Nadella</a></figcaption></figure><p>The investment has made Microsoft the “exclusive” provider of cloud computing services to OpenAI, and the two companies will work together to develop new technologies, to explore artificial intelligence’s<em> magic formula.</em></p><h3>GPT -3 and capitalism; Could there be a boom followed by a bust?</h3><p>We’ve noticed that politicians are lured by the enormous tax revenue generated by CEOs, as well as the popularity gained by paying their voters’ wages (UBI). Politicians believe Sam is no different from any other capitalist attempting to persuade the government to accept an oligarchy.</p><p><a href="https://medium.com/u/22acd7905c72">Sam Altman</a> suggested <strong>wealth taxation strategies</strong>, mentioning</p><blockquote>“The American Equity Fund would be capitalised by taxing companies above a certain valuation 2.5% of their market value each year, payable in shares transferred to the fund, and by taxing 2.5% of the value of all privately-held land, payable in dollars.”</blockquote><p>To this, Daron Acemoglu, an economist at MIT and Richard Miller, founder of tech consultancy firm Miller-Klein Associates, told <strong>CNBC</strong>:</p><blockquote>“Altman’s post feels muddled and the model is unfettered capitalism.”</blockquote><blockquote>“There is an incredible mistaken optimism of what AI is capable of doing.”</blockquote><h3>Recipe to Copilot:Architecture of GPT-3</h3><p>GitHub Copilot has been trained on billions of lines of publicly available code. While we amusingly explore the consequences, the processing behind it is ultimately based on GPT-3, so let’s see how it works.</p><p><em>To understand the architecture of Transformer based Language Model better, I want you to follow the questions that lead to the following explanations :</em></p><h3>How does the model take input and what are the inputs and outputs of GPT?</h3><h4>1. Defining Input and output :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/0*SYmpp7HiMPSWVC70.png" /><figcaption><a href="https://arxiv.org/abs/1706.03762v5">Source</a></figcaption></figure><blockquote>Input sequence — get the next word.</blockquote><p><strong>Input:</strong></p><p>For GPT-3, the input is a sequence of 2048 words (a.k.a tokens).</p><p><strong>Output:</strong>Once we give the input, we get the guess for the next word as the output.</p><p>The GPT output is not just a single guess, it’s a sequence (length 2048) of guesses (a probability for each likely word). Keep repeating to get long generated texts. <a href="https://dugas.ch/blog.html">Example</a> :</p><blockquote><strong>Not all heroes wear capes</strong> -&gt; <em>but</em><br><strong>Not all heroes wear capes but</strong> -&gt; <em>all</em><br><strong>Not all heroes wear capes but all </strong>-&gt; <em>villains</em><br><strong>Not all heroes wear capes but all villains</strong> -&gt; <em>do</em></blockquote><h3>Machine does not understand words, so how do we turn words into vectors?</h3><h4>2. Byte Pair Encoding</h4><p>GPT-3 uses byte-level Byte Pair Encoding (BPE) tokenization for efficiency. This indicates that the vocabulary’s “words” aren’t whole words, but rather groups of characters (for byte-level BPE, bytes) that appear frequently in text.</p><ul><li><strong>Step 1:</strong> Keep a vocabulary of all words, which will allows us to give each word a value.</li><li><strong>Step 2:</strong> Turn each word into a <strong>one-hot encoding vector .</strong></li></ul><blockquote><strong><em>GPT has a vocabulary of 50257 words, </em></strong>we do this for every word in the sequence which results in a 2048 x 50257 matrix of ones and zeroes.</blockquote><h3>The matrix generated is mostly filled with zeros, which is a lot of wasted space. How will we solve this?</h3><h4>3.Embedding</h4><p><strong><em>GPT uses 12288 dimensions.</em></strong></p><p>Dimensions are basically properties<strong>(king/queen)</strong> of the data, hence for text embedding we must know exactly which word is meant to find relations between them according to the features<strong>(he/she) </strong>as shown in <em>figure</em>.</p><p>This helps to store the information of the word’s meaning to a smaller dimensional space, by value for each property.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*QV7zakC8FBGmxp19.png" /><figcaption>Vector representation of 2D embedding. <a href="https://res.cloudinary.com/practicaldev/image/fetch/s--R97eB0d_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://www.nexmo.com/wp-content/uploads/2020/10/word2viz-queen.png">Source</a></figcaption></figure><blockquote>W<a href="https://dugas.ch/index.html">e multiply the 2048 x 50257 sequence-encodings matrix with the 50257 x 12288 embedding-weights matrix (learned) and end up with a 2048 x 12288 sequence-embeddings matrix.</a></blockquote><h3>For output in the sequence, how to predict which input word to focus on and how much?</h3><h4>4. Attention :</h4><p><strong>What attention does :</strong></p><p>It maps a query and a set of key-value pairs to an output in the form of vectors. The output is computed as a weighted sum of the values.</p><p><strong>Multi-head attention in GPT :</strong> GPT model presented by the authors, they use multi-head attention. Which means is that the <em>attention</em> process is repeated many times (96x in GPT-3), each with a different learned query, key, value projection weights.</p><p><strong>Sparse attention in GPT-3 : </strong>Transformers are powerful sequence models, it gets impractical to compute a single attention matrix, for very large inputs. hence in <strong>sparse attention patterns, each output position only computes weightings from a subset of input positions.</strong></p><p>It gives the following benefits:</p><p>It gives a lower loss than full attention, in addition to being significantly faster for the re-computation of attention matrices to save memory.</p><blockquote>Sparse attention is algorithmic improvement of the <em>attention</em> mechanism to extract patterns from sequences 30x longer than possible previously.</blockquote><h4>5. Feed Forward</h4><p>Its a simple multi-layer-perceptron with 1 hidden layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/0*5BBQwzYLpnGGQXlj.png" /><figcaption>Working of Feed Forward: It takes input, multiply with learned weights, add learned bias, applies to ReLU(activation function) and gets result.</figcaption></figure><h4>6. Add &amp; Norm</h4><p>After both the Multi-Head attention and the Feed Forward blocks, the input of the block is added to it’s output, and the result is normalized.</p><h3>After the input is processed into a huge matrix, it contains information about the next predicted word. How do we extract this information?</h3><h4>7. Decoding</h4><p>To obtain information, the mapping done in the embeddings section is reversed to transform our output 12288-vector embedding back into a 50257-word-encoding.</p><p>After a quick <em>softmax</em>, we can then treat the resulting values as probabilities for each word.</p><h3>GitHub Copilot Recitation</h3><h4>The GitHub Copilot is a tool which is designed to work with a programmer rather than independently (i.e pair programmer). This is a huge achievement. Kudos to the <a href="https://medium.com/u/8df3bf3c40ae">GitHub</a> Copilot team!</h4><h3>Guillermo Rauch on Twitter: &quot;&quot;GitHub Copilot is a code synthesizer, not a search engine: the vast majority of the code that it suggests is uniquely generated and has never been seen before.&quot;The future.https://t.co/eqApkWlyB7 / Twitter&quot;</h3><p>&quot;GitHub Copilot is a code synthesizer, not a search engine: the vast majority of the code that it suggests is uniquely generated and has never been seen before.&quot;The future.https://t.co/eqApkWlyB7</p><p>Copilot is going to be an indispensable part of the programmer tool belt similar to IDE autocomplete for many people.</p><p>It guesses the exact code we want to write about one in ten times, and the rest of the time it suggests something <strong>rather good</strong>, or <strong>completely off</strong>. But when it guesses right, it feels like it’s reading our mind!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0KuJ_-TwJ-m5bDYW.png" /><figcaption>Working of the GitHub Copilot as a VS Code extension: The editor plugin sends comments and code to the GitHub Copilot service, then synthesises and offers ideas to the user using OpenAI Codex. [<a href="https://copilot.github.com/diagram.png">Source</a>]</figcaption></figure><h3>Is GitHub Copilot architecture, language agnostic?</h3><p>According to GitHub, Copilot <strong>supports a wide range of frameworks and languages</strong>, although as we see the technical preview, it is mainly focused with dynamic languages like <em>Python, JavaScript, TypeScript, Ruby, </em>and <em>Go.</em></p><p>It uses the<strong> same GPT-3 architecture for different languages </strong>as it was trained using huge corpus of GitHub’s public source codes as a data source for dataset, which contains multiple languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/0*ICVugj3QcGiRH7zE" /><figcaption>Representing bubble graph: GPT-3 was trained using multiple language datasets Source</figcaption></figure><p>This speciality can remove barriers to entry. It can help with learning new languages, and for folks working on <a href="https://thenewstack.io/netflix-builds-pipeline-polyglot-programming/">polyglot</a> codebases.</p><h3>Hands-on Demo of GitHub Copilot and it’s impact on domain specific edges.</h3><h3><strong>In the field of development:</strong></h3><p>Github Copilot is really great for those developers who properly document and comment their code. According to me, this allow <em>programmers</em> to become more productive, which will <em>increase</em> demand for the skills.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRDlJzdwkVnE%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRDlJzdwkVnE&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRDlJzdwkVnE%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a9e6a98941f21f7bcb4398120d42298/href">https://medium.com/media/1a9e6a98941f21f7bcb4398120d42298/href</a></iframe><h4>1. API development Domain:</h4><ul><li>It is capable of writing a Javscript code and It can also write codes for the development of complete API and Python scripts.</li><li>It can be used to write different algorithms such as binary search, binary sort, merge sort, etc.</li><li>It is useful in repetitive low signal to noise ratio languages like React or Java, but is less appealing if you’re writing code in <a href="https://clojure.org/about/functional_programming">Clojure</a> environment.</li></ul><p>The best thing about Github Copilot here is that it automatically produces the best codes keeping in mind the Time Complexity.</p><h4>2. Web Development Domain:</h4><ul><li>Github Copilot does a pretty good job while predicting the <strong>React codes</strong>.</li><li>When we tried to predict <strong>React+Typescript</strong> the Github Copilot was not able to perform efficient prediction, it went on to to predict the most common use case.</li><li><strong>Redux </strong>requires a lot of Boilerplate functions. While trying Github Copilot on Redux, it often wasn’t sure about the relevancy. It doesn’t knows what is relevant and what is not.</li></ul><h4>3. <strong>App Development Domain:</strong></h4><ul><li>While working with flutter the basic boilerplates codes are easy to predict for Github Copilot.</li><li>As the code starts to get complicated and we need to create API with a query and response, it fails to predict the complete code accurately.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FqswDPjMUaR8&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DqswDPjMUaR8&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FqswDPjMUaR8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/992a6357763de375364541625e21dd1f/href">https://medium.com/media/992a6357763de375364541625e21dd1f/href</a></iframe><ul><li>Platforms such as <a href="https://medium.com/u/158bf765767a">Geeks For Geeks</a> and <a href="https://medium.com/u/33f6b9a1a861">LeetCode</a> don’t have to spend time and capital in hiring people to write codes for teaching. This will result in more effecting teaching and learning for the freshers.</li></ul><p>According to the tests by <a href="https://medium.com/u/d67809141db2">Krish Naik</a> to write a function to find the smallest five elements in an array sorted in an ascending order and the Github; <strong>Copilot took only 2 seconds to create the function.</strong></p><h4>3. Data Science for Machine Learning pipelines and Deep Learning:</h4><ul><li>It is <strong>good at basic data preprocessing and exploratory data analysis.</strong></li><li>It is able to write a function for a basic neural network.</li><li>It struggles to write the basic machine learning model code.</li><li>There are multiple times when we pass the comment , it is not able to define a specific Machine Learning model.</li><li>Every time we asked <a href="https://medium.com/u/8df3bf3c40ae">GitHub</a> Copilot to write the code for basic machine learning, often it gave a basic code structure of what machine learning code should look like as an output.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fm0OEHRA1Kfs%3Fstart%3D518%26feature%3Doembed%26start%3D518&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dm0OEHRA1Kfs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fm0OEHRA1Kfs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/26368b9bc9238789d0804a7c3e6e8774/href">https://medium.com/media/26368b9bc9238789d0804a7c3e6e8774/href</a></iframe><ul><li><strong>The Github Copilot isn’t as great for Data Science as it is for writing DSA and Python scripts.</strong> When it comes to writing codes which require some serious thinking like writing complicated deep neural networks then Github Copilot fails to stand up to the expectations.</li></ul><h3>Opinions on Most trending discussions</h3><h4><strong>Drawbacks observed :</strong></h4><ul><li>The code that Copilot suggests is not <strong><em>tested</em></strong>. The code will almost always fail to compile or run.</li><li>The Github Copilot only stores a <strong><em>limited amount</em></strong> of context. Any source file longer than 100 lines is clipped, and the tool only looks at the context that comes before it. Hence, while writing long codes it will be really disappointing to expect <strong><em>continuous performance</em></strong> from the tool.</li><li>GitHub Copilot has been trained on billions of lines of publicly available code, it is extremely accurate. As a result, the proposed code and the code that is informed by it may have a direct link.</li></ul><h3>Further thoughts and ideas :</h3><h4>Generating short summary at the end of generated code for better understanding,</h4><p>Auto-code completion can make it difficult for novices to comprehend the code created by GitHub Copilot and for big codes written by AI, it takes more time to understand</p><p>I’m wondering whether we might make a modified version that creates a brief written summary at the conclusion of the generated code using sequence by sequence codes as input.</p><h4>Taking input functions in the form of Speech for easy accessibility</h4><p>We can do this by performing the task of speech-to-text conversion using state-of-the-art ASR models such as Wav2Vec2.</p><p>Although, prior to this step we will need to train the model on a large dataset which contains technical conversation about different programming languages and computer science related domains.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*pdITLTKSlFuancju.jpg" /></figure><h4>Thank you readers for staying with me until the end; Did my blog motivated you to join the GitHub Copilot waiting list? Please let me know how it works out for you in the comments!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68eb6d30a993" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/speed-up-the-development-with-advanced-pair-programming-github-copilot-68eb6d30a993">Speed up the development with advanced pair programming: GitHub Copilot</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Evolving with BERT: Introduction to RoBERTa]]></title>
            <link>https://medium.com/analytics-vidhya/evolving-with-bert-introduction-to-roberta-5174ec0e7c82?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5174ec0e7c82</guid>
            <category><![CDATA[facebook-research]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 12:43:30 GMT</pubDate>
            <atom:updated>2021-07-09T06:46:00.391Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>No matter how good it is, it can always get better, and that’s the exciting part.</blockquote><p>In this article, I will discuss the &quot;exciting part,&quot; which was how the <a href="https://medium.com/u/25aae929dbb1">Facebook Research</a> AI agency modified the training procedure of the existing <a href="https://medium.com/u/be36e94a7e47">Google</a> BERT, proving to the world that there is always room to improve.</p><p>Let’s look at the development of a robustly optimized method for pretraining natural language processing (NLP) systems(RoBERTa).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/0*E_2thD-NBl8stZ4P.jpg" /></figure><h3>Open Source BERT by Google</h3><p>Bidirectional Encoder Representations from Transformers, or BERT, is a self-supervised method <strong>released by Google in 2018.</strong></p><blockquote>BERT is a tool/model which understand language beter than any other model in the history. It’s freely available and it is increadibly versatily as it can solve a large number of problems related to lanugage tasks. You have used bert without even knowing!</blockquote><blockquote>If you have used google serch, you have already used BERT</blockquote><h3>Architecture:</h3><h4>Transformer model — a foundational concept for BERT</h4><p>BERT is based on the Transformer model architecture</p><p>Examining the model as if it were a single black box, a machine translation application would take a sentence in one language and translate it into a different language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IJiWAHP7JDZFkWO_.png" /><figcaption>Transformer performing the task of machine translation [<a href="https://jalammar.github.io/illustrated-transformer/">Source</a>]</figcaption></figure><ul><li>Basic Transformer consists of an encoder to read the text input and a decoder to produce a prediction for the task.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*YjoWwblcP0y9IOTq.png" /><figcaption>The basic structure of a transformer [<a href="https://jalammar.github.io/illustrated-transformer/">Source</a>]</figcaption></figure><ul><li>Since BERT’s goal is to generate a language representation model, it only needs the encoder part. hence, <strong>BERT is basically a trained Transformer Encoder stack</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*e-QE-c3jZewtMubA.png" /><figcaption>A basic structure of encoder block [<a href="https://jalammar.github.io/illustrated-transformer/">Source</a>]</figcaption></figure><p>To understand further implementations of its mechanism, refer to my previous <a href="https://medium.com/nerd-for-tech/key-feature-extraction-from-classified-summary-of-a-text-file-using-bert-c1472f7b493">blog</a></p><h4>Training of BERT</h4><p>During pretraining, BERT uses two objectives: masked language modeling and next sentence prediction.</p><ul><li><strong>Masked Language Modeling(</strong><strong>MLM) </strong>basically <strong>masks</strong> 80% of the 15% of the randomly selected input tokens and uses the other tokens to attempt to predict the mask (missing word).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*7fxNNDYzIVPSZoTa.png" /></figure><ul><li><strong>Next Sentence Prediction(</strong><strong>NSP)</strong>is a <strong>binary classification loss</strong> for predicting whether two segments follow each other or are from a different document to create a semantic meaning.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/0*fIJ71CLrjyhh799K.png" /></figure><h3><strong>B</strong>eginning of the Optimization of BERT: Introduction to RoBERTa</h3><h4><strong>Room for improvement in BERT</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*5e3Y7sqvfCnG5Qk4.gif" /><figcaption>Learning scope of improvements in BERT</figcaption></figure><h4>BERT is significantly undertrained and the following areas stand the scope of modifications.</h4><h4><strong>1. Masking in BERT training:</strong></h4><p>The masking is done only once during data preprocessing, resulting in a single static mask. Hence, the same input masks were fed to the model on every single epoch.</p><h4><strong>2. Next Sentence Prediction:</strong></h4><ul><li>The original input format used in BERT is <strong>SEGMENT-PAIR+NSP LOSS.</strong></li><li>In this, each input has a pair of segments, which can each contain multiple natural sentences, but the total combined length must be less than 512 tokens.</li><li>It is noticed that individual sentences hurt performance on downstream tasks, which according to the hypothesis is because the model was not able to learn long-range dependencies, hence the authors could experiment with removing/adding NSP loss to see the effect in the model’s performance.</li></ul><h4>3. Text Encoding:</h4><ul><li>The original BERT implementation uses a character-level BPE vocabulary of size 30K.</li><li>BERT uses the WordPiece method a language-modeling-based variant of Byte Pair Encoding.</li></ul><h4>4. Training batch size:</h4><p>Originally BERT is trained for <strong>1M steps with a batch size of 256 sequences</strong>, which shows room for improvement in perplexity on the Masked Language Modelling objective.</p><h3>Altering the training Procedure:</h3><h4>1. Replacing Static masking with Dynamic Masking :</h4><p>To avoid masking the same word multiple times, Facebook used Dynamic masking; the training data was repeated 10 times and every next time, the masked word would be different, meaning the sentence would be the same but the words masked would be different.</p><h4><strong>2. Removing NSP :</strong></h4><p><strong>TEST 1: Feeding the following alternate training formats.</strong></p><p><strong>2.1. Retain NSP Loss:</strong></p><ul><li><strong>SENTENCE-PAIR+NSP:</strong> Each input contains a pair of natural sentences, sampled from a contiguous portion of one document or separate documents. The NSP loss is retained.</li></ul><p><strong>2.2. Remove NSP loss:</strong></p><ul><li><strong>FULL-SENTENCES: </strong>Each input is packed with full sentences sampled contiguously from single or cross documents, such that the total length is at most 512 tokens. We remove the NSP loss.</li><li><strong>DOC-SENTENCES</strong>: Inputs are constructed similarly to FULL-SENTENCES, except that they may not cross document boundaries. Inputs sampled near the end of a document may be shorter than 512 tokens, so we dynamically increase the batch size in these cases to achieve a similar number of total tokens as FULLSENTENCES. We remove the NSP loss.</li></ul><h4>Results —</h4><ul><li>Removing the NSP loss matches or slightly improves downstream task performance, in contrast to the original BERT with NSP loss.</li><li>The sequences from a single document (DOC-SENTENCES) perform slightly better than packing sequences from multiple documents (FULL-SENTENCES) as shown in<em> Table 1.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*cJEC3OtdCe5B2ZjD.png" /><figcaption>Table 1: Comparison of performance of models with and without NSP loss (image is taken from the <a href="https://arxiv.org/abs/1907.11692">paper</a>)</figcaption></figure><h4>3. Training with large mini-batch:</h4><p>It is noticed that training a model with large mini-batches improves the perplexity of MLM objective and end-accuracy.</p><blockquote>1M steps, batch size of 256 has equivalent computational cost to 31K steps, batch size of 8K.</blockquote><p>Large batches are also easier to parallelize via distributed parallel training.</p><h4>4. Byte-Pair Encoding:</h4><ul><li>Here Byte-Pair Encoding is used over raw bytes instead of Unicode characters.</li><li>The BPE subword vocabulary is reduced to 50K (still bigger than BERT’s vocab size) units.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*5qI5ioaSVV5YhU9-" /><figcaption>A quick example of Byte Pair Encoding (BPE) [<a href="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQCZAAiwMGZi20JrcMwe9Nn-L4Vn2enNZj4lA&amp;usqp=CAU">Source</a>]</figcaption></figure><p>Despite degrading the performance of end-task performance in some cases, this method was used for encoding as it is a universal encoding scheme that doesn’t need any preprocessing and tokenization rules.</p><h4>5. Increasing Training Data:</h4><p>It was observed that training BERT on larger datasets, greatly improves its performance. Hence the training data was increased to 160GB of uncompressed text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*FWXcIIW7vFqLsBRa.jpeg" /><figcaption>Google’s BERT after seeing the modifications!<a href="https://www.google.com/search?q=bert+meme+face&amp;tbm=isch&amp;hl=en&amp;sa=X&amp;ved=2ahUKEwi4uv7VnrrxAhU-xHMBHRwTCGIQrNwCKAJ6BAgBEHE&amp;biw=1519&amp;bih=722#imgrc=3M9gWEsbSbq_NM&amp;imgdii=nOdlVxygRt-izM"> source</a></figcaption></figure><h3>Facebook’s RoBERTa: An optimized method for pretraining self-supervised NLP systems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZ7gFAzfRFa3Cua87DJE7w.gif" /></figure><p>The issues discussed above were identified by Facebook AI Research (FAIR), and hence, they proposed an ‘optimized’ and ‘robust’ version of BERT.</p><blockquote>RoBERTa is actually robust across NLU tasks, the absolute geniuses at Facebook actually did it, and it’s not a clickbait!</blockquote><blockquote>RoBERTa is part of Facebook’s ongoing commitment to advancing the state-of-the-art in self-supervised systems that can be developed with less reliance on time- and resource-intensive data labeling.</blockquote><p>It acts more like just giving context to the rest of the input (the actual context), which is a reason why RoBERTa works better than models that were actually fine-tuned on tasks like SQUAD.</p><h3>Why RoBERTa matters?</h3><p>The world appreciates Google as they have made this natural language processing program open source to the world<strong>. Through RoBERTA, we see this move to open source BERT has brought a drastic change to NLP.</strong></p><p>The study demonstrates how sufficiently pre-trained models lead to improvements with trivial tasks when scaled to extremities.</p><blockquote><strong>Improving the potential of BERT, is hugely impacting the the market and economic outlook.</strong></blockquote><ul><li>BERT and RoBERTa are used in<br>the improvement in NLP tasks as they make use of embedding vector space that is rich in context.</li><li>Using RoBERTa for preprocessing the data, is of major advancement to all the small product to big multinational companies as mainly work to incorporate data for analysis to extract information.</li></ul><p>This is significant because, as a result of these studies, experiments, and developments, we are getting closer to the larger challenge for NLP models, which is to achieve a human-level understanding of language!</p><h3>How RoBERTa is different from BERT</h3><h4>The authors of RoBERTa suggest that BERT is largely undertrained and hence, they put forth the following improvements for the same.</h4><ul><li>More <strong>training data</strong> (16G vs 160G)</li><li>Uses <strong>dynamic masking pattern</strong> instead of <strong>static masking pattern.</strong></li><li>Replacing the <strong>next sentence prediction</strong> objective with <strong>full sentences without NSP.</strong></li><li>Training on <strong>Longer Sequences.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*z_N15lPs46CDIxpb" /><figcaption>Accuracy vs Number of training steps plot for different models. [<a href="https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRLIXLOFlu96piyukpXw3zCBlt5h5ohpuU4tw&amp;usqp=CAU">Source</a>]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*wJraZazbjkr2avUc.png" /><figcaption>Comparison of BERT, RoBERTa, DistilBERT, XLNet in terms of training strategies</figcaption></figure><h3>Zero-Shot Learning with RoBERTa</h3><h4>Zero-Shot Learning :</h4><p>It is a type of Machine Learning technique, where the model is used without fine-tuning on a particular task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/0*62JP-PqRYhycsUKu.png" /><figcaption>Explaining Zero-Shot Learning</figcaption></figure><h4>You can use the following examples to implement any text sequence classification task (One-Shot Classification) by simply following the steps. It is extensively used also for sequence regression tasks.</h4><h4><strong>Load RoBERTa from torch.hub</strong></h4><p>Loading pre-trained RoBERTa Large model using Pytorch</p><pre>import torch<br>roberta = torch.hub.load(&#39;pytorch/fairseq&#39;, &#39;roberta.large&#39;)<br>roberta.eval()</pre><h4>Roberta For Sequence Classification:</h4><p>RoBERTa Model transformer is with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.</p><h4>1. Download RoBERTa that is already finetuned for MNLI</h4><p>RoBERTa is also pretrained on MNLI (Multi-Genre Natural Language Inference is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information).</p><pre>roberta = torch.hub.load(&#39;pytorch/fairseq&#39;, &#39;roberta.large.mnli&#39;)<br>roberta.eval()  # disable dropout for evaluation</pre><h4>2. Encode a pair of sentences and make a prediction</h4><pre>tokens = roberta.encode(&#39;Roberta is a heavily optimized version of BERT.&#39;, &#39;Roberta is not very optimized.&#39;)<br>roberta.predict(&#39;mnli&#39;, tokens).argmax()</pre><h4>3. Encode another pair of sentences</h4><pre>tokens = roberta.encode(‘Roberta is a heavily optimized version of BERT.’, ‘Roberta is based on BERT.’)<br>roberta.predict(‘mnli’, tokens).argmax()</pre><h3>Further Discussion</h3><p><strong>Transformers are extremely memory intensive. </strong>Hence, there is quite a high probability that we will run out of memory or above the runtime limit while training larger models or for longer epochs.</p><p>Hence, in my next blog, I’ll be discussing and implementing some <strong>promising well-known, and impactful out-of-the-box strategies</strong> <strong>to speed up transformer with optimization strategies to reduce the training time.</strong></p><h3>Creating bonds, stories, and magic together!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*XTN2BDeDBcKTDkWX.png" /><figcaption>That’s me connecting with you through the screens! <a href="https://blog.textnow.com/blog/2020/04/23/distance-learning-how-textnow-is-helping-teachers-stay-connected-pt-2/">source</a></figcaption></figure><blockquote>I am aware that technology is expanding, that minds are becoming more inquisitive, and that the results are becoming more interesting. And it is because of this that we continue to learn and grow!</blockquote><p><strong>Congratulations for making it to the end of the blog, that was me!</strong></p><p>If you enjoyed learning about RoBERTa today, let me know what tech stacks you use in the comments below, and feel free to reach out for any queries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5174ec0e7c82" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/evolving-with-bert-introduction-to-roberta-5174ec0e7c82">Evolving with BERT: Introduction to RoBERTa</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto-code generation using GPT-2]]></title>
            <link>https://medium.com/geekculture/auto-code-generation-using-gpt-2-4e81cb05430?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e81cb05430</guid>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gpt]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 14:54:48 GMT</pubDate>
            <atom:updated>2021-06-18T07:43:18.672Z</atom:updated>
            <content:encoded><![CDATA[<h4>Code faster with AI</h4><h3>About GPT-2</h3><blockquote>GPT-2 stands for “<strong>Generative Predictive Transformer</strong>”. It is an open-source model trained on an over 1.5 Billion parameters for generating the next sequence of text, for a give sequence.</blockquote><blockquote>The GPT-2 has a remarkable and amazing competence to generate texts, much beyond the expectations of conventional language models.</blockquote><h3>“Too dangerous to be released.”</h3><h4>A phrase published the press statement by OpenAI to accompany their announcement of the release of their GPT-2 language model in February 2019.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*BQ5pUlbr_wYOrzVSdPbOTQ.jpeg" /></figure><blockquote>The threat of AI-driven misinformation has become a huge issue that remains unresolved in today’s post-factual information ecosystem, especially with the recent release of the more powerful GPT-3.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3DxtoxyhZZddjRVg.jpeg" /><figcaption>Performance of GPT on different datasets</figcaption></figure><blockquote>“Today, the United Nations has called for the immediate withdrawal of all nuclear weapons from the world.”</blockquote><p><strong><em>The sentence you just read was neither written by me, nor was it written by the editor. This sentence was written by GPT-2</em></strong></p><blockquote>GPT-2 is a 1.5-bit transformer-based language model, trained in a database of 8 million web pages. It was trained to simply predict the next word in 40GB of Internet text. Due to some issues a very small model for researchers was released to experiment with.</blockquote><h3>Working mechanism of GPT-2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UmykBKt7OHzTSUoG.png" /><figcaption>Working og GPT-2 [<a href="https://openai.com/blog/better-language-models/">Source</a>]</figcaption></figure><ul><li><strong>Generative:</strong> This means that model was trained to predict the next token in a given sequence of tokens. The model is given a lot of raw text and then it is asked to generate more text using the statistical features which ofcourse involves implementation of different layers and mechanism such as RNN-LSTM and Attention mechanism.</li><li><strong>Pre-trained:</strong> OpenAI trained a large and a powerful language transformer model to use it for tasks like summarization, neural machine translation, etc. Now, the model was trained on 40 GB of text, known as WebText.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SaSwwhWtd04v0Ys0.png" /><figcaption>Transformer architecture</figcaption></figure><ul><li><strong>Transformer:</strong> The GPT-2 is build using layers of decoder transformer blocks.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/0*PIFgY3d16zYH37k1.jpg" /><figcaption>The text in blue is given as an input and the text in red is the predicted output</figcaption></figure><p><strong>The GPT-2 architecture is based on the concepts of the transformers.</strong></p><p>The mechanism on which GPT-2 works involves transformer based encoder decoder architecture to learn the input and output dependencies.</p><p>To generate the next output in a given sequence the model need to have the previously generated data as an input.</p><ul><li>The GPT-2 has a great ability to adapt to the context of the text and thus generates realistic and coherent output.</li><li>The model works by adding each token to the sequence of inputs as it is created. In the next step, that new sequence becomes the model’s input. This is an idea called <strong>“auto-regression”</strong>. This is one of the ideas that made RNNs unreasonably effective.</li></ul><h3>What is the meaning of automatic code generation?</h3><h4>The automatic code generation is basically involves the completion of a sequeunce of codes based upon the previous inputs and past habits of the user.</h4><ul><li>Many commercial platforms such as <strong>TabNine and Kite</strong> are already available in the market for this task. These both use GPT-2 to predict the next sequence of codes based upon the previous inputs provided by the users.</li><li>Here is a short video which demonstrates the amazing capabilities of an automatics code generation process:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FtwPtvZuBrAg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtwPtvZuBrAg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtwPtvZuBrAg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d96592beede6e8ff7c778b04fb28af6f/href">https://medium.com/media/d96592beede6e8ff7c778b04fb28af6f/href</a></iframe><h3>Steps to fine-tune GPT-2 for code generation:</h3><h4>1. Cloning the required repositories</h4><ul><li>Here, we are cloning the auto_coding repository which contains code and scripts to fine tune the GPT-2 model for automatic code generation.</li><li>We need to provide the training examples in the form of scripts (Examples: Python, C, C++, Java, and Javascript).</li><li>For fine-tuning our GPT-2 model we have used scripts from scikit-learn examples.</li></ul><pre>!rm -rf auto_coding</pre><pre>!git clone <a href="https://github.com/aasthaengg/auto_coding">https://github.com/aasthaengg/auto_coding</a> &amp;&amp; cd auto_coding &amp;&amp; pip install -r requirements.txt &amp;&amp; cd dataset &amp;&amp; git clone <a href="http://github.com/pytorch/examples.git">http://github.com/scikit-learn/examples.git</a> &amp;&amp; python convert.py — segment_len 256 — stride 10 — dev_size 0.1</pre><h4><strong>2. Downloading the required scripts</strong></h4><p>Now, we are downloading the <strong>convert.py script.</strong></p><p>This script contains the code to convert our examples into the training data in a format expected by our model.</p><pre>!wget <a href="https://raw.githubusercontent.com/aasthaengg/auto_coding/master/dataset/convert.py">https://raw.githubusercontent.com/aasthaengg/auto_coding/master/dataset/convert.py</a> &amp;&amp; cp convert.py /content/auto_coding/dataset/</pre><h4>3. Navigating to the required directing</h4><p>Here, we are executing the convert.py script with <strong>segment length as 256, strides of 10, and development size of 10%.</strong></p><p>The 90% of the data will be used for training and the remaining 10% will be used for testing our model.</p><pre>!cd auto_coding &amp;&amp; cd dataset &amp;&amp; python convert.py — segment_len 256 — stride 10 — dev_size 0.1</pre><h4><strong>4. Executing the training scripts and selecting the model as distilgpt2</strong></h4><p>We have different versions of models available for fine-tuning. Here,<strong> distilgpt2 </strong>is selected to be fine-tuned. If one has enough computing resources and huge dataset then they may go for version with larger number of parameters.</p><pre>!cd /content/auto_coding &amp;&amp; python train.py — model_select distilgpt2</pre><h4>5. The model is now trained. Let’s check out the model</h4><p>After the training get’s completed we need to execute <strong>interact.py script.</strong> This script will run the model for testing purpose. The input needs to be provided and the model will predict the sequence.</p><pre>!python interact.py</pre><h3>How this use case is helping tech professionals?</h3><ul><li>This solution is already being used in the industry. It allows developers to code <strong>faster with 47% less keystrokes</strong>. This helps developers to become more productive.</li><li>Also, if a block of code is already written earlier then the user just need to make few strokes of identical words and the user will get an automatically completed block of codes.</li><li>According, to some of the reviews, this has helped developers reduce a chunk of time, since they are writing just 70% to 80% codes and the rest 20% to 30% of the code is automatically generated.</li><li>With all these advantages, the industry is also saving a significant amount of time.</li></ul><h3>Limitations of GPT-2</h3><ul><li>GPT-2 cannot be used handle complex and long language formations.</li></ul><p>If someone wants to generate a sequence of text related to a particular domain such as literature, finance, or medicine, then it won’t be able to perform well.</p><ul><li>There are certain limitations in terms of computing resources. To train such a huge model with billions of parameter we require very expensive computing resources to train so that the model can perform better.</li></ul><h3>Writer’s perespective on GPT architecture</h3><ul><li>The GPT is no doubt an amazing invention in the field of Natural Language Processing but it’s capabilities are still unknown since the complete pre-trained model was never released by OpenAI because of some serious threats.</li><li>For performing this task on automatic code generation I used distil-GPT2 because of it’s small size and relatively less expensive model fine-tuning. We limited the sequence generation size to 15 for more effective and crisp prediction.</li><li>If one has thought of using GPT-2 model with higher number of parameters then the size of sequence can be increased accordingly, keeping the computing resources in mind.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*jzzbjTcYcSn6X2Xt" /><figcaption>Here is a tweet from VP of AI at Facebook</figcaption></figure><p>I’m quite aware of the additional risks, such as the possibility of GPT-3’s human-like text generation capacity being used for phishing, scamming, spamming, distributing false news, or other fraudulent actions. Hence, one should use such models keeping ethics in mind.</p><p><strong>We should use artificial intelligence to make our lives better and not by doing anytype of criminal activities.</strong></p><h3>Conclusion</h3><p>Congrats for making it all the way to the end of this blog! Thank you very much for taking the time to read this. I hope it was useful in getting you up and going.</p><p><strong>Did you like GPT-2’s superpowers? Please let me know in the comments section where all thoughts and insights are eagerly appreciated.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e81cb05430" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/auto-code-generation-using-gpt-2-4e81cb05430">Auto-code generation using GPT-2</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to NumPy]]></title>
            <link>https://aastha-eng.medium.com/introduction-to-numpy-68e3f0ee7206?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/68e3f0ee7206</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[moonshot-factory]]></category>
            <category><![CDATA[numpy]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nvidia]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Sat, 05 Jun 2021 14:54:26 GMT</pubDate>
            <atom:updated>2021-06-06T00:14:11.912Z</atom:updated>
            <content:encoded><![CDATA[<h4>If you like Numpy Numpy likes you.</h4><h3>In a world without NumPy</h3><blockquote>NumPy is gold. In a world without a numeric library like Numpy, this age of Artificial Intelligence wouldn’t exist as we would not have been able to train Machine Learning algorithms.</blockquote><p><strong>Simple addition of two lists without NumPy:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*IjtlGvbMeT89fQdYXwKk2g.png" /></figure><blockquote>You can anticipate that if we have a lot of data or a multi-dimensional value, we will require two or more layers of<em> </em><strong>for loop</strong><em>,</em> which will slow down computation.</blockquote><h3>About NumPy — see the light!</h3><p><strong>NumPy deservedly bills itself as the fundamental package for scientific computing with Python.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZXOX-BIvwRmVdADI.jpg" /></figure><blockquote>NumPy is a module for Python. The name is an acronym for “Numeric Python” or “Numerical Python”.</blockquote><ul><li>NumPy is a merger of two Python modules, it is mostly written in C. <strong>Numeric</strong> is like NumPy, a Python module for high-performance, numeric computing. <strong>Numarray</strong>, is a complete rewrite of Numeric but is now deprecated.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*OcHifdrzTN2rOcTNnz2DIQ.png" /></figure><blockquote>Jim Hugunin wrote Numeric (Numerical Python) in 1995 with help from a bunch of other people, and Travis Oliphant.</blockquote><blockquote>In 2005, Travis Oliphant created NumPy by incorporating features of the competing Numarray into Numeric.</blockquote><h3>Core Python vs NumPy with Python</h3><blockquote>Python was not originally intended for numerical computing. The demand for quick numeric calculation developed as individuals began to use Python for diverse tasks</blockquote><h4>The advantages of Core Python:</h4><ul><li>It has high-level number objects: integers, floating point</li><li>containers: lists with cheap insertion and append methods, dictionaries with fast lookup.</li></ul><h4>Advantages of using NumPy with Python:</h4><ul><li>It does array oriented computing</li><li>It efficiently implements multi-dimensional arrays</li><li>It is designed for scientific computation</li></ul><h3>Python list vs Python array</h3><h4>A list in Python is a collection of items which can contain elements of multiple data types, which may be either numeric, character logical values, etc.</h4><ul><li>It cannot directly handle arithmetic operations.</li><li>It’s flexibility allows easy modification (addition, deletion) of data.</li><li>The entire list can be printed without any explicit looping.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*6uUIeK9U6-gpdEyeQLYWcQ.png" /></figure><h4>An array is a vector containing homogeneous elements allocated with contiguous memory locations allowing easy modification.</h4><ul><li>Can directly handle arithmetic operations.</li><li>Less flexibility since addition, deletion has to be done element wise.</li><li>A loop has to be formed to print or access the components of array.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*iYB2nC40IYfoBKU6Qz6r6A.png" /></figure><h3>Built-in array vs NumPy array</h3><blockquote><strong>Built-in array : <em>The objects stored in the array are of a same type and is determined by typecode. Type codes are single characters.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*NriOXLa1hdysKBjiYoPkWg.png" /><figcaption><strong>array(typecode [, initializer])</strong> returns an array.</figcaption></figure><ul><li>Built-in array module does not typecast data implicitly (example : if here we provide a float input, it will give us a TypeError as it from float to int)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*K6cfEK5PQ6cGvopbAjAj7w.png" /><figcaption>Typecode ‘i’ indicates , objects stored in the array must be of ‘int’ type. Trying to store floating-point value, caused a TypeError.</figcaption></figure><blockquote><strong>NumPy array :</strong> <strong><em>NumPy module in python is generally used for matrix and array computations.</em></strong></blockquote><blockquote><strong><em>Using the NumPy module, built-in function ‘array’ is used to create an array.</em></strong></blockquote><ul><li>NumPy array is flexible in the matter of typecasting, it will upcast or downcast and try to store the data at any cost. (Float was converted to int, even if that resulted in loss of data after decimal)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*2g3bRsVz0tkl5RwXIEXJTg.png" /><figcaption>Here floating point data was typecasted into int</figcaption></figure><h3>Python list vs NumPy array , since 2005</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/0*tu2427nXM0VPEnur" /></figure><ul><li>The Python core library provided <a href="https://webcourses.ucf.edu/courses/1249560/pages/lists"><strong>Lists</strong></a>. A list is a collection of data type that can be of homogeneous or heterogeneous.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*5IwiNhzq2PgQ77FXAgY9YQ.png" /></figure><ul><li>NumPy array is a grid of orderly allocated values of the homogeneous data types. Element wise operation is possible.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*k0Tm3MLEoQ7hwySbPkq1Jg.png" /></figure><h3>If You Want To Get Ahead, Get a Speed of NumPy!</h3><h4>How fast is Numpy?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*yg4kXlPf_ww25NH5e6ffIw.png" /><figcaption>Numpy array performs 5 times faster than Python List for Scalar Addition</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*YKNSmofA9hhn0rhGZUYTsA.png" /></figure><h4>Why is it so fast?</h4><h4>1. Memory Consumption</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zpzIESW-SjqVDSMN2_vScQ.jpeg" /><figcaption>Array in RAM</figcaption></figure><p><strong>Numpy arrays store one defined type of data as one contiguous block of memory.</strong></p><p>With the array’s initial memory address known, we can simply add the index to jump to any element.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2mLlse6wkN0EjoM7zguOaw.jpeg" /><figcaption>List’s memory allocation</figcaption></figure><p><strong>Python lists are actually arrays of pointers. A pointer is a reference to an object in memory.</strong></p><p><strong>Python takes two steps </strong>when retrieving the first entry in your list which decreases performance: Obtain the pointer first. Second, go to the pointer’s memory address to retrieve the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*YSouvfQ-fWd2kkMaTDJJ1w.png" /></figure><h4>2. Computation process</h4><ul><li>Numpy is able to divide a task into multiple subtasks and process them.</li><li>NumPy combines C, C++, and Fortran code. In comparison to Python, these programming languages have an extremely short execution time.</li></ul><h3>Why does speed matter?</h3><p>The small differences in runtime become amplified with repeated function calls, which cumulates to a large inference time hence slower computation.</p><ul><li>We understand now that Loops slow our code! So, let’s write a code that takes two arrays and performs element-wise multiplication.</li><li>if we are using NumPy arrays, we do not need to write a loop. We can simply do this like shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*ySsLVWHZ_r61QH3SxJomkg.png" /><figcaption>Preparing multiplication of lists and the NumPy arrays</figcaption></figure><ul><li>When using the NumPy’s solution, there are significant speedups of nearly 20–30 times.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*mWBY_3JFBoacqADh2eQI7Q.png" /><figcaption>It took 79 μs for a loop to multiply list, whereas 3.4 μs to multiply NumPy array</figcaption></figure><blockquote>This practice of replacing explicit loops with array expressions is commonly referred to as vectorization.</blockquote><blockquote><strong>Limitation of vectorization</strong> : It can be only be done on a loop which performs operations on two arrays of equal size or between an array and a scalar.</blockquote><h3>What happens if we want to vectorize a loop that involves arrays of different sizes?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*V0QjxGl2I5TrGjd13j_zCw.png" /><figcaption>Addition matrix of shape (3,4) and column vector</figcaption></figure><p><strong>Metod 1 : To loop over the columns of matrix, and add each columns.</strong></p><p><strong>Drawback :</strong> Since we are looping, the code described above will e<em>xecute slowly</em> if the number of columns in the original array gets increased to a large number.</p><p><strong>Method 2 : Make a matrix of the same number of columns as the original array.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bZ5J22322g00TiHOxi_VLQ.png" /></figure><p>This gives us a much faster solution. This important NumPy abstraction is called <strong>Broadcasting.</strong></p><blockquote>“Subject to certain constraints, the smaller array is “<a href="https://numpy.org/doc/stable/user/basics.broadcasting.html">broadcast</a>” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python”</blockquote><h3>NumPy to its full capacity!</h3><p><strong>This blog will now also show you how to utilise NumPy to its best potential by exhibiting how to use Vectorization and Broadcasting.</strong></p><h4>Understanding Rules for Broadcasting</h4><p>NumPy’s <strong><em>broadcasting rule relaxes this constraint when the arrays’ shapes meet certain constraints</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aNti5gxJWKblP9_F.png" /></figure><p>For just two arrays, NumPy compares the shape of the two arrays dimension-by-dimension starting from the trailing dimensions of the arrays working it’s way forward. (from right to left)</p><p><strong>Two dimensions are compatible when:</strong></p><ul><li><strong>Dimensions are equal</strong></li><li><strong>One of them is 1</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BTa1NkNWM0rQ8B2R.png" /><figcaption>Equal Ranks and equal values</figcaption></figure><p>In case of example on the right, starting from left, when we arrive at the 2nd dimension, there’s a difference and neither of them is 1. Therefore, trying to do an operation with them leads to an error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*WP7OszEN-m8-W75xUyKhTQ.png" /><figcaption>Unequal ranks</figcaption></figure><p>In order to compare two such arrays, Numpy <em>appends </em>dimensions of size 1 to the smaller array so that it has a rank equal to the larger array to make it compatible.</p><h3>Conclusion</h3><h4>Woo Hoo! Congrats for making it all the way to the end of this blog.</h4><p>Thank you very much for taking the time to read this. Despite the fact that it was only a tooth in NumPy’s gear wheel, I hope it was useful in getting you up and going.</p><p><strong>Did you like NumPy’s superpowers? Please let me know in the comments section where all thoughts and insights are eagerly appreciated.</strong></p><p>Stay tuned for the next blog in which we will continue exploring and learning NumPy by delving deeper into its features such as broadcasting, vectorization, indexing, ufunc and Numba as well as how they help in real world scenarios.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68e3f0ee7206" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Key Feature extraction from classified summary of a Text file using BERT]]></title>
            <link>https://aastha-eng.medium.com/key-feature-extraction-from-classified-summary-of-a-text-file-using-bert-c1472f7b493?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1472f7b493</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Mon, 31 May 2021 00:18:25 GMT</pubDate>
            <atom:updated>2021-05-31T00:18:25.873Z</atom:updated>
            <content:encoded><![CDATA[<h3>Harnessing the power of BERT embeddings</h3><p>In this post, I’ll show you how BERT solves a basic text summarization and categorization issue.</p><blockquote><strong>About BERT(</strong>Bidirectional Encoder Representations from Transformers)</blockquote><blockquote>BERT, in a nutshell, is a model that understands how to represent text. You feed it a sequence, and it scans left and right a number of times before producing a vector representation for each word as an output.</blockquote><blockquote>BERT and other Transformer encoder architectures have been wildly successful on a variety of tasks in NLP (natural language processing).</blockquote><h3><strong>Structure of BERT</strong></h3><h4><strong>1. The BERT summarizer</strong></h4><ul><li>It has 2 parts: a BERT encoder and a summarization classifier.</li><li>In the encoder, we learn the interactions among tokens in our document while in the summarization classifier, we learn the interactions among sentences.</li></ul><p>To assign each sentence a label , we need to add a token [CLS] before each sentence indicating whether the sentence should be included in the final summary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/0*4dWSd7wQgBTPQegM.png" /><figcaption>BERT structure for summarization</figcaption></figure><h4><strong>2. The BERT Classifier</strong></h4><p>Input — there’s [CLS] token (classification) at the start of each sequence and a special [SEP] token that separates two parts of the input.</p><p>Output — for classification, we use the output of the first token (the [CLS] token). For more complicated outputs, we can use all the other tokens output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/0*9j7r1HGOAJ-k7WVa.JPEG" /></figure><h3><strong>Comparing BERT with </strong>XLNet &amp; GPT-2, <strong>for Text Summarization based on performance</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-5OmHMcNGG17Kt5v2tZNA.png" /><figcaption>Comparison after installing bert-extractive-summarizer, transformers==2.2.0, spaCy</figcaption></figure><p><strong>Results :</strong></p><ul><li>Terms of performance — GPT-2-medium is the best</li><li>Terms of time taken — XLNet (11 s) GPT-2 medium (35s) Bert (30s)</li><li>Terms of ease of use — BERT</li></ul><h4><strong>Step 1: Choosing the BERT Model</strong></h4><p>There are multiple BERT models available.</p><ul><li><a href="https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/3">BERT-Base</a>,</li><li><a href="https://tfhub.dev/google/collections/bert/1">Small BERTs</a></li><li><a href="https://tfhub.dev/google/collections/albert/1">ALBERT</a></li><li><a href="https://tfhub.dev/google/collections/experts/bert/1">BERT Experts</a></li><li><a href="https://tfhub.dev/google/collections/electra/1">Electra</a></li></ul><blockquote><strong>Final model used : DistilBERT</strong></blockquote><blockquote>It is a small, fast, cheap and light Transformer model trained by distilling BERT base.</blockquote><blockquote>It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT’s performances as measured on the GLUE language understanding benchmark.</blockquote><h4><strong>Step 2: Text classification using BERT</strong></h4><blockquote>Your mind must be racing with all of the possibilities that BERT has opened up. We can use BERT’s vast knowledge repository in a myriad of contexts for our NLP applications!</blockquote><h4><strong>1. Let’s Setup!</strong></h4><p>I have used the AdamW optimizer from <a href="https://github.com/tensorflow/models">tensorflow/models</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*mGSTKmOZaBoEMMuYX3dWvg.png" /></figure><h4><strong>2. </strong>Importing and Preprocessing the Dataset</h4><p>Source : <a href="https://www.kaggle.com/cfpb/us-consumer-finance-complaints">Kaggle</a></p><blockquote>Dataset consistes of consumers’ complaints sent by <a href="http://www.consumerfinance.gov/data-research/consumer-complaints/">the CFPB</a> about financial products and services to companies for response to help improve the financial marketplace.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*m6Ft2cFzafuJCaB5eJTlPg.png" /><figcaption>Loading the dataset</figcaption></figure><p><strong>2.1. Feature Selection</strong></p><p>I have selected the columns that were directly related to resloving the issues and classifying them into the product classes</p><p>The output below shows that our dataset has 555,957 rows and 18 columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*HJT3W_Nbmbv-SopGyd345g.png" /><figcaption>Selected 2 out of 18 features.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*3U3hLhFGoUr3Fnp91uVrbw.png" /><figcaption>Issues Classified into 10 product categories</figcaption></figure><p><strong>2.2. Label encoding</strong></p><p>I have label encoded the <strong><em>Product</em></strong> column to convert the text format into label format using <strong>LabelEncoder</strong> .</p><blockquote><strong>LabelEncoder</strong>: It allows to assign ordinal levels to categorical data.</blockquote><blockquote><strong>fit_transform</strong>(y): Fit label encoder and return encoded labels.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*sy2ttWETCKrUqyrA_szxEQ.png" /></figure><h4>3. Creating a BERT Tokenizer</h4><blockquote>Text inputs need to be transformed to <strong>numeric token ids</strong> and arranged in several <strong>Tensors</strong> before being input to BERT.</blockquote><blockquote>Tokenization refers to dividing a sentence into individual words. To tokenize our text, we will be using the BERT tokenizer.</blockquote><p><strong>Importing the pre-trained model and tokenizer which is specific to BERT</strong></p><ul><li>Create a BERT embedding layer by importing the BERT model from hub.KerasLayer</li><li>Retrieve the BERT<strong><em> vocabulary file</em></strong> in the form a<strong><em> numpy array.</em></strong></li><li>Set the text to lowercase and pass our vocab_file and do_lower variables to the BertTokenizer object.</li><li>Initialise tokenizer_for_bert.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*-8Vpfb4Ojf_GaRtqfXo14A.png" /></figure><h4>4. Defining helper function for text preprocessing</h4><ul><li>The <strong><em>encode_text</em></strong> function is converting raw text data into encoded text(‘CLS’+token+ ‘SEP’)which is fitted and converted to token</li><li>To create sentences of equal length, I have padded the <strong><em>token_ids, mask_ids, segment_ids </em></strong>to truncate the tokens with the provided batch size.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*eg0neut_OntPtzyolxdE6g.png" /></figure><ul><li>The model will take strings as input, and return appropriately formatted objects which can be passed to BERT.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*6OxLZFHar-nXzd93RDFVsg.png" /><figcaption>Passing text in <strong>test_text</strong> to <strong>encode_text</strong> function</figcaption></figure><blockquote>Since this text preprocessor is a TensorFlow model, It can be included in any model directly.</blockquote><h4>5. Defining the Model</h4><ul><li>Create a very simple fine-tuned model, with the preprocessing model, the selected BERT model, one Dense and a Dropout layer for regularization.</li><li>As you can see, there are 3 outputs from the preprocessing that a BERT model would use (input_words_id, input_mask and segment_ids).</li></ul><p><em>Batch size = 40 implies that if the input is &gt;than 40, it will be truncated to 40 tokens and if the input is &lt;40 it will pad it to 40 tokens.</em></p><figure><img alt="Here the model will be defined using the Keras functional API" src="https://cdn-images-1.medium.com/max/1024/1*wv0i_svqzgw1ZOsp24ut4A.png" /></figure><h4><strong>6. Converting the train text in encoded format</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2YLH0Bh8Eo1VCxlm6t0_A.png" /></figure><h4>7. Fine-Tuning the model for text classification</h4><p>Fine-tuning follows the optimizer set-up from BERT pre-training: It uses the <strong><em>AdamW</em></strong> optimizer</p><blockquote>BERT was originally trained with: the “Adaptive Moments” (Adam). This optimizer minimizes the prediction loss and does regularization by weight decay.</blockquote><p>To increase the accuracy, increase the no. of epochs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M6c91OCs2HnnRIajRKEx7A.png" /></figure><h3>Building Pipeline</h3><blockquote><strong>Flow of Pipeline :</strong></blockquote><blockquote><strong>Text Summarization using BERT&gt;Text Classification using BERT &gt;Name Entity Recognition using spaCy</strong></blockquote><p><strong>For Text Summarization:</strong></p><p><strong>Extractive, abstractive, and mixed summarization</strong> strategies are most commonly used.</p><ul><li><em>Extractive strategies</em> — It selects the top N sentences that best represent the article’s important themes.</li><li><em>Abstractive summaries </em>— It attempts to rephrase the article’s main ideas in new words.</li></ul><ol><li><strong>Installing bert-extractive-summarizer :</strong></li><li><strong>Installing spaCy : </strong>The smallest English language model takes only a moment to download as it’s around 11MB</li></ol><blockquote>This tool utilizes the <strong>HuggingFace </strong>Pytorch transformers library to run extractive summarizations.</blockquote><blockquote><strong>This works by first embedding the sentences, then running a clustering algorithm, finding the sentences that are closest to the cluster’s centroids</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*TNqoWUnkXhuNcMJIpb-AdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*1vdwALleFfgQ0QDVxpkBzw.png" /></figure><p><strong>3. Defining the pipeline function</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*odj9MmB7-nzl0-ZvX7gbMg.png" /></figure><h3>Testing the Model</h3><p>Passing Input to the trained model to summarize and then classify the text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*gnKTffLGxR7p2PQ9gz3AiA.png" /></figure><h3>Key Feature Extraction using spaCyNER</h3><blockquote>About spaCy Named Entity Recognition</blockquote><blockquote>spaCy’s Named Entity Recognition (NER ) locates and identifies the named entities present in unstructured text into the standard categories such as person names, locations, organizations, time expressions, quantities, monetary values, percentage, codes etc.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1Qc27lKofPVmtL2xpqc5A.png" /><figcaption>Structure of spaCy NER</figcaption></figure><p><strong>Accessing the Entity Annotations on the generated summary of the text</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*s_W6tzMmAWdvCh30p34Vkw.png" /></figure><p><strong>Doc.ents</strong> are token spans with their own set of annotations</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*HQ7H0sbUoiVgmX6RjtroWg.png" /><figcaption>Entity Annotations</figcaption></figure><h3>Further Thoughts</h3><p>For a much faster approach, I can directly extract the key features by extracting <strong>noun phrases</strong> from the generated text summary using spaCy.</p><p>This would help to get the <strong>most common nouns, verbs, adverbs </strong>and so on by counting frequency of all the tokens in the text file.</p><p>Feel free to play around with spaCy as there is a lot more built-in functionality available. I will be doing this in my next blog. Stay connected!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1472f7b493" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[YOLO: You Only Look Once]]></title>
            <link>https://aastha-eng.medium.com/yolo-you-only-look-once-65ea86104c51?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/65ea86104c51</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Sat, 29 May 2021 00:59:43 GMT</pubDate>
            <atom:updated>2021-05-29T00:59:43.358Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The “You Only Look Once,” or YOLO, family of models are a series of end-to-end deep learning models designed for fast object detection, developed by <a href="https://pjreddie.com/">Joseph Redmon</a>, et al. and first described in the 2015 paper titled “<a href="https://arxiv.org/abs/1506.02640">You Only Look Once: Unified, Real-Time Object Detection</a>.”</blockquote><h3>What is YOLO object detection?</h3><p>YOLO is a deep learning based approach of object detection. There are two types of object detection algorithm in the field on deep learning. These two types are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*pP8_Um1rRy1nAQUS.jpg" /><figcaption>Detection using R-CNN (A classification based algorithm)</figcaption></figure><ol><li><strong>Classification based algorithms</strong>: There are two steps involved in these types of algorithms.</li></ol><p><strong>Step1</strong> : involves selecting a group of <strong>Region of Interest (ROI) </strong>in the images where the chances that object is present are high.</p><p><strong>Step2</strong> : involves application of <strong>Convolutional Neural Networks (CNN)</strong> to the regions selected in first step to detect the presence of an object.</p><blockquote>Problem associated with these types of algorithms is that we need to execute a detector in each of the ROI, and this makes the process of object detection very slow and highly expensive in terms of computation.</blockquote><ol><li><strong>Regression-based algorithms: </strong>These types of algorithms are <strong>faster</strong> then the above algorithm. There is no selection of the ROI, instead the bounding boxes and the labels are predicted for the whole image at once. This makes the detection faster than classification algorithms. One of the most famous type of regression algorithms is <strong>YOLO (You Only Look Once)</strong>. Since, the inception of YOLO, it has been used in healthcare,self-driving cars, etc.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sWAB2sgDf8UkTqfh.png" /><figcaption>Detection using YOLO (A regression based algorithm)</figcaption></figure><h3>Working of YOLO: A quick walkthrough</h3><blockquote><strong>Why YOLO is such an useful algorithm!</strong></blockquote><ul><li>YOLO first takes image as an input</li><li>The framework divides a photo into a grid of <strong>NxN </strong>grids. (Let’s us suppose a 3x3 grid)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*BgV5JvtaOyg1UYjX.jpg" /></figure><ul><li>Now, on each grid the task of image classification and localization is applied.</li><li>Then, the YOLO algorithm will predict the bounding boxes and the class probabilities of each objects respectively.</li></ul><h3>Training YOLO on a custom dataset</h3><blockquote>We will follow the following steps to train YOLO v5 model (PyTorch version) on a custom dataset.</blockquote><blockquote>Dataset: The dataset we have chosed here is a blood object detection dataset</blockquote><p><strong>Cloning YOLOv5 repository</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*iq0EmONjNknlYprLIgD2lA.png" /></figure><p><strong>Installing all the necessary dependencies</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*luciS7ZJqyIcmVj-Bbv9Iw.png" /></figure><p><strong>Loading the custom dataset annotated on Roboflow in YOLO v5 (PyTorch) format.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*mGBG6XWAUFWcrY4WE3ihzw.png" /></figure><blockquote>This is the YAML file which Roboflow wrote for public to use their platform to create annotated dataset and then train YOLO effectively.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*u8399c5wogALgHlaihwDBQ.png" /></figure><p>The dataset contains 874 images and the objects to be detected are ‘Platelets’, ‘WBC’, and ‘RBC’ in blood microscopic images.</p><p><strong>Defining model configuration and architecture</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*zTW9nRpqzoWbWth_wN9fCA.png" /></figure><p><strong>6. Customizing iPython writefile so we can write variables</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*Y7sqjF9km5Z79PWjdQPIeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*1CBFysp89lVG84_mCDxFug.png" /></figure><blockquote><strong>Using Nvidia-smi to retrieve gpu info and stats. Here, we are using Google Colaboratory so we have been provided with Nvidia Tesla T4</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/758/1*FlpXHV803VJ_XaFJx96wiw.png" /></figure><h4><strong>Model Training and Evaluation</strong></h4><p>Now, we will be starting the model training and evaluation. For evaluation we will be using<strong><em> Mean Average Precision mAP@0.5.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYV4FPGHNHqUnDhGb1pysQ.png" /></figure><p><strong>Let’s have a quick of our ground truth data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*IpDZu975VSahKdDBXpI7uw.png" /></figure><p><strong>Printing the augmented training data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*9U1z-wXMUFj_gYqCJNhOXQ.png" /></figure><p><strong>Printing the evaluation results</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*_KSIUaA5ubi7weCZPDYfaw.png" /></figure><h3><strong>Conclusion</strong></h3><p>I hoped you enjoyed training custom YOLO v5 object detector!</p><p>YOLO v5 is lightweight and extremely easy to use. YOLO v5 trains quickly, inferences quickly, and performs well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65ea86104c51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discussing Neural Networks]]></title>
            <link>https://aastha-eng.medium.com/discussing-neural-networks-4627956dfeb?source=rss-5f308360d2e5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4627956dfeb</guid>
            <dc:creator><![CDATA[Aastha Singh]]></dc:creator>
            <pubDate>Thu, 18 Feb 2021 18:05:27 GMT</pubDate>
            <atom:updated>2021-05-30T00:33:28.788Z</atom:updated>
            <content:encoded><![CDATA[<h3>Welcome Readers!</h3><p>For most marketers, 2020 was a year where…well, let’s just say it was a year of surprises for everyone. It’s 2021 now and taking the year by the storm is the buzzword, Artificial Intelligence and Machine learning. It is not just the tech world’s internet sensation, but has open up new possibilities in art and entertainment, enabling rich and deeply interactive experiences. <br>I think it is incredible how writers and directors has put the complicated thing like an AI into the drama as easy as casual conversations. Although the term may sound intimidating to the millennials and gen z, but it has been in use for decades and its applications are more common than we might imagine. In the minds of most of the common coders when they hear artificial intelligence and/or machine learning, the first thing that pops up is neural networks.</p><p>With an idea of being open to different perspectives, connecting with all the coders and libertarians I welcome you to my first blog!</p><h3>Talking about Neural Networks</h3><p>Neural networks are designed to work just like the human brain does. It has the ability to derive meaning from a data, by extracting patterns and detecting trends that are too complex to be noticed by either humans or other computer techniques. <br> Neural networks helps in recognition, for example you see a face and detect if it’s a ‘female’ or ‘male’ and you do so by analysing the visible features and patterns. Being sensitive about the topic of genders, let me be very frank, assuming someone’s gender is transphobic and we should never assume anyone’s gender based on how they ‘look’, hence the AI goes under research and development to create a sync between both body- and gender-recognition system as mentioned by Dr. Małgorzata Izabela Rećko<strong> </strong>in <a href="https://www.researchgate.net/publication/343658844_The_Subjectivity_of_Body_Perception_and_Gender_Identity_in_the_field_of_Artificial_Intelligence_Research"><em>The Subjectivity of Body Perception and Gender Identity in the field of Artificial Intelligence Research</em></a></p><p>So, basically <strong>you have an intelligent brain if you are a great pattern detector! </strong>Providing further references that you must check out is about mathematician John Nash, whose life and work was detailed in the movie and book, <a href="https://en.wikipedia.org/wiki/A_Beautiful_Mind_(film)"><strong>“A Beautiful Mind</strong></a><strong>” </strong>in 2001, that has a scene early in the movie John Nash, is taken to a room in the Pentagon and shown a wall filled with seemingly random digits. “The computer can’t detect a pattern, but I’m sure it’s code,” Nash stares for a long time at the digits, and he turns to the general. “I need a map,” he says. He has found geographic information in the patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*wswuGsOallyh5m75xUWkBw.jpeg" /><figcaption>John Nash in Pentagon behind big screen of random digits</figcaption></figure><p>Coming ahead in time, to 2016 from movies about neural networks to films entirely written by an artificial intelligence bot using neural networks <a href="https://www.thereforefilms.com/films-by-benjamin-the-ai.html"><strong><em>Sunspring</em></strong> </a>is a magnificent experimental science fiction that took <br>The script of the film is authored by a recurrent neural network called long short-term memory (LSTM) by an AI bot named Benjamin is the world’s first automatic screenwriter, it is a self-improving LSTM RNN machine intelligence trained on human screenplays conceived by Goodwin and Sharp. It was trained to write the screenplay by feeding it with a corpus of dozens of sci-fi screenplays found online.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*x2Ax1VydNhX3p1o6sdnN2Q.png" /><figcaption>Film synopses by Benjamin, a robot by Ross Goodwin &amp; Oscar Sharp</figcaption></figure><h3><strong>How do Neural Networks work?</strong></h3><p>Neural Networks consist of a number of highly interconnected elements or nodes, called ‘neurons’, which are organised in layers which processes the information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*GjULk1RHRdYFJC_nlem4wg.png" /></figure><p>Patterns to be recognised are introduced to the neural network by the <em>input</em> <em>layer </em>where each<em> </em>neuron computes a weighted sum of its inputs-the larger an input’s weight, the more input that affects the neuron’s output. This weighted sum is then fed into a non-linear function called an <em>activation function </em>— a step that enables neural networks to model complex non-linear phenomena and is communicated to one or more <em>hidden layers</em> present in the network. It is in the hidden layers where all the processing actually happens and there can be multiple hidden layers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*IT0YK1B3cUpnnmo4Hl_k1A.png" /><figcaption>Neural networks are organised in layers, with inputs from one layer connected to outputs from the next layer.</figcaption></figure><p>Let’s understand this with an example. Suppose you have a set of labeled images of diseased and non-diseased crops, and you need to create a neural network that would classify the two into their respective categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iTDR81wBm5P4V0p8p8c0lA.jpeg" /><figcaption>Plant disease detection by image classification</figcaption></figure><p>&gt;The image of the crops is first broken into pixels, then these pixels are then represented as matrices to make them eligible to enter the input layer of the neural network.</p><p>&gt;The <a href="https://towardsdatascience.com/what-the-hell-is-perceptron-626217814f53">perceptrons</a> of the artificial neural network accept and process the input by passing them from the input layer to the hidden layers and finally to the output layer.</p><p>If the network classifies an image correctly, weights contributing to the correct answer are increased, while other weights are decreased. If the network misclassifies an image, the weights are adjusted in the opposite direction.This procedure allowed early neural networks to “learn” in a way that superficially resembled the behavior of the human nervous system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4627956dfeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>