<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nikhila Munipalli on Medium]]></title>
        <description><![CDATA[Stories by Nikhila Munipalli on Medium]]></description>
        <link>https://medium.com/@nikhilamunipalli?source=rss-2cbe77e8d3e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*fjuoj-PGP53ve7vr.</url>
            <title>Stories by Nikhila Munipalli on Medium</title>
            <link>https://medium.com/@nikhilamunipalli?source=rss-2cbe77e8d3e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:01:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nikhilamunipalli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Word2Vec on 100GB+ Datasets — 10X faster and 50X Less RAM!]]></title>
            <link>https://medium.com/@nikhilamunipalli/word2vec-on-100gb-datasets-10x-faster-and-50x-less-ram-4af5d30a9ad3?source=rss-2cbe77e8d3e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4af5d30a9ad3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nikhila Munipalli]]></dc:creator>
            <pubDate>Sat, 17 Sep 2022 12:55:29 GMT</pubDate>
            <atom:updated>2022-09-17T13:06:11.382Z</atom:updated>
            <content:encoded><![CDATA[<h3>Word2Vec on 100GB+ Datasets — 10X faster and 50X Less RAM!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*qBnWngVEzC4rqGui_FstDw.jpeg" /><figcaption>A sick word2vec joke!</figcaption></figure><p>Gensim Word2Vec has been the most popular word embedding model for years. It hardly takes 5 lines of code to process raw text, train the model from scratch, and perform essential functions with the trained model. However, with huge datasets, the available functions to preprocess and train demand high computational power and significant time.</p><p>I hear you! I struggled with figuring out a better way too! This tutorial aims at outlining a few hacks I used to process raw text and train word2vec models in around 8GB RAM and within 12 hours&#39; time on a 150 GB-sized dataset.</p><h3>Setup:</h3><h4>A brief about my dataset:</h4><p>I had 52 CSV files corresponding to every week of a year, each row of the file representing 1 job posting. There are 20+ extracted/ raw fields for each posting, out of which 1 column represented the raw text of the job posting. This JobText column should be used to build the word2vec model.</p><h4>Specification of my server:</h4><p>I had access to a pretty powerful company server with 1.5TB RAM, 32 Core CPU, 48GB GPU, and 100TB disk space.</p><h4>Preprocessing function:</h4><p>I couldn’t use the available Gensim preprocessing function as I had some custom tokenization to be done on the text. Hence, I defined a function that can be applied on all job postings which would return the tokenized form of each sentence for a given description.</p><pre>def preprocess(description):<br>    //split into sentences <br>    //regex,stemming etc on each sentence<br>    // split each processed sentence into list of tokens/words<br>    return [[token1, token2, ... ], [token1, token2, ...], ]</pre><h3>Experiments:</h3><h4>Approach 1 — List Comprehensions:</h4><p>My initial bruteforce approach was to have a global list in RAM to which every list of tokenized sentence in every job description is appended using the preprocess function. I used list comprehensions to convert every JobText to a list of lists.</p><pre>processed_sentences = []<br>for week in tqdm(range(1,53)):<br>    week_data = pd.read_csv(data_path+ week+&#39;.csv&#39;)<br>    processed_sentences.append([preprocess(x) for x in tqdm(list(week_data[&#39;tokenized&#39;])]</pre><h4>Approach 2 — Pandas Apply:</h4><p>I soon realized that pandas apply would be much faster than a list comprehension.</p><pre>processed_sentences = []<br>for week in tqdm(range(1,53)):<br>    week_data = pd.read_csv(data_path+ week+&#39;.csv&#39;)<br>    week_data[&#39;tokenized&#39;] = week_data.apply(preprocess)<br>    processed_sentences.append(list(week_data[&#39;tokenized&#39;])) </pre><p>Pandas apply was relatively faster than a list comprehension and could reduced a week data’s processing time from nearly 2hrs to 45mins.</p><p>I found out this library — swifter which is known to make pandas 10X faster with a single word addition.</p><pre>import swifter<br>processed_sentences = []<br>for week in tqdm(range(1,53)):<br>    week_data = pd.read_csv(data_path+ week+&#39;.csv&#39;)<br>    week_data[&#39;tokenized&#39;] = week_data.swifter.apply(preprocess)<br>    processed_sentences.append(list(week_data[&#39;tokenized&#39;]))</pre><p>However, this couldn’t make processing any faster as I got to know that swifter is not equiped to optimize string processing.</p><p>Here’s the main obstacle! The size of the main list grew so much that it occupied 90% of the RAM by the end. With just 10% of the RAM remaining for the Word2Vec model to train, memory errors were all on the way.</p><p>For the years of inflation and Covid-19 (the dataset size was a little less than 100 GB) It took almost 20 hrs to generate the list of training samples and 18 more hours to train the word2vec model. Models for other years couldn’t be built with the memory issue.</p><h4>Approach 3 — Incremental Learning:</h4><p>As a naive deep learning developer, I looked for options that facilitate online training or incremental learning so that all the data will not be loaded into RAM at a time.</p><p>Gensim has the support for online training. The latest update of Gensim provides a parameter : update which when set to True helps in updating the model weights and vocabulary vectors with new words. It took me an average of 15hrs to trainthe model incrementally with data not being loaded wholly into the RAM.</p><p>This seemed to be one of the most vital optimization but little did I know about catastrophic forgetting! There are 2 main drawbacks of training word2vec incrementally:</p><ol><li>Vocabulary is mostly fixed/ dependent on the first batch of data sent to the model.</li><li>Training neural networks sequentially often results in something called catastrophic forgetting where the weights obtained by training on the previous batch are lost/ forgotten with a new batch of data.</li></ol><p>This brings us to the the end of all the bad decisions I made 😕! The below methods discuss how to tackle these problems!</p><h4>1. 50X Memory Optimized Approach:</h4><p>Efficiency is often defined in terms of time and space complexity (cliché)! My main goal was to optimize space, as for years when the job market was flourishing, my dataset was nearly 200GB which after preprocessing could not fit into my 1.5TB RAM 😢.</p><p>I later discovered that Gensim can take up an alternate format of sentences (not just the list of list of tokens). This format is a file/ files of ‘\n’ separated sentences where each sentence is a processed space separated line.</p><p>This still requires processing but instead of returning a list of list of tokens, my process function should return just a string in which the sentences are first processed with the required steps and then joined by ‘\n’. This processed string is then stored in a single file of a designated directory.</p><pre>def preprocess(description):<br>    //split into sentences <br>    //regex,stemming etc on each sentence<br>    //join all processed sentences with &#39;\n&#39; <br>    //return joined string</pre><p>Instead of appending to a global list, each week’s processed strings are directly stored in a text file which ensures that only 1 week’s data is present in RAM at a time.</p><pre>for week in tqdm(range(1,53)):<br>    week_data = pd.read_csv(data_path+ week+&#39;.csv&#39;)<br>    week_data[&#39;tokenized&#39;] = week_data.apply(preprocess)<br>    processed_sentences = list(week_data[&#39;tokenized&#39;]))<br>    text_file = open(output_path+ week+&#39;.txt&#39;, &#39;w&#39;)<br>    text_file.write(processed_sentences)<br>    text_file.close() </pre><p>Once all the files are created, Gensim’s PathLineSentences function is used to train the word2vec model.</p><pre>from gensim.models import Word2Vec<br>from gensim.models.word2vec import PathLineSentences<br>from gensim.models.callbacks import CallbackAny2Vec<br>import multiprocessing</pre><pre>model = Word2Vec(PathLineSentences(input_dir), <br>                  vector_size=300, <br>                  window=5,<br>                  workers=multiprocessing.cpu_count() * 2,<br>                  compute_loss=True, <br>                  callbacks=[callback()],<br>                  epochs=5)</pre><p>On the whole, processing each file and using PathLineSentences took close to 8GB RAM, solving all memory issues!</p><h4>2. 10X Time and 50X Memory Optimized Approach:</h4><p>Time taken to process and train is still an area that needed improvement in the previous approach.</p><p>Pandas apply function is the main step that was taking most of the time. I looked for several ways of making this apply faster. The one that worked is: <a href="https://modin.readthedocs.io/en/stable/">Modin</a>.</p><p>Modin is an act-in replacement to pandas meaning that all code written using pandas can be used without any changes just by replacing the import function. Instead of pandas, modin uses an immutable data structure that is wrapped with all pandas-like functionalities. It internally uses distributed computing frameworks such as Dask and Ray which handle multi-threading without any hastle!</p><pre>import modin.pandas as mdpd<br>for week in tqdm(range(1,53)):<br>    week_data = mdpd.read_csv(data_path+ week+&#39;.csv&#39;)<br>    week_data[&#39;tokenized&#39;] = week_data.swifter.apply(preprocess)<br>    processed_sentences = list(week_data[&#39;tokenized&#39;]))<br>    text_file = open(output_path+ week+&#39;.txt&#39;, &#39;w&#39;)<br>    text_file.write(processed_sentences)<br>    text_file.close()</pre><p>Surprise! The processing of all 52 weeks took just 2 hrs! Training the model on the whole data after processing took close to 10hrs, which could finally make my model ready in 12hrs!</p><p>One small issue with modin or ray is the garbage collection. modin doesn’t automatically handle the deletion of repetitive variables or overwrite memory when the variable is re-assigned. The garbage collection module (GC) of python is the savior here!</p><pre>import gc<br>gc.collect()</pre><p>Including these 2 lines after every function call overcomes this issue and efficiently handles memory.</p><h4>Bonus — Multiple Iterations</h4><p>Word2Vec also has another prominent issue which is — Determinism. Gensim’s Word2Vec is based on random initialization of neural weights. If carefully observed one can figure out that the embeddings generated on one run are likely to vary from those on another run on the same data. This makes a single run less trustworthy. Hence, I have 10 runs on the whole data and took an average of the embeddings with a confidence interval to report results.</p><p>And that’s it! We have word2vec trained efficiently and accurately! Thanks for reading! Won’t mind a clap 😃 🎆!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af5d30a9ad3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Starter Pack for Deep Learning in PyTorch — for Extreme Beginners -by a beginner!]]></title>
            <link>https://medium.com/@nikhilamunipalli/starter-pack-for-deep-learning-in-pytorch-for-extreme-beginners-by-a-beginner-330f3fdefcc4?source=rss-2cbe77e8d3e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/330f3fdefcc4</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Nikhila Munipalli]]></dc:creator>
            <pubDate>Thu, 18 Jul 2019 12:22:36 GMT</pubDate>
            <atom:updated>2019-07-23T15:31:57.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*FSD8imlH_Qnt3pfI8cUpOg.jpeg" /></figure><h3>All You Need To Know For Your First Ever Project in PyTorch !</h3><p>For Extreme Beginners — By a Beginner!</p><p><strong>Prerequisites :</strong></p><p>Welcome deep learning learners! This article is a kick start for your first ever deep learning project in pytorch.</p><p>To turn eligible to read this story, make sure any of these points best describe you :</p><ol><li>You must be someone who has “DL Enthusiast” in your LinkedIn profile but built models only with keras! Congrats! this article is completely dedicated to you!</li><li>You must have had hard time understanding sessions, variables, classes, etc in tensorflow and planning to switch to pytorch. Yep! you are in the right place!</li><li>You are someone who has built significantly complex enough models in pytorch and looking for some detailed stuff. Bye bye! you can skip to last section!</li></ol><h3>The Actual Start !</h3><p>If you are reading this, Congratulations! you are completely qualified to proceed further!</p><h4><strong><em>The First ever question :</em></strong></h4><p><strong><em>What is pytorch and how is it different from keras ?</em></strong></p><p>As a beginner, it is quite easy to build a neural network by adding Dense layers and some dropouts just by these 2 commands :</p><pre>from keras.models import Sequential </pre><pre>from keras.layers import Dense, Dropout</pre><p>Well it is as easy as saying….</p><pre>import Andrew as ng !</pre><p>If you have to deal with tensors, build models which are complex enough, create some custom loss functions, or understand what exactly is happening within every single layer, pytorch is under rescue! It is a perfect tool for n-dimensional data !It has that simplistic keras-i edge, paired with loads of useful documentation.</p><blockquote>It i<strong>s beginner-friendly!</strong></blockquote><p>Pytorch is known for<strong> </strong>its debugging capacities, great community support and low level abstraction. Faster implementations are better done with pytorch as it doesn’t involve heavy computational junk as in keras.</p><h3><strong>Diving deep into it !</strong></h3><p><strong><em>Step 1</em></strong> : <strong><em>Hardware setup</em></strong> : Deep learning models are always accompanied with lots of data. Obviously, a gpu is must. for cuda and nvidia setup : refer <a href="https://missinglink.ai/guides/deep-learning-frameworks/complete-guide-deep-learning-gpus/">this</a> great article.</p><p><strong><em>Step 2</em></strong> : <strong><em>Installation</em></strong> : Pytorch is installed within few seconds, with one line of ctrl+C — ctrl+V from their official website. Go to their <a href="https://pytorch.org/">website</a>, scroll down, select your setup specifications, copy that link and paste it in your terminal! Done! ready to use it! Also install tensorboardX for visualization of results. Refer this <a href="https://github.com/lanpa/tensorboardX">github repo for</a> installation instructions.</p><p><strong><em>Step 3 : Basics : </em></strong>There are great courses on udemy that discuss basic syntax, usage and functions. Trust me! pytorch doesn’t require that much of effort! It is very similar to Numpy and has many predefined functions. You can discover things once you start coding.</p><p><strong><em>The fun part !</em></strong></p><p><strong><em>Step 3</em></strong> :<strong><em> Importing necessaries</em></strong> :</p><pre>import torch</pre><pre>import torch.nn as nn</pre><pre>from torch.utils import data</pre><pre>import torch.nn.functional as F</pre><pre>from torch.autograd import Variable as V</pre><pre>import torch.optim as optim</pre><pre>from tqdm import tqdm</pre><pre>from tensorboardX import SummaryWriter </pre><p>These are the libraries essential for any deep learning model. nn module has all the necessary loss functions, layers, sequential models, activation functions etc. nn.functional is a specific segment used for activation functions. Remaining are discussed as you proceed further.</p><p><strong>The second ever question :</strong></p><p><strong><em>What is the difference between TorchTensor and a Variable ?</em></strong></p><p>You must have noticed that we imported Variable library from torch.autograd . Tensors are just nd-arrays. For functions such as back propagation, tensors have to be coupled with their associating gradient. A torch Variable does the same. It is a tensor with its corresponding gradient, making all tensor operations quite easy! Have a look at<a href="https://discuss.pytorch.org/t/what-is-the-difference-between-tensors-and-variables-in-pytorch/4914"> this </a>q&amp;a for details.</p><p>With all this foundation, lets start building our model!</p><h3>First Milestone : The Network class :</h3><p><strong><em>step 1 : Inheritance :</em></strong> To build an architecture of neural network, a class must be created, that is inherited from nn.Module. nn.Module gives a structure for creating our network.</p><pre>class Network(nn.Module):</pre><pre>      def __init__(self):</pre><pre>          super(Network, self).__init__()</pre><p><strong><em>step 2: Layers </em></strong><em>: </em>Different types of layers such as linear, LSTM, dropout, etc are directly present in nn module. If you are used to keras sequential model setup, nn.sequential is just like that. Personally, i suggest not to use sequential as it won’t bring out the true purpose of using pytorch. Better way to add layers to your model is :</p><ol><li>Create one layer with nn and assign it to a private member of your network class.</li></ol><p>It looks something like this :</p><pre>self.linear = nn.linear(input dim, output dim)</pre><p>Declare all your layers as class variables in init of your Network class.</p><p><strong><em>Step 3 : The forward function :</em></strong> This is the heart and soul of your network. Whenever you are calling your architecture for training or predicting, the steps you write in forward function are executed. Here, we are overwriting the forward function from nn.Module class, which tells the model exactly what to do. If you mention any parameters for your forward function apart from self, those have to be passed everytime the model is called.</p><p>For example ,</p><pre>def forward(self,x): </pre><pre>     out =  self.linear1(x)</pre><pre>     out = self.linear2(out)</pre><pre>return out</pre><p>One thing you can notice here is connection between 2 layers is established by passing outputs as torch tensors or Variables.</p><p><strong><em>Step 4 : Additional functions : </em></strong>With the above steps, all “to dos” are done! Sometimes, when your model has LSTM layers, it requires a functionality of initializing hidden layers. Likewise, if you are trying to build a Boltzmann machine network, sampling hidden and visible nodes is required. so new member functions oto our Network class can be created and used.</p><p><strong><em>Step 5 : Parameters of your class </em></strong>: While building the network with a class, make sure you are using <strong>minimum or no hard coded values</strong>. Learning rates, sizes of hidden layers, etc can be passed while instantiating your class.</p><p>If you successfully understood all the above steps and could visualize how exactly your network class looks,</p><p>🎉congrats! a 100/100 on your first milestone!!👏</p><p>Since this milestone was long enough, lets look at a small yet an important one!</p><h3>Second Milestone : The Custom DataLoader :</h3><p>Ever done train test split, train val split, batch size splits, shuffles etc for your data from scratch? Keras has parameter “batch size” that takes cares of irregular sizes of batches. But, if you have to do it in pytorch, it takes quite a lot of effort.</p><p>Dont worry! custom DataLoader is here!</p><p>DataLoader is a term used to create training, testing and validating data from your given data, with features such as batch size, shuffle and no of workers.</p><p>pytorch has a default dataloader class, which can be imported with torch.utils.data.dataloader. You can directly make your data model ready!</p><p>Instead of using the default DataLoader, it is quite easy to build a custom DataLoader as per your requirement. Let’s see how to do that!</p><p>Do you remember importing data from torch.utils? Hopefully yes!</p><p>Building a custom dataloader is as simple as executing the below steps:</p><p><strong><em>Step 1 : Inheritance :</em></strong> Similar to the way we inherited nn.Module for our network class,we are gonna inherit data.Dataset class for our DataLoader. You don’t have to call init constructor as done before.</p><p>Something like this :</p><pre>class dataset(data.Dataset):</pre><p><strong><em>Step 2. sequences and labels :</em></strong> data.Dataset class has members called sequences, which refer to X data or training data and labels refer to y. You can create an option called partition that would return its corresponding data.</p><p>for example,</p><pre>class dataset(data.Dataset):</pre><pre>     def __init__(self, partition):</pre><pre>         if partition==’train’:</pre><pre>            self.sequences = X_train</pre><pre>            self.labels = y_train</pre><p>You can make sure that your data is of torch.tensor type by converting it intially.</p><pre> self.sequences = torch.tensor(self.sequences)</pre><p><strong><em>Step 3 : the len method:</em></strong> __len__() is a method of data.Dataset class, that has to be overwritten for our dataloader to work. Just return the length of sequences generated, under this method.</p><p><strong><em>Step 4 :the getitem method : </em></strong>__getitem__ (self, index) is a method that returns sequences and labels at a given index. This has to be overwritten to make a fully functional dataloader.</p><p><strong><em>Step 5: generators and parameter dict : </em></strong>With the above steps, creating the custom dataloader class is done. Now, it’s time to use it!</p><p>Data, which is being sent by dataloader is used in the form of generators. So, a parameter dict has to be created, based on which, data is generated. this dict contains keys which are : batch_size, shuffle, num_workers. object for dataloader class will be instantiated and passed to a generator along with parameter dict:</p><pre>params= {</pre><pre>‘batch_size’ : 10, </pre><pre>‘shuffle’ : True,</pre><pre>‘num_workers’ : 20}</pre><pre>training_set = dataset(X, y, ‘train&#39;)</pre><pre>training_generator = torch.utils.data.dataloader(training_set, **params)</pre><p>Look at the official documentation for more parameters<a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html"> here</a>.</p><p>Got your data quite effortlessly ?! if yes,</p><p>🎊Kudos!! Another 100/100 on you second milestone!👏</p><p>Now that we have both architecture and data , our model is ready to rock and roll! So let’s move on to our 3rd milestone !</p><h3>Third Milestone : The Train Function :</h3><p>You have 2 classes now, one for architecture and the other for data. Our train function is something that is gonna combine both and let our model learn.</p><p><strong><em>Step 1 : Parameters : </em></strong>The first and most important parameter of our train function is the object of our Network class. Next come training generator and validation generator from dataloader. After these compulsory parameters, number of epochs, learning rate, batch size, clip, loss, etc can be passed. As mentioned, do not hard code values. Instead pass them as parameters.</p><p><strong><em>Step 2 : Initialization : </em></strong>Considering “net” to be the object of our network class. To initiate training, it has to be set to train mode. This is done by the command :</p><pre>net.train()</pre><p>Loss functions and optimizers have to be assigned using nn module.</p><p>For example :</p><pre>criterion = nn.MSELoss() </pre><pre>optimizer = torch.optim.Adam(net.parameters(), lr=lr)</pre><p>Now since we initialized values, model is ready to get trained.</p><p><strong><em>step 3 : the epoch loop :</em></strong> Start a loop that iterates n times, where n refers to number of epochs. At the end of loop, print losses (both train and validation ) corresponding to that epoch.</p><p><strong><em>Step 4: the tqdm loop : </em></strong>Loops may run forever! Especially for deep learning models with lots of data! It is quite difficult to figure out what is happening inside and how much time would it take to finish. tqdm is here! It is a progress bar for every iteration. Look at its official documentation <a href="https://tqdm.github.io/">here</a>. An iterator must be assigned to tqdm and used in loop.</p><p>For every epoch, our function iterates over batch size. So our iterator here is training generator.</p><p>tqdm loops look something like this :</p><pre>t = tqdm(iter(training_generator), leave=False, total=len(training_generator))</pre><pre>for i, batch in enumerate(t): </pre><pre> x_batch, y_batch = next(iter(training_generator))</pre><p><strong><em>Step 5: gpu availability : </em></strong>Variables, objects, etc can be transfered to gpu by one single pytorch command.</p><p>Include one line :</p><pre>torch.cuda.set_ device(0)</pre><p>in the start of your code. 0 refers to gpu number to be used.</p><p>Whenever you feel necessary that some variable must be transfered to cuda,</p><p>do : X = X.cuda()</p><p>However, Our code should run irrespective of underlying hardware. If gpu doesn’t exist, you can’t transfer variables to cuda. The code shows an error in that line. In order to get rid of these dependencies, consider a variable, that holds a boolean value of gpu availability.</p><p>Your code can be simply modified as :</p><pre>if gpu_available:</pre><pre>    X= X.cuda()</pre><p><strong><em>Step 6: Train procedure:</em></strong> After setting up everything , training is started with the architecture class object as :</p><pre>output = net(x) </pre><p>Here the parameters passed are those mentioned in forward function.</p><p>After outputs are obtained, loss is computed as :</p><pre>loss = criterion(output, targets)</pre><p>Back propagation is performed with :</p><pre>loss.backward()<br>optimizer.step()</pre><p>Optimizer.step() is used to update parameters( weights and biases ) based on loss which is back propagated.</p><p>You can print the progress of loss with tqdm iterator as :</p><pre>t.set_postfix(loss=train_loss)</pre><p><strong><em>Step 7: Evaluation : </em></strong>After training, evaluation can be done, for which model should be set to evaluation mode:</p><pre>net.eval()</pre><p>At the end of evaluation, make sure you set back the mode to training with net.train().</p><p><strong><em>step 8: saving the model and state_dict :</em></strong> After you successfully finish training your model, you have to save it for future use. There are 2 prominent ways in which a trained model can be saved in pytorch.</p><ol><li><strong>torch.save() :</strong> This method saves the entire model, with its directory structure. It can be used with 2 lines of code:</li></ol><pre>torch.save(net, path)       #saving<br>net = torch.load(path)      #loading</pre><p>2. <strong>model.state_dict() : </strong>state_dict function stores just the parameters of your model (weights, biases, etc) instead of saving the entire model. To use the saved model, an object of the architecture class must be created and corresponding parameters must be assigned.</p><pre>torch.save(net.state_dict(), path)       #saving<br>net = Network(*args, **kwargs)           #loading<br>net.load_state_dict(torch.load(path))</pre><h4>The third ever question :</h4><blockquote><strong>When to use torch.save() and when to use model.state_dict() ?</strong></blockquote><p>As mentioned above, torch.save saves the entire model. To load it, you don’t have to instantiate your Network class. In cases when your model should work on a completely different platform irrespective of underlying code, torch.save() is highly favorable.</p><p>Contrarily, if you are performing experiments, state_dict() is much favorable as it just saves the parameters of your model and is flexible for any further modifications. it is relatively memory friendly.</p><p>Look at <a href="https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch">these</a> stackoverflow answers for more details.</p><p>Congrats! if you are here, you almost built your model to perfection! Just to organize your code and perform experiments, let’s look at one final milestone. btw,</p><p>🎉a 100/100 for this milestone too!! 🎊</p><h3>Final Milestone : Organizing</h3><p>As numerous experiments, parameter tunings are often done on a deep learning model, it is very important to store them in a proper directory structure. These are few steps to be followed :</p><p><strong><em>Step 1 : Config file :</em></strong> Parameters such as learning rate, loss function, no of epochs are always changed. To keep track of all these parameters, create a config file and mention all those parameters which have scope to play around with. Instead of mentioning values while instantiating a class or calling a function, you can import these variables as :</p><pre>from config import * </pre><p>and use. When you have to perform various trial and error methods, changing your config file would suffice.</p><p><strong><em>Step 2 : Tensorboard :</em></strong> Remember importing SummaryWriter from tensorboardX? As mentioned before, visualization of loss, accuracies etc can be done with a tensorboard.</p><p>To include tensorboard just add one line before your train function :</p><pre> writer = SummaryWriter(path)</pre><p>In a loop that goes on for every epoch, include :</p><pre>writer.add_scalar(‘loss’, loss, epoch_number)<br>writer.add_scalar(&#39;accuracy&#39;, accuracy, epoch_number)</pre><p>You can add as many graphs as you want, with various parameters.</p><p>At the end of training, close writer with :</p><pre>writer.close()</pre><p>Remember to change path of subsequent tensorboard writings as graphs may be overwritten or overlapped.</p><p><strong><em>Step 3 : requirements file :</em></strong> May be too cliché to mention, but it is a good practice to have a requirements file that contains all the libraries with their versions used. A single command would deal with all your dependencies.</p><p>Look at this<a href="https://stackoverflow.com/questions/31684375/automatically-create-requirements-txt"> link </a>to know how to freeze all your requirements.</p><p><strong><em>Step 4 : A readme : </em></strong>After all the hardwork, its your right to brag about your work and guide people to use your code! A readme does the same. Wind up your project by adding a qualitative and quantitative readme!</p><p>Now, done!!</p><p>Finallyy!!! Congratulations for successfully finishing all the four milestones! You receive another 100/100 for final milestone and overall :</p><blockquote>🎊“The best beginner in pytorch” award !!🎉</blockquote><h3>Additional : Running Multiple Experiments — The JSON Trick !</h3><p>For large scale experiments, it takes lot of human intervention to modify config file everytime you run an experiment. To minimize this, one set of config parameters can be converted to a single json object.</p><p>Considering your config file looks like this :</p><p>lr = 0.01</p><p>epochs =100</p><p>……</p><p>Below code converts whole text to a json object.</p><pre>import json<br>data = dict(zip(parameter, value))   #create a dictionary<br>json.dumps(data)    #json object</pre><p>Multiple JSON objects can be created, each representing an individual experiment. These will form an array of JSON objects which can be recursively traversed by passing to main only once !</p><h3>A Better Conclusion !</h3><p>Let’s quickly look at the complete outline of our project structure :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PFRCpZWrVSYSMVMgAFoq7A.jpeg" /></figure><p>Hope your project structure looks the same! Feel free to comment any doubts!</p><p>End of this article! Take a break, build a network! 😬</p><p>P.S : I thank <a href="https://medium.com/u/1b64daa76e79">A.Manoj Ghuhan</a> and Niyati Chhaya for guiding all the way, correcting my silly mistakes and turn me confident in the topic!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=330f3fdefcc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple Guide To “KALDI” — an efficient open source speech recognition tool for Extreme Beginners —…]]></title>
            <link>https://medium.com/@nikhilamunipalli/simple-guide-to-kaldi-an-efficient-open-source-speech-recognition-tool-for-extreme-beginners-98a48bb34756?source=rss-2cbe77e8d3e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/98a48bb34756</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Nikhila Munipalli]]></dc:creator>
            <pubDate>Tue, 29 May 2018 16:57:30 GMT</pubDate>
            <atom:updated>2018-05-30T02:21:04.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*WXvew0lSjoQaboklPnB7ZQ.jpeg" /></figure><h3>Simple Guide To “KALDI” — an efficient open source speech recognition tool for Extreme Beginners — by a beginner!</h3><p>For those who are completely new to speech recognition and exhausted searching the net for open source tools, this is a great place to easily learn the usage of most powerful tool “KALDI” with perfectly defined steps.</p><p>Note: This tutorial is for Ubuntu 16.04. It uses tedlium english dataset for ease. Uses docker and gstreamer.</p><p><strong>To turn eligible for reading this story, make sure these points fit for you :</strong></p><ol><li>Make sure you atleast know the names of available open source tools with their accuracies. For lazy ones like me I state few popular free speech recognition tools below :</li></ol><p>a. Kaldi</p><p>b. CMU Sphinx</p><p>c. Deep Speech</p><p>d. HTK</p><p>e. Simon</p><p>these tools are ordered according to their popularity, efficiency and complexity ;).</p><p>2. Be ready to work hard and to witness many many errors! Few errors can be resolved by a simple stackoverflow solution and few make you re-install kaldi all together!</p><p>3. before you start with the below mentioned procedure, I highly recommend you to struggle for atleast 2 days with traditional way of using kaldi. If that works well (no pun intended ;)..), you dont need to follow this one! Below I mention links which you are advised to follow step wise for using kaldi.</p><p>a. Downloading and installation : <a href="http://jrmeyer.github.io/asr/2016/01/26/Installing-Kaldi.html">http://jrmeyer.github.io/asr/2016/01/26/Installing-Kaldi.html</a></p><p>If you feel contented after running the yesno example in the above link, wait for the monster to eat your brain away!</p><p>b. Starting with your own dataset : <a href="http://kaldi-asr.org/doc/kaldi_for_dummies.html">http://kaldi-asr.org/doc/kaldi_for_dummies.html</a></p><p>Once you succeed executing the run.sh file, Congrats! you finished reading my story!! If run.sh just lands you into errors and errors, proceed further :P .</p><p>c. Repeat above tutorial until you are extremely tired and give up. The next alternative you land into is running the available examples in “egs” folder.</p><p>Running examples : <a href="http://kaldi-asr.org/doc/tutorial_running.html">http://kaldi-asr.org/doc/tutorial_running.html</a></p><p>Atleast if this works (or you have an LDC subscription) , you are free to leave this story!</p><p>d. After exploring the “rm” data as used in the official link, you realize that freely available datasets do some good for you. Below are few links for free datasets.</p><p>voxforge : <a href="http://www.voxforge.org/home/downloads">http://www.voxforge.org/home/downloads</a></p><p>AMI : <a href="http://groups.inf.ed.ac.uk/ami/download/">http://groups.inf.ed.ac.uk/ami/download/</a></p><p>librispeech : <a href="http://www.openslr.org/12/">http://www.openslr.org/12/</a></p><p>If you have a good GPU with fast internet connection and 20–25 GB space for a dataset, you can go with the links.</p><p>If all steps bring you back, Congrats! you are completely qualified for reading this tutorial.</p><p><strong>Kaldi- made easy steps start here :</strong></p><p><strong>step 1</strong> : Before you start with kaldi learn the foundation of docker with this simple video tutorial. : <a href="https://www.youtube.com/watch?v=pGYAg7TMmp0">https://www.youtube.com/watch?v=pGYAg7TMmp0</a></p><p><strong>step 2</strong> : installation of docker : <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">https://docs.docker.com/install/linux/docker-ce/ubuntu/</a></p><p>That was a great intro to docker. I advise you to learn docker usage with the official documentation : <a href="https://docs.docker.com/get-started/">https://docs.docker.com/get-started/</a>.</p><p>As already said lazy ones can skip learning.</p><p><strong>step 3</strong> : downloading tedlium dataset :<a href="https://phon.ioc.ee/%7Etanela/tedlium_nnet_ms_sp_online.tgz">https://phon.ioc.ee/%7Etanela/tedlium_nnet_ms_sp_online.tgz</a></p><p>This file must be stored in media/kaldi_models directory.</p><p>to access media, go to computer&gt;media</p><p>open terminal there and make directory by running this command.</p><p>sudo mkdir kaldi_models</p><p>move the downloaded file, after extraction, from downloads to this directory by this command.</p><p>sudo mv /downloads/english media/kaldi_models/</p><p>this dataset is 1.4GB which is neither too big nor too small!</p><p><strong>step 4</strong> : pulling the docker image from dockerHub :</p><p>sudo docker pull jcsilva/docker-kaldi-gstreamer-server</p><p>run this command to download the image.</p><p><strong>step 5</strong> : yaml file download : <a href="https://github.com/alumae/kaldi-gstreamer-server/blob/master/sample_english_nnet2.yaml">https://github.com/alumae/kaldi-gstreamer-server/blob/master/sample_english_nnet2.yaml</a></p><p>from the above link download yaml file and name it nnet2.yaml.</p><p>store the file in english directory.</p><p><strong>imp :open the file in the text editor and replace test/models with opt/models.</strong></p><p>this will be explained later.</p><p>In the file, comment out the line</p><p>“full-post-processor: ./sample_full_post_processor.py”</p><p>as you wont have the sample_full_post_processor.py file . This wont effect the functionality of yaml file.</p><p>for further information about yaml follow this great video : <a href="https://www.youtube.com/watch?v=cdLNKUoMc6c">https://www.youtube.com/watch?v=cdLNKUoMc6c</a></p><p><strong>step 6</strong>: sudo docker container ls</p><p>this command gives the list of available containers. if you find a container with image : jcsilva/docker-kaldi-gstreamer-server:latest, your container allocation is successful under the port 8080:80.</p><p><strong>step 7</strong> : getting inside the container :</p><p>docker run -it -p 8080:80 -v /media/kaldi_models:/opt/models jcsilva/docker-kaldi-gstreamer-server:latest /bin/bash</p><p>this gets you inside the container which can almost be used as a normal linux terminal. Docker partitions memory for its container. place the yaml file in opt/models.</p><p><strong>step 8 </strong>: starting master and worker by docker :</p><p>./start.sh -y /opt/models/nnet2.yaml</p><p>this will create master.log and worker.log in opt.</p><p><strong>step 9</strong> : run ls -l to see the available items in the container.</p><p>if the list contains :</p><p>gst-kaldi-nnet2-online, kaldi, kaldi-gstreamer-server, master.log, models, start.sh, stop.sh, worker.log</p><p>then your ./start was executed properly.</p><p><strong>step 10</strong> : run cat worker.doc to find whether worker is working.</p><p>you probably should encounter an error showing no path found which can be eliminated by modifying the conf files.</p><p>ls models/english/tedlium_nnet_ms_sp_online/conf/</p><p>this gives all the files in conf.</p><p>vi models/english/tedlium_nnet_ms_sp_online/conf/&lt;file&gt;</p><p>replace the &lt;file&gt; with the name of the file in conf folder. modify the test/models path anywhere if found to opt/models. View each file of conf folder by running the same command.</p><p>run cat worker.doc again to test the functionality. This time the error must be resolved and worker available message must be displayed.</p><p><strong>step 11</strong> : websocket url : <a href="http://www.websocket.org/echo.html">http://www.websocket.org/echo.html</a></p><p>enter the web page. in the location enter : ws://localhost:8080/client/ws/status and press connect. On connecting, if you find the message : RECEIVED: {“num_workers_available”: 1, “num_requests_processed”: 9} in log, then the connection is perfect.</p><p>Congratulations!! now you have a working kaldi speech recognizer(english) with gstreamer and docker.</p><p><strong>The fun part — showing it off!! :</strong></p><p>To show off the working of “your” kaldi, follow the steps.</p><p><strong>step 1</strong> : install the free konele app on any of your android devices from the play store. go to settings and choose the mode konele(fast recognition) in recognition services.</p><p><strong>step 2</strong> : For the konele app, you need to get the public link for your local kaldi. So to generate a public url, it must be hosted on a server. If no server is available, use free private hosting sites. A good one found is ngrok.</p><p>become a ngrok member : <a href="https://dashboard.ngrok.com/user/signup">https://dashboard.ngrok.com/user/signup</a> sign up in the above link and follow the installation guide.</p><p>download and unzip. run the command :</p><p>./ngrok authtoken &lt;token value&gt;</p><p>replace &lt;token value&gt; with your token. This adds your account to .yml file.</p><p>run the below command to know your public hosting link.</p><p>./ngrok http 80</p><p><strong>step 3</strong> : in the fast recognition mode, modify the WebSocket URL with the ngrok.</p><p>for example : if terminal contains :</p><p>forwarding http://<em>68ddf1fd.ngrok.io</em> -&gt;localhost : 8080</p><p>enter : ws://68ddf1fd.ngrok.io/client/ws/speech in the WebSocket URL section of konele.</p><p><strong>step 4</strong> : close the settings and start konele. tap and speak. let it transcribe and look at the results!</p><p>Finally, this tutorial has a happy ending!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98a48bb34756" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>