<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ankit Kothari on Medium]]></title>
        <description><![CDATA[Stories by Ankit Kothari on Medium]]></description>
        <link>https://medium.com/@ankitkothari_92911?source=rss-b9fe6cfaaee3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6BxMCUYNqzcrFMbm</url>
            <title>Stories by Ankit Kothari on Medium</title>
            <link>https://medium.com/@ankitkothari_92911?source=rss-b9fe6cfaaee3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 23:25:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ankitkothari_92911/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Docker: How much do we need to know as Data Scientists?]]></title>
            <link>https://medium.com/@ankitkothari_92911/docker-how-much-do-we-need-to-know-as-data-scientists-d5f695061ee8?source=rss-b9fe6cfaaee3------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5f695061ee8</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-scientist]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ankit Kothari]]></dc:creator>
            <pubDate>Mon, 23 Mar 2020 13:53:58 GMT</pubDate>
            <atom:updated>2020-03-24T06:04:23.986Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y6CvfE6WUgoIdT8Mp0Ev_g.png" /></figure><p>Docker is a platform to develop, deploy, and run applications inside containers. Docker is essentially synonymous with containerization.</p><p>Docker is very portable and can be used in a local machine or passed on to a different machine or a worker or stored somewhere in the cloud. It should behave the same.</p><p>Docker container has several mechanisms for interfacing with the outside world. It has ports that can be opened for interacting through the browser. You can configure it to interact with data through the command line.</p><p>In the case of a Docker container, an offsite registry keeps an image, which is like mold, for your container. Then when you need a container you can make one from the image.</p><p>With Docker, you can run multiple containers simultaneously on a host machine. And like other software programs, Docker containers can be run, inspected, stopped, and deleted. Images are the immutable master template that is used to pump out containers that are all exactly alike.</p><p><strong>Images</strong> are the immutable master template that is used to pump out containers that are all exactly alike.</p><p>A <a href="https://docs.docker.com/engine/reference/builder/"><strong>Dockerfile</strong></a> is a file with instructions for how Docker should build your image.</p><h3>Docker Container</h3><p>A Docker image plus the command docker run image_name creates and starts a container from an image.</p><p>The recipe (Dockerfile) tells us what we’re going to do. Here’s the plan:</p><ul><li>The crust is preformed and immutable, it’s like a basic ubuntu parent image. It’s the <em>bottom layer</em> and gets built first.</li><li>Then you’ll add some cheese. Adding this second layer to the pizza is like <em>installing an external library </em>— for example, NumPy.</li><li>Then you’ll sprinkle on some basil. The basil is like the <em>code in a file</em> that you wrote to run your app.</li></ul><h3>Docker Glossary Terms:</h3><blockquote><a href="https://docs.docker.com/engine/docker-overview/#the-docker-platform"><strong>Docker Platform</strong></a><strong> </strong>is<strong> </strong>Docker’s software that provides the ability to package and run an application in a container on any Linux server.</blockquote><blockquote><a href="https://www.docker.com/products/docker-engine"><strong>Docker Engine</strong></a><strong> </strong>is the client-server application.</blockquote><blockquote><a href="https://docs.docker.com/engine/docker-overview/"><strong>Docker Client</strong></a> is the primary way you’ll interact with Docker. When you use the <a href="https://docs.docker.com/engine/reference/commandline/cli/">Docker Command Line Interface (CLI)</a> you type a command into your terminal that starts with docker. Docker Client then uses the Docker API to send the command to the Docker Daemon.</blockquote><blockquote><a href="https://docs.docker.com/engine/docker-overview/"><strong>Docker Daemon</strong></a><strong> </strong>is the Docker server that listens for Docker API requests. The Docker Daemon manages images, containers, networks, and volumes.</blockquote><blockquote><a href="https://docs.docker.com/storage/volumes/"><strong>Docker Volumes</strong></a> are the best way to store the persistent data that your apps consume and create.</blockquote><blockquote>A<strong> </strong><a href="https://hub.docker.com/"><strong>Docker Registry</strong></a><strong> </strong>is the remote location where Docker Images are stored.<strong> </strong>You push images to a registry and pull images from a registry.</blockquote><blockquote>A<strong> </strong><a href="https://docs.docker.com/docker-hub/repos/"><strong>Docker Repository</strong></a> is a collection of Docker images with the same name and different tags. The <em>tag</em> is the identifier for the image.</blockquote><h3>Docker Advanced Terms:</h3><p>The following four concepts relate to using multiple containers at once.</p><blockquote><strong>Docker Networking </strong>allows you to connect Docker containers together. Connected Docker containers could be on the same host or multiple hosts.</blockquote><blockquote><strong>Docker Compose </strong>is a tool that makes it easier to run apps that require multiple Docker containers. Docker Compose allows you to move commands into a docker-compose.yml file for reuse. The Docker Compose command-line interface (cli) makes it easier to interact with your multi-container app. Docker Compose comes free with your installation of Docker.</blockquote><blockquote><strong>Docker Swarm</strong> is a product to orchestrate container deployment.</blockquote><blockquote><strong>Docker Services</strong> are the different pieces of a distributed app. From the docs:</blockquote><blockquote><em>Services are really just “containers in production.” A service only runs one image, but it codifies the way that image runs — what ports it should use, how many replicas of the container should run so the service has the capacity it needs, and so on. Scaling a service changes the number of container instances running that piece of software, assigning more computing resources to the service in the process.</em></blockquote><blockquote><a href="https://kubernetes.io/"><strong>Kubernetes</strong></a><strong> </strong>automates deployment, scaling, and management of containerized applications.</blockquote><p>Docker Commands</p><pre>$ docker container ls<br>$ docker container ls<br>$ docker container stop webserver<br>$ docker container ls -a<br>$ docker container rm webserver<br>$ docker image ls<br>$ docker image rm nginx</pre><p>Docker Containers are lightweight as compared to virtual machines because they don&#39;t need the extra load of the operating system but run directly within the host machine kernel</p><p>Each Docker image contains a file named <em>Dockerfile </em>with no extension. The Dockerfile is assumed to be in the current working directory when docker build is called to create an image. A different location can be specified with the file flag (-f).</p><p>FROM, RUN, COPY, and ADD create layers in the final image. Other instructions configure things, add metadata, or tell Docker to do something at run time, such as expose a port or run a command.</p><p>ARG defines a variable to pass from the command line to the image at build-time. A default value can be supplied for ARG in the Dockerfile, as it is in the example: ARG my_var=my_default_value.</p><p>Unlike ENV variables, ARG variables are not available to running containers. However, you can use ARG values to set a default value for an ENV variable from the command line when you build the image.</p><p>.dockerignore is similar to .gitignore. It’s a file with a list of patterns for Docker to match with file names and exclude when making an image.</p><p>There isn’t any need to use EXPOSE instruction in your own Dockerfile if the docker image is simply a web app. You might want to use it when you are distributing your Dockerfile to others and letting them connect to the application.</p><p>-p is short for --port. The port is the interface with the outside world.1000:8000 maps the Docker port 8000 to port 1000 on your machine. If you had an app that output something to the browser you could then navigate your browser to localhost:1000 and see it.</p><h3>A Dozen Dockerfile Instructions</h3><blockquote>FROM — specifies the base (parent) image.<br>LABEL —provides metadata. Good place to include maintainer info.<br>ENV — sets a persistent environment variable.<br>RUN —runs a command and creates an image layer. Used to install packages into containers.<br>COPY — copies files and directories to the container.<br>ADD — copies files and directories to the container. Can upack local .tar files.<br>CMD — provides a command and arguments for an executing container. Parameters can be overridden. There can be only one CMD.<br>WORKDIR — sets the working directory for the instructions that follow.<br>ARG — defines a variable to pass to Docker at build-time.<br>ENTRYPOINT — provides command and arguments for an executing container. Arguments persist.<br>EXPOSE — exposes a port.<br>VOLUME — creates a directory mount point to access and store persistent data.</blockquote><pre><em>##FROM command marks the start of the Dockerfile<br>###Here python is used as the base image with a </em><strong><em>tag</em></strong><em> of 3.7.3 and python is the </em><strong><em>repository, no tag assumes latest version.</em></strong><em> </em></pre><pre><strong>FROM</strong> python:3.7.3</pre><pre><em>##LABEL provides information about the Dockerfile something like READEME in github. It is usually used to put maintainer info <br>LABEL</em></pre><pre><strong>LABEL</strong> maintainer=&quot;ankit256@gmail.com&quot;</pre><pre><em>##ENV is nice for setting constants. If you use a constant several places in your Dockerfile and want to change its value at a later time, you can do so in one location.</em></pre><pre><strong>ENV ADMIN=&quot;ankit&quot;</strong></pre><pre><em>##RUN creates a layer at build-time. Docker commits the state of the image after each RUN. RUN is often used to install packages into an image. example: apt upgrade tells Docker to update the packages from the base image.</em></pre><pre><em>The following command is used to create a folder named &quot;</em><strong>docker_practice_directory&quot;</strong></pre><pre><strong>RUN [&quot;mkdir&quot;, &quot;/docker_practice_directory&quot;]</strong></pre><pre><em>### So the following command will setup our working directory in the container for the COPY, RUN, CMD, ENTRYPOINT instructions. <br>We can also use multiple WORKDIR instructions. If relative paths are provided, then each WORKDIR instruction changes the current working directory.</em></pre><pre><strong>WORKDIR /docker_practice_directory<br></strong></pre><pre><em>## COPY command copies all the contents from the current folder in your local where all your files would be like your code and requirements.txt to the &quot;/docker_practice_directory&quot; we just created<br></em>Copy will create the target directory if it doesn’t exist. ADD in place of copy does the same thing except it can copy files from remote URL&#39;s. </pre><pre><em># Copy code from your local context to the image working directory<br>COPY . . if you have already set the working directory</em></pre><pre><strong>COPY ./ /docker_practice_directory</strong></pre><pre><em>so if run<br></em><strong>RUN pip install -r requirements.txt</strong><em> now it will execute this command and reference the file we just copied into the image folder &quot;/docker_practice_directory&quot; from the current working directory on our local. Also &quot;-r&quot; flag make sure&#39;s it executes the most updated </em>requirements.txt instead of using a stale one in it&#39;s cache</pre><pre><em>###Only one CMD per Dockerfile, It must contain an executable file like &quot;.py&quot; for python if not an ENTRYPOINT instruction must exist.</em></pre><pre><em>ENTRYPOINT provides the defaault arguments and commands when the container starts. Difference between ENTRYPOINT and CMD is the earlier is not overwritten instead appended where as CMD can be overwritten. More to clarify later. <br></em></pre><pre><strong>CMD [&quot;python&quot;, &quot;./dockerp.py&quot;]<br>or<br>ENTRYPOINT [&quot;python&quot;,&quot;./dockerp.py&quot;]</strong></pre><p>Running the Docker CLI</p><pre><em>### The following command will create an &quot;image named:&quot; &quot;first_docker_image&quot; with the tag: &quot;first&quot;  based on the Dockerfile in the current working directory acessed through &quot;.&quot; , -t tags the image with the provided tag</em></pre><pre><strong>docker image build -t first_docker_image:first .<br>or<br>docker build . -t dockertutorial</strong></pre><pre>Output will be something like</pre><pre>Sending build context to Docker daemon  41.47kB<br>Step 1/7 : FROM python:3.7.3<br>3.7.3: Pulling from library/python<br>6f2f362378c5: Pull complete <br>494c27a8a6b8: Pull complete <br>7596bb83081b: Pull complete <br>372744b62d49: Pull complete <br>615db220d76c: Pull complete <br>39aa0c89bda1: Pull complete <br>ac275157d894: Pull complete <br>98d16dec829a: Pull complete <br>c8514b1c6524: Pull complete<br>.....<br>.....<br>Successfully built 9230db86ea62<br>Successfully tagged first_docker_image:first</pre><pre><em>###</em><strong><em>to connect it to a repository or docker hub to push your image.</em></strong></pre><pre><strong>docker image ls<br></strong>REPOSITORY                TAG     IMAGE ID     CREATED         SIZE<br>first_docker_image<strong>       </strong> first 9230db86ea62  37minutesago    1.26GB<br>dockertutorial            latest 9230db86ea62 37 minutes ago  1.26GB</pre><pre>ankitkothari$ <strong>docker run first_docker_image</strong></pre><pre>Output of dockerp.py executing                                                                        Context<br>Tell me about your personality      Just think of me as the ace up your sleeve.<br>I want to know you better         I can help you work smarter <br>...                                                                         ...</pre><pre><em>##gives a lot of details</em></pre><pre><strong>docker image history first_docker_image about the image</strong></pre><pre><em>##gives a lot of metadata</em><strong><br>docker image inspect  first_docker_image </strong></pre><pre><em>###To delete an image</em><br>ankitkothari$ <strong>docker image rm first_docker_image -f</strong><br>Untagged: first_docker_image:latest<br>Deleted: sha256:c5530470cea8d7652ca051d1d86a1138a2134a34fc377a9e2a1b408f53c61dad</pre><p>Running specific Download commands</p><pre><strong>RUN python -m nltk.downloader punkt</strong></pre><pre><strong>docker container rm 56ee<br>docker image rm 9230db86ea62 &lt;image_id&gt;</strong></pre><pre>So <strong>EXPOSE</strong> helps in inter-container communication. What if, there’s a need to bind the port of the container with that of the host machine on which the container is running?</pre><pre>Pass the <strong>-p </strong>(lower case p) as a option to the <a href="https://docs.docker.com/engine/reference/run/"><strong>docker run</strong></a> instruction as follows</pre><pre><strong>docker run -p &lt;HOST_PORT&gt;:&lt;CONTAINER:PORT&gt; IMAGE_NAME<br>docker run -p 5000:4000 chatbot &lt;image name&gt;</strong></pre><pre><strong>docker run -p 5000:4000 --name slackbot chatbot:v1<br>here name will be used to name the image chatbot:v1 as slackbot </strong></pre><pre><strong>docker exec -it slackbot /bin/sh is used to get inside the running container and it&#39;s folders <br>COMMAND + D to get out of the container</strong> </pre><pre><strong>docker logs slackbot shows the logs of the container </strong></pre><pre><strong>docker image rm $(docker images -a -q)</strong> — Delete all images. Careful with this one! Note that images that have been pushed to a remote registry will be preserved — that’s one of the benefits of registries. 😃</pre><h3>Containers</h3><p>Use docker container my_command</p><p>create — Create a container from an image.<br>start — Start an existing container.<br>run — Create a new container and start it.<br>ls — List running<em> </em>containers.<br>inspect — See lots of info about a container.<br>logs — Print logs.<br>stop — Gracefully stop running container.<br>kill —Stop main process in container abruptly.<br>rm— Delete a stopped container.</p><h3>Images</h3><p>Use docker image my_command</p><p>build — Build an image.<br>push — Push an image to a remote registry.<br>ls — List images.<br>history — See intermediate image info.<br>inspect — See lots of info about an image, including the layers.<br>rm — Delete an image.</p><h3>Misc</h3><p>docker version — List info about your Docker Client and Server versions.<br>docker login — Log in to a Docker registry.<br>docker system prune — Delete all unused containers, unused networks, and dangling images.</p><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker container run -it botty</strong><br>[nltk_data] Downloading package stopwords to /root/nltk_data...<br>[nltk_data]   Unzipping corpora/stopwords.zip.<br>[nltk_data] Downloading package punkt to /root/nltk_data...<br>[nltk_data]   Unzi</pre><pre><strong>docker container create --name slacbot botty:latest</strong><br>543365ce6ddab47d8492ca7170e00e7362c714b3693c67cbc2947510d9bf38c2</pre><pre><strong>docker container run -d my_image</strong></pre><pre>-d is short for --detach. Run the container in the background. Allows you to use the terminal for other commands while your container runs.</pre><pre><strong>docker container ls</strong> — List running<em> </em>containers. Also provides useful information about the containers.</pre><pre><strong>docker container ls -a -s</strong></pre><pre>-a is short for -all. List all containers (not just running ones).</pre><pre>-s is short for --size. List the size for each container.</pre><pre><strong>docker container inspect my_container</strong> — See lots of info about a container.</pre><pre><strong>docker container logs my_container</strong> — Print a container’s logs.</pre><pre><strong>docker container stop my_container</strong></pre><pre><strong>docker container rm my_container</strong> — Delete one or more containers.</pre><pre><strong>docker container rm $(docker ps -a -q)</strong> — Delete all containers that are not running.</pre><pre><strong>docker container logs &lt;container id&gt;</strong></pre><p><strong>Dockerfile</strong></p><pre>FROM python:3.7.3 <strong>### Building th base image from python </strong></pre><pre>LABEL maintainer=&quot;ankit256@gmail.com&quot; <strong>##code maintainer</strong></pre><pre>RUN pip install virtualenv </pre><pre>RUN mkdir docker <strong>##creating a folder named docker</strong></pre><pre>RUN mkdir nltk_data <strong>##creating the folder nltk_data (will download all the nltk libraries into this folder)</strong></pre><pre>RUN chown -R 1000:500 /docker <strong>#setting permissions to access these</strong> folders </pre><pre>RUN chown -R 1000:500 <strong>/nltk_data #setting permissions to access these folders</strong></pre><pre>WORKDIR <strong>/docker ##making this as a working directory</strong></pre><pre>COPY ./ /docker <strong>##copying all the files from the host folder to this folder &quot;docker&quot; </strong></pre><pre>RUN pip install -r requirements.txt <strong>#downloading all the python dependencies</strong></pre><pre>RUN mkdir /docker/metrics/ <strong>##creating this folder to maintain data written in the script through volums and it has to be declared before the volume command</strong></pre><pre>RUN touch /docker/metrics/metrics.json <strong>##creating the json file in which the script will write data</strong></pre><pre>RUN echo {} &gt; /docker/metrics/metrics.json <strong>##declaring a dictionary in the file</strong></pre><pre>VOLUME [&quot;/docker/metrics/&quot;] <strong>##declaring the volume folder</strong></pre><pre>CMD [&quot;python&quot;, &quot;./ops_bot.py&quot;] <strong>##running the script through CMD command</strong></pre><p><strong>Build an image</strong></p><h3>1. Cd into the current working directory where all the contents and the docker file is</h3><pre><strong>docker build . -t slackbotty:v12</strong></pre><h3>2. Running the docker container</h3><pre><strong>docker container run -it slackbotty:v12</strong></pre><h3>3. Look at the running containers</h3><pre>ankitkothari:~ $ <strong>docker container ls</strong></pre><pre>CONTAINER ID        IMAGE               COMMAND                 CREATED             STATUS              PORTS               NAMES</pre><pre>7b161f15485a        slackbotty:v12      &quot;python ./ops_bot.py&quot;   52 seconds ago      Up 51 seconds                           xenodochial_hofstadter</pre><pre>6062bf0b00ea        slackbotty:v11      &quot;python ./ops_bot.py&quot;   23 hours ago        Up 18 minutes                           sleepy_dhawan</pre><pre>ankitkothari:~ $</pre><h3>4. get inside the folders of the running container</h3><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker exec -it 7b161f15485a  /bin/bash</strong><br>root@7b161f15485a:/docker# ls -ltra<br>total 104<br>-rw-r--r-- 1 root root    73 Feb  7 08:29 requirements.txt<br>drwxr-xr-x 2 root root  4096 Feb  7 08:29 nltk_data<br>-rw-r--r-- 1 root root  6212 Feb  7 08:29 ebill.csv<br>-rw-r--r-- 1 root root   101 Feb  7 08:29 dockerp.py<br>-rw-r--r-- 1 root root    13 Feb  7 08:29 README.md<br>-rw-r--r-- 1 root root     5 Feb  7 08:29 .gitignore<br>-rw-r--r-- 1 root root    12 Feb  7 08:29 .dockerignore<br>-rw-r--r-- 1 root root   616 Feb  7 08:30 Jenkinsfile<br>-rw-r--r-- 1 root root    71 Feb  8 06:49 secrets.env<br>-rw-r--r-- 1 root root    91 Feb  8 07:41 docker-compose.yml<br>-rw-r--r-- 1 root root   427 Feb  9 05:09 Dockerfile<br>-rw-r--r-- 1 root root 36465 Feb  9 05:23 dialog_talk_agent.csv<br>-rw-r--r-- 1 root root  5578 Feb  9 05:26 ops_bot.py<br>drwxr-xr-x 1 1000  500  4096 Feb  9 05:27 .<br>drwxr-xr-x 1 root root  4096 Feb 10 05:05 ..<br>drwxr-xr-x 2 root root  4096 Feb 10 05:05 metrics<br>root@7b161f15485a:/docker# cd metric<br>bash: cd: metric: No such file or directory<br>root@7b161f15485a:/docker# cd metrics<br>root@7b161f15485a:/docker/metrics# ls -ltra<br>total 12<br>drwxr-xr-x 1 1000  500 4096 Feb  9 05:27 ..<br>-rw-r--r-- 1 root root    3 Feb  9 05:27 metrics.json<br>drwxr-xr-x 2 root root 4096 Feb 10 05:05 .<br>root@7b161f15485a:/docker/metrics#</pre><h3>5. To stop the running container</h3><pre>ankitkothari:~ $ <strong>docker container ls</strong></pre><pre>CONTAINER ID        IMAGE               COMMAND                 CREATED             STATUS              PORTS               NAMES</pre><pre>6062bf0b00ea        slackbotty:v11      &quot;python ./ops_bot.py&quot;   23 hours ago        Up 23 minutes                           sleepy_dhawan</pre><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker container stop 6062bf0b00ea</strong><br>6062bf0b00ea</pre><pre>ankitkothari:~ $ <strong>docker container ls</strong></pre><pre>CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES</pre><h3>6. To start a stopped container</h3><pre><strong>docker start -i 6062bf0b00ea</strong></pre><pre>CONTAINER ID        IMAGE               COMMAND                 CREATED             STATUS              PORTS               NAMES</pre><pre>6062bf0b00ea        slackbotty:v11      &quot;python ./ops_bot.py&quot;   23 hours ago        Up 46 seconds                           sleepy_dhawan</pre><h3>7. Manage Volumes</h3><p>A Docker volume is a directory (or collection of files) that lives on the host file system and is not a part of the container’s UFS. It is within these volumes that containers are capable of saving data. With the <em>docker volume</em> command, you can easily manage volumes to expand your containers well beyond their basic capability.</p><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker volume inspect </strong>0d92cd39a58895ff736e534c6a27ae82540a9352a66d12d37645c8e43b7dcd05<br>[<br>    {<br>        &quot;CreatedAt&quot;: &quot;2020-02-09T05:13:43Z&quot;,<br>        &quot;Driver&quot;: &quot;local&quot;,<br>        &quot;Labels&quot;: null,<br>        &quot;Mountpoint&quot;: &quot;/var/lib/docker/volumes/0d92cd39a58895ff736e534c6a27ae82540a9352a66d12d37645c8e43b7dcd05/_data&quot;,<br>        &quot;Name&quot;: &quot;0d92cd39a58895ff736e534c6a27ae82540a9352a66d12d37645c8e43b7dcd05&quot;,<br>        &quot;Options&quot;: null,<br>        &quot;Scope&quot;: &quot;local&quot;<br>    }<br>]<br>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $</pre><h3>8. docker version</h3><pre>support $ <strong>docker --version</strong><br>Docker version 19.03.5, build 633a0ea<br>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $</pre><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker-compose --version</strong><br>docker-compose version 1.24.1, </pre><h3>9. .evn file</h3><pre>S_TOKEN=&lt;your token for the enviorment variable&gt;</pre><h3>10. docker-compose.yml</h3><pre>version: &#39;3&#39;<br>services:<br>bot:<br>  image: slackbotty:v21<br>  environment:<br>      SLACK_BOT_TOKEN: &quot;$S_TOKEN&quot;</pre><h3>11. docker-compose commands</h3><p><strong>Build services: </strong><strong>docker-compose build [SERVICE...]<br>Generate and run containers: </strong><strong>docker-compose up [SERVICE...]<br>View running containers: </strong><strong>docker-compose ps [SERVICE...]</strong></p><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ <strong>docker-compose config</strong><br>services:<br>  bot:<br>    environment:<br>      SLACK_BOT_TOKEN: &lt;token from the .env file&gt;<br>    image: slackbotty:v21<br>version: &#39;3.0&#39;</pre><pre>ankitkothari:~/Documents/chatbot/ops-bot/ops-support $ d<strong>ocker-compose up bot</strong><br>Recreating ops-support_bot_1 ... done<br>Attaching to ops-support_bot_1</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5f695061ee8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git Crash Course that will make your life easy!!]]></title>
            <link>https://medium.com/@ankitkothari_92911/git-crash-course-that-will-make-your-life-easy-97273565c95e?source=rss-b9fe6cfaaee3------2</link>
            <guid isPermaLink="false">https://medium.com/p/97273565c95e</guid>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Ankit Kothari]]></dc:creator>
            <pubDate>Mon, 23 Mar 2020 13:43:25 GMT</pubDate>
            <atom:updated>2022-05-24T01:18:41.774Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HmtkzxNYchDkG3n9IIbCZg.png" /></figure><p>Git is a potent tool for collaboration across teams, But being a Programmer Analyst, I have always been working on my projects and never really used Git as much, but and I kept forgetting the commands and the process. And then came the real-time when I had to use it a production environment across teams, and I saw myself struggling. So the best way I thought was to use it for my day to day projects to be in touch, and it kind of sinks into me.</p><p>The significant benefits I saw was:</p><ol><li>Keeps my work tidy and clean, saves much time</li><li>And also, when you are working for a big organization and work across teams, it’s a convenient tool to know.</li></ol><p>The scope of this article is as follows:</p><ol><li>Create a repository, adding files, through the UI</li><li>Accessing it through the terminal (Mac)</li><li>Creating multiple branches and making changes, deleting branches</li><li>push and pull between remote and local</li><li>squashing and rebasing it</li><li>In the end, merging your pull request to master.</li></ol><p>Three main terminologies that I want to stress:</p><ol><li>modified: The file has been changed but it is still yet not ready to be merged.</li><li>staged: The file is ready (after all the changes made) and ready to be committed.</li><li>committed: The current version of the file has been added to the commit and git has stored and recorded those changes.</li></ol><h3><strong>Step 1: Registering your user name and email address</strong></h3><p>Let&#39;s start by setting up your user name and email address. This will be used in every commit you make behind the scenes from that particular system. And use this command without — global parameter if you want to use another username and id for a specific project.</p><pre>git config — global user.name “&lt;user-name&gt;”<br>git config — global user.email “&lt;email-address&gt;”</pre><h3><strong>Step 2: Creating the GitHub repo:</strong></h3><p>This can be achieved in two ways:</p><p><strong>a) Using the GitHub UI </strong><a href="https://github.com/ankit-kothari?tab=repositories"><strong>https://github.com/<em>&lt;user-name&gt;</em>?tab=repositories</strong></a><strong> and replacing <em>&lt;user-name&gt; </em>with the actual user-name.</strong></p><p>i. Click on the create button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CoIFDUH9SwB7b5q406qRSQ.png" /><figcaption>Creating a new repository through the UI</figcaption></figure><p>ii) Created three files in my repository “github_demo” through the UI and also added the context of this repository on the “README” file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rMVZrtlRUipW7Aw_awMUhw.png" /></figure><p><strong>b) Using the command line to create a repository on your local machine.</strong></p><p>i) Navigate to the folder where you would like to create a repository. It will be the place that will sync to your remote UI and a place to add all your files and folders for that particular project and repo.</p><p>For example, I have created the folder “git_demo” for this project and I have navigated to that folder and it can be seen it’s an empty folder now.</p><pre>Ankits-MBP-2:git_demo ankitkothari$ mkdir gitdemo<br>Ankits-MBP-2:git_demo ankitkothari$ cd gitdemo<br>Ankits-MBP-2:gitdemo ankitkothari$ ls -a<br>. ..</pre><p>ii) Create a repository in this folder using the following command and you will see that now you have a “.git” file in your folder which confirms that a repository has been created locally.</p><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git init</strong></pre><pre>Initialized empty Git repository in /Users/ankitkothari/git_demo/gitdemo/.git/<br>Ankits-MBP-2:gitdemo ankitkothari$ ls -a</pre><pre>. .. .git</pre><p>iii) Adding files to the repo “gitdemo”. We can use nano command, it will create a new file if doesn’t exist and you can write and save the content. I am creating a README.md file, adding a file from another folder to Github folder “gitdemo”. Now we have three files in our repository.</p><pre>Ankits-MBP-2:git_demo ankitkothari$ nano README.md<br>Ankits-MBP-2:git_demo ankitkothari$ nano nlp<br>Ankits-MBP-2:git_demo ankitkothari$ cp /Users/ankitkothari/Desktop/addingfiles.txt.rtf /Users/ankitkothari/git_demo</pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ cp /Users/ankitkothari/Desktop/addingfiles.txt.rtf /Users/ankitkothari/gitdemo</pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ ls -a</pre><pre>.   ..   .README.md  addingfiles.txt.rtf nlp.txt</pre><p>Unlike in the UI, where we can merge it directly to the master or the branch we are working in, we will need to use a few commands here to get that done.</p><p><strong><em>git status:</em></strong> all the files which have been modified since the last commit (modifying)</p><p><strong><em>git add:</em></strong> adding all the files which are ready to be committed (staging part)</p><p><strong><em>git commit:</em></strong> committing the final changes to the branch you are working on (committed)</p><pre>Ankits-MBP-2:gitdemo ankitkothari$ git status<br>On branch master<br>No commits yet</pre><pre>Untracked files:</pre><pre>(use &quot;git add &lt;file&gt;...&quot; to include in what will be committed)</pre><pre>.README.md<br>addingfiles.txt.rtf<br>nlp.txt</pre><pre>nothing added to commit but untracked files present (use &quot;git add&quot; to track)</pre><pre><strong>### . means all the contents of the current folder you are, in my case gitdemo</strong></pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git add .</strong></pre><pre><strong>### commiting my files or adding files in this case to the master branch(locally on my system)</strong></pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git commit -m</strong> &quot;adding my first set of files into gitdemo&quot;</pre><pre>[master (root-commit) 8603f1a] adding my first set of files into gitdemo</pre><pre>3 files changed, 22 insertions(+)<br>create mode 100644 .README.md<br>create mode 100644 addingfiles.txt.rtf<br>create mode 100644 nlp.txt</pre><pre><strong>##Now we can check the status again, no modified files after the commit</strong></pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git status</strong></pre><pre>On branch master<br>nothing to commit, working tree clean</pre><pre>Ankits-MBP-2:gitdemo ankitkothari$</pre><h3><strong>Step 3: Making changes to the file</strong></h3><p><strong>a) Through the UI</strong></p><p>i) Making changes to the file ml_to_know by using the edit button in the UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9FX0mS3Hu6vbNRdlbLsoNw.png" /></figure><p>ii) The second step is committing these changes to whatever branch you are working on, currently, we are working directly on master but it will change soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k97xZyzuX-vQm2-a7gTORw.png" /></figure><p>iii) The next steps are to track these changes and shows what was added and subtracted in the file, in our case it was only added</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y0huS10NjXim_z2jJf5azA.png" /></figure><p><strong>b)Using the command line, the following are steps taken to make changes, followed by the usual steps, we need to (git add, and git commit)</strong></p><pre>Ankits-MBP-2:gitdemo ankitkothari$ nano nlp.txt<br>Ankits-MBP-2:gitdemo ankitkothari$ git status</pre><pre>On branch master<br>Changes not staged for commit:<br>(use &quot;git add &lt;file&gt;...&quot; to update what will be committed)<br>(use &quot;git checkout -- &lt;file&gt;...&quot; to discard changes in working directory)</pre><pre><strong>modified:   nlp.txt</strong></pre><pre><strong>###to track the differences or changes made to the file in the file</strong> </pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git diff nlp.txt</strong></pre><pre><strong>diff --git a/nlp.txt b/nlp.txt<br>index b89cd9d..8ddb741 100644<br>--- a/nlp.txt<br>+++ b/nlp.txt<br></strong>@@ -9,4 +9,9 @@ Keras<br>Tensorflow<br>Pytorch</pre><pre><strong><em>+adding to the list<br>+caffe<br>+BERT<br>+ELAMO<br>+</em></strong></pre><pre><strong>###commiting the changes to the master branch locally</strong></pre><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git add nlp.txt</strong><br><strong>### To see the difference the staged file will make</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ git diff --staged<br>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git commit -m</strong> &quot;making changes to nlp.txt&quot;</pre><pre>[master 1337e54] making changes to nlp.txt<br>1 file changed, 5 insertions(+)</pre><p>Now there can be numerous commits and changes by many team members to the same repo. So to keep track of each of the commits</p><ol><li>Click on History button for that file in the UI, It will give information for all the previous commits.</li><li>On the command line, we can use the following command, It will give us the commit id, when it was made, which file was changed.</li></ol><pre>Ankits-MBP-2:gitdemo ankitkothari$ <strong>git log ##git log --stat for more details</strong></pre><pre>commit 1337e5478b3a1ddaba8be186611cf6b9ed0c795e (<strong>HEAD -&gt; master</strong>)<br>Author: ankit-kothari <br>Date:   Thu Jan 23 16:54:39 2020 -0500<br>making changes to nlp.txt</pre><pre>commit 8603f1adddb4634ba008b99b150cdef83186b891<br>Author: ankit-kothari <br>Date:   Thu Jan 23 16:32:39 2020 -0500<br>adding my first set of files into gitdemo</pre><h3><strong>STEP 4: Cloning a branch from Github</strong></h3><pre><strong>### Making a folder locally on your machine where we want to copy the contents of a repo</strong></pre><pre>Ankits-MBP-2:~ ankitkothari$ mkdir clonedemo<br>Ankits-MBP-2:~ ankitkothari$ cd clonedemo</pre><pre><strong>### cloning the repo (esentially means copying the contents of a remote repo to your local machine)</strong></pre><pre>Ankits-MBP-2:clonedemo ankitkothari$ <strong>git clone</strong> https://github.com/ankit-kothari/github_demo.git</pre><pre>Cloning into &#39;github_demo&#39;...<br>remote: Enumerating objects: 18, done.<br>remote: Counting objects: 100% (18/18), done.<br>remote: Compressing objects: 100% (16/16), done.<br>remote: Total 18 (delta 5), reused 0 (delta 0), pack-reused 0<br>Unpacking objects: 100% (18/18), done.</pre><pre>Ankits-MBP-2:github_demo ankitkothari$ ls -a<br>.    .git    git_to_know.txt   other_things_to_know.txt<br>..    README.md   ml_to_know</pre><p>So after I cloned the repo, I made changes to the “git_to_know.txt” file locally and added a few more commands to learn for git, and you have to add and commit it like we always do to enforce the changes on the branch you are working on.</p><pre>Ankits-MBP-2:github_demo ankitkothari$ git status<br>On branch master<br>Your branch is up to date with &#39;origin/master&#39;.</pre><pre>Changes to be committed:<br>(use &quot;git reset HEAD &lt;file&gt;...&quot; to unstage)<br>modified:   git_to_know.txt</pre><p>Once you have “committed” the file, you will see the following difference this time, which means the changes you made to the “git_to_know.txt” file is only on your local copy of that repo, It is yet not known to the remote (origin) that we have made these changes.</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git status</strong></pre><pre>On branch master<br>Your branch is ahead of &#39;origin/master&#39; by 1 commit.<br>(use &quot;git push&quot; to publish your local commits)</pre><p>To make these changes on the remote and make sync these changes we need to use the following commands.</p><p>First, we need to identify what is the remote repo associated with our local git repo</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git remote -v</strong><br>origin https://github.com/ankit-kothari/github_demo.git (fetch)<br>origin https://github.com/ankit-kothari/github_demo.git (push)</pre><p>and now pushing it to the origin</p><pre><strong>###here orign refers to remote(repo) and master refers to the local copy on your computer</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin master<br></strong>Enumerating objects: 5, done.<br>Counting objects: 100% (5/5), done.<br>Delta compression using up to 12 threads<br>Compressing objects: 100% (3/3), done.<br>Writing objects: 100% (3/3), 407 bytes | 407.00 KiB/s, done.<br>Total 3 (delta 1), reused 0 (delta 0)<br>remote: Resolving deltas: 100% (1/1), completed with 1 local object.<br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>8bd992c..7fe7cae  master -&gt; master</pre><p>so few important commands which might come handy during troubleshooting, I will introduce the term “HEAD” now in this article. So when we make commits after commits, the repo keeps updating itself incorporating the changes of the commit. In the end, it points to the latest change unless(we want to point it some other previous commit)</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git log</strong></pre><pre><strong>## so currently our repo is upto-date, and has all the changes made uptil commit id= 7fe7caed34ee4ea6172fe7d37cc40a91eb0ecd1e</strong></pre><pre><strong>commit 7fe7caed34ee4ea6172fe7d37cc40a91eb0ecd1e</strong> (<strong>HEAD -&gt; master</strong>, <strong>origin/master</strong>, <strong>origin/HEAD</strong>)</pre><pre>Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Thu Jan 23 17:37:22 2020 -0500<br>adding more git commands to learn</pre><pre>commit 8bd992cb7ce66a827a3ff05665ed0535f3b2eae0<br>Author: ANKIT KOTHARI &lt;37350298+ankit-kothari@users.noreply.github.com&gt;<br>Date:   Thu Jan 23 16:57:33 2020 -0500<br>adding list of new models</pre><pre>###<strong>just want to see where the HEAD is, will give the same id as above</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git rev-parse HEAD</strong></pre><pre><strong>7fe7caed34ee4ea6172fe7d37cc40a91eb0ecd1e</strong></pre><p>and to see what were the changes made in that commit id we can use</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git show 7fe7caed34ee4ea6172fe7d37cc40a91eb0ecd1e</strong></pre><pre>commit 7fe7caed34ee4ea6172fe7d37cc40a91eb0ecd1e (<strong>HEAD -&gt; master</strong>, <strong>origin/master</strong>, <strong>origin/HEAD</strong>)<br>Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Thu Jan 23 17:37:22 2020 -0500<br>adding more git commands to learn<br><strong>diff --git a/git_to_know.txt b/git_to_know.txt<br>index c150349..01b702e 100644<br>--- a/git_to_know.txt<br>+++ b/git_to_know.txt<br></strong>@@ -1,3 +1,7 @@<br>This file is about gGithub<br>The things we need to know<br>Push, Pull, Remote, Squashing, Rebase, Merge, Commit, Stage, Clone<br>+<br>+adding new commands to learn<br>+<br>+fetch, remote, rebase, squash, origin, master</pre><p>Now I want to investigate the changes that happened between two different commits at a different point in time</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git rev-parse HEAD~1</strong></pre><pre>8bd992cb7ce66a827a3ff05665ed0535f3b2eae0</pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git rev-parse HEAD~2</strong></pre><pre>e0c32275ec3042381f1d6c7d05aa770233227567</pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git --no-pager diff 8bd992cb7ce66a827a3ff05665ed0535f3b2eae0  e0c32275ec3042381f1d6c7d05aa770233227567</strong></pre><pre><strong>diff --git a/ml_to_know b/ml_to_know<br>index 5b92769..5fecc0a 100644 --- a/ml_to_know +++ b/ml_to_know</strong>@@ -3,9 +3,3 @@ SVM,</pre><pre>KNN,<br>Random Forest,<br>Logistic Regression</pre><pre><strong>###the following is the difference between the two commits and that is what was added when we added more models to the file adding list of new models</strong></pre><pre><strong>-##adding new models<br>-PCA<br>-Decision Trees<br>-Linear regression<br>-t-sne</strong></pre><p>But now realize this is a wrong list and we do not want these changes into the file and want to revert it back to the previous version</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git reset --hard e0c32275ec3042381f1d6c7d05aa770233227567</strong></pre><pre><strong>###can also be done git reset --hard HEAD~2</strong></pre><pre><strong>## if we use only the --soft flag, it will reset the history but not make any changes to the working directory.</strong></pre><pre>HEAD is now at e0c3227 Create other_things_to_know.txt<br>Ankits-MBP-2:github_demo ankitkothari$ nano ml_to_know</pre><pre><strong>## Now you will notice in the file the following models have been deleted from the list and it is reverted back to the original state</strong></pre><pre><strong>-##adding new models<br>-PCA<br>-Decision Trees<br>-Linear regression<br>-t-sne</strong></pre><p>But since we last synced with our remote branch it still has the new components and that has not been reverted back, so it shows we are behind the origin by 2 commits</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git status</strong></pre><pre>On branch master<br>Your branch is behind &#39;origin/master&#39; by 2 commits, and can be fast-forwarded.</pre><p>To sync this again we need to run the following command</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin master -f</strong><br>Total 0 (delta 0), reused 0 (delta 0)<br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>+ 7fe7cae...e0c3227 master -&gt; master (forced update)<br>Ankits-MBP-2:github_demo ankitkothari$ git status<br>On branch master<br>Your branch is up to date with &#39;origin/master&#39;.</pre><p>But there can be one other scenario, while you are working and there are other commits to the master branch, that might create conflict with the work we are doing or we need those changes to be reflected, we need to use the following command to get sync with the remote repo</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git status</strong><br>On branch master<br>Your branch is behind &#39;origin/master&#39; by 1 commit, and can be fast-forwarded.<br>(use &quot;git pull&quot; to update your local branch)</pre><p>So the changes have been incorporated in the local as well</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git pull origin master</strong><br>From <a href="https://github.com/ankit-kothari/github_demo">https://github.com/ankit-kothari/github_demo</a><br>* branch            master     -&gt; FETCH_HEAD<br>Updating 02cd57c..e0c3227<br>Fast-forward<br>other_things_to_know.txt | 5 +++++<br>1 file changed, 5 insertions(+)<br>create mode 100644 other_things_to_know.txt<br>Ankits-MBP-2:github_demo ankitkothari$ git status<br>On branch master<br>Your branch is up to date with &#39;origin/master&#39;.</pre><h3>STEP 5: Creating Branches, Merging, Deleting</h3><p>Creating a new branch and switching to that branch</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch branchdemo</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git checkout branchdemo</strong></pre><pre>Switched to branch &#39;branchdemo&#39;<br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch</strong><br>* branchdemo<br>master</pre><pre>###This can also be done in one command <strong>git checkout -b branchdemo</strong></pre><p>So I made the changes to the file “git_to_know.txt” and committed to the branch “branchdemo” as we usually do, Now we need to sync our remote as well to make sure this branch is visible there so others can collaborate there or comment/review on it before we merge it to the master,</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin branchdemo</strong><br>Enumerating objects: 5, done.</pre><pre>### To see all the branches that exist in remote <br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch -r</strong></pre><pre>origin/HEAD -&gt; origin/master<br>origin/branchdemo<br>origin/master</pre><pre>### To see all the branches that exist locally<br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch -a</strong></pre><pre>* branchdemo<br>master<br>remotes/origin/HEAD -&gt; origin/master<br>remotes/origin/branchdemo<br>remotes/origin/master</pre><p>Now I want to try out the following scenarios:</p><ol><li>Create two branches through the UI</li><li>Create a copy of those branches in my local from the Github UI and make changes in that branch and push it to the remote branches in the UI</li></ol><p>So I have created 2 new branches in the Github UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GxFqpRp8PbLYtxmT4luS3A.png" /></figure><p>But my local doesn’t know if they exist, To do that sync, we need to do the following command.</p><pre><strong>### Fetching all the branches available on the local</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git fetch --all</strong></pre><pre><strong>###ankitkothari$ git fetch --all --prune</strong></pre><pre>Fetching origin<br>remote: Enumerating objects: 12, done.<br>remote: Counting objects: 100% (12/12), done.<br>remote: Compressing objects: 100% (9/9), done.<br>remote: Total 9 (delta 2), reused 0 (delta 0), pack-reused 0<br>Unpacking objects: 100% (9/9), done.<br>From <a href="https://github.com/ankit-kothari/github_demo">https://github.com/ankit-kothari/github_demo</a><br>* [new branch]      repodemo       -&gt; origin/repodemo<br>* [new branch]      ui_branchdemo  -&gt; origin/ui_branchdemo<br>* [new branch]      ui_branchdemo1 -&gt; origin/ui_branchdemo1</pre><pre><strong>### Shows all the branches availabe remotely</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch -r</strong><br>origin/HEAD -&gt; origin/master<br>origin/branchdemo<br>origin/master<br>origin/repodemo<br>origin/ui_branchdemo<br>origin/ui_branchdemo1<br></pre><pre><strong>### Shows all the branches available on your local machine</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch -a</strong><br>* branchdemo<br>master<br>remotes/origin/HEAD -&gt; origin/master<br>remotes/origin/branchdemo<br>remotes/origin/master<br>remotes/origin/repodemo<br>remotes/origin/ui_branchdemo<br>remotes/origin/ui_branchdemo1</pre><p>Now we go into the branchdemo in the UI on Github and make a change there but our local doesn’t know we made a change, I know it’s getting a bit tricky but just go along with me, I am pretty sure you will have a solid understanding in different scenarios.</p><p>To accomplish that we use the following commands, Please note the changes will be made to the branch “branchdemo” locally and not the master locally because we are working out of the “branchdemo”. So after this, ONLY our branches will be synced.</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git pull origin branchdemo</strong><br>remote: Enumerating objects: 5, done.<br>remote: Counting objects: 100% (5/5), done.<br>remote: Compressing objects: 100% (3/3), done.<br>remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0<br>Unpacking objects: 100% (3/3), done.<br>From <a href="https://github.com/ankit-kothari/github_demo">https://github.com/ankit-kothari/github_demo</a><br>* branch            branchdemo -&gt; FETCH_HEAD<br>879769b..2e69332  branchdemo -&gt; origin/branchdemo<br>Updating 879769b..2e69332<br>Fast-forward<br>git_to_know.txt | 1 +<br>1 file changed, 1 insertion(+)</pre><p>Now we do the opposite, we will make the changes locally and sync it with the Github branch remotely. I updated the file “git_to_know.txt” locally and now for the remote to sync to it, we need to use the following command</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin branchdemo</strong><br>Enumerating objects: 5, done.<br>Counting objects: 100% (5/5), done.<br>Delta compression using up to 12 threads<br>Compressing objects: 100% (3/3), done.<br>Writing objects: 100% (3/3), 373 bytes | 373.00 KiB/s, done.<br>Total 3 (delta 2), reused 0 (delta 0)<br>remote: Resolving deltas: 100% (2/2), completed with 2 local objects.<br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>2e69332..43f97c6  branchdemo -&gt; branchdemo</pre><p>So now we have worked on these branches both locally and remotely but nothing has been updated so, Let&#39;s get the master updated both locally and remotely</p><pre><strong>###Switching back to the master folder </strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git checkout master</strong><br>Switched to branch &#39;master&#39;<br>Your branch is up to date with &#39;origin/master&#39;.</pre><pre><strong>###Difference between the master and the branch &quot;branchdemo&quot;</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git diff masterbranchdemo</strong><br><strong>diff --git a/git_to_know.txt b/git_to_know.txt<br>index c150349..7d08fca 100644<br>--- a/git_to_know.txt<br>+++ b/git_to_know.txt<br></strong>@@ -1,3 +1,9 @@<br>This file is about gGithub<br>The things we need to know<br>Push, Pull, Remote, Squashing, Rebase, Merge, Commit, Stage, Clone<br>+</pre><pre><strong>+##Working on branchdemo<br>+Task1: Changing in the branch on the Github UI and pulling it on the local copy of the branch<br>+Task2: Making the change in the local branch and sending it to remote</strong></pre><pre><strong>###Merging the change to the local master<br></strong>Ankits-MBP-2:github_demo ankitkothari$ <strong>git merge branchdemo</strong><br>Updating e0c3227..23317d1<br>Fast-forward<br>git_to_know.txt | 6 ++++++<br>1 file changed, 6 insertions(+)</pre><pre><strong>###Pushing the changes to the remote master</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin master</strong><br>Total 0 (delta 0), reused 0 (delta 0)<br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>e0c3227..23317d1  master -&gt; master</pre><p>Now since we have merged all our changes in “branchdemo” we need to delete it both locally and remotely to keep our working repo clean and not to have too many active branches</p><p>To do it locally,</p><pre><strong>### Deleted the branch locally but we still see a remote branchdemo</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git branch -d branchdemo</strong><br>Deleted branch branchdemo (was 23317d1).<br>Ankits-MBP-2:github_demo ankitkothari$ git branch -a<br>* master<br>remotes/origin/HEAD -&gt; origin/master<br>remotes/origin/branchdemo<br>remotes/origin/master<br>remotes/origin/repodemo<br>remotes/origin/ui_branchdemo<br>remotes/origin/ui_branchdemo1</pre><p>To delete it for remote repo</p><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin --delete branchdemo</strong><br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>- [deleted]         branchdemo<br>Ankits-MBP-2:github_demo ankitkothari$ git branch -r<br>origin/HEAD -&gt; origin/master<br>origin/master<br>origin/repodemo<br>origin/ui_branchdemo<br>origin/ui_branchdemo1</pre><p>Now to check the difference between commits between two branches</p><pre>Ankits-MBP-2:github_demo ankitkothari$ git rev-list --left-right --count master rebase</pre><pre>0 19</pre><pre>Ankits-MBP-2:github_demo ankitkothari$ git rev-list --left-right --count origin/master origin/rebase</pre><pre>0 18</pre><pre>Ankits-MBP-2:github_demo ankitkothari$</pre><h3><strong>STEP 6: Squashing Multiple commits into one</strong></h3><p>So we have created a branch “final” added three files in 3 commits, and when we see the logs, we see three different commits, but it doesn’t make sense to have 3 commit’s for the changes in the same branch, instead of everything can be squashed in one.</p><pre><strong>##We can see seperate commits on the same branch</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git log</strong><br>commit 5292e8ad70e891617040b4ed1dd831bf8ff78e3d (<strong>HEAD -&gt; final</strong>)Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Fri Jan 24 17:03:56 2020 -0500<br>text2.txt</pre><pre>commit c2c41551e967f7f5227f2f280f06a93327035cb9<br>Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Fri Jan 24 17:01:56 2020 -0500<br>text1.txt</pre><pre>commit 6b4eb397afd246ddb716ebd05567e6262e43e716<br>Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Fri Jan 24 17:01:40 2020 -0500<br>text.txt</pre><pre>commit 3f9f647d895759781787eae132eeef0be1dc41dc (<strong>upstream/master</strong>, <strong>master</strong>)</pre><pre><strong>##Squashing 3 commits into one commit.</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git reset --soft HEAD~3</strong><br>Ankits-MBP-2:github_demo ankitkothari$ <strong>git log<br></strong>Ankits-MBP-2:github_demo ankitkothari$ <strong>git commit -m &quot;squashing all text files&quot;</strong><br>[final 8949b1c] squashing all text files<br>3 files changed, 4 insertions(+)<br>create mode 100644 text.txt<br>create mode 100644 text1.txt<br>create mode 100644 text2.txt</pre><pre><strong>##Squashed commit</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git log</strong><br>commit 8949b1ca12b5a9727d94a3165bea360bbd11b7d5 (<strong>HEAD -&gt; final</strong>)<br>Author: ankit-kothari &lt;ankit256@gmail.com&gt;<br>Date:   Fri Jan 24 17:10:37 2020 -0500<br>squashing all text files</pre><pre><strong> ##Pushing the squashed commit to the remote &quot;final&quot; branch</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ <strong>git push origin final -f</strong><br>Enumerating objects: 6, done.<br>Counting objects: 100% (6/6), done.<br>Delta compression using up to 12 threads<br>Compressing objects: 100% (2/2), done.<br>Writing objects: 100% (5/5), 366 bytes | 366.00 KiB/s, done.<br>Total 5 (delta 1), reused 0 (delta 0)<br>remote: Resolving deltas: 100% (1/1), completed with 1 local object.<br>remote:<br>remote: Create a pull request for &#39;final&#39; on GitHub by visiting:<br>remote:      <a href="https://github.com/ankit-kothari/github_demo/pull/new/final">https://github.com/ankit-kothari/github_demo/pull/new/final</a><br>remote:<br>To <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a><br>* [new branch]      final -&gt; final</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8ko7b56dm8H0CqIaCLSSQ.png" /></figure><h3>STEP 7: REBASING YOUR BRANCH</h3><pre>Ankits-MBP-2:github_demo ankitkothari$ git remote add upstream <a href="https://github.com/ankit-kothari/github_demo.git">https://github.com/ankit-kothari/github_demo.git</a></pre><pre><strong>###git remote add upstream &lt;url of the master branch on github&gt;</strong></pre><pre><strong>### checkout on the folder/branch you want to rebase<br>and then run the following command to get the latest copy of the master and sync it to your branch</strong></pre><pre>Ankits-MBP-2:github_demo ankitkothari$ git fetch upstream</pre><pre>Ankits-MBP-2:github_demo ankitkothari$ git rebase upstream/master<br>First, rewinding head to replay your work on top of it...<br>Applying: Update final.txt<br>Ankits-MBP-2:github_demo ankitkothari$ git log</pre><pre><strong>###after rebasing your branch is ahead only by the commit you have made and is current with the most recent version of master i.e, time beween you created the branch and the commits which were made after that are all synced up to the most recent version </strong></pre><pre>commit 73c8a55801ddd4fa959e917c465925a155527047 (<strong>HEAD -&gt; rebasetry</strong>)<br>Author: ANKIT KOTHARI &lt;37350298+ankit<br>kothari@users.noreply.github.com&gt;<br>Date:   Fri Jan 24 18:00:49 2020 -0500<br>Update final.txt</pre><pre>commit 157a100f9760cc7a56a0d1a908f636d597016b68 (<strong>upstream/master</strong>, <strong>origin/master</strong>, <strong>origin/HEAD</strong>)<br>Merge: 618ce80 e95b789<br>Author: ANKIT KOTHARI &lt;37350298+ankit-kothari@users.noreply.github.com&gt;<br>Date:   Fri Jan 24 18:00:06 2020 -0500<br>Merge pull request #6 from ankit-kothari/gitrepo2<br>Update rebasefile.txt</pre><pre><strong>##pushing that changed/rebased version to remote</strong><br>Ankits-MBP-2:github_demo ankitkothari$ git push origin rebasetry -f<br>Enumerating objects: 5, done.<br>Counting objects: 100% (5/5), done.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97273565c95e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hyperparameters in Deep Learning]]></title>
            <link>https://medium.com/@ankitkothari_92911/hyperparameters-in-deep-learning-ce320858b6a1?source=rss-b9fe6cfaaee3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce320858b6a1</guid>
            <category><![CDATA[hyperparameter]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[activation-functions]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ankit Kothari]]></dc:creator>
            <pubDate>Wed, 26 Feb 2020 19:45:55 GMT</pubDate>
            <atom:updated>2020-02-26T19:53:52.129Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*qf77nxDUzYhcSUMyWj5LkA.jpeg" /></figure><h3>Activation Functions</h3><p><em>It is used to determine the output of neural network like yes or no. It maps the resulting values in between 0 to 1 or -1 to 1 etc. (depending upon the function).</em></p><p>If a function f(x) is differentiable on the interval (a,b) and belongs to one of the four considered types (i.e. it is increasing, strictly increasing, decreasing, or strictly decreasing), this function is called <strong>monotonic</strong> on this interval.[2]</p><h3>Why derivative/differentiation is used?</h3><p>When updating the curve, to know in which direction and how much to change or update the curve depending upon the slope.</p><h3>Sigmoid Function (Multi-class classification problems)</h3><p>The main reason why we use the sigmoid function is that it exists between <strong>(0 to 1). </strong>Therefore, it is especially used for models where we have to <strong>predict the probability</strong> as an output. Since the probability of anything exists only between the range of <strong>0 and 1,</strong> sigmoid is the right choice.</p><p>The <strong>softmax function</strong> is a more generalized logistic activation function that is used for multiclass classification.</p><p><strong>Difference between Sigmoid and Softmax</strong></p><p>When designing a model to perform a classification task (e.g. classifying diseases in a chest x-ray or classifying animals) we want to tell our model whether it is allowed to choose many answers (e.g. both pneumonia and abscess) or only one answer (e.g. the animal “cow.”) [1]</p><p><strong>The sum of the probabilities produced by softmax are equal to 1</strong> whereas <strong>the sum probabilities produced by sigmoid are independent and need not be equal to 1,</strong> if we are using a softmax, in order for the probability of one class to increase, the probabilities of at least one of the other classes have to decrease by an equivalent amount.</p><h3><strong>Tanh Function (Classification between two classes)</strong></h3><p>The advantage is that the negative inputs will be mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph because( it ranges from -1 to 1</p><h3><strong>ReLU (</strong>convolutional neural networks or deep learning)</h3><p>f(z) is zero when z is less than zero and f(z) is equal to z when z is above or equal to zero.</p><h3>Leaky ReLU</h3><p>It is an attempt to solve the dying ReLU problem. For values, less than 0 the function becomes y=ay where a (0.01) unlike ReLU when it becomes absolutely 0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/0*UXh3Bh2TbHdUaEXy.png" /></figure><h3><strong>Loss Functions</strong></h3><p>The loss function helps in optimizing the parameters of the neural networks. Our objective is to minimize the loss for a neural network by optimizing its parameters(weights). The loss is calculated using loss function by matching the target(actual) value and predicted value by a neural network. Then we use the gradient descent method to update the weights of the neural network such that the loss is minimized. This is how we train a neural network.</p><h3><strong>1. Binary Crossentropy</strong></h3><p>In BCE there is <strong>only one final node</strong> and that will predict one of the two classes, We will have to use the <strong>sigmoid activation</strong> to get the probability(0–1), If the probability is greater than the set threshold, for example, if the probability is greater than 0.6(can be defined depending upon sensitivity) then it&#39;s a spam else, not a spam based on how we have trained the network. During training target value fed to the network should be 1 for Spam.</p><h3>2. <strong>Categorical Crossentropy</strong></h3><p>In CLE can be used in a multiclass classification task, the number of final nodes should be equal to the number of classes in the model, for example, if we have the MNIST dataset to identify numbers from 0 to 9, we should have 10 nodes at the final layer and it should be <strong>passed through a softmax activation</strong> so we can get the total sum of probability 1, where one will have to be low to compensate for a high value if the network is classifying properly, with whichever class node has the highest probability score, the image is classified into that class.</p><p>Important Notes</p><p><strong>Target values are one hot coded</strong> (for example 1 in MNSIT will be like [0 1 0 0 0 0 0 0 0 0 0] and 3 will be like [0 0 0 1 0 0 0 0 0 0], Target vector should be of the same size as the number of classes.</p><h3>3. Sparse Categorical Crossentropy</h3><p>When we are using SCCE loss function, you do not need to one hot encode the target vector. If the target image is of a [cat, rat, dog] you simply pass [0, 1, 2]. Basically, whichever the class is you just pass the index of that class.[cat, rat, dog, rat, dog] would be [0,1,2,0,2]</p><p>sparse categorical cross-entropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical cross-entropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]).</p><p>This allows for conserving time and memory. Consider the case of 10000 classes when they are mutually exclusive — just 1 log instead of summing up 10000 for each sample, just one integer instead of 10000 floats.[5]</p><h3>DEFINING DISTANCE BETWEEN TWO TEXTS</h3><ol><li>Hamming Distance: It is the count of the difference between two vectors where the corresponding position in the 2nd vector is different. One thing to note the length of the two vectors or strings should be the same.</li></ol><pre>1   (0 0 0 1)<br>4   (0 1 0 0)<br>Hamming Distance=2</pre><p>2. Levenshtein distance: Number of operations like insert, delete, add, needed to make two strings similar.</p><p>3. Cosine Similarity: pairwise.cosine_similarity(vector1, vector2): Central to the Cosine Distance, or the more often used Cosine Similarity, is the idea of vectorizing your text: converting a text into numerical data.</p><h3>LAYERS IN DEEP LEARNING:</h3><pre><strong>T = 8 ### T is the sequnces lenght<br>D = 2 ### D is the vector length dimensionality<br>M = 3 ### hidden layer size<br>Y = 4 ## Number of output classes</strong></pre><p>To better understand the shapes and outputs of all these layers please follow this GitHub repo: <a href="https://github.com/ankit-kothari/Deep_Learning/blob/master/Deep_Learning_Layers_Shapes.ipynb">https://github.com/ankit-kothari/Deep_Learning/blob/master/Deep_Learning_Layers_Shapes.ipynb</a></p><h3><strong>Dense (Feedforward Neural Network)</strong></h3><pre>In a feedforward network, each neuron connects to every other neuron in the network, so it creates a large number of weights, which makes it computationally tough to solve.</pre><pre>Input-to-hidden weight is size (TXD) (TXM) = 512 numbers<br>Hidden-to-output weights is size (TXM) (TXK)= 768 numbers<br>Total = 1280 weights</pre><h3><strong>RNN</strong></h3><pre><strong>In RNN the sequences is passed as a time series,one by one</strong><br>input to hidden = 6 (D)X (M)<br>hidden-to hidden = 9 (M)X(M)<br>hidden-to-output = 12 (M)X(Y)<br>Total= 27 weights</pre><pre><strong>Data is always rectangualry shaped in RNN when using Keras and TF</strong><br>NXTXD (N=# number of samples. T = sequence lenght, D = embedding dimnsion from Glove or word2vec)</pre><h3><strong>LSTM</strong></h3><p>They are used often as compared to RNN’s as they are better suited for long term dependencies and vanishing gradient problems in RNN. three outputs from a GRU layer namely output, the hidden state, and the cell state.</p><pre><strong>output, h, c</strong> = LSTM(128, return_state=True, return_sequences=True) (input)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23yV2rgKxisAWr-eufviUw.png" /></figure><h3><strong>GRU</strong></h3><p>They are used often as compared to RNN’s as they are better suited for long term dependencies and vanishing gradient problems in RNN, two output from a GRU layer namely output and the hidden state.</p><pre><strong>output, h</strong> = GRU(128, return_state=True, return_sequences=True) (input)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qj9j_QWegTK6seWAXaG5xQ.png" /></figure><h3><strong>CONV-1D</strong></h3><p>The convolutional layer is the very first layer where we extract features from the images in our datasets. Due to the fact that pixels are only related to the adjacent and close pixels, convolution allows us to preserve the relationship between different parts of an image. Convolution is basically filtering the image with a smaller pixel filter to decrease the size of the image without losing the relationship between pixels. When we apply convolution to 5x5 image by using a 3x3 filter with 1x1 stride (1-pixel shift at each step). We will end up having a 3x3 output (64% decrease in complexity).</p><p>In <strong>1D CNN</strong>, kernel moves in <strong>1</strong> direction. Input and output data of 1D CNN is <strong>2</strong> dimensional. Mostly used on <strong>Time-Series</strong> data.</p><pre>model.add(Conv1D(&lt;filter_size&gt;, &lt;<strong>kernel_size&gt;, </strong>&lt;stride_length&gt;, activation=&#39;relu&#39;))</pre><ul><li><strong>filters</strong>: Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).</li><li><strong>kernel_size</strong>: An integer or tuple/list of a single integer, specifying the length of the 1D convolution window.</li><li><strong>strides</strong>: An integer or tuple/list of a single integer, specifying the stride length of the convolution. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/0*5C_OXfLdPy8BlIXu.png" /></figure><h3><strong>CONV-2D</strong></h3><p>Filtering:</p><p>Line up the filter/feature and the image patch, Multiply each image pixel by the corresponding feature pixel. Add them up, Divide the total number of pixels in the feature. That’s where the filter size (stride length and the filter window comes into play). <strong>Every filter creates a new image, so we have a stack of filtered images.</strong></p><pre>convolution = Conv2D(&lt;filter_size&gt;, &lt;<strong>kernel_size&gt;, </strong> &lt;stride_length&gt;, activation=&#39;relu&#39;, name=&#39;convolution&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HrbziWVzpDax0YVgl_Y5MQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*TVqw-MGbPaq1K90mvEK1lA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LvImdepx8Zidmrd2Jsve1g.png" /></figure><p>In <strong>2D CNN</strong>, kernel moves in <strong>2</strong> directions. The input and output data of 2D CNN is <strong>3</strong> dimensional. Mostly used on <strong>Image </strong>data.</p><h3><strong>CONV-3D</strong></h3><p>In <strong>3D CNN</strong>, kernel moves in <strong>3</strong> directions. The input and output data of 3D CNN is <strong>4</strong> dimensional. Mostly used on <strong>3D</strong> <strong>Image</strong> data (MRI, CT Scans).</p><h3>MAX-Pooling</h3><p>When constructing CNNs, it is common to insert pooling layers after each convolution layer to reduce the spatial size of the representation to reduce the parameter counts which reduces the computational complexity. In addition, pooling layers also helps with the overfitting problem. Basically we select a pooling size to reduce the amount of the parameters by selecting the maximum, average, or sum values inside these pixels. <strong>A stack of filtered images become a stack of smaller images.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdSA-JSlaJgIhOCe74ZbGg.png" /></figure><h3>Normalization:</h3><h4>Everywhere the value is negative change it to 0, just to prevent math to blow up and simplified when the images passed through “ReLU” activation. After which the goal is to make the image smaller as far as it makes sense and then flattens it out where every vote is counted and contributes towards the prediction of a class.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pe5MpyfMfG-MovWWPjVsxA.png" /></figure><h4>Embedding Layer</h4><p>It is used to create a vector relationship between the words in the corpus, There are a number of options,</p><pre><strong># embedding layer dimensions and parameters: </strong><br>DIMENSION (can be 50, 100), NUM_UNIQUE_WORDS in the corpus, <strong>Total parameters = DIMENSION*NUM_UNIQUE_WORDS</strong></pre><ol><li>Word2Vec</li></ol><pre><strong>model = Word2Vec(sentences=&lt;an array of tokenized sentences&gt;, size=64, sg=1, window=10, min_count=5, seed=42, workers=8)<br>model.save(&#39;raw_gutenberg_model.w2v&#39;)<br>model = gensim.models.Word2Vec.load(&#39;raw_gutenberg_model.w2v&#39;)</strong></pre><pre>size=64 refers to number of dimensions in vector space<br>min_count = takes into account words which are presnt at least 5 times in the corpus<br>window = refers to 10 words at a time <br>methodology can be CBOW or skip-gram<br>workers=8 is to parallize this task</pre><pre><strong>N.B. also we can download the pretrained Glove and Word2Vec embeddings and not needed to train our own. </strong></pre><p>Similarly, we can also use,</p><p>2. Glove 3. BERT</p><h3>Bi-Directional LSTM</h3><p>In this architecture we the sequence once from T(1) to T(8) and then we start from T(8) to T(1). This proves really helpful in remembering long term dependencies.</p><pre>rnn = Bidirectional(LSTM(M, return_state=True, return_sequences=True)), Here the output matrix is (TX2M)<br>The following is the output, Here you would notice <br><strong>that h1 is equal to the first 3 numbers of the output of the last cell and </strong><br><strong>h2 is equal to the last 3 numbers, output of the 1st Bidirectional(LSTM) cell </strong></pre><pre>o: [[[ 0.04978516 -0.08433709 -0.04692979  <strong>0.13628519 -0.04636889<br>    0.17559886</strong>]<br>  [ 0.12263913  0.11880868 -0.02853395 -0.07217801  0.04473321<br>   -0.03797236]<br>  [ 0.02871868 -0.06394435 -0.05262146 -0.00632159  0.32471615<br>   -0.12384429]<br>  [-0.08453789 -0.14362018 -0.05468569 -0.1310425   0.25726965<br>   -0.17572744]<br>  [-0.0215911   0.08495823  0.04642646 -0.20351063  0.11268553<br>   -0.25346553]<br>  [-0.1542023   0.04310239  0.05129007 -0.06772143  0.12287466<br>   -0.09438805]<br>  [-0.02211632  0.16745287  0.06010206 -0.06737933 -0.05280914<br>    0.00872847]<br>  [<strong>-0.01559592  0.11286419  0.02341494</strong>  0.04245665 -0.02800092<br>    0.04601097]]]<br><strong>o.shape: (1, 8, 6)</strong><br>h1: [[<strong>-0.01559592  0.11286419  0.02341494</strong>]]<br>h1: (1, 3)<br>c1: [[-0.03238782  0.20910858  0.04949831]]<br>c1: (1, 3)<br>h2: [[ <strong>0.13628519 -0.04636889  0.17559886</strong>]]<br>c2: [[ 0.20058024 -0.08406101  0.2490933 ]]</pre><h3>Multi-net Conv</h3><p>In this architecture, we concatenate the output of parallel convolutional layers who are individually passed through an activation function and then it’s passed through a Dense layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZelbYY51ksX16S5V4DY7Vg.png" /></figure><pre>input_layer = Input(shape=(max_review_length,), dtype=&#39;int16&#39;, name=&#39;input&#39;) </pre><pre>embedding_layer = Embedding(n_unique_words, n_dim, input_length=max_review_length, name=&#39;embedding&#39;)(input_layer)</pre><pre>conv_1 = Conv1D(n_conv_1, k_conv_1, activation=&#39;relu&#39;, name=&#39;conv_1&#39;)(embedding_layer)<br>maxp_1 = GlobalMaxPooling1D(name=&#39;maxp_1&#39;)(conv_1)</pre><pre>conv_2 = Conv1D(n_conv_2, k_conv_2, activation=&#39;relu&#39;, name=&#39;conv_2&#39;)(drop_embed_layer)<br>maxp_2 = GlobalMaxPooling1D(name=&#39;maxp_2&#39;)(conv_2)</pre><pre>conv_3 = Conv1D(n_conv_3, k_conv_3, activation=&#39;relu&#39;, name=&#39;conv_3&#39;)(drop_embed_layer)<br>maxp_3 = GlobalMaxPooling1D(name=&#39;maxp_3&#39;)(conv_3)</pre><pre><strong>concat = concatenate([maxp_1, maxp_2, maxp_3])</strong></pre><pre>dense_layer = Dense(n_dense, activation=&#39;relu&#39;, name=&#39;dense&#39;)(concat)</pre><h3>Stacked</h3><p>In Stacked LSTM, we can keep stacking LSTM because when we do return_sequences= True it will out vector of the size of the hidden dimension, which will be the out to the next LSTM, 3 in this case.</p><pre>def stacked():<br> input_ = Input(shape=(T, D))<br> layer1 = LSTM(M, return_sequences=True)<br> x = layer1(input_)<br> layer2 = (LSTM(M))<br> y = layer2(x)</pre><pre>model = Model(inputs=input_, outputs=y)<br> o = model.predict(X)<br> print(&quot;o:&quot;, o)<br> print(&quot;o.shape:&quot;, o.shape)</pre><pre><strong>o: [[ 0.00107461  0.0108855  -0.01437271]]<br>o.shape: (1, 3)</strong></pre><h3>Flatten</h3><p>It will convert a 2D array into a 1D array, for example, a [64, 100] will become an array of [6400]</p><pre>model.add(Embedding(n_unique_words, n_dim, input_length=max_review_length))<br>model.add(Flatten())<br>model.add(Dense(n_dense, activation=&#39;relu&#39;))<br></pre><h3>Gradient Descent, Learning Rate, Batch Size, and epochs:</h3><p>Gradient descent has a parameter called the learning rate. It is the size of steps taken to reach the bottom or minimum. As you can see above (left), initially the steps are bigger that means the learning rate is higher and as the point goes down the learning rate becomes smaller by the shorter size of steps. Also, the Cost Function is decreasing or the cost is decreasing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/0*oGbyVTKJPHJllPsk.gif" /></figure><p>We need terminologies like epochs, batch size, iterations only when the data is too big which happens all the time in machine learning and we can’t pass all the data to the computer at once</p><h3>Epochs</h3><p>One Epoch is when an ENTIRE dataset is passed forward and backward through the neural network only ONCE Since one epoch is too big to feed to the computer at once we divide it into several smaller batches.</p><h3>Batch Size</h3><p>The total number of training examples present in a single batch.</p><h3><strong>Iterations</strong></h3><p>Iterations are the number of batches needed to complete one epoch.</p><p>Let’s say we have 2000 training examples that we are going to use.</p><blockquote><strong><em>We can divide the dataset of 2000 examples into batches of 500 then it will take 4 iterations to complete 1 epoch.</em></strong></blockquote><blockquote><strong>Where Batch Size is 500 and Iterations is 4, for 1 complete epoch.</strong></blockquote><h3>2nd order derivative (Adam)</h3><p>AdaGrad</p><p>RMSprob</p><p>ADAM</p><h3>Initialization:</h3><p>How to initialize weights and bias in the beginning, 0 doesn’t work well.</p><pre>glorot uniform: The weights are from a uniform distribution</pre><pre>glorot normal: Take values from a normal distributiom </pre><h3>Reduce Overfitting:</h3><h3>L1/L2 regularization lambda</h3><p>Logistic Regression then the loss is log-loss if the model is Support Vector Machine the loss is hinge-loss. If the model is a <strong>neural network then it will be some form of cross-entropy loss.</strong></p><p>The regularization parameter (lambda) serves as a <strong>degree of importance that is given to miss-classifications intuitively</strong>, as lambda grows larger the less the wrongly classified examples are allowed (or the highest the price the pay in the loss function). Then when lambda tends to infinite the solution tends to the hard-margin (allow no miss-classification). When lambda tends to 0 (without being 0) the more the miss-classifications are allowed.</p><p>In Keras, I can directly apply regularization to any layer using the regularizers. I have applied regularizer on a dense layer having 100 neurons and “relu” activation function.</p><pre>#l2 example adding it to a dense layer<br> regularizermodel.add(Dense(500,kernel_regularizer=regularizers.l2(0.01),activation=&quot;relu&quot;))</pre><h3>Dropout</h3><p><strong>Dropout</strong> is a <a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)">regularization</a> technique patented by Google<a href="https://en.wikipedia.org/wiki/Dropout_(neural_networks)#cite_note-pat-1">[1]</a> for reducing <a href="https://en.wikipedia.org/wiki/Overfitting">overfitting</a> in <a href="https://en.wikipedia.org/wiki/Neural_networks">neural networks</a> by preventing complex co-adaptations on training data.</p><pre>drop_dense_layer = Dropout(0.2, name=&#39;drop_dense&#39;)(x)<br>where x is the output of the previous layer</pre><pre>#another form<br>model.add(SpatialDropout1D(0.2))</pre><h3>SpatialDropout1D</h3><pre>example:<br>drop_embed_layer = SpatialDropout1D(0.2, name=’drop_embed’)(embedding_layer)</pre><p>The noise shape</p><p>In order to understand SpatialDropout1D, you should get used to the notion of the <strong>noise shape</strong>. In plain vanilla dropout, each element is kept or dropped independently. For example, if the tensor is [2, 2, 2], each of 8 elements can be zeroed out depending on a random coin flip (with certain &quot;heads&quot; probability); in total, there will be 8 independent coin flips and any number of values may become zero, from 0 to 8.</p><p>Sometimes there is a need to do more than that. For example, one may need to drop the <em>whole slice</em> along 0 axis. The noise_shape in this case, is [1, 2, 2] and the dropout involves only 4 independent random coin flips. The first component will either be kept together or be dropped together. The number of zeroed elements can be 0, 2, 4, 6 or 8. It cannot be 1 or 5.</p><p>Another way to view this is to imagine that input tensor is in fact, but each value is double-precision (or multi-precision). Instead of dropping the bytes in the middle, the layer drops the full multi-byte value.</p><h3>Early Stopping</h3><p>In Keras, we can apply early stopping using the <strong>callbacks</strong> function. Below is the implementation code for it. I have applied early stopping so that it will stop immediately if validation error will not be decreased after 3 epochs.</p><pre>from keras.callbacks import EarlyStopping<br><strong>earlystop</strong>= EarlyStopping(monitor=&#39;val_acc&#39;, patience=3)epochs = 20 # batch_size = 256</pre><pre>history = model.fit_generator(datagen.flow(x_train,y_train, batch_size=batch_size),                              epochs = epochs, validation_data = (x_test,y_test),                              verbose = 1, steps_per_epoch=x_train.shape[0] // batch_size                              , callbacks=[<strong>earlystop</strong>])</pre><h3>Expand Data</h3><p>add noise to the sound, rotate the images, move the digits, bring the noise to the images.</p><h3>CONFUSION METRIX</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nRUUzIv5rnvomOG0IYP2LA.png" /></figure><p>N.B. Should not be the only metrics we should be using if we have imbalanced data</p><pre><strong>Accuracy:</strong> True Positive+ True Negative/ TP+TN+FP+FN<br>Accuracy is best to use when the dataset is balanced otherwise we should use sampling method first. </pre><pre><strong>Precision:</strong> Focused on predicted positive<br>Out of all the points predicted to be positive, How many of them are actually positive.</pre><pre>Precision = True Positive/ (TP+FP)</pre><pre><strong>Recall:</strong> Recall focus on actual positives value in our dataset<br>Out of the actually labelled positive, how many of them are correctly predicted positive</pre><pre>Recall (Sensitivity) = True Positive/ TP + FN (To determine if we are doing a good job in predicting a positive value without regard to how are we doing on actual negative values)</pre><pre>Recall (Specificity) = True Negative/ TN + FP (To determine if we are doing a good job in predicting a negative value without regard to how are we doing on actual positive values)</pre><h3><strong>AUC-ROC</strong></h3><pre><strong>from sklearn.metrics import roc_auc_score&quot;{:0.2f}&quot;.format(roc_auc_score(y_valid, y_hat)*100.0)</strong></pre><pre>ROC is a probablity curve, AUC is a measure of seperatability, It tells how much the model is capable of distinguishing between classes</pre><pre>AUC= 1, It has a good measure of seperatabliity <br>AUC= 0  It is reciprocating <br>AUC= 0.5 It means model has no class speration capacity whatsoever. </pre><pre>The graph between True Positive Rate (TPR) and False poitive rate (FPR) </pre><h3>PICKLE</h3><p>To save your machine learning model</p><h4>If you are working with Scikit-Learn Machine Learning Models</h4><h4>2.1 Save The Model</h4><p>Use Pickle to serialize and save the models</p><pre>from sklearn.linear_model import LogisticRegression<br>import pickle<br>model = LogisticRegression()<br>model.fit(xtrain, ytrain)# save the model to disk<br>pickle.dump(model, open(model_file_path, &#39;wb&#39;))</pre><h4>2.2 Load The Model</h4><p>Use Pickle to deserialize and save the models</p><pre>model = pickle.load(open(model_file_path, &#39;rb&#39;))<br> result_val = model.score(xval, yval)<br>result_test = model.score(xtest, ytest)</pre><h3>Using pickle to save TOKENIZER obejct:</h3><pre>import pickle<br><br># saving<br>with open(&#39;tokenizer.pickle&#39;, &#39;wb&#39;) as handle:<br>    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)<br><br># loading<br>with open(&#39;tokenizer.pickle&#39;, &#39;rb&#39;) as handle:<br>    tokenizer = pickle.load(handle)</pre><h4>2.3 Using Keras and Modelcheckpoint</h4><pre>from keras.callbacks import ModelCheckpoint<br>modelcheckpoint = ModelCheckpoint(filepath=output_dir+&quot;/weights.{epoch:02d}.hdf5&quot;)<br>if not os.path.exists(output_dir):<br>    os.makedirs(output_dir)</pre><pre>model.load_weights(output_dir+&quot;/weights.00.hdf5&quot;) # zero-indexed<br>y_hat = model.predict_proba(x_valid)</pre><pre><strong>Another example:</strong><br>keras.callbacks.callbacks.ModelCheckpoint(filepath, monitor=&#39;val_loss&#39;, verbose=0, save_best_only=<strong>False</strong>, save_weights_only=<strong>False</strong>, mode=&#39;auto&#39;, period=1)</pre><pre>Save the model after every epoch.</pre><pre>filepath can contain named formatting options, which will be filled with the values of epoch and keys in logs (passed in on_epoch_end).</pre><pre>For example: if filepath is weights.{epoch:02d}-{val_loss:.2f}.hdf5, then the model checkpoints will be saved with the epoch number and the validation loss in the filename.</pre><blockquote><strong>filepath</strong>: string, path to save the model file.</blockquote><blockquote><strong>monitor</strong>: quantity to monitor.</blockquote><blockquote><strong>verbose</strong>: verbosity mode, 0 or 1.</blockquote><blockquote><strong>save_best_only</strong>: if <em>save_best_only=True</em>, the latest best model according to the quantity monitored will not be overwritten.</blockquote><blockquote><strong>save_weights_only</strong>: if True, then only the model’s weights will be saved (<em>model.save_weights(filepath)</em>), else the full model is saved (<em>model.save(filepath)</em>).</blockquote><blockquote><strong>mode</strong>: one of {auto, min, max}. If <em>save_best_only=True</em>, the decision to overwrite the current save file is made based on either the maximization or the minimization of the monitored quantity. For <em>val_acc</em>, this should be <em>max</em>, for <em>val_loss</em> this should be <em>min</em>, etc. In <em>auto</em> mode, the direction is automatically inferred from the name of the monitored quantity.</blockquote><blockquote><strong>period</strong>: Interval (number of epochs) between checkpoints.</blockquote><h3>References</h3><ol><li><a href="https://towardsdatascience.com/classification-sigmoid-vs-softmax-2a3585ff740f">https://towardsdatascience.com/classification-sigmoid-vs-softmax-2a3585ff740f</a></li><li><a href="https://www.math24.net/increasing-decreasing-functions/">https://www.math24.net/increasing-decreasing-functions/</a></li><li><a href="https://towardsdatascience.com/3-text-distances-that-every-data-scientist-should-know-7fcdf850e510">https://towardsdatascience.com/3-text-distances-that-every-data-scientist-should-know-7fcdf850e510</a></li><li><a href="https://towardsdatascience.com/understanding-different-loss-functions-for-neural-networks-dd1ed0274718">https://towardsdatascience.com/understanding-different-loss-functions-for-neural-networks-dd1ed0274718</a></li><li><a href="https://datascience.stackexchange.com/questions/41921/sparse-categorical-crossentropy-vs-categorical-crossentropy-keras-accuracy">https://datascience.stackexchange.com/questions/41921/sparse-categorical-crossentropy-vs-categorical-crossentropy-keras-accuracy</a></li><li><a href="https://www.udemy.com/course/deep-learning-advanced-nlp">https://www.udemy.com/course/deep-learning-advanced-nlp</a></li><li><a href="http://debajyotidatta.github.io/nlp/deep/learning/word-embeddings/2016/11/27/Understanding-Convolutions-In-Text/">http://debajyotidatta.github.io/nlp/deep/learning/word-embeddings/2016/11/27/Understanding-Convolutions-In-Text/</a></li><li><a href="https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-dataset-54c35b77a38d">https://towardsdatascience.com/image-classification-in-10-minutes-with-mnist-dataset-54c35b77a38d</a></li><li><a href="https://www.youtube.com/watch?v=FmpDIaiMIeA">https://www.youtube.com/watch?v=FmpDIaiMIeA</a></li><li><a href="https://medium.com/analytics-vidhya/regularization-in-machine-learning-and-deep-learning-f5fa06a3e58a">https://medium.com/analytics-vidhya/regularization-in-machine-learning-and-deep-learning-f5fa06a3e58a</a></li><li><a href="https://towardsdatascience.com/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9">https://towardsdatascience.com/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9</a></li><li><a href="https://medium.com/fintechexplained/how-to-save-trained-machine-learning-models-649c3ad1c018">https://medium.com/fintechexplained/how-to-save-trained-machine-learning-models-649c3ad1c018</a></li><li><a href="https://keras.io/callbacks/">https://keras.io/callbacks/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce320858b6a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>