<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[CARBON CONSULTING - Medium]]></title>
        <description><![CDATA[On the road which is going to product from idea,    Think. Design. Deliver. - Medium]]></description>
        <link>https://medium.com/carbon-consulting?source=rss----de447e4c9973---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>CARBON CONSULTING - Medium</title>
            <link>https://medium.com/carbon-consulting?source=rss----de447e4c9973---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:16:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/carbon-consulting" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to deploy NextJS project on Github pages ?]]></title>
            <link>https://medium.com/carbon-consulting/how-to-deploy-nextjs-project-on-github-pages-32ab20bf2d2?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/32ab20bf2d2</guid>
            <category><![CDATA[github-pages]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[nextjs]]></category>
            <dc:creator><![CDATA[Cemre Acar]]></dc:creator>
            <pubDate>Mon, 16 Jan 2023 06:44:18 GMT</pubDate>
            <atom:updated>2023-01-16T06:44:18.669Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*Cb-3di6u-RUuEMiG.jpeg" /></figure><h3>How to deploy NextJS project on Github pages ?</h3><p>In this article, I actually wanted to talk about the GitHub pages that I researched and tried completely for my needs. For a long time, I had in mind to prepare a small portfolio page and open it live. I sat down and created this using next js and tailwind. I will not share the codes here, but if you visit and like it, you can access all the codes on GitHub.</p><p>Coming to our topic, GitHub pages actually enable the related content to be hosted instantly through a created GitHub repository and seen as a website. We will raise the NextJS project that we have created here with the NextJS deploy <em>.yml</em> file, which is already ready in GitHub.<br>Let’s get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*CxMPvoE3sSbaaigp.gif" /></figure><p>First we create a GitHub repo with your GitHub username. For example, my GitHub username is <em>cmracar</em><strong> </strong>and the repository I created is <em>cmracar.github.io</em>. After the repository is created, let’s move on to the <em>Actions</em> section through the tabs. Then let’s proceed with the <strong>New Workflow</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dRMxupwRYjqDgOAcLtl21g.png" /></figure><p>Here, when we type “<strong><em>next</em></strong>” in the search box, the workflow we want will appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OOJhaS0aIb_w5-ynj2xglA.png" /></figure><p>After selecting the workflow, a file structure in the form of .github/workflows was created in our repo. In fact, if you have experience in creating a <em>.yml</em> file, you can manually add your file under .github/workflows. If we go back to the code, we have a structure like the following.</p><pre># Sample workflow for building and deploying a Next.js site to GitHub Pages<br>#<br># To get started with Next.js see: https://nextjs.org/docs/getting-started<br>#<br>name: Deploy Next.js site to Pages<br><br>on:<br>  # Runs on pushes targeting the default branch<br>  push:<br>    branches: [&quot;main&quot;]<br><br>  # Allows you to run this workflow manually from the Actions tab<br>  workflow_dispatch:<br><br># Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages<br>permissions:<br>  contents: read<br>  pages: write<br>  id-token: write<br><br># Allow one concurrent deployment<br>concurrency:<br>  group: &quot;pages&quot;<br>  cancel-in-progress: true<br><br>jobs:<br>  # Build job<br>  build:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout<br>        uses: actions/checkout@v3<br>      - name: Detect package manager<br>        id: detect-package-manager<br>        run: |<br>          if [ -f &quot;${{ github.workspace }}/yarn.lock&quot; ]; then<br>            echo &quot;manager=yarn&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;command=install&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;runner=yarn&quot; &gt;&gt; $GITHUB_OUTPUT<br>            exit 0<br>          elif [ -f &quot;${{ github.workspace }}/package.json&quot; ]; then<br>            echo &quot;manager=npm&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;command=ci&quot; &gt;&gt; $GITHUB_OUTPUT<br>            echo &quot;runner=npx --no-install&quot; &gt;&gt; $GITHUB_OUTPUT<br>            exit 0<br>          else<br>            echo &quot;Unable to determine packager manager&quot;<br>            exit 1<br>          fi<br>      - name: Setup Node<br>        uses: actions/setup-node@v3<br>        with:<br>          node-version: &quot;16&quot;<br>          cache: ${{ steps.detect-package-manager.outputs.manager }}<br>      - name: Setup Pages<br>        uses: actions/configure-pages@v2<br>        with:<br>          # Automatically inject basePath in your Next.js configuration file and disable<br>          # server side image optimization (https://nextjs.org/docs/api-reference/next/image#unoptimized).<br>          #<br>          # You may remove this line if you want to manage the configuration yourself.<br>          static_site_generator: next<br>      - name: Restore cache<br>        uses: actions/cache@v3<br>        with:<br>          path: |<br>            .next/cache<br>          # Generate a new cache whenever packages or source files change.<br>          key: ${{ runner.os }}-nextjs-${{ hashFiles(&#39;**/package-lock.json&#39;, &#39;**/yarn.lock&#39;) }}-${{ hashFiles(&#39;**.[jt]s&#39;, &#39;**.[jt]sx&#39;) }}<br>          # If source files changed but packages didn&#39;t, rebuild from a prior cache.<br>          restore-keys: |<br>            ${{ runner.os }}-nextjs-${{ hashFiles(&#39;**/package-lock.json&#39;, &#39;**/yarn.lock&#39;) }}-<br>      - name: Install dependencies<br>        run: ${{ steps.detect-package-manager.outputs.manager }} ${{ steps.detect-package-manager.outputs.command }}<br>      - name: Build with Next.js<br>        run: ${{ steps.detect-package-manager.outputs.runner }} next build<br>      - name: Static HTML export with Next.js<br>        run: ${{ steps.detect-package-manager.outputs.runner }} next export<br>      - name: Upload artifact<br>        uses: actions/upload-pages-artifact@v1<br>        with:<br>          path: ./out<br><br>  # Deployment job<br>  deploy:<br>    environment:<br>      name: github-pages<br>      url: ${{ steps.deployment.outputs.page_url }}<br>    runs-on: ubuntu-latest<br>    needs: build<br>    steps:<br>      - name: Deploy to GitHub Pages<br>        id: deployment<br>        uses: actions/deploy-pages@v1</pre><p>I did not make any additional edits in the file in my own use, but you can make additional edits if you wish. As a note, I can add that I wanted the deploy operation to be triggered when I open a pull request to the main branch and lines 9 and 10 in my <em>.yml</em> file. I edited the lines accordingly. But GitHub says it can only do this for push to main branch, so I didn’t make this change either.</p><p>After creating our file, we had a new workflow. There is a different workflow called GitHub Pages that GitHub already offers, we need to remove it because we have 2 workflows running constantly and we don’t want our readme file to go live.</p><p>Let’s go to the <strong>Settings</strong> tab in the repository and select it from the <strong>Pages</strong> tab as follows. Here, GitHub Pages offered by GitHub are selected and we change it to GitHub Actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ph3QW-15JO5rs5P1vHYG-Q.png" /></figure><p>After all these operations, we can observe the flow process in the Actions tab with a new push operation. If a successful deploy operation has taken place, we will see that our website is up. Afterwards, we can visit our website from the browser with the repository name we created. For example <a href="https://cmracar.github.io/">https://cmracar.github.io</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S6OPNlY1ZEBpEbbRYRAx7Q.png" /></figure><p><strong>Sources:</strong><em> </em><a href="https://docs.github.com/en/actions">GitHub Actions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32ab20bf2d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/how-to-deploy-nextjs-project-on-github-pages-32ab20bf2d2">How to deploy NextJS project on Github pages ?</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[React Auth with High Order Component]]></title>
            <link>https://medium.com/carbon-consulting/react-auth-with-high-order-component-dffd3d5514d?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/dffd3d5514d</guid>
            <category><![CDATA[high-order-component]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Cemre Acar]]></dc:creator>
            <pubDate>Thu, 26 May 2022 05:12:23 GMT</pubDate>
            <atom:updated>2022-05-26T05:12:23.647Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lmmttUJvm2im-p74" /></figure><p>In this article, I will try to answer the questions of what is High Order Component (HOC) and how can I use it with Auth.</p><p>When I tried to do auth with React, I went through various searches and met HOC. What is this HOC ? As the name suggests, you can think of it as a high priority component. It can basically take a function and return a function. The goal is to add additional functionality to our functions.</p><p>So why should I use HOC ? Sometimes we want to define a common behavior for some components, such as the user control that I will talk about in this article. It becomes much more advantageous to create a structure that can be managed from a single place instead of querying it one by one in all components. Moreover, it has a very simple structure.</p><p>With the example below, you will see that it has a simple structure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d6e84cbe72485e6a3f0c721a51b3cad/href">https://medium.com/media/1d6e84cbe72485e6a3f0c721a51b3cad/href</a></iframe><p>As in the example, there is a function in our component and we return it. What we return in the function is entirely up to us, but in this example, I set up a context and userAuth structure. So here I will check the existence of user very simply. Now let’s take a look at how the context structure is.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7661264622b3f9bf38857f83d64ad3c/href">https://medium.com/media/b7661264622b3f9bf38857f83d64ad3c/href</a></iframe><p>We set up a basic context structure above. To explain a little bit, we will basically try to do our operations with user, loading and login. In order to see all the steps, we wait 3 seconds in the login function and do the operations that way. We would like to see a loading screen during this time. And finally, we check the loading status and take action accordingly and send our values in the provider.</p><p>After finishing our work on the context side, it remains to integrate it on the HOC side. We are making some changes on the first HOC code we wrote. We include the user state from the context structure we created. Then we check the existing user and take action accordingly. In my example, if the user is not specific, it is redirected to the Login page so that he can login. If there is a certain user, it is directed to our component that we receive as props in Auth.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24267da59f346b312b41fc1c5b0590c5/href">https://medium.com/media/24267da59f346b312b41fc1c5b0590c5/href</a></iframe><p>Finally, in cases where the user is undefined, we wrap components which we want to protect with the HOC we created.</p><p>As an example, I created a dashboard component and I don’t want it to be accessed before the login process takes place. I can protect it in the following way.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92b50a0c84c8a449a0cad734c0c61dab/href">https://medium.com/media/92b50a0c84c8a449a0cad734c0c61dab/href</a></iframe><p>That’s it ! You can access all these codes from the relevant link in my github profile. <a href="https://github.com/cmracar/react-auth-with-hoc-and-context">Here!</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*yjE5d1Qb7_CZ9iFj.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dffd3d5514d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/react-auth-with-high-order-component-dffd3d5514d">React Auth with High Order Component</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Steps to Coding Pythonic]]></title>
            <link>https://medium.com/carbon-consulting/3-steps-to-coding-pythonic-90b8f1edcd20?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/90b8f1edcd20</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[pythonic]]></category>
            <category><![CDATA[clean-code]]></category>
            <dc:creator><![CDATA[Salih Talha Akgün]]></dc:creator>
            <pubDate>Thu, 20 Jan 2022 11:33:23 GMT</pubDate>
            <atom:updated>2022-01-20T11:33:23.567Z</atom:updated>
            <content:encoded><![CDATA[<p>Pythonic comes from Idiomatic.</p><p><strong>Idiomatic </strong>means something that can’t be translated one-to-one but everyone understands what it means, like “hit the books” means “study hard”. The one-to-one translation is not enough to understand that phrase, you have to understand it with some prior knowledge.</p><p><strong>Pythonic </strong>means a code is written with idiomatic style, you can’t understand what it does with one-to-one translation (you can’t use python basics to understand it), you have to understand it with some prior knowledge. This article aims to give you some of that knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iD6FXwrLSntrv-KzIqYB2w.jpeg" /><figcaption>Image from Real Python [5]</figcaption></figure><h3>1- Built-in Functions</h3><p>Let’s start with an easy one.</p><p>To get a slice from a list, you will probably use this annotation:</p><p>array[:2]</p><p>This annotation actually calls a built-in function called <strong>slice. </strong>As it is obvious in this example, using <em>for loops</em> to get a slice of your data will result in a bad usage of resources.</p><p>So for any algorithm you developed, you should first check if you can use built-in functions [1] instead of trying to implement them by yourself.</p><h3>2- Context Managers</h3><p>There will always be some data that will be used in your code. This data can be coming from a file or database.</p><p>When you finished the job with your file or database, you may want to close the file or disconnect from the database. This is why people use <strong><em>with </em></strong>statement in python. But there is always a probability for an exception to happen. Are you going to handle it with lots of <em>try and except</em>s?</p><p>A good way to handle these situations is to use a class with __enter__ and __exit__ magic functions. These are the functions that will be called when entered or exited the runtime context related to the object of that class with the <strong><em>with </em></strong>statement.</p><p>Here is an example code from the book Clean Code in Python[2]. This code is trying to disconnect from the database to take a backup. To handle any unexpected errors, it rewrites the __exit__ magic function to connect again with the database when the backup ended or when any exception is thrown.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c521361bd518f4d00a36e16fd87293d/href">https://medium.com/media/9c521361bd518f4d00a36e16fd87293d/href</a></iframe><p>This is definitely better than writing lots of code to handle all of the exceptions that may happen. The exception arguments in __exit__ function will be None if no exception happens.</p><p>Also, there is a better version of this that uses contextmanager.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85cf171a86144e6d441a9cc4939976bf/href">https://medium.com/media/85cf171a86144e6d441a9cc4939976bf/href</a></iframe><p>This is what I call a Pythonic code. You first have to know how context managers work and second, you have to know that <strong><em>try </em></strong>represents <strong>__enter__</strong>, <strong><em>finally </em></strong>represents <strong>__exit__</strong>.</p><p>Contextlib is a python standard library having utilities for with-statement contexts. You can check all python standard libraries here: <a href="https://docs.python.org/3/library/">https://docs.python.org/3/library/</a></p><p>Also, these codes are from the book[2], and I highly recommend you to read the book. It has more examples on this topic which you may want to learn.</p><h3>3 — Data Classes</h3><p>Data classes are classes that are meant to hold data. The purpose of dataclass is to make classes easier to use as data containers. It makes your code more compact and understandable. Here is an example of dataclass from a great article[3].</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d49e56e6a18254b181c3a8f8a5697016/href">https://medium.com/media/d49e56e6a18254b181c3a8f8a5697016/href</a></iframe><p>Let’s look at it one by one.</p><ul><li>You don’t have to define an __init__ function, instead, you give the variable names and their types, it handles the rest for you.</li><li>It forces you to use type annotations, which is good for many things.</li><li>You can easily give default values (not mandatory), if you wanted to use this with default classes, you’d have to use lots of if else statements.</li><li>You can easily compare two instances if they have same values for same variables. If you wanted to use this with default classes, you’d have to overwrite __eq__ magic function.</li></ul><p>These are just the beginning, you can check more details on this topic at [3] and [4]. There are much more things that can be done with dataclass but this example is enough to start using it.</p><p>I hope you enjoyed and learned something. I highly recommend you implement these yourself and then read the references to get a deeper understanding.</p><p>If you have read this far, you may want to check other articles we have been publishing as Carbon Consulting.</p><p><a href="https://medium.com/carbon-consulting">https://medium.com/carbon-consulting</a></p><p>References:</p><p>1 —Python Built-in Functions, <a href="https://docs.python.org/3/library/functions.html">https://docs.python.org/3/library/functions.html</a></p><p>2 — Clean Code in Python, Develop Maintainable and Efficient Code, 2nd Edition, Mariano Anaya <a href="https://www.amazon.com/Clean-Code-Python-maintainable-efficient/dp/1800560214">https://www.amazon.com/Clean-Code-Python-maintainable-efficient/dp/1800560214</a></p><p>3— 9 Reasons Why You Should Start Using Python Dataclasses, <a href="https://towardsdatascience.com/9-reasons-why-you-should-start-using-python-dataclasses-98271adadc66">https://towardsdatascience.com/9-reasons-why-you-should-start-using-python-dataclasses-98271adadc66</a></p><p>4— Data Classes in Python 3.7+ (Guide), <a href="https://realpython.com/python-data-classes/">https://realpython.com/python-data-classes/</a></p><p>5 — <a href="https://realpython.com/learning-paths/writing-pythonic-code/">https://realpython.com/learning-paths/writing-pythonic-code/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90b8f1edcd20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/3-steps-to-coding-pythonic-90b8f1edcd20">3 Steps to Coding Pythonic</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[General Bash Usage — 2 (Grep-Awk)]]></title>
            <link>https://medium.com/carbon-consulting/general-bash-usage-2-grep-awk-c40155041095?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/c40155041095</guid>
            <category><![CDATA[shell-script]]></category>
            <category><![CDATA[grep]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[awk]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Cemre Acar]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 13:21:46 GMT</pubDate>
            <atom:updated>2022-01-18T13:21:46.674Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rpQtxCfIzr06g8UtFziRIw.jpeg" /></figure><h3>General Bash Usage — 2 (Grep-Awk)</h3><p>In the previous, I published an article about basic shell programming based on general bash usage. In this article, I will talk about <em>Regular Expressions</em> to support the use of bash. Regular expressions play an important role in computer science applications. For example, in apps that contain text, users may want to search for lines that match certain things. Regex provides us with a powerful method for this, and utilities and text editors such as grep, awk provide mechanisms for definitions of patterns using regular expressions. In this article, I’ll show you how to use regular expressions with <strong>grep</strong> and<strong> awk</strong>.</p><h4>What is Grep? (Global Regular Expression Print)</h4><p>If we explain the logic briefly, it allows us to select the word you want in a text or to mark it.</p><h4>Basic Operations with Grep</h4><p>Let’s have a <em>.txt</em> file and I’ll list the places where the word “<strong>linux</strong>” is mentioned in this file. For this, it will be enough to give the following command.</p><pre>grep “linux” test.txt</pre><p>With this command, the lines containing the word “<strong>linux</strong>” were printed on the screen. Well, we learned the process of choosing the word. Let’s look at the process of excluding the given word.</p><pre>grep -v “linux” test.txt</pre><p>With this command, we have excluded the lines where the word “<strong>linux</strong>” is mentioned and listed all the remaining lines. We may have a detail here, I have excluded the word “<strong>linux</strong>”, but the word may be written in the form of “<strong>Linux</strong>” or “<strong>LINUX</strong>”. We can control this as follows.</p><pre>grep -i “linux” test.txt</pre><p>In the same context, we can list files, not words. Let’s do this with the following command.</p><pre>grep -l “a” *.txt</pre><p>With this command, we have listed all <em>.txt</em> files with “<strong>a</strong>” in their name.</p><p>grep has a wider variety of uses, such as the ones above.</p><p>With the command below, we are looking for a 4-letter word that ends with s, without noticing the initial letter in the text in the <em>essay.txt</em>, and we color it with the color command.</p><pre>grep –color ‘[a-z]..s’ test.txt</pre><p>We can use the same coloring process when searching for a file in our directory. In the example below, let’s also list according to the first letter and the last letter.</p><pre>ls | grep –color ‘^[Aa]’</pre><p>by initial letter or</p><pre>ls | grep –color ‘[Aa]$’</pre><p>We have listed them according to the last letter with the command. Here, we can also use [0–9] instead of [Aa] to list files containing numerical expressions.</p><p>We can also use various classification expressions expressed as POSIX character classes. Let’s take an example to illustrate this. If we will use the ls command again, let’s list our files that contain only lowercase letters as follows.</p><pre>ls | grep — color ‘[[:lower:]]’</pre><p>So far we’ve talked about simple regular expressions. Let’s talk a little bit about extended grep (egrep). Simple regular expressions as we know ^ $ . It consisted of [ ] and * characters. In addition to the extended grep ( ) { } ? + | contains characters.</p><p>Let’s take an example of using extended grep. Let’s have a <em>student.txt</em> file and include student numbers and surnames.</p><pre><em>Text File Content:</em></pre><pre>20170602038 Cemre<br>20170602042 Canan<br>20160342467 Ali<br>20142425001 Ayşe</pre><p>In such a file, we can color the numbers with the following command.</p><pre>egrep — color ‘^0?[0–9]+’ students.txt</pre><p>With a different command in the same file, we can check and list only the first 8 numbers among the numbers.<br>There is an additional expression we use here, which is the { } metacharacters. This expression, called quantifiers, allows us to specify max or min numbers. ‘^0?[0–9]{8}’</p><pre>egrep — color ‘^0?[0–9]{8}’ students.txt</pre><h4>What is awk?</h4><p>Awk is similar to grep and sed. Besides Awk, there are two advanced versions named NAWK and GAWK. It takes its name from the initials of its developers, Aho, Weinberger, and Kernighan.</p><p>Awk is expressed as a ‘…’ file.</p><h4>Basic Operations with Awk</h4><p>We can see its various uses in detail through examples.</p><p><strong>E.g:</strong><br>We can select certain parts of our files that we want to list with the ls command in any of our directories.</p><pre>ls -l | awk ‘{print “File Name: “$9” File Permissions: “$1}’</pre><p>Or suppose we have a name file. My names are in a text file called <em>file.txt</em> and they are all under one another.</p><pre>awk ‘{print $1}’ file.txt</pre><p>I am able to access my names using this command.</p><p>We can also use the -F argument as follows.</p><pre>awk -F”\t+” ‘{print $2}’ file.txt</pre><p>To mention the NF, NR and length arguments that we can use in commands;<br><strong>NF :</strong> The number of spaces in the line separated by spaces,<br><strong>NR :</strong> Number of rows,<br><strong>length :</strong> Returns the number of characters in the line.</p><p>We can use these arguments as follows.</p><pre>awk ‘{print NF}’ file.txt</pre><pre>awk ‘{print NR}’ file.txt</pre><pre>awk ‘{print length}’ file.txt</pre><p>Additionally, we may want to specify a condition in the commands, below we want it to suppress all content with two field numbers (NF).</p><pre><em>Text File Content:</em></pre><pre>Cemre 809<br>Canan 905<br>Mustafa 997<br>Erol 720 Additional Area<br>Huriye 800<br>Günnur 600 Additional Field<br>Azra 650<br>Burcu 307 <br>Halim 345 Additional Domain</pre><p>When we run the following command for this text file, we do the filtering correctly.</p><pre>awk ‘{if (NF==2) {print}}’ file.txt</pre><p>In yet another example, let’s create a structure that allows us to query the data in our file.</p><p><strong>E.g:</strong><br>Let’s say we have a file like the one below. I set the name as file.txt.</p><pre><em>Text File Content:</em></pre><pre>1200<br>Cemre<br>Izmir</pre><p>Let’s query this file with an awk file. I opened a file called <em>myfile.awk</em> and determined its contents as follows.</p><pre>/[0–9]+/ {print “This line consists of a number.”}</pre><pre>/[A-Za-z]+/ {print “This line consists of text.”}</pre><pre>/^$/ {print “This line is a blank line.”}</pre><p>Next, let’s type the required command.</p><pre>awk -f myfile.awk file.txt</pre><p>After running this command, we will have printed information about the data in the lines of our <em>.txt</em> file.</p><p>We can also go over the examples below for programming with Awk.</p><p><strong>E.g:</strong><br>Let’s open a new blank <em>.txt</em> file and make 4–5 lines of space in it. Then when we give the following command, it will count us our blank lines.</p><pre>awk ‘/^$/ { print x++ }’ newfile.txt</pre><p>We can calculate grades with awk as an example from a different angle.<br>Let’s create a <em>note.txt</em> file similar to the following structure.</p><pre><em>Text File Content:</em></pre><pre>Ahmet 56 67 83 90 75<br>Selim 75 88 92 100 60<br>Meric 45 38 90 72 81</pre><p>And then let’s create a <em>calculate.awk</em> file and make its contents as follows.</p><pre>{<br>total = $2 + $3 + $4 + $5 + $6<br>average = total / 5<br>print $1 , average<br>}</pre><p>Now, finally, let’s give our command and print the average grade values of the people in the <em>.txt</em> file to the screen.</p><pre>awk -f calculate.awk note.txt</pre><p>Finally, as a small example, I tried to show you how to find large and small values with awk below, I think it will be useful for you.</p><p><strong>Finding the Biggest and Smallest Value:</strong></p><pre>awk ‘BEGIN {max=-999; min=999} { {if ($1 &gt;= max) {max = $1} } {if ($1 &lt;= min) {min = $1} } } END {print max, min}’ file.txt</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c40155041095" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/general-bash-usage-2-grep-awk-c40155041095">General Bash Usage — 2 (Grep-Awk)</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[General Bash Usage — 1]]></title>
            <link>https://medium.com/carbon-consulting/general-bash-usage-1-e57c9f37fdf?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/e57c9f37fdf</guid>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[shell-script]]></category>
            <category><![CDATA[bash-script]]></category>
            <category><![CDATA[terminal]]></category>
            <dc:creator><![CDATA[Cemre Acar]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 06:50:53 GMT</pubDate>
            <atom:updated>2021-12-06T06:50:53.354Z</atom:updated>
            <content:encoded><![CDATA[<h3>General Bash Usage — 1</h3><figure><img alt="General Bash Usage — 1" src="https://cdn-images-1.medium.com/max/1024/1*n3AQo-yf76Q4ACcSx1P5og.jpeg" /></figure><p>In this article, I will talk about the concept of Linux Bash. What is Bash? What is Bash Expansion? It’s a good list of Linux Variables , Linux Alias , Linux Pipe Operations... It goes on and on.</p><p>Bash is one of the most important and most used shells developed for UNIX. It is a shell or command language interpreter for the GNU operating system. Derived from the initials of the words Bourne Again Shell.</p><h4>Shell Variables</h4><p>Since Bash is also an interpreted programming language, basic programming constructs (Variables, Loops, etc.) are also available. A variable is determined by simply naming it and its type is not found.</p><p><strong>E.g :</strong></p><pre>variable = “Cemre”<br>echo $variable</pre><pre><em>Cemre</em></pre><h4>Alias</h4><p>We can define our own commands or shortcuts using aliases. Before creating a shortcut, let’s query if an alias is used with the name we want.</p><pre>type myVar</pre><p>If there is no definition, let’s create it with the alias command.</p><p><strong>E.g :</strong></p><pre>alias myVar = ‘cd ~ ‘</pre><p>When we give the myVar command with this command, it will send us to the home directory. To remove this alias also</p><pre>unalias myVar</pre><p>We can give the command.</p><h4>Pipe Operations</h4><p>Pipe is used to send the output of any command to another command that can process that output. Pipe is used with the ( <strong>|</strong> ) character. This character sends the output of the commands that come before it to the command that comes after it.</p><p><strong>E.g :</strong></p><p>Let’s list all the files in any directory in detail and send it to the <strong>lpr</strong> command, which allows us to print it from our printer.</p><pre>ls -l | lpr</pre><p>Here, all the information that the <strong>ls -l</strong> command will print on our screen has been extracted from our printer with the <strong>lpr</strong> command.</p><p>Now, let’s mention another topic…<br>In Pipe operations, we sent the output of the command to another command. Here, the purpose is to direct the command we give to a file. If we use the list command again, let’s assume that we have a text file named files.txt. We direct our listed files to this text file as follows.</p><p><strong>E.g :</strong></p><pre>ls -l &gt; files.txt</pre><p>this command has written the output of our listing command to our files.txt file.</p><pre>cat files.txt</pre><p>We can read quickly with the command. <br>What if I gave a wrong command? Will I be able to see this error in the file? I need to give a different command for this.</p><p><strong>E.g :</strong></p><p>Let’s try to list a file that is not in our directory.</p><pre>ls /wrong 2&gt; files.txt</pre><p>The different “<strong>2&gt;</strong>” in this command allowed us to suppress the standard error to our files.txt file.</p><p>If we make this work a little more colorful, let’s want it to be written in error.txt if the command we give will produce an incorrect output, and in the result.txt files if it will produce a standard output. Let’s use the ls command in our directory and see that it prints the result to <strong>result.txt</strong>. For this, it will be enough to run the following command.</p><pre>ls -l 2&gt;error.txt &gt; result.txt</pre><p>We see that the output is written to result.txt. Well, let’s try to do an incorrect operation in the same command.</p><pre>ls /non-folder 2&gt;error.txt &gt;result.txt</pre><p>Now we can see the standard error written to <strong>error.txt</strong>. There is a small point we should mention here. When using the <strong>&gt;</strong> sign when redirecting Standard output or Standard error, if you do not have a file with that name, it will be created and the output will be written to the file. If the file exists, its contents are destroyed and the output of the command is written as the new content of the file. In short, your existing information will be lost. If you want to add the output of a new command on top of the existing file, you should use the <strong>&gt;&gt;</strong> signs.</p><p><strong>E.g :</strong></p><pre>~ text = “Hello Bash”<br>~ echo $text &gt;&gt; result.txt</pre><p>Now, when we view our <strong><em>result.txt </em></strong>file with the <em>cat</em> command, you will see that the text “<strong>Hello Bash</strong>” has been added.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*n8rd9wdkE5OKe6TlQUmAOg.gif" /></figure><h4>Processes</h4><p>UNIX is a multitasking operating system. Processes are a running program. Each process can run in the background or under the control of the shell and has its own Process ID (PID). When we run a command in the shell, the Parent Process creates a Child Process. Here, fork() and wait() system calls, which are the main topics of Linux Operating Systems, are used. I won’t go too far on this theoretical part, so as not to confuse it.</p><p>We can use the ps command to view these processes in the shell. When this command is used, PID, TTY, TIME and CMD fields appear.</p><p>PID, Process ID ,<br>TIME, CPU time consumed,<br>TTY, terminal where the process is run,<br>CMD stands for the name of the program entered from the command line.</p><h4>Background Process</h4><p>I have said before that every process runs in the background or under the control of the shell. In order to run a command in the background, we put an “<strong>&amp;</strong>” sign at the end of the command we give. After giving the command with this character, we will continue from the shell screen again.</p><p><strong>E.g :</strong></p><pre>sleep 100 &amp;</pre><p>When we issue the command, the shell we are in goes to sleep for 100 seconds, but we continue our operations in another shell. If we give the ps command here again, we will see two processes running, and the process we put to sleep will appear under the CMD with the text sleep 100. It will automatically expire after 100 seconds.</p><p>Well, let’s say we give the sleep command in the existing shell. I entered 1000 by mistake, I will be on hold for a very long time and I don’t want to wait for it. For this, I need to interrupt or terminate the program.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e57c9f37fdf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/general-bash-usage-1-e57c9f37fdf">General Bash Usage — 1</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customizing Training with TensorFlow in Python]]></title>
            <link>https://medium.com/carbon-consulting/custom-training-with-tensorflow-in-python-dae0dcc693bd?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/dae0dcc693bd</guid>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[vgg16]]></category>
            <dc:creator><![CDATA[Salih Talha Akgün]]></dc:creator>
            <pubDate>Wed, 27 Oct 2021 11:51:21 GMT</pubDate>
            <atom:updated>2021-10-27T11:51:21.620Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve searched the internet for an easy and complete introduction to custom model building in TensorFlow but I didn’t find it. So I decided to share it and the best reason is, the code explains itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*aouBWf3ffmOCORzJ.jpg" /><figcaption>Just show me the code!</figcaption></figure><p>Sequential API is a very easy way to create sequential fully connected neural networks. I recommend you to read about it if you don’t know, before continuing this article.</p><p>There are several ways to customize your training. Let’s start with custom flows.</p><h3>1. Custom Flows</h3><p>You can create custom flows with Functional API. This helps you make models where you have two outputs like MaskRCNN where you have bounding box and class predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*ssAOlVnS5sGtQ9c3.jpg" /><figcaption>An Output of the MaskRCNN Model</figcaption></figure><p>For this, you have to use Functional API as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*GFTxazWUSsomlXavNBdVbA.png" /><figcaption>Custom Model Flow</figcaption></figure><p>The base model will not give the best results but the final model will output a bounding box and a classification prediction. You can change the base model as you want.</p><h3>2. Custom Loss Function</h3><p>In the training of a neural network, the loss function is very important to find good results. If you want to customize your loss function, I’ll assure you it’s the easiest customization you can make.</p><p>To show it, we’ll implement RMSE (root mean squared error), even though it already exists in TensorFlow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*bPfMNZ3BBX0JK40t1tIxhw.png" /><figcaption>Custom Loss Function Definition</figcaption></figure><p>You can change the definition of the function as you want. Don’t forget to pass it to the compile function. I recommend you to check <a href="https://www.tensorflow.org/api_docs/python/tf/keras/losses/Loss">this site</a> if you want to create it as a class.</p><h3>3. Custom Layers</h3><p>Here we’ve come to the most important parts.</p><p>For this article, we’ll define a quadratic layer wherein for every neuron, we’ll use <strong>w0 * x² + w1 * x + b</strong> instead of <strong>w0 * x + b.</strong></p><p>To make it, we define a class that inherits from the Layer class and overwrites three functions.</p><ul><li>The <strong>__<em>init__ </em></strong>function will be called when the layer we defined is called. Here we will need to <strong>get the variables we want to give to the Layer</strong>.</li><li>The <strong><em>build </em></strong>function will be called to create the first state of our variables. Don’t forget to define them as trainable.</li><li>The <strong><em>call </em></strong>function will be called when we want to get the output of this layer. It <strong>computes the values that we want to pass to the next layer</strong> and returns them.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*IhQ3lEWLRyXWmEcrtzOgGA.png" /><figcaption>Custom Model Class Definition</figcaption></figure><p>After that, you can just use this layer just like a Dense layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*roTvoBQ-FUawikqTih629Q.png" /><figcaption>Model Definition</figcaption></figure><p>I highly recommend you try to change it as default, where you use the linear function <strong>w0 * x + b.</strong></p><h3>4. Custom Model</h3><p>Let’s create a custom <a href="https://towardsdatascience.com/vgg-neural-networks-the-next-step-after-alexnet-3f91fa9ffe2c">VGG model</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/0*hGbWyJrgw9o_70l0.png" /><figcaption>VGG Model — source: <a href="https://paperswithcode.com/method/vgg">https://paperswithcode.com/method/vgg</a></figcaption></figure><p>I’ll first create a Layer that has <strong>Max Pooling + k * Convolution + ReLU. </strong>Then I’ll create the whole model by repeating it and giving how much convolution layer I want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*SWc_KsFQ0x266UEPTnVAFg.png" /><figcaption>Definition of a VGG Block</figcaption></figure><p>After I defined the Block, I’ll create the MyVGG model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*YNBPUQ1PNLiCZlHJInKxCA.png" /><figcaption>MyVGG Model Definition</figcaption></figure><p>Thus we can customize our model by giving different filters and repetitions to Block or changing the last layers of the VGG. We can customize it as we want, but if you define another layer just don’t forget to call it in the call function otherwise it’ll not change.</p><p>And that was the last one. In the last couple of weeks, I’ve finished the <a href="https://www.coursera.org/specializations/tensorflow-advanced-techniques">TensorFlow: Advanced Techniques Specialization</a> and most of the source code of this article is from the assignments in the course. They have great courses, I highly recommend them to everyone. You can check the all codes from my Github.</p><p><a href="https://github.com/SalihTalha/tensorflow_notes/tree/main/coursera%20assignments">tensorflow_notes/coursera assignments at main · SalihTalha/tensorflow_notes</a></p><p>Ask me if you have any questions. Please give your feedback. Thank you for reading to the end.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dae0dcc693bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/custom-training-with-tensorflow-in-python-dae0dcc693bd">Customizing Training with TensorFlow in Python</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer Models Hyperparameter Optimization With the Optuna]]></title>
            <link>https://medium.com/carbon-consulting/transformer-models-hyperparameter-optimization-with-the-optuna-299e185044a8?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/299e185044a8</guid>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[optuna]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Tue, 21 Sep 2021 17:46:05 GMT</pubDate>
            <atom:updated>2022-11-23T21:03:48.609Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rYt51yajCVHQaTwc" /><figcaption>Photo by <a href="https://unsplash.com/@alschim?utm_source=medium&amp;utm_medium=referral">Alexander Schimmeck</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before the beginning: Transformers library already has a function called “<strong><em>hyperparameter_search</em></strong>” in the <strong>Trainer</strong> object. But the aim of this article is to learn the Optuna by using Transformers.</p><p>Some codes look bad but to make copy-paste easier, I add the code with the screenshot of the good-looking code!</p><h3>What is Hyperparameter Optimization?</h3><p>A hyperparameter is a parameter whose value is used to control the learning process. By contrast, the values of other parameters (typically node weights) are learned. By changing these parameters, we can improve or worsen the accuracy of the model.</p><p>As a consequence, choosing the right hyperparameters are important. Hyperparameter optimization is the process of the choosing right values for the hyperparameters through searching.</p><h3>What is Optuna?</h3><blockquote><em>Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.</em></blockquote><blockquote><em>— From the official website</em></blockquote><p>With Optuna, it is easier to search for the best hyperparameters. Let’s start with a simple example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1dca4f5b75cbf8ea47e3ea7fc136b5a4/href">https://medium.com/media/1dca4f5b75cbf8ea47e3ea7fc136b5a4/href</a></iframe><p>Here we have:</p><ol><li>An objective function to optimize</li><li>A trial object to suggest some hyperparameter values</li><li>A study object to run the “optimize” method and find the best value from the 100 trials.</li></ol><p>Let’s dive in and learn with a simple text classification example:</p><h3>Get the dataset</h3><p>I will use the “Adverse Drug Reaction Data v2” dataset from the Huggingface Datasets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/084374d0bdfa233ad995e79a18e62d59/href">https://medium.com/media/084374d0bdfa233ad995e79a18e62d59/href</a></iframe><p>Normally, to train the model you would first load the model and the tokenizer and tokenize the dataset using the tokenizer then decide to train arguments and use all of these in the Trainer object to start training:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9078b2cbf64c1da732d7f31846d975ca/href">https://medium.com/media/9078b2cbf64c1da732d7f31846d975ca/href</a></iframe><p>For this example, what we want to do is find the best hyperparameters to get the minimum training loss. We will create a function named “objective”. This function will initialize the model, use different training argument values for N times and return the training loss using Optuna.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6117b65a3cb71ffee7be2570fd2fa35/href">https://medium.com/media/d6117b65a3cb71ffee7be2570fd2fa35/href</a></iframe><p>To optimize this “objective”, we need to create a study and call it’s “optimize” function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/685b40fe0845afdfece74bda37b80ae0/href">https://medium.com/media/685b40fe0845afdfece74bda37b80ae0/href</a></iframe><p>This is the complete code example:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87c07073e95a3e2a1a1ba2dbb098adcb/href">https://medium.com/media/87c07073e95a3e2a1a1ba2dbb098adcb/href</a></iframe><p>This is just a simple example; there are lots of things to learn, such as callbacks, samplers, and pruners! Thanks for reading!</p><p>Note from a reader: By default, during the first 10 trials, the random sampler is used via TPEsampler, so we might see more improvement when we increase n_trials &gt; 10.</p><h3>References</h3><ol><li><a href="https://en.wikipedia.org/wiki/Hyperparameter_optimization">https://en.wikipedia.org/wiki/Hyperparameter_optimization</a></li><li><a href="https://optuna.readthedocs.io/en/stable/index.html">https://optuna.readthedocs.io/en/stable/index.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=299e185044a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/transformer-models-hyperparameter-optimization-with-the-optuna-299e185044a8">Transformer Models Hyperparameter Optimization With the Optuna</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should serve your Tensorflow model using TF Serving?]]></title>
            <link>https://medium.com/carbon-consulting/why-you-should-serve-your-tensorflow-model-using-tf-serving-cb62c8a49370?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/cb62c8a49370</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[tfx]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Thu, 09 Sep 2021 05:32:25 GMT</pubDate>
            <atom:updated>2021-09-09T05:32:25.038Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SOXVzi838R4LTpJE.png" /></figure><p>Let’s say you have an AI task to solve. You get the data, clean the data, trained a model with it, and find the best hyperparameters. After these steps, you need to deploy your model or it won’t be useful, right? Once you have developed and trained a model that you’re happy with, it’s now time to deploy it. You can deploy your model with 3 options:</p><ol><li>Using Tensorflow Serving in your server</li><li>Using Tensorflow Lite to embed your model to mobile apps or in IoT components.</li><li>Using Tensorflow JS you can use your model right in the user’s browser.</li></ol><p>In this article, I will focus on Tensorflow Serving.</p><h3>Tensorflow Serving</h3><p>According to the TFX guide, TensorFlow Serving (TFS) is a flexible, high-performance serving system for machine learning models, designed for production environments. It consumes a SavedModel and will accept inference requests over either REST or gRPC interfaces. It runs as a set of processes on one or more network servers, using one of several advanced architectures to handle synchronization and distributed computation.</p><p>What are the benefits? Why should I use Tensorflow Serving instead of calling my model inside FastAPI(as an example, because it is fast.)?</p><p>Because TensorFlow Serving is specially designed and optimized for “Serving” your model, it is a lot faster than using in any python based backend-framework. In the end, I will share some test results to answer this question crystal clear.</p><p>While we are using GPU for training and predicting, we have some trouble when we try to use our model in a backend framework. Let’s say you have some requests from 20 different users. To handle more requests from more unique users, what would you do is incrementing worker counts like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HMvPVAjOD0xmSSw4X7sNUQ.png" /></figure><p>But as you run your service you will get an error after you get a request from more than 1 user at the same time. The issue is GPU synchronization. When you separate your service and your model with Tensorflow Serving, you can increment your worker count as much as you want and Tensorflow Serving will handle everything for you, the speed enhancement is very big and a nice bonus.</p><p>Now it’s time to share some impressive results of Tensorflow Serving. I’ve tested a model with a pre-trained BERT-base for the transfer learning part. I’ve used Locust to test the RPS rate and median response time. I have RTX 2060 GPU and Ryzen 5 1600AF CPU. In the tables below, RPS = Request per second and MRT = Median Response Time</p><h4>TF Serving using CPU (Worker count = User count)</h4><pre>+-----------------------+-------+---------+<br>|        Test           | RPS   | MRT(ms) |<br>+-----------------------+-------+---------+<br>| 1 User 5000 request   |     9 |     100 |<br>| 10 User 5000 request  | 15.5  |     620 |<br>| 50 User 5000 request  |    18 |    2900 |<br>+-----------------------+-------+---------+</pre><p>I don’t know the reason but you don’t have any problem with worker count if you use CPU to run your model.</p><h4>Model in FastAPI using CPU(Worker count = User count)</h4><pre>+-----------------------+------------+------------+<br>|        Test           |    RPS     |  MRT(ms)   |<br>+-----------------------+------------+------------+<br>| 1 User 5000 request   | 7.1        | 100        |<br>| 10 User 5000 request  | 16         | 620        |<br>| 50 User 5000 request  | PC crashed | PC crashed |<br>+-----------------------+------------+------------+</pre><h4>Tensorflow Serving using GPU(Worker count = User count)</h4><pre>+-----------------------+------+---------+<br>|        Test           | RPS  | MRT(ms) |<br>+-----------------------+------+---------+<br>| 1 User 5000 request   |   45 |      21 |<br>| 10 User 5000 request  | 105  |      87 |<br>| 50 User 5000 request  |  120 |     330 |<br>+-----------------------+------+---------+</pre><p>Reminder for next table, worker count must be 1, otherwise, you will get an error!</p><h4>Model in FastAPI using GPU(Worker count = 1)</h4><pre>+-----------------------+-------+---------+<br>|        Test           | RPS   | MRT(ms) |<br>+-----------------------+-------+---------+<br>| 1 User 5000 request   | 20.7  |      45 |<br>| 10 User 5000 request  | 22    |     450 |<br>| 50 User 5000 request  | 22-24 |    2300 |<br>+-----------------------+-------+---------+</pre><h3>References</h3><ol><li><a href="https://www.tensorflow.org/tfx/guide">https://www.tensorflow.org/tfx/guide</a></li><li><a href="https://www.tensorflow.org/tfx">https://www.tensorflow.org/tfx</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cb62c8a49370" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/why-you-should-serve-your-tensorflow-model-using-tf-serving-cb62c8a49370">Why you should serve your Tensorflow model using TF Serving?</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use one MongoDB instance in multiple FastAPI routers?]]></title>
            <link>https://medium.com/carbon-consulting/how-to-use-one-mongodb-instance-in-multiple-fastapi-routers-810c288b1c51?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/810c288b1c51</guid>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[pymongo]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Sun, 08 Aug 2021 15:35:34 GMT</pubDate>
            <atom:updated>2021-08-08T15:35:34.653Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*du7p50wS_fIsaC_lR18qsg.png" /><figcaption><a href="https://fastapi.tiangolo.com/">https://fastapi.tiangolo.com</a></figcaption></figure><blockquote>When I begin to use FastAPI, I was fascinated by how it is easy to use and how it is possible to have an organized backend structure with it.</blockquote><p>For example, when I have endpoints for<strong><em>/item</em></strong> and <strong><em>/users</em></strong> I would like to have them in separate files. With this approach, I could get rid of the problem of having one huge file to maintain and have a cleaner structure. I have just one problem with this approach: I have only one MongoDB instance and I need to use that one instance from every router file. After trying lots of bad or useless solutions, today I finally find the best and cleaner way to use the same MongoDB instance inside different routers and I want to share it with everyone. <a href="https://github.com/michaldev/fastapi-async-mongodb">This</a> is where I find the solution and I will explain this implementation using PyMongo( not an async solution ). I will try to keep it simple and small. This is my file structure:</p><pre>app</pre><pre>   --- db<br>         --- __init__.py<br>         --- mongo_manager.py<br>   --- routers<br>         --- __init__.py<br>         --- users.py<br>   --- main.py</pre><p>This is what we have in mongo_manager.py:</p><pre>import logging<br>from pymongo import MongoClient<br>from pymongo.database import Database<br><br><br>class MongoManager:<br>    client: MongoClient = None<br>    db: Database = None<br><br>    def connect_to_database(self, path: str):<br>        logging.info(&quot;Connecting to MongoDB.&quot;)<br>        self.client = MongoClient(path)<br>        self.db = self.client.main_db<br>        logging.info(&quot;Connected to MongoDB.&quot;)<br><br>    def close_database_connection(self):<br>        logging.info(&quot;Closing connection with MongoDB.&quot;)<br>        self.client.close()<br>        logging.info(&quot;Closed connection with MongoDB.&quot;)<br><br>    def get_users(self):<br>        users_query = self.db.users.find({}, {&#39;_id&#39;: 0})<br>        return list(users_query)</pre><p>It is a pretty simple class, given connection string, it connects to the database and assigns the client and db to the class variables. We only have one db operation which is get_users() and it returns all users in the database without their id in the database. db/__init__.py is next:</p><pre>from app.db.mongo_manager import MongoManager<br><br>db = MongoManager()</pre><p>That’s all :) We will use this db object from the main.py and users.py! So, main.py is next!</p><pre>import uvicorn<br>from fastapi import FastAPI<br><br>from app.db import db<br>from app.rest import users<br><br>app = FastAPI(title=&quot;FastAPI&quot;)<br><br>app.include_router(users.router, prefix=&#39;/api/users&#39;)<br><br><br>@app.on_event(&quot;startup&quot;)<br>async def startup():<br>    await db.connect_to_database(path=&quot;YOUR MONGODB CONNECTION STRING&quot;)<br><br><br>@app.on_event(&quot;shutdown&quot;)<br>async def shutdown():<br>    await db.close_database_connection()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    uvicorn.run(app, host=&quot;0.0.0.0&quot;, port=8000)</pre><p>As you can see from the 3. line we import the db object from the previous file we looked at. On the fastAPI <strong>startup event</strong>, we connect to the MongoDB database using the <strong><em>connect_to_database()</em></strong> function and we close the connection on <strong>shutdown. </strong>Now time to go to the users.py.</p><pre>from fastapi import APIRouter<br><br>from app.db import db<br><br>router = APIRouter()<br><br><br>@router.get(&#39;/&#39;)<br>def all_users():<br>    users = db.get_users()<br>    return users</pre><p>I don’t know if it is the magic of the fastAPI or not but when we import the same db object from here, it doesn’t give us a newly initialized MongoManager object. What we get is a MongoManager object we configure in the startup!</p><p>I guess that’s all! You can use the same db object in the different routers too! Thanks for reading and I hope it helps someone out there because I waste lots of time until I found this method :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=810c288b1c51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/how-to-use-one-mongodb-instance-in-multiple-fastapi-routers-810c288b1c51">How to use one MongoDB instance in multiple FastAPI routers?</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BERT: Encoder Stack Is All You Need]]></title>
            <link>https://medium.com/carbon-consulting/bert-encoder-stack-is-all-you-need-f1483cfe2e07?source=rss----de447e4c9973---4</link>
            <guid isPermaLink="false">https://medium.com/p/f1483cfe2e07</guid>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[language-model]]></category>
            <dc:creator><![CDATA[Muhammet Nusret Özateş]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 06:50:38 GMT</pubDate>
            <atom:updated>2021-06-24T06:50:38.061Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wb6M3smJkNB83QrAzBsDeQ.png" /><figcaption>Source: codemotion.com</figcaption></figure><p>In this article, I will talk about the BERT model and I will add sources to learn more about the BERT. To understand how BERT works, you need to understand how transformer models work. You can read <a href="https://medium.com/carbon-consulting/transformer-architecture-how-transformer-models-work-46fc70b4ea59">my article</a> about Transformers or you can learn it from <a href="http://jalammar.github.io/illustrated-transformer/">here</a>.</p><p>BERT [1] (Bidirectional Encoder Representations from Transformers) is a new language representation model. It is one of the ways of creating word embeddings and sentence representation vectors. BERT uses the encoder stack of the transformer model to output the representation of each token in the given input. Additionally, it has a special token “[CLS]” at the beginning of the inputs to use in classification tasks. BERT has two steps:</p><ol><li>Pre-training</li><li>Fine-tuning</li></ol><p>The pre-training step is where the model is trained to learn the given language(s) and output meaningful representation of the given input sequence (“sequence” term especially used here to emphasize given input can be more than one sentence.). To do this step, the model is trained using unlabeled text data using two different tasks. For fine-tuning, labeled data is needed and the model is fine-tuned to do given downstream tasks such as classification or question answering.</p><h3>Pre-training</h3><p>Pre-training is done with two different tasks:</p><ol><li>Masked Language Model</li><li>Next Sentence Prediction</li></ol><h4>Masked Language Model</h4><p>According to the authors of the BERT model, deep bidirectional models are powerful than unidirectional models. The problem with bidirectional models is, it allows to each word indirectly “see itself” and the model can easily predict the target word. To solve this problem, the authors mask some percentage(%15 in the article) of the input tokens randomly by changing the word with the special token “[MASK]” and the model is tasked with the prediction of the masked tokens. This is called the “Masked Language Model” but it is called a “cloze test” (fill in the blanks questions) in literature. The output of the hidden vectors (only at the masked tokens’ positions) are given into a softmax layer with the neuron size is equal to vocabulary size to predict real words.</p><p>The “[MASK]” token is used for pre-training but not in fine-tuning section. This causes a mismatch and to solve this problem, the authors choose to not using the “[MASK]” token every time. The selected token <em>i </em>with the %80 probability, will be replaced with the “[MASK]” token, with %10 probability it will be replaced by a random token, and with %10 probability, it will remain unchanged. Then the token in the <em>i’th</em> position will be used to predict the original token.</p><figure><img alt="An example of masked language model from Jay Alammar[2]" src="https://cdn-images-1.medium.com/max/1024/1*R_femtoSuJ3Pp12B6wqvFQ.png" /><figcaption>An example of the masked language model from Jay Alammar [2]</figcaption></figure><h4>Next Sentence Prediction</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MQcZ-pEtFmkl27L_LeEc5g.png" /><figcaption>An example of the next sentence prediction model from Jay Alammar [2]</figcaption></figure><p>Some NLP tasks such as Question Answering and Natural Language Inference(Determining whether a “hypothesis” is true (entailment), false (contradiction), or undetermined (neutral) given a “premise”.) need to understand the relation between sentences. This relation cannot be captured with the masked language model task. To train a model that can capture the relationship between two sentences, the authors give two sentences “Sentence A” and “Sentence B” as input. With the %50 probability, “Sentence B” is the actual sentence that comes after “Sentence A” and with the %50 probability it is a random sentence. As in the example above, the special “[CLS]” token is used to predict if the second sentence comes after the first sentence.</p><h3>Fine-tuning</h3><p>It is a very simple step, by swapping the input depending on the task (single sentence or two sentences)<br>and connecting the output to an appropriate classification layer. For a sentence classification task, the output vector of the model’s “[CLS]” token could be connected to a softmax layer and</p><ol><li>Freeze the BERT weights and train the latest classification weights</li><li>Fine-tune all weights</li></ol><h3>Code Example</h3><p>Using Tensorflow Hub, training or fine-tuning BERT models is very easy. In the following steps, I will show you how you can use a BERT model to detect toxicity in texts for the <a href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/overview">Toxic Comment Classification Challenge</a>. Download the train data first and create a Tensorflow dataset, separate it as train and validation set using this code:</p><pre>import tensorflow as tf</pre><pre>dataset = tf.data.experimental.make_csv_dataset(<br>    &#39;data/kaggle_toxic_comments/train.csv&#39;, batch_size=batch_size, num_epochs=1<br>    , select_columns=[&#39;comment_text&#39;, &#39;toxic&#39;, &#39;severe_toxic&#39;, &#39;obscene&#39;, &#39;threat&#39;, &#39;insult&#39;, &#39;identity_hate&#39;])<br><br>dataset = dataset.unbatch()<br>validation = dataset.take(10000)<br>train = dataset.skip(10000)<br><br>train = train.map(lambda data: (</pre><pre>    dataset_preprocessing(data[&#39;comment_text&#39;]),<br>    [data[&#39;toxic&#39;],<br>     data[&#39;severe_toxic&#39;],<br>     data[&#39;obscene&#39;],<br>     data[&#39;threat&#39;],<br>     # data[&#39;sexual_explicit&#39;],<br>     data[&#39;insult&#39;],<br>     data[&#39;identity_hate&#39;],<br>     ])).batch(batch_size).cache().prefetch(tf.data.AUTOTUNE)<br><br>validation = validation.map(lambda data: (<br>    dataset_preprocessing(data[&#39;comment_text&#39;]),<br>    [data[&#39;toxic&#39;],<br>     data[&#39;severe_toxic&#39;],<br>     data[&#39;obscene&#39;],<br>     data[&#39;threat&#39;],<br>     # data[&#39;sexual_explicit&#39;],<br>     data[&#39;insult&#39;],<br>     data[&#39;identity_hate&#39;],<br>     ])).batch(batch_size).cache().prefetch(tf.data.AUTOTUNE)<br><br>return train, test</pre><p>In the code above, I did some preprocessing but you can just use “data[“comment_text”]”.</p><p>Let’s create the BERT model using the Tensorflow Hub:</p><pre>from tensorflow.keras.layers import *<br>import tensorflow_hub as hub</pre><pre>text_input = tf.keras.layers.Input(shape=(), dtype=tf.string)<br>preprocessor = hub.KerasLayer(<br>    &quot;https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3&quot;)<br>encoder_inputs = preprocessor(text_input)<br>encoder = hub.KerasLayer(<br>    &quot;https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/4&quot;,<br>    trainable=False)<br>outputs = encoder(encoder_inputs)<br>pooled_output = outputs[&quot;pooled_output&quot;]  # [batch_size, 512].<br>sequence_output = outputs[&quot;sequence_output&quot;][:, 0, :]</pre><p>Now we have two different outputs, pooled output, and sequence output. The pooled output represents each input sequence as a whole, and the sequence output represents each input token in context. Either of those can be used as input to further model building. For this task I want to use the output of the [CLS] token so I connect the sequence_output to a sigmoid layer and create the model like this:</p><pre>classification_output = Dense(6, activation=&#39;sigmoid&#39;)(sequence_output)<br><br>embedding_model = tf.keras.Model(text_input, classification_output)</pre><p>Now I can just compile and fit the model:</p><pre>embedding_model.compile(<br>    optimizer=tf.keras.optimizers.Nadam(learning_rate=0.025),<br>    loss=tf.keras.losses.BinaryCrossentropy(),<br>    metrics=[tf.keras.metrics.AUC()],<br>    run_eagerly=False<br><br>)<br><br>embedding_model.summary()<br><br>embedding_model.fit(x=train_data, validation_data=test_data, epochs=2)</pre><p>Thanks for reading!</p><h3>References</h3><ol><li><a href="https://arxiv.org/pdf/1810.04805.pdf">https://arxiv.org/pdf/1810.04805.pdf</a> BERT paper</li><li><a href="http://jalammar.github.io/illustrated-bert/">http://jalammar.github.io/illustrated-bert/</a> Awesome and more detailed explanation of BERT model</li><li><a href="https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/4">https://tfhub.dev/tensorflow/bert_en_uncased_L-24_H-1024_A-16/4</a> An example BERT model from Tensorflow Hub</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1483cfe2e07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carbon-consulting/bert-encoder-stack-is-all-you-need-f1483cfe2e07">BERT: Encoder Stack Is All You Need</a> was originally published in <a href="https://medium.com/carbon-consulting">CARBON CONSULTING</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>