<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sagar Gadewar on Medium]]></title>
        <description><![CDATA[Stories by Sagar Gadewar on Medium]]></description>
        <link>https://medium.com/@sagar-gadewar?source=rss-b436be0199a3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FYblgGDj7EyhfPsBVxy4dw.jpeg</url>
            <title>Stories by Sagar Gadewar on Medium</title>
            <link>https://medium.com/@sagar-gadewar?source=rss-b436be0199a3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:23:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sagar-gadewar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deploy Django Application on Windows IIS Server]]></title>
            <link>https://blog.nonstopio.com/deploy-django-application-on-windows-iis-server-93aee2864c41?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/93aee2864c41</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[windows-server]]></category>
            <category><![CDATA[iis-server]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:49:10 GMT</pubDate>
            <atom:updated>2021-07-06T11:33:02.776Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*-lptGfDAx-0lrNVa8lCulA.jpeg" /></figure><p>Hello geek,</p><p>This blog is basically focused on deploying the Django application, on Windows Server and IIS as the production environment.</p><p>Sometimes client requirement is they are more comfortable with Windows rather than Linux OS.</p><h4><strong>Step 1: Create VM</strong></h4><p>We need a Windows server on which we can host our Django server. for that Either we can create a virtual machine or host on Cloud or can use a bare-metal server.</p><p>reference URL: <a href="https://medium.com/@sagar.gadewar/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88">https://medium.com/@sagar.gadewar/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88</a></p><p>recommended: <strong>Windows server 2019</strong></p><h4>Step 2: Download the Python 3 Installer</h4><p>To get started working with Python 3, you’ll need to have access to the Python interpreter.</p><ol><li>Open a browser window and navigate to the <a href="https://www.python.org/downloads/windows/">Download page for Windows</a> at <a href="https://www.python.org/">python.org</a>.</li><li>Underneath the heading at the top that says <strong>Python Releases for Windows</strong>, click on the link for the <strong>Latest Python 3 Release — Python 3.x.x</strong>. (As of this writing, the latest in Python 3.6.5.)</li><li>Scroll to the bottom and select either <strong>Windows x86–64 executable installer</strong> for 64-bit or <strong>Windows x86 executable installer</strong> for 32-bit. (See below.)</li><li>Once you have chosen and downloaded an installer, simply run it by double-clicking on the downloaded file. A dialog should appear that looks something like this:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ka-mFfxWVjPRlOiAlLzLig.png" /></figure><p>5. In the Python Setup dialog that appears, click “<strong>Customize Installation”.</strong></p><p>6. Make sure <strong>Add Python 3.8 to the path</strong> checkbox selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOA4vHe4FGVSlQeaVmMUUw.png" /></figure><p><strong>[Optional]</strong></p><p>7. On the Optional Features step, leave the default settings and click “Next”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkYbxQNzzZX7rGYmMlEOmQ.png" /></figure><p>8. On the Advanced Options step, make the following changes:</p><ul><li>Check the “<strong>Install for all users</strong>” box (note that this will also check the “Precompile standard library” box).</li><li>Check “<strong>Add Python to environment variables</strong>”.</li><li>In the input box below “<strong>Customize install location</strong>” change the value to the following: <strong>C:\Python&lt;Version&gt;.</strong></li></ul><p>9. Click “Install”</p><p>10. When the installation is complete, click the <strong>Close </strong>button.</p><p><strong>Verify Python Installation</strong></p><ul><li>Open a new Command Prompt. Note that if you already had a command prompt open you will need to close and reopen it, as the Python installation process added new environment variables and any open command prompts will not have the new environment variables available.</li><li>Type python and press Enter. This should open a Python interpreter:</li></ul><pre>c:\&gt;  python<br>&gt;&gt;&gt;<br>&gt;&gt;&gt; exit()</pre><blockquote>If instead of seeing the Python interpreter you see the error “‘python’ is not recognized as an internal or external command …” this likely means the “Add Python to environment variables” box was not checked on the Advanced Options step of the installation process. If this occurs you can uninstall and reinstall Python and make sure to check the “Add Python to environment variables” checkbox, or add the following to the System environment “Path” variable:<br>C:\Python38;C:\Python38\Scripts</blockquote><h4>Step 3: Git for Windows stand-alone installer</h4><ol><li>Download the latest <a href="https://git-for-windows.github.io/">Git for Windows installer</a>.</li><li>When you’ve successfully started the installer, you should see the <strong>Git Setup</strong> wizard screen. Follow the <strong>Next</strong> and <strong>Finish</strong> prompts to complete the installation. The default options are pretty sensible for most users.</li><li>Open a Command Prompt (or Git Bash if during installation you elected not to use Git from the Windows Command Prompt).</li><li>Run the following commands to configure your Git username and email using the following commands, replacing Emma’s name with your own. These details will be associated with any commits that you create:</li></ol><pre>$ git config --global user.name &quot;Sagar Gadewar&quot;<br>$ git config --global user.email &quot;eparis@atlassian.com&quot;</pre><h4>Step 4: Configure Database</h4><p><strong>To configure Database on bare metal (on Windows Server)</strong></p><p>Refer URL:<strong> </strong><a href="https://www.postgresqltutorial.com/install-postgresql/">https://www.postgresqltutorial.com/install-postgresql/</a></p><p><strong>To configure RDS</strong></p><p>Refer URL:<strong> </strong><a href="https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/">https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/</a></p><h4>Step 5: Configure Python Virtual Environment</h4><p><a href="https://packaging.python.org/key_projects/#venv">venv</a> (for Python 3) and <a href="https://packaging.python.org/key_projects/#virtualenv">virtualenv</a> (for Python 2) allow you to manage separate package installations for different projects. They essentially allow you to create a “virtual” isolated Python installation and install packages into that virtual installation. When you switch projects, you can simply create a new virtual environment and not have to worry about breaking the packages installed in other environments. It is always recommended to use a virtual environment while developing Python applications.</p><p>1. Open a <strong>command prompt</strong> and run the following commands</p><pre><br>c:\Users\sagar&gt;  <strong>cd C:\</strong></pre><pre>c:\&gt;  <strong>mkdir pyvirtualenv</strong></pre><pre>c:\&gt;  <strong>cd pyenv</strong></pre><pre>c:\pyenv&gt;  <strong>python -m venv Eric</strong></pre><pre>c:\pyenv&gt;  <strong>cd C:\pyvirtualenv\Eric</strong></pre><pre>C:\pyenv\Eric&gt;  <strong>Scripts\activate.bat</strong></pre><pre>(Eric) C:\pyenv\Eric&gt;  <strong>python -m pip install --upgrade pip<br></strong>Collecting pip<br>Downloading<br><a href="https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da1">https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da1</a><br>42ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)|████████████████████████████████| 1.4MB 6.8MB/s<br>Installing collected packages: pip<br>Found existing installation: pip 19.2.3Uninstalling pip-19.2.3:<br>Successfully uninstalled pip-19.2.3<br>Successfully installed pip-20.0.2</pre><pre>(Eric) C:\pyenv\Eric&gt; <strong> cd ..</strong></pre><pre>(Eric) C:\pyenv&gt; <strong> git clone &lt;django project repository url&gt;</strong></pre><pre>(Eric) c:\pyenv&gt;  <strong>cd &lt;django-project&gt;</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt;  <strong>pip install wfastcgi</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt;  <strong>pip install -r requirement.txt</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt; <strong>python  manage.py makemigrations</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt;<strong> python  manage.py migrate</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt;<strong> python  manage.py collectstatic</strong></pre><pre>(Eric) c:\pyenv\&lt;django-project&gt;&gt;<strong> python  manage.py createsuperuser<br>admin: &lt;admin name&gt;<br>email:<br>password: &lt;password&gt;</strong></pre><blockquote>Make sure in setting.py file under your project folder you do:</blockquote><blockquote>1. DEBUG = False</blockquote><blockquote>2. ALLOWED_HOSTS= [‘localhost’,’server ip’]</blockquote><h4>Step 6: Install IIS</h4><p>Even if you already have IIS installed, you may need to go through these steps to install the CGI feature if it wasn’t installed as part of the original IIS installation.</p><ul><li>Open the Control Panel</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HeGd4gSoR_HuOSEsN3fR7g.png" /></figure><ul><li>In the search box in the top right, type “windows features” (without the quotes)</li><li>In the search results under “Programs and Features” click “<strong>Turn Windows features on or off</strong>” This launches the Add Roles and Features Wizard.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ROl5x9rnIDEkJG5TIM30ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLwWQIxLiVJcnzGH4HkSqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zLwWQIxLiVJcnzGH4HkSqA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5qvYVfV3ju1SYZPbp-mkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5qaPyVyzOHRo62QKu2J-Nw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fiLEbrkoarXTFLaH-Mi4dg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IU7IDy0XCGwmOSQELIV_2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SLNXI-jzO9Me0OlOHmsbAQ.png" /></figure><p><strong>Verify the IIS Installation</strong></p><ul><li>Open a web browser on the server.</li><li>Enter <a href="http://localhost/"><em>http://localhos</em>t</a>/ in the address bar and press Enter. You should see the default IIS page.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*SfdOeR1ZcGzyRSqw.png" /></figure><ul><li>If you don’t see the default IIS page:</li></ul><blockquote><em>1. Open Control Panel</em></blockquote><blockquote><em>2. Type “services” in the search box</em></blockquote><blockquote><em>3. Under “Administrative Tools” click “View local services”</em></blockquote><ul><li>Scroll to the bottom of the list and ensure you see “World Wide Web Publishing Service” listed, and that the status is “Running”.</li></ul><h4>Step 7: Configure IIS</h4><h4>Configure FastCGI in IIS</h4><ol><li>Open the IIS Manager by clicking the Windows button, clicking on Administrative Tools, and double-clicking Internet Information Services (IIS) Manager</li><li>Click on the name of the server in the list on the left. If you see the following dialog box, check the box “Do not show this message” and click “No.” (You can always get to this later if necessary by clicking “Get New Web Platform Components” under “Actions” on the right-hand side of IIS Manager.)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*SLHvLPQ1P6LP-7_a.png" /></figure><p>3. Double-click the “FastCGI Settings” icon</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*yS4HcVJnF5LHjyXd.png" /></figure><p>4. Under “Actions” on the right-hand side click “Add application …”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*h5jpRWX5lHU5zXBqGcBZBw.png" /></figure><p>5. In the Add FastCGI Application dialog, in the “Full Path” box, type the path to the Python executable for the application’s virtual environment:<br><em>C:\virtualenvs\Eric\Scripts\python.exe</em></p><p>6. In the Arguments input box, type the path to the wfastcgi.py file in the application’s virtual environment:<br><em>C:\virtualenvs\Eric\Lib\site-packages\wfastcgi.py</em></p><p>7. At this point your settings in the Add FastCGI Application dialog should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h9jORNnHelfynTNr9PSppg.png" /></figure><p>8. With the Add FastCGI Application dialog box still open, under the “General” section, click on the “<strong>Environment Variables</strong>” line, then click the gray “…” button that appears next to (Collection) on the right-hand side of the line. This opens the Environment Variables Collection Editor dialog.</p><p>9. In the Environment Variables Collection Editor dialog, click “Add”</p><p>10. In the “Name properties” section on the right, click the input box to the right of “Name,” remove the “Name” text that is already in the input box and enter <strong>DJANGO_SETTINGS_MODULE</strong> (note that this MUST be entered in ALL CAPS)</p><p>11. Click the input box to the right of “Value” and enter Ericsson.settings</p><p>12. Click “Add” again and enter the following:<br>Name: <strong>PYTHONPATH</strong><br>Value: C:\User_details\Ericsson</p><p>13. Click “Add” again and enter the following:<br>Name: <strong>WSGI_HANDLER</strong><br>Value: Ericsson.wsgi.application</p><p>14. At this point you will have three environment variables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*rGrgVedDTV_BJE1a.png" /></figure><blockquote><strong><em>NOTE:</em></strong><em> All of these settings are CASE-SENSITIVE. They must be entered with exactly the case indicated here in order to work.</em></blockquote><ul><li>Click “OK” to close the EnviromentVariables Collection Editor.</li><li>Click “OK” to close the Add FastCGI Application dialog.</li></ul><h4>Step 8: Create and Configure a New IIS Web Site</h4><p>Next, we need to create a new website in IIS for the Django application and add a Handler Mapping to the newly created website so that requests coming to the IIS website are passed off to the Django application for processing.</p><ul><li>Open IIS Manager</li><li>On the left-hand side under Connections, expand the tree under the server name by clicking on the arrow to the left of the server name</li><li>Right-click on the Sites folder and click “Add Website …”</li><li>For the site, name enter Ericsson</li><li>For the physical path, type the following: <em>C:\User_details\Ericsson</em></li><li>For the purposes of this example configuration, change the Port to 81, since the Default site is running on port 80. For a real-world application, you’ll likely want to use name-based virtual hosting by adding bindings and run the site on port 80.</li><li>Under the Binding section, the IP address parameter dropdown selects the private IP address.</li></ul><p>You may leave the “Hostname” blank. At this pIn short:</p><p>Step 1:oint the Add Website dialog should look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*vCg3MdmzFhOO6alT.png" /></figure><ul><li>Click “OK”</li></ul><p>Next, we’ll add a FastCGI handler mapping to this site so the requests coming into the site will be processed by the Django application.</p><ul><li>In IIS Manager, expand the Sites folder on the left-hand side and click on the site</li><li>On the right, double-click “<strong>Handler Mappings</strong>”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*_FtoubqyOfgCWpw5.png" /></figure><p>On the right, under “Actions,” click “Add Module Mapping”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*s9VAD_sGYCGWIVWa20zfTg.png" /></figure><ul><li>In the “Request path” box enter an asterisk: *</li><li>Click the arrow on the right-hand side of the “Module” box and select “FastCgi Module”</li></ul><blockquote><em>NOTE: Make sure to select </em><strong><em>FastCgi Module</em></strong><em>, NOT CgiModule</em></blockquote><ul><li>In the “Executable” box, enter the following:<br>C:\virtualenvs\Eric\Scripts\python.exe|C:\virtualenvs\Eric\Lib\site-packages\wfastcgi.py</li></ul><blockquote><em>Note that the character after python.exe is a pipe (|), which is entered by pressing Shift-\ on your keyboard</em></blockquote><ul><li>In the “Name” box, enter Django Handler (you can call this whatever you want; it’s merely a friendly name for the module mapping)</li><li>The Add Module Mapping dialog should now look like this:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*6AMhmz5TRfVbfrI6.png" /></figure><ul><li>Click the “Request Restrictions” button and uncheck the “Invoke handler only if the request is mapped to the” checkbox</li><li>Click “OK” to close the Request Restrictions dialog</li><li>Click “OK” to close the Add Module Mapping dialog</li><li>When prompted “Do you want to create a FastCGI application for this executable?” click “No” since we created the application earlier.</li></ul><blockquote><em>Note that you CAN have it create the FastCGI application for you from the module mapping, but the settings seem to be different and the end result isn’t fully editable. I also detailed how to create the FastCGI application to be as thorough as possible with all the various pieces involved.</em></blockquote><ul><li>You will now see the Django Handler listed in the Handler Mappings for the foo website:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*p0iuWtXcvIPNGmL_.png" /></figure><p>At this point everything should be working, so verify by loading the application in a browser.</p><ul><li>Open a browser on the server</li><li>Browse to <a href="http://localhost:81/">http://localhost:8</a>000/</li></ul><h4>Step 9: Configure the Windows Firewall</h4><ol><li>Navigate to Control Panel, System and Security, and Windows Firewall.</li><li>Select Advanced settings and highlight Inbound Rules in the left pane.</li><li>Right-click Inbound Rules and select New Rule.</li><li>Add the port you need to open and click Next.</li><li>Add the protocol (TCP or UDP) and the port number into the next window and click Next.</li><li>Select Allow the connection in the next window and hit Next.</li><li>Select the network type as you see fit and click Next.</li><li>Under the profile section, selects all checkboxes (Domain, Private &amp; Public).</li><li>Name the rule something meaningful and click Finish.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*LCeMUAU_xZdvSg22BIz3PQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*ugVk7OkfzCJYiUopjoeGkA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ukERBYg9rxJt6U4ypsvmFQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sZW6q7nAnol8d09mCcmPtg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JV0eGUOKPznCA8dkxAnbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nQ9whzGo5_Xdfq2oA71EQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LDTF43cPuqagT-ycZJB6g.png" /></figure><p>Make sure Users should have access to read/write access on the <strong>debug.log</strong> file in the Django project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*5roDURHJ7ePYuW7ITN5NWQ.png" /></figure><h4>Step 10: Create static applications for serving static content</h4><p>On left side <strong>expand sites website</strong></p><p>Right Click on main Django applications select Edit bindings</p><p>make sure you selected private IP</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*M_hCS1LbSXFpCwWUFlpa4g.png" /></figure><p>Right-click on static folder select <strong>convert into the application</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*9y_Sfz6HWl3LZ3-AwSOIYA.png" /></figure><blockquote>Fill out Add Application form with the</blockquote><blockquote><strong>Site name(Project Name), Application pool(Project Name), Physical path(Absolute path of static folder)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WgVToLq9uZWs6MU72yDpKg.png" /></figure><p>After static application converted into an application. <strong>Choose the static application</strong> and Open <strong>Static Home Handler Mappings.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*nbLtDyXQ8os5FmRRzcho4Q.png" /></figure><p><strong>Remove Django Handler for static application</strong></p><p>Right-click Django Handler entry select <strong>Remove</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*O_HPOZAzICn7h0IxCHsGaw.png" /></figure><p>I hope this blog is helpful to you……..</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F3473690&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fsponge-bob-square-pants-animation-cartoon-comedy-complete-gif-3473690&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fimages%2Fcab2c7e1bf5a9eb7fe5455c7be0001bf%2Fraw&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fab2067ceeaf7895696474db889130a/href">https://medium.com/media/3fab2067ceeaf7895696474db889130a/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93aee2864c41" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/deploy-django-application-on-windows-iis-server-93aee2864c41">Deploy Django Application on Windows IIS Server</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MongoDB Performance Testing Using JMeter]]></title>
            <link>https://blog.nonstopio.com/mongodb-performance-testing-using-jmeter-f98382844fbc?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/f98382844fbc</guid>
            <category><![CDATA[jmeter]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:36:27 GMT</pubDate>
            <atom:updated>2021-07-06T11:31:37.691Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z1FOwiebUZ-n2NYL.jpg" /></figure><p>Hello,</p><p>Sometimes we have to make some configuration changes in database configuration or add encryption or migrate DB platforms like from data center to the cloud than the most common question we had that what would be the effect on database performance and how could we measure this. How to define this as a good config is a bad config for us. I also faced the same kind of scenario.</p><p>I would like to share this experience with you. There could be multiple ways to test the performance of MongoDB. one of which is using JMeter.</p><p>Performance testing is a non-functional software testing technique that determines how the stability, speed, scalability, and responsiveness of an application hold up under a given workload.</p><p><strong>MongoDB</strong> is a cross-platform, document-oriented database that provides, high performance, high availability, and easy scalability. MongoDB works on the concept of collection and document.</p><p>The <strong>Apache JMeter™</strong> application is open-source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.</p><p><strong>Database Load testing</strong> involves simulating real-life user <strong>load</strong> for the target <strong>Database</strong> application. It helps you determine how your <strong>Database</strong> application behaves when multiple users hit it simultaneously.</p><p>In this blog we’ll try to create samplers to test the basic MongoDB operations:<br>connect to a database<br>create a document<br>read this document<br>modify the document<br>delete this document</p><h4>Setup JMeter</h4><p>Let&#39;s say I have Ubuntu 18.04 as the host machine</p><ol><li>Install Java</li></ol><pre>$ sudo apt install default-jdk</pre><pre>## Verify<br>$ java -version</pre><p>2. Download JMeter binary (tar.tz file) from the official page <a href="https://jmeter.apache.org/download_jmeter.cgi">here</a></p><pre>$ wget <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz</a><br>$ tar -xvzf <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">apache-jmeter-5.4.1.tgz</a><br>$ ls <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">apache-jmeter-5.4.1</a>/</pre><p>3. Check java driver compatibility on MongoDB docs<br>refer <a href="https://docs.mongodb.com/drivers/java/">https://docs.mongodb.com/drivers/java/</a></p><p>4. Download mongo-java driver jar files<br><strong>refer</strong> <a href="https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver">https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver</a></p><p>5. Go to extracted folder of JMeter follow lib folder <br>Delete existing mongo-java-driver and copy newly downloaded mongo-java-driver</p><p>6. Open terminal &amp; lanuch JMeter</p><pre>$ ./bin/jemter.sh</pre><h4>Create test plan</h4><ol><li>Right click add Thread Group =&gt; Follow add Sampler =&gt; Select JS2233_sampler<br>rename to MongoDBCreate<br>Copy following MongoDB connect and create a database with collection code in script section</li></ol><pre>import com.mongodb.*<br>import com.mongodb.client.MongoClients;<br>import com.mongodb.client.MongoClient;<br>import com.mongodb.MongoClientSettings;<br>import com.mongodb.ServerAddress;<br>import com.mongodb.client.MongoCollection;<br>import com.mongodb.client.MongoDatabase;<br>import org.bson.Document;<br>import java.util.Arrays;</pre><pre>try {<br> MongoClientSettings settings = MongoClientSettings.builder()<br>  .applyToClusterSettings {builder -&gt; <br>   builder.hosts(Arrays.asList(new ServerAddress(vars.get(&quot;mongoHost&quot;),vars.get(&quot;mongoPort&quot;).toInteger())))}<br>  .build();<br> <br> MongoClient mongoClient = MongoClients.create(settings);<br> <br> MongoDatabase database = mongoClient.getDatabase(vars.get(&quot;databaseName&quot;));<br> MongoCollection&lt;Document&gt; collection = database.getCollection(vars.get(&quot;collectionName&quot;));<br> <br> vars.putObject(&quot;collection&quot;, collection);<br> <br> return &quot;Connected to &quot; + vars.get(&quot;collectionName&quot;);<br>}<br>catch (Exception e) {<br> SampleResult.setSuccessful(false);<br> SampleResult.setResponseCode(&quot;500&quot;);<br> SampleResult.setResponseMessage(&quot;Exception: &quot; + e);<br>}</pre><blockquote><strong>Note:</strong> Add another JSR223 Samplers &amp; repeat above same step for MongoDB_Insert, MongoDB_Query, MongoDB_Update, MongoDB_Delete samplers in JMeter</blockquote><blockquote>Add a View Results Tree Listener.</blockquote><blockquote><em>Right Click-&gt;Add-&gt;Listener-&gt;View Results Tree</em></blockquote><p><strong><em>Code for MongoDB_Insert:</em></strong></p><pre>import com.mongodb.client.MongoCollection;<br>import org.bson.Document;<br>import java.util.Arrays;</pre><pre>try {<br> MongoCollection&lt;Document&gt; collection = vars.getObject(&quot;collection&quot;);<br> <br> Document document = new Document(&quot;firstName&quot;, &quot;Expert&quot;)<br>  .append(&quot;lastName&quot;, &quot;Protocolson&quot;)<br>  .append(&quot;age&quot;, 37)<br>  .append(&quot;occupation&quot;, &quot;DevOps&quot;)<br>  .append(&quot;skills&quot;, Arrays.asList(&quot;System Administration&quot;, &quot;Linux&quot;))<br>  .append(&quot;adress&quot;, new Document(&quot;city&quot;, &quot;Systemberg&quot;)<br>  .append(&quot;street&quot;, &quot;Data Line&quot;)<br>  .append(&quot;house&quot;, 42));<br> <br> collection.insertOne(document);</pre><pre>return &quot;Document inserted&quot;;<br>}<br>catch (Exception e) {<br> SampleResult.setSuccessful(false);<br> SampleResult.setResponseCode(&quot;500&quot;);<br> SampleResult.setResponseMessage(&quot;Exception: &quot; + e);<br>}</pre><p><strong><em>Code for MongoDB_Query:</em></strong></p><pre>import com.mongodb.client.MongoCollection;<br>import static com.mongodb.client.model.Filters.*;<br>import org.bson.Document;<br>import org.bson.types.ObjectId;</pre><pre>try {<br> MongoCollection&lt;Document&gt; collection = vars.getObject(&quot;collection&quot;);<br> <br> Document result = collection.find(eq(&quot;firstName&quot;, &quot;Expert&quot;)).first();</pre><pre>vars.put(&quot;exampleDocumentId&quot;, result.get(&quot;_id&quot;).toString());<br> <br> return &quot;Document with id=&quot; + result.get(&quot;_id&quot;) + &quot; found&quot;;<br>}<br>catch (Exception e) {<br> SampleResult.setSuccessful(false);<br> SampleResult.setResponseCode(&quot;500&quot;);<br> SampleResult.setResponseMessage(&quot;Exception: &quot; + e);<br>}</pre><p><strong><em>Code for MongoDB_Update:</em></strong></p><pre>import com.mongodb.client.MongoCollection;<br>import static com.mongodb.client.model.Filters.*;<br>import static com.mongodb.client.model.Updates.*;<br>import org.bson.Document;<br>import org.bson.types.ObjectId;</pre><pre>try {<br> MongoCollection&lt;Document&gt; collection = vars.getObject(&quot;collection&quot;);<br> <br> collection.updateOne(<br>  eq(&quot;_id&quot;, new ObjectId(vars.get(&quot;exampleDocumentId&quot;))),<br>  combine(set(&quot;occupation&quot;, &quot;Project Manager&quot;), set(&quot;adress.city&quot;, &quot;New Codeshire&quot;), currentDate(&quot;lastModified&quot;)));</pre><pre>return &quot;Document with id=&quot; + vars.get(&quot;exampleDocumentId&quot;) + &quot; modified&quot;;<br>}<br>catch (Exception e) {<br> SampleResult.setSuccessful(false);<br> SampleResult.setResponseCode(&quot;500&quot;);<br> SampleResult.setResponseMessage(&quot;Exception: &quot; + e);<br>}</pre><p><strong><em>Code for MongoDB_Delete:</em></strong></p><pre>import com.mongodb.client.MongoCollection;<br>import static com.mongodb.client.model.Filters.*;<br>import org.bson.Document;</pre><pre>try {<br> MongoCollection&lt;Document&gt; collection = vars.getObject(&quot;collection&quot;); <br> collection.deleteOne(eq(&quot;occupation&quot;, &quot;Project Manager&quot;));</pre><pre>return &quot;Document deleted&quot;;<br>}<br>catch (Exception e) {<br> SampleResult.setSuccessful(false);<br> SampleResult.setResponseCode(&quot;500&quot;);<br> SampleResult.setResponseMessage(&quot;Exception: &quot; + e);<br>}</pre><h4>Add MongoDB configuration</h4><p>Click on Test Plan &amp; under <em>User-Defined Variables </em>specify the necessary variables:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rv6MOgx5Uru4DZ91KBxvpg.png" /></figure><p>Click on Thread Group under Thread Properties update as per your requirement. For more info refer to <a href="https://jmeter.apache.org/usermanual/test_plan.html">https://jmeter.apache.org/usermanual/test_plan.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*S_Eub914T5s-OHOv1cEb6g.png" /></figure><p>Run the script and see the results in the listener:</p><p>You can also run via command line, let&#39;s say you save the test plan as MongoDB_CRUD.jmx file</p><pre>$ ./apache-jmeter-5.3/bin/jmeter -n -t <strong>MongoDB_CRUD.jmx</strong> -l output.jtl -e -o local_out/</pre><p>Open output folder</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iErwn4_UEO96HMuu0g5Pbw.png" /></figure><h4>That’s it !!!!!!!!!!!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*0t3I-XgDcpEGV45V.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f98382844fbc" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/mongodb-performance-testing-using-jmeter-f98382844fbc">MongoDB Performance Testing Using JMeter</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load CSV Data Into Elasticsearch]]></title>
            <link>https://blog.nonstopio.com/load-csv-data-into-elasticsearch-56e7fb02a232?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e7fb02a232</guid>
            <category><![CDATA[bulk-import]]></category>
            <category><![CDATA[csv]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:35:01 GMT</pubDate>
            <atom:updated>2021-07-06T10:26:54.630Z</atom:updated>
            <content:encoded><![CDATA[<h4>Import CSV Into AWS Elasticsearch Service Using Bulk import or Logstatsh</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDCP9mWw1F4tgeybapyGGA.png" /></figure><p>Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.</p><p><a href="https://aws.amazon.com/elasticsearch-service/">Amazon Elasticsearch Service</a> is a fully managed service that makes it easy for you to deploy, secure, and run Elasticsearch cost-effectively at scale.</p><h4>Key Concepts</h4><p>The key concepts of Elasticsearch are as follows −</p><p><strong>Node: </strong>It refers to a single running instance of Elasticsearch. A single physical and virtual server accommodates multiple nodes depending upon the capabilities of their physical resources like RAM, storage, and processing power.</p><p><strong>Cluster: </strong>It is a collection of one or more nodes. A cluster provides collective indexing and search capabilities across all the nodes for the entire data.</p><p><strong>Index: </strong>It is a collection of different types of documents and their properties. The index also uses the concept of shards to improve performance. For example, a set of documents contains data of a social networking application.</p><p><strong>Document: </strong>It is a collection of fields in a specific manner defined in JSON format. Every document belongs to a type and resides inside an index. Every document is associated with a unique identifier called the UID.</p><p><strong>Shard: </strong>Indexes are horizontally subdivided into shards. This means each shard contains all the properties of a document but contains fewer JSON objects than an index. The horizontal separation makes the shard an independent node, which can be store in any node. The primary shard is the original horizontal part of an index and then these primary shards are replicated into replica shards.</p><p><strong>Replicas: </strong>Elasticsearch allows a user to create replicas of their indexes and shards. Replication not only helps in increasing the availability of data in case of failure but also improves the performance of searching by carrying out a parallel search operation in these replicas.</p><p>There are multiple ways to load CSV data into Elasticsearch Cluster. I want to discuss two cases depends on the size of a file</p><ol><li>If file size is of few KB’s (kilobytes)</li><li>If file size in MBs or in GB</li></ol><p>Prerequisites:</p><p><strong>a) Create Elasticsearch</strong></p><p>There are only a few basic steps to getting an Amazon Elasticsearch Service domain up and running:</p><p>Define your domain =&gt; Configure your cluster =&gt; Set up access =&gt; Review</p><p><strong>refer: </strong><a href="https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html">https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-createupdatedomains.html</a></p><p>You can also <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html">install</a> Elasticsearch locally or host it on any virtual machine</p><p>Let&#39;s assume I have an Ubuntu machine locally</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Jw6etRmEJ0xemPkXyVbiLw.jpeg" /></figure><h3>Case I</h3><p><strong>CSV file size &lt; few KB’s</strong></p><p>Elasticsearch expects some specific format of JSON file as input. To convert the CSV file into the required format we need to run the below script.</p><p>Install python3, Run below python script &amp; Run curl command</p><p>Note: Provide file path of <strong>input_csv_path &amp; input_json_path.</strong></p><pre><strong>csv_to_es_json.py</strong></pre><pre>import csv <br>import json <br>import collections<br>orderedDict = collections.OrderedDict()<br>from collections import OrderedDict</pre><pre>def csv_to_json(csvFilePath, jsonFilePath):<br>    jsonArray = []<br>    x = OrderedDict([(&#39;index&#39;, {})])      <br>    jsonString = json.dumps(x)  <br>    with open(csvFilePath, encoding=&#39;utf-8&#39;) as csvf: <br>        with open(jsonFilePath, &#39;w&#39;, encoding=&#39;utf-8&#39;) as jsonf:<br>            csvReader = csv.DictReader(csvf) <br>            for row in csvReader: <br>                jsonf.write(jsonString)<br>                jsonf.write(&quot;\n&quot;)<br>                y = json.dumps(row)<br>                jsonf.write(y)<br>                jsonf.write(&quot;\n&quot;)<br>          <br>csvFilePath = r&#39;<strong>&lt;input_csv_path&gt;</strong>&#39;<br>jsonFilePath = r&#39;<strong>&lt;input_json_path&gt;</strong>&#39;<br>csv_to_json(csvFilePath, jsonFilePath)</pre><p><strong>Output:</strong></p><pre><strong>## input.csv<br></strong>Title1,Title2,Title3<br>one,two,three<br>example1,example2,example3</pre><pre><strong>## output json<br></strong>{ &quot;index&quot;: {} }<br>{&quot;Title1&quot;: &quot;one&quot;, &quot;Title2&quot;: &quot;two&quot;, &quot;Title3&quot;: &quot;three&quot;} <br>{ &quot;index&quot;: {} }<br>{&quot;Title1&quot;: &quot;example1&quot;, &quot;Title2&quot;: &quot;example2&quot;, &quot;Title3&quot;: &quot;example3&quot;}</pre><p><em>Note: Provide </em><strong><em>ELASTIC_PASSWORD, DOMAIN_URL, INDEX_NAME DOC_TYPE, input_json_path</em></strong><em> these variables values in following command and run.</em></p><p><em>$ curl -u admin:</em><strong><em>&lt;ELASTIC_PASSWORD&gt;</em></strong><em> -H ‘Content-Type: application/json’ -XPOST ‘</em><strong><em>&lt;DOMAIN_URL&gt;</em></strong><em>/</em><strong><em>&lt;INDEX_NAME&gt;</em></strong><em>/&lt;</em><strong><em>DOC_TYPE&gt;</em></strong><em>/_bulk?pretty’ — data-binary @</em><strong><em>&lt;input_json_path&gt;`</em></strong></p><p>eg.</p><p>curl -u admin:<strong>my_elastic_pass</strong> -H ‘Content-Type: application/json’ -XPOST ‘<a href="https://my-search-hieswgdbdgi8907fazkabdjknh.ap-south-1.es.amazonaws.com/">https://my-search-hieswgdbdgi8907fazkabdjknh.ap-south-1.es.amazonaws.com</a>/myindex/_doc/_bulk?pretty&#39; — data-binary @/path/output.json</p><h3>Case II</h3><p><strong>CSV file size in MBs or GB</strong></p><p>Create one VM or EC2 (at least 2 vCPU 4 GB RAM) of ubuntu 18..04</p><p><strong>Install Logstash</strong></p><blockquote>sudo apt update<br>wget -qO - <a href="https://artifacts.elastic.co/GPG-KEY-elasticsearch">https://artifacts.elastic.co/GPG-KEY-elasticsearch</a> | sudo apt-key add -<br>sudo apt-get install apt-transport-https<br>echo &quot;deb <a href="https://artifacts.elastic.co/packages/7.x/apt">https://artifacts.elastic.co/packages/7.x/apt</a> stable main&quot; | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list<br>sudo apt-get update &amp;&amp; sudo apt-get install logstash<br> sudo apt-get install default-jdk</blockquote><p>Create configuration file at location <strong><em>/etc/logstash/conf.d/import_csv.conf</em></strong></p><pre>input {<br>    file {<br>        path =&gt; &quot;<strong><em>input_json_path</em></strong>&quot;<br>        start_position =&gt; beginning<br>    }<br>}<br>filter {<br>    csv {<br>        columns =&gt; [&quot;Title1&quot;,&quot;Title2&quot;,&quot;Title3&quot;]<br>        separator =&gt; &quot;,&quot;<br>        }<br>}<br>output {<br>elasticsearch {<br>  hosts =&gt; [&quot;<strong><em>DOMAIN_URL</em></strong>:443&quot;]<br>  index =&gt; &quot;<strong><em>INDEX_NAME</em></strong>&quot;<br>  document_type =&gt; &quot;<strong><em>DOC_TYPE</em></strong>&quot;<br>  ilm_enabled =&gt; false<br>  ssl =&gt; true<br>  user =&gt; &quot;admin&quot;<br>  password =&gt; &quot;<strong><em>ELASTIC_PASSWORD</em></strong>&quot;<br>}<br>}</pre><p>Restart Logstash service</p><pre><strong>sudo systemctl restart logstash.service</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/0*TXOpkW2FM8CjgCCD.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e7fb02a232" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/load-csv-data-into-elasticsearch-56e7fb02a232">Load CSV Data Into Elasticsearch</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying a Node.js app using Ansible]]></title>
            <link>https://blog.nonstopio.com/deploying-a-node-js-app-using-ansible-cfe7dfeddcac?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfe7dfeddcac</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 05 Jul 2021 07:28:53 GMT</pubDate>
            <atom:updated>2021-07-06T11:36:19.058Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*AiJucVlnH50phBm41lGgrQ.png" /></figure><p>Hello Techies,</p><p><strong>Node.js</strong> is a platform built on <a href="https://code.google.com/p/v8/">Chrome’s JavaScript runtime</a> for easily building fast and scalable network applications.</p><p><a href="https://www.ansible.com/"><strong>Ansible</strong></a> is a software tool that provides simple but powerful automation for cross-platform computer support.</p><p><strong>Playbooks</strong> are one of the core features of Ansible and tell Ansible what to execute. They are like a to-do list for Ansible that contains a list of tasks.<br><strong>Module</strong>: Basically, a module is a command or set of similar Ansible commands meant to be executed on the client-side<br><strong>Task</strong>: A task is a section that consists of a single procedure to be completed<br><strong>Inventory</strong>: File containing data about the ansible client servers.</p><p>The intention of this blog is not to tell you how to deploy node applications on servers. Might be you know better than me. I am trying to share my experience that might help you to get familiarity with some amazing ansible modules like git, npm, file, etc. some interesting keywords like with_items/loop, with_together.</p><p><strong>Launch EC2 Instance on AWS (optional). You could follow step 1</strong></p><p><a href="https://medium.com/nonstopio/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88">Create and Integrate S3 and RDS with EC2 in AWS</a></p><p>Create inventory file</p><pre><strong>## inventory</strong></pre><pre>[webserver]<br>web ansible_host=1.2.3.4 ansible_connection=ssh ansible_ssh_user=ubuntu  ansible_ssh_private_key_file=./web-key.pem</pre><p>Let&#39;s start writing ansible-playbook</p><pre><strong>## playbook.yaml</strong></pre><pre>---<br>- hosts: localhost<br>  become: yes<br>  gather_facts: no<br>  vars:<br>      git_repo_url: https://example.org/path/to/hello-world.git<br>      repo_path: /your/local/repo/path<br>      repo_branch: master</pre><p>We need to install some packages like git, curl, wget, nginx, nodejs. You could install any node version you like <a href="https://github.com/nodesource/distributions/blob/master/README.md"><strong>refer</strong></a><strong>. </strong>I’m choosing node 14.x</p><pre>tasks: <br>  - name: Run the equivalent of &quot;apt-get update&quot; as a separate step<br>    apt:<br>     update_cache: yes</pre><pre>  - name: Ansible shell module multiple commands<br>    shell: &#39;curl -fsSL <a href="https://deb.nodesource.com/setup_14.x">https://deb.nodesource.com/setup_14.x</a> | sudo -E bash -&#39;</pre><pre>  - name: Install reqired packages<br>    apt: name={{ item }} state=present<br>    with_items:<br>      - git<br>      - curl<br>      - wget<br>      - nginx<br>      - nodejs</pre><p>Clone your node app repository on the server. It’s optional you could just copy your build to EC2.</p><p>Make sure your EC2 instance may have the right permission to clone the repository</p><pre>  - name: Clone a private repository<br>    git:<br>     repo: &quot;{{ git_repo_url }}&quot;<br>     dest: &quot;{{ repo_path }}&quot;<br>     version: &quot;{{ repo_branch }}&quot;<br>     accept_hostkey: yes</pre><p>Install the dependencies into the node_modules folder. &amp; build</p><pre>  - name: Install packages based on package.json using the npm<br>    npm:<br>      path: &quot;{{ repo_path }}&quot;<br>      state: present</pre><pre>  - name: Build app<br>    command: npm run build<br>    args:<br>      chdir: &quot;{{ repo_path }}&quot;</pre><p>Copy build folder to /var/www/</p><pre>  - name: Copy folder to /var/www<br>    copy:<br>     src: &quot;{{ repo_path }}/build&quot;<br>     dest: /var/www<br>     <strong>remote_src: no</strong></pre><blockquote><strong>Note</strong>: Use remote_src: yes. If you are not fan of git clone on EC2 you could locally build your node app and copy to remote server</blockquote><p>create nginx configuration file</p><pre><strong>## example.conf<br></strong>server {<br>    listen    443 ssl;<br>    server_name example.com;<br>    ssl_certificate     /etc/nginx/SSL/example.crt;<br>    ssl_certificate_key /etc/nginx/SSL/example.key;<br>    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;<br>    ssl_ciphers         HIGH:!aNULL:!MD5;</pre><pre>location / {<br>   root /var/www/build;<br>   index index.html;<br>   try_files $uri /index.html;<br>  }<br>}</pre><p>Configure the Nginx server.</p><pre>  - name: Recursively change ownership of a directory<br>    file:<br>        path: /etc/nginx/SSL<br>        state: directory<br>        recurse: yes</pre><pre>  - name: Copy OpenSSH keypair with permissions<br>    copy:<br>        owner: root<br>        group: root<br>        src: &quot;{{ item.0 }}&quot;<br>        dest: &quot;{{ item.1 }}&quot;<br>        mode: &quot;0400&quot;<br>      with_together:<br>        - [&#39;local_path/example.crt&#39;, &#39;local_path/example.key&#39;, &#39;example.conf&#39;]<br>        - [&#39;/etc/nginx/SSL/example.crt&#39;, &#39;/etc/nginx/SSL/example.key&#39;, &#39;/etc/nginx/sites-available/example.conf&#39;]</pre><pre>  - name: Remove files<br>    file:<br>     path: &quot;{{ item }}&quot;<br>     state: absent<br>    with_items:<br>      - /etc/nginx/sites-enabled/default<br>      - /etc/nginx/sites-available/default</pre><pre>  - name: Create a symbolic link<br>    file:<br>      src: /etc/nginx/sites-available/example.conf<br>      dest: /etc/nginx/sites-enabled/example.conf<br>      state: link</pre><pre>  - name: Restart nginx service<br>    service:<br>      name: nginx<br>      state: restarted</pre><p>Finally, run this playbook using the command</p><pre><strong># ansible-playbook -i /path/inventory  /path/playbook.yaml</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*o2kDnQz8ExAByL7z.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfe7dfeddcac" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/deploying-a-node-js-app-using-ansible-cfe7dfeddcac">Deploying a Node.js app using Ansible</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Jenkins CICD Stages]]></title>
            <link>https://blog.nonstopio.com/jenkins-cicd-stages-ff4aa4c4dd57?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff4aa4c4dd57</guid>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[jenkins-stage]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[jenkins]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 05:11:26 GMT</pubDate>
            <atom:updated>2021-07-01T04:01:55.546Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*K4q6gBZiXCfA0pDb" /></figure><p>Hello techies,</p><p>The intention behind witting this blog is to share my experience what I think should be steps you should consider before creating any CICD pipeline. In this blog, I am not explaining in-depth steps, instead, I’ll provide reference links that I used and applied in my journey. <strong>Learn from the best.</strong></p><h3>Jenkins</h3><p><strong>Build great things at any scale</strong></p><p>The leading open-source automation server, Jenkins provides hundreds of plugins to support building, deploying, and automating any project.</p><h4><strong>1. Code Checkout on specific branch commit</strong></h4><p>Either you could use Generic Webhook Trigger or CrumbIsusser.</p><ul><li><a href="https://plugins.jenkins.io/generic-webhook-trigger/">Generic Webhook Trigger</a></li><li><a href="https://cloudbooklet.medium.com/jenkins-how-to-trigger-build-if-only-a-push-is-made-to-a-specific-branch-on-bitbucket-4e615c691630">Jenkins — How to trigger build if only a push is made to a specific branch on Bitbucket</a></li><li><a href="http://www.inanzzz.com/index.php/post/jnrg/running-jenkins-build-via-command-line">Running jenkins jobs via command line</a></li></ul><h4><strong>2. Code Build</strong></h4><p>Compiles an Angular app into an output directory named dist/ at the given output path. Must be executed from within a workspace directory.</p><pre>ng <strong>build</strong> &lt;project&gt; [options]<br>OR <br>npm run build</pre><h4>3. Code Unit Testing</h4><p><strong>refer: </strong><a href="https://livebook.manning.com/book/testing-angular-applications/chapter-11/42">https://livebook.manning.com/book/testing-angular-applications/chapter-11/32</a></p><h4>4. SonarQube Integration with Jenkins</h4><p>Install SonarQube on the same Jenkins server.</p><p><a href="https://www.digitalocean.com/community/tutorials/how-to-ensure-code-quality-with-sonarqube-on-ubuntu-18-04">How To Ensure Code Quality with SonarQube on Ubuntu 18.04 | DigitalOcean</a></p><p>Login to Jenkins GUI console and install “ SonarQube scanner” plugin</p><ul><li>Manage Jenkins &gt; Manage Plugins &gt; Available &gt; typeSonarQube scanner</li></ul><p>Configure SonarQube scanner home path</p><ul><li>Manage Jenkins &gt; Global Tool Configuration &gt; SonarQube Scanner</li><li>Name : sonar_scanner</li><li>SONAR_RUNNER_HOME : /opt/sonar_scanner</li></ul><p>Configure SonarQube server name and authentication token</p><ul><li>Manage Jenkins &gt; Configure Systems &gt; SonarQube Servers</li><li>Name : SonarQube</li><li>ServerURL : http://&lt;Sonarqube_server&gt;:9000/sonar</li></ul><h4><strong>5. Code Deploy</strong></h4><p>To deploy code on the remote server you need ssh connection</p><p>Either you write your own shell scripts or use Ansible playbooks to deploy your build.</p><ul><li><a href="https://plugins.jenkins.io/ssh/">SSH</a></li><li><a href="https://plugins.jenkins.io/ansible/">Ansible</a></li></ul><p>To begin using Ansible as a means of managing your server infrastructure, you need to install the Ansible software on the Jenkins server that will serve as the Ansible control node.</p><p>From your control node, run the following command to include the official project’s PPA (personal package archive) in your system’s list of sources:</p><pre>sudo apt-add-repository ppa:ansible/ansible</pre><p>Press ENTER when prompted to accept the PPA addition.</p><p>Next, refresh your system’s package index so that it is aware of the packages available in the newly included PPA:</p><ul><li>Following this update, you can install the Ansible software with:</li></ul><pre>sudo apt update<br>sudo apt install ansible</pre><p>Step 6: Run Selenium test cases for UI testing</p><p><a href="https://medium.com/nonstopio/run-selenium-tests-in-bitbucket-pipeline-64e0dcdd1a1f">Run selenium tests in Bitbucket pipeline</a></p><p>Refer above story and install the required package like Apache Maven, java chrome &amp; chrome webdriver, xvfb</p><p>Now, you just need to run the <strong>maven</strong> <strong>test </strong>command in your Jenkins pipeline</p><h4><strong>6. Load/Performance Testing using JMeter</strong></h4><p>Login to Jenkins GUI console and install “ SonarQube scanner” plugin</p><ul><li>Manage Jenkins &gt; Manage Plugins &gt; Available &gt; search JMeter you would find plugin name as Performance and install &amp; then restart Jenkins</li></ul><p>Download JMeter binary (tar.tz file) on Jenkins server from the official page <a href="https://jmeter.apache.org/download_jmeter.cgi">here</a></p><pre>$ mkdir /var/lib/jenkins/jmeter<br>$ cd /var/lib/jenkins/jmeter<br>$ wget <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz</a><br>$ tar -xvzf <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">apache-jmeter-5.4.1.tgz</a><br>$ ls <a href="https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.4.1.tgz">apache-jmeter-5.4.1</a>/<br>$ chown -R jenkins:jenkins /var/lib/jenkins/jmeter</pre><p>Add following command in Jenkins pipeline</p><pre>$ /var/lib/jenkins/jmeter/apache-jmeter-5.3/bin/jmeter -jjmeter.save.saveservice.output_format=xml -n -t /var/lib/jenkins/jmeter/<strong>TestPlan</strong>.<strong>jmx</strong>  -l /var/lib/jenkins/jmeter/<strong>output.jtl</strong></pre><h4><strong>7. Slack Webhook Integration with Jenkins</strong></h4><p>Login to Jenkins GUI console and install “ Slack Notification” plugin</p><ul><li>Manage Jenkins &gt; Manage Plugins &gt; Available &gt; search for slack &gt;Check the box next to install</li></ul><p>Get a Slack account: <a href="https://slack.com/">https://slack.com/</a></p><p>Configure the Jenkins integration: <a href="https://my.slack.com/services/new/jenkins-ci">https://my.slack.com/services/new/jenkins-ci</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vd__MfZwXSduXKM9WiKczQ.png" /></figure><p>Select channel or OR create a new channel</p><p>Click on Add Jenkins CI integration button</p><p>Expand setup instructions</p><p>Add post-build Actions in freestyle job</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-f_-IM4gpysRRo3WvKvGg.png" /></figure><p>OR you could use the following script in the pipeline</p><pre>post {<br>    success {<br>      slackSend (color: &#39;#00FF00&#39;, message: &quot;SUCCESSFUL: pipeline &#39;${env.JOB_NAME} [${env.BUILD_NUMBER}]&#39; (${env.BUILD_URL})&quot;)<br>     }</pre><pre>failure {<br>      slackSend (color: &#39;#FF0000&#39;, message: &quot;FAILED: pipeline &amp; Triggering ROLLOUT pipeline &#39;${env.JOB_NAME} [${env.BUILD_NUMBER}]&#39; (${env.BUILD_URL})&quot;)<br>     }</pre><pre>aborted {<br>      slackSend (color: &#39;#808080&#39;, message: &quot;ABORTED: pipeline &#39;${env.JOB_NAME} [${env.BUILD_NUMBER}]&#39; (${env.BUILD_URL})&quot;)</pre><pre>     }<br>}</pre><ul><li><a href="https://plugins.jenkins.io/slack/">Slack Notification</a></li><li><a href="https://sagar-gadewar.medium.com/create-slack-webhooks-854c85c6903b">Create slack webhooks</a></li></ul><p>I hope you find this article helpful. See ya…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*0Rmie0Cx1q6x_6FCjBLUlQ.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff4aa4c4dd57" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/jenkins-cicd-stages-ff4aa4c4dd57">Jenkins CICD Stages</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrate Ansible with Bitbucket Pipeline]]></title>
            <link>https://blog.nonstopio.com/integrate-ansible-with-bitbucket-pipeline-4ae9821ce6fd?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ae9821ce6fd</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[bitbucket]]></category>
            <category><![CDATA[bitbucket-pipelines]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 05:03:08 GMT</pubDate>
            <atom:updated>2021-07-01T04:04:51.162Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*sV8pi5txXJiFOJfJ.png" /></figure><p><a href="https://bitbucket.org/product/features/pipelines">Bitbucket Pipelines</a> is an integrated CI/CD service, built into Bitbucket. It allows you to automatically build, test, and even deploy your code based on a configuration file in your repository.</p><p>BitBucket Pipelines is a relatively similar product to <em>Bamboo, Travis, and Jenkins. It is built into the BitBucket system, (though you pay for more build minutes).</em></p><p>Before starting first of all I would like to explain why I need this use case. Nowadays the bitbucket pipeline is getting popular for CICD. We already have a standard plan of bitbucket repository which includes default get 200 minutes per month for the pipeline. So we have basically two options to set up the CICD pipeline from scratch either we can use Jenkins or we can use bitbucket pipeline credits. for Jenkins, we have to maintain one dedicated server which I think is not cost-effective for small steps such as deployment, unit testing, and web testing.</p><p><strong>Problem statement</strong>: Let’s say we have one EC2 (of OS <strong>Ubuntu 18.04) </strong>in AWS. I want to configure the CICD pipeline for that. Whenever a developer commits to the <strong>development</strong> <strong>branch</strong> server needs to update.</p><p>Lets’s have an example of ansible inventory and playbook file. We need to add this file to our code.</p><pre><strong>inventory<br></strong>[webserver]<br>dev ansible_host=2.7.31.21 ansible_connection=ssh ansible_ssh_user=ubuntu</pre><pre><strong>playbook.yml<br></strong>---<br>- hosts: webserver<br>  gather_facts: no<br>  tasks:<br><strong>  -</strong> name<strong>:</strong> Git Pull<br>    ansible.builtin.git<strong>:</strong><br>      repo<strong>:</strong> &#39;https://foosball.example.org/path/to/repo.git&#39;<br>      dest<strong>:</strong> /mnt<br>      version<strong>:</strong> development</pre><pre>... omitted...</pre><pre>  - name: Restart service nginx<br>    service:<br>      name: nginx<br>      state: restarted<br>...</pre><h4>Configuring Bitbucket</h4><p>After your repository is all set and done, all that’s left to do is to enable Pipelines for your repo through the settings. Scroll down to the PIPELINES section and click on <strong>Settings</strong>. You’ll see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R3KNKq04Pe9NOXlE.png" /></figure><p>Enable Pipelines and follow the Configuration button</p><p>Check the toggle and you’ll get the option to configure the bitbucket-pipelines.yml file. This file will tell Bitbucket what commands to run after the code is pushed to your repository. Clicking on the “<em>Configure bitbucket-pipelines.yml</em>” and paste these lines.</p><pre><strong>bitbucket-pipeline.yml</strong></pre><pre>image: python:3.7<br>options:<br> max-time: 10<br>pipelines:<br>  branches:<br>    development:<br>      - step:<br>          name: Deploy to DEV server<br>          caches:<br>            - pip<br>          script:<br>            - pip install ansible<br>            - ansible-playbook -i &lt;inventory_file_path&gt;  &lt;ansible_playbook_path&gt;</pre><p><strong>max-time:</strong> [value could be 0- — 120 ] maximum amount of minutes pipeline could execute. If it crosses this max-time, then the pipeline fails automatically.</p><p><strong>branches: </strong>specific branch on which whenever user commit pipeline get triggered.</p><p><strong>caches</strong>: Re-downloading dependencies from the internet for each step of a build can take a lot of time. Using a cache they are downloaded once to our servers and then locally loaded into the build each time.</p><p><strong>Note: </strong>In order to run successfully pipeline bitbucket-pipeline.yml, inventory, playbook.yml needs to present in the specific branch you mentioned (in our case development branch</p><p>Now is the question of how to connect bitbucket to the EC2 instance. There are multiple ways.</p><ol><li>Use <strong>ansible_ssh_private_key_file</strong> in inventory file.</li><li>Use <strong>ansible_ssh_pass</strong>=***** in inventory file.</li><li>Passwordless connection between bitbucket and EC2.</li></ol><h4><strong>The passwordless connection between bitbucket repo and EC2.</strong></h4><p><strong>To generate a new SSH key pair</strong></p><ol><li>In <strong>Repository settings</strong>, go to <strong>SSH keys</strong> under ‘Pipelines’.</li><li>Click <strong>Generate keys</strong> to create a new SSH key pair.</li><li>Copy public key, and save it at a secure location</li><li>SSH to remote server and paste these copied public keys from step 3 at <strong>~/.ssh/authorized_keys</strong> file</li><li>Add public IP in known hosts address &amp; click on fetch</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCu0DhZGOO8ubhy76qe52g.png" /></figure><p>Note: Use could also use parameter <strong>host_key_checking = False </strong>in <strong>ansible.cfg</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/0*zZptuAPNrnyf_K7y.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ae9821ce6fd" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/integrate-ansible-with-bitbucket-pipeline-4ae9821ce6fd">Integrate Ansible with Bitbucket Pipeline</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Run selenium tests in Bitbucket pipeline]]></title>
            <link>https://blog.nonstopio.com/run-selenium-tests-in-bitbucket-pipeline-64e0dcdd1a1f?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/64e0dcdd1a1f</guid>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[bitbucket-pipelines]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[headless-chrome]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Tue, 05 Jan 2021 06:04:52 GMT</pubDate>
            <atom:updated>2021-01-12T05:11:25.323Z</atom:updated>
            <content:encoded><![CDATA[<h3>Run Selenium Tests In Bitbucket Pipeline</h3><h4>using docker image in chrome headless mode</h4><p>Hello viewers,</p><p><a href="https://bitbucket.org/product/features/pipelines">Bitbucket Pipelines</a> is an integrated CI/CD service, built into Bitbucket. It allows you to automatically build, test, and even deploy your code based on a configuration file in your repository.</p><p>BitBucket Pipelines is a relatively similar product to <em>Bamboo, Travis, and Jenkins. It is built into the BitBucket system, (though you pay for more build minutes).</em></p><p>Before starting first of all I would like to explain why I need this use case. Nowadays the bitbucket pipeline is getting popular for CICD. We already have a standard plan of bitbucket repository which includes default get 200 minutes per month for the pipeline. So we have basically two options to set up the CICD pipeline from scratch either we can use Jenkins or we can use bitbucket pipeline credits. for Jenkins, we have to maintain one dedicated server which I think is not cost-effective for small steps such as deployment, unit testing and web testing. that&#39;s why selenium came into the picture.</p><h4>Configuring Bitbucket</h4><p>After your repository is all set and done, all that’s left to do is to enable Pipelines for your repo through the settings. Scroll down to the PIPELINES section and click on <strong>Settings</strong>. You’ll see the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iLlaox0EiVdCLAZi.png" /></figure><p>Enable Pipelines and follow the Configuration button</p><p>Check the toggle and you’ll get the option to configure the bitbucket-pipelines.yml file. This file will tell Bitbucket what commands to run after code is pushed to your repository. Clicking on the “<em>Configure bitbucket-pipelines.yml</em>” and paste these lines.</p><p><strong>bitbucket-pipeline.yml</strong></p><pre>pipelines:<br>  custom:<br>   selenium-test:<br>    - step:<br>        script:<br>          - docker version<br>          - docker build . -t sagargadewar/selenium-test-local<br>          - docker run -w /usr/src sagargadewar/selenium-test-local mvn clean verify<br>        services:<br>          - docker<br>        caches:<br>          - docker</pre><p>Create file name as Dockerfile<strong> </strong>and place beside the bitbucket-pipeline.yml file</p><p><strong>Dockerfile</strong></p><pre>FROM sagargadewar/selenium-test<br>WORKDIR /usr/src<br>COPY . .</pre><p>You could see your repository looks like following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*5bTLmtWGH78MIb-ERK4uhQ.png" /></figure><p>There are a few changes you need in your code before pushing in bitbucket.</p><p><strong>Set chrome driver path as /usr/bin/chromedriver</strong></p><pre>System.setProperty(“webdriver.chrome.driver”,”/usr/bin/chromedriver”);</pre><p><strong>Configure </strong><a href="https://www.selenium.dev/"><strong>Selenium</strong></a><strong> to launch Chrome in headless mode:</strong></p><pre>ChromeOptions options = new ChromeOptions().setHeadless(true);<br>WebDriver driver = new ChromeDriver(options);</pre><pre>                 <strong>OR</strong></pre><pre>ChromeOptions chromeOptions = new ChromeOptions();<br>chromeOptions.addArguments(&quot;headless&quot;);   <br>driver = new ChromeDriver(chromeOptions);</pre><p><strong>[Optional]</strong></p><p>If you don’t want to use my <strong>sagargadewar/selenium-test </strong>you can create your own image for that following the below steps.</p><p><strong><em>Step 1: Install </em></strong><a href="https://docs.docker.com/engine/install/"><strong><em>Docker Engine</em></strong></a></p><p><strong><em>Step 2: Create a file name as Dockerfile</em></strong></p><pre>FROM maven:3.6.3-jdk-8</pre><pre># Google Chrome</pre><pre>ARG CHROME_VERSION=87.0.4280.88–1<br>RUN wget -q -O — <a href="https://dl-ssl.google.com/linux/linux_signing_key.pub">https://dl-ssl.google.com/linux/linux_signing_key.pub</a> | apt-key add — \<br> &amp;&amp; echo “deb <a href="http://dl.google.com/linux/chrome/deb/">http://dl.google.com/linux/chrome/deb/</a> stable main” &gt;&gt; /etc/apt/sources.list.d/google-chrome.list \<br> &amp;&amp; apt-get update -qqy \<br> &amp;&amp; apt-get -qqy install google-chrome-stable=$CHROME_VERSION \<br> &amp;&amp; rm /etc/apt/sources.list.d/google-chrome.list \<br> &amp;&amp; rm -rf /var/lib/apt/lists/* /var/cache/apt/* \<br> &amp;&amp; sed -i ‘s/”$HERE\/chrome”/”$HERE\/chrome” — no-sandbox/g’ /opt/google/chrome/google-chrome</pre><pre># ChromeDriver</pre><pre>ARG CHROME_DRIVER_VERSION=87.0.4280.88<br>RUN wget -q -O /tmp/chromedriver.zip <a href="https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip">https://chromedriver.storage.googleapis.com/$CHROME_DRIVER_VERSION/chromedriver_linux64.zip</a> \<br> &amp;&amp; unzip /tmp/chromedriver.zip -d /opt \<br> &amp;&amp; rm /tmp/chromedriver.zip \<br> &amp;&amp; mv /opt/chromedriver /opt/chromedriver-$CHROME_DRIVER_VERSION \<br> &amp;&amp; chmod 755 /opt/chromedriver-$CHROME_DRIVER_VERSION \<br> &amp;&amp; ln -s /opt/chromedriver-$CHROME_DRIVER_VERSION /usr/bin/chromedriver</pre><p><strong><em>Step 3: Build and push a container image to Docker Hub from your computer</em></strong></p><p><strong><em>Refer: </em></strong><a href="https://docs.docker.com/docker-hub/"><strong><em>https://docs.docker.com/docker-hub/</em></strong></a></p><p><strong><em>Step 4: </em></strong><em>Replace sagargadewar/selenium-test with your image name in bitbucket-pipeline.yml and Dockerfile files</em></p><blockquote>Note: Get the appropriate chrome driver version, chrome stable version and update Dockerfile if needed</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*CBh_cbjVRBycpEYg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64e0dcdd1a1f" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/run-selenium-tests-in-bitbucket-pipeline-64e0dcdd1a1f">Run selenium tests in Bitbucket pipeline</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up Terraform Development Environment In AWS]]></title>
            <link>https://blog.nonstopio.com/setting-up-terraform-development-environment-in-aws-25f8f6d0cf0d?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/25f8f6d0cf0d</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Sat, 08 Aug 2020 10:51:14 GMT</pubDate>
            <atom:updated>2020-08-08T11:14:27.989Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create and Integrate S3 and RDS with EC2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*6N52-vm-0qRbk9pb.png" /></figure><p>Hello viewers! It’s time to automate your infrastructure.</p><p>I hope you have seen my earlier <a href="https://medium.com/nonstopio/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88">blog</a> in which I have manually created all services in AWS (ED2-RDS-S3). However, what if we want to replicate the environment in another AWS account? Or in the worst-case what if the setup crashes and you have to recreate it by following all manual steps again?. <br>It’s risky! So I thought how we automate this process and I quest to automate the infrastructure setup lead me to discover the concept of “Infrastructure as a Code”</p><h4>What is Terraform?</h4><p>Terraform is an open-source “Infrastructure as Code” tool, created by HashiCorp.</p><p>A <em>declarative</em> coding tool, Terraform enables developers to use a high-level configuration language called HCL (HashiCorp Configuration Language) to describe the desired “end-state” cloud or on-premises infrastructure for running an application. It then generates a plan for reaching that end-state and executes the plan to provision the infrastructure.</p><blockquote><strong>What is the development environment?</strong></blockquote><blockquote>The development environment helps the developers to develop the application or product using a set of processes and programming tools.</blockquote><blockquote>A development environment provides developers an interface and convenient view of the development process which includes writing code, testing the same, and packaging the build so that it can be deployed.</blockquote><h4><strong>Prerequisite</strong>:</h4><ol><li>AWS account</li><li>IAM user whose having programmatic access (access key &amp; secret access key)</li></ol><h4>Local setup</h4><p><strong>Step 1: Installing Terraform</strong></p><p>Refer this <a href="https://www.terraform.io/downloads.html">URL</a> and download the latest version of Terraform (0.12.29). Kindly download the proper package for your operating system and architecture.</p><p>Terraform is distributed as a single binary. Install Terraform by unzipping it and moving it to a directory included in your system’s <a href="https://superuser.com/questions/284342/what-are-path-and-other-environment-variables-and-how-can-i-set-or-use-them">PATH</a>.</p><p>Verify terraform version using $ <strong>terraform — version </strong>command.</p><p><strong>Step 2: Install AWS CLI [Optional]</strong></p><blockquote>The AWS Command Line Interface (AWS CLI) is an open source tool that enables you to interact with AWS services using commands in your command-line shell.</blockquote><blockquote>Refer URLs: <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html">https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html</a></blockquote><p><strong>Step 3 SSH key Configure</strong></p><p>Create public and private keys by <strong>ssh-keygen</strong> command</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Etc0-BhvnKIhXrszftSMrA.png" /><figcaption>Expected Infrastructure to be setup</figcaption></figure><h4>Creating our Project</h4><p><strong>Define Variables</strong></p><p>Create a terraform variable file with <strong>.tfvars</strong> extension eg. terraform.tfvars</p><pre>aws_access_key         = &quot;&lt;ACCESS_KEY_VALUE&gt;&quot;<br>aws_secret_key         = &quot;&lt;SECRET_ACCESS_KEY_VALUE&gt;&quot;<br>private_key_path       = &quot;&lt;PATH_OF_SSH_PRIVATE_KEY&gt;&quot;<br>bucket_name_prefix     = &quot;global&quot;<br>environment_tag        = &quot;Dev&quot;<br>project_tag            = &quot;&lt;Project_Name&gt;&quot;<br>ec2_instance_type      = &quot;t2.micro&quot;<br>s3_bucket_name         = &quot;&lt;Gobally_Unique_Bucket_name&gt;&quot;<br>allocated_storage      =  10<br>engine                 = &quot;postgres&quot;<br>engine_version         = &quot;9.6.9&quot;<br>db_instance_class      = &quot;db.t2.micro&quot;<br>name                   = &quot;&lt;DB_NAME&gt;&quot;<br>username               = &quot;&lt;USENAME&gt;&quot;<br>password               = &quot;&lt;Complex_PASSWORD&gt;&quot;<br></pre><p><strong>A Terraform project is just a set of files in a directory containing resource definitions</strong>. Those files, which by convention end in <em>.tf</em>, use <a href="https://www.terraform.io/docs/configuration/index.html">Terraform’s configuration language</a> to define the resources we want to create.</p><blockquote><strong>Note:</strong></blockquote><blockquote>Define all the above variables in the main config file(*.tf) in following formt</blockquote><blockquote><strong>variable “&lt;variable_name&gt;” {}</strong></blockquote><blockquote>eg. variable “aws_access_key” {}</blockquote><p><strong>Provider</strong></p><pre>provider &quot;aws&quot; {<br>  access_key = var.aws_access_key<br>  secret_key = var.aws_secret_key<br>  region     = var.region<br>}</pre><p><strong>Security Groups</strong></p><pre><strong>####### EC2 Security Group ############</strong><br>resource &quot;aws_security_group&quot; &quot;ec2-sg&quot; {<br>  name   = &quot;ec2_sg&quot;</pre><pre># SSH access from anywhere<br>  ingress {<br>    from_port   = 22<br>    to_port     = 22<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre># HTTP access from the VPC<br>  ingress {<br>    from_port   = 80<br>    to_port     = 80<br>    protocol    = &quot;tcp&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre># outbound internet access<br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre>}</pre><pre><strong>####### RDS Security Group ############</strong></pre><pre>resource &quot;aws_security_group&quot; &quot;rds-sg&quot; {<br>  name   = &quot;rds_sg&quot;</pre><pre># SSH access from anywhere<br>  ingress {<br>    from_port   = 5432<br>    to_port     = 5432<br>    protocol    = &quot;tcp&quot;<br>    security_groups = [aws_security_group.django-sg.id]<br>  }</pre><pre># outbound internet access<br>  egress {<br>    from_port   = 0<br>    to_port     = 0<br>    protocol    = &quot;-1&quot;<br>    cidr_blocks = [&quot;0.0.0.0/0&quot;]<br>  }</pre><pre>}</pre><p><strong>IAM Role for EC2 to access S3 bucket</strong></p><ul><li><strong><em>IAM Roles</em></strong><em> are used to granting the application access to AWS Services without using permanent credentials.</em></li><li><em>Create an IAM role by copy-paste the content of a below-mentioned link</em></li><li><strong>assume_role_policy<em> </em></strong><em>— (Required) The policy that grants an entity permission to assume the role.</em></li></ul><pre><em>resource &quot;aws_iam_role&quot; &quot;ec2_s3_role&quot; {<br>  name = &quot;ec2_s3_role&quot;</em></pre><pre><em>  assume_role_policy = &lt;&lt;EOF<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Action&quot;: &quot;sts:AssumeRole&quot;,<br>      &quot;Principal&quot;: {<br>        &quot;Service&quot;: &quot;ec2.amazonaws.com&quot;<br>      },<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Sid&quot;: &quot;&quot;<br>    }<br>  ]<br>}<br>EOF<br>}</em></pre><ul><li><em>This is going to create IAM role but we can’t link this role to AWS instance and for that, we need EC2 instance Profile</em></li></ul><pre><em>resource &quot;aws_iam_instance_profile&quot; &quot;ec2_s3_profile&quot; {<br>  name = &quot;ec2_s3_profile&quot;<br>  role = aws_iam_role.ec2_s3_role.name<br>}</em></pre><ul><li><em>Now if we execute the above code, we have Role and Instance Profile but with no permission.</em></li><li><em>Next step is to add IAM Policies which allows EC2 instance to execute specific commands for eg: access to S3 Bucket</em></li><li><em>Adding IAM Policies</em></li><li><em>To give full access to S3 bucket</em></li></ul><pre><em>resource &quot;aws_iam_role_policy&quot; &quot;ec2_s3_policy&quot; {<br>  name = &quot;ec2_s3_policy&quot;<br>  role = aws_iam_role.ec2_s3_role.id</em></pre><pre><em>policy = &lt;&lt;EOF<br>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Action&quot;: [<br>        &quot;s3:*&quot;<br>      ],<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Resource&quot;: </em>[<br>                &quot;arn:aws:s3:::${var.s3_bucket_name}&quot;,<br>                &quot;arn:aws:s3:::${var.s3_bucket_name}/*&quot;<br>            ]<em><br>    }<br>  ]<br>}<br>EOF<br>}</em></pre><p><strong>Create and Integrate S3 and RDS with EC2</strong></p><pre><strong>######## RDS ############</strong></pre><pre>resource &quot;aws_db_instance&quot; &quot;db&quot; {<br>  allocated_storage      = var.allocated_storage<br>  engine                 = var.engine<br>  engine_version         = var.engine_version<br>  instance_class         = var.db_instance_class<br>  name                   = var.db_name<br>  username               = var.username<br>  password               = var.password<br>  vpc_security_group_ids = [aws_security_group.rds-sg.id]<br>  skip_final_snapshot    = true<br>}</pre><pre><strong>####### EC2 ############</strong></pre><pre>resource &quot;aws_key_pair&quot; &quot;ec2-key&quot; {<br>  key_name   = &quot;deployer-key&quot;<br>  public_key = var.public_key<br>}</pre><pre>resource &quot;aws_instance&quot; &quot;django&quot; {<br>  ami                    = var.ami_id<br>  instance_type          = var.ec2_instance_type<br>  vpc_security_group_ids = [aws_security_group.django-sg.id]<br>  key_name               = aws_key_pair.ec2-key.key_name<br><strong>  </strong>iam_instance_profile   = aws_iam_instance_profile.<em>ec2_s3_profile</em>.name</pre><pre>}</pre><pre><strong>####### S3  ############</strong></pre><pre>resource &quot;aws_s3_bucket&quot; &quot;media_bucket&quot; {<br>    bucket        = var.s3_bucket_name<br>    acl           = &quot;private&quot;<br>}</pre><p><em>It’s time to execute code</em></p><p><em>1: This will initialize the Terraform working directory OR it will download plugins for a provider(example: AWS)</em></p><pre><em>terraform init</em></pre><p><em>2: Let you see what terraform will do before making the actual changes</em></p><pre><em>terraform plan</em></pre><p><em>3: To actually create the instance we need to run terraform apply</em></p><pre><em>terraform apply</em></pre><p>Check your AWS Console and you shall see the infrastructure ready.</p><p>I hope this simple blog has proven beneficial to you :-)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25f8f6d0cf0d" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/setting-up-terraform-development-environment-in-aws-25f8f6d0cf0d">Setting Up Terraform Development Environment In AWS</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure HA PostgreSQL Cluster in Active/Passive Mode]]></title>
            <link>https://blog.nonstopio.com/configure-ha-postgresql-cluster-in-active-passive-mode-b9994aca9d1d?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9994aca9d1d</guid>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[patroni]]></category>
            <category><![CDATA[etcd]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Fri, 15 May 2020 16:49:46 GMT</pubDate>
            <atom:updated>2020-05-27T09:34:35.647Z</atom:updated>
            <content:encoded><![CDATA[<h4>In Active/Passive Mode</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/0*rZjw_V5HceAM77K3.png" /></figure><p><strong>PostgreSQL</strong> is a general purpose and object-relational database management system, the most advanced open-source database system. PostgreSQL was developed based on <a href="http://db.cs.berkeley.edu/postgres.html">POSTGRES 4.2</a> at <a href="http://www.cs.berkeley.edu/">Berkeley Computer Science Department</a>, University of California.</p><h4>Patroni</h4><p>There are many ways to run high availability with PostgreSQL; for a list, see the <a href="https://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling">PostgreSQL Documentation</a>.</p><p>Patroni is a template for you to create your own customized, high-availability solution using Python and — for maximum accessibility — a distributed configuration store like <a href="https://zookeeper.apache.org/">ZooKeeper</a>, <a href="https://github.com/coreos/etcd">etcd</a>, <a href="https://github.com/hashicorp/consul">Consul</a> or <a href="https://kubernetes.io/">Kubernetes</a>. Database engineers, DBAs, DevOps engineers, and SREs who are looking to quickly deploy HA PostgreSQL in the datacenter-or anywhere else-will hopefully find it useful.</p><p>We call Patroni a “template” because it is far from being a one-size-fits-all or plug-and-play replication system. It will have its own caveats. Use wisely.</p><p>reference URL: <a href="https://patroni.readthedocs.io/">patroni.readthedocs.io</a>.</p><h4>Step1: Configure ETCD cluster:</h4><p>We’ll install the ETCD on all nodes. recommended 3 ETCD Nodes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*16a_HkuDJoZtdc6l.jpg" /></figure><p><strong>On all 3 ETCD Nodes:</strong></p><p>Place corresponding parameters value for per each ETCD node</p><pre><strong>&lt;ETCD_Unique_Name&gt;</strong>: Unique Name for each ETCD node<br><strong>&lt;ETCD_Local</strong><strong>_Private_IP&gt;</strong>: Local Private IP on which node<br><strong>&lt;etcd_1</strong><strong>_Private_IP&gt;, &lt;etcd_1</strong><strong>_Private_IP&gt; and &lt;etcd_1</strong><strong>_Private_IP&gt;: </strong>Private IP address of ETCD node1, node2, node3 respectively.</pre><p>Run following commands on each node</p><pre>Make sure that your system is up to date<br><strong>$ sudo apt-get update -y</strong></pre><pre>To install ETCD<br><strong>$ sudo apt-get install -y etcd</strong></pre><pre><strong>$ vi /etc/default/etcd<br></strong>ETCD_INITIAL_CLUSTER=&quot;etcd1=http://<strong>&lt;ETCD_1</strong><strong>_Private_IP&gt;</strong>:2380,etcd2=http://<strong>&lt;ETCD_2</strong><strong>_Private_IP&gt;</strong>:2380,etcd3=http://<strong>&lt;ETCD_3</strong><strong>_Private_IP&gt;</strong>:2380&quot;<br>ETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;<br>ETCD_INITIAL_CLUSTER_TOKEN=&quot;etcd-cluster-01&quot;<br>ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;http://<strong>&lt;ETCD_Local</strong><strong>_Private_IP&gt;</strong>:2380&quot;<br>ETCD_DATA_DIR=&quot;/var/etcd&quot;<br>ETCD_LISTEN_PEER_URLS=&quot;http://<strong>&lt;ETCD_Local</strong><strong>_Private_IP&gt;</strong>:2380&quot;<br>ETCD_LISTEN_CLIENT_URLS=&quot;http://<strong>&lt;ETCD_Local</strong><strong>_Private_IP&gt;</strong>:2379,http://127.0.0.1:2379&quot;<br>ETCD_ADVERTISE_CLIENT_URLS=&quot;http://<strong>&lt;ETCD_Local</strong><strong>_Private_IP&gt;</strong>:2379&quot;<br>ETCD_NAME=&quot;<strong>&lt;ETCD_Unique_Name&gt;</strong>&quot;</pre><p>start etcd on all nodes asap. Because if you start node1 first, it’ll wait for a few seconds to connect other nodes. At this time, other nodes are unavailable then, the etcd will not start. So open two terminals for all the nodes, and run the start command asap.</p><pre><strong># service etcd start</strong></pre><p><strong>Check the cluster health:</strong></p><blockquote><em>Run this command on any one of the nodes<br></em><strong><em># etcdctl cluster-health</em></strong></blockquote><blockquote><em>member 74ea3293214c99ab is healthy: got healthy result from </em><a href="http://10.128.0.114:2379/"><em>http://10.128.0.114:2379</em></a><em><br>member a3e5382602a9a572 is healthy: got healthy result from </em><a href="http://10.128.0.115:2379/"><em>http://10.128.0.115:2379</em></a><em><br>member a3e5382602a9a572 is healthy: got healthy result from </em><a href="http://10.128.0.115:2379/"><em>http://10.128.0.117:2379</em></a></blockquote><blockquote><em>cluster is healthy</em></blockquote><h4>Step 2: Configure HAProxy for ETCD Cluster</h4><p><strong>On etcd-lb node</strong></p><p>You can configure HAProxy by editing /etc/haproxy/haproxy.cfg file:</p><pre><strong># nano /etc/haproxy/haproxy.cfg</strong></pre><pre>global<br>    user haproxy<br>    group haproxydefaults<br>    mode http<br>    log global<br>    retries 2<br>    timeout connect 3000ms<br>    timeout server 5000ms<br>    timeout client 5000ms</pre><pre>frontend etcd<br>    bind <strong>&lt;HAProxy_IP&gt;</strong>:6443<br>    option tcplog<br>    mode tcp<br>    default_backend etcd-cluster-nodes</pre><pre>backend etcd-cluster-nodes<br>    mode tcp<br>    balance roundrobin<br>    option tcp-check<br>    server etcd_01 <strong>&lt;ETCD_1</strong><strong>_Private_IP&gt;</strong>:6443 check fall 3 rise 2<br>    server etcd_02 <strong>&lt;ETCD_2</strong><strong>_Private_IP&gt;</strong>:6443 check fall 3 rise 2<br>    server etcd_03 <strong>&lt;ETCD_3</strong><strong>_Private_IP&gt;</strong>:6443 check fall 3 rise 2</pre><p>Save and close the file. Then, restart HAProxy with the following command:</p><pre><strong># systemctl restart haproxy</strong></pre><h4>Step 3: Install PostgreSQL and Patroni</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*SXowN5c6P3axceZ9.png" /></figure><p><strong>On node postgresql1, postgresql2, postgresql3</strong></p><pre>Make sure that your system is up to date<br><strong>$ sudo apt-get update -y</strong></pre><pre>To install PostgreSQL and Python-PostgreSQL Database Adapter.<br><strong>$ sudo apt-get install -y postgresql python3-psycopg2 python-psycopg2</strong></pre><pre>create symlink for super user<br><strong>$ sudo ln -s /usr/lib/postgresql/9.5/bin/* /usr/sbin/</strong></pre><blockquote><em>To Verify the status of PostgreSQL<br></em><strong><em>$ sudo systemctl status postgresql<br></em></strong><em>● postgresql.service - PostgreSQL RDBMS<br> Loaded: loaded (/lib/systemd/system/postgresql.service; enabled; vendor preset: enabled)<br> Active: active (exited) since Fri 2018-09-21 20:03:04 IST; 1min 7s ago<br> Main PID: 3994 (code=exited, status=0/SUCCESS)<br> CGroup: /system.slice/postgresql.service<br><br>Sep 21 20:03:04 Node1 systemd[1]: Starting PostgreSQL RDBMS...<br>Sep 21 20:03:04 Node1 systemd[1]: Started PostgreSQL RDBMS.<br>Sep 21 20:03:24 Node1 systemd[1]: Started PostgreSQL RDBMS.</em></blockquote><p>Next, stop the PostgreSQL service so that we can manage Patroni</p><pre><strong>$ sudo systemctl stop postgresql</strong></pre><p>Install Patroni</p><pre><strong>$ sudo -H pip3 install --upgrade pip</strong></pre><pre><strong>$ pip install patroni</strong></pre><pre><strong>$ pip install python-etcd</strong></pre><p><strong>On every PostgreSQL nodes</strong></p><p>Place corresponding parameters value for per each PostgreSQL node</p><pre><strong>&lt;Unique_Name&gt;</strong>: Unique Name for each PostgreSQL node<br><strong>&lt;Local_Private_IP&gt;</strong>: Local Private IP on which node<br><strong>&lt;ETCD_LB_IP&gt;: </strong>ETCD Load Balancer IP<br><strong>&lt;Postgresql_1_Private_IP&gt;, &lt;Postgresql_2_Private_IP&gt; &lt;Postgresql_3_Private_IP&gt;</strong>: Private IP address of PostgreSQL node1, node2, node3 respectively.</pre><p>Edit <strong>/etc/patroni.yml</strong></p><pre><strong># nano /etc/patroni.yml</strong></pre><pre>scope: postgres<br>namespace: /db/<br>name: <strong>&lt;Unique_Name&gt;</strong><br><br>restapi:<br>    listen: <strong>&lt;Local_Private_IP&gt;</strong>:8008<br>    connect_address: <strong>&lt;Local_Private_IP&gt;</strong>:8008<br><br>etcd:<br>    host: <strong>&lt;ETCD_LB_IP&gt;</strong>:2379<br><br>bootstrap:<br>    dcs:<br>        ttl: 30<br>        loop_wait: 10<br>        retry_timeout: 10<br>        maximum_lag_on_failover: 1048576<br>        postgresql:<br>            use_pg_rewind: true<br><br>    initdb:<br>    - encoding: UTF8<br>    - data-checksums<br><br>pg_hba:<br>    - host replication replicator 127.0.0.1/32 md5<br>    - host replication replicator <strong>&lt;Postgresql_1_Private_IP</strong><strong>&gt;</strong>/0 md5<br>    - host replication replicator <strong>&lt;Postgresql_2_Private_IP</strong><strong>&gt;</strong>/0 md5<br>    - host replication replicator <strong>&lt;Postgresql_3_Private_IP</strong><strong>&gt;</strong>/0 md5<br>    - host all all 0.0.0.0/0 md5<br><br>    users:<br>        admin:<br>            password: admin<br>            options:<br>                - createrole<br>                - createdb<br><br>postgresql:<br>    listen: <strong>&lt;Local_Private_IP&gt;</strong>:5432<br>    connect_address: <strong>&lt;Local_Private_IP&gt;</strong>:5432<br>    data_dir: /data/patroni<br>    pgpass: /tmp/pgpass<br>    authentication:<br>        replication:<br>            username: replicator<br>            password: password<br>        superuser:<br>            username: postgres<br>            password: password<br>    parameters:<br>        unix_socket_directories: &#39;.&#39;<br><br>tags:<br>    nofailover: false<br>    noloadbalance: false<br>    clonefrom: false<br>    nosync: false</pre><p>Save and close the file, when you are finished.</p><p>Next, create a data directory for Patroni on node1, node2 and node3:</p><pre>mkdir -p /data/patroni</pre><p>Next, change the ownership and permissions of the data directory:</p><pre>chown postgres:postgres /data/patroni<br>chmod 700 /data/patroni</pre><p>Next, you will need to create a startup script for Patroni to manage and monitor Patroni. You can do this with the following command:</p><pre><strong># nano /etc/systemd/system/patroni.service</strong></pre><pre>[Unit]<br>Description=Runners to orchestrate a high-availability PostgreSQL<br>After=syslog.target network.target</pre><pre>[Service]<br>Type=simple</pre><pre>User=postgres<br>Group=postgres</pre><pre>ExecStart=/usr/local/bin/patroni /etc/patroni.yml</pre><pre>KillMode=process</pre><pre>TimeoutSec=30</pre><pre>Restart=no</pre><pre>[Install]<br>WantedBy=multi-user.targ</pre><p>Save and close the file. Then start PostgreSQL and Patroni service on all node1, node2 and node3 with the following command:</p><pre>systemctl start patroni<br>systemctl enable postgresql</pre><p>Next, check the status of Patroni using the following command:</p><pre># systemctl status patroni</pre><pre>● patroni.service - Runners to orchestrate a high-availability PostgreSQL<br>   Loaded: loaded (/etc/systemd/system/patroni.service; disabled; vendor preset: enabled)<br>   Active: active (running) since Fri 2018-09-21 22:22:22 IST; 3min 17s ago<br> Main PID: 3286 (patroni)<br>   CGroup: /system.slice/patroni.service<br>           ├─3286 /usr/bin/python3 /usr/local/bin/patroni /etc/patroni.yml<br>           ├─3305 postgres -D /data/patroni --config-file=/data/patroni/postgresql.conf --max_worker_processes=8 --max_locks_per_transaction=64<br>           ├─3308 postgres: postgres: checkpointer process                                                                                     <br>           ├─3309 postgres: postgres: writer process                                                                                           <br>           ├─3310 postgres: postgres: stats collector process                                                                                  <br>           ├─3315 postgres: postgres: postgres postgres 192.168.0.105(54472) idle                                                              <br>           ├─3320 postgres: postgres: wal writer process                                                                                       <br>           └─3321 postgres: postgres: autovacuum launcher process                                                                              <br><br>Sep 21 22:24:52 Node1 patroni[3286]: 2018-09-21 22:24:52,329 INFO: Lock owner: postgresql0; I am postgresql0<br>Sep 21 22:24:52 Node1 patroni[3286]: 2018-09-21 22:24:52,391 INFO: no action.  i am the leader with the lock</pre><h4>Step 4: Configure HAProxy for PostgreSQL</h4><p>PostgreSQL cluster is now up and running. It’s time to configure HAProxy to forward connection receive from PostgreSQL client to the Master node.</p><p>You can configure HAProxy by editing /etc/haproxy/haproxy.cfg file:</p><pre><strong># nano /etc/haproxy/haproxy.cfg</strong></pre><pre>global<br>    maxconn 100</pre><pre>defaults<br>    log global<br>    mode tcp<br>    retries 2<br>    timeout client 30m<br>    timeout connect 4s<br>    timeout server 30m<br>    timeout check 5s</pre><pre>listen stats<br>    mode http<br>    bind *:7000<br>    stats enable<br>    stats uri /</pre><pre>listen postgres<br>    bind *:5000<br>    option httpchk<br>    http-check expect status 200<br>    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions<br>    server postgresql_01 <strong>&lt;Postgresql_1_Private_IP&gt;</strong>:5432 maxconn 100 check port 8008<br>    server postgresql_02 <strong>&lt;Postgresql_2_Private_IP&gt;</strong>:5432 maxconn 100 check port 8008<br>    server postgresql_03 <strong>&lt;Postgresql_3_Private_IP&gt;</strong>:5432 maxconn 100 check port 8008</pre><p>Save and close the file. Then, restart HAProxy with the following command:</p><pre><strong># systemctl restart haproxy</strong></pre><p>Thus we have configured an HA PostgreSQL Cluster in Active/Passive Mode</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9994aca9d1d" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/configure-ha-postgresql-cluster-in-active-passive-mode-b9994aca9d1d">Configure HA PostgreSQL Cluster in Active/Passive Mode</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create and Integrate S3 and RDS with EC2 in AWS]]></title>
            <link>https://blog.nonstopio.com/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88?source=rss-b436be0199a3------2</link>
            <guid isPermaLink="false">https://medium.com/p/47015743fe88</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[aws-rds]]></category>
            <dc:creator><![CDATA[Sagar Gadewar]]></dc:creator>
            <pubDate>Sat, 09 May 2020 15:37:11 GMT</pubDate>
            <atom:updated>2020-05-27T10:01:42.165Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello guys, In this blog I’ll try to help you integrate S3 and RDS with EC2 instance from scratch. Before that let’s have a brief introduction about services.</p><p><strong>Amazon </strong><a href="https://aws.amazon.com/ec2/"><strong>EC2</strong></a><strong> (Elastic Compute Cloud)</strong>, one of Amazon Web Services’ most well-known services, offers businesses the ability to run applications on the public cloud. Developers can create instances of virtual machines and easily configure the capacity scaling of instances using the EC2 web interface.</p><p>Amazon<strong> Simple Storage Service (Amazon S3)</strong> is a scalable, high-speed, web-based cloud storage service designed for online <a href="https://searchdatabackup.techtarget.com/definition/backup">backup</a> and <a href="https://searchstorage.techtarget.com/definition/archive">archiving</a> of data and <a href="https://searchsoftwarequality.techtarget.com/definition/application">applications</a> on Amazon Web Services. Amazon S3 was designed with a minimal feature set and created to make web-scale computing easier for developers.</p><p>Amazon <strong>RDS</strong> is a managed<strong> Relational Database Service</strong> that provides you six familiar database engines to choose from, including <a href="https://aws.amazon.com/rds/aurora/">Amazon Aurora</a>, <a href="https://aws.amazon.com/rds/mysql/">MySQL</a>, <a href="https://aws.amazon.com/rds/mariadb/">MariaDB</a>, <a href="https://aws.amazon.com/rds/oracle/">Oracle</a>, <a href="https://aws.amazon.com/rds/sqlserver/">Microsoft SQL Server</a>, and <a href="https://aws.amazon.com/rds/postgresql/">PostgreSQL</a>. This means that the code, applications, and tools you already use today with your existing databases can be used with Amazon RDS. Amazon RDS handles routine database tasks such as provisioning, patching, backup, recovery, failure detection, and repair.</p><p>Amazon RDS makes it easy to use replication to enhance availability and reliability for production workloads. Using the <a href="https://aws.amazon.com/rds/details/multi-az/">Multi-AZ</a> deployment option, you can run mission-critical workloads with high availability and built-in automated fail-over from your primary database to a synchronously replicated secondary database. Using <a href="https://aws.amazon.com/rds/details/read-replicas/">Read Replicas</a>, you can scale out beyond the capacity of a single database deployment for read-heavy database workloads.</p><p><strong>In Short:</strong></p><p>Step 1: Launch EC2 Instance on AWS</p><p>Step II: Create S3 Storage</p><p>Step III: Integrate S3 and EC2 instance</p><p>Step IV: Create RDS (PostgreSQL)</p><p>Step V: Integrate RDS (PostgreSQL) with EC2 Instance</p><h3>Step 1: Launch EC2 Instance on AWS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/0*u5ZiM5DSDwLOirU1" /></figure><p>1. Open the Amazon EC2 console at Under compute services</p><blockquote><a href="https://console.aws.amazon.com/ec2/">https://console.aws.amazon.com/ec2/</a>.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*D0zHWTlsufZ2pl2c" /></figure><p>2. From the console dashboard, choose <strong>Launch Instance</strong>.</p><p>3. The Choose<strong> </strong>an<strong> Ubuntu 18.04 </strong>Amazon Machine Image<strong> (AMI)</strong> page</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2y87GWdg-Oxx20Ik" /></figure><p>4. On the <strong>Choose an Instance Type</strong> page, you can select the hardware configuration of your instance. as per your requirement</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LiGcWhntDmhoLMjx" /></figure><p>5. In Configuration settings details page leave default settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/0*yIu4YqU1-PgCKwlm" /></figure><p>6. Click on <strong>Add Storage</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GDyw33rHBmfQmCNv" /></figure><p>[Optional]</p><p>7. Edit Size options if required (minimum 10 GB )</p><p>[Optional]</p><p>8. You can add <strong>tags</strong> if multiple EC2 instances present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/0*KdHVF-inQlgbJibm" /></figure><p>9. Click on the<strong> Configure Security Group</strong> button.</p><p>10. Create a new <strong>security group</strong> with appropriate name add new rules for</p><ul><li>HTTP (80 port)</li><li>HTTPS (443 port)</li><li>SSH (22 port)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/0*2tdb8ssT7Ni0n0mZ" /></figure><p>11. Click on Review and Launch</p><p>12. In the <strong>Select an existing key pair or create a new key pair</strong> dialog box, choose to <strong>Create a new key pair</strong>, enter a name for the key pair, and then choose <strong>Download Key Pair</strong>. This is the only chance for you to save the private key file, so be sure to download it. Save the private key file in a safe place and click on Launch Instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/0*zwx2j_1wagsdY0G2" /></figure><p>13. A confirmation page lets you know that your instance is launching. Choose <strong>View Instances</strong> to close the confirmation page and return to the console.</p><h3><strong>Step II: Create S3 Storage</strong></h3><ol><li>Sign in to the AWS Management Console and open the Amazon S3 console at <a href="https://console.aws.amazon.com/s3/">https://console.aws.amazon.com/s3/</a>.</li><li>Choose <strong>Create bucket</strong>.<br>The Create bucket page opens.</li><li>In <strong>Bucket&#39;s name</strong>, enter a DNS-compliant name for your bucket.<br>The bucket name must:</li></ol><ul><li>Be unique across all of Amazon S3.</li><li>Be between 3 and 63 characters long.</li><li>Not contain uppercase characters.</li><li>Start with a lowercase letter or number.</li></ul><p>4. After you create the bucket, you can’t change its name.</p><p>5. In <strong>Region</strong>, choose the AWS Region where your AWS EC2 instance resides.</p><p>6. In <strong>Bucket settings for Block Public Access</strong>, keep the values set to the defaults.</p><p>7. Choose Create bucket.</p><p>You’ve created a bucket in Amazon S3.</p><h3><strong>Step III: Integrate S3 and EC2 instance</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*Pg75KMAPxcJLilWE.png" /></figure><p>To connect to your S3 buckets from your EC2 instances, you need to:</p><p>1. Create and attach an AWS Identity and Access Management (IAM) profile role to the instance that grants access to Amazon S3.</p><p>2. Confirm that the S3 bucket policy doesn’t have a policy denying access.</p><p>3. Confirm network connectivity between the EC2 instance and Amazon S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*e4MyEz2fTn28Cnzy" /></figure><ol><li>To create IAM role, go to IAM service.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jhcz0uzr3Z_UODrh" /></figure><p>2. To create a new role</p><p>Click on <strong>Create role</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/0*OhT05z_6HWAGdqt9" /></figure><p>3. Select type <strong>AWS service </strong>of<strong> </strong>the<strong> </strong>trusted entity</p><p>and Click on <strong>Next: Permissions </strong>button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/0*RRUVUbTS-YZbDw1t" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bBeLJsHT_4qpdMPV" /></figure><p>4. Search for the <strong>AmazonS3FullAccess</strong> managed policy and then select it. If you want to attach multiple policies to a role, you can do so.</p><p><strong>Note:</strong> To create a custom policy that grants access to specific buckets, see <a href="https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/">Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/0*diHKCiEdwxYzrx_M" /></figure><p>Verify policy permissions in JSON format by clicking on <strong>AmazonS3FullAccess </strong>hyperlink.</p><p>and <strong>Click on Next: Review </strong>button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/186/0*6RfWj_vBK__o_wTX" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sDPsVd455VkWyTuX" /></figure><p>5. Provide a valid name and description and hit <strong>create role </strong>button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/0*Rr_Av_--EKGvWDdH" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*rOMk9FFHxDzqKknI" /></figure><p>6. Open the Amazon EC2 dashboard at Under compute services</p><blockquote><a href="https://console.aws.amazon.com/ec2/">https://console.aws.amazon.com/ec2/</a>.</blockquote><p>Go to running Instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/0*lxX0vrCHYAasoQp8" /></figure><p>To verify EC2 could not access S3 buckets run following command in EC2 instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C1fcPzyn6_Fbgfj_" /></figure><p>7. Select our EC2 instance to which S3 policy want to attach.</p><p>Choose the Actions tab, followed by<strong> Instance Settings </strong>&gt;<strong> Attach/Replace IAM Role </strong>as shown in Fig.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*12i-V_rU4TLkQmKY" /></figure><p>8. Attach IAM role which we created earlier. We get dropdown options as we type IAM role name in <strong>IAM role</strong> text box. and Click on <strong>Apply</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*nfzuSEd-g321oPfI" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mem0OWFP4Hrm-ih0" /></figure><p><strong>Verification: </strong>Verify access to your S3 buckets by running the following command:</p><pre>aws s3 ls</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wlfBFFtDoxdJ_Sde" /></figure><h3><strong>Step IV: Create RDS (PostgreSQL)</strong></h3><p><strong>Refer URL: </strong><a href="https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/">https://aws.amazon.com/getting-started/tutorials/create-connect-postgresql-db/</a></p><h3>Step V: Integrate RDS (PostgreSQL) with EC2 Instance</h3><p><strong>Note: </strong>Recommended</p><p>If you don’t want to connect RDS outside VPN</p><p><strong>Only accessible to Django EC2 Instance</strong></p><p>Set <strong>Public accessibility</strong> parameter to<strong> No</strong></p><p>Go to your RDS instance and click the security group in the <strong>Connectivity &amp; Security</strong> tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/0*IxlYQrYYNZ7-fc_B" /></figure><p>This will take you to configure the security group assigned to your RDS instance.</p><p>Click the Inbound tab, then click the Edit button to change the rules for your security group.</p><p>Change the Type property to <strong>PostgreSQL</strong>, which will update the Protocol and Port Range to the proper values.</p><p>In the value, the group enters the security group that is used for your EC2 instance.</p><p>Click Save.</p><p><strong>Conclusion: </strong>I hope this story helps you in integrating S3 and RDS with EC2 instance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47015743fe88" width="1" height="1" alt=""><hr><p><a href="https://blog.nonstopio.com/create-and-integrate-s3-and-rds-with-ec2-in-aws-47015743fe88">Create and Integrate S3 and RDS with EC2 in AWS</a> was originally published in <a href="https://blog.nonstopio.com/">nonstopio</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>