<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Deepak Puthraya on Medium]]></title>
        <description><![CDATA[Stories by Deepak Puthraya on Medium]]></description>
        <link>https://medium.com/@deepakputhraya?source=rss-fbb83474af40------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rD97ctQeEy3Bo5KGSqfOrg.png</url>
            <title>Stories by Deepak Puthraya on Medium</title>
            <link>https://medium.com/@deepakputhraya?source=rss-fbb83474af40------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:09:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deepakputhraya/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pair programming & Senior Engineers]]></title>
            <link>https://medium.com/@deepakputhraya/pair-programming-senior-engineers-93e89422be4b?source=rss-fbb83474af40------2</link>
            <guid isPermaLink="false">https://medium.com/p/93e89422be4b</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[pair-programming]]></category>
            <dc:creator><![CDATA[Deepak Puthraya]]></dc:creator>
            <pubDate>Sat, 01 Sep 2018 21:13:32 GMT</pubDate>
            <atom:updated>2018-09-01T21:14:23.444Z</atom:updated>
            <content:encoded><![CDATA[<p>From Wikipedia: Pair programming is an agile software development technique in which two programmers work together at one workstation. One, the driver, writes code while the other, the observer or navigator, reviews each line of code as it is typed in. The two programmers switch roles frequently.</p><p>You have probably seen two engineers working together either racking their brains on a bug, discussing how to improve the code quality in a particular area or hacking together a feature! Well, all those are instances of pair programming.</p><p>I had briefly talked about this <a href="https://blog.playment.io/deepak-puthraya-software-developer-story/">once</a> &amp; how pair programming benefited me as a rookie engineer. However, I always wondered what it was for a senior engineer or manager. What is it that they gained from pairing together with a junior developer? It seemed such a waste of their time for me.</p><p>I have been on both the sides now feel that it is a lot more beneficial to senior engineers &amp; I strongly feel that organisations would benefit a lot more with engineers pair programming. Pair programming solves many problems that are sometimes really hard to solve with “processes”, especially for small organisations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*MRJXmrlC2zVOTenTx_XGLA.jpeg" /></figure><h3>Onboarding</h3><p>Companies have unique processes to onboard new engineers. They are generally assigned buddies who help new engineers with their problem. They are given simple tasks or features to complete so that they get accustomed to the engineering processes. Though completing these tasks gives the engineer a sense of fulfilment, it lacks depth. In small startups, documentation could be lacking, there is no explanation as to why there was a hack, or there might be few bad practices from your early days of coding lying around which could get picked up as “best practice” by the engineer.</p><p>You could take a few hours to pair with them to explain some of the best practices used. Warn them about the pitfalls lying around &amp; probably even pointing them out to various other team members for help if needed for a particular section of code.</p><h3>Kent Beck on Twitter</h3><p>pair programming works best with a large uncertain search space of problems and solutions. the closer to a solved problem, the less it helps</p><h3>Code Quality</h3><p>When developing features due to various reasons, there are various times when we incur technical debt. Some of these are fixed over a period when developments happen in the future. Some bad practices could lie around for months.</p><p>When pairing with other engineers, you could take the opportunity to point out these “bad code” parts that could need attention in the future. They would empathise with you after seeing the tech debt the team has incurred and could volunteer to help remove the tech debt.</p><h3>Debugging Bugs</h3><p>Debugging at times seems more like an art than science. They are at times harder than making a feature itself. Debugging can be intimidating. Pairing with junior engineers is a great opportunity to impart great debugging skills &amp; an excellent opportunity to assess the code quality that you probably missed out on in your review.</p><p>Engineer move out of teams or take a more critical role where they are mostly dealing with people or dealing with only essential services or features. At such times they miss out on code reviews and tech debt is occurred without the knowledge of incurring tech debt. Debugging is an opportunity to get more hands-on &amp; assess the current state of the codebase.</p><h3>Kent Beck on Twitter</h3><p>pair programming is not a &quot;go faster&quot; strategy, it is a &quot;waste less&quot; strategy (which often results in going faster)</p><h3>Improving engineering practices</h3><p>As engineers, we are not just building features for our customers. We are also building tools that help us ship features faster. Be it setting up code guidelines or CI/CD pipeline or introducing cool new tech into your stack. When senior engineers pair with other engineers they see things that other engineers might miss out.</p><p>For example, to test a feature manually, it could require a lot of test data or manual inputs. In such cases, others would enter them manually; you would see that as an opportunity to write simple scripts to automate the process &amp; commit them so that others can use them in the future. Alternatively, you noticed a lack of initial data &amp; decide to invest in using embedded datasets or a solution that speeds up development. Pair programming is a great tool to identify such cases.</p><h3>Team bonding</h3><p>It is a great way to connect with your colleagues. When you pair program the sense of superiority is lost &amp; you don’t just complain about the need to put semicolons everywhere, but also share your favourite Rick &amp; Morty episodes.</p><h3>Conclusion</h3><p>Pair programming is a fantastic engineering practice &amp; every organisation should encourage it. It is an excellent way to reduce technical debt, increase team bonding, improve development practices &amp; share knowledge across the organisation.</p><h3>Kent Beck on Twitter</h3><p>pair programming works best with a large uncertain search space of problems and solutions. the closer to a solved problem, the less it helps</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93e89422be4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Say Yes to Technical Debt]]></title>
            <link>https://medium.com/@deepakputhraya/say-yes-to-technical-debt-3fc33d5a0eb6?source=rss-fbb83474af40------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fc33d5a0eb6</guid>
            <category><![CDATA[technical-debt]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Deepak Puthraya]]></dc:creator>
            <pubDate>Sat, 01 Sep 2018 21:12:36 GMT</pubDate>
            <atom:updated>2018-09-01T21:15:27.993Z</atom:updated>
            <content:encoded><![CDATA[<p>Technical Debt is usually referred to as something Bad. Most articles on the topic are about how to get rid of technical debt. Having been influenced by such articles, I shunned hacks, features that were poorly designed or features without much test coverage. However, I have been thinking quite the opposite recently. Is debt bad? What effect would the technical debt have on future development projects? How will it affect related features and tasks? If this is a peripheral feature that is not likely to be used in the coming months, maybe a technical debt is okay, and I could use the extra time to move the product further.</p><p>Think of technical debt as anything about your code that slows you down over the long term. Hard-to-read code, lack of test automation, duplication, tangled dependencies.</p><h3>Jedd Ahyoung on Twitter</h3><p>What is technical debt?&quot; Technical debt is hard to explain, but a picture is worth a thousand words. #programming #softwaredevelopment</p><p>After <a href="https://playment.io/">Playment</a> launched its <a href="https://app.playment.io/">annotation platform</a>, we set out to build an onboarding tool for our platform so that we could mitigate the number of users that were dropping off after signups. Product owner wanted us to build a simple onboarding experience for the first time web users. They wanted us to ship the feature real quick so that they could test &amp; iterate over it. We built a quick &amp; dirty feature that required a developer’s help to set up the onboarding flow, but it worked. The feature was poorly designed, not very extendible &amp; had few test cases. After having released the feature, we could see that many users were now completing the onboarding flow, earning few bucks &amp; continued to do other tasks on our Platform. The code ended up being untouched for over 6 months till we decided to extend the feature into onboarding experience for projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYmLNKpUpG8OmQkZtKCE7Q.png" /><figcaption><em>Sample onboarding flow</em></figcaption></figure><p>However, when we wanted to continue the product line we realised it was good that we left it half done because what we intend to do now over the same feature happened to be a much better thing than what we thought we would do, which wasn’t possible in hindsight at all. It was right for us to have taken the quick &amp; dirty approach to problem-solving rather than the correct &amp; tedious one.</p><h3>Paul Graham on Twitter</h3><p>Technical debt is bad, but (more often than not) optimal.</p><p>As developers, we are quick to attacking a problem &amp; trying to find the right approach. We try to cover cases that would likely not arise or if they do they probably far out in the future. At least, this is the truth for most startups. Sure that database query would break if there were 1M entries users, But we probably need to make lot more features so that we can reach that 1M mark.</p><p>Tech debt stinks, it slows down development &amp; creates quality issues. It can have a negative impact when scaling up. Refactoring code to reduce tech debt is painful &amp; resource distraction.</p><p>At Playment, lead-engineers try to keep a track of tech debt, classify easy fixes to fundamental architecture design flaws. A small percentage of engineering effort goes into lessening tech debt every release. We strive to set expectations that refactoring is part of our job, showcasing excellent refactoring(eg: small demo of the code). At times its fun &amp; feels great after removing large amount dead code.</p><blockquote>Tech debt is often a “damned if you do, damned if you don’t” problem. <br><strong>- Survival to Thrival</strong></blockquote><p>There is a time and a place to reduce technical debt, but it’s not as crucial as most engineers think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fc33d5a0eb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Postgresql backup and recovery with Barman]]></title>
            <link>https://medium.com/@deepakputhraya/postgresql-backup-and-recovery-with-barman-a41c81dd10fd?source=rss-fbb83474af40------2</link>
            <guid isPermaLink="false">https://medium.com/p/a41c81dd10fd</guid>
            <category><![CDATA[disaster-recovery]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[database-administration]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Deepak Puthraya]]></dc:creator>
            <pubDate>Fri, 28 Apr 2017 21:46:59 GMT</pubDate>
            <atom:updated>2018-01-17T10:46:25.552Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Xx9pPtaH1p5Lvyds165SQ.jpeg" /></figure><p>Taking backups is an important administrative task that can have some disastrous consequences if it is not done right. The use of <a href="https://en.wikipedia.org/wiki/RAID">RAID</a> configurations in your storage system, replication between nodes, clustering and trusting 100% that your SAN will be up ARE NOT backup strategies. These measures are necessary for HA(High availability) but do not replace the necessity of taking backups of our databases.</p><p>There are two different types of backup that can be used with PostgreSQL to implement a good backup and restore strategy</p><p><strong>Logical backups:</strong></p><p>They take a snapshot of a database at a given moment.</p><ul><li>Backup individual databases or all databases</li><li>Backup just the schemas, just the data, individual tables, or the whole database (schemas and data)</li><li>Create the backup file in proprietary binary format or plain SQL script</li><li>Can be restored using the pg_restore utility which also ships with PostgreSQL</li><li>Does not offer point-in-time recovery (PITR)</li></ul><p><strong>Physical backups:</strong></p><p>This type of backup takes copies of the files where the PostgreSQL saves the databases.</p><ul><li>Offer point-in-time recovery</li><li>Backup the contents of the PostgreSQL <em>data directory</em> and the <em>WAL</em> (Write Ahead Log) files</li><li>Take larger amounts of disk space</li><li>Use the PostgreSQL pg_start_backup and pg_stop_backup commands. However, these commands need to be scripted, which makes physical backups a more complex process</li><li>Do not back up individual databases, schemas only, etc. It’s an all-or-nothing approach</li></ul><p>PITR is a good strategy when High availability is not the major reason behind your database backup. If you cannot afford to loose almost zero data and can afford to have some downtime, then PITR is the approach for you. We will be looking at how we can setup a PITR recovery backup for a Postgresql instance using <a href="http://docs.pgbarman.org/release/2.1/#general-commands">Barman</a>.</p><p>We will be testing this on a Ubuntu machine with Postgresql 9.6 and Barman 2.1. We will need using three Virtual machines one that would be the database that needs to be backed up, the other machine is where Barman will be installed, and backed-up data would be sitting and the last machine we will be using to restore the database to. I will assume that you already have the database up to get started, if you do not you can create a Postgresql 9.6 database and continue forward.</p><h3>Initial Setup</h3><p>Create a new machine to install, run and manage backups. Let’s call this <em>barman-01</em></p><p><em>On the barman machine install Barman</em></p><pre>sudo sh -c &#39;echo &quot;deb <a href="http://apt.postgresql.org/pub/repos/apt/">http://apt.postgresql.org/pub/repos/apt/</a>$(lsb_release -cs)-pgdg main&quot; &gt; /etc/apt/sources.list.d/pgdg.list&#39;<br>sudo apt-get update<br>sudo apt-get install barman</pre><p><em>Switch to the Barman user on the machine.</em></p><pre>sudo su - barman</pre><p>Authorise the barman user to connect to the database machine(<em>postgres@database-machine-ip</em>) that needs to be backed up. To do this copy the public keys from barman to authorization file in the Postgres users of the database machine.</p><p>On Barman as barman user:</p><pre>ssh-keygen</pre><p>Press enter for all the subsequent entries. Copy the contents on <em>~/.ssh/id_rsa.pub</em></p><p>On the Database machine, switch to the Postgres user and copy the ssh key of the barman into <em>~/.ssh/authorized_keys</em></p><pre>sudo su - postgres<br>echo “copied-ssh-key-string” &gt;&gt; ~/.ssh/authorized_keys</pre><p>The above commands also need to be run between Postgresql machine and the Barman machine as well. To make sure that the machines can be connected.</p><p>Test the connections with the below commands: To connect to the barman machine from the database machine as <em>postgres</em> user:</p><pre>ssh barman@barman-server-ip</pre><p>To connect to database machine from Barman</p><pre>ssh postgres@main-db-server-ip</pre><p>If the above two commands are working, that means the two machines can connect each other without any issue.</p><h3>Configuring Barman for backups</h3><p>Edit <strong><em>/etc/barman.conf</em></strong> to be set the properties as below</p><pre>[barman]<br>barman_home = /var/lib/barman</pre><pre>barman_user = barman<br>log_file = /var/log/barman/barman.log<br>compression = gzip<br>bandwidth_limit = 0<br>immediate_checkpoint = true<br>last_backup_maximum_age = 3 DAYS<br>retention_policy = RECOVERY WINDOW OF 7 days<br>minimum_redundancy = 3</pre><p>What we are specifying here:</p><p><strong>barman_home</strong> : Keeping the default backup location determining where the backup will be saved.</p><p><strong>compression</strong>: WAL(write ahead logs) logs will be compressed, and base backups will use incremental data copying</p><p><strong>last_backup_maximum_age</strong> : The age of the last full backup for a PostgreSQL server should not be older than 3 days</p><p><strong>retention_policy</strong>: We want to be able to recover our database to any time of the last seven days</p><p><strong>minimum_redundancy: </strong>This ensures that we have at least 3 base backups and will make sure that we accidentally do not delete backups.</p><p>What we did above was set the global configurations for all our Postgresql backups. We can override these settings for individual backups in their specific settings.</p><p>Add the below lines to the end of the file <em>/etc/barman.d/pg.conf</em> so as to specify which Postgresql instances need to be backed up:</p><pre>[pg]<br>description = “PG Backup”<br>conninfo = host=10.0.2.2 user=postgres<br>ssh_command = ssh postgres@10.0.2.2<br>backup_method = rsync<br>streaming_archiver = on<br>streaming_conninfo = host=10.0.2.2 user=postgres dbname=db<br>archiver = on<br>slot_name = barman</pre><pre>reuse_backup = link</pre><pre># For base backup retention<br>retention_policy = RECOVERY WINDOW OF 7 DAYS</pre><pre># For archive logs retention<br>wal_retention_policy = main</pre><pre># Retention policies are automatically enforced by cron<br>retention_policy_mode = auto</pre><pre>minimum_redundancy=2</pre><p>Note that in the above setting host=10.0.2.2 is the host address of the Postgresql database.</p><p><strong>reuse_backup: </strong>We specify if the database should take a complete backup of the database or just copy on the changed files. If we <em>link </em>then we can save a lot of disk space locally(about 50%)</p><h3>Gabriele Bartolini on Twitter</h3><p>pgbarman #postgresql #IncrementalBackup #database size: 13.2 TB. Actual size on disk: 5.0 TB (-62.01% #deduplication ratio). Saved 8.2TB!</p><p>We will need to configure the database: We will need the INCOMING_WALS_DIRECTORY on the barman machine for the server, to get this run:</p><pre>barman show-server pg |grep incoming_wals_directory</pre><p>In the <em>pg_hba.conf</em> of the database make an entry to allow Barman access the database:</p><pre>host         all          all         &lt;barman ip&gt;/32           trust</pre><p>Make an entry for ‘replication’ or ‘all’ in <strong><em>pg_hba.conf</em></strong> file. We will also need to make changes in the <strong>postgres.conf</strong> file:</p><pre>listen_addresses = &#39;*&#39;<br>archive_mode = on<br>wal_level = ‘replica’<br>archive_command = ‘rsync -a %p barman@backup:INCOMING_WALS_DIRECTORY/%f’<br>max_wal_senders = 2<br>max_replication_slots = 2</pre><p>Restart the Postgresql instance.</p><p><strong>NOTE</strong>: max_wal_senders and max_replication_slots are just examples.</p><p>On barman once all the configurations are done:</p><pre>barman switch-xlog --force --archive pg<br>barman check pg</pre><p>To create a slot run:</p><pre>barman receive-wal --create-slot pg</pre><p>The check command may show <em>FAILED</em> in <strong>minimum redundancy requirements</strong> and <strong>backup maximum age</strong></p><p>Create a base backup of the Postgresql instance</p><pre>barman backup pg</pre><h3>Cron</h3><p>We need to setup cron to start backup every day at a particular time:</p><pre>crontab -e</pre><p>Add the following entries:</p><pre>30 23 * * * /usr/bin/barman backup pg<br>* * * * * /usr/bin/barman cron</pre><p>The above has created a cron that takes a base backup of the database every day at 11:30 PM</p><h3>Recovery</h3><h4>Creating a Standby Server</h4><p>Installing Postgres on Ubuntu 14.04:</p><pre>sudo add-apt-repository “deb <a href="http://apt.postgresql.org/pub/repos/apt/">http://apt.postgresql.org/pub/repos/apt/</a> trusty-pgdg main”</pre><pre>wget -quiet -O <a href="https://www.postgresql.org/media/keys/ACCC4CF8.asc">- https://www.postgresql.org/media/keys/ACCC4CF8.asc</a> | sudo apt-key add -</pre><pre>sudo apt-get update</pre><pre>sudo apt-get install postgresql-9.6</pre><p>From the barman machine run: To get the list of backup done so far:</p><pre>barman list-backup pg</pre><p>To get the latest backup information we can use a shortcut:</p><pre>barman show-backup pg latest</pre><p>To recover first stop the standby Postgresql instance:</p><pre>service postgresql stop</pre><p>Now run the command below:</p><pre>barman recover -target-time &lt;timestamp&gt; -remote-ssh-command “ssh postgres@&lt;standby-ip&gt;” pg &lt;backup-id&gt; /var/lib/postgresql/9.6/main</pre><p>We can recover to a particular time stamp by providing the <em>— target-time</em> “2017–03–06 15:36:23+05:30”.</p><h3>Debugging:</h3><p>In case you get stuck</p><ol><li>List of slots in Postgresql</li></ol><pre>select * from pg_replication_slots;</pre><p>To know what each slot means you can read the Postgresql <a href="https://www.postgresql.org/docs/9.4/static/catalog-pg-replication-slots.html">documentation</a>.</p><p>2. To delete slots in Postgresql</p><pre>select pg_drop_replication_slot(&#39;pg&#39;);</pre><p>In the above command pg is the slot name.</p><p>3. Creating scripts to send a mail</p><p>In case you want to receive a mail whenever a base backup gets completed you can write a small script that would send you the mail and this can be triggered via the hook scripts configuration. In the pg.conf of barman add this line:</p><pre>post_backup_script = /var/lib/barman/report.sh</pre><p>In the home directory of the barman user create a shell script called <strong><em>report.sh</em></strong></p><pre>#!/bin/bash</pre><pre># Delete the oldest backup<br># It will only be deleted if minimum redundancy criteria is met <br>barman delete pg oldest</pre><pre># Get the information of the latest backup<br>barman show-backup pg latest &gt; /tmp/temp.txt<br>str=$(cat /tmp/temp.txt | while read x; do echo &quot;$x&lt;br&gt;&quot;;done)</pre><pre># Get disk related information<br>echo &#39;&lt;pre&gt;&#39; &gt; /tmp/temp.txt<br>df -H &gt;&gt; /tmp/temp.txt<br>echo &#39;&lt;/pre&gt;&#39; &gt;&gt; /tmp/temp.txt<br>data=$(cat /tmp/temp.txt | while read x; do echo &quot;$x&lt;br&gt;&quot;;done)</pre><pre>backup_date=`date &#39;+%Y-%m-%d&#39;`</pre><pre>str1=&#39;&lt;html&gt;&lt;p&gt;Date: &#39;$backup_date&#39;&lt;/p&gt;&lt;p&gt;Backup Status : &#39;$BARMAN_STATUS&#39;&lt;/p&gt;&lt;p&gt;Backup ID : &#39;$BARMAN_BACKUP_ID&#39;&lt;/p&gt;&lt;p&gt;Error Message : &#39;$BARMAN_ERROR&#39;&lt;/p&gt;&lt;p&gt;Stats : &lt;br&gt;&#39;$str&#39;&lt;/p&gt;&lt;p&gt;Drive Information : &#39;$data&#39;&lt;/p&gt;&lt;/html&gt;&#39;<br></pre><pre>echo &#39;{<br>  &quot;personalizations&quot;: [<br>    {<br>      &quot;to&quot;: [<br>        {<br>          &quot;email&quot;: &quot;someone@company.com&quot;,<br>        }<br>      ],<br>      &quot;subject&quot;: &quot;DB Backup | &#39;$backup_date&#39;&quot;<br>    }<br>  ],<br>  &quot;from&quot;: {<br>    &quot;email&quot;: &quot;<a href="/service/mailto:barman@playment.in">barman@company.com</a>&quot;<br>  },<br>  &quot;reply_to&quot;: {<br>    &quot;email&quot;: &quot;<a href="/service/mailto:barman@playment.in">barman@company.com</a>&quot;<br>  },<br>  &quot;subject&quot;: &quot;DB Backup | &#39;$backup_date&#39;&quot;,<br>  &quot;content&quot;: [<br>    {<br>      &quot;type&quot;: &quot;text/html&quot;,<br>      &quot;value&quot;: &quot;&#39;$str1&#39;&quot;<br>   }<br>  ]<br>}&#39; &gt; &quot;/tmp/temp.txt&quot;</pre><pre>curl -X POST -H &quot;Content-Type: application/json&quot; -H &quot;Authorization: &lt;Token&gt;&quot; -H &quot;Cache-Control: no-cache&quot; -d  @/tmp/temp.txt &quot;<a href="https://api.sendgrid.com/v3/mail/send">https://api.sendgrid.com/v3/mail/send</a>&quot;</pre><pre>rm /tmp/temp.txt</pre><p>The script gives the details of the latest backup taken along with the disk space left in the Barman machine. Make sure to make the script executable and can be executed by barman user. The script uses Sendgrid to send the email.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a41c81dd10fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configure Postgresql for Scale]]></title>
            <link>https://medium.com/@deepakputhraya/configure-postgresql-for-scale-c8c5ef87ef76?source=rss-fbb83474af40------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8c5ef87ef76</guid>
            <category><![CDATA[scale]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Deepak Puthraya]]></dc:creator>
            <pubDate>Sun, 26 Feb 2017 12:40:48 GMT</pubDate>
            <atom:updated>2017-02-26T12:40:48.916Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are using Postgresql as your database or you are looking for a reliable database to get started Postgresql would most likely do the job. For most applications, Relational databases could be an excellent choice. And if you are wondering that your application would need a database to store unstructured database then Postgres is a good alternative. Postgresql supports JSONB type that lets you store unstructured data, you could also go ahead, and index attributes from these columns! PostgreSQL also supports multi-dimensional arrays. It has great tooling, and there are a lot of great extensions that you could add to your database without any changes. If you need more convincing read <a href="http://www.craigkerstiens.com/2012/04/30/why-postgres/">this</a>.</p><p>In any case, if you are using Postgresql here are something that you could probably consider doing to scale up your existing database:</p><h3>Setup Replica</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*2JcGZUCFXaH2n32xVbeIhg.gif" /></figure><p>Most likely you have already done this. If you have not, you should probably set this up. If your company has people running analytical queries directly on the primary database, then one large SQL query could bring down your whole database or significantly affect the performance. You could easily setup a Database replica, and there are a lot of guides on the internet to help set it up. Here is <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-on-postgresql-on-an-ubuntu-12-04-vps">one</a> to help get started.</p><h3>Use PgAnalyze</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Rdm0C4WPsO49GwKmkZH3Uw.png" /><figcaption>Report from PgAnalyze</figcaption></figure><p><a href="https://pganalyze.com/"><strong>Pganalyze</strong></a> is a great tool to help analyze slow queries, and it also provides some insight on how you could speed things up. You need to install their extension and configure the database a bit. You will be able to monitor and get some insight as to which queries possible need tuning, columns of tables that need indexes. They also provide monitoring tools to view disk usage, CPU usage and IO usage. Pganalyze is free for use, they do have paid plans on which they provide weekly reports and notify you when there are any critical issues.</p><h3>Use PgPool</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*v7voJ4QhMeokDCZ28saqfw.png" /></figure><p><a href="https://wiki.postgresql.org/wiki/Pgpool-II"><strong>Pgpool</strong></a> is a great tool that you could use to get a sudden boost in your application performance if you run a lot of reads and write queries. Pgpool sits on top of your primary and replicas; it load-balances all the read queries on the machines thus removing most of the load from the primary database. It is Nginx load balancer for databases. Note that all write queries still go to the primary database.</p><h3>Setup Materialized views</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*TCqXGvCNQ60XBpA5CuWr5w.png" /></figure><p>Most reports are generated by querying the database directly without much business, And they could be overwhelming slow, substantially slowing down the rest of the queries that are running on the database. Materialized views are similar to views, but they are real tables. One cannot insert into a materialized view, but the tables can be indexed. Pre-generated materialized views can be used by analysts, to generate reports quickly. The predicament with Materialized views is that you would have to refresh them manually which could be a deal breaker for many. There is a workaround to refresh materialized views by using triggers. Setup a trigger to update the materialized view every couple of hours or you could update the view whenever there is any change in the tables the view is concerned. Note: Do use this feature carefully as it could lead to increase the load on the database if setup incorrectly.</p><h3>Configure the Database.</h3><p>If your company does not have a DBA then most likely you are using the stock Postgresql. You could not have bothered with changing the configuration of the database for the hardware. Configuring the database is application and system specific, it is NOT easy. But there is still some bit of tuning that you could do to get more performance out of the machine. There are tools available to help configure PostgreSQL for your machine’s hardware, <a href="http://pgtune.leopard.in.ua/">PGTune</a> is one.</p><h3><strong>Citus for Sharding</strong></h3><p>Quoting from <a href="https://www.citusdata.com/">CitusData</a>:</p><blockquote>Citus is a distributed database that scales across commodity servers using transparent sharding and replication. Citus extends the underlying database rather than forking it, giving developers and enterprises the power and familiarity of a relational database. As an extension, Citus supports new PostgreSQL releases and allows you to benefit from new features while maintaining compatibility with existing PostgreSQL tools.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IC0CFoDmzltqg0XDZ_rYtw.png" /><figcaption>Cloudflare using Citus for analytics</figcaption></figure><p>Sharding is a technique of partitioning large tables into multiple smaller, faster, more easily managed tables. Citus is a great extension that you can use to shard tables if your database has grown over a couple of hundreds of gigabytes. Citus takes care of managing the queries and you do not need to do any changes in your applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8c5ef87ef76" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>