<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sole Techie on Medium]]></title>
        <description><![CDATA[Stories by Sole Techie on Medium]]></description>
        <link>https://medium.com/@soletechie?source=rss-139e7b07d7ff------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*z4Kn77ocbpLfp5g-Ez9X8Q@2x.jpeg</url>
            <title>Stories by Sole Techie on Medium</title>
            <link>https://medium.com/@soletechie?source=rss-139e7b07d7ff------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:41:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@soletechie/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Fixing issues related to S3 COPY command to Redshift]]></title>
            <link>https://medium.com/@soletechie/fixing-issues-related-to-s3-copy-command-to-redshift-702d9978d505?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/702d9978d505</guid>
            <category><![CDATA[aws-migration]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 16:21:55 GMT</pubDate>
            <atom:updated>2021-12-18T16:21:55.688Z</atom:updated>
            <content:encoded><![CDATA[<p>We use s3 COPY command to move data from S3 to Redshift table. And here are the problems that we might face while using S3 COPY command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*SMAjqZpyxxnDujn1MlZgcw.png" /></figure><p>Make sure the schema for Redshift table is created before running your COPY command. I’m writing this post to log all errors related to the COPY command that I faced which might help others in saving their time.</p><p>Problem 1: <strong>Faced issue with delimiters on CSV file</strong>. As we know the delimiter for any CSV file will be comma separators, but the problem on keeping this as a delimiter is that it will fail with below error if the data also have commas in it.</p><p><strong>Error Code: 1213, Error Description: Missing data for not-null field</strong></p><p>AWS Documentation link that helped in solving this issue:<br><a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#load-from-csv">https://docs.aws.amazon.com/redshift/latest/dg/r_COPY_command_examples.html#load-from-csv</a></p><p>Solution 1.1: As the issue is with comma as a delimiter, I have<strong> specified the file format as CSV and ignored the headers</strong>. This fixed the issue.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec0042c80b845238563c0dc48e4ca43c/href">https://medium.com/media/ec0042c80b845238563c0dc48e4ca43c/href</a></iframe><p>Solution 1.2: <strong>Using ESCAPE, and remove quotes should work as well</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2339613da5d7aed1d5c03a506d426356/href">https://medium.com/media/2339613da5d7aed1d5c03a506d426356/href</a></iframe><p>Problem 2: <strong>Redshift doesn’t support materialized views</strong>. I have tried copying views directly to Redshift from S3, which resulted in below error:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98fcd7cdd65cbccae516f6ca44afffbb/href">https://medium.com/media/98fcd7cdd65cbccae516f6ca44afffbb/href</a></iframe><p>Solution 2.1: <strong>Create views as a table and then COPY, if you don’t care its either a view or a table on Redshift.</strong></p><p>Solution 2.2: <strong>First COPY all the underlying tables, and then CREATE VIEW on Redshift.</strong></p><p>FYI, ***<strong>Problem with JSON format</strong>****<br>The format of the file which you are trying to copy also plays an important role in fixing the issues related to the COPY command. I’m using CSV files for most of the test runs because I faced a limitation of 4MB while dealing with JSON formats. JSON parsers related to redshift have more problems to deal with. FYI, here is the error info I got related to JSON:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d96fb1b6fddbaed6ea552805824cdff/href">https://medium.com/media/9d96fb1b6fddbaed6ea552805824cdff/href</a></iframe><p>Problem 3: <strong>Delimiter not found while running copy command</strong></p><p>Try with <strong>FILLRECORD </strong>— where missing columns will be with zero-length strings or NULLs</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/106e49b9d6001f46e2adeeea87db1f92/href">https://medium.com/media/106e49b9d6001f46e2adeeea87db1f92/href</a></iframe><p>Problem 4: <strong>When there are issues with timestamps and the file format is not csv but comma separated</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/377d1b7e1000081d1133941a19d79a98/href">https://medium.com/media/377d1b7e1000081d1133941a19d79a98/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=702d9978d505" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connecting Presto with Redshift]]></title>
            <link>https://medium.com/@soletechie/connecting-presto-with-redshift-77883b907eda?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/77883b907eda</guid>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[aws-migration]]></category>
            <category><![CDATA[presto]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 16:02:16 GMT</pubDate>
            <atom:updated>2021-12-18T16:02:16.860Z</atom:updated>
            <content:encoded><![CDATA[<p>This post helps you in adding redshift connector to presto so that you can directly access redshift using Presto.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*spzczXIUN7No-tfFalD4gQ.png" /><figcaption>Adding presto connectivity to Redshift</figcaption></figure><p>Majority of the data resides on Redshift and our end users can only access Presto. So, tried on adding redshift connector to presto. By following below steps:</p><p>1.<strong> Add redshift properties under /etc/presto/conf/catalog/ (This will add redshift catalog)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85d303e79c1836fd4cd2c715f657578d/href">https://medium.com/media/85d303e79c1836fd4cd2c715f657578d/href</a></iframe><p>2. <strong>Now, go to presto-CLI: to find redshift catalog as well</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/983f36a3647df9f945aeecef7090de87/href">https://medium.com/media/983f36a3647df9f945aeecef7090de87/href</a></iframe><p>NOTE:</p><p>- Restart presto-server if you cannot find redshift catalog</p><p>- Better to add these properties on all master and all the slave nodes</p><p>3. <strong>But, while trying to access schemas from redshift, I’m getting connection attempt failed error. Maybe because of the security group issues.</strong></p><blockquote>presto&gt;<strong> show schemas from redshift;</strong></blockquote><blockquote>Query 20190724_115839_00002_kcxs8 failed: The connection attempt failed.</blockquote><p><strong><em>The issue is with the jdbc url, added to the redshift properties file. After fixing it, it worked fine.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77883b907eda" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exporting data from Hive to DynamoDB]]></title>
            <link>https://medium.com/@soletechie/exporting-data-from-hive-to-dynamodb-e2fedc41ab5f?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2fedc41ab5f</guid>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[hive]]></category>
            <category><![CDATA[aws-migration]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 15:43:20 GMT</pubDate>
            <atom:updated>2021-12-18T22:10:20.849Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*GWVC2T9mXYwmqrh18MnwEg.png" /><figcaption>Exporting Data from Hive to Dynamo DB</figcaption></figure><p>Follow below steps to move data from Hive to Dynamo DB.<br>Source link: <a href="https://docs.amazonaws.cn/en_us/amazondynamodb/latest/developerguide/EMRforDynamoDB.html">https://docs.amazonaws.cn/en_us/amazondynamodb/latest/developerguide/EMRforDynamoDB.html</a></p><p>Below is just a quick summary from the above documentation link.</p><p>1. <strong>Create a hive table. (Ignore this step if hive table is already created)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7aac25a8b22330c5a2de734d31d2106c/href">https://medium.com/media/7aac25a8b22330c5a2de734d31d2106c/href</a></iframe><p>2. <strong>Now create another external hive table which just references to the hive table (but provides the mapping between Hive and DDB tables)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f40ad7e8fbc6dd826aa9b316b25c48d0/href">https://medium.com/media/f40ad7e8fbc6dd826aa9b316b25c48d0/href</a></iframe><p>3. <strong>Now insert data into Dynamo DB table from Hive table (for which Hive internally uses Map-Reduce Job to do this copy)</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f66b551a1eb49434365c334ffdb8a217/href">https://medium.com/media/f66b551a1eb49434365c334ffdb8a217/href</a></iframe><p>4. <strong>we can use hive QL to query DDB table</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48cec429b56ce757a4a3271c0e14538b/href">https://medium.com/media/48cec429b56ce757a4a3271c0e14538b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2fedc41ab5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to copy data from S3 to Aurora Postgres?]]></title>
            <link>https://medium.com/@soletechie/how-to-copy-data-from-s3-to-aurora-postgres-dd243b112acd?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd243b112acd</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws-migration]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 15:21:33 GMT</pubDate>
            <atom:updated>2021-12-18T15:21:33.795Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Scenario 1: To copy data from S3 to Aurora Postgres (greater than v9 or latest)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MKCsUJ8XXt2GToCenzwYA.png" /><figcaption>using aws_s3.table_import_from_s3 package</figcaption></figure><p><strong>How</strong>?: We can use <strong><em>aws_s3.table_import_from_s3 function </em></strong>(to migrate the data from S3 to Aurora Postgres).</p><p><strong>Steps:</strong></p><p>A sample file with columns — id, prefix, mstr_id is copied to S3.</p><p><strong>Create schema on Aurora Postgres (with the required columns).</strong></p><pre>drop table core.mstr;</pre><pre>CREATE TABLE core.mstr (<br>	id varchar(300) NULL,<br>	prefix varchar(300) NULL,<br>	mstr_id float8 NULL<br>);</pre><p><strong>Copy command to transfer the data from S3 to Aurora Postgres</strong></p><pre>SELECT aws_s3.table_import_from_s3(<br>   &#39;core.MSTR&#39;, <br>   &#39;id,prefix,mstr_id&#39;, <br>   &#39;(format csv, header true)&#39;,<br>   &#39;&lt;bucket-name&gt;&#39;,<br>   &#39;MSTR_DATA/part_file_00.csv&#39;,<br>   &#39;us-east-2&#39;,<br>   &#39;&lt;secret key&gt;&#39;, &#39;&lt;access key&gt;&#39;<br>);</pre><p><strong>Note: If IAM roles are given, we need not specify access keys.</strong></p><pre>SELECT aws_s3.table_import_from_s3(<br>   &#39;core.MSTR&#39;, <br>   &#39;id,prefix,mstr_id&#39;, <br>   &#39;(format csv, header true)&#39;,<br>   &#39;&lt;bucket-name&gt;&#39;,<br>   &#39;MSTR_DATA/part_file_00.csv&#39;,<br>   &#39;us-east-2&#39;<br>);</pre><p>Note: aws_s3.table_import_from_s3 function is not supported for versions earlier than Aurora Postgres Version 9.</p><h3><strong>Scenario 2: To copy data from S3 to Aurora Postgres (older versions than 9)</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XILENNAmLEr-pKsZgHuzFg.png" /><figcaption>Run sqoop job on EMR</figcaption></figure><p><strong>How? </strong>: Use SQOOP Job to export the data from S3 to Aurora Postgres.</p><p><strong>Steps</strong>:</p><p><strong>Upload data files to s3</strong>.</p><p><strong>Create schema on Aurora Postgres (with the required columns).</strong></p><pre>drop table core.mstr;</pre><pre>CREATE TABLE core.mstr (<br>	id varchar(300) NULL,<br>	prefix varchar(300) NULL,<br>	mstr_id float8 NULL<br>);</pre><p><strong>Spin-up EMR Cluster</strong></p><ul><li>Must have access to S3 and Aurora Postgres</li><li>Check for connectivity:</li><li>Whether EMR cluster and RDS reside in same VPC</li><li>dig &lt;Aurora hostname&gt;</li><li>nc -vz &lt;hostname&gt; (must get a message: connectivity looks good)</li><li>Make sure the security groups are properly assigned to the EMR Cluster.</li><li>The security group must ALLOW traffic from CORE node of EMR Cluster</li><li>Must be added to the RDS Security Group as a source</li></ul><p><strong>Sqoop Job</strong></p><pre>echo &quot;Exporting MSTR table from S3 to Postgres&quot;<br>echo &quot;Sqoop Job in Progress&quot;</pre><pre>sqoop export <br>--connect &lt;jdbc URL&gt;<br> --input-null-string NULL <br> --input-null-non-string NULL <br> --table mstr <br> --export-dir s3://&lt;bucket_name&gt;/MSTR/ <br> --username user_etl <br> --P<br> -- --schema test</pre><pre>echo &quot;Sqoop Job is executed successfully&quot;<br>echo &quot;Data will now be available on Aurora Postgres&quot;</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd243b112acd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2min overview of terraform lifecycle]]></title>
            <link>https://medium.com/@soletechie/2min-overview-of-terraform-lifecycle-5b908bc36ee8?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b908bc36ee8</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 12:50:35 GMT</pubDate>
            <atom:updated>2021-12-18T12:50:35.503Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pNoOaq8eYH2oRCtapxc8PA.png" /><figcaption>Image from Terraform documentation — <a href="https://www.terraform.io/">https://www.terraform.io/</a></figcaption></figure><p>If we are using terraform, terraform state file is the heart of all the infrastructure that we spin up using terraform templates.</p><p>There are several ways to deploy the infrastructure using terraform:</p><p>1. Using CLI (setup terraform and then run terraform commands)</p><p>2. Automated Build (terraform scripts integrated as part of your jenkins pipeline)</p><p>No matter of the way we chose, we must make sure that we are using the same terraform state file, so that we are having a sync and proper checklists of the resources that we used.</p><p>Here are the most frequently used terraform commands:</p><p><strong>terraform init</strong> =&gt; the basic/starting command which initializes the terraform (make sure the proper provider is provided. In my case, I use AWS).</p><p><strong>terraform workspace select &lt;workspace name</strong>&gt; =&gt; (creates a new workspace, useful in scenarios where we have different terraform modules — database, servers, logs, storage)</p><p><strong>terraform state list</strong> =&gt; shows the list of terraform resources that are created (uses the state file)</p><p><strong>terraform plan</strong> =&gt; creates a terraform plan and will give us a list of changes without actually deploying them (make sure you specify -out tfplan.plan if we need to store the output of this plan command)</p><p><strong>terraform apply</strong> =&gt; to actually deploy based on the terraform plan</p><p><strong>terraform destroy </strong>=&gt; to destroy the resources which we created</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b908bc36ee8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Must use VS-Code Extensions for Cloud Developers/Engineers]]></title>
            <link>https://medium.com/@soletechie/must-use-vs-code-extensions-for-cloud-developers-engineers-2bf683afaca1?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bf683afaca1</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[vscode-extension]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[vscode]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 12:32:39 GMT</pubDate>
            <atom:updated>2021-12-18T12:32:39.360Z</atom:updated>
            <content:encoded><![CDATA[<p>Here are the list of VS Code extensions that anyone working on cloud technologies can use to speed up their development.</p><p>To download any extension, refer to the extension tab on your VS code window:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*GFMeRMjmk6eqgt1Y.png" /></figure><p>As we will manage all our cloud resources using Terraform, we will start with Terraform Autocomplete Extension.</p><p>1. <strong>Terraform</strong> <strong>Extensions</strong></p><p><strong>Terraform: </strong>to manage terraform resources directly from VS Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*cAIdmmHI6w7O5sAe.png" /></figure><p><strong>Terraform Autocomplete</strong>: useful when we are creating terraform resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*kp2B033k95mub66D.png" /></figure><p>2. <strong>Docker</strong>: To build, manage and deploy docker containers from VS Code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*vXPVbJJrt7TRChef.png" /></figure><p>3. <strong>Python</strong>: extension that provides python interpreter</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*CpyYK3MSDa6Z9_Mn.png" /></figure><p>4. <strong>Prettier-Code formatter</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*POtVeETDCDzLKHNA.png" /></figure><p>5. <strong>Markdown Preview</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*YDnJGudqZSukMwji.png" /></figure><p>6. <strong>Git</strong>:</p><p><strong>Git History</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*SFmKotHPDiKESmZ0.png" /></figure><p><strong>Git Graph</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*mnpZ0raScOc20u1Z.png" /></figure><p>Now we can select the below extensions, and click on install.</p><h3><strong>AWS VSCode Extensions:</strong></h3><p>1. <strong>AWS Toolkit</strong>: To interact with AWS resources directly from VS Code. Helpful in taking a look of AWS resources without having to login into console, provides us with a very cool UI to get a quick overview of our resources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*_mTaaDy4T3h9PHIE.png" /></figure><p>Upon successful installation, we can find AWS on the left toolbar as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*ksj0q_Ww6vJ6--1Q.png" /></figure><p>2. <strong>AWS CLI Configure</strong>: To use the AWS profiles directly, will be very handy when we want to use multiple AWS accounts and want to manage them separately. Realtime use-case would be when we want to access AWS resources from different environments like PROD environment or DEV environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*gkqOYSPLJw4vMBxp.png" /></figure><p>3. <strong>AWS boto3</strong>: boto3 is a python library that will help us communicate with AWS resources</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*jWvQ__vQingIiCdg.png" /></figure><p>4. <strong>Sort AWS IAM Policy</strong>: will be a lot of help when we want to prepare IAM document especially when we are dealing with too many AWS resources in the same document. Unless they are really sorted, IAM policy can quickly become a mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*xt21zrZufS3qLpa-.png" /></figure><p>5. <strong>AWS Step Functions Constructor</strong>: Helps us to visualize the AWS step functions directly on the VSCode, without having to check the document definition on the console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*EiROzguh5r5N7qxO.png" /></figure><h3><strong>Azure VSCode Extensions</strong>:</h3><p><strong>1. Azure Account</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*LOf6EQIjNo7_eAk3.png" /></figure><p>2. <strong>Azure Tools</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*dQKXiYtL7xwZDDN3.png" /></figure><p>The above extension is a package installer — will install or download the following Azure extensions as well:</p><p>- Azure Functions</p><p>- Azure Resources</p><p>- Azure CLI Tools</p><p>- Azure App Service</p><p>- Azure Resource Manager (ARM) tools</p><p>- Azure Databases</p><p>- Azure Storage</p><p>- Azure Pipelines</p><p>- Azure Virtual Machines</p><p>- ARM Template Viewer</p><h3><strong>Google Cloud (GCP) Extensions</strong>:</h3><p>1. <strong>Cloud Code</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*bldhI_a2VMZEEKkH.png" /></figure><p>2. <strong>Google Cloud Spanner Driver</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*AbIp2QMkEH3OZvxS.png" /></figure><p>This will be a work in progress. Will be adding more extensions. Feel free to contribute by posting a comment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bf683afaca1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with AWS]]></title>
            <link>https://medium.com/@soletechie/getting-started-with-aws-cfd3717de073?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfd3717de073</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[getting-started]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 12:27:49 GMT</pubDate>
            <atom:updated>2021-12-18T12:29:06.773Z</atom:updated>
            <content:encoded><![CDATA[<p>Compiled instructions for beginners who wants to get started with AWS.</p><p><strong>Here are the getting started guides that I followed along to setup my new AWS account</strong>:</p><p>https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Get_to_Know_the_AWS_Cloud</p><p><a href="https://aws.amazon.com/getting-started/guides/setup-cdk/">https://aws.amazon.com/getting-started/guides/setup-cdk/</a></p><p><a href="https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Launch_Your_First_Application">https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Launch_Your_First_Application</a></p><h3><strong>Setting up AWS account</strong>:</h3><p>1. <strong>Create Amazon Free Tier account</strong>: <a href="https://portal.aws.amazon.com/billing/signup?refid=ps_a131l0000085ejvqam&amp;trkcampaign=acq_paid_search_brand&amp;redirect_url=https%3A%2F%2Faws.amazon.com%2Fregistration-confirmation#/start">https://portal.aws.amazon.com/billing/signup?refid=ps_a131l0000085ejvqam&amp;trkcampaign=acq_paid_search_brand&amp;redirect_url=https%3A%2F%2Faws.amazon.com%2Fregistration-confirmation#/start</a></p><p>- Provide your details (email, username, billing information, and make sure you select basic support-free option).</p><p>- Upon successful signup, we will be seeing a confirmation like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*Zqupfkf4TyInfZhT.png" /><figcaption>AWS Account Signup Successful!</figcaption></figure><p>2.<strong> Sign-in as root user</strong>: provide your login information (email, password) and we will be able to see our AWS dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*5hlqPM3-na-08bSv.png" /><figcaption>Login to AWS Console as root user</figcaption></figure><p>3. <strong>Access AWS Management Console</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*iKm-NxxW9N7QPfvD.png" /><figcaption>Accessing AWS Management Console</figcaption></figure><p>4. Follow this documentation provided by AWS: <a href="https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Get_to_Know_the_AWS_Cloud">https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Get_to_Know_the_AWS_Cloud</a></p><p>Using the above documentation link, we can find the best practices on how to setup our AWS cloud account.</p><p>Let us start making progress by following this guide:</p><p><strong>Setting up AWS environment</strong>: <a href="https://aws.amazon.com/getting-started/guides/setup-environment/">https://aws.amazon.com/getting-started/guides/setup-environment/</a></p><p><strong>Adding MFA:</strong></p><p>- select “IAM” service, and add <strong>MFA (Multi Factor Authentication)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*rBYuZDY88LCLd-TO.png" /><figcaption>IAM Dashboard (before enablingMFA)</figcaption></figure><p>- once we select “<strong>Add MFA</strong>”, it will take us to this page- and we need to select “<strong>Activate MFA</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*yLa7eZ7727GFLJjI.png" /><figcaption>Activate MFA</figcaption></figure><p>- select “<strong>Virtual MFA device</strong>” and hit Continue:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*W6uUD4P8AYWw2-E9.png" /><figcaption>Managing Virtual MFA Device — Google Authenticator App</figcaption></figure><p>- I used “<strong>Google Authenticator</strong>” app as my MFA device. Scan the QR Code using the app, and enter 2 MFA codes. Once we successfully add the device, we can see our device under MFA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*lTBpHfFpZGnCrkOu.png" /><figcaption>After enabling MFA, we can see virtual device added</figcaption></figure><p>- once we add the MFA, we can see the IAM dashboard as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*nYJZUn4Oz5l9-yDQ.png" /><figcaption>IAM dashboard (after enabling MFA)</figcaption></figure><p><strong>Create IAM Group</strong> -&gt;</p><p>Now, we can proceed with creating user groups. As it is not advised to use root user for everything. We have to follow the least access privilege principle to keep our accounts more secure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*lzEXVsJf5ZJ3wsXu.png" /><figcaption>Creating a new IAM Group</figcaption></figure><p><em>Enter user group name</em>: admins</p><p><em>Attach permission policies</em>: search for “administrator access” &amp; select it</p><p>Now we can see admins group getting created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*6UIkkTb9YRyq6lTf.png" /><figcaption>Created new IAM User Group — “admins”</figcaption></figure><p><strong>Create IAM User</strong> -&gt;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*Y8E4L8S8VHusYaM3.png" /><figcaption>Creating a new IAM User</figcaption></figure><p>select “<strong>Add users</strong>”</p><p><strong>username</strong>: soletechie</p><p>enable both programmatic access (to use AWS resources using CLI) and password — to access management console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*TG0cxz5Ejo16xGpw.png" /><figcaption>Enabling programmatic and management console access to IAM user</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/0*lO2TU-yGSYeeBEPW.png" /><figcaption>Attaching IAM User to IAM Group</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*EFTH3OEl-to0Br9F.png" /><figcaption>Specifying Tags — helps in resource management, billing.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*qTC3OVyEmzfAQWFV.png" /><figcaption>Review of the IAM User profile which we are trying to add</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*C01SUSG2MQAsc9XZ.png" /><figcaption>Created a new IAM User</figcaption></figure><p>Make sure the IAM User details are stored. We will be using AWS Access key ID and Secret access key while configuring our AWS CLI.</p><p><strong>Note</strong>:</p><p>- <strong>we can create our account alias</strong> if we don’t want to use our account ID to login to AWS console</p><p>- To create alias, go to IAM dashboard, and to your right, you can find your AWS account ID information, where you will have the option to create alias.</p><p>- Aliases must be unique, once you give a unique alias name, you will be able to sign in to AWS management console using this alias.</p><h4>**************************************************************<strong>Setting up AWS-CLI: </strong>**************************************************************</h4><p>- Use this link to setup AWS CLI (latest version v2) based on your operating system: <a href="https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html">https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html</a></p><p>- To download macOS package file: <a href="https://awscli.amazonaws.com/AWSCLIV2.pkg">https://awscli.amazonaws.com/AWSCLIV2.pkg</a></p><p>- once you run the installer, we will be able to see the software installed successfully.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*8srcz7JVz_BoTSdp.png" /><figcaption>Installing AWS CLI V2 package on macOS</figcaption></figure><p>- To verify if AWS CLI is successfully installed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*2RWzMDAecAbbJu-g.png" /><figcaption>Verifying if AWS CLI V2 is properly installed</figcaption></figure><p>Time to <strong>CONFIGURE</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*UjEb7lhjCgi-2pp5.png" /><figcaption>aws configure command</figcaption></figure><p>- type command — “<strong>aws configure</strong>” and provide your access key id, aws secret access key, default region name and default output format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*1WiFi7-yYW6mWGZB.png" /><figcaption>configuring AWS profile (default) using the IAM User credentials information</figcaption></figure><p>More detailed information on how to configure AWS CLI:<a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html">https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html</a></p><h4>**************************************************************<strong>Setting up Cloud9 — Browser based IDE </strong>**************************************************************</h4><p>- use this link to setup Cloud9. <a href="https://aws.amazon.com/getting-started/guides/setup-environment/module-four/?refid=ps_a131l0000085ejvqam&amp;trkcampaign=acq_paid_search_brand">https://aws.amazon.com/getting-started/guides/setup-environment/module-four/?refid=ps_a131l0000085ejvqam&amp;trkcampaign=acq_paid_search_brand</a></p><p>- <strong>cloud9</strong> is a free cloud based IDE that we can run using our browser. It supports programming languages including <strong>Python</strong>, <strong>JavaScript</strong>. So, we can work on our project basically using our browser rather than dealing with environment setups specific to our home/office laptops.</p><p>- AWS CLI command to spin up &amp; access &amp; destroy cloud9 environment. we use environment ID to access and delete the cloud9 environment.</p><blockquote><strong>aws cloud9 create-environment-ec2 — name getting-started — description “Getting started with AWS Cloud9.” — instance-type t3.micro — automatic-stop-time-minutes 60</strong></blockquote><blockquote><strong>{ “environmentId”: “8a34f51ce1e04a08882f1e811bd706EX” }</strong></blockquote><blockquote><strong>aws cloud9 delete-environment — environment-id &lt;environmentID&gt;</strong></blockquote><p>- To access the cloud 9 environment: <a href="https://console.aws.amazon.com/cloud9/ide/%3Cenvironment%20ID%3E?region=us-west-2">https://console.aws.amazon.com/cloud9/ide/&lt;environment ID&gt;?region=us-west-2</a></p><p>Note:</p><p>To dive deeper: <a href="https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Dive_Deeper">https://aws.amazon.com/getting-started/?e=gs2020&amp;p=console/#Dive_Deeper</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfd3717de073" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Glue as Common Metadata Store — Hive, Presto, Redshift]]></title>
            <link>https://medium.com/@soletechie/using-glue-as-common-metadata-store-hive-presto-redshift-bf8aa85bd17?source=rss-139e7b07d7ff------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf8aa85bd17</guid>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[presto]]></category>
            <category><![CDATA[glue]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Sole Techie]]></dc:creator>
            <pubDate>Mon, 22 Jul 2019 14:54:30 GMT</pubDate>
            <atom:updated>2019-07-22T14:54:30.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Glue as Common Metadata Store — Hive, Presto, Redshift</h3><p>From this post, you will learn how to use Glue to read the schema from the S3 file using its crawlers and build a common metadata store to other AWS services like Hive, Presto and Redshift Spectrum.</p><p>Before Glue Data Catalog (GDC) feature, it is painful to create schemas for the same file as the schema definitions vary from Hive and Redshift. Creating multiple schemas for the same file structure not only challenges the data integrity but also involves a lot of manual work. Most of us are already familiar with Glue as ETL framework on AWS. But its cataloging features will quickly add value to the business as well as help to streamline the technical workflows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*U-PLGdFnF59KDtqd.png" /><figcaption>Populating Glue Data Catalog (Source: AWS documentation <a href="https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html">https://docs.aws.amazon.com/glue/latest/dg/populate-data-catalog.html</a>)</figcaption></figure><p>In AWS, no matter what kind of data we deal with, S3 will be the primary ingestion/landing zone for our ETL processing. The Glue crawlers will read data from S3 and build a data catalog, and we can use the same data catalog as our hive meta-store which will be integrated with Presto as well.</p><p>Here are the steps:</p><ul><li>Drop the dataset on S3 or assume data is already there on S3.</li><li>Setup a Glue Crawler to read data from the file on S3.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LXJHswOmnpiXnSaTdx6jFQ.png" /><figcaption>Glue Crawler to read the schema from S3 file</figcaption></figure><p>Once the glue crawler is run, we must be able to see the glue database/catalog getting created. We can get the schema definition, by accessing the tables under that glue catalog.</p><ul><li>Now its time to spin up your EMR cluster and make sure these configuration settings are set for the cluster: set up hive-site to use glue data catalog meta-store, enable presto-connector-hive and spark-hive-site as well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*ijfm7I4tTEjXxScxLTFCmg.png" /><figcaption>EMR Cluster Configuration Details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NxKe4sXaz55fjTK3VgfJBg.png" /><figcaption>EMR Cluster Configuration Settings — to integrate Hive and Presto with GDC(Glue Data Catalog)</figcaption></figure><p>Note:</p><p>~ If you are unable to use the Master public DNS to do SSH, please try SSH with the private IP address. (Get the IP address of the master node by going to Hardware tab -&gt; Master Instance Details -&gt; there you will find your EC2 instance ID along with status, public DNS name, private DNS name, public IP address, and private IP address).</p><p>~ Download the private key file (.pem) to launch the cluster. Then, use: ssh -i &lt;name of your .pem file&gt;.pem hadoop@&lt;DNS Name/IP Address)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*TQJiWLkMXyt009XlfC2VTg.png" /><figcaption>SSH to the EMR Cluster</figcaption></figure><ul><li>Access HIVE on the EMR cluster, we must be able to see the glue databases here and access the tables on glue as hive tables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sl1ZSLswNunlSC9wO6o1pw.png" /><figcaption>Accessing Hive Console</figcaption></figure><p>Now, we must be able to see the Glue Data Catalog Database as one of the Hive Databases. Use that database, to run all your queries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yZaPEZrBWlFBug4FUytbw.png" /><figcaption>Running a simple count(*) on the hive table</figcaption></figure><ul><li>Access PRESTO-CLI on EMR, and we can use glue catalogs as our catalog and start querying the dataset on S3 directly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*iWVPvv2GyQ_D1yUj8tGyHQ.png" /><figcaption>Accessing Presto-CLI on EMR</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3btyYgm6E17kaZud88__lA.png" /><figcaption>Running a simple select count(*) on presto</figcaption></figure><ul><li>Now create external tables on redshift using IAM role (which should have permissions to access s3, glue services) as we will create and access redshift tables using glue catalog.</li></ul><p>create external schema &lt;Redshift schema name&gt;<br>from data catalog<br>database ‘&lt;glue catalog DB name&gt;’<br>region ‘us-east-1’ <br>iam_role ‘&lt;iam role arn&gt;’;</p><p>Note: Creating external schemas or tables on redshift, is nothing but we are using Redshift spectrum features, so we will be billed based on spectrum charges (usually based on a scanned query)</p><p>Thank you for your time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf8aa85bd17" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>