<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Onel Harrison on Medium]]></title>
        <description><![CDATA[Stories by Onel Harrison on Medium]]></description>
        <link>https://medium.com/@onelharrison?source=rss-410e9ddcf6f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OrPRlNIxnH5_gSphxZ3KQg.png</url>
            <title>Stories by Onel Harrison on Medium</title>
            <link>https://medium.com/@onelharrison?source=rss-410e9ddcf6f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 06:39:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@onelharrison/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deploying Containerized AWS Lambda functions with Terraform]]></title>
            <link>https://medium.com/akava/deploying-containerized-aws-lambda-functions-with-terraform-7147b9815599?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7147b9815599</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Tue, 17 Jan 2023 16:56:23 GMT</pubDate>
            <atom:updated>2023-01-17T20:00:09.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nf7Lgosu4_cJxfishLG1-A.png" /></figure><p>In <a href="https://medium.com/akava/simple-aws-lambda-deployment-with-terraform-1f51c08c6f4">Simple AWS Lambda Deployment with Terraform</a>, I demonstrated how to deploy a class of AWS Lambda functions I call the simple lambda function. Simple lambda functions are deployed by packaging source code as a zip archive and uploading it directly to the AWS Lambda service. These functions tend to be small (&lt;50MB), use only the built-in modules for the programming languages they are written in, and optionally also include the AWS SDK, which is available in all AWS Lambda runtimes.</p><p>Developing and deploying containerized applications has become common place in the software industry, and for good reasons (e.g. deployment ease and consistency, portability, etc.)! In this article, I demonstrate how to deploy containerized AWS Lambda functions using Terraform.</p><blockquote>The complete source code used in this post is <a href="https://github.com/onelharrison/terraform-containerized-lambda-deployment">available on GitHub</a>.</blockquote><p>The Python code below is for a lambda function that, when invoked, produces faked profile information as JSON response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ab12b0164dace8184bb66589595684d/href">https://medium.com/media/2ab12b0164dace8184bb66589595684d/href</a></iframe><p><strong>Example</strong></p><p>Input event</p><pre>{}</pre><p>Output (response)</p><pre>{<br>  &quot;statusCode&quot;: 200,<br>  &quot;headers&quot;: {<br>    &quot;Content-Type&quot;: &quot;application/json&quot;<br>  },<br>  &quot;body&quot;: &quot;{\&quot;profile\&quot;: {\&quot;name\&quot;: \&quot;Jennifer Hawkins\&quot;, \&quot;address\&quot;: \&quot;294 Dominic Coves Apt. 336\\nSmithchester, WI 70267\&quot;}}&quot;<br>}</pre><p>We begin by configuring the terraform provider for AWS. The code below assumes that your AWS credentials are available in the environment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82c951bedaa94ee849e81814b366cd79/href">https://medium.com/media/82c951bedaa94ee849e81814b366cd79/href</a></iframe><p>Notice that the AWS provider is configured to use the us-east-2 region. This is important because (at the time of this writing) AWS Lambda functions must be deployed in the same region where their container images are hosted in Amazon ECR. (Here is a <a href="https://github.com/aws/containers-roadmap/issues/1281">GitHub issue</a> tracking a discussion about making it so that AWS Lambda functions can use images hosted in any region, in any AWS account. AWS has so far <a href="https://aws.amazon.com/blogs/compute/introducing-cross-account-amazon-ecr-access-for-aws-lambda/">delivered on the cross-account part</a> of the feature request.)</p><p>The following terraform code creates a lambda function by using a container image hosted in an existing AWS ECR repository.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd1e1dadd430f897ab4ddb4d8d2c0ff8/href">https://medium.com/media/cd1e1dadd430f897ab4ddb4d8d2c0ff8/href</a></iframe><p>The file begins with a line declaring a variables for the environment, which is used for naming resources. The last few lines of the file create an aws_iam_role for the lambda function. But the core lines in this code are those for the aws_lambda_function resource and the aws_ecr_repository data source.</p><p>The aws_lambda_function resource is used to provision (as the name suggests) the AWS Lambda function using information from the aws_iam_role resource and aws_ecr_repository data source.</p><p>The aws_ecr_repository data source allows us to access the URI for the app’s container image. You’ll need to create an AWS ECR repository called profile-faker for this to work. (<a href="https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html">See these step-by-steps instruction for creating an AWS ECR repository</a>.)</p><p>The app’s source code is located in the ./aws_lambda_functions/profile_faker directory. The directory structure below shows how the essential files in this example are organized.</p><pre>.<br>├── aws.tf<br>├── aws_lambda_functions<br>│   └── profile_faker<br>│       ├── Dockerfile<br>│       ├── Makefile<br>│       ├── main.py<br>│       └── requirements.txt<br>└── providers.tf</pre><p>The Dockerfile for creating an image of the profile faker app is very basic as shown below.</p><pre>FROM public.ecr.aws/lambda/python:3.9<br><br># Install the function&#39;s dependencies using file requirements.txt<br># from your project folder.<br><br>COPY requirements.txt  .<br>RUN  pip3 install -r requirements.txt --target &quot;${LAMBDA_TASK_ROOT}&quot;<br><br># Copy function code<br>COPY main.py ${LAMBDA_TASK_ROOT}<br><br># Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)<br>CMD [ &quot;main.handler&quot; ]</pre><p>Once this Dockerfile exists, we merely need to create a container image for the app and push it to an AWS ECR repository.</p><p>I like using Makefiles to help with these things. (You will need to set environment variable values for AWS_ECR_ACCOUNT_ID and AWS_ECR_REGION.)</p><pre>APP_NAME = profile-faker<br>APP_VERSION = 0.0.1<br><br>AWS_ECR_ACCOUNT_ID ?=<br>AWS_ECR_REGION ?=<br>AWS_ECR_REPO = $(APP_NAME)<br><br>TAG ?= $(APP_VERSION)<br><br><br>.PHONY : docker/build docker/push docker/run docker/test<br><br>docker/build :<br>  docker build -t $(APP_NAME):$(APP_VERSION) .<br><br><br>docker/push : docker/build<br>  aws ecr get-login-password --region $(AWS_ECR_REGION) | docker login --username AWS --password-stdin $(AWS_ECR_ACCOUNT_ID).dkr.ecr.$(AWS_ECR_REGION).amazonaws.com<br>  docker tag $(APP_NAME):$(APP_VERSION) $(AWS_ECR_ACCOUNT_ID).dkr.ecr.$(AWS_ECR_REGION).amazonaws.com/$(AWS_ECR_REPO):$(TAG)<br>  docker push $(AWS_ECR_ACCOUNT_ID).dkr.ecr.$(AWS_ECR_REGION).amazonaws.com/$(AWS_ECR_REPO):$(TAG)<br><br><br>docker/run :<br>  docker run -p 9000:8080 $(AWS_ECR_ACCOUNT_ID).dkr.ecr.$(AWS_ECR_REGION).amazonaws.com/$(AWS_ECR_REPO):$(TAG)<br><br><br>docker/test :<br>  curl -XPOST &#39;http://localhost:9000/2015-03-31/functions/function/invocations&#39; -d &#39;{}&#39;</pre><p>Now I can do this to push the image to AWS ECR.</p><pre>make docker/push TAG=dev</pre><p>And do this to test the lambda function locally.</p><pre>make docker/run<br>make docker/test</pre><p>The output should look similar to this.</p><pre>{<br>  &quot;statusCode&quot;: 200,<br>  &quot;headers&quot;: {<br>    &quot;Content-Type&quot;: &quot;application/json&quot;<br>  },<br>  &quot;body&quot;: &quot;{\&quot;profile\&quot;: {\&quot;name\&quot;: \&quot;Jennifer Hawkins\&quot;, \&quot;address\&quot;: \&quot;294 Dominic Coves Apt. 336\\nSmithchester, WI 70267\&quot;}}&quot;<br>}</pre><p>I encourage you to init , plan , and apply the <a href="https://github.com/onelharrison/terraform-containerized-lambda-deployment">terraform code</a> used in this post and play with deploying containerized lambda functions using terraform.</p><pre>terraform init<br>terraform plan -var=&quot;env_name=dev&quot;<br>terraform apply -var=&quot;env_name=dev&quot;</pre><blockquote>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to <a href="/service/mailto:bd@akava.io">bd@akava.io</a> and reference this post.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8Yx3N4lsIgMuOdjF.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7147b9815599" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/deploying-containerized-aws-lambda-functions-with-terraform-7147b9815599">Deploying Containerized AWS Lambda functions with Terraform</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adopting GitHub Actions // More Than Just a CI Tool!]]></title>
            <link>https://medium.com/akava/adopting-github-actions-more-than-just-a-ci-tool-68f9bb06e9c3?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f9bb06e9c3</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 16 Nov 2022 16:27:10 GMT</pubDate>
            <atom:updated>2022-11-16T16:27:10.711Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Xhd9l-Sd1Yd-diwh.png" /></figure><p>Earlier this year, I joined a team that had been tasked with working on some exciting green-field projects to support better flow of data within our organization. Our team took the opportunity to step into the present and adopt some more modern and feature-rich tools, such as GitHub for remote code hosting and GitHub Actions to support our software development workflows. (We were previously using Bitbucket and a self-managed Jenkins instance for continuous integration and delivery.)</p><p>This post shares a few things I learned along the way as we leaned into GitHub and GitHub Actions.</p><h3>Contents</h3><ol><li>More than a continuous integration tool</li><li>Working with workflows</li><li>Code reuse</li><li>Summary</li></ol><h3>More than a continuous integration tool</h3><p>GitHub Actions has been a live product for a few years now, but this year was the first time I used it in a project. Based on many of the blog and video headlines about GitHub Actions I had seen since its launch, I thought GitHub Actions was simply a newer continuous integration and delivery (CI/CD) tool that had tight integration with the GitHub platform. While this is accurate, it isn’t the full story. GitHub Actions is not <strong>just </strong>a CI/CD tool. It can do so much more.</p><blockquote>GitHub Actions is an event-driven workflow automation product for supporting the software development process in the GitHub environment.</blockquote><p>The examples of products most similar to GitHub Actions that readily come to mind are Zapier and IFTTT, but with the events that trigger an automated workflow originating from the GitHub platform. GitHub events are published when we perform activities such as forking a repository, pushing code to a remote branch, or opening a pull request. GitHub Action workflows are configured to listen for specific events and do something. That something could be anything, programmer’s choice.</p><h3>Working with workflows</h3><p>In the context of GitHub Actions, a workflow is an automated process you configure in a GitHub repository. Workflows are configured using YAML in files stored in a .github/workflows/ directory. A workflow is comprised of one or more “actions” that run in response to a GitHub event. Here’s an example of a workflow that builds, tests, and deploys some code when it is merged/pushed to the main branch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/051e5066c3ec0cdd4c8091c2844af5a4/href">https://medium.com/media/051e5066c3ec0cdd4c8091c2844af5a4/href</a></iframe><p>This workflow file would be located at .github/workflows/release.yaml. It starts out with a name for the workflow, “Release,” followed by the event configuration that will cause the configured actions to execute (on code push to the main branch). Below that, the actions to execute are grouped in a job called “release”. This job (the sequence of actions/steps) will execute on an Ubuntu instance. First, an action called checkout (made available by GitHub on the GitHub Actions Marketplace) is used to checkout the repository, then we build, test, and deploy the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*4Rk7c_kSKH2r1-WwECaILQ.png" /><figcaption>Structure of a GitHub Actions workflow (Image credit: Author)</figcaption></figure><p>It’s worth noting that in workflows with multiple jobs, jobs are executed concurrently. However, some amount of sequence can be established by <a href="https://docs.github.com/en/actions/using-jobs/using-jobs-in-a-workflow#defining-prerequisite-jobs">configuring dependencies among jobs</a>. You can also share information among jobs using <a href="https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs#overview">job output variables</a>.</p><h3>Code reuse</h3><p>As you write more workflows across your repositories, you’ll quickly notice that there are common tasks you want to perform in your workflows, or that there is some code duplication. Naturally, your engineering instinct will nudge you to investigate ways to write reusable code for your workflows. There are two main ways to reuse GitHub Actions code: <a href="https://docs.github.com/en/actions/creating-actions/creating-a-composite-action">composite actions</a>, and <a href="https://docs.github.com/en/actions/using-workflows/reusing-workflows">reusable workflows</a>.</p><h4>Composite actions</h4><p>actions/checkout@v3 as shown in the example workflow above is an example of a composite action. GitHub saw that checking out a repository was something many of us would be doing, so they created a composite action called checkout for us to use (and reuse) for that purpose. In this case, we’re using the 3rd version of the checkout action.</p><blockquote>A composite action is a packaged series of tasks to execute that can be reused in GitHub Action workflows.</blockquote><p>When creating composite actions, you need to consider the security risk of script injection. And when using third-party composite actions, you need to consider the security risk that you may be using malicious code. For this reason, you should get your composite actions from trusted sources, such as verified actions on the GitHub Actions Marketplace. You also have the ability to <a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#allowing-select-actions-and-reusable-workflows-to-run">constrain what composite actions and reusable workflows are allowable for use</a> in your repositories through your repository’s settings for added security and stability.</p><h4>Reusable workflows</h4><p>Sometimes, you will write workflows that are so similar, you will begin to wonder if you could have just written it once, parametrize it, and reuse it in other GitHub Action workflows. The answer is yes! Yes, you can. For instance, you may have a release workflow that you want to run for different environments (e.g. dev, stage, prod), each of which have different secrets and input variables such as their environment names and cloud account IDs. The GitHub Actions documentation goes in depth about <a href="https://docs.github.com/en/actions/using-workflows/reusing-workflows#creating-a-reusable-workflow">how to create reusable workflows</a> and some of the limitations on them. Reusable workflows are structurally very similar to standard workflows, with a few added configuration items such as input variables and secrets.</p><h3>Summary</h3><p>GitHub Actions is an event-driven workflow automation product for supporting the software development process in the GitHub environment. It is more than just a tool for continuous integration and delivery. The concepts of workflows and actions are at the heart of the GitHub Actions product. Workflows are configured using YAML to perform a specific set of actions (tasks) in response to an event originating from the GitHub platform. As you write more workflows, code deduplication and reusability will become concerns, and composite actions as well as reusable workflows are excellent tools to address those concerns.</p><blockquote>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to <a href="/service/mailto:bd@akava.io">bd@akava.io</a> and reference this post.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8Yx3N4lsIgMuOdjF.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f9bb06e9c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/adopting-github-actions-more-than-just-a-ci-tool-68f9bb06e9c3">Adopting GitHub Actions // More Than Just a CI Tool!</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exporting Snowflake Query Results]]></title>
            <link>https://medium.com/akava/exporting-snowflake-query-results-abb013a2d29b?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/abb013a2d29b</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 05 Oct 2022 16:17:33 GMT</pubDate>
            <atom:updated>2022-10-05T16:17:33.024Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to download data from Snowflake</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YicDVFyT8BNXNqsJm4OdZw.png" /></figure><p>Snowflake is a cloud data platform that offers solutions to a variety of data management problems faced by modern organizations. Snowflake is most commonly used by organizations as a data warehouse to perform <a href="https://en.wikipedia.org/wiki/Online_analytical_processing">online analytical processing (OLAP)</a>.</p><p>In this respect, Snowflake can be used to solve the same class of problems that other cloud data warehouses such as Google Big Query and Amazon Redshift can solve. However, Snowflake provides its customers with several distinguishing features and capabilities such as multi-cloud flexibility, separation of storage and compute which allows for each to scale independently of the other, and many data privacy, security, and sharing features.</p><p>While Snowflake may be a great solution fit (or not) for your organization, many users will likely encounter challenges extracting data from the platform when necessary. This post highlights a few ways of extracting data from Snowflake should the need arise.</p><h3>Contents</h3><ol><li>Web interfaces</li><li>SnowSQL</li><li>Unloading data with stages</li><li>Using third-party clients</li><li>Summary</li></ol><h3>Web interfaces</h3><p>Many Snowflake users primarily interact with the platform through its web interfaces. Snowflake’s <a href="https://docs.snowflake.com/en/user-guide/ui-using.html">Classic Web Interface</a> was the only option users had for a long time, but Snowflake’s more modern web interface called <a href="https://docs.snowflake.com/en/user-guide/ui-snowsight.html">Snowsight</a> is now concurrently available to users. Snowsight addresses limitations of the Classic Web Interface, and provides additional features such as allowing eligible users to access their <a href="https://docs.snowflake.com/en/user-guide/data-exchange.html">Data Exchange</a>.</p><p>One of the Classic Web Interface limitations that Snowsight addresses is the limit on how much of a query result can be downloaded. In the Classic Web Interface, users can only download the first 100 MB of their query result as a CSV file. In contrast, Snowsight allows users to download the full result of their queries as a CSV file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OIHArimTbuyo61PDm9AoSg.png" /><figcaption>Snowflake Classic Web Interface showing download button and 100MB limitation on downloading query results (Image credit: Author)</figcaption></figure><p>If you have encountered this limitation on the amount of data you can download in the Classic Web Interface, making the switch to Snowsight is the fastest and easiest way to overcome it. Furthermore, while the Classic Web Interface is currently fully supported, Snowflake may deprecate it in the future. Snowsight is the present and (foreseeable) future of Snowflake’s web interface. I highly recommend making the switch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_SKM2qyThM4rIaYXYNbg8Q.png" /><figcaption>Snowsight UI showing download button for query results (Image credit: Author)</figcaption></figure><h3>SnowSQL</h3><blockquote>SnowSQL is the command line client for connecting to Snowflake to execute SQL queries and perform all DDL and DML operations, including loading data into and unloading data out of database tables. (<a href="https://docs.snowflake.com/en/user-guide/snowsql.html">SnowSQL — Snowflake Documentation</a>)</blockquote><p>You can <a href="https://docs.snowflake.com/en/user-guide/snowsql-install-config.html">download, install</a>, and <a href="https://docs.snowflake.com/en/user-guide/snowsql-config.html">configure</a> SnowSQL, Snowflake’s CLI tool for interacting with their data platform to download data from Snowflake. Once SnowSQL is correctly configured, you can query and download results interactively or non-interactively.</p><h4>Non-interactive</h4><p>The following SnowSQL shell command executes the query in the query_all_songs.sql file and outputs the result to a CSV file called output_file.csv in the current local directory.</p><pre>$ snowsql -f query_all_songs.sql -o output_format=csv -o header=true -o timing=false -o friendly=false  &gt; output_file.csv</pre><p>You can learn more about SnowSQL options <a href="https://docs.snowflake.com/en/user-guide/snowsql-use.html">here</a>, but here is a summary of the options used in the above command.</p><pre>-f query_all_songs.sql # specifies file with SQL query to execute<br>-o output_format=csv # output file should be CSV<br>-o timing=false # remove query timing information<br>-o friendly=false # remove startup and exit messages</pre><h4>Interactive</h4><p>To interactively work with SnowSQL, execute the SnowSQL shell command with correct Snowflake credentials, for example (assumes a default connection is configured in the SnowSQL config file):</p><pre>$ snowsql</pre><pre>&gt; !set output_format=csv<br>&gt; !set header=true<br>&gt; !set timing=false<br>&gt; !set friendly=false<br>&gt; !set output_file=output_file.csv</pre><pre># if SQL query is in a file</pre><pre>&gt; !source query_all_songs.sql</pre><pre># if directly writing query</pre><pre>&gt; SELECT * FROM kaggle.dl_course_data_v5.songs;</pre><h3>Unloading data with stages</h3><p>Snowflake’s versatile <a href="https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html">COPY INTO command</a> has a dizzying amount of options that allow users to load and unload data in a variety of ways. In addition to Snowflake’s internal and external STAGEs, and SnowSQL, users can download their query results to an authorized cloud location (using an external stage) or to a local directory (using an internal stage and <a href="https://docs.snowflake.com/en/sql-reference/sql/get.html">SnowSQL’s GET command</a>).</p><p><a href="https://stackoverflow.com/users/8767570/rich-murnane">Rich Murnane</a> provides <a href="https://stackoverflow.com/questions/58632241/how-download-more-than-100mb-data-into-csv-from-snowflakes-database-table">a template for performing the latter</a>. The former varies quite a bit because of the different cloud service providers Snowflake supports, but the <a href="https://docs.snowflake.com/en/user-guide-data-unload.html">Snowflake documentation</a> provides excellent details for this functionality.</p><h3>Using third-party clients</h3><p>Many database uses connect to databases using clients not provided by their database vendor. These third-party clients, which are too many to enumerate or discuss in detail, often provide functionality for downloading query results to a local file.</p><h3>Summary</h3><p>Downloading or extracting data from Snowflake, and especially downloading query results, can seem daunting for the uninitiated. The easiest way to download your data is by using Snowsight, Snowflake’s modern web interface. Users who work more at the command line can use SnowSQL, Snowflake’s CLI tool, while users steeped in a cloud environment may opt to use Snowflake’s COPY INTO command with external stages. Some Snowflake users primarily interact with the platform through their favorite third-party database clients, and they can use whatever functionality provided by the client to download data.</p><h3>References</h3><ol><li><a href="https://docs.snowflake.com/en/user-guide/ui-using.html">Classic Web Interface — Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html">COPY INTO &lt;table&gt; — Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/user-guide/ui-snowsight-query.html#exploring-the-results-table">Querying Data Using Worksheets — Snowflake Documentation. Exporting the Results Table.</a></li><li><a href="https://docs.snowflake.com/en/user-guide/snowsql-use.html">SnowSQL — Snowflake Documentation</a></li><li><a href="https://docs.snowflake.com/en/user-guide-data-unload.html">Unloading Data from Snowflake — Snowflake Documentation</a></li></ol><blockquote>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to <a href="/service/mailto:bd@akava.io">bd@akava.io</a> and reference this post.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8Yx3N4lsIgMuOdjF.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abb013a2d29b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/exporting-snowflake-query-results-abb013a2d29b">Exporting Snowflake Query Results</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple AWS Lambda Deployment with Terraform]]></title>
            <link>https://medium.com/akava/simple-aws-lambda-deployment-with-terraform-1f51c08c6f4?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f51c08c6f4</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 24 Aug 2022 17:23:57 GMT</pubDate>
            <atom:updated>2022-08-25T21:07:51.997Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mg-b3TtTCpERb-PEraxfDw.png" /></figure><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda</a> is an Amazon Web Services (AWS) compute service offering that allows developers to write code in a variety of programming languages and deploy their code on a highly available and completely managed compute infrastructure.</p><p>AWS Lambda functions have become an essential component in many modern cloud applications and systems. Today, developers use AWS Lambda functions to service API requests for web and mobile applications, perform scheduled tasks, and bridge disparate applications and systems to list a few common uses.</p><p>It is therefore critical that good development and deployment practices be used with such a common system component. In another post, we shared <a href="https://medium.com/akava/6-tips-for-working-with-aws-lambda-954f6b40e6a1">6 Tips for Working with AWS Lambda</a> that highlight some of these best practices. One practice that was not discussed in that post was the deployment and change management of AWS Lambda functions and their associated resources using an infrastructure-as-code (IaC) tool like <a href="https://www.terraform.io/">Terraform</a>. This post provides an example of how to do this for a class of AWS Lambda functions I have taken to calling the <strong><em>simple lambda function.</em></strong></p><p>A simple lambda function is one whose functionality is implemented primarily using the built-in modules for a particular programming language. Simple lambda functions may also use the <a href="https://aws.amazon.com/getting-started/tools-sdks/">Amazon SDK</a> which is available in all AWS Lambda runtime environments. Furthermore, simple lambda functions are under 50 MB in size and can be zipped and directly uploaded to AWS Lambda.</p><blockquote>The complete source code used in this post is <a href="https://github.com/onelharrison/terraform-simple-lambda-deployment">available on GitHub</a>.</blockquote><p>The Python code below is for a simple lambda function that takes as input an event object with a text field containing a string value and returns a scrambled version of the input string in a JSON response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dcc2bfc393fbf86c8be230b89166eec/href">https://medium.com/media/2dcc2bfc393fbf86c8be230b89166eec/href</a></iframe><p><strong>Example</strong></p><p>Input event</p><pre>{ &quot;text&quot; : &quot;rice and beans&quot; }</pre><p>Output (response)</p><pre>{ &quot;statusCode&quot; : 200,<br>  &quot;headers&quot; : { &quot;Content-Type&quot; : &quot;application/json&quot; },<br>  &quot;body&quot; : &quot;{\&quot;result\&quot;: \&quot;ircnnes a bdea\&quot;}&quot;</pre><p>We begin by configuring the terraform provider for AWS. The code below assumes that your AWS credentials are available in the environment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81aa041a4c57430d3a4d8eb349cf6696/href">https://medium.com/media/81aa041a4c57430d3a4d8eb349cf6696/href</a></iframe><p>The following terraform code creates a zip archive of the simple lambda function’s source code and uploads it to the AWS Lambda service.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbb6f5efce4ac1c3f8c9378a5d13435b/href">https://medium.com/media/cbb6f5efce4ac1c3f8c9378a5d13435b/href</a></iframe><p>The file begins with a few lines declaring variables and initializing local identifiers that are used throughout. The last few lines of the file create an aws_iam_role for the lambda function. But the core lines in this code are those for the aws_lambda_function resource and the archive_file data source.</p><p>The aws_lambda_function resource is used to provision (as the name suggests) the AWS Lambda function. It utilizes the variables and local identifiers that were initialized and pulls information from the aws_iam_role resource and archive_file data source.</p><p>The archive_file data source creates the zip archive of the simple lambda function’s source code located in the ./aws_lambda_functions/text_scrambler directory. The directory structure below shows how the essential files in this example are organized.</p><pre>.<br>├── aws.tf<br>├── aws_lambda_functions<br>│   └── text_scrambler<br>│       └── main.py<br>└── providers.tf</pre><p>The configuration for the archive_file data source to create the zip archive of the Lambda function’s source code is minimal in this example owing to the simplicity of the function’s code — i.e. the function only has one file in its source directory. However, the archive_file data source is quite flexible and can accommodate archiving a more complex organization of code. For example, you can exclude certain files and set the octal file mode for all the archived files. Check out its <a href="https://registry.terraform.io/providers/hashicorp/archive/latest/docs/data-sources/archive_file">documentation</a> to learn more.</p><p>I encourage you to init , plan , and apply the <a href="https://github.com/onelharrison/terraform-simple-lambda-deployment">terraform code</a> used in this post and play with deploying simple lambda functions using terraform.</p><pre>terraform init<br>terraform plan -var=&quot;env_name=dev&quot;<br>terraform apply -var=&quot;env_name=dev&quot;</pre><h3>Summary</h3><p>Infrastructure-as-code (IaC) tools such as terraform allow us to provision, manage, and change cloud infrastructure with much ease. AWS Lambda functions are a common component in many cloud-based applications and systems, and stand to gain all the benefits associated with IaC when managed using tools like terraform. This post demonstrated how to deploy a class of AWS Lambda functions I call the <strong><em>simple lambda function </em></strong><em>— i.e.</em><strong><em> </em></strong>AWS Lambda functions whose functionalities are implemented primarily using the built-in modules for a particular programming language and optionally the <a href="https://aws.amazon.com/getting-started/tools-sdks/">Amazon SDK</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzynGw-wDZBQJJ8aXxcmTA.png" /></figure><p>You can view the <a href="https://github.com/onelharrison/terraform-simple-lambda-deployment">complete source code</a> used in this post on GitHub.</p><blockquote>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to <a href="/service/mailto:bd@akava.io">bd@akava.io</a> and reference this post.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*8Yx3N4lsIgMuOdjF.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f51c08c6f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/simple-aws-lambda-deployment-with-terraform-1f51c08c6f4">Simple AWS Lambda Deployment with Terraform</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Functional Programming in Python]]></title>
            <link>https://medium.com/akava/functional-programming-in-python-e492f2ad1e37?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/e492f2ad1e37</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Tue, 12 Jul 2022 15:48:24 GMT</pubDate>
            <atom:updated>2022-07-12T15:48:24.709Z</atom:updated>
            <content:encoded><![CDATA[<h4>Essential concepts, patterns, and modules</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pU8l4-u0s1flFQiU.png" /></figure><p>Contents</p><ol><li>What is functional programming?</li><li>Functions, functions, functions</li><li>Immutable data types</li><li>Useful modules</li><li>Conclusion</li><li>References</li></ol><p>Python is a general purpose, multi-paradigm programming language. That means a programmer using Python can write just about any type of program using one or more programming paradigms.</p><p><strong>Programming paradigms</strong> are a manner of programming that emphasize or constrain particular programming practices, patterns, or techniques. In other words, programming paradigms represent an opinionated view of how to write programs.</p><p>Among the paradigms afforded by Python are the imperative, object-oriented, and functional programming paradigms. This article focuses on the last of these, which in recent years has become more popular in industry settings.</p><h3>What is functional programming?</h3><p>Functional programming is a style of programming that centers the use of functions and immutable data types. These functions perform operations on data available to them as <strong>bound variables</strong> (also known as <strong>parameters</strong>), <strong>free variables</strong> (that exist outside the functions’ scope), and <strong>local variables </strong>(defined inside the function).</p><p>Also, these functions are ideally stateless, meaning they do not maintain any internal state that could change the functions’ output, regardless of how often the functions are called. All that to say, the outputs of these functions depend solely on their inputs.</p><p>The emphasis on immutable data types also contributes to this discipline as it prevents problems that stem from functions sharing mutable state such as race conditions.</p><p>Additionally, <strong>side-effects</strong> (which are effects caused by a running program that are external to the program) are treated with much intention. Side-effects are necessary in most useful programs, but they can also present several challenges such as carefully managing effect sequencing. Functions that do not have any side-effects are called <strong>pure functions </strong>while those that produce side-effects are called <strong>impure functions.</strong></p><p>To solve problems using functional programming is to decomposed them into a set of these ideally stateless, pure (and minimally impure) functions that are then composed into a final solution (Kuchling, n.d). The result of programming in the functional style is often highly composable and modular programs that are easy to debug and test (Kuchling, n.d.).</p><h3>Functions, functions, functions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*YbbnH9bxCORtCyLxrF4OsQ.jpeg" /><figcaption>Image credit: Author</figcaption></figure><h4>First-class functions</h4><p>In some languages, functions are merely defined or called. However, languages that enable functional programming support <strong>functions as first-class citizens</strong>. In such languages, functions are ordinary values, which means functions can be stored in variables and passed to or returned from other functions, just as other values like integers and strings can be. The code snippet below demonstrates these very things.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bfdb210222c38d6de1d9f5aa192680b/href">https://medium.com/media/0bfdb210222c38d6de1d9f5aa192680b/href</a></iframe><p>The presence of first-class functions in a language is foundational to its ability to enable functional programming. Keep this in mind while reading the rest of this post as you will notice that many of the language features discussed herein would not be possible without first-class functions.</p><h4>Higher-order functions</h4><p>A <strong>higher-order function</strong> is a function that takes at least one function as an argument, returns a function as its output, or does both. In the code snippet above, the filter and make_adder functions are examples of higher-order functions. In addition to filter, Python has other frequently-used built-in higher-order functions such as map and sorted.</p><p><a href="https://wiki.python.org/moin/PythonDecorators">Python decorators</a> are also syntax sugar over higher-order functions as shown in the code snippet below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77a9a867aa2d912fa3c9e8478353ea41/href">https://medium.com/media/77a9a867aa2d912fa3c9e8478353ea41/href</a></iframe><h4>Anonymous functions</h4><p>An <strong>anonymous function</strong> is a function without a name. In Python, they are created using the <strong>lambda</strong> keyword. Python only allows anonymous function to include a single expression, which befits another name for an anonymous function — <strong>function expression.</strong></p><p>In Python, anonymous functions are often used in conjunction with high-order . The code snippet below shows anonymous functions being used with common built-in higher-order functions in Python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/21634dfcb54148e1902d2f48a41d22e1/href">https://medium.com/media/21634dfcb54148e1902d2f48a41d22e1/href</a></iframe><p>The output of the above snippet is shown below.</p><pre>[&#39;Marie&#39;, &#39;Katherine&#39;, &#39;Ada&#39;]<br>[Person(name=&#39;Ada Lovelace&#39;, age=36)]<br>[Person(name=&#39;Ada Lovelace&#39;, age=36), Person(name=&#39;Katherine Johnson&#39;, age=101), Person(name=&#39;Marie Curie&#39;, age=66)]<br>[Person(name=&#39;Ada Lovelace&#39;, age=36), Person(name=&#39;Marie Curie&#39;, age=66), Person(name=&#39;Katherine Johnson&#39;, age=101)]</pre><h4><strong>Pure vs impure functions</strong></h4><p>As previously mentioned, functions that do not have any side-effects (e.g. I/O operations and global state modification) are called <strong>pure functions </strong>while those that produce side-effects are called <strong>impure functions.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a08b15f042a7cb031a9c6acfebc36dc2/href">https://medium.com/media/a08b15f042a7cb031a9c6acfebc36dc2/href</a></iframe><h4>Predicate functions</h4><p>Predicate functions map their arguments to a true or false value. They have been use multiple times in this post so far. The named is_even function and the lambda p: p.name.startswith(&quot;A&quot;) anonymous function shown in the <em>Higher-order functions</em> and <em>Anonymous functions</em> sections, respectively, are examples of predicate functions.</p><h3>Immutable data types</h3><p>Python has many built-in immutable data types. Most of these are primitive-like data types such as strings, ints, floats, and bool. But there are also immutable collection, container, and sequence data types such as tuples and ranges.</p><p><em>⚠️Warning! Immutable collection/container objects such as tuples may contain mutable objects such as lists, dictionaries, and custom collection/container-like objects. For instance, while a tuple itself may be immutable, the objects it contains may be mutable and can be modified.</em></p><p>Because mutation is minimized in the functional paradigm, it is easy to view programs as performing a series of reproducible data transformations. In the functional style, data-transforming functions and operations that seem to perform mutations typically result in the creation of new objects. For these reason, functional programming languages have historically been criticized for being less performant than their imperative counterparts.</p><p>However, many advancements in optimizing compilers and data structures (such as immutable <a href="https://en.wikipedia.org/wiki/Persistent_data_structure">persistent data structures</a>) have quelled inefficiency concerns for the common programming scenarios, which allows programmers to get all the benefits of using the functional style without fear of significant performance loss.</p><h3><strong>Useful modules</strong></h3><h4>operator</h4><blockquote>The <a href="https://docs.python.org/3/library/operator.html#module-operator">operator</a> module exports a set of efficient functions corresponding to the intrinsic operators of Python… The functions fall into categories that perform object comparisons, logical operations, mathematical operations and sequence operations.<br>- <a href="https://docs.python.org/3/library/operator.html">operator</a></blockquote><p>The operator module provides the standard Python operators such as addition and multiplication as functions. For example, there are the operator.add and operator.mul functions for the addition and multiplication operators, respectively.</p><p>The provided operators-as-functions allow us to use Python’s operators more flexibly and in ways that fit the functional style more closely.</p><h4><strong>functools</strong></h4><blockquote>The <a href="https://docs.python.org/3/library/functools.html#module-functools">functools</a> module is for higher-order functions: functions that act on or return other functions. In general, any callable object can be treated as a function for the purposes of this module.<br>- <a href="https://docs.python.org/3/library/functools.html">functools</a></blockquote><p>As its description highlights, the functools module has some useful functions for working with higher-order functions. The ones I tend to reach for most often include <a href="https://docs.python.org/3/library/functools.html#functools.cache">cache</a>, <a href="https://docs.python.org/3/library/functools.html#functools.reduce">reduce</a>, <a href="https://docs.python.org/3/library/functools.html#functools.partial">partial</a>, and <a href="https://docs.python.org/3/library/functools.html#functools.wraps">wraps</a>.</p><p>Among these, perhaps the most important one to highlight is partial as it enables us to easily perform partial application, a critical technique to grasp in functional programming.</p><blockquote><strong>Partial application</strong> (or <strong>partial function application</strong>) refers to the process of fixing a number of arguments to a function, producing another function of smaller <a href="https://en.wikipedia.org/wiki/Arity">arity</a> [i.e., number of arguments].<br>- <a href="https://en.wikipedia.org/wiki/Partial_application">Wikipedia (Partial application)</a></blockquote><p>The arguments we fix when partially applying a function are those whose values we don’t anticipate will change across several invocations of the function. Consequently, partial application leads to terser code because each subsequent invocation of the partially applied function usually takes up less screen space.</p><p>Additionally, we may want to partially apply a function to give it an arity that would allow us to compose it with other functions.</p><h4>itertools</h4><blockquote>This module implements a number of <a href="https://docs.python.org/3/glossary.html#term-iterator">iterator</a> building blocks inspired by constructs from APL, Haskell, and SML… The [itertools] module standardizes a core set of fast, memory efficient tools that are useful by themselves or in combination… These tools and their built-in counterparts also work well with the high-speed functions in the <a href="https://docs.python.org/3/library/operator.html#module-operator">operator</a> module.</blockquote><p>The itertools module has a lot of functions that allow Python programmers to conveniently and efficiently work with collection and sequence data types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*os7l0aChC0_-olQv.png" /></figure><h3>Conclusion</h3><p>Python is a general purpose, multi-paradigm programming language. Among the many paradigms it supports is the functional programming paradigm, a style of programming that centers the use of functions and immutable data types. This post highlights essential functional programming concepts, patterns, and tools to begin writing more functionally-styled Python.</p><p>Essential concepts and patterns highlighted in this post include higher-order functions, first-class functions, pure and impure functions, predicate functions, and partial application. Built-in modules such as operator , functools , and itertools that fit well with the functional style are also brought to the fore. Third-party libraries such as <a href="https://github.com/pytoolz/toolz/">toolz</a> may also be of interest.</p><p>Finally, if your curiosity is still raging, other topics to investigate include <a href="https://en.wikipedia.org/wiki/Closure_(computer_programming)">closure</a>, <a href="https://en.wikipedia.org/wiki/Currying">currying</a>, <a href="https://en.wikipedia.org/wiki/Lazy_evaluation">lazy</a> and <a href="https://en.wikipedia.org/wiki/Evaluation_strategy#Eager_evaluation">strict</a> evaluation, and immutable <a href="https://en.wikipedia.org/wiki/Persistent_data_structure">persistent data structures</a>.</p><h3>References</h3><ol><li>Kuchling, A. M. (n.d.). <em>Functional Programming HOWTO (v. 0.32)</em>. Python 3 Documentation. Retrieved from <a href="https://docs.python.org/3/howto/functional.html">https://docs.python.org/3/howto/functional.html</a></li><li><a href="https://docs.python.org/3/library/operator.html">Python 3 operator module</a></li><li><a href="https://docs.python.org/3/library/functools.html">Python 3 functools module</a></li><li><a href="https://docs.python.org/3/library/itertools.html">Python 3 itertools module</a></li><li><a href="https://wiki.python.org/moin/PythonDecorators">Python decorators</a></li></ol><blockquote>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to <a href="/service/mailto:bd@akava.io">bd@akava.io</a> and reference this post.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*BVg7pHRxOn6TwrdD.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley technology transformation consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e492f2ad1e37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/functional-programming-in-python-e492f2ad1e37">Functional Programming in Python</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 Tips for Working with AWS Lambda]]></title>
            <link>https://medium.com/akava/6-tips-for-working-with-aws-lambda-954f6b40e6a1?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/954f6b40e6a1</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 04 May 2022 15:16:05 GMT</pubDate>
            <atom:updated>2022-05-04T15:16:05.919Z</atom:updated>
            <content:encoded><![CDATA[<h4>Write and deploy AWS Lambda functions with more confidence</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fbYP68PV0OYxEmm9.jpg" /><figcaption>AWS Lambda (Image credit: Amazon Web Services, Inc.)</figcaption></figure><h3>Contents</h3><ul><li>Introduction</li><li>Tip #1: CPU power and memory scale proportionally</li><li>Tip #2: Simplicity is a virtue</li><li>Tip #3: Mind your concurrency</li><li>Tip #4: Use native handling of partial batch failures</li><li>Tip #5: Necessary runs only</li><li>Tip #6: Avoid using AWS Lambda</li><li>Conclusion</li><li>References</li></ul><h3>Introduction</h3><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda</a> is a highly scalable compute service offered by Amazon Web Services (AWS). It allows developers to run code in a variety of languages on a highly available and completely managed compute infrastructure. It can also run almost any type of application or backend service, but it is best suited to event-based, short-running processes. This is because the execution time for each invocation of a Lambda function is constrained to a <a href="https://aws.amazon.com/about-aws/whats-new/2018/10/aws-lambda-supports-functions-that-can-run-up-to-15-minutes/">maximum of 15 minutes</a>.</p><p>Cost management is an ever-present concern when using cloud service providers, so keep the following in mind as you read further.</p><blockquote>The number of executions and execution time of a Lambda function are its main cost variables. Additionally, the execution time cost for a given Lambda function depends on how much memory is allocated to it. Check the <a href="https://aws.amazon.com/lambda/pricing/">AWS Lambda pricing guide</a> for details.</blockquote><p>This post shares 6 tips for writing and deploying Lambda functions with more confidence. The tips include cost optimization, performance configuration, and operational suggestions.</p><h3>Tip #1: CPU power and memory scale proportionally</h3><p>If you have a Lambda function that is running longer than expected, the most obvious remedy may be to increase its timeout setting so that the function can run to completion. However, if the function is CPU-bound, you can improve its performance and reduce its execution time by increasing its memory size. This is because CPU power for Lambda functions scale proportionally with their allocated memory.</p><p>Alas, increasing a function’s timeout setting remains the only method for ensuring that IO-bound functions, such as those that spend most of their time waiting on responses from databases or APIs, have a chance of completion.</p><h3>Tip #2: Simplicity is a virtue</h3><p>There are multiple advantages to ensuring that your Lambda functions are as simple as necessary. Minimizing cold-start times and making functions more maintainable are two of the most glaring benefits.</p><p>You can use <a href="https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html">Lambda layers</a> to share code across multiple Lambda functions and keep deployment package sizes to a minimum. Lambda layers can make Lambda functions more maintainable as they ensure only unique, essential logic is contained in the function’s body. They also make updating shared logic is easier, since only the layer needs updating to have the effect of a change applied to all functions that use the changed layer.</p><blockquote>If you don’t need a dependency or artifact at runtime, it should not be part of the Lambda function’s deployment package.</blockquote><p>Furthermore, smaller deployment package (and container image) sizes mean faster start up times (cold-start times) for Lambda functions. The golden rule here is, “if you don’t need a dependency or artifact at runtime, it should not be part of the Lambda function’s deployment package.” Smaller package size may also mean easier deployment procedures, since <a href="https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html">zipped packages below 50 MB</a> can be directly upload to AWS Lambda instead of employing Amazon S3 for storing zipped packages or <a href="https://aws.amazon.com/ecr/">Amazon Elastic Container Registry (Amazon ECR)</a> for storing container images.</p><h3>Tip #3: Mind your concurrency</h3><p>Lambda functions can almost instantaneously go from zero to a thousand concurrently running invocations. This is great for scalability in general but can also disastrous for the systems a Lambda function depends on. It is a good practice to set a maximum concurrency limit for Lambda functions. Determining what that number is depends on the AWS resource quotas a Lambda function may affect or the load allowances of services the function depends on (e.g. databases and APIs).</p><p>There is also an upper limit on the number of Lambda function invocations that can be ran concurrently in an AWS account region. A group of one of more Lambda functions with an unset concurrency limit may quickly reach the upper limit and prevent other functions in the account-region from executing.</p><p>You will also want to set appropriate alarms to ensure that the right people are notified when quotas are nearing their limits, anticipated costs are exceeded, and failures prop up.</p><p><em>There are two types of concurrency often discussed regarding Lambda functions, reserved and provisional. The concurrency recommendations made in this section refer to reserved concurrency. Learn more about </em><a href="https://aws.amazon.com/blogs/compute/managing-aws-lambda-function-concurrency/"><em>managing AWS Lambda function concurrency</em></a><em>.</em></p><h3>Tip #4: Use native handling of partial batch failures</h3><p>Amazon SQS is one of the most frequently used event sources for Lambda functions. But until recently, processing a batch of multiple messages from a SQS queues meant introducing more complexity in a Lambda function code’s than desirable. An entire batch of messages needed to be successfully processed by a function for the invocation to be considered a total success. This meant that should any message in a batch fail to be processed, the whole batch would be retried at a later time, which is configurable using the visibility timeout setting. There were workarounds to prevent double processing and other accompanying ills, but none of the workarounds were particularly satisfying. This made adopting a one message per batch policy a highly preferred alternative for many people writing Lambda functions.</p><p>In case you missed the <a href="https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-partial-batch-response-sqs-event-source/">AWS announcement in late 2021</a>, they released an update to the AWS Lambda SQS event source that allows a partial batch response. This ensures that when a partial batch failure occurs, reprocessing is attempted only for the individual messages that were unsuccessfully processed. Learn more about <a href="https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting">reporting batch item failures</a>.</p><h3>Tip #5: Necessary runs only</h3><p>Recall that the main pricing variables for Lambda functions are number of executions and execution time. Therefore, one way of reducing cost is to ensure that there are no unnecessary executions. A Lambda function should only attempt to process requests and events it is capable of handling.</p><p>In late 2021, AWS added the <a href="https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/">ability to filter events for Amazon SQS, DynamoDB, and Kinesis event sources</a>. These filters are applied before a Lambda function is invoked, so if you were previously skipping undesirable events in your Lambda function code, now your function code can be simplified. You also reduce cost by no longer running the function for those undesired events.</p><h3>Tip #6: Avoid using AWS Lambda</h3><p>AWS Lambda is an attractive service for many reasons, but sometimes it is not the appropriate choice. If you need more control over the underlying compute resources your applications or services run on, take a look at <a href="https://aws.amazon.com/ec2/">Amazon Elastic Compute Cloud (Amazon EC2)</a> or <a href="https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html">AWS Elastic Beanstalk</a>. As previously mentioned, AWS Lambda is best suited for applications and services that only need to run for a short while (15 minutes) to service requests and process events. Applications and services that need more time to do that may be better ran on other compute services such as <a href="https://aws.amazon.com/ecs/">Amazon Elastic Container Service (Amazon ECS)</a> or <a href="https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html">AWS Fargate</a>. Also, if the average cold-start time for AWS Lambda functions is not acceptable for your application or service, consider running your application or service in an always-on mode with any of the previous mentioned compute services.</p><h3>Conclusion</h3><p>AWS Lambda is a great choice for creating applications and services that are event-based and require minimal time to service requests and process events. Additionally, it is a highly scalable compute service that requires no management of its underlying compute resources on the part of function programmers. But despite its apparent simplicity, there are many things function programmers must keep in mind when thinking about, writing, and deploying Lambda functions. I covered 6 of these things in this post<strong>, </strong>which generally include tips for cost and configuration optimizations as well operational suggestions. However, several other more concerns and practices were not discussed that you may find discussed in the <a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">AWS Lambda documentation</a> and some of the references. Take a peek and let me know what surprises you!</p><h3>References</h3><ol><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/welcome.html">What is AWS Lambda?</a></li><li><a href="https://aws.amazon.com/about-aws/whats-new/2018/10/aws-lambda-supports-functions-that-can-run-up-to-15-minutes/">AWS Lambda enables functions that can run up to 15 minutes</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-package.html">Lambda deployment packages</a></li><li><a href="https://aws.amazon.com/ecr/">Amazon Elastic Container Registry (Amazon ECR)</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html">Managing Lambda reserved concurrency</a></li><li><a href="https://aws.amazon.com/blogs/compute/managing-aws-lambda-function-concurrency/">Managing AWS Lambda Function Concurrency</a></li><li><a href="https://aws.amazon.com/blogs/architecture/best-practices-for-developing-on-aws-lambda/">Best Practices for Developing on AWS Lambda</a></li><li><a href="https://aws.amazon.com/blogs/compute/working-with-lambda-layers-and-extensions-in-container-images/">Working with Lambda layers and extensions in container images</a></li><li><a href="https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-partial-batch-response-sqs-event-source/">AWS Lambda now supports partial batch response for SQS as an event source</a></li><li><a href="https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting">Using Lambda with Amazon SQS — Reporting batch item failures</a></li><li><a href="https://aws.amazon.com/ecs/">Amazon Elastic Container Service</a></li><li><a href="https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html">What is AWS Fargate?</a></li><li><a href="https://aws.amazon.com/blogs/compute/filtering-event-sources-for-aws-lambda-functions/">Filtering event sources for AWS Lambda functions</a></li><li><a href="https://aws.amazon.com/about-aws/whats-new/2021/11/aws-lambda-event-filtering-amazon-sqs-dynamodb-kinesis-sources/">AWS Lambda now supports event filtering for Amazon SQS, Amazon DynamoDB, and Amazon Kinesis as event sources</a></li></ol><blockquote><em>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to </em><a href="/service/mailto:bd@akava.io"><em>bd@akava.io</em></a><em> and reference this post.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cPsCvp8NFJO6sAzsqOx-UQ.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=954f6b40e6a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/6-tips-for-working-with-aws-lambda-954f6b40e6a1">6 Tips for Working with AWS Lambda</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Visualization with Yellowbrick]]></title>
            <link>https://medium.com/akava/machine-learning-visualization-with-yellowbrick-42e3142c41b9?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/42e3142c41b9</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 20 Apr 2022 15:47:53 GMT</pubDate>
            <atom:updated>2022-05-17T13:41:57.100Z</atom:updated>
            <content:encoded><![CDATA[<h4>Visually explore, tune, and select machine learning models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vbk3Bm0WihD9CmkT.jpg" /></figure><h3>Contents</h3><ul><li>Introduction</li><li>Classification demo</li><li>Conclusion</li><li>References</li></ul><h3>Introduction</h3><p>It is well known that data is a critical part of machine learning (ML) projects — especially, the data that is used as input to ML algorithms. However, there is also the data that gets generated during the model development process that gets a bit less attention.</p><p>While developing models, a lot of data is generated that needs to be distilled to inform future iterations and experiments, as well as select models that are most likely to achieve their desired ends within a set of constraints (e.g. model size, time-to-train, acceptable error rate, etc).</p><p>Practitioners may simply print or tabularize some of this data, but visualizations are very natural way to quickly make sense of much of the data produced during model development. Unfortunately, libraries like matplotlib provide too low-level an interface for us to be highly productive and seaborn is better suited to statistical visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hwhTnVzpe6BwvSxe.png" /><figcaption>Yellowbrick extends the Scikit-Learn API to make model selection and hyperparameter tuning easier. Under the hood, it’s using Matplotlib.</figcaption></figure><p>This post features Yellowbrick, a Python library that presents a high-level interface built atop matplotlib and scikit-learn to help us plot common visualizations relevant to model training, tuning, and evaluation.</p><h3>Classification demo</h3><p>In this section, I demonstrate a highly visual approach to developing ML models using scikit-learn and Yellowbrick by training and evaluating two decision tree classifiers for the <a href="https://archive.ics.uci.edu/ml/datasets/wine">WINE dataset</a> [4] using scikit-learn [5] and Yellowbrick [1].</p><h4>Imports and constant initialization</h4><pre>import numpy as np<br>from sklearn.datasets import load_wine<br>from sklearn.model_selection import (<br>    cross_val_score,<br>    train_test_split<br>)<br>from sklearn.tree import DecisionTreeClassifier<br>from yellowbrick.classifier import confusion_matrix<br>from yellowbrick.model_selection import (<br>    learning_curve,<br>    validation_curve,<br>    cv_scores<br>)<br>from yellowbrick.target import class_balance</pre><pre>RANDOM_STATE = 42<br>CV = 10<br>NJOBS = -1<br>SCORING = &quot;accuracy&quot;</pre><pre>%matplotlib inline</pre><h4>Load the dataset</h4><pre>wine = load_wine()<br>X, y = wine.data, wine.target</pre><h4>Check class balance</h4><pre>class_balance(y)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*ntdq1A-_mcgoAbYsmZgNAQ.png" /><figcaption>Class balance for full WINE dataset (Image credit: Author)</figcaption></figure><p>Yellowbrick’s class_balance function [6] lets us easily visualize a histogram of class frequencies for our dataset. The histogram shows that WINE is a tiny dataset with a meager 178 examples and that its classes are fairly balanced. With this visualization, we can quickly determine the presence (or absence) of class imbalances, which in turn informs our choice of evaluation metrics and whether or not we may need sampling techniques such as oversampling minority classes or under-sampling majority classes.</p><p>Because the WINE classes are fairly balanced, we won’t need any sampling techniques, and the standard accuracy metric is a decent choice for this demonstration.</p><h4>Split data</h4><p>Here we perform a stratified split of the WINE dataset to use 75% of its data for model training and the remaining 25% for testing. Scikit-learn’s train_test_split function performs a stratified split by default, which preserves the proportions of class occurrences between the full dataset and the split data.</p><pre>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.25, random_state=RANDOM_STATE<br>)</pre><p>We can see this at a glance by passing both the training and test labels to the class_balance function.</p><pre>class_balance(y_train=y_train, y_test=y_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*pyvKjuKwPWUJFKdk7bzBYg.png" /><figcaption>Class balance for train and test splits of the WINE dataset (Image credit: Author)</figcaption></figure><h4>Train models</h4><pre>dt = DecisionTreeClassifier(random_state=RANDOM_STATE)<br>dt.fit(X_train, y_train)</pre><p>Decision trees are simple, easy-to-interpret models with many desirable properties such as requiring little data preparation and being able to handle multi-class prediction problems. However, decision tree learners are notorious for learning overly complex trees that tend to result in models that overfit and don’t generalize well.</p><p>We can use a learning curve to get an idea of whether or not the decision tree we learned shows signs of overfitting (or underfitting).</p><blockquote>A learning curve shows the relationship of the training score versus the cross validated test score for an estimator with a varying number of training samples. [2]</blockquote><p>We can use a learning curves to determine if our model is likely to improve with additional data, as well as determine our model’s sensitivity to error due to bias or variance.</p><pre>learning_curve(<br>    estimator=dt,<br>    X=X_train,<br>    y=y_train,<br>    cv=CV,<br>    train_sizes=np.linspace(0.1,1.0,5),<br>    n_jobs=NJOBS,<br>    random_state=RANDOM_STATE,<br>    scoring=SCORING,<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*fCQZnWtWkLWl2L22ZDeDEg.png" /><figcaption>Learning curve for a fully-grown DecisionTreeClassifier (Image credit: Author)</figcaption></figure><p>This learning curve tells us that our decision tree classifier is not sensitive to error due to bias as indicated by the lack of variability in the training scores. It also shows that the model overfits the training data from the outset and continues to do so as more training instances are added. This is expected of fully-grown decision trees and doesn’t tell us the full story of whether or not the model is likely to generalize well when presented with unseen examples.</p><p>The cross validation scores tell us the remainder of the story. In this instance, there is consistent high variability in the cross validation scores as the number of training instances increase, which is indicative of the classifier being sensitive to error due to variance. The training and cross validation scores also appear to be converging at a high point as the cross validation score improves with more training instances.</p><p>The plot suggest that we have a high variance model that may benefit from additional data. We also have the option of using a less complex model of the same type or a different type of model altogether. With any of these options, we would be looking for potential decreases training score and improvement in cross validation score (upward trend with reduced variability).</p><p>We do not have more data to feed our decision tree learner, so we will explore training a simpler model of the same type. One way of reducing the complexity of a decision tree is to prune it. We will try using a pre-pruning strategy — that is, we will preemptively stop the tree from growing to its max depth.</p><p>First, let us view the max depth of the current tree. It’s not a deep tree by any stretch of the imagination, and this should be expected given the size of the WINE dataset.</p><pre>print(dt.tree_.max_depth) # 4</pre><p>Then, we can use a validation curve to plot the influence of a the max_depth hyperparameter on the training and cross validation scores to learn if the model may be underfitting or overfitting for some of max_depth values [3].</p><pre>validation_curve(<br>    estimator=dt,<br>    X=X_train,<br>    y=y_train,<br>    param_name=&quot;max_depth&quot;,<br>    param_range=np.arange(1, 5),<br>    cv=CV,<br>    n_jobs=NJOBS,<br>    random_state=RANDOM_STATE<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*_DrR27xePG8K6YKF6OeFdA.png" /><figcaption>Validation curve for max_depth for DecisionTreeClassifier (Image credit: Author)</figcaption></figure><p>Similar to what we observed on the learning curve, there appears to be consistently high variability in cross validation scores. We also see that, all else held equal, a decision tree with a max_depth of 3 (a simpler model, although by not much) may yield similar results as the fully-grown tree.</p><p>We could also visualize the influence of other hyperparameters such as min_samples_split and min_samples_leaf, but you will find that their influence pales in comparison to that of max_depth for this problem.</p><p>Now, let us learn a decision tree with a max depth of 3 and see how it compares to the fully grown tree with a max depth of 4.</p><pre>dt_md3 = DecisionTreeClassifier(<br>    max_depth=3, random_state=RANDOM_STATE<br>)</pre><pre>learning_curve(<br>    estimator=dt_md3,<br>    X=X_train,<br>    y=y_train,<br>    cv=CV,<br>    train_sizes=np.linspace(0.1,1.0,5),<br>    n_jobs=NJOBS,<br>    show=True,<br>    random_state=RANDOM_STATE,<br>    scoring=SCORING,<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*ks2KkbOTMNogAppKUrvJhA.png" /><figcaption>Learning curve for DecisionTreeClassifier with max_depth=3 (Image credit: Author)</figcaption></figure><p>This learning curve closely resembles the learning curve of the unpruned decision tree. There is some perturbation in the training and cross validation scores, but the curve still points to this new tree exhibiting the same characteristics as the first. The model remains more sensitive to error due to variance as the training and cross validation scores converge at a high point. More training data and a model type that better addresses this variance still remain the indicated best avenues for learning a model that is most likely to generalize well.</p><p>The plots that follow show us just how similar the two learned decision trees are. Their cross validation scores and confusion matrices are very similar. There is not much more that can be done to significantly improve the performance for a decision tree classifier on this dataset.</p><p>Still, the preceding steps can be repeated for different model types (e.g. support vectors and knn) to find a best-performing model.</p><p><strong>Comparing cross validation scores</strong></p><pre>cv_scores(<br>    dt,<br>    X_train,<br>    y_train,<br>    cv=CV,<br>    n_jobs=NJOBS,<br>    scoring=SCORING,<br>    random_state=RANDOM_STATE<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*doK2l8IS4UZENp74336CDA.png" /><figcaption>Cross Validation Scores for fully-grown DecisionTreeClassifier (Image credit: Author)</figcaption></figure><pre>cv_scores(<br>    dt_md3,<br>    X_train,<br>    y_train,<br>    cv=CV,<br>    n_jobs=NJOBS,<br>    scoring=SCORING,<br>    random_state=RANDOM_STATE<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*3Tu6lqjI7HLx3ZA_QpEO2Q.png" /><figcaption>Cross Validation Scores for DecisionTreeClassifier with max_depth=3 (Image credit: Author)</figcaption></figure><p><strong>Comparing confusion matrices</strong></p><pre>confusion_matrix(dt, X_train, y_train, X_test, y_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*xVENXdRSAuKY0Eqw38T_PQ.png" /><figcaption>Confusion Matrix for fully-grown DecisionTreeClassifiers (Image credit: Author)</figcaption></figure><pre>confusion_matrix(dt_md3, X_train, y_train, X_test, y_test)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*ChPlzJw9wKWxJAu3lLO2_g.png" /><figcaption>Confusion Matrix for DecisionTreeClassifier with max_depth=3 (Image credit: Author)</figcaption></figure><h3>Conclusion</h3><p>The data generated during model development is necessary to inform future iterations and experiments. Visualizations are a natural way of making sense of this data, and Yellowbrick is an easy-to-use Python library with a high-level interface for plotting common visualizations relevant to training, tuning, and evaluating models with scikit-learn.</p><p>This post exclusively featured what Yellowbrick calls quick methods (the visualization functions demonstrated), but there is a slightly more verbose object-oriented interface that may afford some more-than-basic interactions. There are also several other visualizations that Yellowbrick affords for a variety of purposes such as regression, clustering, feature analysis, and text modeling. I highly encourage checking out the library’s documentation to learn more if interested.</p><h3>References</h3><ol><li><a href="https://www.scikit-yb.org/en/latest/">Yellowbrick: Machine Learning Visualization</a></li><li><a href="https://www.scikit-yb.org/en/latest/api/model_selection/learning_curve.html">Yellowbrick: Machine Learning Visualization — Learning Curve</a></li><li><a href="https://www.scikit-yb.org/en/latest/api/model_selection/validation_curve.html">Yellowbrick: Machine Learning Visualization — Validation Curve</a></li><li>Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. <a href="https://archive.ics.uci.edu/ml/datasets/wine">WINE data set</a>.</li><li><a href="https://scikit-learn.org/stable/index.html">Scikit-learn: Machine Learning in Python</a></li><li><a href="https://www.scikit-yb.org/en/latest/api/target/class_balance.html">Yellowbrick: Machine Learning Visualization — Class Balance</a></li></ol><blockquote><em>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to </em><a href="/service/mailto:bd@akava.io"><em>bd@akava.io</em></a><em> and reference this post.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cPsCvp8NFJO6sAzsqOx-UQ.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42e3142c41b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/machine-learning-visualization-with-yellowbrick-42e3142c41b9">Machine Learning Visualization with Yellowbrick</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grokking Time Travel in Snowflake]]></title>
            <link>https://medium.com/akava/grokking-time-travel-in-snowflake-7b06aebf156f?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b06aebf156f</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 13:36:07 GMT</pubDate>
            <atom:updated>2022-04-06T13:36:07.282Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQ7t2Ih5R-_c0_rsBaq6Rg.png" /></figure><h3>Contents</h3><ol><li>Introduction</li><li>Overview</li><li>Configuring Time Travel</li><li>Querying/cloning historical data/objects</li><li>Undropping historical objects</li><li>Conclusion</li><li>References</li></ol><h3>Introduction</h3><p>Snowflake is a cloud-based data warehouse similar to others such as Amazon Redshift and Google BigQuery. However, Snowflake differentiates itself quite a bit by supporting multi-cloud configurations, separation of storage and compute, fine-grained role-based access controls among many other data security features, and temporal data access.</p><p>In this post, I provide an overview of <a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html">Time Travel</a>, Snowflake’s feature that enables temporal data access, and highlight some affordances of the feature.</p><h3>Overview</h3><p>As mentioned, Time Travel is Snowflake’s feature that enables temporal data access. Users can access historical data that’s up to 90 days old starting with Snowflake’s Enterprise edition. The lowest edition in their tiered offering is the Standard edition, which allows access to 1-day old historical data.</p><blockquote>Everyone gets to Time Travel! You get to Time Travel!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/0*8Jbi9Pn9G-d_UOj9" /></figure><blockquote>We all get to take advantage of this feature to do things like query historical data, create point-in-time snapshots of our data, and recover from accidental data loss.</blockquote><p>Time Travel is enabled by default for all Snowflake accounts, but the DATA_RETENTION_TIME_IN_DAYS parameter that controls how far back we can access data is initialized at 1. Let’s take a moment to discuss how we can change the default Time Travel configuration.</p><h3>Configuring Time Travel</h3><p>Consider the image below. It shows a linear top-down hierarchy of the Snowflake account, database, schema, and table objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*iadQUCY4f-e8ZJ2L-WVoAQ.png" /><figcaption>Account, database, schema, table linear top-down hierarchy (Image credit: Author)</figcaption></figure><p>The DATA_RETENTION_TIME_IN_DAYS parameter can be configured for each of these objects using an ALTER statement.</p><pre>ALTER [ACCOUNT|DATABASE|SCHEMA|TABLE]<br>SET DATA_RETENTION_TIME_IN_DAYS = &lt;value&gt;;</pre><p>By default, the DATA_RETENTION_TIME_IN_DAYS parameter is set to 1 at the account level, and its value is inherited by the objects lower in the hierarchy that do not have the parameter set for itself or an intermediate ancestor object. The principle is that an object’s data retention period is that of the one explicitly set on it, and if not set, inherited from its nearest ancestor.</p><p>If you are ever not sure what the parameter configuration for a Snowflake object is, you can use the <a href="https://docs.snowflake.com/en/sql-reference/sql/show-parameters.html">SHOW PARAMETERS statement</a>.</p><pre>SHOW PARAMETERS FOR [ACCOUNT|DATABASE|SCHEMA|TABLE] &lt;object_name&gt;;</pre><blockquote>It’s worth noting that there are different types of tables such as temporary, transient, and permanent and the allowable data retention values vary by type.</blockquote><p>Temporary and transient tables can have a data retention value of 0 or 1, while permanent tables may have a value ranging from 0 to 1 (Standard edition) or 90 (Enterprise edition and beyond). Snowflake provides a summary of this information as a <a href="https://docs.snowflake.com/en/user-guide/data-cdp-storage-costs.html#temporary-and-transient-tables">table</a> in their documentation.</p><h3>Querying/cloning historical data/objects</h3><p>Once we have configured the DATA_RETENTION_TIME_IN_DAYS parameter for objects in our Snowflake account and have some data stored, we can start writing queries that fetch historical data or clone historical objects. We can write time sensitive SELECT queries as well as CREATE…CLONE queries by using the AT or BEFORE clause.</p><p>The clauses are appropriately named as they convey precisely what they do. We can query historical data from an object as it was at an exact point in time or right before a point in time. The value we provide to the AT or BEFORE clauses may be a timestamp, offset from the present moment in seconds, or statement ID. These queries will fail, however, if we provide values that exceed the data retention period for an object or any of its children.</p><h4>SELECT queries</h4><p>The structure of a SELECT query with the AT or BEFORE clause is as follows.</p><pre>SELECT *<br>FROM &lt;table&gt;<br>[AT|BEFORE]([TIMESTAMP|OFFSET|STATEMENT] =&gt; &lt;value&gt;);</pre><p>Here’s an example showing a how we might query historical data from a table at the point in time before a query with the specified ID was executed.</p><pre>SELECT name, email<br>FROM users<br>BEFORE(STATEMENT =&gt; &#39;82b3fd4z-3598-255d-116q-fc931111z00r&#39;);</pre><h4>CREATE…CLONE queries</h4><p>The structure of a CREATE…CLONE query with the AT or BEFORE clause is like this.</p><pre>CREATE [DATABASE|SCHEMA|TABLE] &lt;new_object_name&gt; <br>CLONE &lt;existing_object_name&gt;<br>[AT|BEFORE]([TIMESTAMP|OFFSET|STATEMENT] =&gt; &lt;value&gt;);</pre><p>We can use the following query to clone a table as it was 3 days ago.</p><pre>CREATE TABLE users_3days_ago<br>CLONE users<br>AT(OFFSET =&gt; -60*60*24*3);</pre><p>The <a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html#querying-historical-data">Snowflake</a> <a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html#cloning-historical-objects">documentation</a> has a few more examples if you need more to get started.</p><h3>Undropping historical objects</h3><p>We all make mistakes. We hope these mistakes don’t include things like dropping database objects that people and systems rely on, but in the event this does happen, Snowflake allows us to hit undo with the UNDROP statement that allows for the most recent version of a dropped database, schema, or table to be restored.</p><pre>DROP [DATABASE|SCHEMA|TABLE] &lt;object_name&gt;; -- OOPS!<br>UNDROP [DATABASE|SCHEMA|TABLE] &lt;object_name&gt;; -- I&#39;M BACK!</pre><blockquote>Dropped objects stick around in the system for the amount of time for which Time Travel is configured for a given object.</blockquote><p>Additionally, objects can only be restored in the current database or schema so it’s a good practice to specify the database or schema to use before running the UNDROP statement. This is necessary even if an object’s fully-qualified name is used.</p><pre>-- Undrop a database<br>UNDROP DATABASE &lt;database_name&gt;;</pre><pre>-- Undrop a schema<br>USE DATABASE &lt;database_name&gt;;<br>UNDROP SCHEMA &lt;schema_name&gt;;</pre><pre>-- Undrop a table<br>USE DATABASE &lt;database_name&gt;;<br>USE SCHEMA &lt;schema_name&gt;;<br>UNDROP TABLE &lt;table_name&gt;;</pre><p>If you’ve somehow managed to dig further into your mistake and not only dropped a database object but also created a substandard replacement with the same name, you can still recover the dropped object. However, you will have to rename the substandard replacement before undropping the original object.</p><pre>ALTER [DATABASE|SCHEMA|TABLE] &lt;object_name&gt; RENAME &lt;another_name&gt;;<br>/*<br> * ... appropriate USE statements as previously shown<br> */<br>UNDROP [DATABASE|SCHEMA|TABLE] &lt;object_name&gt;;</pre><blockquote>Allowing users to recover from errors and mistakes is an essential design principle and Snowflake nailed it!</blockquote><p>You can read more details about restoring dropped database objects in the <a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html#restoring-objects">Snowflake documentation</a>.</p><h3>Conclusion</h3><p>In this post, I provided an overview of Snowflake’s <a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html">Time Travel</a> feature that allows users to query historical data. I also showed how to configure Time Travel using the DATA_RETENTION_TIME_IN_DAYS parameter on Snowflake account, database, schema, and table objects. SELECT, CREATE…CLONE, and UNDROP statements that leverage Time Travel capabilities were also highlighted.</p><p>While everyone gets to time travel, some of us will be able to time travel further into the past than others due to our subscription tier or storage cost constraints. Check out the Snowflake documentation for details on how <a href="https://docs.snowflake.com/en/user-guide/data-cdp-storage-costs.html">storage costs</a> are affected by Time Travel configurations.</p><h3>References</h3><ul><li><a href="https://docs.snowflake.com/en/user-guide/data-availability.html">Snowflake. Snowflake Time Travel &amp; Fail-safe</a></li><li><a href="https://docs.snowflake.com/en/user-guide/data-cdp-storage-costs.html">Snowflake. Storage Costs for Time Travel and Fail-safe</a></li><li><a href="https://docs.snowflake.com/en/user-guide/data-time-travel.html">Snowflake. Understanding &amp; Using Time Travel</a></li></ul><blockquote><em>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to </em><a href="/service/mailto:bd@akava.io"><em>bd@akava.io</em></a><em> and reference this post.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cPsCvp8NFJO6sAzsqOx-UQ.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b06aebf156f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/grokking-time-travel-in-snowflake-7b06aebf156f">Grokking Time Travel in Snowflake</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Backups To Amazon S3 Using Terraform]]></title>
            <link>https://medium.com/akava/snowflake-backups-to-amazon-s3-using-terraform-4519ddfc0c0c?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4519ddfc0c0c</guid>
            <category><![CDATA[data-recovery]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[backup]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Thu, 03 Mar 2022 13:28:18 GMT</pubDate>
            <atom:updated>2022-03-07T16:43:12.523Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUY9zqJRMP5bH7FXOClisA.png" /></figure><p><strong>Contents</strong></p><ol><li>Introduction</li><li>Prerequisite setup</li><li>Create an Amazon S3 bucket</li><li>Create a storage integration</li><li>Create an external stage</li><li>Create a stored procedure</li><li>Schedule stored procedure execution using tasks</li><li>Recovery</li><li>Conclusion</li><li>References</li></ol><h3>Introduction</h3><p>Snowflake offers multiple layers of native data recovery features such as <a href="https://docs.snowflake.com/en/user-guide/data-availability.html">Time Travel and Fail Safe</a>, which allow querying data up to 90 days old and Snowflake’s team to recover historical data up to 7 days old, respectively. Time Travel also allows us to <a href="https://docs.snowflake.com/en/sql-reference/sql/undrop-table.html">undrop tables</a> in case we mistakenly drop a table, and we can configure <a href="https://docs.snowflake.com/en/user-guide/database-replication.html">database replication</a> across regions.</p><p>Unloading data via Snowflake’s COPY INTO statement to an object store like Amazon S3 is yet another option to consider when implementing a Snowflake data recovery strategy. In this post, we demonstrate how to do just that using Terraform.</p><p>This post is inspired by Chris Herther’s <a href="https://calogica.com/sql/snowflake/dbt/2019/09/09/snowflake-backup-s3.html"><em>How To Backup Snowflake Data to S3 or GCS</em></a><em>, </em>so if you are looking for a SQL-centric approach, I recommend you take a look at his post.</p><h3>Prerequisite setup</h3><h4>Terraform provider configuration</h4><p>The following code snippet shows the terraform provider configuration we are using to interact with Amazon Web Services and Snowflake.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc07835c608406681866c8dbf5fa148a/href">https://medium.com/media/cc07835c608406681866c8dbf5fa148a/href</a></iframe><p>The snowflake.sys_admin provider alias is mainly used to create snowflake resources, the snowflake.security_admin alias to issue GRANTs on those resources, and the snowflake.account_admin alias to perform account-level operations and other operations conveniently executable by the ACCOUNTADMIN snowflake role.</p><p>Furthermore, both the Snowflake and AWS terraform providers are additionally configured using environment variables as shown via the following script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fa83864d2c12445b21624e004602c46/href">https://medium.com/media/4fa83864d2c12445b21624e004602c46/href</a></iframe><p>Once the above environment variables are populated, we can source the file and initialize terraform.</p><pre>source env.sh<br>terraform init</pre><h4><strong>Basic snowflake environment setup</strong></h4><p>Before we configure and perform any kind of backup procedure, we need some Snowflake resources that are typically already defined in an organization. Using the terraform code that follows, we perform operations that lay the foundation off which we will configure backups to Amazon S3.</p><ul><li>CREATE a database named SANDBOX</li><li>CREATE a schema named ACTIVITY</li><li>CREATE tables named USERS and EVENTS</li><li>CREATE a role named SANDBOX_RW for performing read-write operations in the SANDBOX database</li><li>CREATE a virtual warehouse</li><li>Apply appropriate GRANTs to each resource</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/875aaeb148e8e7ad6dcb179e67fc1c17/href">https://medium.com/media/875aaeb148e8e7ad6dcb179e67fc1c17/href</a></iframe><p>After applying the above terraform code, the highlighted resources in the image below should be visible, usable, and queryable in the Snowflake web console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IPg5TBarjUgwfOav6vSf7g.png" /><figcaption>Snowflake web console showing usable and queriable resources (Image credit: Author)</figcaption></figure><p>Run the following statement to add a bit of data to the USERS table so that we will have something to backup. We only insert records into the USERS table to later demonstrate that the COPY INTO &lt;location&gt; FROM &lt;table&gt; statement does not unload empty tables like our EVENTS table.</p><pre>INSERT INTO sandbox.activity.users VALUES (&#39;1&#39;, &#39;Jenkins&#39;);</pre><h3>Create an Amazon S3 bucket</h3><p>We will need some place to store the database backup files, and an object store like Amazon S3 is a cheap enough and reasonable place to store them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8090b840e4ab7585a5b911e8558c61f2/href">https://medium.com/media/8090b840e4ab7585a5b911e8558c61f2/href</a></iframe><p>In the code above, we create an Amazon S3 bucket. We also ensure that versioning is enabled, week old files are transitioned to cheap Amazon Glacier storage, and 30-day-old files are expired. Consider your own versioning, transition, and expiration policies and make the necessary adjustments.</p><p>Next, we create a Snowflake storage integration and external stage to access Amazon S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nobXpk50iJbHTV65.png" /><figcaption><a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html">Snowflake storage integration, external stage, and s3 bucket</a> (Image credit: Snowflake)</figcaption></figure><h3>Create a storage integration</h3><p>The terraform code that follows creates and configures an IAM role with sufficient permissions for Snowflake to interact with the S3 bucket we created.</p><p>It also creates a TOOLS schema in the SANDBOX database that will house Snowflake resources such as storage integrations, stored procedures, and tasks as you will see later on.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd774c1427d56040b0cec3088a120232/href">https://medium.com/media/fd774c1427d56040b0cec3088a120232/href</a></iframe><h4>Inspect the created storage integration</h4><pre>DESCRIBE INTEGRATION SNOWFLAKE_S3_BACKUP;</pre><h3>Create an external stage</h3><p>Here we create an external stage connected to our S3 bucket via the storage integration. The files in the stage will be GZIP compressed CSV files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/060c74a11b22a33a290ee3bc6cc815d6/href">https://medium.com/media/060c74a11b22a33a290ee3bc6cc815d6/href</a></iframe><h4>Inspect the created stage</h4><pre>DESCRIBE STAGE SANDBOX.TOOLS.SNOWFLAKE_S3_BACKUP;</pre><h3>Create a stored procedure</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a0f9f7d923d50d56d17106a95e9d39d/href">https://medium.com/media/8a0f9f7d923d50d56d17106a95e9d39d/href</a></iframe><p>The body of the stored procedure is written in JavaScript as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02b894541ffb552dc71386fd4b30614f/href">https://medium.com/media/02b894541ffb552dc71386fd4b30614f/href</a></iframe><p>There are a few things to note about this stored procedure.</p><ul><li>The capitalization of the DATABASE procedure parameter is required.</li><li>Only base tables are unloaded.</li><li>Unloaded tables are split into one or more files matching the data_.*[.]csv.gz pattern, each of which are at most 100 MB in size per <a href="https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare.html#file-sizing-best-practices-and-limitations">Snowflake’s File Sizing Recommendations</a>.</li><li>Unloaded files contain a header row and are overwritten each time the procedure is executed if the procedure is executed multiple times in a day.</li><li>The COPY INTO statement does not unload any data for empty tables.</li></ul><h4>Inspect the created stored procedure</h4><pre>DESCRIBE PROCEDURE SANDBOX.TOOLS.SPROC_BACKUP_DATABASE(VARCHAR);</pre><h4>Test the created stored procedure</h4><pre>CALL SANDBOX.TOOLS.SPROC_BACKUP_DATABSE(&#39;SANDBOX&#39;);</pre><p>You will know that the stored procedure works if you find the relevant files in the S3 bucket. In this case, there should be files located in S3 that match the following pattern.</p><pre>s3://&lt;bucket&gt;/&lt;YYYY-MM-DD&gt;/&lt;db&gt;/&lt;schema&gt;/&lt;table&gt;/data.*[.]csv.gz</pre><h3>Schedule stored procedure execution using tasks</h3><p>Here we create a TASKADMIN role for managing and operating tasks, a task that calls the stored procedure we created with the SANDBOX database as a parameter on a cron schedule, and update exiting GRANTs to include the TASKADMIN role.</p><p>The code is effectively a terraform translation of the SQL found on <a href="https://docs.snowflake.com/en/user-guide/tasks-intro.html#creating-a-task-administrator-role">Snowflake’s Introduction to Tasks — Creating a Task Administrator Role</a> documentation page.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7c66a9d3a98926eccd3b9b2c140f90e/href">https://medium.com/media/a7c66a9d3a98926eccd3b9b2c140f90e/href</a></iframe><h4>Inspect the created task</h4><pre>USE ROLE TASKADMIN;<br>DESCRIBE TASK SANDBOX.TOOLS.TASK_CALL_SPROC_BACKUP_DATABASE;</pre><h4>View a history of task execution</h4><p>The <a href="https://docs.snowflake.com/en/sql-reference/account-usage/task_history.html">account usage TASK_HISTORY view</a> allows us to view the history of task executions.</p><pre>USE ROLE ACCOUNTADMIN;</pre><pre>SELECT *<br>FROM snowflake.account_usage.task_history<br>ORDER BY completed_time DESC<br>LIMIT 10;</pre><p>It would have been nice to have the ability to manually trigger a task to test that it is configured correctly, but there’s no way to do that (at the time of writing). One workaround is to adjust the specified cron schedule to run the task every minute and review the TASK_HISTORY view and the resulting backup files in S3.</p><p><em>⚠️ If you use this workaround, remember to readjust the cron schedule to something more reasonable to avoid unwanted Snowflake credit usage.</em></p><h3>Recovery</h3><p>We can view the backup files available in our stage — i.e., in the S3 bucket — using the LIST statement. Notice that there aren’t any files for the empty EVENTS table as previously alluded to.</p><pre>LIST @SANDBOX.TOOLS.SNOWFLAKE_S3_BACKUP;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LncHCIB9n7E3pS2dbK_yYw.png" /></figure><p>Here we see that there are backups of the USERS table for 2022–01–14 and 2022–01–15. Let’s query the data in the latest backup.</p><pre><br>SELECT<br>  $1 as ID<br>  , $2 as NAME<br>FROM <a href="http://twitter.com/SANDBOX">@SANDBOX</a>.TOOLS.SNOWFLAKE_S3_BACKUP/2022-01-15/SANDBOX/ACTIVITY/USERS</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lS6I8tiJ2DbVrDrqNbaOYg.png" /></figure><p>We can then create a table to store and inspect the recovered data more closely using a <a href="https://docs.snowflake.com/en/sql-reference/sql/create-table.html">CTAS</a> statement.</p><pre>CREATE TABLE sandbox.activity.users_recovered AS<br>SELECT<br>  $1 as ID<br>  , $2 as NAME<br>FROM <a href="http://twitter.com/SANDBOX">@SANDBOX</a>.TOOLS.SNOWFLAKE_S3_BACKUP/2022-01-15/SANDBOX/ACTIVITY/USERS</pre><p>Having to specify an alias for each column may be tedious for wider tables, so there is a two-step approach we can use that includes a CTL and INSERT statement.</p><pre>CREATE TABLE sandbox.activity.users_recovered LIKE sandbox.activity.users;</pre><pre>INSERT INTO sandbox.activity.users_recovered <br>SELECT $1, $2<br>FROM <a href="http://twitter.com/SANDBOX">@SANDBOX</a>.TOOLS.SNOWFLAKE_S3_BACKUP/2022-01-15/SANDBOX/ACTIVITY/USERS</pre><p>Once we have inspected the recovered table to ensure that things are as expected, we can replace the original table with the recovered table.</p><pre>ALTER TABLE sandbox.activity.users RENAME TO sandbox.activity.users_old;</pre><pre>ALTER TABLE sandbox.activity.users_recovered RENAME TO sandbox.activity.users;</pre><pre>DROP TABLE sandbox.activity.users_old;</pre><h3>Conclusion</h3><p><a href="https://www.youtube.com/watch?v=r4tAyiTgwRw">Some teams</a> have opted to manage a portion of their Snowflake environment via infrastructure-as-code tools such as Terraform. Such a decision may lead to data recovery solutions similar to the one presented in this post.</p><p>However, this method has some limitations such as not accurately reflecting the full database in the backup files since empty tables are not unloaded. (Although, there is <a href="https://community.snowflake.com/s/question/0D50Z00009JKB9gSAH/wanted-unload-empty-file-with-header">a workaround</a> for this should you want to have an unloaded file with only headers for empty tables.)</p><p>It’s also reasonable to expect that you may want to exclude certain schema or particular tables from being unloaded. While the solution presented in this post does not allow for this, you can further parametrize and modify the backup stored procedure to add this functionality.</p><p>Additionally, because the <a href="https://docs.snowflake.com/en/sql-reference/stored-procedures-javascript.html#synchronous-api">Snowflake API for stored procedures is synchronous</a>, an executing query must be completed before another can begin. This means the stored procedure used in this post may be sufficient for backing up a database with relatively small tables while backing up databases with large tables may consume significantly more compute credits than is desirable. A more production-ready solution would involve writing client code that uses a <a href="https://docs.snowflake.com/en/user-guide/conns-drivers.html">Snowflake connector</a> to run the COPY INTO queries asynchronously so that they are executed in parallel.</p><p>Finally, restoring data from backups may not be as straightforward as presented here as there may be schema changes between when a backup was created and the time of recovery. Schema information such as what’s found in the information_schema.columns in each database may be helpful to understand the differences in schema between the backups and current tables. The stored procedure can be modified to unload a database’s information_schema.columns table alongside the other tables in the database as well.</p><p>You may view the <a href="https://github.com/onelharrison/terraform-snowflake-s3-backups">complete source code</a> used in this post on GitHub.</p><h3>References</h3><ul><li><a href="https://www.snowflake.com/blog/snowflake-introduces-javascript-stored-procedures/">Artin Avares. Snowflake. Snowflake Introduces JavaScript Stored Procedures</a></li><li><a href="https://calogica.com/sql/snowflake/dbt/2019/09/09/snowflake-backup-s3.html">Claus Herther. How to Backup Snowflake Data to S3 or GCS</a></li><li><a href="https://www.youtube.com/watch?v=r4tAyiTgwRw">Managing Snowflake Architecture with Terraform and dbt, Ritual</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/sql/create-storage-integration.html">Snowflake. Create Storage Integration</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/sql/create-table.html">Snowflake. Create Table</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/stored-procedures-javascript.html">Snowflake. Working with Stored Procedures in JavaScript</a></li><li><a href="https://docs.snowflake.com/en/user-guide/conns-drivers.html">Snowflake. Connectors and Drivers</a></li><li><a href="https://docs.snowflake.com/en/user-guide/database-replication.html">Snowflake. Database Replication</a></li><li><a href="https://docs.snowflake.com/en/user-guide/data-unload-s3.html">Snowflake. Data Unload S3</a></li><li><a href="https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare.html#file-sizing-best-practices-and-limitations">Snowflake. File Sizing Best Practices and Limitations</a></li><li><a href="https://docs.snowflake.com/en/user-guide/tasks-intro.html#creating-a-task-administrator-role">Snowflake. Introduction to Tasks — Creating a Task Administrator Role</a></li><li><a href="https://docs.snowflake.com/en/user-guide/data-load-s3-config-storage-integration.html">Snowflake. Option 1: Configuring a Snowflake Storage Integration to Access Amazon S3</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/account-usage/task_history.html">Snowflake. TASK_HISTORY View</a></li><li><a href="https://docs.snowflake.com/en/sql-reference/sql/undrop-table.html">Snowflake. Undrop Table</a></li></ul><blockquote><em>Akava would love to help your organization adapt, evolve and innovate your modernization initiatives. If you’re looking to discuss, strategize or implement any of these processes, reach out to </em><a href="/service/mailto:bd@akava.io"><em>bd@akava.io</em></a><em> and reference this post.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cPsCvp8NFJO6sAzsqOx-UQ.png" /></figure><p><em>Onel is a Technology Advisor at </em><a href="https://akava.io/"><em>Akava</em></a><em>, a Silicon Valley consultancy. He is a software engineer with broad experience building data analytics products and platforms in technology verticals such as social media and education technology. In addition to advising the Akava client portfolio on cloud-based data architectures, technologies, and application implementations, Onel is an avid technical writer. When he’s not writing or helping teams make better technology choices, he’s mentoring budding technologists or trying new technologies.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4519ddfc0c0c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/akava/snowflake-backups-to-amazon-s3-using-terraform-4519ddfc0c0c">Snowflake Backups To Amazon S3 Using Terraform</a> was originally published in <a href="https://medium.com/akava">Akava</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Python Recipe for Generating Subsets]]></title>
            <link>https://python.plainenglish.io/a-python-recipe-for-generating-subsets-a4a4e191df3d?source=rss-410e9ddcf6f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4a4e191df3d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Onel Harrison]]></dc:creator>
            <pubDate>Wed, 25 Aug 2021 06:10:00 GMT</pubDate>
            <atom:updated>2021-09-24T03:04:09.380Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently found myself needing to generate subsets of collections. In one situation I needed to generate all combinations of possibly interacting diseases (up to 6 diseases from a set of ~24 diseases) to diagnose a given set of symptoms. And in another, I needed to generate combinations of insufficient input data to assert that a system under test would report an error given the insufficient data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RuVaPtfmX6EdHEdKqDJ1A.png" /><figcaption>Image credit: Author</figcaption></figure><p>At first, I didn’t see these problems as set problems. I saw them exclusively as combination problems, but the set perspective turned out to be the right lens through which to view them.</p><p>The recipe in this article is a succinct procedure for generating subsets that resulted from my mulling over the aforementioned problems and enlisting the help of some Internet friends to put a name to some code I had written — a <a href="https://en.wikipedia.org/wiki/Power_set">power set</a>-producing procedure, which I later generalized to what is shown below.</p><h3>The recipe</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9801b104a34c6d082da093045b5895b7/href">https://medium.com/media/9801b104a34c6d082da093045b5895b7/href</a></iframe><h3>Articles you should watch next</h3><ul><li><a href="https://onelharrison.medium.com/watch-set-basics-in-python-d734ee273e8a">Set Basics in Python</a></li><li><a href="https://onelharrison.medium.com/watch-data-structures-stacks-and-queues-in-python-39ed90724a04">Data Structures, Stacks, and Queues in Python</a></li><li><a href="https://onelharrison.medium.com/watch-demystifying-init-self-in-python-b331e01688e5">Demystifying __init__(self, …) in Python</a></li></ul><h3>More Computing resources</h3><p>Watch videos covering a variety of topics in Computing at <a href="https://oneltalkstech.com/">OnelTalksTech.com</a></p><p><em>More content at </em><a href="http://plainenglish.io/"><strong><em>plainenglish.io</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4a4e191df3d" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/a-python-recipe-for-generating-subsets-a4a4e191df3d">A Python Recipe for Generating Subsets</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>