<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Darren Reading on Medium]]></title>
        <description><![CDATA[Stories by Darren Reading on Medium]]></description>
        <link>https://medium.com/@dreading?source=rss-a2329e0e2a8a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EnWHG4xuauPB-sqWoQaJiw.jpeg</url>
            <title>Stories by Darren Reading on Medium</title>
            <link>https://medium.com/@dreading?source=rss-a2329e0e2a8a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:23:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dreading/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Strategies for cloud cost management and optimisation]]></title>
            <link>https://medium.com/@dreading/strategies-for-cloud-cost-management-and-optimisation-c6e1d1d5d5b1?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6e1d1d5d5b1</guid>
            <category><![CDATA[multi-cloud-management]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cio]]></category>
            <category><![CDATA[saving]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 03:21:37 GMT</pubDate>
            <atom:updated>2024-07-28T03:52:14.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ReWmX-gN-DAP3-T" /><figcaption>Photo by <a href="https://unsplash.com/@andriklangfield?utm_source=medium&amp;utm_medium=referral">Andrik Langfield</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is the first in a multi-part series aimed at how organisations can manage and optimise costs to get the best of their cloud infrastructure. At <a href="https://momenton.com.au/">Momenton</a>, we place a strong focus on leveraging the cloud whilst also seeking to optimise cost management.</p><p>Public cloud adoption continues to accelerate, with cloud spending rising as companies adopt multi-cloud strategies and expand into the cloud. Public cloud is now a significant line item in IT budgets, especially among larger organisations. Cloud spends are expected to increase by half in the next year and are typically over budget¹.</p><p>COVID-19 will further increase cloud use. The extra capacity will be needed to meet increased demand as online usage grows. Migration from data centres to the cloud will accelerate in response to reduced headcount, difficulties accessing data centre facilities and delays in hardware supply chains¹.</p><p>Organisations struggle to handle this growing cloud spend, not taking advantage of strategies to reduce cloud costs. They are wasting an estimated 30 per cent of cloud spend⁶, even though one of the leading cloud initiatives is to optimise the existing use of the cloud and save costs¹.</p><blockquote>Managing cloud spend is one to top challenges that organisations face¹.</blockquote><h3>Managing cloud spend</h3><p>Cloud spending comes from multiple sources, such as virtual machines, container management, and serverless computing. Each cloud provider structures its pricing differently and offers different discount types for savings opportunities¹:</p><ul><li>AWS offers reserved instances, spot instances, Enterprise discounts, Saving Plans, and ad hoc negotiated discounts.</li><li>Azure offers enterprise agreements, reserved instances, hybrid benefits and low-priority VMs.</li><li>Google has committed use discounts, ad-hoc negotiated discounts and preemptible VMs.</li></ul><p>This makes understanding your cloud spending complex and challenging to manage, especially for organisations with multi-cloud strategies.</p><p>Organisations are not taking advantage of all the available discounts. For example, only around half of AWS and Azure users leverage reserved instances¹<em>. </em>Cloud providers have recognised that users want simplified discounting, with AWS now offering a simplified AWS Saving Plan².</p><blockquote>Taking a close look at provider discounts could uncover opportunities to reduce costs.</blockquote><h3>Optimising and Monitoring</h3><p>Most organisations have at least a hundred virtual machines, and it is not uncommon for enterprises to run thousands of virtual machines¹. Although there is a growing trend to use PaaS, compute instances will still be one of the largest cost items for companies. We see that compute contributes 40% to 70% of the overall cloud spend from our experience.</p><blockquote>To achieve cost benefits for compute resources, organisations must optimise or right-size instances by selecting the best instances for their workloads.</blockquote><p>Cloud vendors offer a wide selection of compute, memory, and storage-optimised instances, allowing users to pick and configure the best fit for provisioning their workloads. Further, AWS spot instances can save up to 90% compared to the cost of On-Demand Instances³ or Google’s Preemptible VMs, costing up to 70 per cent less than the company’s standard instances. However, the instances terminate after 24 hours or when the resources are required for other tasks⁴.</p><p>Once provisioned, instances should be tuned for proper performance and optimal usage. The number one cause of companies spending too much on their cloud services is failing to understand how their instances operate. You need to know how your workloads function at a granular level.</p><p>Workloads should be modelled to provide insights into their usage⁵, which requires:</p><ul><li>monitoring compute resources to understand CPU, disk I/O, and memory utilisation,</li><li>monitoring system idle time and underutilisation,</li><li>monitoring network bandwidth utilisation, throughput and response times,</li><li>periodic sampling of data and analysis of workloads, and</li><li>identifying trends to pinpoint cost drivers.</li></ul><p>Cloud vendors provide tools to monitor and evaluate costs and utilisation to identify opportunities for optimisation. Cloud usage reports help you address workload inefficiencies by determining more precisely the memory, virtual CPU cores, or other resources your workloads need. They also provided recommendations for right-sizing. For example, AWS provides tools such as Amazon CloudWatch, AWS Cost Explorer, EC2 Right-Sizing, and AWS Trusted Advisor to monitor and evaluate costs and utilisation to identify opportunities for optimisation.</p><h3>Automation</h3><p>Organisations should use automation to monitor and optimise spend continually. Automated optimisation policies for cloud costs can save time while ensuring organisations monitor their environments consistently to eliminate waste.</p><p>Automated policies can be leveraged to shut down workloads after hours or when not required. For example, a batch processing job with infrequent high utilisation would be suitable to be turned off when inactive, so you are not paying for CPU cycles you do not need.</p><p>Turning off instances is not always feasible. Organisations with geographically distributed teams across different timezones or with volatile infrastructure — causing frequent issues with tearing down and restarting infrastructure — will have drastically reduced productivity, outweighing the gain of turning off infrastructure when not required. Also, Google offers sustained usage discounts, so turning off instances can be more expensive than leaving them on.</p><p>Rather than tearing down infrastructure, using container-as-a-service offerings (AWS ECS/EKS, Azure Container Services or Google GKE) with Kubernetes provides flexibility to scale clusters up and down as required to support always-on workloads and optimise costs.</p><blockquote>Containers offer more flexibility over instances, so container usage can also help control costs because they allow more efficient use of infrastructure¹.</blockquote><p>Kubernetes will optimise resource utilisation by using its bin packaging feature. Kubernetes treats instances as a pool of CPU and memory and effectively allocates resources from this pool, allowing you to focus on your cloud at a high level and lets Kubernetes take care of the hard part of allocating resources.</p><h3>Summary</h3><p>Organisations looking to optimise their spending should:</p><ul><li>take a close look at provider discounts to uncover opportunities to reduce costs,</li><li>use automation to monitor and evaluate costs and utilisation continually,</li><li>use right-size instances by selecting the best instances for workloads,</li><li>use Kubernetes for flexibility and more efficient use of infrastructure.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0kYb_vgR6R5CpAjzYrXIQ.png" /><figcaption>Figure 1: Steps to optimise cloud costs</figcaption></figure><h3>References</h3><p>¹: <a href="https://www.flexera.com/blog/industry-trends/trend-of-cloud-computing-2020">https://www.flexera.com/blog/industry-trends/trend-of-cloud-computing-2020</a></p><p>²: <a href="https://aws.amazon.com/savingsplans/">https://aws.amazon.com/savingsplans/</a></p><p>³: <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html</a></p><p>⁴: <a href="https://cloud.google.com/compute/docs/instances/preemptible">https://cloud.google.com/compute/docs/instances/preemptible</a></p><p>⁵: <a href="https://mc.ai/simplifying-cloud-right-sizing-with-ai-enabled-environment-matching-techniques-a-deepcloud-case/">https://mc.ai/simplifying-cloud-right-sizing-with-ai-enabled-environment-matching-techniques-a-deepcloud-case/</a></p><p>⁶: <a href="https://www.cio.com/article/3228247/right-sizing-public-cloud-instances-can-cut-costs-20-40.html">Right-sizing Public Cloud Instances Can Cut Costs 20%-40%</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6e1d1d5d5b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Slack notifications for Google Cloud Billing]]></title>
            <link>https://medium.com/@dreading/slack-notifications-for-google-cloud-billing-7bf72bcbace?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bf72bcbace</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud-billing]]></category>
            <category><![CDATA[cloud-build]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Sun, 21 Apr 2019 22:58:48 GMT</pubDate>
            <atom:updated>2023-05-20T00:36:32.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sQ0FgJ4zSmOB-57U" /><figcaption>Photo by <a href="https://unsplash.com/@fancycrave?utm_source=medium&amp;utm_medium=referral">Fancycrave</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://cloud.google.com/">Google Cloud Platform</a> gives you easy access to computing resources and services that you can use to build and scale your business. These come at a cost, and it is important to actively monitor how much you are spending to keep your costs under control. You can do this by using <a href="https://console.cloud.google.com/billing">Google Cloud Billing</a> alerts. We are going to show how to use these alerts to send your current budget’s spending to a Slack channel via Slack notifications. This is a great way to help you stick to your budget and allows you to take action when your budget is out of control.</p><p>Budget alerts can be applied to either a billing account or a project. Alerts can be set at a specific amount or matched to the previous month’s spend. The alerts will be sent when spending exceeds a percentage of your budget. Alerts can be configured to send an email or publish an event to a Cloud Pub/Sub. Emails are sent to billing administrators and billing account users once the threshold is reached. The email is a standard template that cannot be customised. While for the Cloud Pub/Sub, an event is published on a topic at regular intervals of around 20 mins. This data in the event contains the billing account id, current threshold limit reached, budget limit and cost accrued so far for the month.</p><p>The Cloud Pub/Sub option gives greater control over how the budget alert is consumed. For example, you can subscribe to the billing alert’s topic to create custom email templates, log the data or send a Slack notification. The one drawback with publishing alerts to Cloud Pub/Sub is that events are regularly triggered, and you may receive a message multiple times as Cloud Pub/Sub only guarantees at-least-once delivery. If you naively send every event to users, around one every 20 minutes, you will quickly inundate them with alerts. We will look at a simple solution for our Slack notifications using BigQuery to only trigger alerts once per budget threshold.</p><h3>Architecture</h3><p>The architecture for the solution is shown below in Figure 1. Billing Alerts raise events that are pushed to a Cloud Pub/Sub topic. A Cloud Function subscribes to the topic and processes the billing alert event. This function saves the data to a BigQuery table and then sends a Slack notification if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zi5V-R8dylkhaWDOUxRNtw.png" /><figcaption>Figure 1: Architecture</figcaption></figure><p>We have also included the deployment pipeline, using Cloud Build to automatically deploy the code changes from GitHub.</p><h3>Setup</h3><p>Before we write the code for our notifications, we first need to set up the billing alerts, create a BigQuery table and set up an access token for Slack.</p><ol><li>Create a new Cloud Pub/Sub topic called <em>billing-alerts</em>. This will topic will be used for the billing alert publisher.</li><li>Navigate to Billing in the Google Cloud Console and <a href="https://cloud.google.com/billing/docs/how-to/budgets">create your budget</a>. When creating your budget in the section “Manage notifications” configure the Cloud Pub/Sub topic <em>billing-alerts </em>that were created in the previous step¹.</li><li>In BigQuery, add a new dataset called <em>billing</em> and create a new table called <em>budget</em>. The schema for the table is shown below:</li></ol><pre>[<br> {<br> “name”: “createdAt”,<br> “type”: “TIMESTAMP”,<br> “mode”: “REQUIRED”<br> },<br> {<br> “name”: “costAmount”,<br> “type”: “NUMERIC”,<br> “mode”: “REQUIRED”<br> },<br> {<br> “name”: “budgetAmount”,<br> “type”: “NUMERIC”,<br> “mode”: “REQUIRED”<br> },<br> {<br> “name”: “budgetName”,<br> “type”: “STRING”,<br> “mode”: “REQUIRED”<br> },<br> {<br> “name”: “threshold”,<br> “type”: “NUMERIC”,<br> “mode”: “REQUIRED”<br> }<br></pre><p>4. Create a new Slack application and set up an access token: refer to <a href="https://api.slack.com/slack-apps">https://api.slack.com/slack-apps</a>. Keep the <em>Bot User OAuth Access Token</em> handy, as you will need it below to configure access from our Cloud Function to Slack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_ZBCU3yBPmKmZUp40rXfOg.png" /><figcaption>Figure 2: Slack Application OAuth &amp; Permissions</figcaption></figure><h3>Sending Notifications</h3><p>We use a cloud function to subscribe to the Billing Alerts Cloud Pub/Sub topic, write the event data to BigQuery and send a notification.</p><p>Navigate to Cloud Function in the Google Cloud Console and create a new Cloud function with the following values:</p><ul><li>Name: slack-billing-notification</li><li>Trigger: Cloud Pub/Sub</li><li>Topic: billing-alerts</li><li>Source code: inline editor</li><li>Runtime: Node.js 8</li><li>Function to execute: notifySlack</li></ul><p>Before adding code for our function, we will set up the BOTTOKEN environment variable that contains <em>Bot User OAuth Access Token</em><strong> </strong>from configuring the Slack Application from Step 4 in the Setup section. In the Advanced Options, add an environment variable for the BOTTOKEN. Copy the value of the <em>Bot User OAuth Access Token.</em></p><p>In the package.json file, add the following dependencies:</p><pre>{<br> &quot;name&quot;: &quot;slack-billing-notifications&quot;,<br> &quot;version&quot;: &quot;0.0.1&quot;,<br> &quot;dependencies&quot;: {<br> &quot;slack&quot;: &quot;^11.0.1&quot;,<br> &quot;@google-cloud/bigquery&quot;: &quot;^2.0.6&quot;,<br> &quot;@google-cloud/pubsub&quot;: &quot;^0.18.0&quot;<br> }<br>}</pre><p>For index.js, enter the following code to create the of our notification function:</p><pre>const slack = require(&#39;slack&#39;);</pre><pre>const {BigQuery} = require(&#39;<a href="http://twitter.com/google">@google</a>-cloud/bigquery&#39;);<br>const bigquery = new BigQuery();</pre><pre>const CHANNEL = &#39;gcp-notifications&#39;;<br>const DATASET = &#39;billing&#39;<br>const TABLE = &#39;budget&#39;;<br>const PROJECT = process.env.GCP_PROJECT;<br>const DATASET_LOCATION = &#39;australia-southeast1&#39;;<br>const BOT_ACCESS_TOKEN = process.env.BOTTOKEN;</pre><pre>exports.notifySlack = async (data, context) =&gt; { <br>    // code to send notification to slack<br>};</pre><p>After we import the libraries for Slack and BigQuery, we define the constants for the function. Notably, we define PROJECT as the inherited GCP_PROJECT environment variable and BOTTOKEN as the eponymous environment variable defined for our Cloud Function above.</p><p>Let’s flesh out the implementation details of our <em>notifySlack</em> function. The <em>notifySlack</em> function is configured to be called when our Cloud Pub/Sub topic receives an event. The event data is passed in the first argument as a Base64 encoded JSON string. We parse this data and perform some basic formatting:</p><pre> const pubsubMessage = data;<br> const pubsubData = JSON.parse(Buffer.from(pubsubMessage.data, &#39;base64&#39;).toString());<br> const formatter = new Intl.NumberFormat(&#39;en-US&#39;, {style: &#39;currency&#39;, currency: &#39;USD&#39;, minimumFractionDigits: 2})<br> const budgetId = pubsubMessage.attributes.budgetId;<br> const costAmount = formatter.format(pubsubData.costAmount);<br> const budgetAmount = formatter.format(pubsubData.budgetAmount);<br> const budgetName = pubsubData.budgetDisplayName;<br> const createdAt = new Date().toISOString();<br> let threshold = (pubsubData.alertThresholdExceeded*100).toFixed(0);<br>  <br> if (!isFinite(threshold)){<br>   threshold = 0;<br> }</pre><p>Save this parsed data to our BigQuery table:</p><pre> const rows = [{createdAt: createdAt, <br>                    costAmount: pubsubData.costAmount, <br>                    budgetAmount:pubsubData.budgetAmount, <br>                    budgetId: budgetId,<br>                    budgetName: budgetName,<br>                    threshold: threshold}]</pre><pre>await bigquery.dataset(DATASET).table(TABLE).insert(rows);</pre><p>Once we have saved our data, we then check if we have any other data for this budget threshold. We count the number of events that have been saved for the current month, matching the budget and threshold. If this count is only 1, then this is the first event for the threshold for the month, and we need to send a notification, otherwise, we can stop processing and exit the function.</p><pre>const query = `SELECT count(*) cnt<br>                    FROM \`${PROJECT}.${DATASET}.${TABLE}\`<br>                    WHERE createdAt &gt;  TIMESTAMP( DATE(EXTRACT(YEAR FROM CURRENT_DATE()) , EXTRACT(MONTH FROM CURRENT_DATE()), 1)) <br>                    AND Threshold = ${threshold} <br>                    AND BudgetId = &#39;${budgetId}&#39;<br>                    `;</pre><pre>const options = {<br>        query: query,<br>        location: DATASET_LOCATION,<br>    };</pre><pre><strong>const </strong>[job] = <strong>await </strong>bigquery.createQueryJob(options);<br><br>// Wait for the query to finish<br><strong>const </strong>[results] = <strong>await </strong>job.getQueryResults();<br><br><strong>if </strong>(results.length &gt; 0 &amp;&amp; results[0].cnt &gt; 1 ){<br>    <strong>return</strong>;<br>}</pre><p>The last block of code posts a simple message to Slack. We add a 🔥 emoticon if 90% of the budget has already been used. Even though the composition of our message is relatively simple, Slack notifications support a rich set of features².</p><pre>const emoticon = threshold &gt;= 90 ? &#39;:fire:&#39; : &#39;&#39;;</pre><pre>notification = `${emoticon} ${budgetName} \nThis is an automated notification to inform you that your billing account has exceeded ${threshold}% of the monthly budget of ${budgetAmount}.\nThe billing account has accrued ${costAmount} in costs so far for the month.`</pre><pre>const res = await slack.chat.postMessage({<br>        token: BOT_ACCESS_TOKEN,<br>        channel: CHANNEL,<br>        text: notification<br>    });</pre><p>The completed code listing is available on <a href="https://github.com/momenton/gcp-billing-notifications">GitHub</a>.</p><h3>Testing</h3><p>Once we have saved index.js, we can test our function. In our Cloud Function, select the <em>Testing </em>tab. This displays an input where we can enter a JSON string to test our function.</p><p>We will test our function by triggering a 50% budget threshold alert. First we need to Base64³ encode our test data:</p><pre>{<br> “costAmount”: 501,<br> “budgetAmount”: 1000,<br> “budgetDisplayName”: “test”,<br> “alertThresholdExceeded”: 0.50<br> }</pre><p>Once the above data is encoded we can create the JSON to send to our function:</p><pre>{<br> “data”: “ew0KCQkiY29zdEFtb3VudCI6IDUwMSwNCgkJImJ1ZGdldEFtb3VudCI6IDEwMDAsDQoJCSJidWRnZXREaXNwbGF5TmFtZSI6ICJ0ZXN0IiwNCgkJImFsZXJ0VGhyZXNob2xkRXhjZWVkZWQiOiAwLjUwDQoJfQ==”,<br> “attributes”: {<br>       “budgetId”: “test”<br> }<br>}</pre><p>Note the value for the data is our Base64 encoded test data. Click the <em>Test the Function</em> button. This will execute the Cloud Function with the test data. If successful, you will see a Slack notification appear in your Slack channel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*Z6vSdhS_gU_kXRpuFHIJ2Q.png" /><figcaption>Figure 3: Slack notification for test data</figcaption></figure><p>Review the Stackdriver logs to debug any issues if the notification does not appear.</p><h3>Deployment</h3><p>Up to this point, our code has been manually deployed. This is less than ideal as we must remember to deploy any changes we make in our code repository. It would be better to automatically deploy our code when it changes in GitHub. We will use Cloud Build to achieve this.</p><p>For Cloud Build, we need to create a configuration file that will define the steps that need to be executed when the code in the repository is updated. For our Cloud Function, we just need the following steps:</p><ol><li>Run npm install,</li><li>Run npm test, and</li><li>Deploy our cloud function.</li></ol><p>The configuration file, CloudBuild.yaml, for these steps, is:</p><pre># CloudBuild.yaml<br>steps:<br>- name: &#39;gcr.io/cloud-builders/npm&#39;  <br>  args: [&#39;install&#39;]<br>- name: &#39;gcr.io/cloud-builders/npm&#39;  <br>  args: [&#39;test&#39;]<br>- name: &#39;gcr.io/cloud-builders/gcloud&#39;  <br>  args: [&#39;beta&#39;,          <br>         &#39;functions&#39;,          <br>         &#39;deploy&#39;,          <br>         &#39;slack-billing-notification&#39;,          <br>         &#39;--entry-point=notifySlack&#39;,        <br>         &#39;--source=.&#39;,          <br>         &#39;--runtime=nodejs8&#39;,          <br>         &#39;--trigger-topic=billing-alert&#39;]</pre><p>With our Cloud Builder configuration file, we can now create a trigger to deploy our code when it changes in GitHub. In the Google Cloud Console, select <em>Cloud Build &gt; Triggers</em>. Click the <em>Add Trigger</em> button to display the first step for creating a trigger.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*g5PQkZb0nCAbwKXJzEtO6w.png" /><figcaption>Figure 4: Selecting a source for Cloud Build trigger</figcaption></figure><p>Since we are storing our code in the GitHub repository <em>momenton/gcp-billing-notifications</em>, we select GitHub as the source and select the checkbox for the consent. Google with authenticate with the repository service and display a list of repositories for your user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*osodKeMGJtxab96oaqAUQw.png" /><figcaption>Figure 5: Selecting a repository for Cloud Build trigger</figcaption></figure><p>Select the code repository and click the <em>Continue</em> button to display the trigger settings. In <em>Build Configuration</em>, select the <em>Cloud Build configuration</em> file option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*fd0EZ4DJp1Rh8_up9oZ1MA.png" /><figcaption>Figure 6: Selecting build configuration for Cloud Build trigger</figcaption></figure><p>Click the <em>Create Trigger</em> button to finish the process.</p><p>We can test that the trigger is working by pushing a commit to GitHub and then navigating to <em>Cloud Build -&gt; Build History</em>. If successful, you will see the build details displayed; see Figure 7 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BIVTxzdTHm84mABQfKv2LQ.png" /><figcaption>Figure 7: Cloud Build history</figcaption></figure><h3>Summary</h3><p>We have shown how to use Billing Alerts to send your current budget’s spending to a Slack channel via Slack notifications, using BigQuery to only trigger alerts once per budget threshold.</p><p>We can extend this example to take actions to enforce cap costs and stops usage for a Google Cloud project by disabling billing⁴. This will cause all Google Cloud services to terminate non-free tier services for the project.</p><h3>Notes:</h3><p>¹: To be able to create billing alerts, you must be a billing administrator. For information about billing administrators and billing permissions, see <a href="https://cloud.google.com/billing/docs/how-to/billing-access">Overview of Access Control</a>.</p><p>²: Check out <a href="https://api.slack.com/messaging/composing">https://api.slack.com/messaging/composing</a> for an overview of what is possible for composing Slack messages.</p><p>³: You can easily Base64 encode this data using one of the online encoding sites, such as <a href="https://www.base64encode.net/">https://www.base64encode.net/</a>.</p><p>⁴: It is recommended that if you have a hard funds limit, you set your budget below your available funds to account for delays. Due to usage latency from when a resource is used to the time that the activity is billed, additional costs might be incurred for usage that hasn’t arrived when all services are stopped.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bf72bcbace" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Typesetting neural network diagrams with TeX]]></title>
            <link>https://medium.com/@dreading/typesetting-neural-network-diagrams-with-tex-4920b6b9fc19?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/4920b6b9fc19</guid>
            <category><![CDATA[latex]]></category>
            <category><![CDATA[typesetting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[tex]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 13:40:42 GMT</pubDate>
            <atom:updated>2023-05-20T00:33:57.272Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*32nbIa1_OibDhaAo" /><figcaption>Photo by <a href="https://unsplash.com/@raphaelphotoch?utm_source=medium&amp;utm_medium=referral">Raphael Schaller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://en.wikipedia.org/wiki/TeX">TeX</a> is the <em>lingua franca</em> of scientific publishing. It is powerful and precise, a 70’s throwback that is a testament to its creator Donald Knuth&#39;s genius. Whilst TeX is great, it can be very complex once you start doing more than just simple typesetting, and there is not a lot of documentation to help.</p><p>I wanted to use TeX to typeset some diagrams of neural networks for an upcoming paper. Nothing too complex, a few symbols and a number of circles with directed lines connecting them; see Figure 1 below. I managed to ‘TeX’ the diagrams and was happy with the results, however, it was a non-trivial process. This story is to share the solution and help out anyone looking to typeset neural networks with TeX.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*KTI79d_FQKgv7MLSTHF9rg.jpeg" /><figcaption>Figure 1: Diagrams of neural network layers and pipeline</figcaption></figure><p>I used the <a href="http://www.texample.net/media/pgf/builds/pgfmanualCVS2012-11-04.pdf"><strong><em>TikZ</em></strong></a> package to typeset the diagrams. Tikz is a powerful but complex graphics package for TeX. It is a set of high-level macros built with PGF, a low-level language for producing vector graphics from geometric/algebraic descriptions. The <a href="http://mirror.lagoon.nc/pub/ctan/graphics/pgf/base/doc/pgfmanual.pdf">manual</a> for TikZ/PGF is around 1280 pages, and this <a href="http://www.texample.net/tikz/examples/all/">gallery</a> has a number of examples of what is possible with TikZ.</p><p>To get started with TikZ, you need to instruct TeX to use the Tikz package and load Tikz libraries in the document preamble:</p><pre>\usepackage{tikz}<br>\usetikzlibrary{matrix,chains,positioning,decorations.pathreplacing,arrows}</pre><p>Once the preamble is configured, you can declare <em>tikzpicture</em> environments with the description of the diagram to be drawn. The <em>tikzpicture</em> environment can be enclosed inside a figure or similar environment.</p><p>TikZ offers two libraries that make positioning easier when arranging a large number of nodes: the matrix library and the chains library. The first allows you to arrange multiple nodes in rows and columns, while the second is mostly useful for creating “chains of nodes” and, more generally “flows”.</p><p>Listing 1 below contains the code to generate the first diagram. We use the matrix library to build a 10 by 3 matrix to hold the nodes. There are 3 columns, one for each of the layers and 10 rows; one for the headings, 5 for neurons in the first column and four blanks for spacing. We use ‘|[clear]|’ to remove the default <em>net</em> styling on the matrix cells. After the matrix is defined, we draw lines between the neurons by using, for example:</p><pre>draw[-&gt;] (mat-\ai-1) — (mat-\aii-2);</pre><p>This draws a directed arrow <strong><em>-&gt;</em></strong> between the matrix’s cells in row \ai column 1 and row \aii column 2. Here \ai and \aii are loop variables with index references for the matrix being 1-based.</p><pre>% Listing 1: Tex for neural network layers</pre><pre>\begin{tikzpicture}[<br>     % define styles <br>     clear/.style={ <br>         draw=none,<br>         fill=none<br>     },<br>     net/.style={<br>         matrix of nodes,<br>         nodes={ draw, circle, inner sep=10pt },<br>         nodes in empty cells,<br>         column sep=2cm,<br>         row sep=-9pt<br>     },<br>     &gt;=latex<br>]</pre><pre>% define matrix mat to hold nodes</pre><pre>% using net as default style for cells<br>\matrix[net] (mat)<br>{<br>% Define layer headings<br>|[clear]| \parbox{1.3cm}{\centering Input\\layer} <br>    &amp; |[clear]| \parbox{1.3cm}{\centering Hidden\\layer} <br>    &amp; |[clear]| \parbox{1.3cm}{\centering Output\\layer} \\<br>         <br>$\alpha_{0}^{0}$  &amp; |[clear]|        &amp; |[clear]| \\<br>|[clear]|         &amp; $\alpha_{0}^{1}$ &amp; |[clear]| \\<br>$\alpha_{1}^{0}$  &amp; |[clear]|        &amp; |[clear]| \\<br>|[clear]|         &amp; |[clear]|        &amp; |[clear]| \phantom{$a_{0}^{0}$} \\<br>$\alpha_{2}^{0}$  &amp; $\alpha_{1}^{1}$ &amp; $\alpha_{0}^{2}$ \\<br>|[clear]|         &amp; |[clear]|        &amp; |[clear]|  \phantom{$a_{0}^{0}$} \\<br>$\alpha_{3}^{0}$  &amp; |[clear]|        &amp; |[clear]| \\<br>|[clear]|         &amp; $\alpha_{2}^{1}$ &amp; |[clear]| \\<br>$\alpha_{4}^{0}$  &amp; |[clear]|        &amp; |[clear]| \\ <br>};</pre><pre>% left most lines into input layers<br>\foreach \ai in {2,4,...,10}<br>    \draw[&lt;-] (mat-\ai-1) -- +(-2cm,0);</pre><pre>% lines from a_{i}^{0} to each a_{j}^{1}<br>\foreach \ai in {2,4,...,10} {<br>    \foreach \aii in {3,6,9}<br>        \draw[-&gt;] (mat-\ai-1) -- (mat-\aii-2);<br>        }</pre><pre>% lines from a_{i}^{1} to a_{0}^{2}<br>\foreach \ai in {3,6,9}<br>  \draw[-&gt;] (mat-\ai-2) -- (mat-6-3);<br>    <br>% right most line with Output label<br>\draw[-&gt;] (mat-6-3) -- node[above] {Output} +(2cm,0);</pre><pre>\end{tikzpicture}</pre><p>The output for the above code is shown in Figure 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*7rZtnDDnYqoocQMUK0NBGg.png" /><figcaption>Figure 2: Output for neural network layers diagram</figcaption></figure><p>Listing 2 contains the code to generate the second diagram. As for the first diagram, we start by defining the styles. However, rather than using the matrix library, we will use Tikz chain’s library and create 3 chains for each path or flow through the pipeline.</p><pre><br>% Listing 2: Tex for neural network pipeline</pre><pre>\begin{tikzpicture}[<br>    % define styles    <br>    init/.style={ <br>         draw, <br>         circle, <br>         inner sep=2pt,<br>         font=\Huge,<br>         join = by -latex<br>    },<br>    squa/.style={ <br>        font=\Large,<br>        join = by -latex<br>    }<br>]</pre><pre>% Top chain x1 to w1</pre><pre>\begin{scope}[start chain=1]<br>    \node[on chain=1] at (0,1.5cm)  (x1) {$x_1$};<br>    \node[on chain=1,join=by o-latex] (w1) {$w_1$};<br>\end{scope}</pre><pre>% Middle chain x2 to output</pre><pre>\begin{scope}[start chain=2]<br>    \node[on chain=2] (x2) {$x_2$};<br>    \node[on chain=2,join=by o-latex] {$w_2$};<br>    \node[on chain=2,init] (sigma) {$\displaystyle\Sigma$};<br>    \node[on chain=2,squa,label=above:{\parbox{2cm}{\centering Activation\\ function}}]   {$f_{act}$};<br>    \node[on chain=2,squa,label=above:Output,join=by -latex] {$y_{out}$};<br>\end{scope}</pre><pre>% Bottom chain x3 to w3</pre><pre>\begin{scope}[start chain=3]<br>    \node[on chain=3] at (0,-1.5cm) <br>    (x3) {$x_3$};<br>    \node[on chain=3,label=below:Weights,join=by o-latex]<br>    (w3) {$w_3$};<br>\end{scope}</pre><pre>% Bias</pre><pre>\node[label=above:\parbox{2cm}{\centering Bias \\ $b$}] at (sigma|-w1) (b) {};</pre><pre>% Arrows joining w1, w3 and b to sigma</pre><pre>\draw[-latex] (w1) -- (sigma);<br>\draw[-latex] (w3) -- (sigma);<br>\draw[o-latex] (b) -- (sigma);</pre><pre>% left hand side brace<br>\draw[decorate,decoration={brace,mirror}] (x1.north west) -- node[left=10pt] {Inputs} (x3.south west);</pre><pre><br>\end{tikzpicture}</pre><p>The output for the above code is shown in Figure 3:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*C1PlBAP8sLSp3YSjeNhZwg.png" /><figcaption>Figure 3: Output for neural network pipeline diagram</figcaption></figure><p>The complete code is available on GitHub: <a href="https://github.com/dreading/tex-neural-network">https://github.com/dreading/tex-neural-network</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4920b6b9fc19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why are deeper neural networks harder to train?]]></title>
            <link>https://medium.com/@dreading/why-are-deeper-neural-networks-harder-to-train-587edf422ac2?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/587edf422ac2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-neural-networks]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 02:11:10 GMT</pubDate>
            <atom:updated>2023-05-20T00:38:33.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KrAqH8kOuZRrZAkb" /><figcaption>Photo by <a href="https://unsplash.com/@craftedbygc?utm_source=medium&amp;utm_medium=referral">Green Chameleon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Deep neural networks are an important class of neural networks that have been applied to numerous machine learning areas, such as natural language processing [WE:2016], computer vision [KR:2012] and speech recognition [HI:2012]. Training of such networks is often successfully performed by minimising a high-dimensional non-convex objective function. In a theoretical sense, we have only scratched the surface of this optimisation problem, and a number of important questions remain to be proved about its behaviour.</p><p>A recent paper, “<a href="https://arxiv.org/pdf/1811.03804.pdf">Gradient Descent Finds Global Minima of Deep Neural Networks</a>” by Du, Lee, Li, Wang and Zhai [DU:2018] sheds light on two unexplained behaviours of deep neural networks:</p><ul><li>you can achieve zero training loss for random initialised first-order methods, even if the labels are arbitrary, and</li><li>deeper networks are harder to train.</li></ul><p>Du et al.’s paper consider three deep neural network architectures:</p><ul><li>multilayer fully-connected neural networks,</li><li>deep residual network (ResNet)*, and</li><li>convolutional ResNet.</li></ul><p>They use a randomly initialised gradient descent algorithm to find the global minimiser of the empirical loss for learning. Their paper focuses on proving how much over-parameterisation is needed to ensure the global convergence of gradient descent. This provides a necessary condition for zero training loss for the three specified architectures and as well as an understanding of why deeper networks are harder to train.</p><h4>Zero training loss for random initialised first-order methods</h4><p>It is well-known that using random initialised first-order methods, like gradient descent for deep learning, can achieve zero training loss, even for arbitrary labels [ZH:2016]. Over-parameterisation is the generally agreed reason for this, since if the neural network has a sufficiently large capacity, it is possible for the neural network to fit all the training data. Highly over-parameterised architectures are common in practice, such as Wide Residual Networks that have a factor of one hundred times more parameters than the number of training data [ZA:2016].</p><p>Theorems 3.1, 4.1 and 5.1 [DU:2018] provide sufficient conditions for guaranteeing that gradient descent achieves zero training loss time for deep over-parameterised neural networks for each of the three architectures. These theorems state that if the width is large enough and the step size is set appropriately, then the gradient descent will converge to the global minimum with zero loss at a linear rate.</p><p>Du et al. require the following conditions:</p><ul><li>the last layer’s Gram matrix is strictly positive definite,</li><li>every two by two sub-matrix of every layer for multilayer fully-connected neural networks and the first layer for ResNet, has a lower bounded eigenvalue, and</li><li>the activation function is Lipschitz and smooth.</li></ul><p>These are relatively weak assumptions. The first condition provides a non-degeneracy condition on the matrix. The second condition is a stability assumption guaranteeing that if the width is large, the Gram matrix at the initialisation phase will be close to the population Gram matrix. The last condition holds for many activation functions and allows for non-linear functions such as soft-plus.</p><h4>Deeper networks are harder to train</h4><p>Heuristically, we know deeper networks are harder to train. Various methods have been proposed to deal with this problem. He et al. [HE:2016] proposed the deep residual network (ResNet) architecture. ResNet uses a randomly initialised first-order method to train neural networks with an order of magnitude more layers. Du et al. note that “Theoretically, Hardt and Ma [HA:2016] showed that residual links in linear networks prevent gradient vanishing in a large neighbourhood of zero, but for neural networks with non-linear activations, the advantages of using residual connections are not well understood.” [DU:2018]. Why does using residual connections in a deep residual network (ResNet) architecture offer a better convergence to the fully-connected feedforward networks?</p><p>The answer to this question falls out of Du et al.’s analysis. The bounds provided in Theorems 3.1, 4.1 and 5.1 [DU:2018] depend on the number of neurons per layer. The nature of this dependency differs between the different architectures; feed forward networks scale exponentially with depth while ResNet only requires polynomially scaling. For feed-forward networks, “ the exponential dependency results from the amplification factor of multilayer fully-connected neural network architecture” [DU:2018]. These theorems clearly demonstrate the advantage of using residual connections and the underlying complexity of the multilayer fully-connected neural network.</p><h4>Summary</h4><p>Du et al.’s paper [DU:2018] proves that gradient descent on deep over-parametrised networks can obtain zero training loss with some relatively weak assumptions on the neural network. They also provide an explanation of why deeper networks are harder to train and why ResNet is better than a multilayer fully-connected neural network in terms of the convergence of the training loss.</p><p>In practice, stochastic gradient descent is more likely to be used instead of gradient descent. However, we note that Du et al. expect that their analysis can be extended to stochastic gradient methods with similar convergence rates.</p><p>They focus on training loss and do not look at test loss. It is important to be able to prove that gradient descent can also find low test loss solutions. Unfortunately, existing research in this area is limited, and this remains an open question.</p><h4>Notes</h4><p>* Du et al.’s ResNet architecture is modified, without loss of generality, to skip-connections at every layer, in contrast to the standard ResNet that skip-connections of every two layers. This provides an easier analysis, and the results in the paper can be generalised to standard architectures.</p><h4>References</h4><p>[DU:2018] Du Simon S.,Lee, Jason D., Li Haochuan, Wang Liwei, Zhai Xiyu: <strong>Gradient Descent Finds Global Minima of Deep Neural Networks</strong> <em>arXiv preprint arXiv:1811.03804., </em>2018.</p><p>[HA:2016] Hardt Moritz, Ma Tengyu: <strong>Identity matters in deep learning.</strong> <em>arXiv preprint arXiv:1611.04231</em>, 2016.</p><p>[HE:2016] He Kaiming, Zhang Xiangyu, Ren Shaoqing, Sun Jian: <strong>Deep residual learning for image recognition.</strong> In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>, pages 770–778, 2016.</p><p>[HI:2012] Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdel-rahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al.: <strong>Deep neural networks for acoustic modelling in speech recognition: The shared views of four research groups. </strong>IEEE Signal Processing Magazine, 29(6):82–97, 2012.</p><p>[KR:2012] Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.: <strong>Imagenet classification with deep convolutional neural networks.</strong> In Advances in neural information processing systems, pp. 1097–1105, 2012.</p><p>[WU:2016] Wu, Yonghui, Schuster, Mike, Chen, Zhifeng, Le, Quoc V., Norouzi, Mohammad, Macherey, Wolfgang, Krikun, Maxim, Cao, Yuan, Gao, Qin, Macherey, Klaus, Klingner, Jeff, Shah, Apurva, Johnson, Melvin, Liu, Xiaobing, Kaiser, Lukasz, Gouws, Stephan, Kato, Yoshikiyo, Kudo, Taku, Kazawa, Hideto, Stevens, Keith, Kurian, George, Patil, Nishant, Wang, Wei, Young, Cliff, Smith, Jason, Riesa, Jason, Rudnick, Alex, Vinyals, Oriol, Corrado, Greg, Hughes, Macduff, and Dean, Jeffrey: <strong>Google’s neural machine translation system: Bridging the gap between human and machine translation.</strong> CoRR, abs/1609.08144, 2016</p><p>[ZA:2016] Zagoruyko Sergey, Komodakis Nikos: <strong>Wide residual networks. </strong><em>NIN</em>, 8:35–67, 2016.</p><p>[ZH:2016] Zhang Chiyuan, Bengio Samy, Hardt Moritz, Recht Benjamin, Vinyals Oriol: <strong>Understanding deep learning requires rethinking generalization.</strong> <em>arXiv preprint arXiv:1611.03530</em>, 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=587edf422ac2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Massively Parallel Computations using DataProc]]></title>
            <link>https://medium.com/@dreading/massively-parallel-computations-using-dataproc-1f806940baa8?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f806940baa8</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dataproc]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Sun, 18 Nov 2018 23:43:01 GMT</pubDate>
            <atom:updated>2018-11-18T23:43:01.841Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*143qCDpww31xSFrimYJlqg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/Wpnoqo2plFA?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Mika Baumeister</a> on <a href="https://unsplash.com/search/photos/data-processing?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><a href="https://cloud.google.com/dataproc/">DataProc</a> is Google Cloud’s Apache <a href="https://hadoop.apache.org/">Hadoop</a> managed service. It is a quick, easy and relatively inexpensive way to build Hadoop clusters from a few instances to hundred of thousands. Whilst Hadoop excels at processing big data, we will look at its ability to perform calculations for a class of high throughput and high latency computations. This opens up the ability to perform numerical experiments at a scale that previously would have been difficult or restricted to the domain of supercomputers.</p><p>Recently Andrew Sutherland from MIT used Google cloud platform “to <a href="https://cloud.google.com/blog/products/gcp/220000-cores-and-counting-mit-math-professor-breaks-record-for-largest-ever-compute-engine-job?hl=nl">explore</a> generalisations of the Sato-Tate Conjecture and the conjecture of Birch and Swinnerton-Dyer to curves of higher genus”[1]. He used 580,000 cores and broke the record for largest ever Compute Engine job. Although he did not use DataProc, it demonstrates the ability of Google’s Cloud Platform to do high performance computing.</p><p>The future of cloud computing will include dedicated high performance computing (HPC) services. Alibaba is already targeting this space, offering <a href="https://www.alibabacloud.com/product/ehpc">E-HPC</a>, its HPCaaS cloud platform providing an all-in-one high-performance public computing service.</p><h3><strong>Monte Carlo Sampling</strong></h3><p>We will look at using Monte Carlo sampling to approximate the value of a definite integral. This is a basic method that will serve to demonstrate the concept without getting bogged down with mathematics. Monte Carlo methods are a class of stochastic algorithms that can be used in a wide range of applications from business, engineering, science, mathematics and other fields. Monte Carlo uses repeated random sampling to create a probability distribution for a variable to provide approximations that would otherwise be difficult to compute. We look at using it to compute definite integrals of the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*eEU-7pZRgYHmwzh8_KafVg@2x.png" /></figure><p>For the Monte Carlo method, we generate a random sample of uniformly distributed points <strong>x₁</strong>, <strong>x₂</strong>…. in Ω and approximate the integral by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*OaDsbummi1A7bI6z4Hgdsg.png" /></figure><p>where V is the the volume of the region Ω.</p><p>By the law of large numbers, the limit of the approximation exists and by the Central Limit Theorem, it will converge with N^-½. Even though the convergence rate is slow, it is independent of the number of dimensions. This is an important difference with deterministic methods, which are dependent on the size of the dimension and suffer from the curse of dimensionality for large dimensions.</p><p>In particular, we will compute the value of the integral with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*sjTbYVPEmagyknHROZbhSQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*HckEn_h665EUBgvslTQBgw.png" /><figcaption>Unit sphere first quadrant</figcaption></figure><p>This integral gives us the volume of a unit sphere in the first quadrant centred at the origin, the value of which equals π/6 or approximately 0.523598.</p><h3>DataProc</h3><p>Using DataProc is straightforward; provision a cluster and then execute code on the it. The code can be run via an interactive shell or submitted as a job. We will run a job on our Dataproc cluster, using the Cloud SDK’s gcloud dataproc job submit command.</p><p>To create a cluster, refer to the Google’s <a href="https://cloud.google.com/dataproc/docs/guides/create-cluster">documentation</a>. There is one important flag --num-preemptible-workers that specifies the number of preemptible workers in the clusters. Preemptible instances are short lived instances with a maximum life of 24 hours, that can be handed back to Google at any time. The advantage of preemptible workers is that they are considerably cheaper, at around 80% of a normal worker. We do not care if our instances get destroyed as we are performing random simulations.</p><p>For our numerical experiment, we will create a cluster with 10,000 workers. It only takes a few minutes yo create this cluster with 10,000 workers.</p><p>The following code will be used to create our approximation:</p><pre><em>#!/usr/bin/python<br></em><strong>import </strong>pyspark<br><strong>import </strong>random<br><strong>import </strong>time<br><strong>from </strong>operator <strong>import </strong>add<br><br>def integrand(x):<br>    return  1 if sum([i*i for i in x]) &lt;=1 else 0</pre><pre><br>def sampler(seed)<strong>:</strong><br>    random.seed(seed)<br>    x = [random.random(), random.random(), random.random()]<br>    return  integrand(x)<br><br>sampleSize = 1000000*10000<br>sc = pyspark.SparkContext()<br><br>print sc.parallelize([time.time() + i <strong>for </strong>i <strong>in </strong>xrange(sampleSize)])\<br>     .map(sampler).reduce(add)/sampleSize</pre><p>The parallelize operation creates a <a href="http://spark.apache.org/docs/1.0.1/programming-guide.html#resilient-distributed-datasets-rdds">resilient distributed dataset (RDD)</a> optimised for parallel processing. In this case, the RDD contains seeds that are based on the current system time. When creating the RDD, Spark slices the data based on the number of workers and the number of cores.</p><p>To run the code, we save it as file <em>integrate.py </em>on our cloud bucket and then submit the following job:</p><pre>gcloud dataproc jobs submit pyspark \<br>    --cluster &lt;cluster-name&gt; --region &lt;region&gt; \<br>    gs://&lt;bucket-name&gt;/integrate.py</pre><p>where <em>&lt;cluster-name&gt;</em>, <em>&lt;region&gt;</em> and <em>&lt;bucket-name&gt;</em> are placeholders for their respective values.</p><p>After we run the job, we get the following in the terminal output after a few seconds:</p><pre>Waiting for job output... <br>…</pre><pre>0.523597843</pre><pre>…</pre><pre>Job finished successfully.</pre><p>We have successfully computed the approximate value of the integral π/6 to six decimal places.</p><h3>Conclusion</h3><p>We have presented a rather naive example but it shows the power of DataProc to massively scale computations to tens of thousands of workers with minimal effort. We could quite as easily used a hundred thousand or two hundred thousand nodes. Although not all algorithms are suited for this approach, it works well with high throughput computations that do not need low latency between the nodes.</p><p>Being able to effortless provision resources at scale will change high performance computing: “It changes your whole outlook … when you can ask a question and get an answer in hours rather than months…You ask different questions.”[1]</p><p>[1]: <em>220,000 cores and counting: MIT math professor breaks record for largest ever Compute Engine job </em><a href="https://cloud.google.com/blog/products/gcp/220000-cores-and-counting-mit-math-professor-breaks-record-for-largest-ever-compute-engine-job">https://cloud.google.com/blog/products/gcp/220000-cores-and-counting-mit-math-professor-breaks-record-for-largest-ever-compute-engine-job</a></p><p>If you have an interest in Machine Learning and love solving complex problems come and have a chat with the <a href="https://momenton.com.au/#contact">Momenton</a> crew.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f806940baa8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comprehending Emoticons]]></title>
            <link>https://medium.com/@dreading/comprehending-emoticons-55f1532ae351?source=rss-a2329e0e2a8a------2</link>
            <guid isPermaLink="false">https://medium.com/p/55f1532ae351</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[comprehend]]></category>
            <category><![CDATA[emoticons]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Darren Reading]]></dc:creator>
            <pubDate>Mon, 03 Sep 2018 11:25:43 GMT</pubDate>
            <atom:updated>2018-11-01T23:39:45.389Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TRR86orEc9AX51F-" /><figcaption>“smiley paint on gray ground in front of people” by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>AWS’s <a href="https://aws.amazon.com/comprehend">Comprehend</a> provides a natural language processing service that uses machine learning to identify a text’s language, extract key phrases, understand sentiment (how positive or negative the text is) and automatically organise text by topic. You can easily analyse text and apply the results in a wide range of applications, finding insights and relationships in text. No need to understand how it works, just call an API and get the results. Simple?</p><p>Machine learning algorithms not only need to be accurate but also understandable and explainable. It is important to test how they behave for your specified domain.</p><p>I was interested in analysing user feedback. This can include emoticons and often does. I was curious how Comprehend would handle these ubiquitous little characters.</p><p>To test this out, I called the <a href="https://docs.aws.amazon.com/comprehend/latest/dg/API_DetectSentiment.html">Detect Sentiment</a> api with a test string: “Comprehend makes me happy”. The results were what you would expect, with the API returning a positive sentiment that was 93% positive:</p><pre>{ “Sentiment”: { “Sentiment”: “POSITIVE”, “SentimentScore”: { “Positive”: 0.932614266872406, “Negative”: 0.008453690446913242, “Neutral”: 0.05423713102936745, “Mixed”: 0.004694902338087559 } } }</pre><p>Replacing the text ‘happy’ with an emoticon ‘😀’, the updated test string “Comprehend makes me 😀”, resulted in a neutral sentiment overall and only 8% positive:</p><pre>{ “Sentiment”: { “Sentiment”: “NEUTRAL”, “SentimentScore”: { “Positive”: 0.08551795035600662, “Negative”: 0.03741350769996643, “Neutral”: 0.8686702847480774, “Mixed”: 0.00839831680059433 } } }</pre><p>What went wrong? To understand why, let’s call the <a href="https://docs.aws.amazon.com/comprehend/latest/dg/API_DetectSyntax.html">Detect Syntax</a> api. The response from this API, shows us that the emoticon is being interpreted as punctuation:</p><pre>{<br> “TokenId”: 4,<br> “Text”: “😀”,<br> “BeginOffset”: 20,<br> “EndOffset”: 21,<br> “PartOfSpeech”: {<br> “Tag”: “PUNCT”,<br> “Score”: 0.9972658157348633<br> }</pre><p>This is not ideal and any text that contains emoticons is not being analysed correctly.</p><p>Fortunately the solution is simple, before consuming the text with Comprehend’s API, pre-process the text to replace any emoticons with their corresponding text. For example 😀 to happy, 💗 to love and so on.</p><p>Notes:</p><ol><li>If you are using Python, there is a great little<a href="https://pypi.org/project/emoji/"> emoticon project</a> that will ‘demojize’ text.</li><li>There have been other <a href="https://github.com/aws-samples/aws-big-data-blog/blob/master/aws-blog-firehose-lambda-elasticsearch-near-real-time-discovery-platform/lambda-s3-twitter-to-es-python/tweet_utils.py">examples</a> that deal with emoticons by assigning each emoticon a sentiment. If the text contains an emoticon and it is a ‘positive’ emoticon then the text is positive. This approach is flawed, as text could contain negative modifiers (not😀), which would not be correctly interpreted.</li></ol><p>If you have an interest in Machine Learning and love solving complex problems come and have a chat with the <a href="https://momenton.com.au/#contact">Momenton</a> crew.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55f1532ae351" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>