<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Janaka Bandara on Medium]]></title>
        <description><![CDATA[Stories by Janaka Bandara on Medium]]></description>
        <link>https://medium.com/@janaka_bandara?source=rss-ca4b0f9c3cd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4pZOgYAjZdZiRfNcliP8rA.jpeg</url>
            <title>Stories by Janaka Bandara on Medium</title>
            <link>https://medium.com/@janaka_bandara?source=rss-ca4b0f9c3cd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 12:40:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@janaka_bandara/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Integrating regulatory submissions with the FDA ESG]]></title>
            <link>https://medium.com/@janaka_bandara/integrating-regulatory-submissions-with-the-fda-esg-ff7b7c0cf811?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff7b7c0cf811</guid>
            <category><![CDATA[fda]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[as2]]></category>
            <category><![CDATA[regulatory-compliance]]></category>
            <category><![CDATA[health-canada]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Wed, 26 Jul 2023 18:30:52 GMT</pubDate>
            <atom:updated>2024-03-27T13:34:11.133Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ql2kCtU2eutqmz0V.jpg" /></figure><p>If you operate in the healthcare domain within US jurisdictions, the US Food and Drug Authority (FDA) naturally becomes a vital part of your compliance process. Being able to perform timely and error-free submissions to FDA, seamlessly from your organization in-house systems, will make your life much easier — not to mention the competitive edge your business can gain from quicker approvals, shorter time-to-market, and a happier C-level.</p><h3>Importance of AS2 for FDA regulatory submissions</h3><p>As described in one of our <a href="https://aayutechnologies.com/blog/what-is-the-fda-esg/">previous articles</a>, FDA accepts submissions through two channels: <em>WebTrader</em> and <em>AS2</em>. WebTrader is recommended for low-volume manual submissions, but for any medium-to-large-scale organizations, and even for small businesses with an ambitious growth plan and projections, AS2 is the more future-proof submission method.</p><p>FDA accepts AS2 submissions over their Electronic Submissions Gateway (ESG). Integrating regulatory submissions with the FDA ESG involves a number of steps, but most of them are one-time tasks and can be performed by any IT personnel with a basic knowledge in network communications and public/asymmetric key based security.</p><h3>Steps to integrate your regulatory submissions with FDA</h3><h4>1. Register with the FDA ESG</h4><p><a href="https://www.fda.gov/industry/create-esg-account/setting-as2-account-checklist">E-mail your registration details</a> to FDA helpdesk (ESGHelpDesk@fda.hhs.gov), mentioning “New AS2 Test Account Registration” on the subject line, to commence the onboarding process. Provide your company name, contact name and phone, submission method (as “AS2”), and a <a href="https://www.fda.gov/industry/about-esg/appendix-g-letters-non-repudiation-agreement">Letter of Non Repudiation</a>.</p><p>The approval process is manual and may take a few days. FDA will open a ticket in their <a href="https://esghelpdesk.fda.gov/">ESG HelpDesk</a> and update you on the progress.</p><h4>2. Find an AS2 provider</h4><p>In AS2, both parties involved in a connection must have an AS2 application that can both send and receive traffic. Usually it is quite simple, economical and maintenance-free to go with a <a href="https://mftgateway.com/">cloud-based AS2 provider</a>, who will manage the data and connectivity on your behalf, where you would simply upload the submission files/archives to their platform, and download the response files when sent back by FDA. However if your organization has any strict security or regulatory compliance requirements regarding data retention or transfer, you can install an <a href="https://aayutechnologies.com/solutions/as2-gateway/">in-house or on-premise AS2 solution</a> and fine-tune it for your needs.</p><p>In either case, the AS2 solution has to have additional capabilities on top of the standard AS2 protocol support, in order to be compatible with FDA ESG submission process:</p><ul><li>ability to include <em>routing headers</em> (<a href="https://aayutechnologies.com/blog/as2-fda/#transmission-and-routing">HTTP request headers</a> indicating the destination FDA center/department and type of the submission) on a per-submission basis — especially if you wish to deal with multiple<a href="https://www.fda.gov/industry/about-esg/center-submission-types">centers and/or submission types</a></li><li>ability to <em>resume interrupted transfers</em>, if you wish to make large sized (VLF, “very large file”) submissions in gigabyte ranges; especially if you plan on submitting multimedia content like video files</li><li>ability to receive response traffic over <em>HTTPS</em>, on port <em>4080</em></li></ul><h4>3. Create an AS2 identity (station) for yourself, and get it registered with FDA</h4><p>This involves:</p><ol><li>choosing an <a href="https://aayutechnologies.com/docs/as2/as2-protocol/#44-what-is-an-as2-identifier-or-id"><em>AS2 identifier</em></a> to represent your organization; also referred to as “routing ID” by FDA</li><li>creating an AS2 (<em>private key</em> and <em>public certificate</em>) representing your organization; FDA accepts<a href="https://aayutechnologies.com/docs/as2/certificates-and-keys/#self-signed-certificates">self-signed</a> certificates, so there is no real need to get a CA (cartification authority) involved — unlike usually is the case with HTTPS/SSL certificates</li><li>obtaining the <em>AS2 receiver/listener URL</em> and its <em>HTTPS certificate</em> from your AS2 provider; this may be a common/shared one mentioned in their documentation or user interface (esp. in case of cloud AS2 platforms), or it may be customized/tailored for you (e.g. with a subdomain of your organization domain name)</li><li>submitting above details (commonly referred to as your <em>partner profile</em>), via the support ticket opened in FDA’s ESG HelpDesk (from the first step). Make sure that you<strong>do not</strong> share the <strong>private key</strong> from 2; FDA requires only the <em>public certificate</em>.</li></ol><p>FDA attends to these tickets quickly, usually within 24–48 hours.</p><h4>4. Register FDA as a partner on your AS2 provider</h4><p>Upon completion of step 3, FDA will share its AS2 identity details for the <em>test environment</em> with you, over the same support ticket. Similar to the partner profile that you shared, this will include:</p><ol><li>FDA’s AS2 identifier; usually ‘ZZFDATST’ for test and ‘ZZFDA’ for production, but may vary for some centers</li><li>FDA’s public AS2 certificate</li><li>FDA’s AS2 receiving URL, along with its HTTPS/TLS certificate</li></ol><p>Now you can use these details to create a partner entity on your AS2 platform. Partner creation process varies widely across platforms, but is generally <a href="https://aayutechnologies.com/docs/product/as2-gateway/adding-as2-partner/">well-documented by most providers</a>.</p><h4>5. Get the AS2 connection “activated” from FDA side</h4><p>In some cases, even with both ends configured correctly, FDA will continue to return HTTP 500 error code when you transmit files to them. You need to request FDA to “refresh” your account, through a support ticket. This may not always happen, but it is worth looking into — especially if you receive a 500 error during your first few submission attempts.</p><h4>6. Complete the test transmissions back and forth</h4><p>FDA may request you to perform one or two tests, based on the nature of your submissions:</p><ol><li>a “connect test”: transmit a small file to their GWTEST center (with routing headers X-Cyclone-Metadata-FdaCenter: GWTEST and X-Cyclone-Metadata-FdaSubmissionType: ConnectTest)</li><li>if you wish to make large-sized submissions, a “size test” where you submit a 7 GB file (of the same type that you wish to submit in future), to same GWTEST center — but with a different submission-type header value — SizeTest). This test may take several hours to complete, and transmission may be interrupted - this is where you can verify your platform&#39;s AS2-restart capabilities as well.</li></ol><p>For each of these, FDA will return an (AS2-level MDN receipt) and (a response text file indicating that the center successfully unpacked your submission).</p><p>FDA will review the test submission(s) — which could take up to 1 week — and if all looks good, will allow you to proceed to the next (and final) step.</p><h4>7. Set up the FDA production connection</h4><p>FDA will send you instructions on setting up your <em>production</em> account. Once the account is approved (which, according to FDA, could take up to 48 hours), you will receive their production AS2 connection profile; similar to the test connection, this will include their AS2 ID, URL and certificates.</p><p>There are no separate tests required for this step; once you set up FDA production system parameters as a partner on your end, you are good to go.</p><h4>Final thoughts</h4><p>If you reached here, congratulations! You now have a working AS2 connection and submission channel with FDA. Now you can start making regulatory submissions over the production channel.</p><p>Making well-formed, error-free FDA regulatory submissions, however, requires some additional, careful design steps upon your data processing pipeline, such as:</p><ul><li>creating well-formed tar.gz archive files</li><li>obscure issues caused by presence of some MIME types and AS2 headers</li><li>processing multi-level acknowledgements that FDA returns, and correlating them with each original submission</li></ul><p>You can learn about these, and common pitfalls that may arise due to negligence of any steps, in <a href="https://aayutechnologies.com/blog/as2-fda/">our dedicated guide on the subject</a>.</p><p>Getting up to speed with a smooth regulatory submission process with the FDA ESG, is much easier, faster and trouble-free, when you go with a custom-tailored <em>Regulatory Submissions Platform</em> rather than a generic AS2 solution. Check out what <a href="https://aayutechnologies.com/solutions/fda/">Aayu Technologies has in store for you</a>, and get started right away with a zero-commitment <em>free trial</em>.</p><p>You can sign up for an account with a one-month free trial (No credit card necessary) on the hosted SaaS version of the <a href="https://mftgateway.com/">MFT Gateway</a>. Alternatively, if you prefer an on-premise deployment, you can request to evaluate the <a href="https://aayutechnologies.com/docs/product/as2-gateway/enterprise/">AS2 Gateway</a> by contacting Aayu Technologies at <a href="https://aayutechnologies.com/contact/">https://aayutechnologies.com/contact/</a></p><p><em>Originally published at </em><a href="https://aayutechnologies.com/blog/integrating-regulatory-submissions-with-the-fda-esg/"><em>https://aayutechnologies.com</em></a><em> on July 26, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff7b7c0cf811" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Try out Serverless projects — online, in your browser!]]></title>
            <link>https://medium.com/@janaka_bandara/try-out-serverless-projects-online-in-your-browser-35bb5239b822?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/35bb5239b822</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[serverless-framework]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Thu, 04 Jul 2019 09:54:21 GMT</pubDate>
            <atom:updated>2019-07-04T09:54:21.270Z</atom:updated>
            <content:encoded><![CDATA[<h3>Try out Serverless projects — online, in your browser!</h3><h4>Sigma IDE can now open Serverless Framework projects!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hkfHgoeNnyXs0mQP" /><figcaption>Photo by <a href="https://unsplash.com/@jplenio?utm_source=medium&amp;utm_medium=referral">Johannes Plenio</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://serverless.com/framework/">Serverless Framework</a> is the unanimous leader in serverless tooling. Yet, there is no easy way to try out Serverless Framework projects online; you do need a decent dev set-up, and a bit of effort to set up sls, npm etc.</p><p>To be precise, you <em>did</em> — until now.</p><h3>Serverless project — in your browser?!</h3><p><strong>Sigma</strong> — <a href="https://slappforge.com/sigma?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=landing"><em>the</em> cloud based IDE for serverless application development</a> — can now open, edit and deploy Serverless projects, online — all in your browser!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/0*2D8WveCPlNFt5lbS.png" /><figcaption>Sigma: Think Serverless!</figcaption></figure><p>Nothing to install, nothing (well, to be extra honest: <a href="https://slappforge.com/docs/sigma/getting_started.html">very little</a>) to configure, and very little to worry about!</p><ul><li>Fire up Sigma.</li><li>On the <a href="https://sigma.slappforge.com/#/project"><strong>Projects</strong> page</a>, you’ll see a new <strong>import a Serverless Framework project</strong> option at the bottom.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*bIvUiQed4Q33VN7F.png" /><figcaption>Projects page: Import a Serverless Framework Project!</figcaption></figure><ul><li>Enter the path to your serverless.yml file (or the project root).</li><li>Goes without saying: click that ⚡ thunderbolt!</li></ul><h3>Serverless projects online: <a href="https://sigma.slappforge.com/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=landing">Sigma</a>’s insider story</h3><p>Internally, Sigma converts your Serverless template and opens it as a Sigma project. From there onwards, you can enjoy all Sigma goodies on your Serverless project; add dependencies, drag-n-drop coding, one-click deploy, <a href="https://hackernoon.com/how-we-streamlined-serverless-testing-8f5e0bbdc7e5">sub-second testing</a>, and more!</p><p>We are still working on improving support for all sorts of serverless.yml variations, but many of the generic ones should work fine.</p><p>By the way, one important thing to note: although we <em>import</em> from Serverless format ( serverless.yml), we don&#39;t <em>save</em> content in that format - yet. So if you import a project, make some changes and save it, things will get saved in Sigma&#39;s internal format.</p><p>(You can — and probably <em>should</em> — always pick a different repository to save your project, to prevent the original Serverless Framework repo from getting messed up.)</p><h3>Serverless on Sigma: the missing pieces</h3><p>Well, as with any new feature, the usual disclaimers apply — this is highly experimental, and could fail to load most, if not all, of your project; it could crash your IDE, kill your cat, bla bla bla.</p><p>And, on top of all that, we still do need to:</p><ul><li>provide support for externalized parameters, based on options (${opt:...}) and external environment variables; currently we take the default value if one is available</li><li>work something out for <a href="https://serverless.com/plugins/">plugins</a></li><li>add support for a ton of options like <a href="https://www.alexdebrie.com/posts/lambda-custom-authorizers/">API Gateway authorizers</a> and various trigger types that Sigma does not currently support</li><li>do something about all sorts of things that we find in the <a href="https://serverless.com/framework/docs/providers/aws/guide/serverless.yml/">custom field</a></li></ul><h3>What it all means — for you</h3><p>We guess this would be a good opportunity for folks to quickly try out Serverless apps and projects “off the shelf” — without actually installing anything on their own systems.</p><p>This would be great news for Sigma users as well; because it literally “explodes” the number of samples you can try out on Sigma!</p><h3>But wait — there’s more!</h3><p>In parallel, we have (correction: we <em>had to</em>) introduced a few other cool improvements:</p><h4>More control on utility files</h4><p>Now you have the key utility files ( package.json, .gitignore, README.md etc.) exposed at project root. Earlier they were internally managed by Sigma - hidden, out of your sight... but now you can add your own NPM configs, dependencies, scripts and whatnot; write your readme right inside Sigma; and much more! Any dependencies you add via Sigma&#39;s built-in <em>Dependency Manager</em> will be automatically added to package.json so you&#39;re covered.</p><h4>Custom resources for your Sigma project!</h4><p>You can add custom resource definitions to your project! Earlier this was limited to IAM role statements (with the cool <em>Permission Manager</em>), but now you can add whatever you like. EC2 instances, CloudFront distros, IoT stuff, AppSync… anything that you can define in <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/">CloudFormation</a> (or GCP’s <a href="https://cloud.google.com/deployment-manager/docs/configuration/syntax-reference">Deployment Manager syntax</a>, for that matter).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/185/0*jIogfnJnkHV8JZ6J.png" /><figcaption>The all-new ‘Customize Resources’ button!</figcaption></figure><p>We do hope to introduce <a href="https://www.terraform.io/">Terraform</a> support as well, although the ETA is nowhere in sight yet… so much cool stuff to do, with so few people!</p><h3>Coming up next…</h3><p>We do hope to support other project formats — like <a href="https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md">SAM</a> and <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/">raw CloudFormation</a> — in Sigma, pretty soon. So, as always, <a href="https://slappforge.com/blog/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=landing">stay tuned</a>; and, more importantly, <a href="https://launchpass.com/slappforge">shout out loud</a> with what <em>you</em> would like to see in the next Sigma release!</p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2019/07/try-out-serverless-framework-projects.html?m=1"><em>http://randomizd.blogspot.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35bb5239b822" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Greasemonkey Script Recovery]]></title>
            <link>https://medium.com/@janaka_bandara/greasemonkey-script-recovery-736468ca24cc?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/736468ca24cc</guid>
            <category><![CDATA[scripting]]></category>
            <category><![CDATA[sqlite]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[firefox]]></category>
            <category><![CDATA[data-recovery]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Thu, 30 May 2019 15:50:46 GMT</pubDate>
            <atom:updated>2019-05-30T16:01:58.835Z</atom:updated>
            <content:encoded><![CDATA[<h4>Getting the Great Monkey back on its feet!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*r5QjpnWUHug2TKSs.png" /><figcaption>GreaseMonkey — the Greatest Script Monkey of All Time! (techspot.com)</figcaption></figure><p>You just lost all your <a href="https://www.greasespot.net/">Greasemonkey</a> scripts? And the complete control of your browser, painfully built over several days of work? Looking all over the place for a Greasemonkey recovery technique posted by some tech magician?</p><p>Don’t worry, I have been there too.</p><h3>Damn you, BSOD! You Ate My Monkey! 😱</h3><p>My OS — Windows 10 — crashed (that infamous <a href="https://www.windowscentral.com/how-troubleshoot-blue-screen-errors-windows-10">“:(“ BSOD</a>) and restarted itself, and when I was back, I could see only a blank square where the Great Monkey used to be:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/52/0*UwVJUSD73Z-syXim.png" /><figcaption>The Monkey was here.</figcaption></figure><p>My natural instinct was thinking that my copy of GM had outlived its max life (I have a notorious track record of running with obsolete extensions); so I just went ahead and updated the extension (all the way from April 2018).</p><p>And then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/52/0*aL0QdN8ozf3DSJiH.png" /><figcaption>The Monkey says: ‘ ‘</figcaption></figure><p>Well said. That’s all I needed to hear.</p><p>Since I’m running <a href="https://www.mozilla.org/en-US/firefox/nightly/all/">Firefox Nightly</a>, I didn’t have the luxury of putting the blame on someone else:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/0*9X7IBnE2ZysOT-Ct.jpg" /></figure><p><em>Nightly is experimental and may be unstable.</em> — <a href="http://forums.mozillazine.org/viewtopic.php?f=23&amp;t=2476841#p12005477">Dear ol’ Mozilla</a></p><h3>No Greasemonkey. Helpless. Powerless. In a world of criminals… 🐱‍💻</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*R_qM8AH5fuGdzn3c" /><figcaption>To champion the cause of the innocent, the helpless, the powerless, in a world of criminals who operate above the law. (GreenWiseDesign)</figcaption></figure><p>So, <a href="https://knight-rider.fandom.com/wiki/Knight_Rider_(original_series)">Knight Rider quotes</a> aside, what are my options? Rewrite all my life’s work?</p><p>Naah. Not yet.</p><p>The scripts are probably still there; for whatever reason, GM cannot read them now.</p><h3>Hunting for the lost GM scripts</h3><p>Earlier, GM scipts were being saved in plaintext somewhere inside the Firefox user profile directory; but <a href="https://stackoverflow.com/questions/48469729/greasemonkey-script-folder-missing">not anymore, it seems</a>.</p><p>So I picked up the 61413404gyreekansoem.sqlite file from {FireFox profile home}/storage/default/moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5/idb/, opened it quick-n-dirty in a text editor, and searched for some of my old script fragments.</p><p>Phew, they are there! But they seem to be fragmented; possibly because it’s SQLite.</p><h3>sqlite3.exe to the rescue</h3><p>So I fired up <a href="https://developer.android.com/studio/command-line/sqlite3">Android SDK’s SQLite tool</a> (I didn’t have anything else lying around), and loaded the file:</p><pre>sqlite&gt; .open &#39;C:\Users\janaka\AppData\Roaming\Mozilla\Firefox\Profiles\a4And0w1D.default\storage\default\moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5\idb\61413404gyreekansoem.sqlite&#39;</pre><pre>sqlite&gt; .dump</pre><pre>PRAGMA foreign_keys=OFF;<br>BEGIN TRANSACTION;<br>CREATE TABLE database( name TEXT PRIMARY KEY, ...<br>...<br>INSERT INTO object_data VALUES(1,X&#39;30313137653167...00001300ffff&#39;);<br>INSERT INTO object_data VALUES(1,X&#39;30313965386367...00001300ffff&#39;);<br>...</pre><p>Looks good! So why can’t GM read this?</p><p>Probably their data model changed <strong>so dramatically</strong> so that my DB file no longer makes any sense?</p><h3>Let’s reboot Greasemonkey ⚡</h3><p>FF <a href="https://developer.mozilla.org/en/docs/Tools/Browser_Console">browser console</a> was also spewing out several errors at load time; I didn’t bother delving into the sources, but it looked like relaunching GM with the existing DB was a no-go.</p><p>So I moved the DB file to a safe location, deleted the whole extension data directory ( C:\Users\janaka\AppData\Roaming\Mozilla\Firefox\Profiles\a4And0w1D.default\storage\default\moz-extension+++d2345083-0b49-4052-ac13-2cefd89be9b5\) and restarted FF.</p><p>( <strong>Correction</strong>: I <em>did</em> back up the whole extension data directory, but it is <a href="https://stackoverflow.com/questions/48469729/greasemonkey-script-folder-missing">probably not necessary</a>.)</p><p>Nice, the Great Monkey is back!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/0*hZ5z8NgcgeCXP3qf.png" /><figcaption>GreaseMonkey clean startup: at least The Monkey is back — but… but…</figcaption></figure><p>FF also opened up the <a href="https://www.greasespot.net/2017/09/greasemonkey-4-announcement.html">Greasemonkey startup page</a> indicating that it treated the whole thing as a clean install. The extension data directory was also back, with a fresh DB file.</p><h3>Greasemonkey script migration: first attempt</h3><p>So, the million-dollar question: how the heck am I gonna load my scripts back to the new DB?</p><p>First, let’s see if the schemas have actually changed:</p><h4><strong>Old DB:</strong></h4><pre>CREATE TABLE database( name TEXT PRIMARY KEY, origin TEXT NOT NULL, version INTEGER NOT NULL DEFAULT 0, last_vacuum_time INTEGER NOT NULL DEFAULT 0, last_analyze_time INTEGER NOT NULL DEFAULT 0, last_vacuum_size INTEGER NOT NULL DEFAULT 0) WITHOUT ROWID;<br>CREATE TABLE object_store( id INTEGER PRIMARY KEY, auto_increment INTEGER NOT NULL DEFAULT 0, name TEXT NOT NULL, key_path TEXT);<br>CREATE TABLE object_store_index( id INTEGER PRIMARY KEY, object_store_id INTEGER NOT NULL, name TEXT NOT NULL, key_path TEXT NOT NULL, unique_index INTEGER NOT NULL, multientry INTEGER NOT NULL, locale TEXT, is_auto_locale BOOLEAN NOT NULL, FOREIGN KEY (object_store_id) REFERENCES object_store(id) );<br><br>// skimmin...<br><br>CREATE INDEX index_data_value_locale_index ON index_data (index_id, value_locale, object_data_key, value) WHERE value_locale IS NOT NULL;<br>CREATE TABLE unique_index_data( index_id INTEGER NOT NULL, value BLOB NOT NULL, object_store_id INTEGER NOT NULL, object_data_key BLOB NOT NULL, value_locale BLOB, PRIMARY KEY (index_id, value), FOREIGN KEY (index_id) REFERENCES object_store_index(id) , FOREIGN KEY (object_store_id, object_data_key) REFERENCES object_data(object_store_id, key) ) WITHOUT ROWID;<br><br>// skim...<br><br>CREATE TRIGGER object_data_delete_trigger AFTER DELETE ON object_data FOR EACH ROW WHEN OLD.file_ids IS NOT NULL BEGIN SELECT update_refcount(OLD.file_ids, NULL); END;<br>CREATE TRIGGER file_update_trigger AFTER UPDATE ON file FOR EACH ROW WHEN NEW.refcount = 0 BEGIN DELETE FROM file WHERE id = OLD.id; END;</pre><h4>New DB:</h4><pre>CREATE TABLE database( name TEXT PRIMARY KEY, origin TEXT NOT NULL, version INTEGER NOT NULL DEFAULT 0, last_vacuum_time INTEGER NOT NULL DEFAULT 0, last_analyze_time INTEGER NOT NULL DEFAULT 0, last_vacuum_size INTEGER NOT NULL DEFAULT 0) WITHOUT ROWID;<br>CREATE TABLE object_store( id INTEGER PRIMARY KEY, auto_increment INTEGER NOT NULL DEFAULT 0, name TEXT NOT NULL, key_path TEXT);<br><br>// skim...<br><br>CREATE TABLE unique_index_data( index_id INTEGER NOT NULL, value BLOB NOT NULL, object_store_id INTEGER NOT NULL, object_data_key BLOB NOT NULL, value_locale BLOB, PRIMARY KEY (index_id, value), FOREIGN KEY (index_id) REFERENCES object_store_index(id) , FOREIGN KEY (object_store_id, object_data_key) REFERENCES object_data(object_store_id, key) ) WITHOUT ROWID;<br><br>// skim...<br><br>CREATE TRIGGER object_data_delete_trigger AFTER DELETE ON object_data FOR EACH ROW WHEN OLD.file_ids IS NOT NULL BEGIN SELECT update_refcount(OLD.file_ids, NULL); END;<br>CREATE TRIGGER file_update_trigger AFTER UPDATE ON file FOR EACH ROW WHEN NEW.refcount = 0 BEGIN DELETE FROM file WHERE id = OLD.id; END;</pre><p>Well, looks like both are identical!</p><p>So theoretically, I should be able to dump data from the old DB (as SQL) and insert ’em to the new DB.</p><h3>no such function: update_refcount? WTF?</h3><h4>Old DB:</h4><pre>sqlite&gt; .dump</pre><pre>PRAGMA foreign_keys=OFF;<br>...<br>INSERT INTO object_data VALUES(1,X&#39;3031...ffff&#39;);<br>...</pre><h4>New DB:</h4><pre>sqlite&gt; INSERT INTO object_data VALUES(1,X&#39;3031...ffff&#39;);</pre><pre>Error: no such function: update_refcount</pre><p>Bummer.</p><p>Is FF actually <a href="http://sqlite2.blogspot.com/2016/03/how-to-edit-indexeddb-file.html?m=1">using non-standard SQLite</a>? I didn’t wait to find out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*MbzBM4sxN-uwE4nj.png" /><figcaption>GreaseMonkey, hang in there buddy… I WILL fix ya! (me.me)</figcaption></figure><h3>Greasemonkey Import/Export, to the rescue!</h3><p>If you haven’t seen or used it before, GM comes with two handy <strong>Export a backup</strong> and <strong>Import a backup</strong> options; they basically allow you to export all your scripts as a zip file, and later import them back.</p><p>And, from earlier, you would have guessed that our “barrier to entry” is most probably the SQLite trigger statement for AFTER INSERT ON object_data:</p><pre>CREATE TRIGGER object_data_insert_trigger AFTER INSERT ON object_data FOR EACH ROW WHEN NEW.file_ids IS NOT NULL BEGIN SELECT update_refcount(NULL, NEW.file_ids); END;</pre><p>So, what if I:</p><ul><li>drop that trigger,</li><li>import the old data dump into the new DB,</li><li>fire up the fox,</li><li>use Greasemonkey’s <strong>Export a backup</strong> feature to export (download) the recovered scripts,</li><li>shut down the fox,</li><li>delete the new DB,</li><li>restart the fox (so it creates a fresh DB), and</li><li>import the zip file back-up?</li></ul><h3>Yay! that worked! 🎉</h3><p>The second step caused me a bit of confusion. I was copying the dumped data right off the SQLite screen and pasting it into the SQLite console of the new DB; but it appeared that some lines were too long for the console input, and those inserts were causing cascading failures. Hard to figure out when you copy a huge chunk with several inserts, paste and run them, and all — except the first few — fail miserably.</p><p>So I created a SQL file out of the dumped data, and imported the file via the .read command:</p><h4>Old DB:</h4><pre>sqlite&gt; .output &#39;C:\Windows\Temp\dump.sql&#39;</pre><pre>sqlite&gt; .dump</pre><pre>sqlite&gt; .quit</pre><h4>Cleanup:</h4><p>Now, open dump.sql and remove the DDL statements; CREATE TABLE, CREATE INDEX, CREATE TRIGGER, etc.</p><h4>New DB:</h4><pre>sqlite&gt; DROP TRIGGER object_data_insert_trigger;<br>DROP TRIGGER object_data_update_trigger;<br>DROP TRIGGER object_data_delete_trigger;<br>DROP TRIGGER file_update_trigger;</pre><pre>sqlite&gt; .read &#39;C:\Windows\Temp\dump.sql&#39;</pre><p>All else went exactly as planned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*aFJPLXwUjXotb-48.jpg" /></figure><h3>Welcome back, O Great Monkey! 🙏</h3><p>So, my Greasemonkey set up is back online, running as smoothly as ever!</p><p>Final words: if you ever hit an issue with GM:</p><ul><li>don’t panic.</li><li>back up your extension data directory right away.</li><li>start from scratch (get the monkey running), and</li><li>carefully feed the monkey (yeah, Greasemonkey) with the old configs again.</li></ul><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2019/05/greasemonkey-script-recovery-getting.html"><em>http://randomizd.blogspot.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=736468ca24cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Cloud Functions: a (long overdue) “hello, world!” on GCP]]></title>
            <link>https://medium.com/@janaka_bandara/google-cloud-functions-a-long-overdue-hello-world-on-gcp-97332ff04eba?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/97332ff04eba</guid>
            <category><![CDATA[functions-as-a-service]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[google-cloud-functions]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Sun, 14 Apr 2019 03:07:18 GMT</pubDate>
            <atom:updated>2019-04-14T03:41:40.048Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>A time-travel, back to the beta days of GCF</h4><p>In early 2018, Google Cloud Functions <a href="https://serverless.com/blog/google-cloud-functions-generally-available-what-means-serverless/">went GA</a>. Some time before that — in March 2018, while it was still in beta — I took some screenshots while taking my first shot at Cloud Functions. But then one thing led to another, and to another, and another, and this blog post was never born.</p><p>Now, after a whole year of downright obsoletion, I present to you my “getting started with Google Cloud Functions [beta]” guide.</p><p>Surely a lot has changed; beta tag gone, hordes of new features introduced: Python runtime, environment variables, in-built test invocations and logs, and a lot that I haven’t even seen yet.</p><h3>Activate billing on your GCP account</h3><p>Now, if you rush to the <a href="https://console.cloud.google.com/functions/list">Cloud Functions dashboard</a>, you may notice that you need to <em>enable the Cloud Functions API</em> — unless you have done so already, with your currently active GCP project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/271/0*fCctWQaVQvdRfk4Z.png" /><figcaption>‘Cloud Functions’ link on the GCP console’s left menu</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/1*kLoE17Ro5RCRubfWgARgRg.png" /><figcaption>‘Cloud Functions API not enabled’</figcaption></figure><p>Cloud Functions need a billing enabled GCP project, so that’s the first thing we need to do. <a href="https://cloud.google.com/billing/docs/how-to/modify-project">Google’s official guide</a> is fairly easy to follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*HP-7crnnk0z8c6n6.png" /><figcaption>filling in payment details for your billing account</figcaption></figure><p>Google promises that we won’t be charged during our <a href="https://cloud.google.com/free/">1-year, $300 free trial</a> so we’re covered here.</p><h3>Create a cloud function</h3><p>When done, go ahead to the <a href="https://console.cloud.google.com/functions/list">Cloud Functions dashboard</a> and click <strong>Create function</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*VwGPsY9pvUCPXHV3.png" /><figcaption>a fresh Cloud Functions dashboard (back from the beta days)</figcaption></figure><p>This will take you to a <strong>Create function</strong> wizard. The first phase would resemble:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/0*Br-AEuibkEOmkNsb.png" /><figcaption>GCP Create Function: configuration phase 1</figcaption></figure><p>Here you define most of the basics of your cloud function: name, max <em>memory</em> limit, <em>trigger</em>, and source code.</p><p>Later on, you get the chance to define the <em>handler/entrypoint</em> (NodeJS function to invoke when the Cloud Function is hit); and other settings like the function deployment <em>region</em> and <em>timeout</em> (maximum running time per request).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*iG9InXE33EByDxN0.png" /><figcaption>GCP Create Function: name (handler), region and timeout</figcaption></figure><h3>Trigger options</h3><p>Triggers can invoke your cloud function in response to external actions: <em>active</em> ones like <a href="https://cloud.google.com/functions/docs/writing/http">HTTP requests</a> or <em>passive</em> ones like events from <a href="https://cloud.google.com/storage/">Cloud Storage</a> buckets or <a href="https://cloud.google.com/pubsub/">Cloud Pub/Sub</a> topics.</p><p>In platforms like AWS this may not make much difference: Lambda configures and handles both event types in the same way. But in GCP they are handled quite differently; so different that the cloud function <a href="https://cloud.google.com/functions/docs/writing/#types_of_cloud_functions">method signature itself is different</a>.</p><p>If you try switching between the trigger types, you would see how the sample code under <strong>Source code</strong> changes:</p><h4>HTTP functions</h4><p>Google generates the signature:</p><pre>/**<br> * Responds to any HTTP request that can provide<br> * a &quot;message&quot; field in the body.<br> *<br> * @param {!Object} req Cloud Function request context.<br> * @param {!Object} res Cloud Function response context.<br> */<br>exports.helloWorld = (req, res) =&gt; {<br>  // Example input: {&quot;message&quot;: &quot;Hello!&quot;}<br>  if (req.body.message === undefined) {<br>    // This is an error case, as &quot;message&quot; is required.<br>    res.status(400).send(&#39;No message defined!&#39;);<br>  } else {<br>    // Everything is okay.<br>    console.log(req.body.message);<br>    res.status(200).send(&#39;Success: &#39; + req.body.message);<br>  }<br>};</pre><p>A HTTP function accepts a <a href="https://expressjs.com/en/api.html#req">request</a> and writes back to a <a href="https://expressjs.com/en/api.html#res">response</a>.</p><p>The HTTP(S) endpoint is automatically provisioned by Google, at https://{region}-{project-name}-gcp.cloudfunctions.net/{function-name}. So there is nothing more to configure in terms of triggers.</p><h4>Event-based functions</h4><p>Google’s signature looks like:</p><pre>/**<br> * Triggered from a message on a Cloud Pub/Sub topic.<br> *<br> * @param {!Object} event The Cloud Functions event.<br> * @param {!Function} The callback function.<br> */<br>exports.subscribe = (event, callback) =&gt; {<br>  // The Cloud Pub/Sub Message object.<br>  const pubsubMessage = event.data;<br><br>  // We&#39;re just going to log the message to prove that<br>  // it worked.<br>  console.log(Buffer.from(pubsubMessage.data, &#39;base64&#39;).toString());<br><br>  // Don&#39;t forget to call the callback.<br>  callback();<br>};</pre><p>Event-based functions accept an <em>event</em> and convey success/failure (and optionally a result) via a <a href="https://cloud.google.com/functions/docs/writing/background#function_parameters"><em>callback</em></a>.</p><p>Here we need to configure an event source (Pub/Sub topic or Storage bucket) to trigger the function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*8iW-uo4lJKyGUM4Z.png" /><figcaption>configuring a Cloud Storage bucket to trigger a function</figcaption></figure><p>Good thing is, GCP allows you to pick an existing topic/bucket, or create a new one, right there inside the cloud function wizard page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*ESaw4mJIKDP-bah4.png" /><figcaption>defining a new Cloud Storage bucket via the ‘Create new bucket’ pop-up</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*DXYyHON49zul2XrK.png" /><figcaption>defining a new Cloud Pub/Sub topic via the ‘Create new topic’ pop-up</figcaption></figure><p>Picking an existing entity is just as easy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*9EJDFryxVLpD_oMR.png" /><figcaption>selecting an existing Cloud Storage bucket</figcaption></figure><h4>Automatic retry</h4><p>These non-HTTP functions also have a retry mechanism: you can configure GCP to redeliver an event back to the function automatically, if the function failed to process it during the last time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*jFBybTj0IbkPTjWZ.png" /><figcaption>Cloud Function retry configuration</figcaption></figure><p>This is good for taking care of temporary failures, but it can be dangerous if the error is due to a bug in your code: GCP will keep on retrying the failing event for up to 7 days, draining your quotas and growing your bill.</p><h3>A function is born</h3><p>Now we are done with the configurations; click <strong>Create</strong>.</p><p>GCP takes you back to the function dashboard, where the new function will be listed; with a “pending” or “creating” status.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*5WMV0Q5bonXQE7oK.png" /><figcaption>Cloud Functions dashboard, while your new function gets created</figcaption></figure><p>It may take a while, but finally you would see the successfully created function in <strong>Active</strong> status:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/0*vIEC6mSYRt4jPrvI.png" /><figcaption>Cloud Function created and in ‘Active’ status</figcaption></figure><p>Back in the beta days, I got some sporadic errors when trying to create functions; for no apparent reason. Hopefully nobody is that much unlucky these days.</p><p>It is a good practice to <em>label</em> your functions, so you can find and manage them easily. GCP allows you to do this right from the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*6Q3dYMZw_CF_wws6.png" /><figcaption>labelling your Cloud Functions, right from the dashboard</figcaption></figure><h3>Function actions</h3><p>Click the three-dots button at the far right end of the function entry, to see what you can do next:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/0*NOOkmqEVvGWEk1Yj.png" /><figcaption>Cloud Function actions</figcaption></figure><p>Back in the beta days, there were some hiccups with some of these options (like <strong>Copy Function</strong>; hopefully they are long gone now!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*KpXerNFFrc0giTP_.png" /><figcaption>Copy Function: ‘no source code location’ error</figcaption></figure><h3>Testing</h3><p><strong>Test Function</strong> gives a nice interface where you can invoke the function with a custom payload, and view the output and execution logs right away. However it still lacks the ability to define and run predefined custom test events, like in AWS or <a href="https://slappforge.com/docs/sigma/features/test/test.html#1-creating-a-test-case">Sigma</a>. Another caveat is that the test invocations also hit the same production function instance (unlike, say, the <a href="https://hackernoon.com/how-we-streamlined-serverless-testing-8f5e0bbdc7e5">test environment of Sigma</a>), so they count towards the logs and statistics of your actual function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*CAHa5iiV0YtZ2WT6.png" /><figcaption>‘Test Function’ tab of the function detailed view</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*C78T5vyb-g4h9YAl.png" /><figcaption>Test result for the standard HTTP function sample</figcaption></figure><h3>Logs</h3><p><strong>View Logs</strong> takes you to the familiar StackDriver Logging page, where you can browse, sort, search stream and do all sorts of things with the logs of your function. It takes a few seconds for the latest logs to appear, as is the case with other platforms as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*SH2QZxreX4HurPD5.png" /><figcaption>‘View Logs’ takes you to StackDriver Logging</figcaption></figure><h3>More function details</h3><p>You can click the function entry to see more details:</p><ul><li><strong>General</strong> tab shows a nice stats graph, along with basic function configs like runtime, memory etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/419/0*prrUgozhSAJ5Cl6X.png" /><figcaption>‘General’ tab of our new function</figcaption></figure><ul><li><strong>Trigger</strong> tab shows the trigger config of the function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/0*gl3pNTm1eX_NnRin.png" /><figcaption>‘Triggers’ tab with an HTTP trigger config</figcaption></figure><p>As of now, GCP doesn’t allow you to edit/change the trigger after you have created the function; we sincerely hope this would be relaxed in the future!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*UOzxToSpinq8bv_T.png" /><figcaption>a read-only trigger config</figcaption></figure><ul><li><strong>Source</strong> tab has the familiar code viewer (although you cannot directly update and deploy the code from there). It also has a <strong>Download zip</strong> button for the code archive.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/0*AH3nv07ZviyEiDQs.png" /><figcaption>‘Source’ tab with function code</figcaption></figure><ul><li><strong>Testing</strong> tab we have already seen. It’s also pretty neat, for something that went GA just a few months ago.</li></ul><h3>So, that’s what a cloud function looks like.</h3><p>Or, to be precise, how it used to be — back in the pre- and post-beta days.</p><p>I’m sure Google will catch up on <a href="https://hackernoon.com/why-serverless-why-now-f09ce43c4767">the serverless race</a> — with more event sources, languages, monitoring and so forth.</p><p>But do we need to wait? Absolutely not.</p><p>Cloud Functions are mature enough for most of your routine integrations. One major bummer is that it doesn’t yet support timer schedules, but <a href="https://medium.com/@earlg3/google-cloud-functions-scheduled-trigger-915b5fb8310f">folks are already using workarounds</a>.</p><p>Plus, many of the leading serverless development frameworks are already supporting GCP!</p><p>So <a href="https://slappforge.com/docs/sigma/components/gcp/getting-started.html">hop in</a> — write your own serverless success story on GCP!</p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2019/04/google-cloud-functions-looong-overdue.html?m=1"><em>randomizd.blogspot.com</em></a><em> on April 5, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97332ff04eba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 5 Reasons Why Your EDI Transfer is Failing]]></title>
            <link>https://medium.com/@janaka_bandara/top-5-reasons-why-your-edi-transfer-is-failing-d4185766ade1?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4185766ade1</guid>
            <category><![CDATA[troubleshooting]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[as2]]></category>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[ide]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Fri, 12 Apr 2019 03:28:46 GMT</pubDate>
            <atom:updated>2022-04-27T03:27:48.076Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Troubleshooting B2B EDI — sounds hard, but it does not have to be!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*q5Afmb4i8D5K4J7z.jpg" /><figcaption>Uh-oh, something’s not right with your EDI transfer engine. (courtesy of <a href="https://upload.wikimedia.org/wikipedia/commons/c/c9/Vw_engine_check.jpg">Wikipedia</a>)</figcaption></figure><p>EDI transfer is a very popular medium for business document exchange; yet, setting up and maintaining an EDI transfer communication link can be fairly challenging. While <a href="https://www.capterra.com/p/175877/AS2Gateway/">SaaS EDI transfer offerings</a> simplify the process to a great extent, there are several pitfalls that need manual intervention; especially during the initial set-up.</p><blockquote><strong>Note</strong>: This article only applies to EDI transfer via the <a href="https://aayutechnologies.com/docs/as2/how-as2-works/">AS2 protocol</a>. If you are using <a href="https://www.edibasics.com/edi-resources/messaging-protocols/">different protocols</a>, check out the guides for the respective protocol; or <a href="https://aayutechnologies.com/docs/as2/importance-of-as2/">consider the benefits</a> of migrating to AS2, the <a href="https://blogs.opentext.com/the-most-popular-b2b-protocols-as-chosen-by-you/">most popular secure EDI exchange protocol</a>.</blockquote><h3>1. There is something wrong with your network communication link.</h3><p>AS2 EDI transfer happens over the <a href="https://www.networkworld.com/article/3172730/security/the-dangers-of-the-public-internet.html">public internet</a>, so both parties (<strong>trading partners</strong> — you and your recipient) must be able to connect to each other. As such, this should be your first checkpoint, whenever something has gone wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*8JIMLhRWB5AcTzO0.jpg" /><figcaption>Is your network up and running? (courtesy of <a href="https://live.staticflickr.com/6196/6088751332_7da4134066_n.jpg">Flickr</a>)</figcaption></figure><h4>(a) Network (IP address) whitelisting — you, your partner, or both</h4><p>Usually applications use <a href="https://en.m.wikipedia.org/wiki/Firewall_(computing)">network firewall rules</a> for controlling EDI/AS2 connectivity access, so allowing (whitelisting) you on your partner’s side would often involve some communication and manual intervention.</p><p>Whitelisting comes at two levels:</p><ul><li>you, on your partner’s side (if you are unable to connect/send to your partner)</li><li>your partner, on your side (if your partner is unable to connect/send to you)</li></ul><p>SaaS providers <a href="https://aayutechnologies.com/docs/product/as2-gateway/">like AS2 Gateway</a> will usually allow access from any network/IP address; and use alternative measures to validate and regulate security. However, if your partner uses a <a href="https://searchnetworking.techtarget.com/definition/value-added-network">legacy system like VAN (Value Added Network)</a> or an on-premise application, you might need to contact them and send them your EDI application’s IP address for whitelisting.</p><p><strong>Finding and whitelisting your IP address (when you cannot connect to your partner)</strong></p><p>The process of finding your IP address can vary:</p><ul><li>For an <a href="https://as2gateway.com/on-premise/">on-premise EDI/AS2</a> installation, you will need to ensure that your organization has a <a href="https://www.lifewire.com/what-is-a-public-ip-address-2625974">public IP address</a>; and that the EDI application is <a href="https://www.neowin.net/forum/topic/949134-set-up-computer-as-public-ip-and-expose-to-internet/">exposed over, or reachable from, that address</a>. Once confirmed, you can find the IP address value via an online service like <a href="https://ipecho.net/plain">IPEcho</a>. Contact your EDI/AS2 vendor or consult your network/infrastructure team for more instructions and support.</li><li>For a cloud or SaaS installation, you will usually be able to find the IP address via its <em>user interface</em> or <em>documentation</em>; for example, the public IP of AS2 Gateway is 52.204.119.191, as mentioned in <a href="https://aayutechnologies.com/docs/product/as2-gateway/troubleshooting-guide/">their documentation</a>. Contact the vendor if you cannot find the IP address in this manner.</li></ul><p><strong>Finding (and whitelisting) your partner’s IP address (when your partner cannot connect to you)</strong></p><blockquote><strong>Note:</strong> This step can be omitted if you go with a SaaS EDI transfer solution like <a href="https://as2gateway.com/">AS2 Gateway</a> which has minimal network restrictions on inbound EDI transfers.</blockquote><p>First, check whether the current configuration works; ask your partner to try connecting to your EDI exchange (e.g. send a test message). If it results in an error like “connection refused” or “connection timed out”, your firewall is probably blocking him from connecting.</p><p>Contact your partner, obtain their EDI transfer IP address (from which he will be connecting to you), and:</p><ul><li>if you’re using an on-premise EDI exchange, <a href="https://help.serversaustralia.com.au/hc/en-us/articles/115002335806-How-To-Whitelist-An-IP-Address-In-IPTables">whitelist the IP</a> on your organization network/firewall.</li><li>if you’re on a SaaS EDI transfer service, contact the service provider for instructions, settings or assistance on whitelisting the partner’s IP address.</li></ul><h4>(b) You are calling the wrong endpoint on your partner’s AS2 system — or the other way around.</h4><p>Each EDI/AS2 exchange system has a specific endpoint (URL) for receiving incoming transmissions. For example, AS2 Gateway receives EDI/AS2 transmissions at:</p><ul><li>http://service.as2gateway.com:8280/service/as2-receiver and</li><li>https://service.as2gateway.com:8043/service/as2-receiver</li></ul><p>If your EDI application is not configured with the correct endpoint of your partner (or his with yours), the sending would fail with HTTP errors; such as:</p><ul><li>301/302: <strong>Redirect</strong>,</li><li>400: <strong>Bad Request</strong>,</li><li>404: <strong>Not Found</strong>,</li><li>405: <strong>Method not allowed</strong>,</li><li>500: <strong>Internal Server Error</strong>, or</li><li>503: <strong>Service Unavailable</strong>.</li></ul><p>You would need to consult your trading partner and configure his EDI transfer endpoint on your side; and send him the AS2 endpoint of your application so that he can set up your endpoint for transfers from his side.</p><blockquote>AS2 Gateway makes this quite easy by allowing you to share your <a href="https://aayutechnologies.com/docs/product/as2-gateway/creating-as2-station/">local trading station details</a> with your partner <a href="https://aayutechnologies.com/docs/product/as2-gateway/creating-as2-station/#31-partner-configuration-of-an-as2-station">via email, in one click</a>. If your partner is also using AS2 Gateway or <a href="https://as2gateway.com/on-premise/">its on-premise variant</a>, he will be able to do the same.</blockquote><h3>2. Your EDI transfer recipient has not properly configured you as one of his trading partners.</h3><p>In EDI transfer over AS2, each party (sender and recipient) should configure the other a trading partner. This allows both parties to trust each other and utilize various security mechanisms during EDI exchange.</p><p>If your transfer is not receiving any response (e.g. causing a “connection closed” error), or resulting in an HTTP error: e.g.,</p><ul><li>400: <strong>Bad Request</strong>,</li><li>403: <strong>Forbidden</strong>,</li><li>404: <strong>Not Found</strong>, or</li><li>500: <strong>Internal Server Error</strong>,</li></ul><p>it might be worth contacting your partner and verifying that he has added you as a correspondence on his EDI/AS2 system.</p><p>Similarly, if your partner is seeing similar issues on his side, you may need to ensure that you have added him as a trading partner on your EDI exchange application. The steps for defining a new trading partner are usually <a href="https://aayutechnologies.com/docs/product/as2-gateway/adding-as2-partner/">included in the provider’s documentation</a>.</p><h3>3. You are (or your partner is) using the wrong certificates.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*F5Blm3e5dLDZ5x1k.png" /><figcaption>Are your certificates all in order? (courtesy of <a href="https://upload.wikimedia.org/wikipedia/commons/a/a4/Text_document_with_red_question_mark.svg">Wikipedia</a>)</figcaption></figure><p>Certificates can help to establish secure channels for EDI/AS2 transfers over the insecure, public internet. An EDI exchange involves quite a few certificate types; it is quite possible to swap them incorrectly, or fail to import some of them, resulting in a bad, non-working configuration.</p><h4>(a) Encryption certificate</h4><p>This is provided by your partner, and your EDI/AS2 application uses it to encrypt and secure the content being sent out to your partner. (Similarly you need to send your own certificate to your partner, for him to be able to communicate with you.)</p><p>This certificate may be missing if your partner does not expect you to encrypt your EDI transfers; in that case you can ignore this (although it would be a security loophole).</p><h4>(b) Signing certificate</h4><p>This, also provided by your partner, is used to produce a digital signature; for ensuring that the EDI content has not been tampered with, during the transfer.</p><p>It is possible to use the encryption certificate itself as the signing certificate, or to disable signing entirely; in which case you can ignore this step.</p><h4>(c) HTTPS or SSL certificate</h4><p>This is used only when the EDI transfer happens over HTTPS; it refers to the certificate that your AS2 application should use for encrypting the overall communication (high-level, not just the EDI content) during transfer.</p><h4>(d) Chain certificates</h4><p>Each of the above certificates can be “signed” by a third party, in order to verify that the certificate in your possession truly belongs to your partner; otherwise a third party could send you a fake certificate, and later intercept and read all your communications intended for your partner.</p><blockquote>Sometimes this can be a “trust chain” where the original certificate is signed by one party, who is in turn signed by another party, and so forth; until there’s a <a href="https://www.namecheap.com/support/knowledgebase/article.aspx/334/38/what-is-certificate-authority-ca">root authority</a> that is implicitly trusted by the EDI/AS2 application, hence validating all the previous trust relationships.</blockquote><p>In such cases, your partner may have provided you with one or more “chain certificates” that would constitute this “trust chain”, which you would then need to import into your EDI/AS2 exchange application.</p><p>If there are no chain certificates, it could mean either:</p><ul><li>your partner’s certificate is signed by a root authority (that <a href="https://www.globalsign.com/en/certificate-authority-root-signing/">any application would generally trust</a>), or</li><li>your partner himself has signed it: a “self-signed” certificate</li></ul><h4>Dealing with self-signed certificates</h4><p>While the latter sounds insecure, if you received the certificate file via trustworthy means, it is secure enough for use in EDI/AS2 transfers.</p><p>Your EDI/AS2 application may complain that the certificate is “self-signed” and “untrusted” (especially on Windows). In this case, you would need to <a href="https://smallbusiness.chron.com/make-computer-trust-certificate-authority-57649.html">add it to the list of trusted certificates</a>; but if the application itself has a certificate store feature, it is safer to add it there, rather than at operating system level.</p><p>By nature, if you fail to configure any of these certificates (e.g. forget to import the chain certificates); or mistakenly swap them (e.g. use the HTTPS certificate in place of the encryption certificate), EDI/AS2 communications would fail.</p><p>Unless your partner has named them properly, it would be difficult to tell these certificates apart. If you know the “owner names” (<a href="https://knowledge.digicert.com/generalinformation/INFO1745.html">distinguished names</a>) of each certificate, you can open the files via a <em>certificate viewer</em> and figure out which is which. Most operating systems provide native certificate viewers (simply double-click the file).</p><blockquote>Online solutions like AS2 Gateway <a href="https://aayutechnologies.com/docs/product/as2-gateway/certificate-management/">provide built-in certificate stores</a> for easy addition and management of such encryption, signing and chain certificates.</blockquote><h3>4. Missing or invalid MDN (receipt notifications)</h3><p>AS2 protocol optionally enables partners to confirm the receipt of EDI/AS2 transfers via <a href="https://aayutechnologies.com/docs/as2/as2-protocol/#25-message-disposition-notification-mdn">message disposition notifications (MDNs)</a>; similar to receipts in monetary transactions.</p><p>If the EDI sender has requested an MDN from the recipient, but the recipient fails to provide one, the EDI transfer is considered as a failure. In such cases, AS2 recommends the sender to retry the EDI/AS2 transfer up to a desired number of times.</p><p>If the recipient has configured his EDI/AS2 application incorrectly, this could lead to repeated re-transfers of the same EDI content, leading to undesired errors.</p><p>Additionally, AS2 allows two types of MDNs. One is <em>synchronous</em> or <em>sync</em>, where the recipient sends back the receipt immediately, as a direct response on the same EDI transfer channel. The other is <em>asynchronous</em> or <em>async</em>, where the recipient briefly acknowledges the original channel, and later sends back a proper receipt on a separate channel. The type of MDN could be <a href="https://aayutechnologies.com/docs/as2/as2-protocol/#251-sending-mdn">explicitly configured at partner level</a>, or requested on a per-message basis via certain AS2 headers.</p><p>However, if one of the two EDI transfer applications does not honor this MDN type convention, it could lead to errors and re-transmissions as described earlier.</p><h3>5. Mismatched EDI transfer configurations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*mV3CmN7yRTAEYrGv.jpg" /><figcaption>So many knobs and switches, so many ways to make a mistake… (courtesy of <a href="https://www.maxpixel.net/Switchboard-Power-Electric-Panel-Plugs-Electrical-2332239">MaxPixel</a>)</figcaption></figure><p>EDI transfer over AS2 requires quite a few configurations, on your side as well as your partner’s side.</p><p>Cloud EDI transfer solutions <a href="https://aayutechnologies.com/docs/product/as2-gateway/creating-as2-station/">have detailed instructions</a> on these; however the steps are error-prone, and one mistake can lead to complete breakdown of the EDI communication channel.</p><h4>(a) Double-checking your end</h4><p>There are a few EDI transfer configurations that you should receive from your partner and configure on your side. Here is a brief summary:</p><p><strong>For sending to your partner:</strong></p><ul><li>Your partner’s unique identifier (<a href="https://aayutechnologies.com/docs/as2/as2-protocol/#44-what-is-an-as2-identifier-or-id"><em>AS2 ID</em></a>)</li><li>The endpoint (URL, address) at which your partner receives EDI/AS2 traffic</li><li>The type of EDI/AS2 traffic your partner is expecting (encrypted? signed? compressed? if compressed, is it <em>before</em> or <em>after</em> encryption?)</li><li>For <em>encryption</em>, your partner’s <em>encryption certificate</em></li><li>With <em>signing</em>, your partner’s <em>signing certificate</em></li><li>If the transfer happens over <em>HTTPS</em>, the HTTPS certificate of your partner’s AS2 software, along with any non-standard signers of it (<em>chain certificates</em>; as described before)</li></ul><p>In addition, you can decide whether to request an <em>MDN (receipt)</em> for the sent EDI/AS2 message; and, if so,</p><ul><li>whether it should be <em>sync</em> or <em>async</em> (see above), and</li><li>whether it should also be <em>signed</em> (like the AS2 message itself).</li></ul><p>While most partners would natively support all these combinations, double-checking could be useful in troubleshooting failures.</p><p><strong>For receiving from your partner:</strong></p><p>On standard EDI/AS2 applications <a href="https://aayutechnologies.com/docs/product/as2-gateway/user-guide/">like AS2 Gateway</a>, this should not usually require additional configuration on your end; other than your partner’s AS2 ID. However, the setup may vary based on your EDI/AS2 software.</p><h4>(b) Double-checking your partner’s end:</h4><p>Here are some EDI transfer configurations that you should send to your partner, and he should configure on his side:</p><p><strong>For your partner to send files to you:</strong></p><p>This is similar to the set of configurations that you would configure on your side, in order to transfer EDI to your partner:</p><ul><li>Your unique identifier (<em>AS2 ID</em>)</li><li>The EDI/AS2 traffic endpoint of your AS2 application</li><li>For <em>encryption</em>, your <em>encryption certificate</em></li><li>With <em>signing</em>, your <em>signing certificate</em> (could be the same as the encryption certificate)</li><li>If the transfer happens over <em>HTTPS</em>, the <em>HTTPS certificate</em> of your AS2 software; along with any <em>chain certificates</em></li></ul><blockquote>As mentioned before, solutions like AS2 Gateway provide convenient options to <a href="https://aayutechnologies.com/docs/product/as2-gateway/creating-as2-station/#31-partner-configuration-of-an-as2-station">share all these details in one click</a>.</blockquote><p><strong>For you to send files to your partner:</strong></p><p>This should only require your AS2 ID on your partner’s end; although your partner’s AS2 application may require other configurations as well.</p><p>Some applications may require your partner to explicitly define the type of messages (encrypted, signed, compressed, etc.) that you would be sending; others like AS2 Gateway will intelligently derive them from the incoming messages themselves.</p><h3>Summary</h3><p>EDI transfers over AS2 could fail due to one or more of a variety of reasons. Checking the underlying error messages is often the key to finding and rectifying these issues.</p><p>These issues could be network-level (firewalls and invalid endpoints); content-level (like invalid or swapped certificates); or simple misconfigurations of partner details (AS2 ID, messaging mode or MDN settings) on either your or your partner’s side.</p><blockquote>Online AS2 providers like AS2 Gateway have <a href="https://as2gateway.com/contact/">support channels</a> where you can obtain further support on any EDI exchange issues experienced by you or your partner.</blockquote><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2019/04/top-5-reasons-why-your-edi-transfer-is.html"><em>randomizd.blogspot.com</em></a><em> on April 12, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4185766ade1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EDI Exchange: 10 Tips to Pick Your Perfect EDI Provider]]></title>
            <link>https://medium.com/@janaka_bandara/edi-exchange-10-tips-to-pick-your-perfect-edi-provider-38686ace4587?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/38686ace4587</guid>
            <category><![CDATA[business-process]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[ide]]></category>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[as2]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Fri, 12 Apr 2019 03:06:50 GMT</pubDate>
            <atom:updated>2022-05-03T03:12:54.026Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Towards a smooth business — and the ultimate B2B nirvana</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qL0bRy8BYhpCN9pl.jpg" /><figcaption>Communication, collaboration, success! (courtesy of <a href="https://www.maxpixel.net/Aerial-Paper-Business-Office-Document-Finance-3213924">Pixabay via MaxPixel</a>)</figcaption></figure><p>Picking the correct <a href="https://aayutechnologies.com/docs/edi/what-is-edi/">EDI (<strong>E</strong>lectronic <strong>D</strong>ata <strong>I</strong>nterchange)</a> document exchange provider is very important for a trouble-free business workflow; EDI exchange is becoming a very common requirement in today’s automated business domain. Things may look complicated, but a simple questionnaire would immensely help you pick and choose between the plethora of EDI providers out there.</p><p>Rather than hosting and maintaining your own EDI exchange infrastructure, it is often cheaper and more convenient to bring a third-party (often SaaS, i.e. online/cloud hosted) EDI communication provider.</p><p>However, there are a few points that need careful consideration before you do so:</p><h3>1. What document types are you dealing with?</h3><p>EDI is just an umbrella term for a wide array of electronic trading document standards: <a href="https://www.edibasics.co.uk/edi-resources/document-standards/edifact/">EDIFACT</a>, <a href="https://www.dcs-is-edi.com/2016/03/knowing-the-differences-between-ansi-x12-and-edifact/">ANSI X12</a>, <a href="https://www.techopedia.com/definition/24197/electronic-business-xml--ebxml">ebXML</a> and so forth. Each of these has dozens of types representing actual documents: <a href="https://service.unece.org/trade/untdid/d11a/trmd/cuscar_c.htm">UN/EDIFACT CUSCAR D.11A</a> for customs cargo reporting, <a href="https://en.m.wikipedia.org/wiki/X12_Document_List#Insurance/Health_Series_(INS)">X12 INS-112</a> for property damage reports, and so forth. In real life you wouldn’t _really_ be exchanging EDI, but one or more of these concrete types. So it is important to know whether your provider supports the expected set of document types.</p><p>Providers (like <a href="https://as2gateway.com/">AS2 Gateway</a>) are <em>payload agnostic</em> — meaning whatever you give them, they will transfer verbatim; regardless of whether it is an X12 INS-112, an UN/EDIFACT D.11A, or even a JPEG image or a ZIP file. This is very convenient because you can simply plug-in such a provider easily into your EDI exchange workflow, and forget about everything else.</p><p>However, if you expect the third-party EDI exchange provider to do additional work based on the document type, you might need to contact the vendor and clarify whether they support it.</p><h3>2. Do you wish to only send, only receive, or do both?</h3><p>While most providers do support both, knowing your communication direction would be important. With some high-tier providers, as well as open-source solutions like <a href="https://medium.com/@manjulapiyumal/getting-start-with-as2-protocol-using-as2gateway-and-openas2-796249cfd3ac">OpenAS2</a>, configuring the receiving path could be quite challenging; in others like <a href="https://www.capterra.com/p/175877/AS2Gateway/">AS2 Gateway</a>, most of the possible receiving configurations would be taken care of by the platform itself.</p><h3>3. Do you want to modify or transform the document, before transmission or after reception? Or is it just a send/receive?</h3><p>Some providers offer EDI processing facilities (composing, interpreting and even autogeneration and auto-reply). However, this usually comes at an additional cost and, more importantly, configuration overhead. If you are not looking at modifying or interpreting the content (e.g. your supply chain management (SCM) or warehouse/inventory management system would already be able to do this for you) it would be simple and cost-effective to go for a provider that simply provides EDI <em>transmission</em> capabilities.</p><h3>4. What is the preferred transmission medium for EDI exchange?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*NU7ksFxiq7GdpF5r.jpg" /><figcaption>Medium of transmission is independent of document type. (thanks <a href="https://upload.wikimedia.org/wikipedia/commons/thumb/7/7a/Dhl.a300b4.oo-dlz.arp.jpg">Wikipedia</a>!)</figcaption></figure><p>As mentioned earlier, EDI is just an umbrella term, and even <em>UN/EDIFACT CUSCAR D11A</em> is a document type. Usually what is more important is the <em>medium</em> of transfer.</p><p>A good analogy is a goods delivery system: you have goods (information) packaged in various ways (formats) — bubble wraps, sachets, boxes, crates, etc. — that are loaded into a vehicle (medium of transport) — van, lorry, freight train, etc. — and carried to the destination.</p><p>Similarly, trading <em>information</em> is packaged into various <em>document formats</em> and <em>transmitted</em> to the destinations (trading partners).</p><p>A few famous transports used for EDI exchange are:</p><h4><a href="https://www.investopedia.com/terms/v/value-added-network.asp">VAN</a> (Value Added Network)</h4><p>Contrary to the similarly-named vehicle type, this is a dedicated end-to-end network for exchange of EDI (and other document types) — like privately-owned <a href="https://en.m.wikipedia.org/wiki/Private_railway">rail tracks</a>. These are quite expensive and difficult to operate and maintain, so most organizations are moving away from them.</p><h4><a href="https://aayutechnologies.com/docs/as2/as2-protocol/#13-ediint-applicability-statements-from-the-ietf">ASn (Applicability Statement)</a> family of standards: AS1, AS2, AS3 and AS4</h4><p>These are attempts to transfer EDI data securely over existing public networks (like transporting valuables in sealed containers, on public highways). AS<em>n</em> falls into four standards, utilizing email, HTTP (same tech used by browsers and most web/mobile applications), FTP (file transfer protocol) and web services (over HTTP). By far the most popular and widely-adopted is <a href="https://aayutechnologies.com/docs/as2/how-as2-works/">AS2, the HTTP-based protocol</a>.</p><h4><a href="https://en.m.wikipedia.org/wiki/OFTP">OFTP</a> (ODETTE File Transfer Protocol)</h4><p>This is another transfer mechanism like AS2 (more advanced, in some aspects). OFTP is more popular in Europe rather than in the Americas; AS2 is more popular in the latter, thanks to the <a href="https://www.computerworld.com/article/2577247/networking/wal-mart-chooses-internet-protocol-for-data-exchange.html">initiative by Wal-Mart</a>. OFTP has its own network protocol, whereas others like AS<em>x</em> are built on already-established protocols like HTTP/S.</p><blockquote>If you are dealing with Wal-Mart, Amazon or any other American partners (large or small), you would probably need to go for <strong>AS2</strong>.</blockquote><h3>5. How much do you intend to transfer (say, per month)?</h3><p>While most providers have extensible plans, they are often quite expensive beyond the basic limit. For example, for a 1000 messages/month package, the cost for 1500 total (500 extra) messages could be significantly higher than the cost of directly going for a 2000 messages/month package.</p><p>Additionally, remember that most cloud-based vendors use shared systems for message exchange. If you tend to receive or send batches (spikes) of documents (e.g. every day at midnight) the transmission performance might be affected, especially if the EDIs are large (several megabytes) in size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*2JyAukiIniLWqLfX.jpg" /><figcaption>Bandwidth limit exceeded! (<a href="https://4.bp.blogspot.com/-Ns2uAl-M8mA/VptO8Kpa0mI/AAAAAAAAA5U/hqCaLHhMQ4s/s1600/http%2Bstatus%2B-%2B509%2Bbandwidth%2Blimit%2Bexceeded.jpg">Blogger</a>)</figcaption></figure><h3>6. How many partners are you dealing with?</h3><p>EDI exchange usually happens between predefined <em>partners</em> — digital identities for easily and reliably communicating with and managing real-world business relationships. If you intend to exchange EDIs with multiple partners, go for a platform and package that supports as many as you want.</p><h3>7. Do you want multiple trading identities?</h3><p>Since EDI communication is generally two-way, you as a trading entity will also have your own identity (referred to as <a href="https://aayutechnologies.com/docs/product/as2-gateway/creating-as2-station/"><em>station</em></a>, <em>local station</em>, <em>local partner</em> etc.). Others will refer to you by this identity when they transfer documents to you. The standards allow you to have multiple such identities (e.g. JOHNDOE for dealing with Wal-Mart and JACKSON for Amazon), so if you intend to leverage it, make sure you pick a platform that supports it.</p><h3>8. Do you have strict security or compliance requirements?</h3><p>Cloud or SaaS EDI transfer services would often work for generic EDI exchange use cases. However, some trading partners may impose tighter limits; network configurations (VPNs, firewalls and other layers of security), advanced encryption and signing algorithms, non-standard headers, etc.</p><p>In other cases, your own EDI/AS2 installation may need to meet various security or compliance requirements; either as specific corporate guidelines or in terms of generic standards like PCI-DSS, SOC or HIPAA. They may be imposed by your partner, or your own organizational policies</p><p>In either case, such scenarios may require you to go for a dedicated EDI exchange set-up; or, in extreme cases, an <a href="https://as2gateway.com/on-premise/">on-premise EDI transfer utility</a> which can then be <a href="https://as2gateway.com/case-studies/eshopbox/">further customized</a> to meet the specific requirements and compliance guidelines.</p><h3>9. How would you like to connect the EDI exchange workflow to your existing trade systems?</h3><p>Manual send/receive may work for up to a few hundred messages per month, but you would need automation if:</p><ul><li>your message counts are in several hundreds, thousands, or beyond</li><li>you already have automated trade systems in place — warehouse management, automated order processing, and so forth</li></ul><p>On-premise solutions usually offer more flexibility for connecting with existing systems; however, modern online EDI exchange software also offer convenient B2B system integration options. For example, AS2 Gateway allows you to send and receive messages via <strong>SFTP</strong> (secure file transfer protocol); place your files into a SFTP outbox directory, and read received files via another SFTP inbox directory. Similar entrypoints will soon be introduced for other storage-driven options such as <a href="https://aws.amazon.com/s3/">AWS S3</a>.</p><p>Meanwhile, <strong>REST APIs</strong> are becoming increasingly popular as integration points for EDI exchange. REST does offer simple and convenient interaction with general entities; yet, it may be complex when handling EDI payloads; especially when uploading and downloading large EDI exchanges. AS2 Gateway also has a beta-stage API, where you can gain access on demand.</p><p>Usually, on-premise EDI exchange solutions can support virtually any type of custom integration. <a href="https://as2gateway.com/on-premise/">AS2 Gateway On-premise</a>, for example, can connect with a wide array of systems, protocols and transports; including SSH/SCP, files, S/FTP, HTTP/REST, <strong>message queues</strong> (AMQP, ActiveMQ, IBM MQ, <a href="https://aws.amazon.com/sqs/">AWS SQS</a>, Kafka etc.), <strong>storage-as-a-service</strong> like AWS S3, Dropbox, and so forth.</p><h3>10. What is your implementation deadline — or go-alive timeline?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*L6qutYhsUcjgxmJ1.jpg" /><figcaption>You cannot run away from deadlines. (courtesy of <a href="https://st3.depositphotos.com/11615252/17390/v/450/depositphotos_173904280-stock-illustration-retro-style-businessman-running-documents.jpg">DepositPhotos</a>)</figcaption></figure><p>On-premise has an installation overhead, while online solutions help you get started with EDI exchange in <a href="https://www.youtube.com/watch?v=64kbJqtuYcw">just a few clicks</a>.</p><p>However, setting up communication is the most time-consuming part of the process. Exchanging the <strong>connectivity information</strong> between the trading parties (e.g. partners in AS2) as early as possible, can greatly simplify and speed up the process.</p><ul><li>party identifiers (e.g. AS2 IDs)</li><li>proper message configurations (does each party need their messages encrypted, signed, and/or compressed? if so, with what algorithms?)</li><li>certificates (encryption, signing, HTTPS, trust chains) and other cryptographic entities</li><li>any special markers (e.g. custom entries, such as <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers">HTTP headers</a>) within the EDI exchange process</li></ul><p>Despite the initial overhead, on-premise offers a <strong>smoother ride on the long term</strong>. Fixes and enhancements can go-alive faster, and you can bring in any desired customizations; even proprietary ones, as long as the EDI exchange vendor can support it. If your trading system is highly dynamic, and evolving and scaling fast, bringing in an <a href="https://as2gateway.com/on-premise/">on-premise EDI exchange</a> can considerably save time (and money) long-term.</p><blockquote>Did we cover all your questions? Do you need to know more? Either way, <a href="/service/mailto:support@as2gateway.com">drop us a line</a> and find the one-stop solution to all your EDI/AS2 needs!</blockquote><h3>In conclusion</h3><p>Setting up your EDI exchange may not be trivial. But when you know the requirements and pick the correct provider, the rest would be fairly smooth. Once set up correctly, the EDI exchange would be one of the most automatable, integration-friendly, resilient and maintenance-free components of your trading infrastructure.</p><blockquote>Feel free to <a href="https://as2gateway.com/contact/">contact us</a> for advice or guidance in setting up an EDI/AS2 exchange. In most cases, you can <a href="https://aayutechnologies.com/docs/product/as2-gateway/">follow our documentation</a> and do it yourself as well, on our cloud AS2 Gateway.</blockquote><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2019/04/edi-exchange-10-tips-to-pick-your.html"><em>randomizd.blogspot.com</em></a><em> on April 12, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38686ace4587" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS: Some Tips for Avoiding Those “Holy Bill” Moments]]></title>
            <link>https://medium.com/hackernoon/aws-some-tips-for-avoiding-those-holy-bill-moments-9e48e0cb1a83?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e48e0cb1a83</guid>
            <category><![CDATA[holy-bill-moments]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[ec2]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Fri, 30 Nov 2018 11:03:41 GMT</pubDate>
            <atom:updated>2019-04-30T04:48:07.133Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Cloud is awesome, but it can hurt really bad when it bites back on your wallet.</h4><p>Cloud is awesome: almost-100% availability, near-zero maintenance, pay-as-you-go, and above all, infinitely scalable.</p><p>But the last two can easily bite you back, turning that awesomeness into a billing nightmare.</p><p>And occasionally you see stories like:</p><p><a href="https://medium.com/@asankha/lambda-programming-errors-that-could-cost-you-thousands-of-dollars-a-day-265dfac354f">Lambda programming errors that could cost you thousands of dollars a day!</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/0*f-IYiXflx9aZp0Dk" /><figcaption>Holy Bill!</figcaption></figure><p>And here I unveil a few tips that we learned from our not-so-smooth journey of building <a href="https://slappforge.com/sigma/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=landing">the world’s first serverless IDE</a>, that could help others to avoid some “interesting” pitfalls.</p><h3>Careful with that config!</h3><p>One thing we learned was to never underestimate the power of a configuration.</p><p>If you read the above linked article you would have noticed that it was a simple misconfiguration: a <a href="https://docs.aws.amazon.com/AmazonS3/latest/user-guide/enable-cloudtrail-events.html">CloudTrail logging config</a> that was writing logs to one of the buckets it was already monitoring.</p><p>You could certainly come up with more elaborate and creative examples of creating “service loops” yielding billing black-holes, but the idea is simple: AWS is only as intelligent as the person who configures it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*RZx_O88UuHXEeqnW" /><figcaption>Welcome to Infinite Loop</figcaption></figure><p>(Well, in the above case it was one of my colleagues who configured it, and <em>I was</em> the one who validated it; so you can stop here if you feel like it ;) )</p><p>So, when you’re about to submit a new config update, try to rethink the consequences. You won’t regret it.</p><h3>It’s S3, not your attic.</h3><p><a href="https://docs.aws.amazon.com/aws-technical-content/latest/cost-optimization-storage-optimization/introduction.html">AWS has estimated</a> that 7% of cloud billing is wasted on “unused” storage — space taken up by content of no practical use: obsolete bundles, temporary uploads, old hostings, and the like.</p><h4>Life in a bucket</h4><p>However, it is true that cleaning up things is easier said than done. It is way too easy to forget about an abandoned file than to keep it tracked and delete it when the time comes.</p><p>Probably for the same reason, S3 has provided <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html">lifecycle configurations</a> — time-based automated cleanup scheduling. You can simply say “delete this if it is older than 7 days”, and it will be gone in 7 days.</p><p>This is an ideal way to keep temporary storage (build artifacts, one-time shares etc.) in check, hands-free.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/148/0*PZ51VdnDBT0BhlwX" /><figcaption>Like that daily garbage truck.</figcaption></figure><p>Lifecycle configs can also become handy when you want to delete a huge volume of files from your bucket; rather than deleting individual files (which in itself would <a href="https://aws.amazon.com/s3/pricing/">incur API costs — while deletes are free, listing is not!</a>), you can simply <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-expire-general-considerations.html">set up a lifecycle config rule</a> to expire everything in 1 day. Sit back and relax, while S3 does the job for you!</p><pre>{<br>    &quot;Rules&quot;: [<br>        {<br>            &quot;Status&quot;: &quot;Enabled&quot;,<br>            &quot;Prefix&quot;: &quot;&quot;,<br>            &quot;Expiration&quot;: {<br>                &quot;Days&quot;: 1<br>            }<br>        }<br>    ]<br>}</pre><p>Alternatively you can move the no-longer-needed-but-not-quite-ready-to-let-go stuff into <a href="https://aws.amazon.com/glacier/pricing/">Glacier, for a fraction of the storage cost</a>; say, for stuff under the subpath archived:</p><pre>{<br>    &quot;Rules&quot;: [<br>        {<br>            &quot;Filter&quot;: {<br>                &quot;Prefix&quot;: &quot;archived&quot;<br>            },<br>            &quot;Status&quot;: &quot;Enabled&quot;,<br>            &quot;Transitions&quot;: [<br>                {<br>                    &quot;Days&quot;: 1,<br>                    &quot;StorageClass&quot;: &quot;GLACIER&quot;<br>                }<br>            ]<br>        }<br>    ]<br>}</pre><p>But before you do that…</p><h4>Ouch, it’s versioned!</h4><p><em>(Inspired by true events.)</em></p><p>I put up a lifecycle config to delete about 3GB of bucket access logs (millions of files, obviously), and thought everything was good — until, a month later, I got the same S3 bill as the previous month :(</p><p>Turns out that the bucket had had versioning enabled, so <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/DeletingObjectVersions.html">deletion does not really delete the object</a>.</p><p>So with versioning enabled, you need to explicitly tell the S3 lifecycle logic to:</p><ul><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex6">discard non-current (deleted) object versions</a>, and</li><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html#lifecycle-config-conceptual-ex7">expire old delete markers</a></li></ul><p>in order to completely get rid of the “deleted” content and the associated <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/DeleteMarker.html"><em>delete markers</em></a>.</p><p>So much for “simple” storage service ;)</p><h4>CloudWatch is your pal</h4><p>Whenever you want to <a href="https://serverfault.com/questions/84815/how-can-i-get-the-size-of-an-amazon-s3-bucket#710080">find out the total sizes</a> occupied by your buckets, just iterate through your <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html">AWS/S3 CloudWatch Metrics namespace</a>. There&#39;s no way—suprise, surprise—to check bucket size natively from S3; even the S3 dashboard relies on CloudWatch, so why not you?</p><p>Quick snippet to view everything? (uses <a href="https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html">aws-cli</a> and bc on bash)</p><pre>yesterday=$(date -d @$((($(date +%s)-86400))) +%F)<br>for bucket in `aws s3api list-buckets --query &#39;Buckets[*].Name&#39; --output text`; do<br>        size=$(aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time ${yesterday}T00:00:00 --end-time $(date +%F)T00:00:00 --period 86400 --metric-name BucketSizeBytes --dimensions Name=StorageType,Value=StandardStorage Name=BucketName,Value=$bucket --statistics Average --output text --query &#39;Datapoints[0].Average&#39;)<br>        if [ $size = &quot;None&quot; ]; then size=0; fi<br>        printf &quot;%8.3f  %s\n&quot; $(echo $size/1048576 | bc -l) $bucket<br>done</pre><h3>EC2: sweep the garbage, plug the holes</h3><p>EC2 makes it trivial to manage your virtual machines — compute, storage and networking. However, its simplicity also means that it can leave a trail of unnoticed garbage and billing leaks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/0*IOVSUVbHRWydeZ_r" /><figcaption>EC2: Elastic Compute Cloud</figcaption></figure><h4>Pick your instance type</h4><p>There’s a plethora of settings when creating a new instance. Unless there are specific performance requirements, picking a <a href="https://aws.amazon.com/ec2/instance-types/t2/">T2-class instance type</a> with <a href="https://aws.amazon.com/ebs/">Elastic Block Store (EBS)-backed storage</a> and 2–4 GB of RAM would suffice for most needs.</p><p>Despite being free tier-eligible, t2.micro can be a PITA if your server could receive compute-or memory-intensive loads at some point; in these cases t2.micro tends to simply freeze (probably has to do with <a href="https://forums.aws.amazon.com/thread.jspa?messageID=789458#jive-message-holder">running out of CPU credits</a>?), causing more trouble than it&#39;s worth.</p><h4>Clean up AMIs and snapshots</h4><p>We habitually tend to take periodic snapshots of our EC2 instances as backups. Some of these are made into <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html">Machine Images (AMIs)</a> for reuse or <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sharingamis-explicit.html">sharing with other AWS users</a>.</p><p>We easily forget about the other snapshots.</p><p>While snapshots <a href="https://aws.amazon.com/ebs/pricing/#Amazon_EBS_Snapshots_to_Amazon_S3">don’t get billed for their full volume sizes</a>, they can add up to significant garbage over time. So it is important to periodically visit and clean up your <a href="https://console.aws.amazon.com/ec2/v2/home#Snapshots">EC2 snapshots tab</a>.</p><p>Moreover, creating new AMIs would usually mean that older ones become obsolete; they can be “deregistered” from the <a href="https://console.aws.amazon.com/ec2/v2/home#Images">AMIs tab</a> as well.</p><p>But…</p><h4>Who’s the culprit — AMI or snapshot?</h4><p>The actual charges are on <em>snapshots</em>, not on AMIs themselves.</p><p>And it gets tricky because <a href="https://serverfault.com/questions/892384/why-aws-snapshot-is-not-removed-after-de-registering-aws-ami">deregistering an AMI does not automatically delete the corresponding snapshot</a>.</p><p>You usually have to copy the AMI ID, go to snapshots, look for the ID in the description field, and nuke the matching snapshot. Or, if you are brave (and lazy), select and delete <em>all</em> snapshots; AWS will prevent you from deleting the ones that are being used by an AMI.</p><h4>Likewise, for instances and volumes</h4><p>Compute is billed while an EC2 instance is running; but its <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumes.html">storage volume</a> is billed all the time — right up to deletion.</p><p>Volumes usually get nuked when you terminate an instance; however, if you’ve played around with volume attachment settings, there’s a chance that detached volumes are left behind in your account. Although not attached to an instance, these still occupy space; and so AWS charges for them.</p><p>Again, simply go to the <a href="https://console.aws.amazon.com/ec2/v2/home#Volumes">volumes tab</a>, select the volumes in “available” state, and hit delete to get rid of them for good.</p><h4>Tag your EC2 stuff: instances, volumes, snapshots, AMIs and whatnot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*Das3GLpOt6LefhLn" /><figcaption>Tag ‘em.</figcaption></figure><p>It’s very easy to forget what state was in the instance, at the time that snapshot was made. Or the purpose of that running/stopped instance which nobody seems to take ownership or responsibility of.</p><p>Naming and tagging can help avoid unpleasant surprises (“Why on earth did you delete that last month’s prod snapshot?!”); and also help you quickly decide what to toss (“We already have an 11–05 master snapshot, so just delete everything older than that”).</p><h4>You stop using, and we start billing!</h4><p>Sometimes, the AWS Lords work in mysterious ways.</p><p>For example, <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html">Elastic IP Addresses (EIPs)</a> are free as long as they are attached to a running instance. But they start getting charged by the hour, as soon as the instance is stopped; or if they get into a “detached” state (not attached to a running instance) in some way.</p><p>Some prior knowledge about the service you’re about to sign up for, can prevent some nasty surprises of this fashion. A quick pricing page lookup or google can be a deal-breaker.</p><h3>Pay-per-use vs pay-per-allocation</h3><p>Many AWS services follow one or both of the above patterns. The former is trivial (you simply pay for the time/resources you actually use, and enjoy a zero bill for the rest of the time) and hard to miss; but the latter can be a bit obscure and quite easily go unnoticed.</p><p>Consider EC2: you mainly pay for <a href="https://aws.amazon.com/ec2/pricing/on-demand/">instance runtime</a> but you also pay for the storage (volumes, snapshots, AMIs) and <a href="https://hackernoon.com/tagged/network">network</a> allocations (like <strong>inactive</strong> Elastic IPs) even if your instance has been stopped for months.</p><p>There are many more examples, especially in the <a href="https://hackernoon.com/why-serverless-why-now-f09ce43c4767">serverless domain</a> (which we ourselves are incidentally more familiar with):</p><ul><li><strong>Kinesis</strong> charges <a href="https://aws.amazon.com/kinesis/data-streams/pricing/">by shard-hour</a> — even if all your shards are idle</li><li><strong>DynamoBB</strong> charges for <a href="https://aws.amazon.com/dynamodb/pricing/">storage and read/write in terms of “capacity units”</a> — thankfully there’s a non-expiring free tier!</li><li><strong>RDS</strong> (very similar to EC2) charges for instance runtime, whether busy or idle (<a href="https://aws.amazon.com/rds/aurora/pricing/">Aurora Serverless</a> seems to be trying to change this to some extent)</li><li><strong>KMS</strong> charges <a href="https://aws.amazon.com/kms/pricing/">a flat fee for each customer-managed key (CMK)</a> whether you use it or not</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*7VbkcarBjciRLYop" /><figcaption>Each block adds a bit more to your cost.</figcaption></figure><p>Meanwhile, some services secretly set up their own monitoring, backup and other “utility” entities. These, although (probably!) meant to do good, can secretly seep into your bill:</p><ul><li>DynamoDB sets up <a href="https://github.com/awslabs/serverless-application-model/issues/507">CloudWatch Alarms</a>; these are left behind even after the corresponding tables have been deleted (at least when managed via CloudFormation).</li><li>RDS automatically creates <a href="https://aws.amazon.com/rds/details/backup/">instance volume snapshots</a>, at termination as well as during daily maintenance (esp. when deployed via the <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-rds-database-instance.html">“default” CloudFormation configs</a>; these can easily add up over your storage quotas</li></ul><p>These are the main culprits that often appear in our <a href="https://hackernoon.com/tagged/aws">AWS</a> bills; certainly there are better examples, but you get the point.</p><h3>CloudWatch (yeah, again)</h3><p>Many services already — or can be configured to — report <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html">usage metrics</a> to CloudWatch. Hence, with some domain knowledge of which metric maps into which billing component (e.g. S3 storage cost is represented by the summation of the BucketSizeBytes metric across all entries of the AWS/S3 namespace), you can build a complete billing and monitoring solution around <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html">CloudWatch Metrics</a> (or delegate the job to a third-party service like <a href="https://www.datadoghq.com/">DataDog</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/0*H-xEWWgpyd_RY1J8" /><figcaption>CloudWatch</figcaption></figure><p>CloudWatch in itself is <a href="https://aws.amazon.com/cloudwatch/pricing/">mostly free</a>, and its metrics have automatic <a href="https://aws.amazon.com/cloudwatch/faqs/#AWS_resource_and_custom_metrics_monitoring">summarization mechanisms</a> so you don’t have to worry about overwhelming it with age-old garbage — or getting overwhelmed with off-the-limit capacity bills.</p><h3>The Billing API</h3><p>Although AWS does have a dedicated <a href="https://console.aws.amazon.com/billing/home">Billing Dashboard</a>, logging in and checking it every single day is not something you would add to your agenda (at least not for API/CLI minds like you and me).</p><p>Luckily, AWS offers a <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ce-api.html">billing API</a> whereby you can obtain a fairly granular view of your current outstanding bill, over any preferred time period — broken down by services or actual API operations.</p><p>Catch is, this API is not free: each invocation costs you $0.01. Of course this is negligible — considering the risk of having to pay several dozens — or even hundreds or thousands in some cases — it is worth having a $0.30/month billing monitor to track down any anomalies before it’s too late.</p><p>Food for thought: with support for <a href="https://developers.google.com/web/updates/2017/04/headless-chrome">headless Chrome</a> offered <a href="https://cloud.google.com/blog/products/gcp/introducing-headless-chrome-support-in-cloud-functions-and-app-engine">for Google Cloud Functions</a>, one might be able to set up a serverless workflow that logs into the AWS dashboard and checks the bill for you. Something to try out during free time (if some ingenious folk hasn’t hacked it together already).</p><h3>Billing alerts</h3><p>Strangely (or perhaps not ;)) AWS doesn’t offer a way to put up a hard limit for billing; despite the <a href="https://news.ycombinator.com/item?id=13072296">numerous user requests</a> and disturbing incident reports all over the web. Instead, they offer <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html">alerts</a> for various billing “levels”; you can subscribe for notifications like “bill at x% of the limit” and “limit exceeded”, via email or SNS (handy for <a href="https://slappforge.com/docs/sigma/components/aws/sns.html#sns-as-a-trigger">automation via Lambda</a>!).</p><p><strong>My advice:</strong> this is a <strong>must-have</strong> for every AWS account. If we had one in place, we could already have saved well over thousands of dollars to date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mhNikTFnXgbrYI_8.jpg" /><figcaption>Don’t wait till they become worthless pieces of plastic.</figcaption></figure><h3>Organizational accounts</h3><p>If you want to delegate AWS access to third parties (testing teams, contract-basis devs, demo users etc.), it might be a good idea to create a sub-account by converting your root account into an <a href="https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidated-billing.html">AWS organization with consolidated billing</a> enabled.</p><p>(While it is possible to do almost the same using an <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html">IAM user</a>, it will not provide resource isolation; everything would be stuffed in the same account, and painstakingly complex IAM policies may be required to isolate entities across users.)</p><p>Our <a href="https://lk.linkedin.com/in/asankha">CEO and colleague Asankha</a> has <a href="https://medium.com/@asankha/creating-isolated-aws-accounts-for-testing-and-experimentation-9795a8d2e2de">written about this quite comprehensively</a> so I’m gonna stop at that.</p><h3>And finally: Monitor. Monitor. Monitor.</h3><p>No need to emphasize on this — my endless ramblings should already have conveyed its importance.</p><p>So, good luck with that!</p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2018/11/aws-some-tips-for-avoiding-those-holy.html?m=1"><em>randomizd.blogspot.com</em></a><em> on November 30, 2018.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e48e0cb1a83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/aws-some-tips-for-avoiding-those-holy-bill-moments-9e48e0cb1a83">AWS: Some Tips for Avoiding Those “Holy Bill” Moments</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serverless Security: Putting it on Autopilot]]></title>
            <link>https://medium.com/think-serverless/serverless-security-putting-it-on-autopilot-74198bf686d2?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/74198bf686d2</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[serverless-architecture]]></category>
            <category><![CDATA[lambda]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Wed, 14 Nov 2018 00:08:00 GMT</pubDate>
            <atom:updated>2018-11-29T09:00:39.112Z</atom:updated>
            <content:encoded><![CDATA[<h4>How serverless security can go wrong, and what we can do about it</h4><p><strong>Ack</strong>: This article is a remix of stuff learned from personal experience as well as from multiple other sources on serverless security. I cannot list down or acknowledge all of them here; nevertheless, special thanks should go to <a href="https://www.theregister.co.uk/2018/05/15/stale_serverless_functions/">The Register</a>, <a href="https://hackernoon.com/many-faced-threats-to-serverless-security-519e94d19dba">Hacker Noon</a>, <a href="https://www.puresec.io/blog/serverless-top-10-released">PureSec</a>, and the <a href="https://serverless.email/">Serverless Status</a> and <a href="https://serverless.curated.co/">Serverless (Cron)icle</a> newsletters.</p><p>We all love to imagine that our systems are secure. And then…</p><h3>BREACH!!!</h3><p>A <em>very</em> common nightmare shared by every developer, sysadmin and, ultimately, CISO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*UiY_acWscYoH567Q.jpg" /><figcaption>You’d better inform the boss…</figcaption></figure><h4>Inevitable?</h4><p>One basic principle of computer security states that <a href="https://www.techrepublic.com/article/an-absolutely-secure-network-is-not-possible-but-the-risk-can-be-managed/">no system can attain absolute security</a>. Just like people: <a href="https://www.goodreads.com/quotes/508462-no-one-is-perfect-that-s-why-pencils-have-erasers">nobody is perfect</a>. Not unless it is fully isolated from the outside; which, by today’s standards, is next to impossible — besides, what’s the point of having a system that cannot take inputs and provide outputs?</p><p>Whatever advanced security precaution you take, attackers will eventually find a way around. Even if you use the most stringent encryption algorithm with the longest possible key size, attackers will eventually brute-force their way through; although it could be time-wise infeasible at present, who can guarantee that a bizarre technical leap would render it possible tomorrow, or the next day?</p><p>But it’s not the brute-force that you should really be worried about: <a href="https://www.itgovernance.co.uk/blog/50-of-the-worst-breaches-caused-by-inadvertent-human-error/">human errors</a> are way more common, and can have devastating effects on systems security; much more so than a brute-forced passkey. Just <a href="https://qz.com/445233/inside-the-irss-massive-data-breach/">have a peek at this story</a> where some guys just walked into the U.S. IRS building and siphoned out millions of dollars, without using a single so-called “hacking” technique.</p><p>As long as systems are made and operated by people — who are error-prone by nature — they will never be truly secure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*YvAvYQOuvR0MyyIE.jpg" /><figcaption>Remember those old slides from college days?</figcaption></figure><h3>So, are we doomed?</h3><p>No!</p><p>Ever seen the insides of a ship?</p><p>How its hull is divided into compartments — so that one leaking compartment does not cause the whole ship to sink?</p><p>People often apply a similar concept in designing software: multiple modules so that one compromised module doesn’t bring the whole system down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/0*q-BnIQxIIrGN6FGK.jpg" /><figcaption>A ship’s watertight hull compartments</figcaption></figure><p>Combined with the <a href="https://en.m.wikipedia.org/wiki/Principle_of_least_privilege">principle of least privilege</a>, this means that a component will compromise the least possible degree of security — ideally the attacker will only be able to wreak havoc within the bounds of the module’s security scope, never beyond.</p><p>Reducing the <a href="https://thenewstack.io/six-security-considerations-for-serverless-environments/"><em>blast radius</em></a> of the component, and consequently the <a href="https://theburningmonk.com/2017/08/many-faced-threats-to-serverless-security/"><em>attack surface</em></a> that it exposes for the overall system.</p><p>A security <em>sandbox</em>, you could say.</p><p>And a pretty good one at that.</p><h3>PoLP: The Principle of Least Privilege</h3><blockquote>Never give someone — or something — more freedom than they need.</blockquote><p>More formally,</p><blockquote>Every module must be able to access only the information and resources that are necessary for its legitimate purpose.</blockquote><blockquote>— <a href="https://en.m.wikipedia.org/wiki/Principle_of_least_privilege">Wikipedia</a></blockquote><p>This way, if the module misbehaves (or is forced to misbehave, by an entity with malicious intent — a <em>hacker</em>, in English), the potential harm it can cause is minimized; without any preventive “action” being taken, and even before the “breach” is identified!</p><h4>It never gets old</h4><p>While the principle was initially brought up in the context of legacy systems, it is even more so applicable for “modern” architectures; SOA (well, maybe not so “modern”), microservices, and FaaS (serverless functions, hence serverless security) as well.</p><p>The concept is pretty simple: use the underlying access control mechanisms to restrict the permissions available for your “unit of execution”; may it be a simple HTTP server/proxy, web service backend, microservice, container, or serverless function.</p><h3>Meanwhile, in the land of no servers…</h3><p>With <a href="https://thenewstack.io/why-serverless-is-being-adopted-at-a-faster-rate-than-expected/">increased worldwide adoption</a> of serverless technologies, the significance of serverless security, and the value of our PoLP, is becoming <a href="https://www.puresec.io/blog/musings-on-serverless-application-security-with-simon-wardley">more obvious than ever</a>.</p><h4>Server-less = effort-less</h4><p>Not having to provision and manage the server (environment) means that serverless DevOps can proceed at an insanely rapid pace. With CI/CD in place, it’s just a matter of code, commit and push; everything would be up and running within minutes, if not seconds. No SSH logins, file uploads, config syncs, service restarts, routing shifts, or any of the other pesky DevOps chores associated with a traditional deployment.</p><h4>“Let’s fix the permissions later.”</h4><p>Alas, that’s a common thing to hear among those “ops-free” devs (like myself). You’re in a haste to push the latest updates to staging, and the “easy path” to avoid a plethora of “permission denied” errors is to relax the permissions on your FaaS entity (<a href="https://aws.amazon.com/lambda/">AWS Lambda</a>, <a href="https://azure.microsoft.com/en-us/overview/serverless-computing/">Azure Function</a>, whatever).</p><p>Staging will soon migrate to prod. And so will your “over-permissioned” function.</p><p>And it will stay there. <a href="https://www.theregister.co.uk/2018/05/15/stale_serverless_functions/">Far longer than you think.</a> You will eventually shift your traffic to updated versions, leaving behind the old one untouched; in fear of breaking some other dependent component in case you step on it.</p><p>And then come the sands of time, covering the old function from everybody’s memories.</p><p>An obsolete function with unpatched dependencies and possibly flawed logic, having full access to your cloud resources.</p><p>A <em>serverless time bomb</em>, if there ever was one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*8jsCvKVzaH33Da8g" /><figcaption>Waiting for the perfect time… to explode</figcaption></figure><h4>Yes, blast radius; again!</h4><p>If we adhere to the least privilege principle, right from the staging deployment, it would greatly reduce the blast radius: by limiting what the function is allowed to do, we automatically limit the “extent of exploitation” upon the rest of the system if its control ever falls into the wrong hands.</p><h3>Nailing serverless security: on public cloud platforms</h3><p>These things are easier said than done.</p><p>At the moment, among the leaders of public-cloud FaaS technology, only AWS has a sufficiently flexible serverless security model. <a href="https://cloud.google.com/functions/docs/concepts/iam#runtime_service_account">GCP automatically assigns a default project-level Cloud Platform service account</a> to all its functions in a given project, meaning that all your functions will be in one basket in terms of security and access control. Azure’s IAM model <a href="https://www.twistlock.com/2018/09/25/iam-roundup-aws-vs-azure-vs-gcp/">looks more promising</a>, but it still lacks the cool stuff like <a href="https://stackoverflow.com/questions/40838619/azure-function-role-like-permissions-to-stop-azure-virtual-machines#comment-68899907">automatic role-based runtime credential assignments</a> available in both AWS and GCP.</p><p>AWS has applied its own <a href="https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html">IAM role-based permissions model</a> for its Lambda functions, granting users the flexibility to define a custom IAM role — with fully customizable permissions — for every single Lambda function if so desired. It has an <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html">impressive array of predefined roles</a> that you can extend upon and has <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html">well-defined strategies</a> for scoping permission to categories of resources or principals; merging rules that refer to the same set of resources or operations; and so forth.</p><p>This whole hierarchy finally boils down to a set of permissions, each of which takes a rather <a href="https://aws.amazon.com/blogs/security/back-to-school-understanding-the-iam-policy-grammar/">straightforward format</a>:</p><pre>{<br>    &quot;Effect&quot;: &quot;Allow|Deny&quot;,<br>    &quot;Action&quot;: &quot;API operation matcher (pattern), or array of them&quot;,<br>    &quot;Resource&quot;: &quot;entity matcher (pattern), or array of them&quot;<br>}</pre><p>In English, this simply means:</p><p><strong>Allow</strong> (or <strong>deny</strong>) an entity (user, EC2 instance, lambda; whatever) that possesses this permission, to perform the matching API operation(s) against the matching resource(s).</p><p>(There are non-mandatory fields Principal and Condition as well, but we&#39;ll skip them here for the sake of brevity.)</p><h4>Okay, okay! Time for some examples.</h4><pre>{<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Action&quot;: &quot;s3:PutObject&quot;,<br>    &quot;Resource&quot;: &quot;arn:aws:s3:::my-awesome-bucket/*&quot;<br>}</pre><p>This allows the assignee to put an object (s3:PutObject) into the bucket named my-awesome-bucket.</p><pre>{<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Action&quot;: &quot;s3:PutObject&quot;,<br>    &quot;Resource&quot;: &quot;arn:aws:s3:::my-awesome-*&quot;<br>}</pre><p>This is similar, but allows the put to be performed on any bucket whose name begins with my-awesome-.</p><pre>{<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Action&quot;: &quot;s3:*&quot;,<br>    &quot;Resource&quot;: &quot;*&quot;<br>}</pre><p>This allows the assignee to do <strong>any</strong> S3 operation (get/put object, delete object, or even <em>delete bucket</em>) against <strong>any</strong> bucket in its owning AWS account.</p><p>And now the <em>silver bullet</em>:</p><pre>{<br>    &quot;Effect&quot;: &quot;Allow&quot;,<br>    &quot;Action&quot;: &quot;*&quot;,<br>    &quot;Resource&quot;: &quot;*&quot;<br>}</pre><p>Yup, that one allows oneself to <strong>do anything</strong> on <strong>anything</strong> in the AWS account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*s7CXbdWPc4wwUzqO.jpg" /><figcaption>The silver bullet</figcaption></figure><p>Kind of like the <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_job-functions.html#jf_administrator">AdministratorAccess</a> managed policy.</p><p>And if your principal (say, lambda) gets compromised, the attacker effectively has admin access to your AWS account!</p><p>A serverless security nightmare. Needless to say.</p><p>To be avoided at all cost.</p><p>Period.</p><p>In that sense, the best option would be a series of permissions of the first kind; ones that are the least permissive (most restrictive) and cover a narrow, well-defined scope.</p><h3>How hard can that be?</h3><p>The caveat is that you have to do this for every single operation within that computation unit — say lambda. <em>Every single one.</em></p><p>And it gets worse when you need to configure <a href="https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html">event sources</a> for triggering those units.</p><p>Say, for an <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-lambda-integration.html">API Gateway-triggered</a> lambda, where the <em>API Gateway service</em> must be granted permission to invoke your lambda in the scope of a specific APIG endpoint (in <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-permission.html">CloudFormation syntax</a>):</p><pre>{<br>  &quot;Type&quot;: &quot;AWS::Lambda::Permission&quot;,<br>  &quot;Properties&quot;: {<br>    &quot;Action&quot;: &quot;lambda:InvokeFunction&quot;,<br>    &quot;FunctionName&quot;: {<br>      &quot;Ref&quot;: &quot;LambdaFunction&quot;<br>    },<br>    &quot;SourceArn&quot;: {<br>      &quot;Fn::Sub&quot;: [<br>        &quot;arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/*/${__Method__}${__Path__}&quot;,<br>        {<br>          &quot;__Method__&quot;: &quot;POST&quot;,<br>          &quot;__Path__&quot;: &quot;/API/resource/path&quot;,<br>          &quot;__ApiId__&quot;: {<br>            &quot;Ref&quot;: &quot;RestApi&quot;<br>          }<br>        }<br>      ]<br>    },<br>    &quot;Principal&quot;: &quot;apigateway.amazonaws.com&quot;<br>  }<br>}</pre><p>Or for a <a href="https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis-example.html">Kinesis stream-powered</a> lambda, in which case things get more complicated: the <em>Lambda function</em> requires access to <em>watch</em> and <em>pull from</em> the stream, while the <em>Kinesis service</em> also needs permission to <em>trigger</em> the lambda:</p><pre>  &quot;LambdaFunctionExecutionRole&quot;: {<br>    &quot;Type&quot;: &quot;AWS::IAM::Role&quot;,<br>    &quot;Properties&quot;: {<br>      &quot;ManagedPolicyArns&quot;: [<br>        &quot;arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole&quot;<br>      ],<br>      &quot;AssumeRolePolicyDocument&quot;: {<br>        &quot;Version&quot;: &quot;2012-10-17&quot;,<br>        &quot;Statement&quot;: [<br>          {<br>            &quot;Action&quot;: [<br>              &quot;sts:AssumeRole&quot;<br>            ],<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>              &quot;Service&quot;: [<br>                &quot;lambda.amazonaws.com&quot;<br>              ]<br>            }<br>          }<br>        ]<br>      },<br>      &quot;Policies&quot;: [<br>        {<br>          &quot;PolicyName&quot;: &quot;LambdaPolicy&quot;,<br>          &quot;PolicyDocument&quot;: {<br>            &quot;Statement&quot;: [<br>              {<br>                &quot;Effect&quot;: &quot;Allow&quot;,<br>                &quot;Action&quot;: [<br>                  &quot;kinesis:GetRecords&quot;,<br>                  &quot;kinesis:GetShardIterator&quot;,<br>                  &quot;kinesis:DescribeStream&quot;,<br>                  &quot;kinesis:ListStreams&quot;<br>                ],<br>                &quot;Resource&quot;: {<br>                  &quot;Fn::GetAtt&quot;: [<br>                    &quot;KinesisStream&quot;,<br>                    &quot;Arn&quot;<br>                  ]<br>                }<br>              }<br>            ]<br>          }<br>        }<br>      ]<br>    }<br>  },<br>  &quot;LambdaFunctionKinesisTrigger&quot;: {<br>    &quot;Type&quot;: &quot;AWS::Lambda::EventSourceMapping&quot;,<br>    &quot;Properties&quot;: {<br>      &quot;BatchSize&quot;: 100,<br>      &quot;EventSourceArn&quot;: {<br>        &quot;Fn::GetAtt&quot;: [<br>          &quot;KinesisStream&quot;,<br>          &quot;Arn&quot;<br>        ]<br>      },<br>      &quot;StartingPosition&quot;: &quot;TRIM_HORIZON&quot;,<br>      &quot;FunctionName&quot;: {<br>        &quot;Ref&quot;: &quot;LambdaFunction&quot;<br>      }<br>    }<br>  },<br>  &quot;KinesisStreamPermission&quot;: {<br>    &quot;Type&quot;: &quot;AWS::Lambda::Permission&quot;,<br>    &quot;Properties&quot;: {<br>      &quot;Action&quot;: &quot;lambda:InvokeFunction&quot;,<br>      &quot;FunctionName&quot;: {<br>        &quot;Ref&quot;: &quot;LambdaFunction&quot;<br>      },<br>      &quot;SourceArn&quot;: {<br>        &quot;Fn::GetAtt&quot;: [<br>          &quot;KinesisStream&quot;,<br>          &quot;Arn&quot;<br>        ]<br>      },<br>      &quot;Principal&quot;: &quot;kinesis.amazonaws.com&quot;<br>    }<br>  }</pre><p>So you see, with this granularity, comes great <strong>power</strong> as well as great <strong>responsibility</strong>. One missing permission — heck, one mistyped letter — and it’s <a href="https://medium.freecodecamp.org/running-around-the-block-a-beginner-meets-aws-lambda-560a1f2849ae">403 AccessDeniedException</a>.</p><p>No easy way; you just have to track down every AWS resource triggering or accessed by your function, look up the docs, pull out your hair, and come up with the necessary permissions.</p><h3>But… but… that’s too much work!</h3><p>Yup, it is. If you’re doing it <em>manually</em>.</p><p>But <a href="https://cars.usnews.com/cars-trucks/best-cars-blog/2016/09/why-are-manual-transmissions-disappearing">who drives manual these days?</a> :)</p><p>Fortunately there are quite a few options, if you’re already into automating stuff:</p><h4>serverless-puresec-cli: thanks PureSec!</h4><p>If you’re using the famous <a href="https://serverless.com/framework/docs/providers/aws/guide/iam/">Serverless Framework</a> — which means you’re already covered on the trigger permissions front — there’s the <a href="https://www.puresec.io/blog/generating-least-privileged-iam-roles-for-aws-lambda-functions-the-easy-way">serverless-puresec-cli</a> plugin from <a href="https://www.puresec.io/blog/author/puresec-team">Puresec</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*OoKFEDbBJrQfNaMX" /><figcaption>Puresec</figcaption></figure><p>The plugin can statically analyze your lambda code and generate a least-privilege role. Looks really cool, but the caveat is that you have to run the serverless puresec gen-roles command before every deployment with code changes; I couldn&#39;t yet find a way to run it automatically - during serverless deploy, for example. Worse, it just prints the generated roles into stdout; so you have to manually copy-paste it into serverless.yml, or use some other voodoo to actually inject it into the deployment configuration (hopefully things would improve in the future :))</p><h4>AWS Chalice: from the Gods</h4><p>If you’re a Python fan, <a href="https://github.com/aws/chalice/blob/master/docs/source/topics/configfile.rst">Chalice</a> is capable of auto-generating permissions for you, natively. Chalice is awesome in many aspects; super-fast deployments, annotation-driven triggers, little or no configurations to take care of, and so forth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/100/0*GaXXbAQatAx7mxZb" /><figcaption>AWS Chalice</figcaption></figure><p>However, despite being a direct hand-down from the AWS gods, it seems to have missed the word “minimal” when it comes to permissions; if you have the code to <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.list_objects">list the contents</a> of some bucket foo, it will generate permissions for listing content of <em>all buckets in the AWS account</em> (&quot;Resource&quot;: &quot;*&quot; instead of &quot;Resource&quot;: &quot;arn:aws:s3:::foo/*&quot;), not just the bucket you are interested in. Not cool!</p><h4>No CLI? go for SLAppForge Sigma</h4><p>If you’re a beginner, or not that fond of CLI tooling, there’s <a href="https://www.slappforge.com/sigma/?utm_medium=blog&amp;utm_source=generic&amp;utm_campaign=feature">Sigma from SLAppForge</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/0*fNcNtW_Kbp2Z75_I.png" /><figcaption>SLAppForge Sigma</figcaption></figure><p>Being a fully-fledged browser IDE, Sigma will automatically analyze your code as you compose (type or <a href="https://www.youtube.com/watch?v=cDGY1D6Chp0">drag-n-drop</a>) it, and derive the necessary permissions — for the Lambda runtime <em>as well as for the triggers</em> — so you are fully covered. The recently introduced <em>Permission Manager</em> also allows you to modify these auto-generated permissions if you desire; for example, if you are integrating a new AWS service/operation that Sigma doesn’t yet know about.</p><p>Plus, with Sigma, you never have to worry about any other configurations; resource configs, trigger mappings, entity interrelations and so forth — the IDE takes care of it all.</p><p>The caveat is that Sigma only supports NodeJS at the moment; but Python, Java and other cool languages are on their way!</p><p>(Feel free to comment below, if you have other cool serverless security policy generation tools in mind! And no, <a href="https://awspolicygen.s3.amazonaws.com/policygen.html">AWS Policy Generator</a> doesn’t count.)</p><h3>In closing</h3><p>Least privilege principle is crucial for serverless security, and software design in general; sooner or later, it <em>will</em> save your day.</p><p>Lambda’s highly granular IAM permission model is ideal for the PoLP.</p><p>Tools like the <em>Puresec CLI plugin</em>, all-in-one <em>Sigma IDE</em> and <em>AWS Chalice</em> can automate security policy generation; making your life easier, and still keeping the PoLP promise.</p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2018/11/serverless-security-putting-it-on.html?m=1"><em>randomizd.blogspot.com</em></a><em> on November 14, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74198bf686d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/think-serverless/serverless-security-putting-it-on-autopilot-74198bf686d2">Serverless Security: Putting it on Autopilot</a> was originally published in <a href="https://medium.com/think-serverless">Think Serverless</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My bots are now homeless.]]></title>
            <link>https://medium.com/@janaka_bandara/my-bots-are-now-homeless-6879b18e1ed8?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/6879b18e1ed8</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[google-apps-script]]></category>
            <category><![CDATA[sigma-ide]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Wed, 18 Jul 2018 04:37:33 GMT</pubDate>
            <atom:updated>2018-07-18T05:15:00.104Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Placeless. Backendless. Serverless.</h4><p>I usually keep an eye on various websites — for latest publications, hot new offers, limited-time games and contests, and the like.</p><p>Most of these do not offer a “clean” notification system, such as an RSS feed. So I often have to <a href="https://en.m.wikipedia.org/wiki/Web_scraping">scrape their HTML</a> to get to what I need.</p><p>Which means I often need to run some custom <a href="https://www.encyclopedia.com/computing/dictionaries-thesauruses-pictures-and-press-releases/string-manipulation">string manipulation</a> magic to get to what I need.</p><p>And I need it to be periodic (who knows when the next hot update would surface?).</p><p>And automatic (I have more important things to do during my day).</p><p>And remotely hosted (I don’t want to keep my laptop running 24×7, with an uninterrupted internet connection).</p><p>So far I have been relying on <a href="https://developers.google.com/apps-script/">Google Apps Script</a> (and more recently, <a href="https://cloud.google.com/appengine/">Google App Engine</a>) for driving these sorts of home-made integration “snippets”; however, with the <a href="https://medium.com/@manjulapiyumal/will-serverless-be-the-future-on-enterprice-application-development-9a1901798b0a">whole world immersing itself in serverless</a>, why shouldn’t I?</p><p>So I set out to migrate one of my scripts, written for monitoring a Chinese retail website. The site occasionally publishes various discounted offers and seasonal games where I can earn nice coupons and credits via daily plays. But for some reason the site does not send out promotional emails to my email address, which means I have to keep checking the site every once in a while just to make sure that I won’t miss something cool.</p><p>And you know the drill.</p><p>I forget things easily. Sometimes, when I’m away from my computer, I miss the reminder as well. Sometimes I’m just too lazy to look things up, because I end up with nothing new, 75–80% of the time. So many excuses…</p><p>Besides, who in their right developer mind wants to do something as boring as that, when you can just set up a bot, sit back, and relax?!</p><p>I started off with <a href="https://aws.amazon.com/lambda/">AWS Lambda</a>, the obvious choice for free serverless computing. Its <a href="https://aws.amazon.com/lambda/pricing/#Lambda_pricing_details">non-expiring free tier</a> gives me an unbelievable <strong>3.2M</strong> (yes, <em>million</em>) seconds of runtime per month — I can virtually keep one lambda running forever, and a little bit more! — across <strong>1M</strong> (<em>million</em> again!) invocations. Previously on Apps Script or App Engine I had just <a href="https://developers.google.com/apps-script/guides/services/quotas#current_quotas">90 minutes</a> per day — a little over 160K seconds per month — meaning that I had to use my quotas very sparingly; but now I can let go of my fears and fully enjoy my freedom of development. Not to mention the fully-fledged <a href="https://www.javacodegeeks.com/2018/02/inside-lambda-runtime-peek-serverless-lair.html">container environment</a> in contrast to the framework confinements of Apps Script or App Engine.</p><p>Enough talk. Let’s code!</p><p>Rather than taking the <a href="https://hackernoon.com/going-serverless-with-amazon-web-services-aws-the-traditional-approach-fb45a7cc5dc2">standard path</a>, I picked <a href="https://medium.com/@asankha/aws-lambda-your-quick-start-guide-to-going-serverless-the-sigma-way-4bc494a9ee84">Sigma from SLAppForge</a> as my development framework; primarily because it had some reputation for supporting external dependencies, and taking care of packaging and deploying stuff on my behalf — including all the external services (APIs, tables, crons and whatnot).</p><p>First I had to <a href="https://medium.com/think-serverless/serverless-getting-started-with-slappforge-sigma-ad2ff8950bd4#e9bd">sign up for Sigma</a>. Although I could have gone ahead with their demo feature (the big yellow button), I already had an AWS account and a GitHub account (not to mention an email address); so why not give it a shot?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4CXsVbUTV1gY1F1Wu9iSNQ.png" /><figcaption>The sign-up form</figcaption></figure><p>After completing the registration, logging in, and <a href="https://slappforge.com/docs/sigma/getting_started.html#granting-aws-access-to-sigma">feeding my AWS credentials to Sigma</a>, I was greeted with a project selection pane, where I opted for a new project with name site-monitor:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqESaTE9-UPLeqNqIHon6Q.png" /><figcaption>Creating ‘site-monitor’</figcaption></figure><p>The app was blazingly fast, and the editor popped up as soon as I hit <strong>Create Project</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQCxy_EHeZo5RH2IhZDXZQ.png" /><figcaption>The Sigma editor</figcaption></figure><p>Without further ado, I grabbed the content of my former Apps Script function and dropped it into Sigma!</p><pre>let AWS = require(&#39;aws-sdk&#39;);</pre><pre>exports.handler = function(event, context, callback) {</pre><pre>    // Here Goes Nothing</pre><pre><strong>    PROPS = PropertiesService.getScriptProperties();<br>    page = UrlFetchApp.fetch(&quot;</strong><a href="http://www.my-favorite-site.com/"><strong>http://www.my-favorite-site.com</strong></a><strong>&quot;)<br>        .getResponseText();<br>    url = page.match(/(lp|\?r=specialTopic)\/[^&quot;]*/)[0];<br>    if (url != PROPS.getProperty(&quot;latest&quot;)) {<br>        GmailApp.sendEmail(&quot;</strong><a href="/service/mailto:janakaud@gmail.com"><strong>janakaud@gmail.com</strong></a><strong>&quot;,<br>            &quot;MyFavSite Update!&quot;, url);<br>        PROPS.setProperty(&quot;latest&quot;, url);<br>    }</strong></pre><pre>    // end of Here Goes Nothing</pre><pre>    callback(null,&#39;Successfully executed&#39;);<br>}</pre><p>(I know, I know, that didn’t work. Bear with me :))</p><p>The next several minutes, I spent transforming my Apps Script code into NodeJS. It was not that hard (both are JS, after all!) once I got the request module added to my project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/168/0*CG4H3inpo6_8nSDU.png" /><figcaption>‘Add Dependency’ button</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*TYPbaP_clzlRe7Ff.png" /><figcaption>Searching for ‘request’ dependency</figcaption></figure><p>But I must say I did miss the familiar, synchronous syntax of the <a href="https://developers.google.com/apps-script/reference/url-fetch/url-fetch-app">UrlFetchApp module</a>.</p><p>Under App Engine I had the wonderfully simple <a href="https://developers.google.com/apps-script/reference/properties/">PropertiesService</a> to serve as the &quot;memory&quot; of my bot. Under Sigma (AWS) things were not that simple; after some look-around I decided to go with <a href="https://slappforge.com/docs/sigma/components/aws/dynamodb.html">DynamoDB</a> (although I still felt it was way much overkill).</p><p>Once I have extracted the URL from the page, I needed to check if I have already notified myself of it; the equivalent of querying my table (formerly the PropertiesService) for an existing entry. In DynamoDB-land this was apparently a <a href="https://slappforge.com/docs/sigma/components/aws/dynamodb.html#get-document-operation">Get Document</a> operation, so I tried dragging in <strong>DynamoDB</strong> into my code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*RKB3gt22SXaPsebr.png" /><figcaption>DynamoDB incoming!</figcaption></figure><p>Once dropped, the DynamoDB entry transformed into a pop-up where I could define my table and provide the code-level parameters as well. Hopefully Sigma would remember the table configuration so I won’t have to enter it again and again, all over my code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/0*giPTwR8gHhIHT2RA.png" /><figcaption>Configuring a new DynamoDB table, and a ‘Get Document’ operation</figcaption></figure><p>Since DynamoDB isn’t a simple key-value thingy, I spent a few minutes scratching my head on how to store my “value” in there; eventually I decided to use a “document” structure of the form</p><pre>{<br>    &quot;domain&quot;: &quot;my-favorite-site.com&quot;,<br>    &quot;url&quot;: &quot;{the stored URL value}&quot;<br>}</pre><p>where I could query the table using a specific domain value for each bot, and hence reuse the table for different bots.</p><p>In my old code I had used a <a href="https://developers.google.com/apps-script/reference/gmail/gmail-app#sendEmail(String,String,String)">GmailApp.sendEmail() call</a> to send myself a notification when I got something new. In Sigma I tried to do the same by dragging and dropping a <a href="https://slappforge.com/docs/sigma/components/aws/ses.html">Simple Email Service (<strong>SES</strong>)</a> entry:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*D2Y4A5RmPe0ljkac.png" /><figcaption>SES: verifying a new email</figcaption></figure><p>Here there was a small hiccup, as it appeared that I would need to verify an email address before I could send something out. I wasn’t sure how bumpy my ride would be, anyway I entered my email address and clicked <strong>Send verification email</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*xrf71CLRE_StWgXq.png" /><figcaption>SES verification email sent!</figcaption></figure><p>Sure enough, I received a verification link via email which, when clicked, redirected me to a <a href="https://aws.amazon.com/ses/verifysuccess/">“Verification successful” page</a>.</p><p>And guess what: when I switched back to Sigma, the popup had updated itself, stating that the email was verified, and guiding me through the next steps!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/0*syg93x3U4A19AjKv.png" /><figcaption>Email verified; SES popup automagically updated!</figcaption></figure><p>I filled in the details right away (<strong>To</strong> myself, no CC’s or BCC’s, <strong>Subject</strong> MyFavSite Update! and <strong>Text Body</strong> @{url} (their own variable syntax; I wish it were ${} though)):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/0*LrvLMI9qk9ii0oLu.png" /><figcaption>SES configuration</figcaption></figure><p>In the callback of SES email sender, I had to update the DynamoDB table to reflect the new entry that was emailed out (so I won’t email it again). Just like the PROPS.setProperty(&quot;latest&quot;, url) call in my original bot.</p><p>That was easy, with the same drag-n-drop thingy: selecting the previously created table under <strong>Existing Tables</strong> and selecting a <a href="https://slappforge.com/docs/sigma/components/aws/dynamodb.html#put-docuement-operation"><strong>Put Document</strong> operation</a> with domain set to my-favorite-site.com (my &quot;search query&quot;; equivalent of &quot;latest&quot; in the old bot) and a url entry set to the emailed URL:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*MFSYA-vpainjo49q.png" /><figcaption>DynamoDB Put Document configuration</figcaption></figure><p>Eventually I ended up with a fairly good piece of code (although it was <strong>way longer</strong> than my dear old Apps Script bot):</p><pre>let AWS = require(&#39;aws-sdk&#39;);<br>const ses = new AWS.SES();<br>const ddb = new AWS.DynamoDB.DocumentClient();<br>const request = require(&quot;request&quot;);</pre><pre>exports.handler = function (event, context, callback) {<br>    <strong>request.get(&quot;</strong><a href="http://www.my-favorite-site.com/"><strong>http://www.my-favorite-site.com</strong></a><strong>&quot;</strong>,<br>        (error, response, body) =&gt; {<br>            if (!body) {<br>                throw new Error(&quot;Failed to fetch homepage!&quot;);<br>            }</pre><pre>            let urls = page.match(/(lp|\?r=specialTopic)\/[^&quot;]*/);<br>            if (!urls) { // nothing found; no point in proceeding<br>                return;<br>            }<br>            let url = urls[0];</pre><pre>            <strong>ddb.get({<br>                TableName: &#39;site-data&#39;,<br>                Key: { &#39;domain&#39;: &#39;my-favorite-site.com&#39; }<br>            }</strong>, function (err, data) {<br>                if (err) {<br>                    throw err;<br>                } else {<br>                    if (!data.Item || data.Item.url != url) {<br>                        <strong>ses.sendEmail({<br>                            Destination: {<br>                                ToAddresses: [&#39;</strong><a href="/service/mailto:janakaud@gmail.com"><strong>janakaud@gmail.com</strong></a><strong>&#39;],<br>                                CcAddresses: [],<br>                                BccAddresses: []<br>                            },<br>                            Message: {<br>                                Body: {<br>                                    Text: {<br>                                        Data: url<br>                                    }<br>                                },<br>                                Subject: {<br>                                    Data: &#39;MyFavSite Update!&#39;<br>                                }<br>                            },<br>                            Source: &#39;</strong><a href="/service/mailto:janakaud@gmail.com"><strong>janakaud@gmail.com</strong></a><strong>&#39;,<br>                        }</strong>, function (err, data) {<br>                            if (err) {<br>                                throw err;<br>                            }<br>                            <strong>ddb.put({<br>                                TableName: &#39;site-data&#39;,<br>                                Item: {<br>                                   &#39;domain&#39;: &#39;my-favorite-site.com&#39;,<br>                                   &#39;url&#39;: url<br>                                }<br>                            }</strong>, function (err, data) {<br>                                if (err) {<br>                                    throw err;<br>                                } else {<br>                                    console.log(&quot;New URL saved successfully!&quot;);<br>                                }<br>                            });<br>                        });<br>                    } else {<br>                        console.log(&quot;URL already sent out; ignoring&quot;);<br>                    }<br>                }<br>            });<br>        });</pre><pre>    callback(null, &#39;Successfully executed&#39;);<br>}</pre><p>Sigma was trying to help me all the way, by providing handy editing assistance (code completion, syntax coloring, variable suggestions…), and even highlighting the DynamoDB and SES operations and displaying tiny icons in front; which, when clicked, displayed (re)configuration pop-ups similar to what I got when I drag-dropped them the first time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*WH-2Zx6lje-Cttu5.png" /><figcaption>DynamoDB operation highlighted, with indicator icon in front</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*80B-Ec368FEj7DxM.png" /><figcaption>DynamoDB operation edit pop-up</figcaption></figure><p>Due to the async, callback-based syntax, I had to move around bits ’n’ pieces of my code several times. Sigma handled it pretty well, re-doing the highlighting stuff a second or two after I pasted the code in its new location.</p><p>Just for fun, I tried editing the code manually (without using the pop-up) and, fair enough, the pop-up understood the change and updated itself the next time I checked. Pretty neat for a newbie who wants to get stuff done without diving into the docs.</p><p>Now, how can I run my bot periodically?</p><p>Sigma shows a red lightning sign near the function header, and highlights the event parameter in the same. Possibly indicating it&#39;s the point of invocation or triggering of the lambda.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*jedFYb41CE-RZoBH.png" /><figcaption>Highlighted ‘event’ variable on function header, with red lightning icon in front</figcaption></figure><p>Yup. Their <a href="https://slappforge.com/docs/sigma/concepts/triggers.html">docs</a> say the same.</p><p><a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html">AWS docs</a> and <a href="https://slappforge.com/docs/sigma/components/aws/cloudwatch.html">Sigma’s own ones</a> pointed me to <strong>CloudWatch scheduled event triggers</strong> that could trigger a lambda with a predefined schedule — like <a href="https://developers.google.com/apps-script/reference/script/trigger">Apps Script triggers</a> but much more powerful; more like <a href="https://cloud.google.com/appengine/docs/flexible/nodejs/scheduling-jobs-with-cron-yaml">App Engine cron expressions</a>.</p><p><a href="https://docs.slappforge.com/concepts/triggers.html#define-trigger">As mentioned in their docs</a>, I dragged a <strong>CloudWatch</strong> entry on to the event variable and configured it like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/0*hdE7EyjcWgTExCih.png" /><figcaption>CloudWatch trigger configuration</figcaption></figure><p>And the whole event thing changed from red to green, possibly indicating that my trigger was set up successfully.</p><p>Right. Time to test it out.</p><p>The toolbar has a <strong>Test</strong> (play) button, with a drop-down to select your test case. <a href="https://developers.google.com/apps-script/overview#try_it_out">Like Apps Script</a>, but much better in the sense that you can defne the input payload for the invocation (whereas Apps Script just runs the function without any input arguments):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/0*xnZBrxqG6k0N5faE.png" /><figcaption>Test button on Sigma toolbar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKz_lykfv1el6aR-pe_g6w.png" /><figcaption>Test case configuration dialog</figcaption></figure><p>As soon as I configured a test case and hit the run button, the status bar started showing a running progress:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*cb-JGIU0EFatbu2N.png" /><figcaption>Test ongoing!</figcaption></figure><p>Few seconds later, a <strong>SigmaTrail</strong> log output window automagically popped up, and started showing some logs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLwyrepJsj2n22KmQGMXxA.png" /><figcaption>SigmaTrail automagically pops up when you run a test!</figcaption></figure><pre><strong>errorMessage:&quot;RequestId: 87c59aba-8822-11e8-b912-0f46b6510aa8 Process exited before completing request&quot;</strong><br>[7/15/2018][5:00:52 PM] Updating dependencies. This might make runtime longer than usual.<br>[7/15/2018][5:00:55 PM] Dependencies updated.<br>[7/15/2018][5:00:57 PM] ReferenceError: page is not defined<br>at Request.request.get [as _callback] (/tmp/site-monitor/lambda.js:13:24)<br>at Request.self.callback (/tmp/site-monitor/node_modules/request/request.js:185:22)</pre><p>Oops, looks like I got a variable name wrong.</p><p>A simple edit, and another test.</p><pre>[7/15/2018][5:04:50 PM] <strong>ResourceNotFoundException: Requested resource not found</strong><br>at Request.extractError (/tmp/site-monitor/node_modules/aws-sdk/lib/protocol/json.js:48:27)<br>at Request.callListeners (/tmp/site-monitor/node_modules/aws-sdk/lib/sequential_executor.js:105:20)</pre><p>Hmm, what does that mean?</p><p>Looks like this one’s coming from the AWS SDK itself.</p><p>Maybe the AWS “resources” I dragged-n-dropped into my app are not yet available on AWS side; besides, many of the <a href="https://medium.com/tag/slappforge">Sigma tutorials</a> mention a “deployment” step before they go into testing.</p><p>Oh well, let’s try deploying this thing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/141/0*ntRjxs-nTIlYQ9T3.png" /><figcaption>Deploy button on Sigma toolbar</figcaption></figure><p>I was hoping a seamless “one-click deploy”, but when I clicked the <strong>Deploy</strong> button I just got a pop-up saying I need to authenticate to GitHub. Sigma might probably be saving my stuff in a GitHub repo and then using it for the rest of the deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*zeUAuFewGRpbyjXT.png" /><figcaption>‘GitHub Authentication’ pop-up</figcaption></figure><p>Seeing no evil, I clicked the sign-in, and authorized their app on the pop-up window that followed. Within a few seconds, I got another pop-up asking me to pick a repo name and a commit message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*qqmq-aqzJtUBZrEW.png" /><figcaption>GitHub commit dialog</figcaption></figure><p>I didn’t have a repo site-monitor in my account, so I was curious to see what Sigma would do. Just as I suspected, after a few seconds from clicking <strong>Commit</strong>, another dialog popped-up asking whether I would like it to create a new repo on my behalf.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*NIHd7Alc-bF3BvwX.png" /><figcaption>GitHub repo creation confirmation dialog</figcaption></figure><p>Sigma was so kind that it even offered to create a private repository; but alas, I didn’t have the luxury, so I just clicked <strong>Create Repo and Commit</strong>.</p><p>From there onwards, things were fairly automated: after the “Successfully commmitted” notification, there was a lightningly fast “build” step (accompanied by a progress bar in the bottom status pane).</p><p>Next I got another pop-up, this time a <strong>Changes Summary</strong>; which, after a few more seconds, populated itself with a kind of “deployment summary”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*xS3cHRZ-_T5HyWit.png" /><figcaption>Deployment summary</figcaption></figure><p>I wasn’t much interested in the low-level detail (though I <em>did</em> recognize the cweOneAM as my cron trigger and siteMonitorLambda as my bot), so I just hit <strong>Execute</strong>; and this time there was a fairly long wait (accompanied by another progress bar, this time within the pop-up itself).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*cINVoZgPPCxTtV_G.png" /><figcaption>Deployment progress</figcaption></figure><p>Once it hit the 100% mark, Sigma stated that my deployment completed with a CREATE_COMPLETE state (sounds good!).</p><p>Now let’s try that testing thing, again.</p><pre><strong>&quot;Successfully executed&quot;</strong><br>[7/15/2018][5:39:34 PM] New URL saved successfully!</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nh-HzmQA18-oxlnsW4lj3Q.png" /><figcaption>SigmaTrail: success!</figcaption></figure><p>Yay!</p><p>Wait, will it resend if I run it again?</p><pre><strong>&quot;Successfully executed&quot;</strong><br>[7/15/2018][5:39:41 PM] URL already sent out; ignoring</pre><p>All good; no duplicates!</p><p>Now to check my inbox, to see if Sigma is telling the truth.</p><p>Initially I was a bit confused because I didn’t actually receive an email; but eventually I found it sitting in my Spam folder (probably because it was sent by a third party (AWS)?), and unmarking it as spam did the trick.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*uDoSXiINCjHI22z6.png" /><figcaption>Email received! (confused? I use <a href="https://mail.google.com/mail/u/0/x/a">Gmail Mobile</a>)</figcaption></figure><p>Hopefully my CloudWatch trigger would fire tomorrow at 1 AM, bringing me the good news if there are any!</p><p>All in all, the graphical IDE is quite slick and recommendable to my colleagues. Except for the deployment time (which I guess is characteristic to serverless apps, or Lambda, or perhaps AWS), I felt almost at home — and even more so, with all the nifty features — autocompletion, drag-n-drop, GUI configs, testing, logs, and so forth.</p><p>Time for a cuppa coffee, and then to start migrating my other bots to Sigma… um… AWS.</p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2018/07/my-bots-are-now-placeless-homeless.html"><em>randomizd.blogspot.com</em></a><em> on July 18, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6879b18e1ed8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google, Here We Come!]]></title>
            <link>https://medium.com/think-serverless/google-here-we-come-4bc3443d143e?source=rss-ca4b0f9c3cd------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bc3443d143e</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[sigma-ide]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[slappforge]]></category>
            <dc:creator><![CDATA[Janaka Bandara]]></dc:creator>
            <pubDate>Mon, 16 Jul 2018 08:03:09 GMT</pubDate>
            <atom:updated>2018-07-18T06:05:26.261Z</atom:updated>
            <cc:license>https://creativecommons.org/publicdomain/mark/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Sigma IDE now supports Google Cloud Platform</h4><p>Google is <a href="https://data-economy.com/aws-vs-google-vs-microsoft-how-cloud-giants-are-battling-it-out-in-the-serverless-arena/">catching up fast</a> on the <a href="https://hackernoon.com/why-serverless-why-now-f09ce43c4767">serverless race</a>.</p><p>And we are, too.</p><p>That’s how <a href="https://medium.com/think-serverless/sigma-the-new-kid-on-the-serverless-block-48b0fa02ad2b">our serverless IDE Sigma</a> recently started supporting integration with <a href="https://cloud.google.com/">Google Cloud Platform</a>, pet-named GCP.</p><h3>What the heck does that mean?</h3><p>To be extra honest, there’s <em>nothing</em> new so far. Sigma could interact with GCP (and any other cloud service, for that matter) right from the beginning; we just didn’t have the scaffolding to make things easy and fun — the nice drag-n-drops and UI pop-ups, and the boilerplate code for handling credentials.</p><p>Now, with the bits ’n’ pieces in place, you can drag, drop and operate on your GCP stuff right inside your Sigma project!</p><h3>(*) Conditions apply</h3><p>Of course, there are some important <em>points</em> (well, let’s face it, <em>limitations</em>):</p><ul><li>GCP access is confined to one project, meaning that all cloud resources accessed in a Sigma project should come from a single GCP project. You can, of course, work around this by writing custom authorization logic on your own, but drag-n-drop stuff will still work only for the first project you configured.</li><li>You can only interact with <a href="https://slappforge.com/docs/sigma/concepts/resources.html#new-resources-vs-existing-resources"><em>existing</em> resources</a>, and just as <a href="https://slappforge.com/docs/sigma/concepts/resources.html#using-resources-as-triggers-and-operations">operations</a> (no <a href="https://slappforge.com/docs/sigma/concepts/triggers.html">trigger</a> support). New resource creation and trigger configurations require deeper integration with GCP, which will become available soon (fingers crossed).</li><li>Only a handful of resource types are currently supported — including <a href="https://cloud.google.com/storage/">Storage</a>, <a href="https://cloud.google.com/pubsub/">Pub/Sub</a>, <a href="https://cloud.google.com/datastore/">Datastore</a>, and <a href="https://cloud.google.com/sql/">SQL</a>. These should, however, cover most of the common scenarios; we are hoping to give more attention to big data and machine learning stuff in the immediate future — which <a href="https://www.marketwatch.com/story/ai-in-2018-google-seeks-to-turn-early-focus-on-ai-into-cash-2017-12-28">Google also seems to be thriving upon</a> — but feel free to <a href="https://launchpass.com/slappforge">shout out</a> so we can reconsider our strategy!</li></ul><h3>So, where can I get this thing?</h3><ul><li><a href="https://slappforge.com/docs/sigma/getting_started.html#loading-a-project">Open up your Sigma project</a> (or <a href="https://slappforge.com/docs/sigma/first_project.html#create-new-project">create a new one</a>, if you haven’t already).</li><li>You’ll notice a new set of buttons at the top of the <a href="https://slappforge.com/docs/sigma/basic_functionalities.html#resources-pane">Resources pane</a>; click on <strong>GCP Resources</strong> (the second one, with the GCP logo).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/0*GHV02vIZy9gdCBGl.png" /><figcaption>‘GCP Resources’ button on the Resources pane</figcaption></figure><ul><li>Now you can provide a <a href="https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount">service account key</a> from the desired <a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">GCP project</a> to access its resources via Sigma. Click <strong>Authorize</strong>, paste the JSON service account key, and press <strong>Save</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/0*oCaKmehvBco7FDjo.png" /><figcaption>GCP Resources pane with ‘Authorize’ button</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*Ni7JMjTa0gseK5N_.png" /><figcaption>GCP Service Account Key pop-up</figcaption></figure><ul><li>The Resources pane will now display the available GCP resource types. You can drag-n-drop them into your code, configure new operations and edit existing ones, just like you would do with <a href="https://slappforge.com/docs/sigma/components/aws/">AWS resources</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/0*JXd0VrbX8bWAxwxh.png" /><figcaption>Resources pane displaying available GCP resource types</figcaption></figure><p>(<strong>Disclaimer</strong>: The list you see may differ, depending on how old this post is.)</p><h3>But you said there’s nothing new!</h3><p>How could I have done this earlier?</p><p>Well, if you look closely, you’ll see what we do behind the scenes:</p><ul><li>We have two environment variables, GCP_SERVICE_TOKEN and GCP_PROJECT_ID, which expose the GCP project identity and access to your Sigma app.</li><li>Sigma has an Authorizer file that reads this content and configures the <a href="https://www.npmjs.com/package/googleapis">googleapis NodeJS client library</a>, using them as credentials.</li><li>Your code imports Authorizer (which runs the abovementioned configuration stuff), and invokes operations via googleapis, effectively operating on the resources in your GCP project.</li></ul><p>See? No voodoo.</p><p>Of course, back in those days, you won’t have seen the cool UI pop-ups or the drag ’n’ drop capability; so we’ve had to do a bit of work, after all.</p><h3>Not cool! I want more!!</h3><p>Relax, the <em>really</em> cool stuff is already on the way:</p><ul><li>Ability to define <strong>new</strong> GCP resources, right within the IDE (drag-drop-configure, of course!)</li><li>Ability to use GCP resources as <strong>triggers</strong> (say, fire a cloud function via a Cloud Storage bucket or a Pub/Sub topic)</li></ul><p>And, saving the best for the last,</p><ul><li>Complete application <strong>deployments</strong> on Google Cloud Platform!</li></ul><p>And…</p><ul><li>A complete <strong>cross-cloud</strong> experience: operating on AWS from within GCP, and vice versa!</li><li>With other players (<a href="https://azure.microsoft.com/en-us/overview/serverless-computing/">MS Azure</a>, <a href="https://topic.alibabacloud.com/Serverless.html">Alibaba Cloud</a> and so forth) joining the game, pretty soon!</li></ul><p>So, <a href="https://medium.com/think-serverless">stay tuned!</a></p><p><em>Originally published at </em><a href="http://randomizd.blogspot.com/2018/07/google-here-we-come.html"><em>randomizd.blogspot.com</em></a><em> on July 16, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bc3443d143e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/think-serverless/google-here-we-come-4bc3443d143e">Google, Here We Come!</a> was originally published in <a href="https://medium.com/think-serverless">Think Serverless</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>