<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Satyajit Paul on Medium]]></title>
        <description><![CDATA[Stories by Satyajit Paul on Medium]]></description>
        <link>https://medium.com/@satyapaul?source=rss-718a52272254------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*YRpikyBWggsdcok0.</url>
            <title>Stories by Satyajit Paul on Medium</title>
            <link>https://medium.com/@satyapaul?source=rss-718a52272254------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:17:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@satyapaul/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Great Reset of the Internet and AI]]></title>
            <link>https://medium.com/swlh/the-great-reset-of-the-internet-and-ai-2cdcc9650f27?source=rss-718a52272254------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cdcc9650f27</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-evolution]]></category>
            <dc:creator><![CDATA[Satyajit Paul]]></dc:creator>
            <pubDate>Sun, 08 Mar 2026 16:09:40 GMT</pubDate>
            <atom:updated>2026-03-08T17:14:30.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Looking back from 2030, it is clear that the world experienced a structural shift in both technology and the global economy between 2026 and 2030. What initially appeared to be an unstoppable AI boom ultimately triggered a redesign of the internet itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cU87SX31ffYi6Na1_SH5ew.png" /></figure><p>The early phase of the AI revolution — now commonly referred to as <strong>AI 1.0</strong> — was dominated by foundation model companies such as OpenAI, the creator of ChatGPT, Anthropic, the creator of Claude and Google with its Gemini. Fueled by explosive demand for AI models to do every piece of work humans could do, these companies became the first in history to cross a <strong>$10 trillion valuation by Q4 2028</strong>. For nearly three years prior, global demand for GPUs, datacenters, and energy infrastructure surged as enterprises rushed to integrate AI into every layer of their business.</p><p>However, the real fuel behind AI 1.0 and the extremely successful models was not compute power alone — it was <strong>unlimited access to the open internet</strong>. The ability to train and improve models on massive amounts of freely available content allowed AI systems to reach astonishing levels of capability in a short period of time.</p><p>But that success carried an unintended consequence.</p><p>By 2026–2027, it had become increasingly clear that the economic benefits of AI were not flowing back to the <strong>original creators of content</strong>. Writers, musicians, filmmakers, journalists, programmers, and researchers found that their work had effectively become raw material for AI systems — often without attribution or compensation. For a period that many now describe as the “Creator Shock” of 2027, entire creative industries experienced disruption and uncertainty.</p><p>By <strong>Q2 2027</strong>, a counter-movement had begun forming.</p><p>A new architecture — later dubbed as <strong>Internet 5.0</strong> — started emerging. Unlike the open and chaotic “digital town square” of Web 2.0, Internet 5.0 introduced <strong>authenticated content ecosystems</strong>, sometimes described as “exclusive digital town squares” or “walled gardens”. In these environments, access to content required <strong>two-way authentication</strong>: verification between the device and the end user (human), and verification between the user’s device and the content platform.</p><p>The system was designed to ensure one crucial principle: <strong>human users could access content, but automated systems could not harvest it without permission</strong>.</p><p>Initially, large technology companies resisted the shift. However, as more publishers, studios, software repositories, and research organizations adopted the new architecture, it became clear that the majority of <strong>high-value original content</strong> was migrating behind authenticated access layers.</p><p>This was led by Aether, a company founded in Q1 2027 to maximize the value creation for the original content creators. This led to meteoric growth of Aether which initially had came up with a new communication protocol for the servers and a browser named Aether that understands the new communication protocol and protects the content from the original content creator. The new protocol doesn’t understand commands like “curl” making all traditional bots toothless. On the browser the end user could not do a view source or run a browser plugin.</p><p>By late 2027, even the largest platforms — including Google, Apple, and Microsoft — joined the ecosystem in order to ensure that their users could still access premium content and services but the transition has been rocky thus far. Aether now serves 70%+ traffic in Internet 5.0. In Q4 2028, Aether rolled out its own handheld device, new OS and an ecosystem of wearable devices like watch, goggles, ear-phone, all powered with advanced AI, AR, VR capabilities.</p><p>This transition fundamentally changed the competitive dynamics of artificial intelligence.</p><p>In the AI 1.0 era, the key advantage was <strong>better model architecture and more compute</strong>. But in the new environment — sometimes called <strong>AI 2.0</strong> — the primary moat shifted to <strong>exclusive access to high-quality content ecosystems</strong>.</p><p>Each authenticated content network now runs its <strong>own AI systems</strong>, trained and updated using its licensed data. These AIs operate as assistants, search engines, and creative tools within their own ecosystem, while <strong>content creators are compensated based on actual usage of their work</strong>.</p><p>As a result, the dominance of early AI leaders began to soften. Since <strong>Q4 2028</strong>, the valuations of OpenAI and Anthropic have corrected by more than 3<strong>5%</strong>, and analysts remain cautious about their long-term growth prospects in a fragmented internet landscape. The trillion dollar question is whether OpenAI and Anthropic can break-into the world of exclusive content. In absence of the new original content, the models are dying slow death as they fail to answer queries from users who are coming from the “walled gardens” of Aether.</p><p>Yet the broader economic story has been more nuanced.</p><p>The wave of productivity gains predicted during the early AI boom did arrive — but not exactly where many expected. One of the biggest transformations occurred in <strong>robotics and physical automation</strong>. Beginning in <strong>Q3 2027</strong>, advances in AI-driven robotics made automated manufacturing dramatically more cost-effective.</p><p>By <strong>Q1 2028</strong>, robotic systems had become <strong>ubiquitous across manufacturing sectors</strong>, from electronics assembly to logistics and construction materials. The result was a sharp reduction in production costs across many categories of goods.</p><p>While job losses did occur — particularly within technology companies and digital services — the feared collapse in living standards never materialized. Many firms became dramatically more productive with smaller workforces, allowing them to <strong>lower prices while maintaining profitability</strong>.</p><p>Consumers ultimately benefited from <strong>cheaper goods and services</strong>, cushioning the economic slowdown that accompanied the restructuring of the technology sector.</p><p>In retrospect, the warnings from early analysts — most famously articulated in the Citrin article of 2026 — turned out to be partially correct. AI did eliminate many jobs, particularly in knowledge work. But it also triggered a profound reorganization of the internet economy where original content creators are now compensated well.</p><p>The SaaS companies didn’t disappear or collapse. In fact, their growth continued for few more years since the publication of the article. The much anticipated risk didn’t come from deluge of competitive offerings triggered by unlimited software engineering bandwidth of AI, rather that came from shrunk problem space or reduced opportunity size due to take-over by agentic workforce of corporate America.</p><p>By 2030, the world now operates on a hybrid digital structure:<br>an ecosystem of <strong>authenticated content networks, specialized AI systems, and highly automated physical industries</strong>.</p><p>The open internet did not disappear — but it is no longer the center of gravity.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Note: Above article was created using the following original content in ChatGPT, followed by edits and updates by human.</p><p>Original Content:</p><ol><li>Its 2029, technology landscape and world economy has undergone a fundamental shift in last 5 years. 1. OpenAI, Claude were the first two companies to reach $10 T by Q1 2028, driven by massive demand in AI compute in previous 3 years.</li><li>Starting 2028, those models were not able to the new internet structure that had started emerging by Q2 of 2027.</li><li>Second, the moat for AI models have shifted from fine tuning the models to access to exclusive contents which essentially shaped internet 3.0</li><li>Iin case you were living in a cave, in internet 5.0, internet is no more a common shared townsquare, it’s all divided and access is privileged</li><li>⁠AI 1.0 thrived on unlimited access to content, that was the real fuel and not the compute capacities of the GPUs.</li><li>But essentially shutout all original content creators — be it movie creators, or music or authors or programmers.</li><li>While all creators became redundant temporarily and their lives were disrupted massively by Q2 2027, a new structure of new exclusive townsquare started emerging starting Q1 2027 where users only human users get access to the content . It’s based on two way authentication. Device to source and device to end user authentication followed by authorization. After initial resistance, Google, Apple and Microsoft all signed up for this in order to ensure access to the original content for their users.</li><li>Aether came up with new protocol that doesn’t support traditional commands like curl making all the bots useless. Aether introduced it’s own browser that can communicate in the new protocol in addition to supporting legacy Http protocol. This gave Aether exclusive rights to the new content from the original content creators.</li><li>Q4 2028, Aether rolled out it’s own handheld device, new OS and an ecosystem of wearable devices like watch, goggles, ear-phone, all powered with advanced AR, VR capabilities.</li><li>⁠Like Internet 5.0, AI 2.0 is also different, every such exclusive town-square now has their own AI which exclusively access the contents and content providers get paid based on the usage of their content.</li><li>Since Q4 2027, Anthropic and OpenAI valuations have corrected by 35%+ and analysts’ forecasts are not very promising.</li><li>⁠what has come true since the famous citrin article is there were job losses in those now old-tech companies, but those companies survived and thrivesd, retained and grown their market share but they don’t employ as many workers now. Productivity gains led to lowering of price and large part of that benefit reached to the hands of consumers.</li><li>SaaS companies didn’t disappear or collapse. In fact, their growth continued for few more years. The much anticipated risk didn’t come from deluge of competitive offerings, rather that came from shrunk problem space or reduced opportunity size due to take-over agentic workforce.</li><li>As a result, economy saw downward spiral but didn’t impact the living standards as much.</li><li>⁠large part or cost reduction in goods and services came from rise of application of robotics starting Q1 2027. By Q4 2027, application of robots for manufacturing was ubiquitous leading lower price at the hands of consumers.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cdcc9650f27" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/the-great-reset-of-the-internet-and-ai-2cdcc9650f27">The Great Reset of the Internet and AI</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is the theory of evolution real?]]></title>
            <link>https://medium.com/@satyapaul/is-the-theory-of-evolution-real-b81cb3f592ca?source=rss-718a52272254------2</link>
            <guid isPermaLink="false">https://medium.com/p/b81cb3f592ca</guid>
            <category><![CDATA[cosmos]]></category>
            <category><![CDATA[neil-degrasse-tyson]]></category>
            <category><![CDATA[evolution]]></category>
            <category><![CDATA[carl-sagan]]></category>
            <category><![CDATA[charles-darwin]]></category>
            <dc:creator><![CDATA[Satyajit Paul]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 01:00:32 GMT</pubDate>
            <atom:updated>2023-04-22T23:17:28.554Z</atom:updated>
            <content:encoded><![CDATA[<p>Ever wondered how we humans came to existence? I guess we all do. This has not just intrigued you and me, it has caught the imagination of humanity for ages. The wiser ones among our ancestors came up with various ideas, theories and suggestions to answer this nagging question. If some theories proposed human beings came from fish eggs (yes, Anaximander of Miletus, a Greek philosopher proposed this around 500 BCE), some believed humans are extra-terrestrial beings that came from a different planet. Theologies and religious books proposed God had hand crafted the first man and women - Adam and Eve and we all flow from them. While all these are good leisurely read for a holiday afternoon but they fail to douse a genuine curiosity. It was not until 1850, humanity had a credible explanation to this quest, when English doctor, biologist, explorer Charles Darwin published his famous book “<em>On the Origin of Species</em>”. In this book, Darwin proposed the theory “Evolution through natural selection”, a scientific explanation to the question that humanity chased for thousands of years. We all have read <em>Darwin’s Theory</em> in school books and likely some of us pursued it further during higher studies.</p><p>While we knew about Charles Darwin’s theory, not many dived deep to build a clear understanding about the topic. I fall in that bracket. For me, this was not an issue until I heard some of the fierce criticism of Darwin’s theory, mostly in debates in socio-religious circles. Some of the counter arguments include — if evolution was true, why we don’t see it happening now; if apes were our ancestors, why we don’t have a single evidence of monkeys turning into human beings in 10,000 years of documented or undocumented history; humans are so perfect that only evolution can not create them without intervention of God etc. I knew most of these criticisms have no scientific basis and are driven by dogmatic beliefs, but I didn’t know a way to counter those arguments in my own mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QOSrbOcitwcyi6eknGGkrQ.jpeg" /><figcaption>Photo Credit to <a href="https://unsplash.com/@jplenio">Johannes Plenio</a></figcaption></figure><p>Recently, my 10 year old daughter and I started watching <em>Cosmos, a Spacetime Odyssey</em> hosted by Astro-Physicist Neil deGrasse Tyson, a must watch in my opinion. That’s when I had my eureka moment. I found <em>Season 1, Episode 6: Some of the Things That Molecules Do </em>an absolute eye opener for me. Neil deGrasse explains it well how micro-organisms evolved through mutation. Three such example struck my mind and I will try to sum them up in this article. Before I go there, let me put few important milestones and their timelines in perspective — Beginning of universe: 13.8 billion years (BY), Sun came to existence: 4.8 BY, Earth Came to existence: 4.5 BY, Emergence of micro-organisms on earth: 3.8 BY, Last common grandmother of humans and chimpanzees: 6 million years, Homo sapiens evolve in East Africa: 200,000 years, Modern Agriculture based society: 13,000 years. A sense of timeline tells us how slow evolution is and how small a speck our existence has been. With that let me move to the last leg of this story where I will try to share the stories from that episode of Cosmos.</p><p>Episode 6 begins with the story of domestication of wild wolves to modern day pet dogs. It’s fascinating how few wild wolves, with lower level of stress hormone were less fearful of human and chose the food given by humans over the stressful work of hunting. Overtime, they evolved reinforcing “natural selection of tameness”. As human started extending preferential treatment to the friendlier and the cuter ones, we ended up having cute little Pugs, Pomeranians and Golden Retrievers of the day. Today wild wolves and dogs are two different species. That’s a story of evolution that took place over last 15000 years, albeit an evolution artificially induced by humans. Similarly, our ancestors are responsible for domestication of wild grasses into corns and wheats of today, staple food for large section of humanity.</p><p>Next story is how bears evolved into polar bears today during the two million years of Ice Age. A mutation in an egg cells resulted into brith of a bear with lighter fur color. This one and its offspring had a critical advantage over its competition — siblings and cousins. Bears with lighter fur were difficult to spot by the prey and hunters alike in a world covered with snow while the ones with darker far color became easy to spot. This led to explosion of bears with lighter far while the ones with darker shades disappeared from the snow-clad areas. This evolution was caused by a freak mutation, followed by natural selection of the fittest ones over millions of years.</p><p>Last story is about how eyes came to existence. As Neil deGrasse put it — “can mindless evolution create a sophisticated organ like eye?”. He then takes us to 4 billion years back when life was getting started in water, and early lives were all about microbes. Microbes ruled the water world of the early earth but they were all blind. During one of those days, some proteins in some microbe mutated and developed light sensitivity and passed on. It helped it to stay away from the intense lights on the surface, as a result they survived in greater number than the blind microbes. The mutation didn’t stop there, continued to evolve and ones with critical advantages from evolution continued to survive in greater number. Cycle of evolution for eye continued over millions of years to have what we experience as eye.</p><p>For me, this episode helped break down the mystery of evolution and finally, now I have some idea on how modern human came to existence!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b81cb3f592ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Authentication & Authorization for Web Apps Using AWS Cognito]]></title>
            <link>https://medium.com/swlh/authentication-authorization-for-web-apps-using-aws-cognito-21548bb3b86?source=rss-718a52272254------2</link>
            <guid isPermaLink="false">https://medium.com/p/21548bb3b86</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[cognito]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Satyajit Paul]]></dc:creator>
            <pubDate>Tue, 31 Dec 2019 05:29:21 GMT</pubDate>
            <atom:updated>2020-01-01T03:04:40.715Z</atom:updated>
            <content:encoded><![CDATA[<p>Every application worth its salt needs Authentication &amp; Authorization and fortunately, that problem is done &amp; dusted by now. It doesn’t make sense for an application developer to reinvent the wheel anymore and I am sure no one is doing it today. When I started exploring AWS Cognito, realized there are multiple pieces to it. It took some time before I could connect various pieces and develop an understanding. So, I thought of writing them down at high level. I have also added a list of articles that I went through as part of my research.</p><p>In this blog we will see how we can use AWS Cognito for Authentication &amp; Authorization for a Web App in completely serverless way. We will walk through the steps for enabling Signup, Signin and Authorization to access to an AWS Resource. At high level these are the things that you need to do —</p><ul><li>Configure a <strong>Cognito User Pool</strong> for <em>User Authentication</em> process.</li><li>Configure a <strong>Cognito Identity Pool</strong> (Federated Identities) for <em>Authorizing </em>the users to AWS resources.</li><li>Configure <strong>IAM Policy</strong> for what AWS Resources that users can access.</li><li>Configure the specific AWS resource, in our case it will be AWS S3.</li><li>Start using the AWS Cognito Authentication/Authorization in your Web App.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3yktz85tuliNEutT" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Create Cognito Userpool</strong></p><ul><li>Create the User Pool in the same region as the WebApp and S3 Bucket. Note down following parameters</li></ul><blockquote><strong>Pool Id</strong> ap-south-1_XXXXX40</blockquote><blockquote><strong>App client id </strong>v9xxxxxxbb6mvyyyyyyvm7hb</blockquote><ul><li>Under <em>General Settings &gt; App Clients</em>, make sure you uncheck “Generate client secret” if you want to use it from browser.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8A0pt1FpGDNpcly2hIRnOQ.png" /></figure><ul><li>Under <em>App Integration &gt; App Client Settings</em>, configure Callback and Sign out URLs. The URLs must be ‘https’, with exception of urls with localhost where ‘http’ is allowed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZI9gnb1qeZmYmkNpPvC1zA.png" /></figure><ul><li>Under <em>App Integration&gt;Domain Name</em>, configure Amazon Cognito domain. This is used in Hosted URL for Signin. You may use your own domain name as well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2LxoqAJtuH4ROD2L-tNbw.png" /></figure><ul><li>Next, under <em>App Integration &gt; App Client Settings</em>, do OAuth2.0 settings based on your need.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*etmBZXxmOU52FR-aaalv5w.png" /></figure><ul><li>“Authorization code grant” enables return of Auth code as response and “Implicit grant” enables return of Tokens like id_token or access_token in the response url.</li><li>Finally, you get a AWS Hosted URL that can be directly used in your Web App. Here are two different types of URL that AWS will provide.</li></ul><blockquote><a href="https://salarymattersin.auth.ap-south-1.amazoncognito.com/login?client_id=v9h8g5e0bb6mvdmt0ps7vm7hb&amp;response_type=token&amp;scope=aws.cognito.signin.user.admin+email+openid+phone&amp;redirect_uri=http://localhost:8080/aws-s3-from-browser.html">https://salarymattersin.auth.ap-south-1.amazoncognito.com/login?client_id=v9h8g5e0bb6mvdmt0ps7vm7hb&amp;<strong>response_type=token</strong>&amp;scope=aws.cognito.signin.user.admin+email+openid+phone&amp;redirect_uri=http://localhost:8080/aws-s3-from-browser.html</a></blockquote><blockquote><a href="https://salarymattersin.auth.ap-south-1.amazoncognito.com/login?client_id=v9h8g5e0bb6mvdmt0ps7vm7hb&amp;response_type=code&amp;scope=aws.cognito.signin.user.admin+email+openid+phone&amp;redirect_uri=http://localhost:8080/aws-s3-from-browser.html">https://salarymattersin.auth.ap-south-1.amazoncognito.com/login?client_id=v9h8g5e0bb6mvdmt0ps7vm7hb&amp;<strong>response_type=code</strong>&amp;scope=aws.cognito.signin.user.admin+email+openid+phone&amp;redirect_uri=http://localhost:8080/aws-s3-from-browser.html</a></blockquote><ul><li>You can do some amount of customization of the default UI in <em>App Integration &gt; UI Customization</em>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKvj7uXQxWk7-ujEaAnFuQ.png" /></figure><p><strong>Create Cognito Identity Pool</strong></p><ul><li>In Identity Pool you need to configure IAM Roles for Unautheticated as well as Authenticated users. We will allow only Authenticated Users and hence, unchecked the checkbox used for enabling the access for Unauthenticated users. The IAM Role created for the Identity Pool needs to have access to the respective AWS Resources.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3Kjy72XhgPylcm0j8NAEA.png" /></figure><ul><li>You need to configure the Authentication Provider here. In my case it was, AWS Cognito i.e. Cognito Userpool created earlier.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTI5aMKapywaD8hipry7_Q.png" /></figure><p><strong>Configure IAM Policy</strong></p><ul><li>Provide AWS Resource Access for the IAM Role used in Identity Pool for the Authenticated users — Cognito_salarymattersAuth_Final_Role in the example above.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q4tzWiqbdoSZFJrokHratA.png" /></figure><ul><li>Like AWS S3, you can configure access to other AWS resources like AWS DynamoDB or Polly here.</li></ul><p><strong>Configure the AWS resource</strong></p><ul><li>Configure CORS Configuration for S3 Bucket. Please ensure the s3 Bucket is in the same region as Cognito User Pool and Identity Pool.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h2fLWbdDhCabk3A0LcE-Nw.png" /></figure><p><strong>Setup your Web App to use Cognito based Auth</strong></p><ul><li>Create a Login Hyperlink/Button to call the Hosted Url with <em>response_type=token</em> shown above.</li><li>User is prompted to login. Post successful login, it redirects to the configured url with parameters like <em>id_token, access_token</em> etc after the #.</li><li>Read <em>id_token</em> from the url &amp; use that to set up the AWS Config.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p4iIs5qpY_jJaBdKBLVh1A.png" /></figure><p>Below are some sample code for illustration pupose to give you an idea on what you may need to code in Javascript. For complete samples, refer to <a href="https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html">this</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TurYDHTc-9dVoTHsIYFWpQ.png" /></figure><p>In this example, we used AWS Cognito Hosted UI for Signup/Signin. But you may like to write your own UI and in that case AWS Cognito provides appropriate APIs to authenticate a user using username/password and setup AWS Cognito credentials.</p><p><strong>Summary</strong></p><p>If you have completed above activities, then you have created a <em>Cognito User Pool</em> which will take care of user Signup &amp; Signin process. A fully functional User Pool will get you — a <em>Hosted URL</em>, <em>User Pool Id</em>, <em>App Client ID. Hosted URL</em> can be directly used in your Web App and you don’t have to do any additional UI code for handling things like — User Registration, Login, Password reset, Multi-Factor Authentication etc. You then use the <em>User Pool Id</em>, <em>App Client ID </em>to configure the Identity Pool (Federated Identities). Once Identity Pool is configured appropriately, you will have <em>Identity Pool Id</em> and a newly created <em>IAM Policy</em>. Configure IAM Policy to get necessary access to AWS Resource, in our case it was Full Access AWS S3. For any actual application, I am sure you will limit the access to a specific an S3 bucket and most likely to limited operations. Finally, to enable access from Web Browser, you configure the CORS Policy for S3. You are all set. In your web app, you will use <em>User Pool Id</em>, <em>Identity Pool Id and id_token from response</em> to setup Cognito Credentials. Finally, you initialize AWS S3 &amp; use that for programmatic access to AWS Resource directly from browser.</p><p><strong>References:</strong></p><blockquote><a href="https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html">https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html</a></blockquote><blockquote>A complete AWS Documentation for using Cognito Authentication/Authorization to upload photos to an S3 Bucket in completely Serverless way.</blockquote><blockquote><a href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-integrating-user-pools-with-identity-pools.html">https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-integrating-user-pools-with-identity-pools.html</a></blockquote><blockquote>Article shows how to access AWS Services Using an Identity Pool After Sign-in, helps with integrating a User Pool with an Identity Pool.</blockquote><blockquote><a href="https://medium.com/@haydnjmorris/uploading-photos-to-aws-s3-getting-started-with-cognito-and-iam-c96ba5b5496d">https://medium.com/@haydnjmorris/uploading-photos-to-aws-s3-getting-started-with-cognito-and-iam-c96ba5b5496d</a></blockquote><blockquote>This article details out how one can upload photos to AWS S3 from browser directly. It shows how to Authenticate using Cognito Identity Pools and then upload to S3 bucket from browser. This approach doesn’t need any AWS Lambda or any other backend coding.</blockquote><blockquote><a href="https://www.mydatahack.com/uploading-file-from-browser-to-s3-with-javascript/">https://www.mydatahack.com/uploading-file-from-browser-to-s3-with-javascript/</a></blockquote><blockquote>Here is one more article that is on the same line as above and it shows the use of both Cognito User Pool &amp; Identity Pools and gives a more complete example including code.</blockquote><blockquote><a href="https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html">https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html</a></blockquote><blockquote>This Article covers two different implementations —</blockquote><blockquote>Use lambda function + API Gateway to generate a temporary access_key and access_key_id and use that from the browser to upload a file to S3 Bucket.</blockquote><blockquote>Use lambda function + API Gateway to generate a pre-signed url and then use the pre-signed url in html form submission to upload a file to S3 Bucket.</blockquote><blockquote><a href="https://medium.com/@ashan.fernando/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db">https://medium.com/@ashan.fernando/upload-files-to-aws-s3-using-signed-urls-fa0a0cf489db</a></blockquote><blockquote>This article discusses two approaches to implement uploading files directly to AWS S3 private bucket from browser, using AWS IAM Temporary Access Credentials and using a pre-signed POST policy.</blockquote><blockquote><a href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html">https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-verifying-a-jwt.html</a></blockquote><blockquote>This article talks about JWT Token Validation — AWS provided client side library takes care of it, it automatically refresh your ID and access tokens if there is a valid (non-expired) refresh token present, and the ID and access tokens have a minimum remaining validity of 5 minutes.</blockquote><blockquote><a href="https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-browser.html">https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/getting-started-browser.html</a></blockquote><blockquote><em>This article will help in case you want to use </em><strong><em>Amazon Polly</em></strong><em> for the Authenticated users.</em></blockquote><blockquote><a href="https://aws.amazon.com/blogs/mobile/building-fine-grained-authorization-using-amazon-cognito-user-pools-groups/">https://aws.amazon.com/blogs/mobile/building-fine-grained-authorization-using-amazon-cognito-user-pools-groups/</a></blockquote><blockquote><em>This article gives step by step guide for calling </em><strong><em>AWS DynamoDB</em></strong><em> services post user Authentication using Cognito.</em></blockquote><blockquote><a href="https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-with-identity-providers.html">https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-user-pools-using-tokens-with-identity-providers.html</a></blockquote><blockquote>Article shows how use ID Token, Access Token &amp; Refresh Token along with Cognito User Pool.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21548bb3b86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/authentication-authorization-for-web-apps-using-aws-cognito-21548bb3b86">Authentication &amp; Authorization for Web Apps Using AWS Cognito</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are the options for using AWS S3 from my Java App]]></title>
            <link>https://medium.com/@satyapaul/what-are-the-options-for-using-aws-s3-from-my-java-app-b5a111ad1748?source=rss-718a52272254------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5a111ad1748</guid>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[s3-bucket]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Satyajit Paul]]></dc:creator>
            <pubDate>Sat, 16 Mar 2019 16:56:14 GMT</pubDate>
            <atom:updated>2019-04-02T01:21:07.853Z</atom:updated>
            <content:encoded><![CDATA[<p>AWS S3 gives multiple options for your Java Application to read, write files into S3 object store. Question is — which is the best option for a Java Application? Answer is — it depends. We will look into the <a href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-objects.html#upload-object">simple example implementation</a> first and then go over all the options one by one. By the end of this article you should have a fairly good idea about which option really suits your need.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DlsvW_CyiAByCcrA" /></figure><p>Simplest option is to use S3Client and upload a single file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t0mIsZwiuQLVUBFA5JcarQ.png" /><figcaption>Exhibit 1</figcaption></figure><p>Above example is good for simple use-cases but things become complicated as you start considering factors like — time it takes to read or write a file (<em>latency</em>), number of files to be transferred in a given time (<em>throughput</em>), sizes of the files to be transferred(<em>transfer rate</em>), multiple java apps need to write to same file, immediate availability in S3, whether multiple java apps must have the same view of the files written (<em>consistency vs eventual consistency</em>), files are one time write vis-a-vis multiple times updates, are all java apps in same network or across geographies (<em>distributed application</em>), whether your app is running in AWS same region/availability zone as S3 bucket or in a different environment (<em>same cloud availability zone vs on-premise</em>), if you want to avoid vendor lock-in and make AWS S3 bucket to be replaceable by other similar service from Google, IBM or Microsoft and last, if it’s an existing application trying to leverage S3 or it’s a net new application being developed.</p><p>As you keep adding each of the factors to your list of requirements, nature of solution also changes. For this article, I will look into only the file write or upload scenarios. So, lets go over the available options for file upload to AWS S3.</p><p><strong><em>Write a single file or a string content to S3 using </em></strong><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjSingleOpJava.html"><strong><em>AWS SDK for Java</em></strong></a></p><p>Good for single file operation, will not scale if you need to upload a large file (&gt; 5 GB) or if you want to upload large number of files. It doesn’t support multipart upload. I tried uploading a 224 MB file from my home MacBook and it took around 60–90 sec to complete the upload. Results were not much different when I tried from servers running outside the specific AWS region. However, when I tried from an EC2 instance in the same region as the S3 bucket, it was way better — 3.7 to 3.8 sec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dpr4_27bUgU22pMKirGbaA.png" /><figcaption>Exhibit 2</figcaption></figure><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html"><strong><em>Multipart file upload</em></strong></a><strong><em> to S3</em></strong></p><p>Good for large file upload, gives flexibility of low level implementation where you can split a large file into multiple smaller parts. Please note AWS S3 doesn’t support uploading file larger than 5 GB in single operation, so you need it if you have to deal with large files. However, it may not be an optimal choice for large number of small files or mixed size files. If you don’t not know the sizes of the files upfront, you will have to take care of the optimization in your code. I tried uploading the same 224 MB file using Multipart upload API from my MackBook, it took around 95–98 sec to complete the upload. The file was broken into multiple parts of 5 MB each for upload, I am guessing my home network was the bottleneck here. Results were better when I tried uploading from servers but not good enough, it still took more than 40 sec. When I tried from an EC2 instance in the same region as the S3 bucket, results were way better — 9.1 to 16.3 sec while S3Client based Simple upload from same EC2 instance was still much better, it took ~4 sec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_MlDhmTNkz59XFQx-b8-cw.png" /><figcaption>Exhibit 3</figcaption></figure><p><strong><em>Using </em></strong><a href="https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-s3-transfermanager.html"><strong><em>AWS Transfer Manager</em></strong></a><strong><em> for file upload to S3</em></strong></p><p>This is the best option if you are using AWS SDK for uploading to S3, can handle single file upload, or a list of files upload or a directory upload. Internally it takes care of different scenarios, gives you options to track progress or to pause an upload. Interestingly, for the same 224 MB file, from the same home network, it took around 38–40 sec to upload. This was performing much better from the regular servers — roughly around 12 to 20 sec and on an EC2 instance in same region, it was around 7 sec.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MItXDn8AeuZPSTdrlIbE9Q.png" /><figcaption>Exhibit 4</figcaption></figure><p><strong><em>Use a FUSE based files system like </em></strong><a href="https://github.com/s3fs-fuse/s3fs-fuse"><strong><em>s3fs-fuse</em></strong></a></p><p>s3fs-FUSE can map an S3 bucket to a local drive. Once that’s done, you can do regular file operations like file copy, listing, delete etc. and it will be reflected on your s3 bucket. There is an option to cache the files on a local temp directory. This is a pretty popular open-source option for mounting an S3 bucket as a local file system, but I was not that lucky — almost 50% of time it didn’t work and when it worked it took around 65–90 sec to complete the upload for the same 224 MB file. Adding a cache location didn’t help much for upload scenario.</p><p>s3fs-FUSE supports AWS S3, Azure Object store, Google object store and IBM Object store.</p><p><strong><em>Use </em></strong><a href="https://github.com/gaul/s3proxy"><strong><em>s3proxy</em></strong></a><strong><em> for file upload</em></strong></p><p>s3proxy uses s3fs-FUSE internally and makes it accessible over http end point. So, users don’t have to mount a drive to the servers where Application is hosted, rather Applications can use the AWS S3 API and use the s3proxy end point to get the s3 objects as regular files. Since it’s built on top of s3fs-FUSE, you can change the underlying object store at anytime you want and Application code can continue to use the S3 API to access the files. I have not run any test for this. I would think it will have an additional overhead on top of s3fs-FUSE performance level due to http layer.</p><p><strong><em>Use AWS Storage Gateway &gt; File Gateway to upload files to S3</em></strong></p><p>AWS <em>File Gateway</em> gives you couple of different options for mounting your S3 bucket, all the options require you to have a dedicated VM that will work as the <em>File Gateway</em>, you can use — an EC2 instance if you are running B2Bi on AWS Cloud, or use VMWare ESXi or Microsoft Hyper-V options if you are running B2Bi on premise. I could try only the option of converting an <a href="https://medium.com/@satyapaul/how-to-use-aws-s3-as-document-storage-for-ibm-b2b-integrator-c008eb838113">EC2 instance into File Gateway</a>. It took around 4 sec for uploading 224 MB file to an s3 bucket in the same region as the EC2 instance.</p><p><strong><em>Use Hardware based File Gateway for S3</em></strong></p><p>There is one off-the-self solution: AWS Storage Gateway &gt; File Gateway Appliance. This is a AWS Storage Gateway pre-loaded on a <a href="https://www.amazon.com/dp/B079RBVX3M">Dell EMC PowerEdge server</a>. This supports NFS mounting like all other File Gateway solutions. I don’t have the latency data for my usual 224 MB file transfer. But given this is a hardware based commercial solution, I would expect this to be the top performing solution. <em>If you have any data based on your experience with this appliance, then I would love to update this post with your findings, needless to say with appropriate credit :)</em>.</p><p>Here is the summary of all performance/latency data I gathered through testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*49uRu51E6IqtQTs7XaVFrg.png" /><figcaption>Exhibit 5</figcaption></figure><p>When you think of all the options above and the time taken to upload a 224 MB file, please keep in mind that it takes less than 1 sec on an EC2 instance to copy a 224 MB files to a local drive using Java IO and it takes around 0.3–0.7 sec using Java NIO.</p><p>So, if you are planning to move to AWS s3 for file storage from regular file system or NFS for your Java App that may run on your on-prem data center as well as in cloud, the biggest factor to consider is <strong>latency</strong>. A jump from 0.5 sec or 2.5 sec to 30 sec or 80 sec is a big jump.</p><p>If you know your application is always going to run on AWS EC2 environments (that includes Lambda, Elastic Beanstalk etc), and you don’t plan to write into a s3 bucket in a different region, then go with appropriate AWS S3 API. If your need is writing a single, small to medium size (&lt; 200 MB) file, you are fine with simple <em>S3Client</em> <em>put</em> operation. However, if you have to deal with multiple files, files of varying sizes (including very large files &gt; 5 GB), or directory upload, go ahead with <em>Transfer Manager</em> option.</p><p>If your application needs to run on AWS cloud as well as on premise and wants to leverage on S3, then go ahead with Storage Gateway &gt; File Gateway options — you have plenty of them including an Appliance. In terms of performance, using Storage Gateway in EC2 environment is comparable to writing to S3 using native AWS SDK APIs (~3.9 sec). Also, this will allow you to continue to use Java File IO/NIO APIs in your application. For an existing applications, you save precious engineering bandwidth by avoiding to re-write your application to use S3 APIs. This will also allow you to use offerings like Dell-EMC — <a href="https://www.dellemc.com/en-us/solutions/storage/ecs/object-storage-with-ecs.htm">ECS Object Storage</a> as NFS mounted drive. Dell-EMC ECS Object Storage (and others in this category) can work with Object Stores from any vendor — Amazon, Google, Azure, IBM. So, with Dell-EMC ECS Object Storage solution and it’s likes, you can avoid vendor lock-in. Last, with all these solutions, your application can update the same file multiple times using the files in cache instead of reading a File (object) from Object store for every small change.</p><p><strong><em>Conclusion</em></strong>:</p><p><strong>I</strong>f your Application runs in EC2 instances in the same region as S3 bucket, then both natively writing to S3 or using the Storage Gateway &gt; File Gateway perform almost at same level ~4 sec for a 224 MB file.</p><p><strong>I</strong>f your application runs in your own data center or in a different AWS region than S3 bucket, then writing to S3, using AWS SDK APIs, is not an option due high network overhead. You must go with either File Gateway or one of the appliances based S3 storage.</p><p><strong>I</strong>f your application requires re-writing multiple times into an existing File, then DON’T use S3. A jump from 0.3 sec to 3.9 sec actually is 13 x jump in time to write. Simply put S3 is not meant for low latency file writes. AWS S3 is good for <em>write-once-read-multiple_times </em>(WORM)<em> </em>usage. No one uses S3 for files used by OS or DB ;).</p><p>Source code used for this article is available under <a href="https://github.com/satyapaul/AWSS3Clients">https://github.com/satyapaul/AWSS3Clients</a> . All of them are sample code shared on AWS website, I made some minor changes.</p><p>Appreciate your comments, feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5a111ad1748" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use AWS S3 as Document storage for IBM B2B Integrator]]></title>
            <link>https://medium.com/@satyapaul/how-to-use-aws-s3-as-document-storage-for-ibm-b2b-integrator-c008eb838113?source=rss-718a52272254------2</link>
            <guid isPermaLink="false">https://medium.com/p/c008eb838113</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[b2b]]></category>
            <category><![CDATA[b2b-integration]]></category>
            <category><![CDATA[ibm-b2b-integrator]]></category>
            <category><![CDATA[sterling-filegateway]]></category>
            <dc:creator><![CDATA[Satyajit Paul]]></dc:creator>
            <pubDate>Sat, 02 Mar 2019 10:53:16 GMT</pubDate>
            <atom:updated>2019-03-20T17:07:12.390Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are running IBM B2B Integrator and mired with the challenges of handling high volume data, one of the options that might have crossed your mind is use of AWS S3 Object store. Will not it be cool if you can use AWS S3 as Document storage for B2Bi/SFG?</p><p>This article looks into the options you have for using AWS S3 and highlights some of the pros and cons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KQYGI2B7LpOv8D9v" /></figure><p>First, lets take a look at the out of the box B2Bi service — <a href="https://www.ibm.com/support/knowledgecenter/en/SS3JSW_6.0.0/integrating/integrating/integrator/Amazon_S3.html">AWS S3 Service</a>, supported starting with v6.0 release. This service is run as part of the <em>Business Process</em> execution and good for lightweight usage. By design, it’s not expected to help with Document storage &amp; Mailbox Payload storage to AWS S3.</p><p>If your need is to use AWS S3 for Document Storage for Adapters and Services (i.e. BPs and Mailboxes), then IBM B2Bi doesn’t support that natively yet.</p><p>Good news is Amazon AWS offers a service that allows you to map an S3 Bucket as a mounted drive on local OS of your VM and then use the mounted drive as the Document Directory ( <em>jdbcService.document_dir</em>) for B2Bi Instance. This configuration can be used irrespective of whether you run B2Bi/SFG On Premise or on AWS Cloud.</p><p>For achieving this, we will use <a href="https://console.aws.amazon.com/storagegateway/home">AWS Storage Gateway</a> Service. The service has three options at high level — <em>File Gateway</em>, <em>Volume Gateway</em> &amp; <em>Tape Gateway</em>. We will use <em>File Gateway</em> to map an S3 Bucket as a locally mounted drive. I know it’s little weird to see the use of phrase “File Gateway” here :), but in terms of functionality it’s very different than IBM File Gateway (SFG).</p><p>For detailed configuration steps, please refer to AWS Blog — <a href="https://aws.amazon.com/blogs/aws/file-interface-to-aws-storage-gateway">File Interface to AWS Storage Gateway</a>. It’s well documented there, so I won’t cover all those details in this article.</p><p>AWS <em>File Gateway</em> gives you couple of different options for mounting your S3 Bucket, all the options require you to have a dedicated VM that will work as the <em>File Gateway</em>, you can use — an EC2 instance if you are running B2Bi on AWS Cloud, VMWare ESXi or Microsoft Hyper-V option if you are running B2Bi on premise and finally, there is a Hardware based Appliance option too. Please note AWS Blog shared above doesn’t list Microsoft Hyper-V or the Hardware based Appliance as options as they were added at a later point in time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XqE6YFDACr37v0RELgrzXw.png" /><figcaption>Screenshot from AWS Storage Gateway</figcaption></figure><p>For this article, I chose the EC2 Instance to map the S3 bucket. I configured one EC2 instance that will be used as AWS File Gateway and will be configured with the intended S3 bucket — <em>b2bi-s3-demo</em>. Once <em>AWS File Gateway</em> instance is configured, this EC2 instance acts as File gateway and used by other EC2 instances to mount the S3 drive. This can be done in as many EC2 client instances as we want within same AWS ‘region’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BoOPVhplVSF5EUOG_iRavg.png" /></figure><p>Here “File-Gateway-Instance” is the one mapped to S3 bucket “b2bi-s3-demo”. In other two EC2 instances (File-Gateway-Client-1 &amp; 2) B2Bi/SFG Application will be hosted.</p><p><strong>A</strong>t high level there are five distinct activities</p><p><strong><em>Step:1 C</em></strong><em>onfigure the EC2 Instance to be used as File Gateway</em></p><p><strong><em>Step:2 C</em></strong><em>onfigure the File Gateway that uses the EC2 instance configured above</em></p><p><strong><em>Step:3 C</em></strong><em>onfigure File Shares that uses the configured File Gateway</em></p><p><strong><em>Step:4 M</em></strong><em>ount the File Share in your EC2 instances hosting B2Bi/SFG</em></p><p><strong><em>Step:5 C</em></strong><em>onfigure B2Bi/SFG to use the mounted S3 bucket</em></p><p>I will cover some of the high level configuration steps and highlight some of the critical steps. For details please refer to <a href="https://docs.aws.amazon.com/storagegateway/latest/userguide/ec2-gateway-file.html">AWS Documentation</a>.</p><p><strong><em>Configure the EC2 Instance to be used as File Gateway</em></strong></p><p>While configuring the EC2 instance for File Gateway Instance, I will recommend to use one of the <a href="https://docs.aws.amazon.com/storagegateway/latest/userguide/ec2-gateway-file.html">AWS Storage Gateway AMI instances</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pKzd7ywSkw10QEGF5DNdCA.png" /></figure><p>Second, while configuring the File Gateway EC2 instance, make sure you add an additional storage drive of minimum 150 GB, to be used for Caching.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WP27UAVlwBZTVxnkWi84pA.png" /></figure><p>Another important point — please change the security group’s default settings to allow inbound traffic. Unless you do this step, the public IP of the EC2 instance will not be accessible from File Gateway configuration UI. I spent lot of time before I figured it out :(.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZzPc5xUUopc3JdBd_xG6qA.png" /></figure><p>Once EC2 instance for File Gateway is ready, you need to create the File Gateway Instance.</p><p><strong><em>Configure the File Gateway that uses the EC2 instance configured above</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApcGs05r6VZOgDb93f0jjw.png" /></figure><p>Public IP of the EC2 instance for AWS File Gateway is used in above configuration.</p><p><strong><em>Configure File Shares that uses the configured File Gateway</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7BBVyk_kdYH4wrzUJZL1-A.png" /></figure><p>This gives you the commands/steps for mounting the File Share to your EC2 instance.</p><p><strong><em>Mount the File Share in your EC2 instances hosting your Application</em></strong></p><p>Next, I mounted the S3 drive in File-Gateway-Client-1 &amp; 2 by running a simple mount command as shown below -</p><p><em>mount -t nfs -o nolock,hard 172.30.2.163:/b2bi-s3-demo /data/s3</em></p><p>172.30.2.163 is the private IP of the EC2 instance that acts as File Gateway.</p><p>With these steps, files in S3 bucket “b2bi-s3-demo” were available in other two EC2 instances as regular files and regular file operations can be done by Java classes using standard java File IO.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iJGib97ijLf3XggVnWIE1w.png" /></figure><p>These are the same files available under S3 bucket as well -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hvcv3yNgT9oEJ4OFNQF4BQ.png" /></figure><p>Any changes done on the mounted drive from any of the EC2 instances get reflected immediately on the other EC2 instance as well as on S3. But changes done directly on S3 bucket doesn’t get reflected on File Gateway immediately. For that one can make a call to “AWS File Gateway” API or UI to refresh the File Gateway as soon as a new object added/deleted in S3 bucket directly i.e. without going through the AWS File Gateway. For B2Bi/SFG use-cases, I don’t expect anyone to write directly into the S3 bucket. The S3 Bucket should be treated as a system resource and only B2Bi/SFG should use that via AWS File Gateway which is nothing but the mounted drive.</p><p>Please note S3 bucket, File Gateway EC2 instance and Application hosting EC2 instances — all must be in same region. When used “US East (N. Virginia)” for everything, it worked fine. I tried cross region as well but that didn’t work.</p><p>Next, lets complete the configuration on B2Bi/SFG side.</p><p><strong><em>Configure B2Bi/SFG to use the mounted S3 bucket</em></strong></p><p>For using S3 for Document storage by B2Bi Adapter &amp; Services, please configure following two parameters in <em>customer_overrides.properties </em>(or use the new <a href="https://www.ibm.com/support/knowledgecenter/en/SS3JSW_6.0.0/administering/administering/integrator/SI_Customizations.html">Customization UI</a> available starting with v6.0).</p><p>jdbcService.document_dir=/data/s3<br>jdbcService.defaultDocumentStorageType=FS</p><p>Additionally, if you want to configure a separate directory where you can restore the document payloads, then please configure following property in the <em>customer_overrides.properties</em></p><p>jdbcService.RESTORE_DOCUMENT_DIR=/data/s3/&lt;a sub directory&gt;</p><p>For On Prem customers, I have not tried configuring the VMWare ESXi or Microsoft Hyper-V images. However, I don’t expect the behaviour and functionalities to be much different compared to what we covered above. For detailed steps on VMWare ESXi setup, you may refer to <a href="https://dzone.com/articles/configuring-aws-s3-storage-gateway-on-vmware-esxi">this article</a>.</p><p>Now, before you take the decision, here are few points to consider -</p><ol><li>You can move to FileSystem for Document storage without worrying about FileSystem getting flooded. Your new File System has the benefits of scalability of AWS S3.</li><li>Moving to FileSystem frees up your Database and help avoid some of the DB performance issues.</li><li>Note AWS File Gateway need it’s own Disks for use by the gateway as cache storage. Usually it’s 20% of your overall storage need on S3.</li><li>On performance front, let me quote from the <a href="https://aws.amazon.com/storagegateway/faqs/">AWS FAQ</a> “The performance you experience depends on what host platform (hardware appliance, virtual machine, Amazon EC2 instance) you are using to run Storage Gateway software and a number of other factors.”</li><li>Java Clients i.e. B2Bi/SFG will write to the <em>Storage Gateway&gt;File Gateway</em> using SMB or NFS, so there will be some overhead due to this.</li><li>I will suggest that you go through <a href="https://medium.com/@satyapaul/what-are-the-options-for-using-aws-s3-from-my-java-app-b5a111ad1748?source=friends_link&amp;sk=e2be826844b15986e5553631731c442f">this blog</a> to understand the performance impact of different options of integration with AWS S3.</li></ol><p>Here are few more points about AWS File Gateway, almost all of it from AWS Documentation.</p><p>Please take a look at the architecture diagram of File Gateway below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExQ_c9FoCaJaVrAGVfK1Kw.png" /></figure><p>AWS File Gateway asynchronously updates the objects in Amazon S3 as you change the files. This should help avoiding any delay due to network latency etc.</p><p>The service optimizes data transfer between the gateway and AWS using multipart parallel uploads or byte-range downloads, to better use the available bandwidth. Local cache is maintained to provide low latency access to the recently accessed data and reduce data egress charges.</p><p>On security front, objects are encrypted with Amazon S3–server-side encryption keys (SSE-S3), this is irrespective of whether you choose Encryption option in B2Bi/SFG. This is configurable at AWS S3 Bucket level. All data transfer is done through HTTPS.</p><p>That’s all for now. You are all set.</p><p>Please leave your comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c008eb838113" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>