<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zamira Jaupaj on Medium]]></title>
        <description><![CDATA[Stories by Zamira Jaupaj on Medium]]></description>
        <link>https://medium.com/@zamirajaupaj?source=rss-fd92b8a209bc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ABSAmA6UDSWF7bkYCjS18g.jpeg</url>
            <title>Stories by Zamira Jaupaj on Medium</title>
            <link>https://medium.com/@zamirajaupaj?source=rss-fd92b8a209bc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:14:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zamirajaupaj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My first trip in Shanghai, China]]></title>
            <link>https://zamirajaupaj.medium.com/my-first-trip-in-shanghai-china-b88cc5678608?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/b88cc5678608</guid>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Fri, 07 Jul 2023 08:32:51 GMT</pubDate>
            <atom:updated>2023-07-07T13:24:52.016Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*5qW_3vkfIar704IqSR637w.jpeg" /></figure><p>I have been fortunate enough to embark on numerous travel adventures over the past five years, exploring breathtaking destinations and immersing myself in different cultures. However, amidst the excitement and awe of each journey, I’ve realized that I’ve neglected to document and share my experiences. It’s almost as if a part of my travel story has remained untold, waiting to be unraveled. But my recent trip to China has left such a profound impact on me that I couldn’t resist the urge to break the silence and share my captivating experience with the world. Traveling to China is an entirely unique and extraordinary adventure, unlike anything I’ve encountered before. From its rich history and captivating landmarks to its vibrant culture and technological innovations, China presents a captivating tapestry of experiences. Join me as I delve into my journey through this incredible country, opening the doors to a world that left me in awe and inspired me to share my travel tales like never before.</p><h3><strong>Before embarking on my journey to China, I made sure to check off a few essential tasks to ensure a smooth and hassle-free experience. Here are some of the things I did before traveling to China</strong></h3><blockquote><strong>VPN</strong>: Since numerous American social media platforms and websites are inaccessible in China, I opted to utilize a VPN. This application is of utmost importance as it enables access to platforms like Google Translate, and Google Maps, facilitating navigation. However, if you possess an iPhone equipped with Apple Maps and Apple Translate, both applications function seamlessly without the need for a VPN.</blockquote><blockquote>WeChat and AliPay accounts: In today’s digital age, e-wallets have revolutionized the way we handle transactions, making cashless payments a breeze. When it comes to China, two prominent e-wallets dominate the scene: Alipay and WeChat Pay. These mobile payment platforms have become an integral part of daily life in China, with almost everything from street vendors to high-end establishments accepting payments through these platforms. To ensure I can make convenient and secure transactions while in China, I installed and activated both WeChat and Alipay accounts before my trip. This involved linking my bank account allowing me to easily pay for goods and services with just a few taps on my phone.</blockquote><p>WeChat, a widely used communication app in China, has long been limited for foreign visitors and residents due to the requirement of a Chinese bank account for WeChat Pay. However, recent developments have brought good news. The Chinese government has approved the addition of foreign cards to WeChat Pay, allowing individuals without a local bank account to utilize this convenient payment method. Despite this exciting development, I personally encountered difficulties with WeChat Pay, leading me to activate an Alipay account as an alternative solution.</p><h3><strong>Transportation</strong></h3><p>Transportation plays a crucial role in any travel experience, and when it comes to exploring China, it’s no different. As I embarked on my journey through this vast and diverse country, I discovered a transportation system that seamlessly combined efficiency, convenience, and a chance to immerse myself in the local culture. From navigating airports to experiencing the high-speed rail and taxi services.</p><p><strong>Air Travel</strong>: Arrive early and embrace modernity for international flights, it’s essential to allow ample time by arriving at the airport three to four hours in advance. The lines at major airports, especially during public holidays, can be quite lengthy. Delays are also not uncommon due to weather conditions or air traffic control, so patience is key. However, China’s airports are modern and well-designed, offering comfortable spaces for relaxation while awaiting departure.</p><p><em>I had the pleasure of flying with KLM, for the outbound flight, and China Eastern Airlines, a prominent Chinese carrier, for the return leg. Both airlines exceeded my expectations, ensuring a smooth and enjoyable travel experience.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CZ0eJ7x0Z4IiCcE5F7D2Bw.jpeg" /><figcaption>Real time flight tracked map</figcaption></figure><p><strong>High-Speed Rail</strong>: A marvel of efficiency and scenic beauty China’s high-speed rail network is a marvel that leaves foreign visitors in awe. With numerous cities interconnected by railways, train travel offers both efficiency and affordability. The ticket prices are modest, and the experience allows you to witness the real China as the landscape whizzes by. Foreign travelers can purchase train tickets on the China Railway’s official website (Chinese only) or directly at the train station by presenting their passport. Rest assured, personal information is handled securely without disclosure to any third parties other than China Railway.</p><p><em>As I arrived at Shanghai airport, I couldn’t contain my excitement to experience the renowned Shanghai Maglev. I had the opportunity to ride the Shanghai Maglev from the airport to the central station, and it was an exhilarating experience. Although the journey lasted only 7 minutes, the speed displayed on the monitor peaked at an impressive 301km/h, leaving me in awe.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*uTG_9ufCx-LFbK-Cj1xx8g.png" /><figcaption>Speed displayed on the monitor peaked at an impressive 301km/h</figcaption></figure><p><strong>Subway and Buses</strong>: Easy city navigation in major cities like Beijing, Shanghai, Xi’an, Guangzhou, and Chengdu, subways provide a convenient and efficient way to get around. With extensive networks and affordable fares, they offer a glimpse into the bustling city life. Some cities also boast an excellent bus network, although it’s important to note that bus stop names are often only displayed in Chinese.</p><p><em>Unfortunately, I didn’t have the chance to experience the subway during my trip, but it’s highly recommended for city exploration.</em></p><p><strong>Taxi &amp; DiDi</strong>: Language barriers and convenient alternatives taxis in China are relatively inexpensive compared to Western standards, but communication can be a challenge as many drivers don’t speak English. It’s advisable to carry a card with your destination or hotel address written in Chinese characters to show the driver. Since Uber withdrew from the Chinese market, DiDi has become the go-to alternative. With an English interface, DiDi offers a convenient and accessible ride-hailing option for travelers.</p><p><em>During my recent one-week stay in Shanghai, I found myself relying on taxis for my daily commute to work. While exploring the local transportation options, I came across DiDi, a popular ride-hailing platform in China. One of the remarkable aspects of DiDi is its seamless integration with AliPay, a widely-used mobile payment platform in China. Unlike other ride-hailing apps that require separate installations, DiDi can be accessed through a mini-app within the AliPay application. This eliminates the need for additional downloads and streamlines the process of ordering a taxi. Simply open AliPay and navigate to the DiDi mini-app to begin your journey.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*9QtO9kKYOnPqRSwCjFXr-Q.png" /><figcaption>Screenshot from Alipay app</figcaption></figure><p><em>Another advantage of using DiDi through AliPay is that it eliminates the need for registration. As a foreign traveler, this was a significant benefit, as I could easily access DiDi without going through the usual sign-up process. However, it’s worth noting that using DiDi as a foreigner does come with a limitation. Due to the restrictions associated with foreign cards, I discovered that I could only order a taxi and not avail myself of the other available options on the platform :). While local residents can enjoy a range of services, such as carpooling or premium car bookings. Despite the limitation imposed on foreign cardholders, the taxi service provided by Didi in Shanghai is efficient and reliable. Once I confirmed my location and destination within the app, I was quickly matched with a nearby taxi driver. The drivers I encountered were courteous and professional, making my daily commutes comfortable and hassle-free. The convenience of ordering a taxi through Didi ensured that I never had to worry about flagging down a cab on the busy streets of Shanghai.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5dEwL8P6uuHH3SeR_IlkhQ.jpeg" /><figcaption>Taxi in Shanghai</figcaption></figure><h3>Security &amp; Comfort</h3><p>My recent trip to Shanghai, China, left a lasting impression on me for a unique reason: the remarkable sense of security and comfort that enveloped me throughout my stay. From the moment I arrived in Shanghai, I noticed a distinct presence of security personnel in every corner. Whether it was the airport, train stations, or popular tourist attractions, there was a visible and reassuring display of police officers. This heightened security presence, combined with frequent identity and physical checks, fostered a sense of safety and protection. It was evident that the Chinese authorities prioritize the well-being of both residents and visitors, creating an environment conducive to exploring without worry. This feeling of being protected and looked after extended to every place I visited, making me feel like an honored guest in China.</p><p>One notable aspect of the security measures in Shanghai was the regularity of identity and physical checks. In train stations and other public areas, security personnel diligently screened individuals, ensuring that only authorized personnel were granted access. While some might find such checks intrusive, I found them to be a testament to the country’s commitment to safety and security.</p><p>Whether exploring the bustling streets of Shanghai or visiting historical landmarks, I always felt secure and protected. This allowed me to fully immerse myself in the beauty and charm of the city, without the constant worry that can sometimes accompany travel.</p><h3>Culture Shock</h3><p>Traveling to a foreign country often leads to encountering cultural differences that can sometimes be surprising, fascinating, or even bewildering. China, a land rich in history and tradition, is no exception. During my trip to this vibrant nation, I encountered several instances of culture shock that left me both amazed and perplexed.</p><ol><li><strong>Language </strong>barriers can be a real challenge. From ordering food to asking for directions, navigating everyday conversations can be daunting. However, there’s no need to fret! Thanks to modern technology, tools like Google Translate or Apple Translate can come to the rescue. If you find yourself in a similar situation, I highly recommend utilizing Apple Translate for a smoother and more enjoyable travel experience.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*6gWni4Bp0pbNPdva9OKpaw.png" /><figcaption>Apple translate app</figcaption></figure><p><strong>2) Tanning</strong>: A contrasting beauty ideal in many western countries, sun-kissed skin is often considered a symbol of beauty, with people flocking to beaches and tanning salons to achieve that coveted bronzed look. However, in China, the cultural perspective on skin tone is quite the opposite. Fair and pale skin is traditionally associated with beauty and wealth. It was fascinating to observe locals carrying umbrellas or wearing long-sleeved clothing on sunny days to shield themselves from the sun’s rays. This stark contrast in beauty ideals reminded me of the diverse perspectives on aesthetics around the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*kmKTI77tPpfHy2phU77Sag.png" /><figcaption>Great wall in Beijing</figcaption></figure><p><strong>3) The Hot Water Myth</strong> Many locals believe that drinking hot water is beneficial for health and digestion. Consequently, finding cold water in restaurants or public places can be a challenge. While I initially found this puzzling, I soon adapted to the practice, realizing that it was an integral part of Chinese culture and respecting the local preferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*0iDxgLIoce06wUhyc3wkjg.png" /><figcaption>The Hot Water</figcaption></figure><p><strong>4) Queuing Etiquette</strong>: A different approach in many western countries, queuing is considered a fundamental social norm, with people patiently waiting their turn in an orderly fashion. In China, however, the concept of queuing can be a bit different. Large crowds, especially in popular tourist attractions or busy transportation hubs, can lead to a more chaotic and fluid approach to queuing. It took some getting used to, but I quickly realized that blending into the flow and being adaptable was the key to navigating these situations smoothly.</p><p><strong>5) Chopsticks</strong>: A Skill to Master, One of the most iconic aspects of Chinese culture is the use of chopsticks when eating. For someone like me, who was accustomed to using a knife and fork, it was an intriguing challenge to learn how to wield these slender utensils. While chopsticks are commonly used for various dishes, including rice, noodles, and dumplings, they can also be employed for picking up smaller food items like vegetables or pieces of meat. It was a joyous experience as I gradually improved my chopstick skills, appreciating the precision and grace with which locals effortlessly handled their meals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*vncxXRXw2vALxD8jU8kiCw.jpeg" /><figcaption>Chopsticks</figcaption></figure><h3>Food and Gastronomy</h3><p>Exploring the food culture of Shanghai as a foreigner was an unforgettable experience that opened my senses to new flavors, textures, and culinary traditions. From the bustling street food markets to traditional Shanghainese eateries, each bite offered a glimpse into the rich tapestry of Shanghai’s gastronomy. Despite the language barriers, the warm hospitality and willingness of locals to share their culinary heritage made the journey even more memorable. If you find yourself in Shanghai, don’t miss the opportunity to embark on your own culinary adventure, embracing the unknown and allowing the city’s vibrant food scene to captivate your palate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46gphANZNqeIufhewnxT_Q.jpeg" /><figcaption>Exploring the food culture of Shanghai</figcaption></figure><h3>Embracing Warmth and Support</h3><p>Visiting a foreign country for business purposes can be an exciting yet challenging experience, especially when the visit is relatively short. However, in China, the warm hospitality and helpful nature of Chinese colleagues can make all the difference. During my one-week stay in China, I had the privilege of witnessing firsthand how my Chinese colleagues went above and beyond to assist and support me throughout my journey.</p><blockquote><strong>Navigating Cultural Differences</strong>: Chinese colleagues understand that navigating cultural differences can be overwhelming for foreigners. From language barriers to unfamiliar customs, they are empathetic and patient in helping me counterparts adapt to the local culture. Whether it’s explaining social norms, offering guidance on appropriate etiquette, or suggesting local customs to embrace, their support helps bridge the gap and promotes a sense of belonging.</blockquote><blockquote><strong>Translation and Interpretation</strong>:Language can often pose a significant challenge when traveling to a foreign country. Chinese colleagues graciously offer their translation and interpretation skills to assist me. From translating important documents to facilitating conversations during meetings, their linguistic proficiency eases communication barriers and ensures effective collaboration. Their willingness to go the extra mile in bridging the language gap is truly commendable.</blockquote><blockquote><strong>Local Recommendations and Exploration</strong>: Chinese colleagues are a treasure trove of local knowledge and recommendations. They take pride in introducing foreigners to the rich cultural heritage, delicious cuisine, and hidden gems of their country. Whether it’s suggesting popular local eateries, sharing insights on tourist attractions, or organizing group outings to experience traditional festivals, their guidance helps visitors immerse themselves in the authentic Chinese experience.</blockquote><blockquote><strong>Logistical Support</strong>: Navigating logistics in an unfamiliar city can be daunting, but Chinese colleagues are quick to lend a helping hand. They assist with transportation arrangements, offer guidance on public transit, and even accompany their foreign counterparts to ensure a smooth and hassle-free commute. Their thoughtfulness and attentiveness make a significant difference in reducing stress and enabling foreigners to focus on their work responsibilities.</blockquote><blockquote><strong>Cultivating Lasting Relationships</strong>:Beyond the professional realm, Chinese colleagues understand the importance of building personal connections. They often extend invitations to social gatherings, allowing foreigners to experience the warmth and hospitality of Chinese culture. These interactions foster trust, create lasting friendships, and contribute to a sense of camaraderie within the workplace.</blockquote><figure><img alt="Eva, is my Chinese colleague and I. The photo was taken during the conference that I was part of my business trip" src="https://cdn-images-1.medium.com/max/1024/1*oWYCqCNI9UimKuNnH_zvAg.jpeg" /><figcaption>Eva, my Chinese colleague and I. The photo was taken during the conference that I was part of my business trip</figcaption></figure><h3>Final words</h3><p>As I reflect on my first trip to Shanghai, China, I am filled with a sense of awe and gratitude. The journey has been nothing short of remarkable, filled with captivating experiences, cultural discoveries, and unforgettable moments. From navigating the language barriers to immersing myself in the vibrant food scene, every aspect of this adventure has been both challenging and rewarding.</p><p>The warmth and hospitality of the local people left an indelible mark on my heart. The genuine curiosity and friendliness extended towards me, as a foreign visitor, made me feel welcome and valued. The encounters with locals, whether through shared meals or chance conversations, provided a glimpse into the rich cultural tapestry of China, leaving me with cherished memories and a deeper appreciation for the country’s heritage.</p><p>As I bid farewell to Shanghai, I carry with me a newfound appreciation for the rich cultural heritage, culinary delights, and warm hospitality that define this remarkable city. My first trip to China has been an enlightening and transformative experience, inspiring me to continue exploring the world and embracing the diverse cultures that shape our global community. I am grateful for the opportunity to have experienced Shanghai as a foreign visitor, and I eagerly look forward to future adventures that await me in this captivating country and beyond.</p><p>See you next time :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b88cc5678608" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Smart AWS Industrial Machine Learning Technology]]></title>
            <link>https://zamirajaupaj.medium.com/smart-aws-industrial-machine-learning-technology-1951b366a0cf?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1951b366a0cf</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Wed, 09 Dec 2020 19:43:43 GMT</pubDate>
            <atom:updated>2020-12-15T18:08:06.808Z</atom:updated>
            <content:encoded><![CDATA[<p>Inspection in the manufacturing industry requires a series of controls such as testing, measuring, and examination of an object, material or activity. These controls are done by technologies or people, or a combination of both with the purpose of determining whether or not it is in proper condition. <em>In this blog, </em><strong><em>I’ll explain the 4 industrial machine learning services provided by AWS</em></strong><em> that helps </em>manufacturers to keep their operations running within specified limits and ensure quality and prioritize safety.</p><blockquote><a href="https://medium.com/p/1951b366a0cf#1a52"><em>1- Amazon Monitron</em></a></blockquote><blockquote><a href="https://medium.com/p/1951b366a0cf#1758"><em>2- Amazon Panorama Appliance</em></a></blockquote><blockquote><a href="https://medium.com/p/1951b366a0cf#dd8d"><em>3- Amazon Lookout for Vision</em></a></blockquote><blockquote><a href="https://medium.com/p/1951b366a0cf#7e74"><em>4- Amazon Lookout for Equipment</em></a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*4MGedB67w5BDoklo" /></figure><h4>#1 — <em>AWS Monitron</em></h4><p>AWS Monitron is a combination of <em>sensors, gateway, and machine learning </em>that allows industries to monitor end-to-end equipment in their production line. Manufacturers can connect these sensors produced by AWS to their legacy equipment to measure the temperature and vibration, etc.</p><h4>How does it work?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*ZHWEdl4bBGCAWM3fiilnSA.png" /><figcaption>End-to-end monitoring to manufacturing</figcaption></figure><p>1- These smart sensors are Bluetooth low energy type, are attached to the equipment to measure the temperature and vibrations, and then send these data to the Gateway.</p><p>2- Gateway is a physical device connected via wifi to the local Internet and automatically gets data from all sensors connected to it. Then this data is sent to the AWS cloud.</p><p>3- All data sent by the gateway are stored in AWS, using a machine learning model to identify all anomalies and defects detected by the sensors and automatically sent the result to a mobile application.</p><p>4- A friendly dashboard is displayed with a list of all the equipment in the mobile application, which a technical person can identify whether the equipment is healthy or unhealthy.</p><p>5- The Quality Control Officer identifies all the sensors that are unhealthy and then can go and do some inspections where unhealthy equipment is set up.</p><h4><em>#2 — AWS Panorama Appliance</em></h4><p>AWS Panorama Appliance is a smart device that allows customers to make predictions based on the images captured by cameras that are added to this device. This device has the capability to use computer vision to improve quality control.</p><h4>How does it work?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*Pky5cYtr22D7mbcbnrot9A.png" /><figcaption>10 camera attached to the appliance</figcaption></figure><p>1- Cameras type H.264 for live streaming sending the data to Appliance.</p><p>2- This device supports up to 10 cameras simultaneously, collects all data, and then sends the data to the cloud.</p><p>3- Machine Learning makes a prediction based on the data collected by Appliance.</p><h4>#3 — <em>Amazon Lookout for Vision</em></h4><p><em>Amazon Lookout for Vision</em> is a Machine vision intelligence for product defect inspection based on deep learning. This enables customers to monitor and find defects in their product line in real-time.</p><p>The following image shows how Machine Learning makes a prediction based on these images and visions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SsCZn0nqU-R3_bssC93_Ew.png" /><figcaption>The camera detects the issue from the images</figcaption></figure><h4>#4 — <em>Amazon Lookout for Equipment</em></h4><p>Amazon Lookout for Equipment allows customers to use their own sensors to detect anomalous equipment, it’s quite the same logic as Amazon Monitron, but in this case, customers are using their own sensors instead of AWS’s. In addition, this service is based on machine learning that trains the data collected by these sensors and performs predictive maintenance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*PBeAIEHb-SsMrHQnKk6jBw.png" /><figcaption>Customers sensors send data to the Cloud</figcaption></figure><p>1- Sensors identify the anomalies in legacy equipment and send the data to the cloud.</p><p>2- Analyze data from inputs like generators, compressors, turbines and, so on.</p><h4><em>Conclusion</em></h4><p>All these smart industrial equipment are very useful in the industrial world today, helping customers to automate all processes and reduce costs. There are simple cost-effective solutions leveraging machine learning capabilities to make predictions in real time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1951b366a0cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning  keynote liveblog re:Invent 2020]]></title>
            <link>https://zamirajaupaj.medium.com/machine-learning-keynote-liveblog-re-invent-2020-9abab404a280?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9abab404a280</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Tue, 08 Dec 2020 18:29:10 GMT</pubDate>
            <atom:updated>2020-12-16T10:06:44.576Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Machine Learning keynote live blog re:Invent 2020</h3><p><strong>Dec 8, 2020 | 4:45 PM — 7:00 PM CET</strong></p><p>This is the first time AWS has introduced a keynote for machine learning(ML).</p><p>4:45–5 PM → Some great music by White Buffalo before the keynote</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtN2Kxt78vHJxDHIf2OkHg.png" /></figure><p>5 PM → VP of machine learning DR Swami Sivasubramanian started an introduction about ML, he explained the three layers that ML offers:</p><blockquote>1- ML frameworks &amp; infrastructure</blockquote><blockquote>2- Amazon SageMaker</blockquote><blockquote>3- AI Services (6 more launch only this year)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sms5xdpmgRMFLhiOaV__SQ.png" /></figure><blockquote>Provide a firm foundation gives you the freedom to reinvent said Dr. Swami Sivasubramanian</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7jUmpnmwiDd-aPp8STm9Q.png" /><figcaption>Three primary frameworks for ML</figcaption></figure><blockquote>90% of Machine Learning runs on TensorFlow and PyTorch. Its important to grow in this area, helping builder to learn faster said Dr. Swami Sivasubramanian</blockquote><h3><strong>Below the list of new machine learning services and features!</strong></h3><blockquote><strong>AWS Inferentia (lower cost inference in the cloud)</strong></blockquote><ul><li>45% lower costs per interface than comparable GPU based instances</li><li>30% higher throughput than comparable GPU based instances</li><li>Amazon Alexa archives 25% lower end to end latency</li></ul><blockquote><strong>AWS Tranium (New machine learning chip customer. designed by AWS to deliver the most cost-effective training in the cloud)</strong></blockquote><ul><li>Most teraflops of any ML instances in the cloud</li><li>Use same Neuron SDK as Inferentia</li><li>Available as Ec2 Instances or Image SageMaker</li></ul><blockquote><strong>[New Launch] Faster Distributed Training on Amazon SageMaker 40% faster training.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F35q_NvmNqHsxsqLeWTF4A.png" /></figure><blockquote>Machine learning is one of the services grows faster in 2020 in AWS said: Dr. Swami Sivasubramanian</blockquote><blockquote><strong>[New Launch] Amazon SageMaker Data wrangeler</strong></blockquote><p>The faster way to prepare data for Ml, training data without a single line of code.</p><blockquote><strong>[New Launch] Amazon SageMaker Clarify</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fd4QUKXBAcMQjIqwYxziuA.png" /></figure><ul><li>Train data for tv show recommendation</li><li>Data labeling and selection</li><li>Model drift</li></ul><p>CPU is one of the most expensive said: Dr. Swami Sivasubramanian</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Fl3FnVDZyubTlBA2nj5Ow.png" /></figure><p><strong>[New Launch] Deep Profiling for SageMaker Debugger</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*SyRDyFwSvk1ZogkCdhbD3A.png" /></figure><p>Debug resources help to identify bottlenecks, to detect bias in machine learning workflow.</p><ul><li>Visualize different systems resources including GPU, CPU, and I/O memory.</li><li>Analyze resources utilization and get recommendations on how to adjust</li><li>Offline training jobs to run at any point in the machine learning workflow</li></ul><blockquote><strong>[New Launch] Amazon SageMaker Pipeline</strong></blockquote><ul><li>Seamlessly manage each step of your end to end workflow</li><li>Share end re-run workflow</li><li>Get started quickly with a preconfigured customizable workflow template</li><li>Compare workflows visually to optimize model performance</li></ul><p>We need a lot of different types of data, then Sagemaker can use this data to training the model said <a href="https://medium.com/u/8f15b369514d">Matt Wood</a> AWS VP Artificial Intelligence</p><blockquote><strong>[New Launch] Amazon SageMaker Edge Manager</strong></blockquote><p>Manage and Monitor ML Model across fleets of Smart Devices</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVP7HPjuhDQdKVHxq6p1LQ.png" /></figure><blockquote><strong>[New Launch] Amazon Aurora ML (Add ML based predeicitons to applicaiton via SQL)</strong></blockquote><blockquote><strong>[New Launch] Amazon Athena ML(add ML based predeicitons toyour queries on S3) → Invovle ML model form SQL queries)</strong></blockquote><blockquote><strong>[New Launch] Amazon Redshift ML(Use SQL to make ML prediciotns form your data warehaouse)</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFvC_EYANBgELCelVPaXZw.png" /></figure><blockquote><strong>[New Launch] Amazon Neptune ML -&gt; predciting for graph application</strong></blockquote><blockquote><strong>[New Launch] Amazon Quick Sight Q(Powerful out of the box ML capabilties)</strong></blockquote><ul><li>Uncover hidden ML insights</li><li>Predict growth and business trends</li><li>Use auto narratives to tell the story of your dashboard</li></ul><h3><strong>Challenging of Anomaly detection in metrics</strong></h3><blockquote><strong>[New Launch] Amazon Lookout for Metrics</strong></blockquote><ul><li>Hight latency of detection</li><li>Failed detection</li><li>Too many false detections</li><li>Lack of actionable restuls</li><li>Adaptability to changes in data streams</li></ul><blockquote><strong>[Andy Jessy New Launch] Amazon Monitron</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8TBThiL8uLeNZ1db9GoXTw.png" /></figure><blockquote><strong>[Andy Jessy New Launch] Amazon Lookout for Equipment</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B-MRlUDrGb3kl9edJQZ3Wg.png" /></figure><blockquote><strong>[Andy Jessy New Launch] Amazon Panorama Appliance</strong></blockquote><ul><li>Ip62 rated ruggedized device, dust and water-resistant</li><li>1U and half-rack wide with chassis points</li><li>Multiple Gigabit Ethernet ports dor redundancy or connection cameras from multiple subnets</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fxUrj-YCiQik08Mrp-O2-w.png" /></figure><blockquote><strong>[Andy Jessy New Launch] Amazon Lookout for Vision</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*FTi2ffCjIb-Djf3ydpON6w.png" /></figure><blockquote><strong>[New Launch] Amazon HealthLake (store, transform and analzye health and lie scencies data in the cloud, or petabyte scale)</strong></blockquote><ul><li>Seamlessly transforms data to understand and extract meaningful medical information</li><li>Organized data in chronological order so that you can look at trends</li><li>Build-in data query, search, and ML capabilities</li><li>Support interoperability, standards like FHIR( Fast interoperability resources) to enable data sharing health systems</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9abab404a280" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Four differences between EKS Fargate and Node Managed]]></title>
            <link>https://zamirajaupaj.medium.com/four-differences-between-eks-fargate-and-node-managed-2c0458884a8d?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c0458884a8d</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[securit]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks-fargate]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Thu, 26 Nov 2020 13:39:35 GMT</pubDate>
            <atom:updated>2020-12-01T13:14:14.616Z</atom:updated>
            <content:encoded><![CDATA[<h3>EKS Node Managed VS EKS Fargate</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F-lv7iNeDoV87ynduqZWVQ.png" /></figure><p><strong>Introduction</strong></p><p>Are you in doubt about what would be the right option to run Kubernetes on AWS? In this blog post you will get the answer, I will show you a few differences between EKS Node Managed and EKS Fargate.</p><p>Containers are not a new concept in the IT world: they have been widely used in the public cloud or on-premises. Most public cloud providers offer different approaches on how to implement the containers. Let’s talk about EKS on AWS. You may ask yourself what does EKS stand for?</p><p>EKS cluster is a container orchestration platform that consists of a set of virtual machines called worker nodes and is designed to manage the lifecycle of containerized applications. Control Manager of EKS manages the nodes and the pods in the cluster. A “pod” is a group of one or more application containers. This platform also provides availability, scaling, and reliability of the pods.</p><p>AWS supports 2 EKS models:</p><blockquote>EKS Fargate: Container as a Service (CaaS) also called “serverless for containers”.</blockquote><blockquote>EKS Node Managed: Infrastructure as a Service (IaaS)</blockquote><p><strong>EKS Fargate. </strong>This model has been designed to give developers the possibility to concentrate only on their workload, pod configurations, and application logic, without worrying about the infrastructure management, availability, fault tolerance, scaling, and patching of host machines. Therefore the control manager and worker nodes are managed by AWS.</p><p><strong>EKS Node Managed.</strong> This model gives developers the freedom to manage not only the workload but also the worker nodes. Worker nodes consist of a group of virtual machines. However, the control manager is always managed by AWS.</p><p>The following drawing shows a high-level difference between EKS Fargate and Node Managed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/0*vOszf9cvtZKKUZ6J" /><figcaption><em>EKS Node Managed vs Fargate</em></figcaption></figure><p><strong>Let me show you a few differences between them:</strong></p><p><strong>1. #SECURITY</strong></p><p>Nowadays, security is a fundamental component. The importance of data protection increases as the amount of data grows, so it is important to have in place a good approach to protect the data and systems from challenging threats. A good way to start is by understanding the shared responsibility model: what are AWS’s and Customer’s responsibilities, and who is responsible to enhance the security of pods, network, incident response, and compliance. Both services are compliant by meeting the security standards, and there is also an SLA 99.99%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/0*QmOY8ILbUhk3m4-j" /><figcaption><em>Shared responsibility EKS model</em></figcaption></figure><p><strong>2. #SCALABILITY</strong></p><p>Your resources increase or decrease according to the system’s workload demands. To better manage the workload, it’s good to have the right scaling strategy allowing systems to scale without interrupting their operation and to ensure that the activities work as it is expected.</p><ul><li>In EKS Fargate, AWS is responsible for managing the scaling of the worker nodes in the cluster, so you don’t need to worry about scalability.</li><li>In Node Managed, the customer is responsible for managing the scalability of the worker nodes.</li></ul><p>There is a tool in AWS that helps you to manage different scaling approaches, this service is called Auto Scaling Group(ASG). It provides vertical (scale-up) and horizontal scaling (scale-out). The scope of both these approaches is to deliver performance and capacity that the workloads require. But these models work differently. You may ask: Which one is a better solution? Let me briefly explain how they handle the resource based on the workload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*9dMpQkekbu6ZBo-A" /><figcaption><em>Scale-out vs scale-up</em></figcaption></figure><blockquote><strong>Scale-out: </strong>Add or remove nodes to the cluster, this means a group of virtual machines can be deployed as dependent pieces.</blockquote><blockquote><strong>Scale-up: </strong>Add or remove capacity to a single node, you can add more CPU, RAM, Storage to this node. Not highly recommended as you don’t have redundancy, and high availability, as this machine will reside in a single availability zone.</blockquote><p><strong>3. #NETWORK</strong></p><p>Running the EKS cluster on AWS requires a pre-existing networking configuration. So, the first step is to set up the virtual private network (VPC) configuration. Keep in mind what workload you may have before deciding the Inter-domain routing (CIDR) of VPC. As your cluster grows and your applications become more complex you may need other resources running on the same network such as load balancers, databases, etc. It’s important to have a scalable solution that fits your requirements, so you won’t be in a position to recreate the resources and network from scratch because your CIDR block is too small in the first place. For both Node Managed and Fargate Control manager is part of the internal network managed by AWS, so you don’t have to worry about that. In the case of Fargate, AWS also manages the worked node network, so you’ll only need to calculate the pod IPs.</p><p>Let me give you two examples of how many IP Addresses you need available to run a simple application in EKS. Here are some combinations of subnet size and machine type.</p><ol><li>One cluster with 5 nodes and 40 pods running on each node.</li><li>Two clusters with 30 nodes and 40 pods running on each node.</li></ol><p>How many available IPs will we have for different Subnet CIDR range setup? In the illustration below are three different scenarios, and each one of them shows how many available IPs we will have for different SIDR ranges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2TngZ3O5szSs23eiaZEGQ.png" /></figure><p>Now let’s imagine we want to use different worker nodes types m5.large and m5.4.xlarge. Can we leverage all available IPs in both scenarios? As you can see in the illustration below there are some limitations on pods availability for different instance types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XqPBY2hcLLNmq9Aa4Qiomw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZH3Aqo3U6-dpa1yniz6CA.png" /></figure><p>In the example # 2 above, you will need at least CIDR /21 to fit the requirements.</p><p><strong>4. #COSTS</strong></p><p>Cost optimization is an important component in any organization, and it requires a good understanding of costs for short and long term. In AWS, you pay for computing capacity per second depending on what cluster you run. Let me show you how much Fargate and Managed Node will cost for one, three and six months. To illustrate this better, I will showcase you two different cases.</p><p><strong>Managed Node Case: 15 nodes, using 2vCPU, 16 GB memory each (3 hours a day and 24 hours).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZRUFY-64w3GWH8swNyirg.png" /></figure><p><strong>Fargate Case: 15 pods, using 2vCPU, 16 GB memory each (3 hours a day and 24 hours)</strong></p><p>Total vCPU charges = # of Pods x # vCPUs x price per CPU-second x CPU duration per day (seconds) x # of days</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jlO-eyqpxQQgq2EE1juKNw.png" /></figure><p>Total memory charges = # of Pods x memory in GB x price per GB x memory duration per day (seconds) x # of days</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hjPaKJZ5zFycH9rDsw85rA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*UjPzJbvrFoliL_xz" /><figcaption><em>Short and long term costs</em></figcaption></figure><p><strong>Conclusion</strong></p><p>Customers may have complex container architecture in their data centers that may face challenges. However, taking advantage of the cloud gives you the flexibility to focus on what you need, and quickly solves scalability, readability, and security.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c0458884a8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Accelerate Your Cloud Adoption with AWS Control Tower]]></title>
            <link>https://zamirajaupaj.medium.com/accelerate-your-cloud-adoption-with-aws-control-tower-fc59230e0879?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc59230e0879</guid>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws-control-tower]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 14:00:51 GMT</pubDate>
            <atom:updated>2020-07-10T12:49:34.348Z</atom:updated>
            <content:encoded><![CDATA[<p>When migrating to the cloud, companies face challenges governing their data and security. In the field of cloud computing, a landing zone is a basic foundation for further cloud deployments. AWS Control Tower is a managed service to implement such a landing zone in a fast and standardized way.</p><p>In this article, I will describe:</p><ol><li><a href="#0ac5"><strong>Main Challenges</strong></a><strong>-</strong> Companies are facing to govern their data and systems</li><li><a href="#579d"><strong>Landing Zone Capabilities</strong></a><strong>-</strong> How a landing zone can help</li><li><a href="#b1cd"><strong>AWS Control Tower</strong></a><strong>-</strong> A managed landing zone solution</li><li><a href="#2bd9"><strong>Next Steps-</strong></a> Using out of the box or custom landing zone</li></ol><h3><strong>Main Challenges</strong></h3><p>Let’s imagine a company organizational structure is made up of teams such as finance, marketing, security, developers, operations, and so on. Each team has certain activities to achieve goals, functionalities, and the processes that guide them cross-functionally. As the company gets bigger, the activities increase, the size and functionality of the teams increase, and new departments come up with new use cases. At the top of the company is a general manager who governs everything, therefore the need for a centralized and well-controlled structure will arise so that everything works perfectly. The same workflow affects our environments, systems, and applications in on-premise or cloud. Due to a large number of applications, systems, and teams, enterprise companies are facing the challenges to manage and govern all of them safely in the cloud.</p><h3><strong>Landing Zone Capabilities</strong></h3><p>In my experience with enterprise companies, one of the biggest challenges is how to find a simple, strategic solution to govern the systems and data in a secure, scalable, and efficient way. They are truly looking for a full-scope process while keeping their systems under control, such as who can access what and when, if there are security and vulnerability scanners in place, regulatory auditors, etc.</p><p>A cloud landing zone is a centralized management solution of accounts, services, and processes between them. Each account or service has certain activities and functionalities, which are governed by centralized governance. Typically, a landing zone offers:</p><ul><li>A good structure for the cloud environment that includes basic IT capabilities</li><li>Change management solutions to handle changes securely</li><li>Centralized account management</li><li>Centralized logging and risk management</li><li>Centralized policy and compliance with a security baseline</li><li>Centralized cost management</li><li>Defense solution to prevent any issues</li><li>Right access control to data and systems</li><li>Scalable, flexible, and efficient solutions</li><li>An easy and fast way to set up</li></ul><h3><strong>AWS Control Tower</strong></h3><p>In our case, the control tower is not a building on the airport, but a service managed by AWS. It offers a highly effective governance solution by providing an automated way of implementing a cloud landing zone.</p><p><strong>What does Control Tower offer?</strong></p><p>Over time, AWS has carefully curated a best practice for landing zone solutions. Control Tower is a managed service to facilitate landing zone setup and ongoing management, and to govern a secure compliance multi-account and services, and access management using an established blueprint. Providing a range of services and mechanisms for different purposes, such as ensuring 24/7/365 monitoring of critical systems, allowing traceability in real-time, audit actions and changes to systems, performance improvements, real-time alerts, and potential outage preventions, offer greater reassurance and efficiency. In the previous blog, I detailed <a href="https://medium.com/@ZamiraJaupaj/setting-up-aws-landing-zone-with-aws-control-tower-a746a99ca76a">Setting Up an AWS Landing Zone</a> with Control Tower — check it out now if you haven’t already.</p><p><strong>Features and capabilities</strong></p><p>You can set up a Control Tower in a centralized management account within a few hours. You only need to enter two emails, one linked to a logging account and the other one to an audit account. Both emails must not be linked to an existing AWS account, otherwise it will fail. Within two hours you will see a full setup of an AWS landing zone as is described in the image below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gU3er_79DlNlHOp2QRBhPA.png" /><figcaption>Capture in Control Tower Dashboard</figcaption></figure><p><strong>Account structure and services</strong></p><p>The image below lists the AWS services linked to each account. From the orange line and above, all services are created automatically during the Control Tower setup and all of them are part of the AWS landing zone solution. Below the orange line may be the services you can add in the landing zone for future purposes or you can add an account based on your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*bvSdP0feTiYN1wmgFSfdOA.png" /><figcaption><em>Basic Landing Zone provided by Control Tower</em></figcaption></figure><p>Let’s see all the AWS services provided in each account and the relationship between them:</p><ol><li>AWS Services deployed to all accounts</li><li>AWS Services deployed to the master account</li><li>AWS Services deployed to shared accounts</li></ol><p><strong>AWS <em>Services deployed</em> <em>to all accounts</em></strong></p><p>In the first part of the table are services such as CloudFormation, IDP, IAM roles, CloudWatch, and CloudTrail, all of which are being deployed to all accounts because they help to connect and grant permissions through accounts and services and to perform certain operations automatically. Also, they make sure that monitoring is in place. Let’s see what their purposes are:</p><p><a href="https://aws.amazon.com/cloudformation/"><strong>AWS CloudFormation templates</strong> </a>— These are established blueprints provisioned by AWS that allows Control Tower to automatically setup accounts and services. These models describe resources in JSON format and there is in-place versioning control.</p><p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers.html"><strong>Identify Provider (IDPs)</strong></a> — This allows accounts to grant permission on cross-account resources. These services are applied to all accounts.</p><p><a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html"><strong>IAM Roles</strong></a> — This defines a set of authorizations and they are associated with different entities. Below are the three most common types created by Control Tower.</p><ul><li><strong><em>Service-linked-role</em></strong> — These types of roles are associated with resources to grant permission to other resources, API.</li><li><strong><em>Cross-account-role</em></strong> — These types of roles grant resources permission in cross accounts, so resources of AWS account A access to resources of AWS account B and vice versa. In this scenario, the Audit and Log account allow the master account to access its resources.</li><li><strong><em>Saml-federation-role</em></strong> — These types of roles delegate access across AWS accounts via SAML.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*zoFMKIJEebp6IjJBLjVcdA.png" /><figcaption>IAM Role cross accounts</figcaption></figure><p><a href="https://aws.amazon.com/cloudwatch/"><strong>CloudWatch logs</strong></a><strong> — </strong>The scope of this service is to monitor the infrastructure, network, and applications, and take control in case of detection. All of the services that are part of the landing zone generate the metrics and send these metrics to CloudWatch.</p><p><a href="https://aws.amazon.com/cloudtrail/"><strong>CloudTrail</strong></a><strong> — </strong>This service also is used for monitoring but has a different scope. It allows you to collect all the history of API through AWS Management Console, AWS SDKs, and command line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*bytZyFsdkehVc3u50xgBcw.png" /><figcaption>ClaudTrail</figcaption></figure><p>Services applied to a master account</p><p>Those services are deployed only to the master account because their purpose is to govern and manage systems and processes in a secure and scalable way. Let’s review them.</p><p><strong>Single Sign-On (SSO) — </strong>This allows you to centrally access management and permission to AWS accounts. You can authenticate an AWS account with a single user and password through a custom, friendly link provided by SSO. Also, you can setup user permissions, account permissions, group permissions, and session duration preferences. Centralized access management avoids creating IAM users and permissions in child accounts.</p><p><strong>Service Catalog — </strong>This allows you to provision portfolios and add the product based on requirements and the specified budget. Control Tower creates its portfolio with an already associated product: “AWS Control Tower Account Factory.” By default there is no budget specified. If you want to launch a new account part of Control Tower, you can do it through Account Factory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*miFJeOIvVQk1KbEWK345Iw.png" /><figcaption>Service Catalog</figcaption></figure><p><strong>AWS Organization — </strong>It is used to set up an account organization structure in groups. In each of them, you can apply a service control policy (SCP), ensuring that accounts stay within your organization’s access control guidelines. Control Tower creates three groups (root, core, and custom). Core and custom are under root. Accounts log and audits are part of the core unit organization. But you also can extend this structure, adding more organization units and accounts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*Y6D7Hbtxh9CuX1eccDK7KA.png" /><figcaption>Organization units</figcaption></figure><p><strong>Guardrails — </strong>There are some basic security and compliance rules that are part of the baseline deployed to all new accounts automatically to prevent and detect certain resources from behaviors that don’t match the rules, and they become non-compliant.</p><p>Services applied to shared accounts</p><p>The purpose of the log account is to collect all the logs between accounts and regions. The purpose of the security account is to audit environments and services by following the protection rules.</p><p><strong>AWS Config — </strong>This allows you to create rules on how to deploy a specific AWS service. In the case that they don’t follow the rules, this ensures that the service doesn’t become a complaint.</p><p><strong>AWS CloudWatch Event Rules — </strong>This allows you to create a rule to trigger an AWS service based on a custom schedule.</p><p><strong>AWS SNS — </strong>This service is used for notifications.</p><p><strong>AWS Lambda Function — </strong>This service allows you to run your code.</p><p>Now let’s see how these services work together in the solution provided by the Control Tower. The following workflow shows when the S3 bucket in log and audit account becomes non-compliant based on the guardrail’s rules. Let’s review the steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*cNz9aVulqXAVFqD8lAWb9Q.png" /><figcaption>Guardrails setups</figcaption></figure><ol><li>A user creates a public bucket or changes the configuration of the existing bucket from private to public.</li><li>AWS Config checks that your S3 buckets do not allow public read access. If an S3 bucket policy or bucket ACL allows public read access, the bucket is non-compliant.</li><li>CloudWatch events trigger SNS topics when the config rules compliance change. In this case, the S3 bucket becomes a non-compliant service.</li><li>SNS topic invokes lambda function.</li><li>Lambda sends a notification to the second SNS topic.</li></ol><p><strong>Costs</strong></p><p>The landing zone created by Control Tower uses AWS resources and therefore generates some costs. For a newly created standard landing zone, this is expected to be:</p><ul><li>$1,86 per month for <a href="https://aws.amazon.com/config/pricing/">AWS Config</a> to record two configuration items (= 31 resources X 2 accounts X 4 regions X 2 configuration state changes).</li><li>$ 0.23 per month for S3 bucket, (45,189.000 PUT, COPY, LIST requests) X $0.005 per 1,000 PUT.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*ZuLTKs68BivPzpZs" /><figcaption>AWS Landing Zone costs the first month</figcaption></figure><h3><strong>Next Steps</strong></h3><p>In summary, Control Tower offers a good multi-environment, access management, network design, and logging strategy solution. It allows you to automate using established blueprints, having the guardrails in place to ensure that all of them are deployed following the policies and rules. It’s a good starting point and, as your company grows, you can extend the landing zone solution by adding more environments, applications, security, and rules based on your needs and requirements. In many cases, Control Tower can help you to get to the cloud quickly and safely. However, Control Tower is a basic standard solution, so it might not fit all your specific needs. In the picture below I specify some limitations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*owR7xT6AvXCe4bc32yxS5A.png" /><figcaption>AWS Control Tower limitations</figcaption></figure><p>If required by your use case, Mobiquity can help you to meet your needs, either by extending Control Tower or with a custom landing zone design for full flexibility.</p><p><a href="https://www.mobiquity.com/cs/c/?cta_guid=bcc0d168-bc9e-4fd0-961d-ce5fc15ae573&amp;placement_guid=3b21c28b-3f7a-4c2c-a32d-5f538f6c3c3f&amp;portal_id=1868764&amp;canon=https%3A%2F%2Fwww.mobiquity.com%2Finsights%2Faccelerate-your-cloud-adoption-with-aws-control-tower&amp;redirect_url=APefjpHqYCBmvpev6Ma8e54vCiqmR6lH20OI6T-MZNyCst-n6QG3S0UBEfOQ66mhGxXuSdoR0m30FOf6u8OaMEy378BOOt_58CoBYQpTgpNFjoo6ULzWFXQZ_yQNSfR5odSN_4Gla6rjlBf77TLfWp0RogoG6hQD3KJeUJIrrdlmDDaoWtZJBCMjEVUhxg6q9_iQEN32lEjx8CKWNXkf_MlwcsPIWR2UgfGClOzvPI-fFxKN_9D7YKM&amp;click=d9077cc1-3e39-4fcf-85af-3d512960a178&amp;hsutk=89766bf23779aa1160779bb47329f458&amp;signature=AAH58kH3uPAXa5TFfoX8j94GJ2NMXZyFfQ&amp;pageId=32021583362"><strong>Contact us today to learn more</strong></a></p><p>Originally published on Mobiquity website: <a href="https://www.mobiquity.com/insights/accelerate-your-cloud-adoption-with-aws-control-tower">https://www.mobiquity.com/insights/accelerate-your-cloud-adoption-with-aws-control-tower</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc59230e0879" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Pyramid Approach to AWS Data Analytics]]></title>
            <link>https://zamirajaupaj.medium.com/a-pyramid-approach-to-aws-data-analytics-503c82c8e126?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/503c82c8e126</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data-analysis]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-analytics]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Wed, 20 May 2020 07:58:27 GMT</pubDate>
            <atom:updated>2020-05-20T08:08:27.160Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yaaU638LR8CXu5ORMiuxug.jpeg" /></figure><p><strong>Overview</strong><br>Nowadays we are producing data every second. According to Domo, “<strong>Data never sleeps</strong>.” There are <a href="https://www.domo.com/learn/data-never-sleeps-5?aid=ogsm072517_1&amp;sf100871281=1">2.5 quintillion bytes of data</a> created each day; however, these values are changing rapidly day by day. I would like to emphasise it is more exponential growth than linear that brings a new challenge and new business strategies in the tech world, where the largest organizations will be engaged in finding an analytics data solution.</p><p>In my <a href="https://medium.com/brain-bites/monitoring-multi-aws-environments-serverless-application-9f0ccdca4a37">previous blog</a> I explained how to collect and transform data located in different AWS accounts and regions. Often, however, the enterprise company has data everywhere, not only in the cloud. One of the most common questions that I get from customers is:</p><p><strong>Which AWS services fit well with my requirements to improve my experience with data analytics?</strong> So here I would like to show you the data model represented in a Pyramid that may be useful and provide the results you’re seeking. I would like to call it the AWS pyramid data model as it focuses only on AWS services. AWS offers a comprehensive set of services dealing with this situation to obtain, store, and analyze data of almost all types of scales and formats. The pyramid illustrates the hierarchical relationship between data and systems. Each layer represents a different workflow of the development process.v</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnDmT8Ax5nXzQG8RNDzP5Q.png" /><figcaption>Pyramid AWS Data Analytics</figcaption></figure><p><em>Let’s examine every layer one by one starting at the bottom of the pyramid.</em></p><p><strong>#1. Collect data from the data producer</strong><br>The data can be collected from information that we share on social media, applications that you keep in your datacenter, applications that you keep in the cloud, from your device, Internet of Things (IoT), financial transactions, website events, application events, database events, etc. Every cloud provider has different strategies on how to collect and analyze the data. Let me show you how to collect data on AWS.</p><p>In general terms, data can be ingested as a stream or in batches. For example, let’s look at the streaming case using Amazon Kinesis Stream. It ingests data from streaming data producers in any format or scale, then the data can be collected by installing a Kinesis agent (AWS API, or API written by you) and the capability of data ingestion depends on how many shards you create. Depending on how much as data you have, you can dynamically increase the number of shards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*kEEKtk9hO4m3DvzDeuKS3A.png" /><figcaption><strong>Collect data from the data producer</strong></figcaption></figure><p><strong>#2. Lake formation (data storage)</strong><br>We are living in the era of Big Data, and it has become challenging to manage the vast amounts of heterogeneous data in your own datacenter as that is both labour -and cost- intensive. Many enterprise companies want to move their data to the cloud, but they are struggling to find a good, secure and cost effective solution. In AWS, there are many alternatives for storing your data in any format. For structured data, the best option is a traditional relational database, whereas noSQL databases are preferred for storing data like logs, text, images and video. Amazon s3 buckets are recommended for static files, allowing you no limit on the quantity of objects; for backup and archive you can use AWS backup.</p><p><strong>#3. Analytics and visualizations</strong><br>Data represents one of the most valuable assets for any company. <strong>Therefore it is instrumental to understand what data analysis might bring, as this analysis may influence strategic decision making. It can enable more effective and efficient work management,</strong> <strong>discover customer needs, and understand problems facing an organization.</strong> Most of the cloud providers are investing in analytics to help customers automate the process of analyzing their data in a secure way and save some costs. However most of the services are still not mature. In AWS there are a lot of services related to analytics. I have selected some of them where these services can be used based on customer needs.</p><p><a href="https://aws.amazon.com/glue/?c=a&amp;sec=srv">AWS Glue</a> is a Serverless ETL AWS Service, so you can focus only on organizing, cleaning, validating and formatting your data. You can build your workflow custom pipeline involving all the activities that you need to process your data. You could have multiple components in series or parallel running in the pipeline and monitor the status of each element. Some of the important components in data pipeline are:</p><blockquote><strong>Crawler</strong> allows you to connect to s3 buckets and extract data from your specific folder, then create a metadata table in a data catalog, and easily store into partitions. The crawler can also classify or identify various types of data based on the format like json, csv, grok, xml, and custom formats.</blockquote><blockquote><strong>Trigger</strong> allows you to trigger a specific activity at a specific time.</blockquote><blockquote><strong>Job</strong> allows you to transform and load data into your favorite formats; you can choose to transform the data using Spark or Python.</blockquote><p><a href="https://aws.amazon.com/athena/?c=a&amp;sec=srv">Amazon Athena</a> is a serverless service that allows you to analyze a variety of data types like unstructured, semi-structured and structured data using SQL. You can create complex queries for your dataset to attain the results you’re after. Data can be located in the s3 bucket or other connections like CloudWatch, Redshift, DynamoDB, MySQL, etc. To extract data from some locations, a connection must be established, which may be performed in different ways:</p><ul><li>Athena uses Amazon Glue data catalog to store metadata to connect with data stored in s3 bucket.</li><li>Athena uses a Lambda Function to connect with data stored to other services listed below.</li></ul><p>You can also use Amazon <a href="https://aws.amazon.com/redshift/?c=a&amp;sec=srv">Redshift</a>, a managed service data warehouse, which allows you to analyze structured data. Amazon Redshift also extracts the data from different sources using JDBC and ODBC. (As opposed to Athena, which is managed by AWS so you don’t have to worry about computation.) Redshift must be managed by you: the more data you have to analyze, the more compute nodes you need, so you can increase compute resources based on your needs. Amazon Redshift allows you to query data on s3 buckets but it is very expensive. I suggest you use Athena in this case; however Redshift is the best service to use if you want to have massive parallel processing, better performance, and scale.</p><p><a href="https://aws.amazon.com/emr/?c=a&amp;sec=srv">Amazon EMR</a> is also a managed service in AWS, helping you to extract, transform and load data, and allowing you to analyze Big Data using Spark or Hadoop.</p><p><a href="https://aws.amazon.com/kinesis/data-analytics/">Amazon Kinesis Analytics</a> is used for real time streaming analytics, allowing you to process data using SQL so you can run queries continuously.</p><p><a href="https://aws.amazon.com/lake-formation/?c=a&amp;sec=srv&amp;loc=2">AWS Lake Formation</a> is another option. We all are concerned about data protection. You might wonder: Is my data encrypted? Are the right access controls applied? Who can access my data? Do I have activity monitoring in place in case something happens? Lake Formation is the right service to help you build a secure data lake, allowing you to manage and maintain data flows, storage, and access control in a secure way.</p><p><a href="https://aws.amazon.com/elasticsearch-service/?c=a&amp;sec=srv">Amazon Elasticsearch Service</a> is suitable for operational analytics for all types of data, including logs, metrics, application, web, database, infrastructure, etc. This service is integrated with Kibana, allowing you to create a dashboard and have a good visualization of the data that helps to perform various kinds of exploration. If you also want to have in place data visualisation with line graphs, charts, lines, numbers, maps etc. AWS has launched a service for this called, <a href="https://aws.amazon.com/quicksight/?c=a&amp;sec=srv">Amazon Quicksight</a>. This service allows you to create custom, strategic analytics, operational, or informational dashboards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*ljWeSmkQ9UCD8alOGktPnA.png" /><figcaption><strong>Analytics and visualizations</strong></figcaption></figure><p><strong>#4. Predictive analytics and machine learning (ML)</strong><br>In the modern era of the digital economy, machine learning use cases are abundant, although most current systems are not mature enough. However, many enterprises have found a high potential in cross industries like healthcare, finance, pharmaceuticals, energy, etc. Machine learning uses a combination of systems and algorithms to make predictions or decisions with minimal human intervention based on the data and automated repetitive learning. In simple terms, a machine learning system has the ability to learn from the data in input to get the results you wish. <a href="https://aws.amazon.com/sagemaker/?c=a&amp;sec=srv">Amazon SageMaker</a> is one of the most powerful machine learning services launched by AWS. Without worrying much about the underlying technology, you can focus on building, training and deploying a machine learning model. It provides you Jupyter Notebook based on the Python language. The workflow is very simple: you can get the input data from s3 bucket, and then start working on this data, creating a model validation, stimulation, and in the end deploy to get the analysis you desire. Sagemaker Ground Truth allows you to build training datasets for machine learning. It labels your content located in an s3 bucket working with human labelers.</p><p><strong>#5 Artificial intelligence (AI)</strong><br>AI is at the top of the pyramid. This is probably the biggest futuristic development so far. It has become more popular today due to increasing data volumes, advanced algorithms, improvement of computer power, and storage. AI is progressing rapidly with human-like characteristics. As humans, we are amazing at being able to transform and drive our life, but also, we want to understand how AI can be integrated within the human experience. AI gets a consistent prediction to interact with a human using their natural languages. AWS offers a variety of services, and all of them are based on deep learning technology, such as vision, speech, text, audio and video analysis, chat bots, personalization, and forecast. You can build an application and integrate them together. For example, in one of my previous blogs I outlined <a href="https://medium.com/@ZamiraJaupaj/from-text-to-speech-with-polly-and-rekognition-b0e12ee072b0">how to transform text-to-speech</a>. It is so easy to integrate those services. You can integrate as many services as you need. The most common capability of those services is to imitate intelligent human behaviour. Sometimes there is confusion about the differences between AI and ML. All machine learning in AWS is around Sagemaker. It allows machines to learn from themselves. AI, on the other hand, is able to carry out tasks similar to humans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W1QSygd3JYA4CO5z" /><figcaption><strong>Artificial intelligence (AI)</strong></figcaption></figure><p><strong>Conclusion</strong><br>Having said all this, there are many ways to analyze your data to any format and scale. A variety of services allow you to collect, transform and categorize data and make it available for machine learning. Artificial intelligence has the capability to perform tasks smartly or imitate human behaviours like understanding language, text analysis, object recognition, video and image processing. The pyramid shows the fact that data engineering constitutes the bulk part of a data science project; ML/AI only becomes viable when the fundamentals are in place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*DUqZYZuD2ALySiIB" /><figcaption>All services together</figcaption></figure><p>Previews published in <a href="https://www.mobiquity.com/insights/a-pyramid-approach-to-aws-data-analytics">https://www.mobiquity.com/resources/from-text-to-speech</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=503c82c8e126" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From text to speech with Polly and Rekognition!]]></title>
            <link>https://zamirajaupaj.medium.com/from-text-to-speech-with-polly-and-rekognition-b0e12ee072b0?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0e12ee072b0</guid>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Mon, 30 Mar 2020 13:44:01 GMT</pubDate>
            <atom:updated>2020-03-30T13:44:01.569Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>From text to speech with Polly and Rekognition!</strong></p><p>Today, Artificial Intelligence (AI) is adding another dimension to improve the quality of life in different fields.</p><blockquote>Let’s imagine a visually impaired person needs to read a label or a street sign but isn’t close enough to see it. Or what if an elderly person is at a restaurant and can’t read the menu or hear the waiter? What if you wanted to capture information from photos or slides and listen to the content later?</blockquote><p>You can easily implement an AI application without needing too much experience as a data scientist.</p><p>In this article, I’ll create a small application which can be applied in the cases above. Plus, I will add a short video guide on how to implement this application.</p><p>The diagram below explains how to extract the text from images and transform them into audio using AWS services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*ALJJyqcnrwdPJuU2" /><figcaption><em>Transform Text to Speech</em></figcaption></figure><p>Before we start, let me give an introduction to what these services are.</p><p><a href="https://aws.amazon.com/rekognition/"><strong>Amazon Rekognition</strong></a> is an AWS managed service based on deep learning technology that allows you to analyze photos and videos, recognize objects, concepts and scenes, and detects and extracts text in the images, among other features. <a href="https://aws.amazon.com/polly/"><strong>Amazon Polly</strong></a> is also based on deep learning technology, and allows you to transform the text into audio using various voices. <a href="https://aws.amazon.com/s3/"><strong>S3</strong></a> is an object storage service that allows you to save static content such as images, etc. <a href="https://aws.amazon.com/dynamodb/"><strong>DynamoDB</strong></a> is a NoSQL database service that is based on key-value and document data structures. <a href="https://aws.amazon.com/lambda/"><strong>Lambda Function</strong></a> is a serverless compute service that runs your code in response to events. <a href="https://aws.amazon.com/cognito/"><strong>Amazon Cognito</strong></a> provides a secure process of authenticating and authorizing users for mobile or web applications.</p><p><strong>Let me explain how these services are integrated together step by step:</strong></p><ol><li>The user uploads the image in S3 bucket.</li><li>Once we have the image in S3 bucket, the bucket invokes the Lambda function and sends the image’s information in input to the Lambda function. Here is an example of a JSON file when S3 invokes a Lambda function.</li></ol><pre>{<br>  &quot;Records&quot;: [<br>    {<br>      &quot;eventVersion&quot;: &quot;2.1&quot;,<br>      &quot;eventTime&quot;: &quot;2020-02-22T16:29:55.059Z&quot;,<br>      &quot;requestParameters&quot;: {<br>        &quot;sourceIPAddress&quot;: &quot;127.0.0.1&quot;<br>      },<br>      &quot;s3&quot;: {<br>        &quot;configurationId&quot;: &quot;test&quot;,<br>        &quot;object&quot;: {<br>          &quot;versionId&quot;: &quot;dGkfynsSVHYbKDgSVJHIOrjRuzLb5M11&quot;<br>          &quot;eTag&quot;: &quot;10f98ebd80631b5fc05a581f3b4c8a96&quot;,<br>          &quot;sequencer&quot;: &quot;005E5157045C9CE682&quot;,<br>          &quot;key&quot;: &quot;beautiful-life-quotes-helen-keller-1531946634.jpg&quot;,<br>          &quot;size&quot;: 53881</pre><pre>},<br>        &quot;bucket&quot;: {<br>          &quot;arn&quot;: &quot;arn:aws:s3:::extract-test-audio-zamira-demo&quot;,<br>          &quot;name&quot;: &quot;extract-test-audio-zamira-demo&quot;,<br>        }<br>       }<br>      },<br>      &quot;awsRegion&quot;: &quot;eu-west-1&quot;,<br>      &quot;eventName&quot;: &quot;ObjectCreated:Put&quot;,<br>      &quot;eventSource&quot;: &quot;aws:s3&quot;<br>    }<br>  ]<br>}</pre><blockquote>Lambda function is the core of the serverless platform in AWS. This function refers to a number of executions of code that are happening at any given time. (from AWS documentation)</blockquote><p>3. In the image below, the first execution of the Lambda invokes the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/rekognition.html#Rekognition.Client.detect_text">detect_text</a> api to detect the text from images. This function has three inputs (bucket, key, version), bucket is the bucket name when into which you’ve uploaded the images, key is the folder within the bucket, and the version is the object version. The function returns the test rekognition’s detect_text API finds in the image. This function in output return text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*1DYCRO84r9nQ5HQk" /><figcaption><strong>Detect the Text Form Images</strong></figcaption></figure><p>4. In the second execution, Lambda invokes <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/polly.html">synthesize_speech</a> api. This is the api of Amazon Polly that allows you to transform the text into audio. This function has only one input(text), this input is coming from the previous function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/0*zh-dedHHykKDrVaQ" /><figcaption><strong>Transform the text in Audio</strong></figcaption></figure><p>5. At the same time, data needs to be stored in DynamoDb, eg. the photo id and text that correspond to this particular photo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*ZjT_7AUqkbmN_J71" /><figcaption><strong>Store the photo id and text to DynamoDB</strong></figcaption></figure><p>6. The last process that Lambda executes is to store the generated audio in S3 via the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html">s3 api</a>, so it is ready to download by end-users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/0*jqNDLA9k4Z3zGSp1" /><figcaption><strong>Store the audio in S3 bucket</strong></figcaption></figure><p>This <a href="https://www.youtube.com/watch?v=glsdhx_Ym8g">video</a> is the tutorial on how you can build and implement this application step by step. You can also find the source code we use in the video in our <a href="https://github.com/zamirajaupaj/serverless-rekognition-polly">code repository</a>. You can also find the source code used in the video <a href="https://github.com/zamirajaupaj/serverless-rekognition-polly">repository</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fglsdhx_Ym8g&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dglsdhx_Ym8g&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/41068de21f046629cd661634854f96eb/href">https://medium.com/media/41068de21f046629cd661634854f96eb/href</a></iframe><p><strong>Conclusion</strong></p><p>As you can tell, it is so easy to integrate AI services in AWS. You can integrate as many services as you need, such as Amazon Cognito, Amazon Translate, and many others.</p><p>Imagine this: you are visiting another country and you don’t speak the language — I believe every one of you has been in this situation. This situation can go from difficult to easy in a matter of minutes. Take a picture with your phone, then your phone authenticates with Cognito and uploads the pictures into an S3 bucket. With this process, now you can recognize the object in the image, or detect the text in the image, and transform this text in audio, or translate the text in another language using another AI AWS service, Amazon Translate.</p><p>Using AI to connect the world is a great use case for this technology!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*fw8CjEDjjpUPWpCE" /></figure><p>Previews published in https://www.mobiquity.com/resources/from-text-to-speech</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0e12ee072b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up AWS Landing Zone with AWS Control Tower]]></title>
            <link>https://zamirajaupaj.medium.com/setting-up-aws-landing-zone-with-aws-control-tower-a746a99ca76a?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a746a99ca76a</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-services]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Mon, 10 Feb 2020 11:40:22 GMT</pubDate>
            <atom:updated>2020-02-11T19:51:35.435Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting Up Landing Zone with AWS Control Tower</h3><p>Many years ago, I was working in a company where everything had to be created from scratch. Defining the requirements, designing the Rack, configuring the firewall, routers, and storage to be able to save data, defining user management, how to create all the infrastructure, and the list went on and on. This was a definitive infrastructure that hosted all customers and then managed them. Today, the way of thinking and drawing the entire infrastructure has changed. It has become much simpler thanks to flexibility and automation tools, but many of the requirements are the same.</p><p>When we start thinking about AWS and what customers want to do on AWS, we need to first think about requirements to better understand what the customers need. For example, which is the right service to use? What about security, governance, and baseline? How many accounts should I create for my customer based on own use cases? How many users, groups, and what permissions should they have?</p><p>Let me introduce you to the concept of a Landing Zone and Control Tower, and why they work well together.</p><p><a href="https://docs.aws.amazon.com/controltower/latest/userguide/what-is-control-tower.html"><strong>AWS Control Tower</strong></a> is an AWS Managed Service which controls the following AWS resources:</p><blockquote>1. AWS Organizations</blockquote><blockquote>2. Identity and Access Management</blockquote><blockquote>3. Guardrails</blockquote><blockquote>4. Service Catalog</blockquote><blockquote>5. Multiple AWS accounts.</blockquote><p>Through the Service Catalog, you can create as many AWS accounts as you want and apply configured rules based on your requirements. In addition, AWS Control Tower can setup a Landing zone in easy and secure way. Or you can create your own solution based on your requirement and your own CloudFormation or Terraform stacks across AWS accounts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/0*SJFajgPbgamzBphq.png" /><figcaption>Landing Zone solution using Control Tower Service</figcaption></figure><p><strong>Master Account</strong></p><p>From the Master account, you can set up an AWS Control Tower that allows you to create:</p><ul><li>Two Organizations Units (OUs), Core Unit and Custom Unit, (AWS Login and Audit are linked with Core Unit and AWS Custom Account are linked with Custom Unit, you can create as you need them)</li><li>Guardrails –Control Tower by default create the rules that are part of the baseline and applied in each AWS Account, you can extend them as well.</li><li>AWS Service Catalog allows you to provision new AWS accounts and assign this account to your favorite Organization Unit.</li></ul><p><strong>Shared Accounts</strong></p><p>Shared Accounts includes Logging Account and Audit Account. They are not provisioned by Service Catalog, but during Control Tower set up. They share resources among all Landing Zone provisioned accounts. It could be services or tools to your organization; for example, centralized directory, Active Directory for SSO integration, perhaps some infrastructure scanning, EBS volumes, golden AMI, or a simple DNS. Cross Account IAM Role is necessary to delegate access across AWS accounts.</p><p><strong>Logging Account</strong></p><p>Control Tower create this account by default, and the purpose of having this account is to collect all the logs from different account to this one. But what kind of logs do we need to collect? The logs could include infrastructure logs, application logs, VPC logs, Security logs, Database logs, etc.. This account become the single point of truth. It should be extremely limited access in that, if you do log into the account, you may want to consider alarming in that condition. It can also be Kinesis stream with Lambda function that collects the log in real-time. For more information, I explained a centralized logging solution in a <a href="https://medium.com/brain-bites/monitoring-multi-aws-environments-serverless-application-9f0ccdca4a37">previous article.</a></p><p><strong>Audit Account</strong></p><p>This account also is set up by Control Tower and the purpose of this is to establish security. In this account is established cross-account role, reports, real-time alerts. Make sure the users, groups, applications has the right permissions</p><p><strong>Functional Account</strong></p><p>How many accounts you should create here depends on your case, and how you want to build and implement your application. To have a significant software development project is the use of multiple environments, and each environment. Below, I’ve listed four AWS accounts (Dev, QA, Pre-Prod and Production) that you should have, at the very least, in your solution. You can scale easily by adding more AWS accounts using the service catalog in the master account.</p><ul><li><strong>Dev</strong>: This environment allows developers to start deploying things, to experiment, to learn, to innovate, and to build things up. Here developers deploy their code and test any newly implemented features or bugs. Once the code is considered stable, they start builds and deploy them to other accounts.</li><li><strong>Pre-Prod</strong>: This is where Quality Assurance (QA) is performed. Testers access the staging environment and ensure that the application works as it should, in this environment they run different tests to detect bugs and ensure that application is ready to deploy in production.</li><li><strong>Prod</strong>: This is the most restrictive environment. It’s where the production applications are made available to end-users. We should have very limited access into the account. We should ensure the right connectivity is there, our logs are going to the logging account, and there may be stricter security rules and policies in place around them.</li><li><strong>DevOps</strong>: This account allows the DevOps team to successfully release applications across AWS accounts (in this case, dev, pre-prod, and prod). It doesn’t matter what your DevOps process looks like and what kind of tools you use, but it is important that these tools are isolated by other environments for security purposes. Services that are hosted here include continuous integration and continuous delivery.</li></ul><p><strong>Baseline Requirements:</strong></p><p>To establish a set of requirements in each AWS account, the first step in establishing a <strong>Landing Zone</strong> that is set up around these account level requirements. We’ve got the concept of accounts, and each account is truly independent from any other account within. Therefore, the baseline requirements we want should include:</p><ol><li>Enable our MFA in our root access. Make sure that’s locked away.</li><li>There are no credentials.</li><li>There’s no access keys for my root account.</li><li>Enable cloud trail which is the API log off activities within the account. It’s API is logged there.</li><li>Access across not just my region that I’m operating, but every other region within my environment. Think about my enterprise roles.</li></ol><p>What is the best way of setting up a landing zone using Control Tower or set up your own Landing Zone?</p><p>In the image below, there are some pros and cons of using a control tower versus setting up your own landing zone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/0*0IG1sU6L4RxHD9sT.png" /></figure><p><strong>Conclusions</strong></p><p>The Landing Zone solution helps customers with large organizations to automate and easily add as many accounts as needed, as many organizations as needed, and guardrails. There are pros and cons using Control Tower to set up Landing Zone, so it’s up to you to determine how you are going to set up Landing Zone based on your case and requirements.</p><p>#AWS #CloudSecury #CloudServices</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a746a99ca76a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The biggest takeaway from re:Invent 2019]]></title>
            <link>https://medium.com/brain-bites/the-biggest-takeaway-from-re-invent-2019-8e32b07e65a6?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e32b07e65a6</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[reinvent]]></category>
            <category><![CDATA[las-vegas]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Thu, 16 Jan 2020 12:56:59 GMT</pubDate>
            <atom:updated>2020-01-16T12:56:59.821Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*RA9a5TIfbGzbEkwJ.jpg" /><figcaption>Andy Jassy Keynote</figcaption></figure><p>Las Vegas is really something beyond a mere dream, famous for gambling, shopping, fine dining, entertainment, and nightlife, but I wasn’t there for all of them, I was there to be part of one of the biggest conferences, AWS’s re:Invent, which is AWS’ annual learning and collaboration conference. At the event, they announce new services, host inspiring and educational sessions, hackathons, trainings, certifications, labs, workshops, and business meetings for partners and customers. There are more than 65,000 attendees and it is a lot of fun.</p><p>AWS re:Invent 2019 is all wrapped up, having announced 77 products, features, and services in just 5 days. Let’s go deeper on the most impactful of this event.</p><p>The 14-hour flight, covering a distance of 5,342 miles, is rather normal when heading to Las Vegas from Amsterdam. As soon as I board the flight, I set my watch and my smartphone for the timezone of the destination. I then mentally assume that timezone. I was so excited until they announced a 2 hour delay. A bit of stress with the hope of not losing the next flight, I didn’t have one direct flight, so I had to stop at Minneapolis−Saint Paul International Airport. How to kill time on a flight? It depends on what you like, watching the movie/TV series, listening to music, sleeping, staring out of the window and philosophizing, or reading a book. I chose to read the book, “<strong><em>14 leadership principles behind the world’s most disruptive company.</em></strong>” If you like business, I strongly recommend it.</p><p><strong>Back to the <em>biggest takeaway from re:Invent 2019!</em></strong> I quickly did the registration at the airport and ran to the first dinner meeting organized by <strong>AWS.</strong></p><p><strong>1- Leader Diversity Scholarship</strong> organized by @<a href="https://www.linkedin.com/in/rossbarich">Ross Barich</a>. Glad to be a part of this meeting for the second time, to sit down with women leaders and hear their inspirational stories and experiences. Mainly we discussed how we can advocate and support each other and our community. It was exceptional meeting women who take initiatives to help their communities grow, especially in some countries where the technology is the last topic to discuss. The strength of these women surprised me and inspired me to think how women can drive technology, transform the business and make innovation happen. They had efforts that have the potential to impact the future of the cloud. Another highlight was discussion with @<a href="https://www.linkedin.com/in/sachathompson">SashaTomson</a> about what AWS can improve and how to support more women over the world. Then there was discussion with @Vanessa from Brazil with the idea of “women speech cross meetup over the world.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ERJ2HoifGFQ7tm1Y" /><figcaption><em>Diversity Scholarship attendees</em></figcaption></figure><p><strong>2- N</strong>owadays being part of the IT world is not difficult, but including “different” people is a great challenge. AWS and @<a href="https://cl.linkedin.com/company/project-alloy">ProjecAlloy</a> have made this challenge easy by giving diverse people the opportunity to participate in re:Invent 2019 and bring their experiences to their local AWS communities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*xEoATFcYxDXglWpB" /><figcaption><em>Meeting organized by Project Alloy</em></figcaption></figure><p><strong>3- AWS User Group Community</strong></p><p>This meeting was organized by @<a href="https://www.linkedin.com/in/rossbarich">Ross Barich</a>. who invited all the AWS User Group Leaders over the world. The Meeting was divided into two parts:</p><p>In the first part, there were several speeches by different leaders. They talked about how to grow the community in their countries by sharing their experiences.</p><p>In the second part, which was also my favorite, there was discussion about the problems that AWS organizations are facing around the world. Even with our different opinions, cultures, and experiences, and after a lot of discussions, we made some decisions on how to improve on these problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6CcH16MR9jT7Y0o8" /><figcaption><em>AWS User Group Leaders</em></figcaption></figure><p><strong>4- Announcements by Andy Jessy &amp; Werner</strong></p><p>I’m going to list only three: <strong>Outposts, Local Zone, and Wavelength</strong>. These all had to do with availability if the user can’t get the capabilities in their AWS Region. They are able to bring AWS services in their datacenter, more local to an application or user, and take advantage of 5G network. There was a lot of curiosity about how the user is going to bring the outpost in their datacenter and, if they are going to make this investment, how they can take advantage of low latency connections.</p><p><strong>5- Workshops and Sessions</strong></p><p>During the week, the sessions start from 7 am to 8 pm. They are located in different hotels, and to manage the time better this year I decided to stay only in Venetian and MGM. Here are some events that I want to emphasize, but it does not mean that those that I have not followed are less important, indeed I will see them all in streaming.</p><ul><li>The serverless journey of Shop.Lego.com by <a href="https://medium.com/u/3fdee8fcc5ef">Danilo Poccia</a> and <a href="https://medium.com/u/33d1209889d5">Sheen Brisals</a></li><li>AWS Lambda function performance tuning by <a href="https://medium.com/u/56b578970060">Alex Casalboni</a></li><li>CodeGuru Session</li><li>DeepComposer Workshop (so fun to play some music!)</li><li>DeepRacer Workshop</li><li>Machine Learning summit (different companies shared their experiences of how they are using AI in different sectors)</li></ul><p>There were also a lot of sessions about Machine Learning, AI, Serverless, etc.</p><p><strong>6- The Expo</strong></p><p><strong>The Expo</strong> at the Venetian hosted the AWS Village, Developer Lounge, sponsor booths, a content theater, a partner theater, Builders Fair, the Health and Wellness Lounge, the Jam Lounge, the We Power Tech Lounge and Startup Central. Let me highlight some of them:</p><p><strong>AWS User Groups Booth</strong> at the Expo Hall: I had a chance to present to the AWS Community, generally to all attendees, and I invited them to join their closest local groups, helped them to identify the AWS User Group, and help them with ideas on how to grow the community in their cities.</p><p><strong>We Power Tech Lounge: </strong>So happy to spend time at this booth and with Sasha to understand how the community can help to increase the equality, tolerance, and diversity, and how to support more women in tech.</p><p><strong>AWS Services Booth: </strong>I spent some time there asking the AWS experts for advice and expertise.</p><p><strong>7- Funny Moments</strong></p><p>During the week there were a lot of games, parties, and activities. AWS gets more advanced every year. re:Play is their end of week party and it was unlike any other party that I have ever been to.t is not just a concert, there is music but also an entire show, games, music, and so much fun stuff to do.</p><p>Author: <a href="https://medium.com/u/fd92b8a209bc">Zamira Jaupaj</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e32b07e65a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brain-bites/the-biggest-takeaway-from-re-invent-2019-8e32b07e65a6">The biggest takeaway from re:Invent 2019</a> was originally published in <a href="https://medium.com/brain-bites">Brain Bites</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring Serverless Applications Across Multiple AWS environments]]></title>
            <link>https://medium.com/brain-bites/monitoring-multi-aws-environments-serverless-application-9f0ccdca4a37?source=rss-fd92b8a209bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f0ccdca4a37</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Zamira Jaupaj]]></dc:creator>
            <pubDate>Thu, 19 Dec 2019 11:06:07 GMT</pubDate>
            <atom:updated>2019-12-19T11:06:07.470Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tcgPO1MX-ntDDUC3AKjT5w.jpeg" /><figcaption><em>Microservices in real world — like small stones under a moving current</em></figcaption></figure><p><em>Welcome to my first blog! </em>Let’s imagine stones in water as microservices. It seems easy to monitor one stone based on the water movements, but how easy will it be to monitor all of them as the current changes (and therefore changes your view)?</p><p>The same situation is presented when we have a simple Serverless application. Imagine an application that is non-provisioning, with no application upgrade, requiring a person to individually log in to the account, navigate CloudWatch to look for any errors or warnings. There is no need to install any agent to collect the logs or setup metrics because there is no server maintenance from you. There aren’t many metrics to worry about scalability and high availability because AWS takes care of it all behind the scenes. Seems so easy, but what happens if we have a complex serverless application running across multiple AWS environments?</p><p>In an enterprise organization, some of the most critical things are controlling and managing the logs, storing the logs in a secure way, and centralizing storage. Centralizing your logs saves time, increases the reliability of your log data, and allows you to filter for the most significant security data, critical for the auditing process and compliance in a large organization. First, let me show you a solution to centralize logging multi AWS accounts.</p><p><strong>Centralized Logging Across Multiple AWS Accounts</strong></p><p>Let’s suppose you have two AWS accounts, the first one is the AWS Application Account, which is used to host the serverless application, and the second one is the destination, AWS Monitoring Account, which is used to collect the logs that are generated by an application hosted in the Application Account.</p><p><strong>Now Let’s Get Started with Monitoring Account</strong></p><p>The diagram shows in input source logs will subscribe to stream all CloudWatch logs to a defined destination in the Monitoring Account via subscription filters. In the destination Monitoring Account, a Kinesis Data Stream is created to receive streamed log data and a log destination is created to facilitate remote streaming, configured to use the Amazon Kinesis Data Firehose Stream as its target. The Amazon Kinesis Data Firehose Stream is created to deliver log data from the data stream to Amazon S3 for debugging and Elasticsearch for Analytics purposes. The delivery stream uses a generic AWS Lambda function for data validation and transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*Ii0a2f7N7oBnJ9Z9v-VZzQ.png" /><figcaption><em>Real-Time log streaming with Kinesis Firehose to diverse endpoints</em></figcaption></figure><p><strong>CloudWatch Destination (Input Logs)</strong></p><p>This service enables you to subscribe that resource to a stream of log events. All the logs that the Application Account put in CloudWatch destination are shifted in Kinesis Firehose.</p><p>The code below shows you how you can define in Terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c0a1c5d51e5a68b161ed88930ea322e/href">https://medium.com/media/7c0a1c5d51e5a68b161ed88930ea322e/href</a></iframe><p><strong>AWS Kinesis Firehose</strong></p><p>Kinesis Firehose allows you to ingest data from different data producers, running a Kinesis agent on a machine, with your own custom application that does this, or CloudWatch destination in the case of serverless application as explained above (refer to CloudWatch Destination above). Amazon Kinesis Firehose is the easiest way to reliably load streaming into different destination end points to accomplish different use-cases in parallel.</p><p>You can also set up rules on how to put the data in destination, buffering data in input by time and by size (e.g., push data in Elasticsearch every 5 minutes and every 100 megabytes). In AWS, there are two services (Kinesis Firehose and Kinesis Streams) that can ingest data in real-time and deliver this data to destinations. To simplify the choices, I will explain some of the differences between the two services to better understand the use-case they fit.</p><p><strong>Kinesis Stream VS Kinesis Firehose</strong></p><p>There are three things to keep in mind when using Kinesis for streaming data: (ingest data, transform/validation and delivering in destination). The way Kinesis Streams and Firehose ingest data is quite similar but delivering the data to the destination has differences:</p><p>Kinesis Stream requires custom code to take the data from the stream and apply the business logic to it; so the stream doesn’t care if this data was used in destination by the customer. Once the stream buffer windows expire (in 24 hours by default), the data is gone from the buffer. Kinesis Stream doesn’t have a manage consumer to take the data on the stream and put the data in the destination.</p><p>Kinesis Firehose, on the other hand, ingests data automatically so you don’t need to write code or manage this. Firehose also allows you to transform/validate data before delivering in the destination.</p><p>So, what service you should use? It depends on your use case. Personally, I prefer Kinesis Firehose as I don’t want to write code, so I want all of this is done automatically by Kinesis.</p><p>Here is example code of Kinesis Firehose using Terraform syntax:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e08519f8f47bc11d02cea0e86f658ba2/href">https://medium.com/media/e08519f8f47bc11d02cea0e86f658ba2/href</a></iframe><p>The code below shows you Kinesis Stream resources using Terraform syntax:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/02fd1e9db44d2ef46b28372f3684578c/href">https://medium.com/media/02fd1e9db44d2ef46b28372f3684578c/href</a></iframe><p><strong>Lambda Function(Validation/Transformation)</strong></p><p>After Kinesis ingestion logs, we need to validate/transform these logs before delivering to the destination. The right service to do that is Lambda Function. Lambda is a serverless service and allows you to run the code without provisioning or managing servers. Firehose trigger Lambda sending the event in input. This function does transformation and validation on that data, which is residing in the Kinesis Firehose and changes it before it’s written into destination. You could actually identify those data components that are not needed in your requirements. Maybe the transformation failed, and you could put it into an S3 bucket if you want to debug later. Lambda function python code shows your log transformation and logs validation before delivering in the destination. Lambda decodes the data attribute records (The <em>data</em> attribute in the Kinesis record is <strong>Base64</strong> encoded and compressed in <strong>gzip</strong> format) and writes the back to Kinesis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bc1725a2fbcf1b501963f936a022116/href">https://medium.com/media/2bc1725a2fbcf1b501963f936a022116/href</a></iframe><p><strong>S3 Bucket(Backup Logs)</strong></p><p>S3 is a managed storage service that provides persistent storage, and can be used to provide backup of the logs for debugging later. Define log retention requirements and lifecycle policies early on, and plan to move log files to cost-efficient storage locations as soon as practical.</p><p>Below code shows you how to define an S3 bucket using Terraform:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f04bbf0d437270b406303881a2d83e9c/href">https://medium.com/media/f04bbf0d437270b406303881a2d83e9c/href</a></iframe><p><strong>Elasticsearch(Destination Logs)</strong></p><p>After validation and transformation of data, we have to shift them to an endpoint for analytics purposes. Elasticsearch is a popular open-source solution for both log analytics and full-text search. Amazon offers you a fully managed solution around that. It takes care of the heavy lifting around deploying, scaling, and Kibana is enabled by default, so you don’t need to install. You can focus on your logs analytics solution, visualizing data using Kibana, and you can query programmatically if you want. Elasticsearch is easy to use, highly scalable, highly available, and automatic failover recovery.</p><p>The below code shows the configuration of Elasticsearch using Terraform syntax:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf2c22e98ed2abea7337497b8fe6eede/href">https://medium.com/media/cf2c22e98ed2abea7337497b8fe6eede/href</a></iframe><blockquote><strong><em>“You can build a data stream, aggregate data, processing streaming data and store and query data. Loading continuously to the destination you choose and processing the data in real time, whatever kind of data will be such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications”</em></strong></blockquote><p><strong>Now Let’s Get Started with application Account</strong></p><p>Let’s assume you have a sample “set time” application in AWS that consists of an API Gateway endpoint and backend AWS Lambda function. Both of these services generate logs in Amazon CloudWatch. The following diagram illustrates the relationship between the frontend and backend layer. In this case, API gateway invokes the Lambda function, IAM Lambda Role, and IAM API Role, allowing us to delegate access to these services to write the event logs in CloudWatch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*-u_JnlSmHMfM3QzDVIuwPw.png" /><figcaption><em>Serverless Application in Monitoring Account</em></figcaption></figure><p><strong>AWS Serverless Application Model (SAM)</strong></p><p>To make deployments easy and fast, I will use AWS SAM, a framework that allows you to build serverless applications on AWS. You can define a resource, outputs, etc. AWS SAM is also an extension of AWS CloudFormation. You get the reliable deployment capabilities of AWS CloudFormation, and you can define resources by using AWS CloudFormation in your AWS SAM template.</p><p>The following code shows you how to define AWS Lambda and API gateway in AWS SAM:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a149b02e1e21a1bf09adb165ebd0bd12/href">https://medium.com/media/a149b02e1e21a1bf09adb165ebd0bd12/href</a></iframe><p>Here is an example of my application “set time” written in NodeJS.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f633133bad51a8410ee83d7796bc085/href">https://medium.com/media/5f633133bad51a8410ee83d7796bc085/href</a></iframe><p>You can deploy easily from your local terminal using this command:</p><pre>sam package -- template-file template.yaml -- s3-bucket BucketName -- output-template-file packaged.yaml -- profile profileName -- region AwsRegion</pre><pre>sam deploy --template-file packaged.yaml -- stack-name myStack --capabilities CAPABILITY_IAM --profile profileName-- region AwsRegion</pre><p><strong>CloudWatch Log Subscription</strong><br>Once you have collected the application logs from the Application Account in Amazon CloudWatch, AWS CloudWatch log subscriptions allow you to deliver the events to an Amazon Kinesis Stream located in the destination Monitoring Account.</p><p>The following code shows you how to define CloudWatch Log Subscription using Terraform syntax:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/655aa4bdfef220fd4f28b40b899b45e6/href">https://medium.com/media/655aa4bdfef220fd4f28b40b899b45e6/href</a></iframe><blockquote><strong>“Cloudwatch acts as the central log management for your applications running on AWS. You can send application logs to Cloudwatch and see how easy it is to move these logs to the Monitoring Account. No matter how many applications or AWS accounts you have, in one minute you can move all the logs to the Monitoring Account.”</strong></blockquote><p><strong>Conclusions</strong></p><p>This is a Cloud-Native centralized logging solution that enables you to collect, analyze, and display logs on AWS across multiple accounts, leveraging a combination of serverless services (enables you to build modern applications with increased agility and lower total cost of ownership) and managed services (enables you to quickly and easily deploy your cloud infrastructure). Leveraging AWS Service ElasticSearch for log analytics that streamed logged data points in a scalable way, with custom business logic. The following image shows you the full centralize logging cross two AWS Account, Application Account and Monitoring Account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*DiBlfRq1clgLxTxUjKuJ8g.png" /><figcaption>Centralized logging across two AWS Accounts</figcaption></figure><p>Once the Monitoring account is set up, you can easily add as many accounts as needed, or deploy logs in different destinations. This solution can easily be adapted with some third-party tools (Splunk, Datadog, SumoLogic, there are some additional costs but you can create or use your own custom analytics solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIN-bB8dQy3XWObjTiKlEQ.png" /><figcaption>Centralized logging across multiple AWS Accounts</figcaption></figure><p>You can visit my GitHub repo to download the templates and scripts for this solution here: <a href="https://github.com/zamirajaupaj/central-logging-multi-account-environments">Monitoring Solution Account GitHub</a> &amp; <a href="https://github.com/zamirajaupaj/serveless-application-sam">Solution Application Account Github</a>.</p><p>#aws #serverless</p><p><strong>Author:</strong> Zamira Jaupaj</p><p><em>Feel free to share and or drop a comment below or send me comments in twitter @</em><a href="https://medium.com/u/fd92b8a209bc"><em>z</em></a><em>amirajaupaj</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f0ccdca4a37" width="1" height="1" alt=""><hr><p><a href="https://medium.com/brain-bites/monitoring-multi-aws-environments-serverless-application-9f0ccdca4a37">Monitoring Serverless Applications Across Multiple AWS environments</a> was originally published in <a href="https://medium.com/brain-bites">Brain Bites</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>