<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ava Chen on Medium]]></title>
        <description><![CDATA[Stories by Ava Chen on Medium]]></description>
        <link>https://medium.com/@avachen2005?source=rss-a252623143d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c5cA5nvm0mL9STGCMOlKug.jpeg</url>
            <title>Stories by Ava Chen on Medium</title>
            <link>https://medium.com/@avachen2005?source=rss-a252623143d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:26:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avachen2005/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Before We Went Live]]></title>
            <link>https://avachen2005.medium.com/before-we-went-live-de5e86751288?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/de5e86751288</guid>
            <category><![CDATA[software-release]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Sun, 22 Jun 2025 14:22:14 GMT</pubDate>
            <atom:updated>2025-06-22T14:22:14.244Z</atom:updated>
            <content:encoded><![CDATA[<p>In the year Barack Obama was first elected President of the United States, I was part of a team responsible for the election coverage page. For the first time in the U.S. history, an African American was nominated as a major party candidate for presidential election. Around the same time, iPad was starting to gain popularity in the market. For those on the move during election and still wanted to stay connected, iPad is a smart choice.</p><p>In order to serve a nation waiting anxiously for the result of the election night, our news platform had to deliver on every front: real-time updates from professional reporters, open forums for public discussion, live polling data, streaming video, and a massive interactive map that allowed people to track results by location. Behind the scenes, ten teams across two time zones in three locations worked for nearly a year to build something that could carry the weight of history. On election night, traffic surged — we have generated the highest simultaneous video streaming load in U.S. internet history at the time — and we held the line. We made it through the night.</p><p>It was not by luck but through careful calculations, countless rehearsals, and strategic decisions made months in advance. The pressure was immense. But the system held. And so did we.</p><p>Before we went live, there was a long list of things that had to be carefully planned. First, we had to define exactly what modules would appear on the landing page. Since we expected most users to visit us through the page, we mapped out every module and its expected traffic pattern. Each module had its own caching strategy, tailored to the nature of its content.</p><p>One module featured real-time updates from on-site reporters. During the day, while voting was still ongoing, this stream could be lightly cached — a lukewarm cache strategy, in a sense. But once the polls closed and results started pouring in, we expected an intense surge in news updates. That’s where things got tricky.</p><p>It’s tempting to build a dynamic caching system that adapts to traffic spikes in real time. But the kind of “smart” cache required far more effort to validate and was prone to subtle and high-impact errors. Instead, we focused on a simpler approach.</p><p>Consider this: a 30-second cache window, and assuming zero cache misses, for a single server that is capable of 20 RPS could handle up to 600 requests per 30 seconds. The real reason we planned the cache is to increase the server capacity when it is necessary. Before the poll closed, traffic was relatively light. RPS obviously was not our bottleneck — user experience was. So instead of chasing a higher throughput, we need more updated news to be distributed to the users. We shortened the cache window from 30 seconds to 5 seconds to make sure users didn’t feel like the news stream was lagging behind real events instead to blindly plan a dynamic cache strategy.</p><p>After finalizing our cache strategy, the next critical step was to estimate how many users we could serve simultaneously. And more importantly — what would happen if we failed?</p><p>As mentioned earlier, we assumed most users would enter through a landing page, which included seven modules. Among them was one video module that auto-played a live stream from a news station — a consistent source of bandwidth pressure.</p><p>Other modules updated at varying intervals:</p><ul><li>A countdown module, also refreshing every <strong>30 seconds</strong>, to fetch the latest polling status</li><li>A live news report module, refreshing every <strong>5 seconds</strong></li><li>A Twitter feed module, refreshing every <strong>30 seconds</strong></li><li>A news list module that pulled in updates from partner news outlets, also on a <strong>30-second interval</strong></li><li>A footer module that serves only static data.</li><li>A header module that allow users to login and share the page</li></ul><p>Each module added weight to the system in different ways. High-frequency updates like the live news feed demanded tight caching and careful tuning. Modules refreshing every 30 seconds gave us more breathing room, but still contributed significantly under scale.</p><p>These refresh intervals weren’t arbitrary — they were carefully planned to balance user experience with system sustainability. With this setup, we could estimate total RPS based on active sessions and refresh frequency, and plan server capacity accordingly.</p><p>After identifying the seven modules on the landing page, the next step was to determine how many users we could serve with one API server and one web server. We factored in browser behavior specifically the number of concurrent API requests a typical browser would make at the time.</p><p>Most browsers limited concurrent connections to the same domain to around <strong>3 simultaneous requests</strong>. Even if all modules fired off requests, only three would be processed at once; the rest was queued until one finished.</p><p>To simulate real-world behavior, we modeled API calls as loading in <strong>batches of three </strong>sequentially. As soon as one batch completed, the next set of module requests would begin. This behavior gave us a more realistic view of RPS (requests per second) during peak usage.</p><p>This approach allowed us to estimate how many active users a single server could support without exceeding its processing limit — considering both cache hit ratios and the latency introduced by browser-level concurrency throttling.</p><p>We had to start with the end in mind. What were we trying to protect? A seamless experience for millions of users. A platform that wouldn’t collapse under peak traffic. A night when failure wasn’t an option. That’s why we ran stress tests — not just to see how many requests our servers could handle, but to surface the hidden cracks before they broke under real pressure.</p><p>Stress testing wasn’t just a technical task; it was our way of rehearsing for impact. We wanted to simulate chaos before chaos came for real — so that when election night arrived, we would not guess. We knew.</p><p>To begin, we started testing from the web server side rather than the API server. Our product managers provided us with an estimated traffic number based on the historical data — but these weren’t just static figures. They represented probabilities.</p><p>The most likely projection was <strong>4 million visitors in a single day</strong>. We didn’t jump straight to the number. We modeled a gradual growing in traffic to observe how our system responded at every stage. We ran these stress tests repeatedly — pushing, tweaking, observing — until we had confidence the platform could withstand the pressure of 4 million users in a day.</p><p>Along the way, we made critical architectural decisions. We decoupled the video streaming service from the core API layer, allowing each to scale independently. We also provisioned fallback resources that could be activated once traffic exceeded our design threshold. In extreme cases, users would still be served — just with a degraded video quality and heavier congestion. But they would not be shut out.</p><p>As it turned out, provisioning a separate video service saved the day. Within the first hour after the polls closed, video streaming traffic surged — peaking at <strong>4 million users</strong>. That single decision to decouple video from our core API infrastructure kept the system stable when it mattered most. Our site managed to functioned normally with a degraded video service throughout the day.</p><p>For many software engineers — myself included — it’s easy to see the job ends when our merge request gets approved. But the reality is, a product has a long journey ahead before it ever goes live. Shipping real software means thinking beyond the code: performance under pressure, edge cases in the wild, users we’ll never meet, and failures we can’t afford. That’s where the real work begins — and where it really counts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de5e86751288" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[To Change the Unchangeable II]]></title>
            <link>https://avachen2005.medium.com/to-change-the-unchangeable-ii-97c7890131b7?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/97c7890131b7</guid>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Mon, 21 Apr 2025 01:39:10 GMT</pubDate>
            <atom:updated>2025-04-21T01:39:10.383Z</atom:updated>
            <content:encoded><![CDATA[<p>Changes started with knowing</p><p>We started out by interviewing the entire product staff with a list of questions,</p><ol><li>What was your previous job?</li><li>What programming languages did you use?</li><li>How many years of experience do you have?</li><li>How long have you been working at this company?</li><li>Can you name one thing that should be improved immediately here in the company?</li></ol><p>The interviews took three days with two full-time resource to perform.</p><p>I started out after the interviews were conducted. Given feedback on the five questions above, a conclusion was drawn. There was less than five percent of the entire staff who is able to adapt modern technology as well as modern software development cycle. Most of them have been working in the current model for more than decades. Almost every made complaints regarding current company working model, company direction, hindrance on individual career development, as well as low pay rate and long working hours. However complaints might sound real, most of the employees would seem to have hard time to find a job else where. Unable to find a job of the same pay grade as they are now, it is simply safe for them to continue working with constant distaste of the company.</p><p>There were three types of employees in the company.</p><p>The first type, employees who have been working in the company forever. It is their first and last job. They have gone through different managements. They are generally not happy about the situation and would occasionally gather together, murmurings about rights and wrongs but do nothing to change the current situation. This is the lukewarm group. It does not hurt to keep these people since they are still functioning 80% of the time but they would occasionally bring out negative feelings of the place.</p><p>The second type, employees who were assigned to this place by a parent company. This is the management group. Air-dropped by someone higher up and respected by no one bottom up. The management is expected to bring the most profit for the holding company instead of the company itself. While there is an obvious conflict of interest in the business assignment, sugar daddy has the final say.</p><p>The third type, employees air dropped by another child company of the same parent company. They are the technical people who were assigned to bring technical direction and support. The other child company was a project-based company originated from the MIS team of the parent company. It is obvious that the third type of employees were not aware of the difference between “product” and “project”. The meaning of “done” observes a totally different level of understanding for this group of people. Obviously, it will be a huge problem for an e-commerce company when ever-changing business model could not be backed by its engineering team.</p><p>In order to help the company to speed up on modernization, we have identified the following strategies,</p><ol><li>Knowledge transfer</li><li>One sample project</li><li>Team transformation</li></ol><p>My first step is to start the learning process. We devised a series of lessons on modernized software development process. From the fundamental continuous integration/development and regular release to scrum process. The must-haves such as git-flow, merge request, unit test, and project management tools are all part of the series.</p><p>While the original team demonstrated a certain degree of openness toward new technologies, many members appeared to be in a transitional or pre-retirement phase. Their skillsets, outdated by approximately 10 to 15 years, posed significant challenges in aligning with modern development practices. Despite extensive efforts to support their upskilling, it became evident over time that they were not the right fit to lead or contribute meaningfully to the next stage of our growth. Even with the availability of modern development tools, their approach remained rigid and manual, often lacking critical thinking in the coding process.</p><p>As we are now standing at the crossroads between AI-assisted coding and traditional legacy development. However, most of the members aren’t even situated on the legacy coding side of that divide. Their understanding of software development is so far behind that they’ve missed even the foundational concepts of what was considered standard a decade ago. As we push toward modern practices empowered by AI, automation, and continuous integration, this gap is not just a slowdown — it’s a liability.</p><p>Our progress was significantly hindered by internal politics and a prevailing resistance to learning. One particularly revealing moment was when a team member openly expressed a preference for using an IDE that had peaked in popularity over 15 years ago, stating he was “too old to adapt new tools.” Faced with such a regressive mindset and unclear political support, we made the difficult decision to withdraw all strategic initiatives and offers until the organizational climate became more stable.</p><p>Unfortunately, these internal struggles were symptomatic of deeper issues. Six months after our withdrawal, the company’s IT department was acquired by another firm. The entire team was redistributed into different divisions of the acquiring company, with most marked for eventual layoff. This outcome underscores a broader problem: in an environment where political maneuvering outweighs technical merit, progress is not only delayed — it becomes unsustainable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97c7890131b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[To Change the Unchangeable I]]></title>
            <link>https://avachen2005.medium.com/to-change-the-unchangeable-i-5e7be2704e38?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e7be2704e38</guid>
            <category><![CDATA[consultancy]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Sun, 16 Jun 2024 13:12:33 GMT</pubDate>
            <atom:updated>2024-07-26T09:00:44.604Z</atom:updated>
            <content:encoded><![CDATA[<p>A Story on Service Migration</p><p>Recently, I had the opportunity to provide a consultation service for a company of a long standing business with a rather out-dated software infrastructure. The company experienced significant losses in its income over the past few years, transitioning from the top of the sellers list in the market to a symbol of obsolescence and slowly going into history. As we grace ourselves into the company, talking to different people of different titles, we realized software infrastructure is rather an outcome of a series of company-wide practices or rather mentalities, then the cause of the company-wide reduction.</p><p>It happens to many companies where we all started with just a handful of people working nights and days for its growth and the building-up of its reputation. As the business grows, organization grows more complicated with less homogeneity and more diversity. When diversity slowly moved in, the company basically faced two options. One, making everyone the same despite of their backgrounds. A top-down chain of command management style is expected to be built up in this case. Two, flatten the organization to embrace diversity and encourage its employees to challenge status quo for good reasons. There are several parameters that would lead to the final decision. It is, however, not a cut-off decision announced publicly. It is a mentality that is gradually enforced to build up a culture specifically for the company.</p><p>The company chose the first path. For years, the company passed from one CEO to the next trying to remediate its earning status as well as reinstalling its glory in the first ten year as the leading company in the market.</p><p>The company, let’s call it the company A, maintained a strict top-down chain of command culture with zero tolerance to mistakes. Top management team maintains a strict audit system to closely monitor company A’s overall condition. Audit process is integrated into every process. Also with zero tolerance policy as its culture, employees maintains a lukewarm attitude towards work since the more they tried, the more likely they got into trouble. And trouble means a death sentence.</p><p>For each layer in the company pyramid, some information is withheld, manipulated, or permanently lost. A lot of unspoken rules are enforced with no questions allowed. This is not to blame on a single person but a general communication protocol where everyone is trained with.</p><p>There were murmurings. Murmurings remain as murmurings. When asked of making changes for those who murmured, 10 out of 10 expressed interest in the need of some changes in the place. However, when a change is announced, people expressed fear, denial, bewildered, and a sense of confusion of which ultimately leads to a simple “no”.</p><p>We started out by conducting a series of interviews with the technical team of 50 including but not limited to engineers. 1/3 of the engineering team came from an outsourcing company with the general understanding where on-site engineers has roughly half under-qualified personnels. Technical leaders are responsible to compile technical solutions and monitor development schedules. Engineering managers oversees engineering development as well as personnel resource management. The head of technical team made guidelines and finalized engineering decisions.</p><p>As one of the standard procedures of consultation, we made a list of universal questions for each technical leader as well as other members of different functions in engineering department. A question of having any prior knowledge or experience in modern project management tools such as JIRA, Kanban, Asana … etc is posted during interviews. Not surprisingly, 90% of the team have bear minimum experience in using modern project management tools and have no prior knowledge or experience on scrum or agile. One of the leaders even gave wrongful explanation on agile and scrum because the source of information is a hearsay from a friend of a friend. For most of the engineering managers, only one of them expressed interest in acquiring new project management techniques with a vague concept on scrum and agile. The rest only wanted to survive each project. For engineers, he/she had no idea on the topic whatsoever and a “no change is needed in this place” comment is a generally accepted answer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e7be2704e38" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turo]]></title>
            <link>https://medium.com/investment-myth/turo-investment-a452b2e1829a?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a452b2e1829a</guid>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 03:06:58 GMT</pubDate>
            <atom:updated>2023-11-10T04:14:17.033Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gLXTsCdIZ3FqU546rvZuNQ.jpeg" /></figure><p>Turo is a car rental platform. Unlike Hertz or Avis. Turo is a platform targeting individual renters to rent out spare cars for both long term and short-term rentals. Prepaid gas, milage limitation, car insurance, and discount for trips last more than a week are all part of Turo provided services.</p><p>Turo has its limitation. It is most suitable for people who need cars for local daily business instead of a long trip. Some car owners also provided custom drop-off service which I found most appealing since I get to pick up the car right outside of the my place.</p><p>Looking around on Turo’s app, I found information that gave out a general estimation on expected return based on different areas and the car to rent for car owners.</p><p>Turo offers three protections for car owners. First, the listing is free. Owners can set up their own price, availability, and rules. Secondly, the platform provides auto insurance plan that covers different level of protection in case of mishap. Thirdly, Turo has 24/7 customer support including road side service. The above information can be retrieved from Turo website for more details.</p><p>First I need to choose a car. I picked out two cars that are most commonly seen on the platform, a 2019 Chevrolet Cruz and 2018 Honda Civic. There are many tools online that provided car depreciation rate. The one I went with provided a 21% depreciation rate for Chevrolet and 16% for Honda Civic. It may seem like 2019 Honda Civic is a better choice for now. However, 2019 Chevrolet Cruz has an estimated value of 11,000 while 2018 Honda Civic is estimated of 16,000. Hence I need to dig further into cost and gains of the car.</p><p>Next I started to look around on the general price posted for these two cars in my residence area. I found $39 to be the lowest price for both cars if rented daily and $37 for renting period longer than a week. So taken $37 into my calculation, if the car is rented out for 100 days, which is less than 1/3 of a year, I will get $3,700 from car rental. If I sell the car at the end of the year, for 2019 Chevrolet and 2018 Honda Civic estimations after depreciations we got,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*ISTJaH6mqO1kr6zJnz236g.png" /><figcaption>1-Year Depreciated Estimations</figcaption></figure><p>Now, I would think there is another number I need to touch on. Usually when I purchase a car, I do financing instead of paying in full. Hence here I would try out two different assumptions. First assumption is that I would put down $5,000 for initial payment and finance the rest while the second assumption I would pay in full upfront without financing. Here is the numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEB7f8XWiStBwlWHOli1MA.png" /><figcaption>breakdown on different cars with different payment methods</figcaption></figure><p>With all the numbers kicked in, it is obvious that with all cash upfront, Chevrolet Cruz stands out. For the best roi, if I made a purchase of a Chevrolet Cruz and rent it out for a year, I will get almost double of my initial investment. Of course, roi does not tell the whole story. If I am to pursue the best profit regardless of my initial investment, Honda Civic with all cash upfront purchase would be the best choice here.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a452b2e1829a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/investment-myth/turo-investment-a452b2e1829a">Turo</a> was originally published in <a href="https://medium.com/investment-myth">Investment Myth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Beginning]]></title>
            <link>https://medium.com/investment-myth/the-beginning-4fe04c3a5fb2?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fe04c3a5fb2</guid>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 03:03:47 GMT</pubDate>
            <atom:updated>2023-11-10T03:03:47.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GjidvPo6XkKXY7ZeqEu5pQ.jpeg" /></figure><blockquote><strong>Genesis 1:1~3</strong> In the beginning, God created heavens and the earth. But the earth became waste and emptiness, and darkness was on the surface of the deep, and the Spirit of God was brooding upon the surface of the waters. And God said, Let there be light, and there was light.</blockquote><p>It could be an overstatement but investment world was created, not by God but by people. It is created and carefully calculated by those very smart and very powerful few. Some in the government while others in the industry. Either way, the games created by the very smart and very powerful few are not easily deciphered.</p><p>As a newbie in the world of investment, I murmured from time to time. The knowledge is not forthcoming and professionals do not talk to amateurs. The only available information is rumors and faith starts out with “I believe”. There is simply no easy answer to investments. Big money and small money have positions in the investment world.</p><p>‘sounds cliche?</p><p>Those who knows do a bookkeeping on the wins and lose. I would called this group of people the amateur-professionals. The amateur-professionals pondering rationally before his/her next move. The rationality, however, may not be totally rational. It could be biased as the data collected may be biased from the beginning. After all, amateur-professionals are still a one man game. The level of knowledge and the scale of professionalism are definitely no comparison to the investment research done by Goldman Sachs’ hundred people team.</p><p>As for those who invest but do not actually know the game, hearsay plays the upper hand. It is a belief system that runs among friends, relatives, or even work places. Win rate from hearsay is not necessarily bad. Compare to those who devised a plan with careful calculations and executions before heading out to bid, hearsay sometimes gets more out of the game. Sometimes!</p><p>My first investment was made to a high return but unstable cashflow target. It was a hearsay and a lesson learnt. Until this day, I reminded myself luck out of hearsay occurs too random and is probably not my thing. A failed investment and a broken trust put a pause on the money game for a few years. Naive is not a good thing when it comes to the adult world. It is not cute. It is a simple stupidity that costs.</p><p>Eight years after my failed experience, I joined a team that works on equity market forecasting and bond market analysis on the side. I initially studied a bunch for my work. However, as it turned out to be a fun learning. I thought instead of a detailed and professional report, I would post some of my ways of thinking for these investments that I cross path with. Let the investment begins.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fe04c3a5fb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/investment-myth/the-beginning-4fe04c3a5fb2">The Beginning</a> was originally published in <a href="https://medium.com/investment-myth">Investment Myth</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use CloudFront Distribution Service and Route53 for My Website Served from S3]]></title>
            <link>https://medium.com/ephod-tech/use-cloudfront-distribution-service-and-route53-for-my-website-served-from-s3-e2a3057b6b06?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2a3057b6b06</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[website]]></category>
            <category><![CDATA[route-53]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-cloudfront]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Mon, 07 Feb 2022 13:34:11 GMT</pubDate>
            <atom:updated>2022-02-07T13:35:15.746Z</atom:updated>
            <content:encoded><![CDATA[<p>I realize there are many service provider offers good deal on full web hosting service with a pretty designer’s website template, cname purchase, SSL certificate, and a decent price lower than 10 bucks per month. However, here I have much more to think about before making just a website.</p><p>In addition to a simple static website, I might want to keep a tracking records on the incoming traffic for analytical purpose that may benefit future business direction decisions for the company. This is certainly achievable by embedding link tracking code in my javascript that can be further aggregated in a visualization tool on <a href="https://www.flurry.com/">Flurry</a> or even something like <a href="https://firebase.google.com/">Firebase</a>. This way I can still benefit from using Wordpress. But it also means more coding in order to plant link tracking code. Furthermore, in order to compile a presentable visualization on <a href="https://www.flurry.com/">Flurry</a> or <a href="https://firebase.google.com/">Firebase</a>, a careful plan needs to be drawn out prior to the deployment of the website.</p><p>The purchase of a cname can be a big thing for a company. In addition to company website hosting, a cloud service can be attached to the purchased cname. If I have an <a href="https://www.ephodtech.com/">http://www.ephodtech.com</a> that leads audience to the content of the company website, I might want to get a https://product.ephodtech.com that directs audience to a web service provided by ephodtech.com. I am pretty sure service like Wordpress would not be able to support advance web application and multiple cname relate re-routing behavior. One might argue, cname management, ssl certification, and re-routing can be hosted on different service providers but I find it more manageable if all internet related works lie with the same cloud service provider.</p><p>CloudFront is a content delivery service by Amazon which hosts many edge locations, i.e. facilities around the world to serve content. CloudFront is easy to set up and fast propagate. There are three price classes representing different propagation methods. If you choose the option to propagate to all edge locations, AWS propagates content to all edge location around the world periodically. The trick here is that propagation does not seem to happen real-time with each updates on the content. From my experience, content propagation on all edge locations usually happens within hours. AWS has servers in Asia, Europe, South America, North America, Africa, and Australia. The other two options do not propagate content to South America and Australia regions. However, content propagation does not mean increase of the price you pay. AWS does not charge by the regions the data is propagating to. CloudFront is charged by the amount of data transfer out into the internet.</p><p>When a CloudFront Distribution is created, the url CloudFront Distribution service generated is not a memorable one. Usually for commercial site usage, we want a meaningful url such as <a href="http://www.facebook.com/">www.facebook.com</a> or <a href="http://www.google.com./">www.google.com.</a> This is not within the scope of CloudFront service. In order to adapt a meaningful cname, we need some help from Route53.</p><p>Here is a tutorial to create a static website with content stored on S3, distributed thru CloudFront, and redirected through Route53. A simple static website with meaningful cname looks something like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*X3NzcN2R0D9Yw8IYkbWMsQ.png" /><figcaption>Static Website Infrastructure</figcaption></figure><p>So let’s start.</p><p>We would go thru the following steps to create our static website.</p><ol><li>Prepare a “Hello, world.” website on S3</li><li>Create a CloudFront distribution</li><li>Create Route53 Zone</li></ol><h3>Prepare a “Hello, world.” Website on S3.</h3><p>Because this is a static site, i.e. the site is aiming to present content without complicated real-time updates. A “Hello World” site is suffice. Simply create an empty text file with the name of <em>index.html</em> and type in</p><pre>&lt;html&gt;Hello World.&lt;/html&gt;</pre><p>Create a bucket on S3 named it <em>static.website</em> and upload <em>index.html </em>to the bucket.</p><h3>Create CloudFront Distribution</h3><p>Navigate to CloudFront service by type in “CloudFront” on top search textbox. Click on the button on the right hand side that says “Create a CloudFront distribution” that will redirect to CloudFront creation page. There are four sections on the page to go through,</p><ol><li>Origin</li></ol><p>Origin section defines where to find the content for the website. Since it is a static website with content stored in S3, click on textbox for <strong>original domain</strong>, S3 url on the dropdown list. There is no need to specify origin path in this case. If for any reason you have all website files stored in a folder in the bucket, this is the place to tell CloudFront where to start looking for website content. The tricky part of this section is the OAI setting. S3 by itself can be a web service. Once public access is enabled on S3 bucket, the bucket is automatically a web service. This is not what we intend to do for this tutorial. Using S3 as a web service does not grant a cname and a cache layer that prevents DDoS attack. In addition, S3 does not have world-wide distribution which may result in long response time if the requester is in different region from S3 bucket region. Hence CloudFront is recommended for website hosting. Of course if this is the scenario where a temporary website is needed with few predictable accesses, there is no need to put CloudFront in the front of S3 bucket. Because CloudFront will be used, we would set use OAI with “update bucket policy” option checked so we do not need to worry about S3 not being given enough permission for CloudFront to access. If for some reason “update bucket policy” is not available, two S3 policies are needed for this setting to work.</p><pre>{<br>    &quot;Version&quot;: &quot;2008-10-17&quot;,<br>    &quot;Id&quot;: &quot;PolicyForCloudFrontPrivateContent&quot;,<br>    &quot;Statement&quot;: [<br>        {<br>            &quot;Sid&quot;: &quot;1&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: {<br>                &quot;AWS&quot;: &quot;arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity &lt;Cloudfront Distribution ID&gt;&quot;<br>            },<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::&lt;S3 bucket name&gt;/*&quot;<br>        },<br>        {<br>            &quot;Sid&quot;: &quot;PublicReadGetObject&quot;,<br>            &quot;Effect&quot;: &quot;Allow&quot;,<br>            &quot;Principal&quot;: &quot;*&quot;,<br>            &quot;Action&quot;: &quot;s3:GetObject&quot;,<br>            &quot;Resource&quot;: &quot;arn:aws:s3:::&lt;S3 bucket name&gt;/*&quot;<br>        }<br>    ]<br>}</pre><p>2. Default cache behavior</p><p>CloudFront provides cache layer and distribution around the world. If nothing is set, CloudFront default in “*” for path pattern. This forces all traffic for the specified domain to be handled by this CloudFront distribution. Keep “Compress objects automatically” to “Yes” that allows content to be compressed. For a simple website, keep all values unchanged in “Viewer” section. However, if CloudFront distribution is created for an API service, “GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE” methods should be allowed instead of just “GET, HEAD”. If website is restricted to specific users, check “Yes” option for “Restrict viewer access”. Leave Cache key and origin requests untouched for the same reason above. Finally there is an option that allows real-time logs. Real-time logs are a bit different from standard logs. Obviously it is aiming fast delivery of the logs. To enable real-time logging, a log configuration needs to be set with corresponding kinesis data stream created ready to receive the logs. After Kinesis data stream receives data from CloudFront real-time logging, there needs to be some other services required to digest log data to make it useful. This is beyond the scope of this article so we could keep real-time log disable for now.</p><p>3. Function association</p><p>If any additional manipulation is required, we could facilitate this requirement with a lambda handler function. Lambda is light weight function but it has 15-minute execution time with a limitation on concurrent executions and computer memory resource <a href="https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/cloudfront-limits.html#limits-functions">usage</a>. We do not need any additional handling at this moment so leave function association as it is for now.</p><p>4. Settings</p><p>Settings section is where aforementioned edge location propagation settings. The best and most recommended solution would be “Use all edge location”. I could not fathom any reason why not to at this moment since propagation does not incur any additional charge. In this section we would also need to specify an “Alternate domain names” and a “Custom SSL certificate”. Alternate domain name is the cname purchased either on AWS or other vendors. As aforementioned, if your registered domain is managed on AWS, SSL certificate could be a lot easier. It took less than a day to receive a SSL certificate. To issue a SSL certificate, navigate to AWS Certificate Manager &gt; Certificates. Type in your domain name on the certificate application page. Submit the request and wait for a day to check if the certificate is issued. Once the qualified domain and SSL are all set, we would need to let CloudFront know the root page for the website. Usually it is something like index.html which is the case here. Turn on “Standard logging” by checking the textbox with a new S3 bucket name. Log bucket needs to be created prior to CloudFront distribution creation. Set off for “Cookie logging” since there is no special needs for cookie information in this scenario. Now click on “Create Distribution” button at the bottom and CloudFront distribution is all set.</p><blockquote>Because CloudFront propagates to all edge locations, updating website content on S3 does not mean your website is updated at a predictable time. If there is time is an issue here, try to “invalidate” the content by switching to “Invalidation” tab on CloudFront Distribution page. This usually propagates result within an hour or so. Furthermore, versioning on css, javascript, and html files is always a good practice in preventing showing of old content.</blockquote><h3>Create Route53 Zone</h3><p>Navigate to Route53 &gt; Hosted zones and click on “Create hosted zone” to create a zone for the registered domain. Creating hosted zone is easy, the only information AWS needs is a domain name. However a zone does not do anything for you. There are distinct types of record that helps to redirect traffic to the correct recipient. After a zone is created, create a Type “A” record which allows Route53 to redirect requests according to “Record name” to “Value”. Leaving Record name empty means this record redirects all traffic for &lt;domain&gt; to where “Value” specifies. Here we would put CloudFront distribution domain name which can be found on CloudFront distribution page on the “Details” section on “General” tab in “Value” textbox. Remember to turn “Alias” switch on so that you can easily select CloudFront distribution from a selection list. There are cases where “Alias” switch is on and CloudFront distribution domain name is no where to be found on the dropdown list. No worries, copy and paste always work in this case. Click on copy button on CloudFront distribution page for its domain name and paste over should do the work. Leave “Routing policy” to “Simple routing”. We are done setting up Route53.</p><h3>Pricing</h3><p>For data transfer within AWS Cloud, the pricing is either free or with big discount. In our case where a user makes a request to CloudFront. CloudFront then forwards request to S3. S3 sends content to CloudFront. CloudFront then sends the content back to the user. The only data transfer charged is the part where CloudFront send the content over to the original requester. Price for purchasing a cname varies according to the popularity of the domain name. Apart from domain pricing, CloudFront gives a pretty good deal as well as S3. This setting generates close to zero cost for the few months from the beginning of our website with extensibility later on when other web service that is to come in the future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2a3057b6b06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ephod-tech/use-cloudfront-distribution-service-and-route53-for-my-website-served-from-s3-e2a3057b6b06">Use CloudFront Distribution Service and Route53 for My Website Served from S3</a> was originally published in <a href="https://medium.com/ephod-tech">Ephod Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[California Childhood Memory]]></title>
            <link>https://medium.com/aveline-avatar/california-childhood-memory-aeffd2372774?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/aeffd2372774</guid>
            <category><![CDATA[california]]></category>
            <category><![CDATA[childhood-memories]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Sat, 01 Jan 2022 12:25:52 GMT</pubDate>
            <atom:updated>2022-01-01T14:56:35.075Z</atom:updated>
            <content:encoded><![CDATA[<p>The older Aveline misses her childhood. There were certain times when I was miserable, my childhood memory in Southern California replays itself. Contradictions between my imagination and the reality brought up a sense of awkwardness that is often hard to digest as the age progressed. The memory of California sunshine and summer bubbles around our apartment calmed me down. I miss the pure happiness in my childhood. There was absolutely no second thought nor hidden agenda at the time. There was only feelings of happiness chasing after bunnies and playing with roly-poly and lady bugs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YhOkvjkenXofDl6urkwd-A.png" /><figcaption>This piece was created with poster paint characterizes its blazing attributes.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aeffd2372774" width="1" height="1" alt=""><hr><p><a href="https://medium.com/aveline-avatar/california-childhood-memory-aeffd2372774">California Childhood Memory</a> was originally published in <a href="https://medium.com/aveline-avatar">Aveline Avatar</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Little Thing on on AWS Lambda, AWS SNS, and Slack]]></title>
            <link>https://medium.com/ephod-tech/step-by-step-on-lambda-deployment-27c74291fa8?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/27c74291fa8</guid>
            <category><![CDATA[aws-sns]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 10:14:31 GMT</pubDate>
            <atom:updated>2021-12-29T06:12:35.611Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*QjKNs9dEShlQ39VI.png" /><figcaption>AWS SNS</figcaption></figure><p>AWS Lambda can be used in many places. As a fully managed service, it is best fit for light-weight, short and easy task such as event trigger, s3 data filter and migration, and recently a rather interesting self-assigned task slash hobby, status reports on slack. The infrastructure is also light-weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*AvShBw-tzYKTyVfCeb4VLA.png" /><figcaption>AWS lambda connects with slack infrastructure</figcaption></figure><p>Major thing I want to achieve here,</p><p>A configuration-based notification, i.e. if there is new notification need, I do not want to create a new event, re-deploy a new lambda code. As a startup that is short of hands, I want to minimize the effort of hooking new notification line to …. let’s say, simply update a json file on s3 without endangering my existing notification line.</p><p>General goal is set, our next research topic would be Slack. An application as such definitely provides some sort of API to access its functionalities. Since Slack is popular among distinct business units worldwide, it is reasonably to believe that API is controlled by advance security settings. Advance security settings is a big word that can be translated as “complicated token scope mechanism”. Now my head started spinning just to think about the infinite scrolls of document to read before I can start typing my first line of code.</p><p>Luckily, Slack has an easy way out for my simple project here. The scenario in this case was to send a message to a slack channel when someone send a message to AWS SNS. There is nothing smart about this behavior. There is no talk back with an attitude bot. It only delivers a message as told. Hence no need of a Slack sdk to invoke a client object before proceeding on the talking part. A webhook, an api service to receive notification sent by AWS Lambda is good enough. Slack webhook document is a peach as it provides clear step-by-step instruction, (<a href="https://api.slack.com/messaging/webhooks#posting_with_webhooks">https://api.slack.com/messaging/webhooks#posting_with_webhooks</a>).</p><p>Without further ado, let’s dive into the world of AWS to set up this scheme. First of all, we need an SNS topic. AWS SNS Topic serves as an intermittent service between sender and receiver. There are few options we need to think about before creating SNS topic. There are two types of SNS topics. In this scenario, the order the message is received is not crucial, Standard typed SNS is suffice. Although display name is optional, setting up a display name makes email subscription more clear for email subscriber. Display name shows on sender field in an email.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pXwYSqm6ye-K-ICCTJ0Dxg.png" /><figcaption>SNS Topic</figcaption></figure><p>Another important section for SNS topic is access policy. Access policy controls who is allowed to send message to SNS topic. Check “Only the specified AWS accounts” under “Define who can public message to the topic” and enter your AWS account id into the field to restrict access of the SNS topic to everyone from the same AWS account. Do the same thing for “Define who can subscribe to this topic” section to apply the same restriction for subscription privilege.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MZIMX70OInQfntox8E5FAA.png" /></figure><p>When a message is post to SNS topic, a Lambda function is triggered. Lambda function is set with two purposes,</p><p>1. Decipher SNS message.</p><p>2. Forward the message to Slack webhook based a configuration file from S3.</p><p>Configuration file should look something like this,</p><pre>[<br>    {<br>        &quot;job_name&quot;: &quot;job name&quot;,<br>        &quot;webhook_url&quot;: &quot;<a href="https://hooks.slack.com/services/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">https://hooks.slack.com/services/x</a>xxxxxx&quot;<br>    }<br>]</pre><p>Here comes a tricky part. In order to read file object from s3, we need to read and process in memory. We use Python 3.8+ in this tutorial, boto3 library is what we need.</p><pre>import boto3</pre><pre>s3 = boto3.client(&#39;s3&#39;)<br>data = s3.get_object(Bucket=&#39;xxx&#39;, Key=&#39;xxx/config.json&#39;)</pre><p>“Body” in data object holds the content of configuration file, we can access content of the configuration file as such,</p><pre>import json</pre><pre>configuration = json.loads(data[&quot;Body&quot;].read())</pre><p>After successfully loading json configuration file, we want to loop through and find a match for sent message. Say if someone post a message like this,</p><pre>import boto3<br>import json</pre><pre>sns = boto3.client(&#39;sns&#39;)<br>message = {<br>    &#39;default&#39;: {<br>        &#39;job_name&#39;: &#39;job name&#39;,<br>        &#39;message&#39;: &#39;{&quot;text&quot;:&quot;success&quot;}&#39;<br>    }<br>}</pre><pre>sns.publish(<br>    TopicArn=&#39;xxxxxxx&#39;,<br>    MessageStructure=&#39;json&#39;,<br>    Message=json.dumps({&#39;default&#39;: json.dumps(message)}),<br>    Subject=&#39;xxx is downloaded&#39;<br>)</pre><p>We want our slack to show like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*KwYgvleO4hPVyGLScIddNQ.png" /></figure><p>Lambda code would generally look as such,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdf6c1d02bca2343283723c5253b0fc8/href">https://medium.com/media/bdf6c1d02bca2343283723c5253b0fc8/href</a></iframe><p>Another challenge we face here is the “request” module that is not provided in pre-compile Lambda environment, i.e. we need to wrap up “request” library along with Lambda python file for Lambda to work. We have the choice of using zipped python package or a container image. In this tutorial, we choose to zip up python module along with our lambda code.</p><p>Creating a python package is as easy as</p><pre>pip install --target ./package requests==2.22.0</pre><p>Two things to remember before uploading our zip file to AWS,</p><ol><li>AWS Lambda uses POSIX file system which only means file needs to be set up with proper permission before zipping them up. Using these two commands to update file permission accordingly.</li></ol><pre>chmod 644 $(find . -type f)<br>chmod 755 $(find . -type d)</pre><p>2. Zip file structure is important. All module folder should be in the root of zip file as well as lambda python file. File structure for zip folder should look like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*JsTiAaZmVPPrETuw_eqhgQ.png" /><figcaption>zip folder structure</figcaption></figure><p>Now that we are all set on Lambda, we can place Lambda zip file on s3 and create a Lambda function using aws cli that points to s3 location.</p><p>We can upload lambda python file and config.json using aws cli as such,</p><pre>aws s3 cp lambda.zip s3://xxxx/<br>aws s3 cp config.json s3://xxxx/</pre><p>And create a lambda function using cli,</p><pre>aws lambda create-function \<br>--function-name &quot;<strong>Lambda Function Name</strong>&quot; \<br>--runtime &quot;python3.8&quot; \<br>--code &#39;{&quot;S3Bucket&quot;:&quot;<strong>Bucket</strong>&quot;, &quot;S3Key&quot;:&quot;<strong>zip file S3 key</strong>&quot;}&#39; \<br>--role &quot;<strong>role arn</strong>&quot; \<br>--handler lambda.handler \<br>--region &quot;ap-northeast-1&quot; \<br>--environment file://lambda_environment.json \<br>--publish \<br>--output &quot;json&quot; &gt; log.json</pre><p>Our Lambda function is ready on AWS now, we can wrap up this tutorial by creating a subscription for the topic. Each time a new notification requirement is initiated, configuration file modification is all we need to perform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*FkFEm1J3dbXxoDtVibuiSg.png" /><figcaption>SNS subscription creation</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27c74291fa8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ephod-tech/step-by-step-on-lambda-deployment-27c74291fa8">Little Thing on on AWS Lambda, AWS SNS, and Slack</a> was originally published in <a href="https://medium.com/ephod-tech">Ephod Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Glue — Develop with Jupyter Notebook]]></title>
            <link>https://medium.com/ephod-tech/aws-glue-develop-with-jupyter-notebook-5bf54752eccc?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bf54752eccc</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Mon, 26 Oct 2020 14:51:03 GMT</pubDate>
            <atom:updated>2020-10-26T14:51:54.729Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Glue — Develop with Jupyter Notebook</h3><h4>We Code In Style</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*JiunKH3NrV0tmyYa" /></figure><p>There are multiple ways to develop on Glue, we will introduce Jupyter Notebook as it is widely used by data scientist these days. Please note that AWS provides Jupyter Notebook in corporation of multiple data processing services to save its users from software installation and possible limitation on local computation resource. Although AWS recommended Zeppline for For more information, you could find tutorial <a href="https://docs.aws.amazon.com/glue/latest/dg/dev-endpoint-tutorial-EC2-notebook.html">here</a>. Keep in mind that when you use local Jupyter Notebook connects with Dev endpoint to perform data computation, you are saving cost on EC2 used to serve Jupyter Notebook on AWS. Dev endpoint charges when it is on as the base of development endpoint is EC2.</p><h3>1. Some tips on Jupyter Notebook</h3><p>Let’s start with Jupyter Notebook Installation. IntelliJ IDEA has clear details on Jupyter Notebook installation. Click <a href="https://jupyter.readthedocs.io/en/latest/install.html">here</a> for Jypyter Notebook installation instructions. Jupyter Notebook is powerful and by far has the most user-friendly interface on the market for data program debugging and development. Package installation and environment variable configuration can perform right on Jupyter Notebook interface. AWS requires AWS_SECRET_KEY_ID, AWS_SECRET_ACCESS_KEY, and REGION environment variables to be set before we get to test with AWS services locally, environment variables can be set on Jupyter Notebook in the following ways,</p><pre>env AWS_REGION={AWS_REGION} AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY} jupyter notebook</pre><p>Jupyter Notebook allows you to develop on python, spark, or R. Jupyter notebook file extension is .ipynb. Keep in mind that this is not to be compiled directly with python. If you want to export python file directly from .ipynb file, this can be achieved by performing the following command on Jupyter Notebook interface,</p><pre>!jupyter nbconvert --to script {name of the file}.ipynb</pre><h4>2. Create a Dev endpoint</h4><p>As aways, in order to set up Dev Endpoint, we need to set up a role with proper permission policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRX9t581ipsKy6WCkrt-4g.png" /><figcaption>Policies for Glue Dev Endpoint</figcaption></figure><p>Next, guide yourself to Glue console page and locate Dev Endpoint category,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*LsxfTpJU0nJKN9usrQdSzQ.png" /><figcaption>Glue Dev endpoints catagory</figcaption></figure><p>Start creating Dev endpoint with click on <strong>add endpoint</strong> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysj2uq4J2BDnT8Zcz-nNlw.png" /><figcaption>Add Dev endpoint</figcaption></figure><p>Assign prior created role to this Dev endpoint, you could also lower down data processing unit if your test does not require huge computation resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*UcN2cjTFM4Mo2PKiv8QYAg.png" /></figure><p>Also make sure to adjust Glue version of your need. Glue version 1.0 has noticeable shorter initiation time than 0.9. If there is any additional package to be installed on Dev endpoint, make sure to facilitate this need with Python library path. Glue accepts .zip file that includes all required python libraries and perform decompression of python files when Dev endpoint is in <strong>“ready”</strong> state. After everything is set up, returning to console page, a <strong>“PROVISIONING”</strong> status is shown as below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PTQow4Gkm1UDFan1Jx6Z9w.png" /></figure><h4>3. Apache Livy</h4><p>Next, you will need to install Apache Livy for port-forwarding purpose. One might wonder, what role does port forwarding play here? Jupyter Notebook allows us to send its execution plan to Dev endpoint for data computation. There is, however, a way to examine Glue code locally without Dev endpoint which we shall introduce in another blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THhQkjyB-QFj7Eosa43ADw.png" /><figcaption>shows on Glue Dev endpoint console, AWS provides sample command for port forwarding</figcaption></figure><p>AWS is thoughtful to provide various sample command lines to access Dev endpoint like the one showing above.</p><p>Here we need to start Apache Livy server first. Initiating Apache Livy server is easy, simply identify the location where Apache Livy code is stored and execute the command line as follows,</p><pre>./bin/livy-server start</pre><p>Start Jupyter Notebook service by the following command line,</p><pre>env AWS_REGION={AWS_REGION} AWS_ACCESS_KEY_ID={AWS_ACCESS_KEY_ID} AWS_SECRET_ACCESS_KEY={AWS_SECRET_ACCESS_KEY} jupyter notebook</pre><p>Change AWS sample port-forwarding command line above to the one below,</p><pre>ssh -i &lt;private-key.pem&gt; -vnNT -L :&lt;Jupyter Notebook port number&gt;:169.254.76.1:9007 <a href="/service/mailto:glue@ec2-3-87-191-113.compute-1.amazonaws.com">glue@&lt;</a>Glue Dev Endpoint host&gt;</pre><h4>4. Verify all setups are done correctly</h4><p>We can use the following code in Jupyter Notebook to verify if connection is set up correctly,</p><pre>from awsglue.transforms import *<br>from awsglue.utils import getResolvedOptions<br>from pyspark.context import SparkContext<br>from awsglue.context import GlueContext<br>from awsglue.job import Job<br>from awsglue.dynamicframe import DynamicFrame<br>from pyspark.sql import SparkSession,SQLContext,HiveContext<br>from pyspark.sql.types import *<br>from pyspark.sql.functions import *</pre><pre>spark = SparkSession.builder.           \<br>  appName(&quot;PA1.1&quot;).  \<br>  config(&quot;spark.databricks.hive.metastore.glueCatalog.enabled&quot;, &quot;true&quot;). \<br>  enableHiveSupport(). \<br>  getOrCreate()<br>sc= spark.sparkContext<br>glueContext = GlueContext(sc)<br>logger = glueContext.get_logger()</pre><p>If Dev Endpoint is set up correctly, you should observe Jupyter Notebook presents as follows running through the blocks of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhL6ZGWJWJo628YmG1kcGg.png" /><figcaption>Jupyter Notebook should look as such if Dev Endpoint is set up correctly</figcaption></figure><p>This is only the first step into Glue development. There will be series of blogs to share some practical knowledge on AWS Glue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bf54752eccc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ephod-tech/aws-glue-develop-with-jupyter-notebook-5bf54752eccc">AWS Glue — Develop with Jupyter Notebook</a> was originally published in <a href="https://medium.com/ephod-tech">Ephod Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Little Things about AWS Glue]]></title>
            <link>https://medium.com/ephod-tech/little-things-about-aws-glue-6809d1d74a68?source=rss-a252623143d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6809d1d74a68</guid>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Ava Chen]]></dc:creator>
            <pubDate>Tue, 06 Oct 2020 07:27:46 GMT</pubDate>
            <atom:updated>2020-11-04T10:08:54.292Z</atom:updated>
            <content:encoded><![CDATA[<h4>It happened and it was not a smooth drive.</h4><figure><img alt="AWS Glue" src="https://cdn-images-1.medium.com/max/830/1*ZqHuhyKYj24gQC1naZq1yA.png" /></figure><p>This year we received a case that we were asked to support to build a data pipeline on AWS Glue. As a heavy EMR user, Glue is not a familiar concept. Through months of struggling, digging, and thinking, here is some of the observations.</p><p>AWS Glue was first announced in 2017. In order to fulfill what was not provided by EMR, AWS released Glue. Glue is a serverless ETL service built on the top of AWS EMR.</p><p>AWS breaks ETL service down into distinct modules that work both independently and coherently. Although AWS Glue was labeled an ETL service, it provides a full data life cycle. From Data catalog to Security settings, AWS Glue intends to provide a fully <strong>managed</strong> data service.</p><p>AWS Glue provides a list of functions to facilitate the needs of data processing at different stages. Some of the fundamental components to draw a full picture of a data pipeline including data application development.</p><figure><img alt="Fundamental Glue components to compose a full picture of data processing." src="https://cdn-images-1.medium.com/max/179/1*oqaPDregZoHEJoGtFxUJ6Q.png" /></figure><h4>Databases &amp; Tables</h4><p>Under <strong>Data catalog</strong> section, you would find databases and tables combo. AWS Glue, as a fully managed service for big data processing, databases and tables combo is not from the concept of relational databases. Both AWS Glue and EMR are built upon Hadoop. Hadoop is a distributed file system that stores and processes massive amount of data across multiple servers. It is obvious that AWS intends to separate data source from its interface. This is a decent design for big data as data is piling up with distinct criterias. However, here is the question we would ask. Are we most <strong>unlikely</strong> to access the entire dataset on a regular basis? If not, why do we need to store data in a managed storage where we pay high maintenance fee. Hence here we have tables to hold definitions of dataset which is alienated from the data source itself. Data is then accessed through pre-defined data definitions and all-time favorite SQL-like syntax when it is needed.</p><h4>Connections</h4><p>In addition to AWS file system, AWS provides one of its oldest service, S3, as a source and destination of a data process. Glue also provides easy interface to access multiple data provider services through <strong>Connections</strong> function. By setting up a connection in Glue, you are able to access data through pre-configured connection in a Glue job without hard-coding secrets and password in a connection string.</p><h4><strong>Crawlers</strong></h4><p>Crawler is a new concept in data processing. It reminds me of a scene in Star Trek series when Voyager meets aliens it needs to use different language patterns to adapt data from alien spaceship. In the good old days we data developers need to know what data to expect before data importing process begins. By using create table syntax, data schema needs to be discussed and planned beforehand. Data importing process breaks when unexpected data is delivered to our system. Now Glue is smart enough to analyze new data real-time. Crawler helps to produce a schema automatically upon a schedule or triggered event. After a crawler finishes its analysis on a dataset, the schema is transformed into a table in Glue.</p><p>Crawler, however, has two-sided stories. It is smart enough to analyze data real-time. If you have unexpected data coming into the system, this would keep data flow working properly even if we do not know the quality of incoming data. Crawler, however, is slow. It does not analyze data format base on a random sample from a dataset as we humans usually do for gigabytes of data. It reads through the entire dataset before drawing a conclusion on data format. The analytical process can be long and not worthwhile when you find most of the data type turn into string.</p><h4>Jobs</h4><p>Glue Job is the center of AWS Glue. Most of the coding is done in a job. AWS Glue job currently supports python and scala with three modes, python shell, spark, and pyspark. Python shell is a bit tricky as it took me a while to finally understand the difference between pyspark and python shell. Python shell is used for non-spark job. If you want to copy one data file from a s3 to another, creating a spark job for this scenario is clearly an over-statement. AWS Glue provides an alternative to avoid overstatement as such. A python shell job is simply a serverless computing unit to execute a python job. Python shell provides basic python libraries but you will need to zip your own python files and specify the location of python zip file for python library path setting on Glue console. It is worthwhile to note that a Glue spark job usually takes minutes to initiate before running the actual job while python shell is almost instantaneous.</p><h4>Triggers</h4><p>AWS also provides a way to kick-off a Glue job. It is called a trigger. A trigger can be an event, a scheduled time, or a manual action. Trigger is important in a workflow. It connects a series of jobs together to build up an ETL service.</p><h4>Workflows</h4><p>AWS Glue Workflows is a real breakthrough for an ETL service. A data pipeline can be as simple as loading data and as complicated as 10 distinct data processes with multiple types of data with distinct type of triggers at the same time. A workflow can implemented with a simple drag and drop on triggers, jobs, and conditions. AWS simplifies a data pipeline to a concept of triggers and jobs. Each Glue job by default return a status that is recognized by Glue. Glue Trigger can then be used to initiate next job based on the execution status of previous job.</p><h4>AWS Glue Studio</h4><p>AWS Glue Studios is a new service came out in Oct. 2020. Provided with a user friendly interface that serves similar purpose as AWS Glue Workflows, AWS Glue Studio added S3 storage in data pipeline configuration which brings more flexibility with a user-friendly Glue-job oriented dashboard.</p><h4>Dev Endpoint</h4><p>Dev Endpoint is a helpful development tool to work Glue. There are different ways to develop on Glue. However, running a spark job may take a while for each small try. If you are Jupyter Notebook type of person, Dev Endpoint will be at your service to speed up your development. Dev Endpoint is a computing service sits on the top of EMR. With support of Jupyter Notebook, development may be as comfortable as it can be.</p><h4>Notebook</h4><p>AWS integrated Jupyter Notebook as part of its big data service. Although different options on Glue development are available, Jupyer Notebook on AWS allows easy-to-access interface without any installation. Jupyter Notebook can also be installed locally on your own computer with constant connection to Glue dev endpoint. AWS also recommends Zeppelin Notebook in combination with Apache Livy and dev endpoint but personally I prefer Jupyter Notebook than Zepline.</p><p>Overall, AWS Glue provides a more convenient way to engage in data process field. Starting from dev endpoint and Jupyter Notebook, AWS Glue job development cycle should be greatly minimized. AWS Glue workflows allows users to set up both simple and complicated data pipeline in simple steps. Nowadays when manual operation is expected to be minimized, AWS Glue is surely a good tidings to big data industry.</p><p>There are downsides on AWS Glue. First of all, AWS Glue costs more than EMR. AWS charges DPU-hour (data point unit hour) for Glue service. On the latest AWS posting, Glue costs 0.44 / DPU-hour. Compare to EMR, a m5.xlarge in US-East Ohio region costs 0.192 / EC2-hour + 0.048 / EMR-hour. Glue costs about three times more than EMR in general. If price is not a big concern, Glue might be a good choice.</p><p>AWS Glue wraps Spark and Hadoop in order to position data programming into context of a data pipeline on AWS. Spark itself is not a mystery as supporting materials can be massively found online. AWS Glue versioned Spark mixed with multiple AWS services can be quite a mystery to general public. There are few traps I stepped on when working on Glue and end up using spark dataframe along. I would recommend Glue for simple data conversion such as csv to parquet format conversion but not for complicated data analysis and computation at the stage of Glue.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6809d1d74a68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ephod-tech/little-things-about-aws-glue-6809d1d74a68">Little Things about AWS Glue</a> was originally published in <a href="https://medium.com/ephod-tech">Ephod Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>