<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alysa Chan on Medium]]></title>
        <description><![CDATA[Stories by Alysa Chan on Medium]]></description>
        <link>https://medium.com/@alysachan830?source=rss-fb1b2b8c75------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*vY9pkLgQLq55RXqsSHf3-Q.jpeg</url>
            <title>Stories by Alysa Chan on Medium</title>
            <link>https://medium.com/@alysachan830?source=rss-fb1b2b8c75------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:08:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alysachan830/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What I have learned by being a cross-disciplinary developer]]></title>
            <link>https://medium.com/@alysachan830/what-i-have-learned-by-being-a-cross-disciplinary-developer-e366ad925b00?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/e366ad925b00</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[personal-growth]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sat, 02 Sep 2023 15:04:39 GMT</pubDate>
            <atom:updated>2023-09-06T15:56:09.968Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BLcvfAK9MrwAX9pAyBbzZg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ed259?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ed 259</a> on <a href="https://unsplash.com/photos/xcrI6CPkkJs?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><a href="#e796">Chinese version 中文版</a></p><h4>Introduction</h4><p>I’m a self-taught software engineer working in a startup company based in Hong Kong for around 2 years. Coming from an arts and design background, I sometimes question how much I can achieve in the software engineering field. Yet, the biggest takeaway from my learnings this year is understanding the power of cross-disciplinary, even though I’ve heard of it since I was in university. I come to believe this is not only making you become a better developer but also having higher achievements.</p><p>When I started to be a developer, I sometimes wished I was from a Computer Science background. This belief implies that “I have to make a correct and life-long decision about my future development as soon as possible if I want to succeed young”. Yet, after working for a while and reading about how great people work, I found that the belief was not valid. Oppositely, cross-disciplinary is an advantage. Great success comes from people who solve problems with extensive understanding in different domains. It’s like using “combos” in a video game which results in having unique insights and positions in the world.</p><h4>To have better predictions on trends, we need cross-disciplinary knowledge to understand the present</h4><p>Most people I admire have cross-disciplinary mindset. Although they initially chose to specialize in a particular field, they remained open to trial and error to accumulate a large amount of experience. Like many others, during high school and college, I tried to learn with a closed mindset of “choosing a field early and delving deep.” I was overly focused on achieving good grades and subconsciously avoided spending time learning seemingly unrelated domains. Since my academic performance in art and design was good, I chose design as my starting point. I thought that by delving deep early, I could gain an early advantage through compounding. I had beliefs like “learning about other fields would be a waste of time” and “I can learn them after I become a senior designer”. <strong>This premature commitment made me perform better in design than other students at that time. However, in the long run, I got trapped in an intense competition that loves chasing the trends. I didn’t know why trends appear and how innovations are developed.</strong> By overly concentrating on becoming the best “screw,” I approached everything from the perspective of a “hammer”</p><blockquote>If all you have is a hammer, everything looks like a nail</blockquote><p><strong>Now, I found that I cannot see the bigger picture of how the world operates or understand the underlying reasons behind the current trends.</strong> For example, why did AI start to rise, but virtual reality could not? Many people want to follow the trend without asking why. But in reality, a small group of people started laying the groundwork for AI development a decade ago. If I can’t understand the fundamentals of the current trends, it’s difficult to observe where the next potential opportunities may arise, let alone make early preparations.</p><p>How can we understand the core reasons behind trends? It relies on how capable we can comprehend different domains outside our expertise. It enables you to observe patterns of change and accurately identify potential opportunities. Here are the examples:</p><ul><li>Ray Dalio, the founder of Bridgewater Associates, seeks to make better market decisions by reflecting on future developments through historical analysis. He tries to uncover the reasons behind economic disasters and shifts in a nation’s fortune. He extensively studies knowledge beyond his field of expertise (business) to understand the historical evolution of China, UK, and Germany, as well as human psychology, politics, and economics. He makes more accurate business decisions with a perspective rooted in the past.</li><li>CEO of OpenAI played poker when he was in school, which taught him psychology and statistics. That was helpful for him to be an angel investor in his career.</li><li>Alan Chan, the founder of Heptabase, developed research-based note-taking software because of his curiosity about how the world works, by drawing relationships between knowledge in different fields.</li></ul><p>Great achievements come from gathering insights from domains outside our expertise. People do great things by solving complicated problems, which requires drawing insights from cross-disciplinary knowledge.</p><h4>Great product is about bringing value to users, not how good the code is written</h4><p>Also, my learning in a startup company for nearly two years has shown me the importance of this mindset. Although I focus on programming, which can be explored in great depth, I have to remind myself that it is a tool made for problem-solving. Programming is a tool that is the same as design, marketing, product development, and recruitment. They are here to create valuable products for users. If we bury ourselves in programming without understanding market demands, even if we write excellent code or have cool designs, it will not be useful because they don’t provide value to the users.<strong> To do the right things, ask the right questions, and understand core problems accurately, we need to go beyond our expertise</strong>, such as delving into economics and user psychology to gain a more accurate understanding and reliable hypotheses. Software engineering is not the value, it is used to serve the ideas of the product. A simple fix on the inquiry page can improve our customer service, but a high-tech feature that no one knows how to use serves no value to the user. To know what problems to be solved by the tool, I have to go beyond programming.</p><h4>Premature career decisions and closed mindsets hinder exploring potential and insights</h4><p>I asked myself, why I fell victim to believing coming from a different background was my weakness.</p><p>The reason is that I had a closed mindset and prematurely drew conclusions when I was too young without enough information. It hindered my explorations. Mainstream education associates subject choices with careers. Choosing subjects was seen as paving the way for future work. You study computer science to be a software engineer. You study design to be a designer. Don’t choose subjects that don’t lead to concrete career paths, such as philosophy or art. In a highly competitive society that favors specialization of labor, the safest approach for normal students is to quickly find a concrete field and deeply cultivate it. Because of the lack of trial and error, I had limited choices to think about. It was common to hear advice like “You get good grades in arts. Go for it!”. That is why I chose visual arts. I thought I would be a graphic designer for the rest of my life. However, later I felt trapped by the fact that I could only do graphics. When I participated in exhibitions, I only had confidence in creating graphic works. I wanted to explore fields beyond a flat surface. Indeed, I felt that a lot of people were sharing the same feelings and competing for opportunities in a highly competitive and noisy environment.</p><p>Why did my choice of a seemingly safe path bring me to a trap? My premature decision and closed mindset hindered me from recognizing opportunities. Luckily, I tried VJ (video jockeying) and started learning creative coding. During the pandemic, I also began exploring software development. After joining a company and working for a while, I gradually expanded my knowledge from software engineering to product design and startup operations from my daily work and reading. I became more curious about problems when I gained more knowledge besides programming, such as UIUX, data analysis, and startup operations. For example, I am curious to know the differences between the mindsets of programmers and designers.</p><p>When I was a designer:</p><ul><li>I used visuals to build brand images and attract public attention to deliver messages. I use different mediums to create my works</li></ul><p>When I am a developer:</p><ul><li>I build logic, break down problems into smaller ones when debugging, validate assumptions, and adopt rapid product iteration.</li></ul><p>I am surprised by how different people can think and act in two different fields. Thanks to the perspective shift, I realized that tools can influence people’s thinking. Overindulging in a single field can easily lead to biased thinking.</p><h4>Shape my career with “combos”</h4><p>Successful people who push the world forward don’t strive for positions in highly competitive mainstream industries. Instead, they approach it like playing a video game. This metaphor is from a podcast I listened to recently, Left Side Escalator, by Jacky Wang. <strong>Those people first choose a skill and then engage in a lot of challenges, along the way, they follow their curiosity to form their “combos”. They create their unique position and insights</strong>. You may find conflicts between fields, which pushes you to question things you used to take for granted. I had a similar experience when I switched from design to software development. I can see more clearly what I lacked as a designer before and also question why I used to design things in a certain way. Some questions may take a long time to answer, but at least I have experienced two different ways of thinking, and they are some of the skills I gained to form my “combo skills”. This set me apart from the noisy competition, similar to what Audrey Tang mentioned before:</p><blockquote>Don’t be afraid of losing at the starting line, because as long as you turn around and run in the opposite direction, you will be in first place.</blockquote><p>With the concept of combos, I learned statistics to analyze data, read books on startups to learn how to innovate, and studied UI/UX to understand user experience. The purpose of these is not to become an expert in those fields but to gain a deeper understanding of problems I am curious about.</p><h4>Take advantage of low-cost risk in youth</h4><p>The cost of making mistakes is low when you are young. As we get older, we take more responsibilities, which results in more risk to bear. Utilizing the advantage of low cost during youth allows for accumulating a large amount of trial and error experience, trying things with a low cost of failure but a high potential for success. In an interview with OpenAI CEO, Sam Altman shared that he tried different projects in his early twenties and took a gap year for more experiments, before starting OpenAI. He admitted that many of his attempts did not succeed, but he believed that the key to finding the right direction was to try a lot and honestly tell himself which projects were worth persevering. This aligns with the idea by Linus Pauling, “If you want to have good ideas, you must have many ideas.”</p><h4>Summary</h4><p>Curiosity and combo are the keys to great achievements. The education system tries to simplify the complexity of problems and gives the impression that all problems have already been answered in textbooks. Indeed, many problems are still waiting to be unearthed. Our drive of learning should not be the sense of inferiority or superiority derived from competing with others, but rather from curiosity about the future.</p><p><strong>啟發</strong></p><p>在這一年的成長中，啟發最大是跨域思考以及試錯心態。大概 2 年前開始，我從藝術設計轉職到軟體工程，我曾經因為自己不是 Computer Science 本科畢業而思考自己在軟體工程這一行業裏能走多遠，也會有懊悔過為什麼當初沒有選資工系來入行，但後來我才發覺這是優勢，我不用過早決定將來發展，而是多建立對不同領域的認知，把握年輕時低成本試錯的優勢，像打電玩一樣用 Combo 技來塑造個人定位。</p><p><strong>透過跨域思考來深入理解潮流現象</strong></p><p>我慢慢發覺自己敬佩的人大多都是有很強的跨域思考能力，他們雖然也在早期選擇了一個領域學習，但卻保持開放試錯來累積大量試錯經驗。我跟很多人一樣，高中到大學期間，嘗試用「盡早選一個領域然後深耕」的封閉心態來學習，因為很想做出成績而過份專注，導致下意識避免花時間學習跟自己看似無關的領域。當時我單純因為對視覺感興趣，而且當時成績較好而選擇了設計，以為愈早深耕就可以用複利方式來跑前一點，覺得學習其他領域的知識就是浪費時間，也會覺得不如等自己在本業站穩才去學就好。這種過早下注的心態，讓我當時看起來在設計表現比一般人好，但長久以來因為過分專注，導致自己缺乏對其他領域的認識。在過分專注在如何成為最好的「螺絲釘」時，就用「鎚子」的角度去看事情，無法宏觀去看整個世界的運作，理解目前社會趨勢的來龍去脈，例如為什麼人工智能在現在才興起？為什麼 VR 不能像 AI 一樣被廣泛使用？很多人想跟上風潮，而沒有問自己因為要解決什麼問題所以要用 AI。但事實上有一小群人十年前就已經開始佈局推動 AI 發展。如果連眼前潮流的來龍去脈都無法搞清楚，那就很難觀察到將來的下一個潛在機會在哪，更枉論提前佈局，最後只能加入後知後覺的跟風群體。要理解風潮現象的核心原因，其一就是取決於自己對本業以外的領域認知有多深多廣，找出不同領域之間如何互相影響，它們可以涉及心理、歷史、經濟學等等。有愈多跨領域的認知，就有愈透徹的理解，才可以有觀察變化規律的能力，更準確找到潛在機會。例如：</p><ul><li>橋水基金創辦人 Ray Dalio，為了讓自己做更好的市場決策，他需要從歷史中來推敲未來發展動向。他要找出為什麼經濟市場會出現災難、一個國家為什麼會由盛轉衰等。他透過大量學習他在本業（商業）以外的知識，理解中國、英文、德國等歷史演變，以及人類心理、政治、經濟學的知識之間找出原因。以古鑑今的心態去更準確做商業決定。</li><li>OpenAI 的 CEO 因為年輕時很愛玩撲克，學習到心理學與統計學的知識，對他之後成為天使投資者很有幫助。</li><li>Heptabase 創辦人 Alan 是因為前期對世界觀很有興趣而大量閱讀以及跨域研究，產生了 Heptabase 這個很好用的筆記軟體。</li></ul><p><strong>好的產品源自為顧客帶來價值，而不是產品裏漂亮的程式碼</strong></p><p>回到個人經驗，我在新創團隊待了接近 2 年，慢慢發覺這思維的重要性。我的本業（程式開發）雖然可以挖很深很深，但它本質是一個解決問題工具，就如設計行銷、產品開發、團隊人才招聘一樣是工具，它們的存在是為了幫助我們持續創造出對用戶有價值的產品。如果沒有對市場需求的認知就埋頭落手寫程式，即使寫得多好，或者我們有多酷炫的設計也沒有用，因為它們沒有為用戶提供價值。這印證了如果要做對的事，問對的問題，就需要有本業以外的知識，無法避免要從經濟、歷史、用戶心理切入了解來更準確找到核心問題，提出可靠的假設，以及知道用什麼方法來驗證成果，最後也才到技術工具。我們用它來實踐想法，這才是它最能體現價值的地方。技術不是愈深就愈有價值，修一個在聯絡方法頁面簡單的 bug，可以提高客戶服務體驗，相反，如果花很多時間去開發一個很複雜但沒有人懂得用的功能，它就沒有價值。</p><p><strong>過早決定職涯方向阻礙個人發展</strong></p><p>另一個啟發是我需要透過累積大量試錯經驗來找到潛在機會以及個人長處，因為前期過早下定論的封閉心態，阻礙了發掘個人潛能。因為不夠了解自己以及社會的機遇，只好加入主流行業跟人互相競爭。無可否認有一小部分人很幸運地很早就找到有極高市場需求的志業。但在主流體制下，在不鼓勵犯錯的教育環境，試錯的經驗並不多，對教育系統來就，犯錯就要受罰，但不鼓勵試錯這件事本身就是違背學習目的。我以前唸書時的主流價值觀是習慣把選科與職業掛勾，選科是為了日後工作鋪路，選電腦科學就是要成為軟體工程師，選設計就是要成為設計師。不建議選沒有具體出路的科目，例如哲學、藝術這類學科，加上大家習慣用文科理科的心態去切割知識領域，並意圖簡化真實世界問題的複雜性。在高度競爭及分工的環境下，最安全的做法是盡快找一個有具體出路，而且在別人跟你說「你做這個好像不錯的領域」來深耕，就像坊間很常說的「贏在起跑線」。我高中時選科因為閱歷不夠，以為自己藝術成績好像不錯而選擇了視覺藝術，走平面設計或藝術這條路，但後來才發覺我發現很不喜歡思維被平面的局限，參加展覽時我也只有信心用平面來創作，很厭倦自己只能做以主觀視覺主導的事。當時的我想尋找視覺以外領域，感覺大家都在一個高度競爭而很吵的環境下爭取機會。</p><p>為什麼當時我看似走上最安全的路，但最後卻感覺被困在一個高度競爭的環境裏？因為試錯經驗不足，而且對未來職業想像很局限，使自己更難認知到其他潛在機會。機緣巧合下，我接觸到 VJ、學習 Creative coding，在疫情下我也開始接觸軟體開發，後來進了公司後工作一段時間後，又從軟體工程慢慢接觸到產品設計、數據分析、以及Startup 運作，我慢慢對很多範疇很好奇，發現可以開始會從設計行銷、軟體技術、產品開發、UIUX 等理解一個問題，而不再只是從設計或者軟體開發來思考時，發現問題變得更立體、更讓人好奇。回過頭來想，單單是平面設計與程式開發的思維、工作模式的差異已經令我覺得很詫異。</p><p>當我是一個設計師：</p><ul><li>如何利用視覺語言去傳達訊息，學習不同藝術媒介，打磨設計細節，吸引到大眾的眼球，並在極短時間內傳遞訊息</li></ul><p>當我是一個開發者：</p><ul><li>建立邏輯，除錯時學習把問題拆分成一個個小問題，審視假設是否合理。要有產品快速迭代思維，追求效能穩定但同時不花過多時間雕琢程式</li></ul><p>這好像兩個不同世界的運作模式，令我發現工具會影響人的思維，過份沉迷在單一領域會容易造成偏面思維。</p><p><strong>用 Combo 技能來探索自己的發展方向</strong></p><p>除了切換領域，也因為閱讀，我發現能夠推動世界前進的人，很多都不是往一個高度競爭的主流裏爭職位，反而像玩遊戲一樣，先選一樣技能，然後大量打怪，沿途組合感興趣或有天分的 Combo 技能，久而久之創造屬於自己的定位。Combo 這個比喻來自我有一次在電扶梯走左邊 with Jacky 的 podcast。因為組合技之間的花火中更易讓你質疑以前覺得習以為常的事，並找到新的機會，就像在兩個程式開發與設計藝術世界的思考模式切換時，我會更清楚看到以前的我缺乏了什麼，甚至質疑為什麼我以前設計要這樣做。這些問題可能要很長時間才有答案，但至少我曾體驗過兩種思維方式，它們都是我 Combo 技的其中一些技能點。即使現在沒有即時看到連結，但我開始相信在某些時候它們會起重要作用，像是唐鳳之前主張：</p><blockquote>別怕輸在起跑點，因為你只要轉身往反方向跑，就是第一名了</blockquote><p>這指的是只要每個人都向自己想跑的方向跑，每個人都是那條跑道的第一名。在組合技的想法下，我為了知道如何分析數據去學統計學、為學習如何創新來閱讀 Startup 的書、為了解用戶體驗而學習 UIUX。這些學習的本意並不是為了讓我成為某某領域的專才，而是讓我更深入理解問題。</p><p><strong>利用年輕時低風險的優勢來大量試錯</strong></p><p>除了知識的擴展，大量親身試錯的經驗也很重要年輕時的試錯成本是比較低，隨著人愈大，責任愈大，犯錯成本自然愈高，利用年輕時低成本優勢可以來大量累積試錯經驗，嘗試失去成本下限少成功上限很高的事。Open AI CEO 在一次分享中提過，他在決心發展 OpenAI 前，20多歲時就大量嘗試不同的項目，並曾經花一年 gap year 的時間來大量嘗試及學習，他坦言那時候很多的嘗試到最後也沒有成功，但他認為找到方向的要訣是大量試錯，並誠實告訴自己什麼項目才是值得繼續堅持。這跟 Linus Pauling 提及的 “If you want to have good ideas, you must have many ideas” 想法相同。</p><p><strong>總結</strong></p><p>因為這些啟發，我開始認同好奇心才會帶人走更遠。教育系統趨向簡化解難思維，讓人覺得所有問題都會有答案，甚至以為所有問題都已經在教科書上。但現實是很多問題都還等待被發掘。推動自己的不應該是與人競爭而帶來的自卑或優越感，而是對未來的好奇心。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e366ad925b00" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Moovup recommends jobs with Amazon Personalize]]></title>
            <link>https://medium.com/moovup/how-moovup-recommends-jobs-with-amazon-personalize-3f1c987ad28a?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f1c987ad28a</guid>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sat, 11 Mar 2023 09:20:08 GMT</pubDate>
            <atom:updated>2023-03-11T17:43:10.543Z</atom:updated>
            <content:encoded><![CDATA[<p>Moovup is an online platform for front-line jobs, such as retail, restaurant, and logistics jobs. To provide jobs that match user’s interests, a personalized job searching experience is the key. For example, if a part-time waiter is searching for jobs in Moovup, we recommend jobs related to food and beverage to him. These personalized jobs are shown in:</p><ul><li>The “Recommended jobs” (推介好工) section on the homepage</li><li>Job searching page</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v3YFL3L6BMTwqlWasHBJew.png" /><figcaption>Recommended jobs on <a href="https://moovup.com/hk/">Moovup’s homepage</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVNmIPoOQjixmeMAcqwvMw.png" /><figcaption>Personalized job ads in the job searching page</figcaption></figure><p>To implement this feature, we integrated Amazon Personalize to our backend. Amazon Personalize is a machine learning service that helps developers easily integrate personalized recommendations to the website with no machine learning expertise required.</p><p>In this article, we share how did we develop a personalized job recommendation feature with Amazon Personalize. To show job recommendations to our users, we have two main steps:</p><ol><li>Create a training model in Amazon Personalize.</li><li>Build a data pipeline to connect with Amazon Personalize.</li></ol><p>The data pipeline needs to continuously:</p><ul><li>Send user’s interaction data to our training model in Amazon Personalize.</li><li>Fetch user’s personalized job recommendation and show it on the front-end side.</li></ul><h4>Part 1: Create a training model in Amazon Personalize</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LCTZ9WneD5Vd73EWVBIHyA.jpeg" /><figcaption>Workflow of creating a training model in Amazon Personalize</figcaption></figure><p>Let’s begin with creating a training model. The diagram above shows the steps of creating a training model in Amazon Personalize. In Amazon Personalize, a training model is called a “solution version”. Here are the details for each step:</p><ol><li>Create a dataset group. It is a container for all of our resources, such as datasets, solutions and campaigns.</li><li>Create three datasets, including items, users and interactions. They are containers for our item, user and interaction data, which are used for training our model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Zj0Ylct4aRXLMHNL_Q6QA.png" /><figcaption><em>Three datasets we created in Amazon Personalize</em></figcaption></figure><p>3. Upload our user, item and interaction data in CSV format to S3 bucket. We upload our job data, user data and user’s clicks data in CSV format to Amazon S3. Amazon Personalize will fetch those data from Amazon S3 for training model later.</p><p>4. Create dataset import jobs to import those CSV files. Specify the Amazon S3 location of the files in the console. Meanwhile, create schemas that fit our datasets.</p><p>For the schema of a dataset, fields should match all column headers in our CSV file. Take “item dataset” as an example, we have fields such as JOB_NAME and JOB_TYPE.</p><p>Additionally, we set these fields with categorical or textual properties to be true. The reason is to let Amazon Personalize train a model based on these fields. JOB_NAME has an infinite set of values which is unstructured data. Yet, JOB_TYPE is not. It has a fixed set of values, part-time or full-time. That is why we set textual for the former but categorical for the latter.</p><pre>{<br> &quot;fields&quot;: [<br>  {<br>   &quot;name&quot;: &quot;JOB_TYPE&quot;,<br>   &quot;type&quot;: [<br>    &quot;null&quot;,<br>    &quot;string&quot;<br>   ],<br>   &quot;categorical&quot;: true<br>  },<br>  {<br>   &quot;name&quot;: &quot;JOB_NAME&quot;,<br>   &quot;type&quot;: [<br>    &quot;null&quot;,<br>    &quot;string&quot;<br>   ],<br>   &quot;textual&quot;: true<br>  },<br>        ...<br> ]<br>}</pre><p>5. Create a solution, which means training a model. We use the <strong>User-Personalization</strong> recipe for the training. To optimize the training model, we can enable HPO (hyperparameter optimization), by adding hpoconfig in the solution config. Amazon Personalize will run many training jobs with different values within the range we specify. The trade-off of it is to have longer training time which causes higher cost.</p><p>After the training is finished, the performance of the training model is shown in solution version metrics. A higher score (closer to 1) means better performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chnvaxxUezxnmCYwiF9sFA.png" /></figure><p>6. Create a campaign to deploy the training model, by selecting the solution version created in the previous step.</p><p>7. Apply a filter to filter out unwanted recommended items. For instance, we can specify only getting promoted job instead of normal jobs, by including an expression Items.JOB_PROMOTION_STATE in (“Posted”) below:</p><pre>Include ItemId WHERE Items.STATE IN (&quot;Posted&quot;) AND <br>Items.IS_JOB_SUSPENDED IN (&quot;false&quot;) AND <br>Items.IS_COMPANY_SUSPENDED IN (&quot;false&quot;) AND <br>Items.JOB_PROMOTION_STATE in (&quot;Posted&quot;)| Exclude ItemId <br>WHERE Interactions.event_type IN (&quot;apply_job_complete&quot;)</pre><p>Now, we have our training model ready to give us job recommendations for different users. By calling Amazon Personalize’s GetRecommendations API with a user id, it returns a list of job ids. We can test the campaign by inputting a random user id 123 in the console:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cg7peJmJeNO3E7POp-z-Iw.png" /></figure><p>It returns a recommendation ID and a list of item ids with their scores.</p><h4>Part 2: Build a data pipeline to connect with Amazon Personalize</h4><p>Next, we build a data pipeline to fetch user’s job recommendations and send their data to our model at the same time. As mentioned at the beginning of the article, <strong>our goal is to recommend jobs that fit user’s interests in near real-time.</strong> Hence, we need two features:</p><ol><li>Collect and send user’s data to Amazon Personalize and update our training model. User’s interaction data tells the training model what kind of jobs the user is interested in.</li><li>Fetch user’s job recommendations from Amazon Personalize and show them to the user when they are searching for jobs in Moovup.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VbZ7EgPGYQEitsH_Zwlqwg.jpeg" /><figcaption>Moovup’s data pipeline of Amazon Personalize</figcaption></figure><p>Here are the steps we have for the data pipeline:</p><ol><li>When a user clicks or applies for a job, our website or mobile application sends an event to Firebase, which is linked with BigQuery.</li><li>Create a scheduled job with Amazon EventBridge. Every 10 minutes, it fetches user, job, and interaction data from PostgreSQL and BigQuery. Then, we send these data in batches (10 records for each batch) to Amazon Personalize by calling the PutEvents, PutItems and PutUsers API.</li><li>Every two hours, Amazon Personalize updates the latest model automatically. It includes the new data we sent to the model before.</li></ol><p>After a while, the user’s interaction data is added to our model in Amazon Personalize. When they are searching jobs in Moovup, we go through the steps below:</p><ol><li>Our server sends a request to fetch a list of job recommendations with their user id, through the GetRecommendations API. Then, Amazon Personalize returns a list of job ids to our server.</li><li>Our server loops through the ids and get job details for each id.</li><li>Our server sends a list of jobs with their details to client. Now the user can see jobs that are recommended by Amazon Personalize.</li></ol><p>What if a new user accesses our application for the first time? They have not clicked or applied for any job in the application yet, which means the training model does not have their interaction data. In this case, Amazon Personalize will return a list of popular jobs instead of personalized job recommendations, as mentioned in the <a href="https://docs.aws.amazon.com/personalize/latest/dg/maintaining-relevance.html">documentation</a>:</p><blockquote>For new users without interactions data, recommendations are initially for only popular items</blockquote><h4>Summary</h4><p>The aim of integrating Amazon Personalize into our backend is to recommend job that matches user’s interest in near real-time. We achieved it by creating a training model in Amazon Personalize, and building a data pipeline to get job recommendations from Amazon Personalize for different users. As a result, we successfully improved our user’s job searching experience and employers’ efficiency of recruitment in Moovup.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f1c987ad28a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moovup/how-moovup-recommends-jobs-with-amazon-personalize-3f1c987ad28a">How Moovup recommends jobs with Amazon Personalize</a> was originally published in <a href="https://medium.com/moovup">Moovup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TCP and TLS handshake: What happens from typing in a URL to displaying a website? (Part 2)]]></title>
            <link>https://medium.com/@alysachan830/tcp-and-tls-handshake-what-happens-from-typing-in-a-url-to-displaying-a-website-part-2-243862438cd9?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/243862438cd9</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sat, 23 Apr 2022 14:30:00 GMT</pubDate>
            <atom:updated>2022-04-23T16:12:00.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bi42FAautSTf2OivvTtxPw.png" /></figure><p>After type in a URL of a website, the browser starts to search for the IP address of the server which hosts the website. This process is called DNS lookup, which is discussed in my previous article.</p><p>In this article, we look at what happens after typing in a URL and going through DNS lookup. The browser can start connecting with the server, with the following steps:</p><ol><li>TCP three-way handshake (SYN, ACK &amp; SYN, SYN)</li><li>TLS handshake (website that has HTTPS)</li></ol><h3>TCP three-way handshake and TLS handshake</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q_gjtOYMjh1D6uCn.png" /></figure><p><em>Image from </em><a href="https://www.cloudflare.com/en-gb/learning/ssl/what-happens-in-a-tls-handshake/"><em>Cloudflare</em></a></p><p>After DNS lookup, the browser (client) starts to establish a connection with the server. TCP is a common protocol used for connection, compared to UDP. <strong>A server must have been listening to a port already, such as 8080, for preparing to handle TCP connection from clients.</strong> How many TCP connections can be handled on a server depends on CPU and memory.</p><p>To start a TCP connection, the first step is three-way handshake.</p><h3>TCP three-way handshake</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g324imkPICu8tw1i.png" /></figure><p><em>Image from </em><a href="https://commons.wikimedia.org/wiki/File:Tcp-handshake.svg"><em>wikimedia</em></a></p><ol><li>SYN: Browser sends a SYN packet to server, with a random sequence number x. The packet also includes TCP flags and options</li><li>SYN-ACK: Server receives the SYN packet from the browser. It needs to return a SYN-ACK packet that includes two sequence numbers. For ACK, it is x+1 which acknowledges the packet sent from the client. For SYN, the server picks a random sequence number y on its side. Then it sends the packet to the client</li><li>ACK: The client receives the SYN-ACK packet. Similarly, the client acknowledges the packet from server, by incrementing the sequence number picked by the server, i.e y+1. Then, the client sends a ACK packet to the server with the sequence numbers y+1 and x+1.</li></ol><p>Now the TCP connection is established and the server reserves some memory to handle this connection.</p><h3>Before sending out HTTP request</h3><p>The TCP connection is initiated and we can start transmitting data now. However, for website with HTTPS, we have one more step to do, which is to ensure the data is transmitted securely by encryption. That is the why TLS comes into play before transmitting data.</p><h3>Why do we need TLS?</h3><p>Answer from RFC 8446 (TLS 1.3):</p><blockquote><em>The primary goal of TLS is to provide a secure channel between two communicating peers; the only requirement from the underlying transport is a reliable, in-order data stream.</em></blockquote><p>For example, if the user sends a POST request to server with credit card information, the information will be seen by the attacker who keeps track of the session. Hence, we need to keep the data secret, by using TLS protocol.</p><h3>TLS properties</h3><p>The secure channel provided by TLS have these properties:</p><ul><li>Authentication Server is always authenticated but client is optionally to be authenticated, by using different cryptography (RSA, ECSDA…)</li><li>Confidentiality Data is only visible to the endpoints.</li><li>Integrity Data cannot be modified.</li></ul><p>Keep these properties in mind. They help us to understand why TLS handshake work with certain steps later.</p><h3>Steps of TLS handshake (TLS 1.2)</h3><p>TLS 1.3 is the latest version now, which is a simplified version of 1.2. Yet, for having a better understanding, let’s start from looking into TLS 1.2.</p><h3>Message flow for a full handshake</h3><h3>Full handshake of TLS 1.2</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*97OsYH1seR4-jHKn.png" /></figure><p><em>Image from </em><a href="https://www.ietf.org/rfc/rfc5246.txt#:~:text=This%20document%20specifies%20Version%201.2,%2C%20tampering%2C%20or%20message%20forgery"><em>RFC5246</em></a><em><br>“*” Indicates optional or situation-dependent messages that are not always sent.</em></p><p>The diagram above describes both essential and optional steps of TLS handshake. A shortened version is also provided below, which shows only the essential steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LI4W_4QccpPsbOBA.png" /></figure><p><em>Image from </em><a href="https://www.ietf.org/rfc/rfc5246.txt#:~:text=This%20document%20specifies%20Version%201.2,%2C%20tampering%2C%20or%20message%20forgery"><em>RFC5246</em></a></p><p>Few important actions we do in TLS handshake:</p><ul><li>Negotiate the cipher suite used in the communication between server and client. Hence, they agree using a specific way to encrypt and decrypt data</li><li>Server sends its SSL certificate to client for verification</li></ul><p>Why do we need these actions? For the first bullet point, the answer lies in the one of the properties of TLS I mentioned before:</p><blockquote><strong><em>Confidentiality: Data is only visible to the endpoints.</em></strong></blockquote><p>To ensure data is only visible to the endpoints, when it is transmitted between client and server, <strong>they need to agree using the same cipher suite</strong>, in other words, negotiate to use the same algorithms and protocol, such as exchanging a key to encrypt and decrypt data, i.e. symmetric encryption.</p><p><strong>Similarly, for the second point, the server needs to prove its identity, by sending its SSL certificate to client.</strong> This action is reasonable when it comes to the following property of TLS:</p><blockquote><strong><em>Authentication Server is always authenticated but client is optionally to be authenticated, by using different cryptography (RSA, ECSDA…)</em></strong></blockquote><p>To understand the steps of how the how browser verifies SSL certificate, this <a href="https://stackoverflow.com/questions/35374491/how-does-the-client-verify-servers-certificate-in-ssl">Stackoverflow</a> discussion maybe helpful for you.</p><p>Now we understand the important steps in TLS handshake and how they relate to the properties of TLS protocol. Let’s go through the details of TLS handshake.</p><h3>TLS handshake (TLS 1.2)</h3><p>The diagram below is a shortened version of TLS handshake with essential steps and <strong>RSA based cipher suite</strong>. In total, the TLS handshake here takes 2 RTT. The first is from step 1–4. The second is from step 5–9. During the handshake, the server and client exchanges message to negotiation and authentication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HbyNE-s9lC6HnBm9.png" /></figure><p>For the detailed description about each steps, <a href="https://www.acunetix.com/blog/articles/establishing-tls-ssl-connection-part-5/">this article</a> or <a href="https://www.ietf.org/rfc/rfc5246.txt#:~:text=This%20document%20specifies%20Version%201.2,%2C%20tampering%2C%20or%20message%20forgery">RFC 5246</a> make a clear explanation on it. Hence, I would not go through all the details, instead, I focus on few questions:</p><ul><li>What are TLS handshake protocol and TLS record protocol?</li><li>What is pre-master secret?</li><li>What is master key? Why do we need it?</li><li>What does Change cipher spec mean?</li></ul><h3>What are TLS handshake protocol and TLS record protocol?</h3><p>TLS protocol is made up with different protocols.</p><p>In the book <a href="https://hpbn.co/transport-layer-security-tls/#tls-record-protocol">High Performance Browser Networking</a>, it mentions the workflow here:</p><ol><li>Record protocol receives application data.</li><li>Received data is divided into blocks: maximum of 214 bytes, or 16 KB per record.</li><li>Message authentication code (MAC) or HMAC is added to each record.</li><li>Data within each record is encrypted using the negotiated cipher.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z9RNVsFx4p_pTeLm.png" /></figure><p>The data comes from application layer, then is passed to session layer and finally is passed to TCP. On the receiver side, it repeats the whole process in a reverse way.</p><h3>What is pre-master secret?</h3><h3>1. The purpose of pre-master secret</h3><p>The purpose of having a pre-master secret is only for generating a master secret. In other words, we don’t encrypt or decrypt data with pre-master secret.</p><h3>2. How pre-master secret is created?</h3><p>The pre-master secret is created based on the cipher suite that server and client agree to use. The way that pre-master is created depends on the chosen cipher suite. For example, a 48-byte premaster secret is generated, if RSA is the key agreement. After that, the value is be encrypted with the public key extracted from server’s SSL certificate and is sent to the server.</p><h3>3. How does server decrypt the value?</h3><p>Server decrypt it by using the private key in the SSL certificate. We can use a private key to decrypt a value encrypted by the public key. That is the key concept of RSA.</p><h3>Master secret</h3><h3>1. The purpose of master secret</h3><p>Server and client encrypt data with master secret before transmitting it to each other.</p><p>In RFC5246, it mentions how master secret is used:</p><blockquote><em>The master secret is expanded into a sequence of secure bytes, which is then split to a client write MAC key, a server write MAC key, a client write encryption key, and a server write encryption key. Each of these is generated from the byte sequence in that order.</em></blockquote><p>We have 4 keys generated from master key:</p><ul><li>client write key</li><li>server write key</li><li>client write MAC key</li><li>server write MAC key</li></ul><p>For understanding what are these keys for, I recommend to read <a href="https://www.cloudflare.com/en-gb/learning/ssl/what-is-a-session-key/">Cloudflare’s article</a> . The explanation is short and simple.</p><h3>2. How master secret is created?</h3><pre>master_secret = PRF(pre_master_secret, &quot;master secret&quot;,<br>                  ClientHello.random + ServerHello.random)</pre><p>After step 5, both server and client have already obtained the necessary values for producing a master secret. They can now generate the same master secret.</p><h3>What does Change cipher spec mean?</h3><p>In step 6 and 8, client and server send change cipher spec message. The answer is revealed in RFC5246:</p><blockquote><em>The ChangeCipherSpec message is sent during the handshake after the security parameters have been agreed upon, but before the verifying Finished message is sent.</em></blockquote><p>So, what is it used for?</p><blockquote><em>The ChangeCipherSpec message is sent by both the client and the server to notify the receiving party that subsequent records will be protected under the newly negotiated CipherSpec and keys. Reception of this message causes the receiver to instruct the record layer to immediately copy the read pending state into the read current state. Immediately after sending this message, the sender MUST instruct the record layer to make the write pending state the write active state.</em></blockquote><p><strong>ChangeCipherSpec message is to notify the other side, from now on, the messages I sent to you will be encrypted by using a symmetric shared key.</strong></p><h3>DH and RSA</h3><p>The concepts of master key and pre-master key mentioned above are for RSA based cipher suite. Besides, Diffie-Hellman (DH) is another choice, which uses DH public key and private key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0WAI0KIGC7dpzz_E.png" /></figure><p><strong>RSA cannot achieve forward secrecy</strong>. The meaning is that RSA is unsafe since all the messages will be decrypted if attacker gets the private key stored in the server. Yet, DH with ephemeral mode (i.e DHE), has forward secrecy. To improve security, RSA is removed and DHE is kept in TLS 1.3.</p><p>Now we understand the overview of TLS 1.2. It’s time to get into the latest TLS handshake, TLS 1.3.</p><h3>The difference between TLS 1.3 and TLS 1.2</h3><p>With understanding TLS 1.2, we can easily realise why TLS 1.3 is faster and safer. Let’s start by looking the overview of full TLS handshake in version 1.3.</p><h3>Full TLS handshake 1.3</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wrwAogoiu1GxicPa.png" /></figure><p>Here are 3 main phrases of the handshake mentioned in RFC 8446:</p><ul><li>Key Exchange: Establish shared keying material and select the cryptographic parameters. Everything after this phase is encrypted.</li><li>Server Parameters: Establish other handshake parameters (whether the client is authenticated, application-layer protocol support, etc.).</li><li>Authentication: Authenticate the server (and, optionally, the client) and provide key confirmation and handshake integrity.</li></ul><h3>Key Exchange</h3><h3>1. Client Hello</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*ltiZ9augvrHK7X8o.png" /></figure><p>RFC 8446</p><p>We would not cover all the details, instead, I would like to focus on the notable differences compared to TLS 1.2. Compared to TLS 1.2, similarly, the first step is always Client Hello, but the parameters look different here.</p><h3>What is key_share?</h3><p>RFC 8446:</p><blockquote><em>In the Key Exchange phase, the client sends the ClientHello (Section 4.1.2) message, which contains … a list of symmetric cipher/HKDF hash pairs; either a set of Diffie-Hellman key shares (in the “key_share” (Section 4.2.8) extension), a set of pre-shared key labels (in the “pre_shared_key” (Section 4.2.11) extension), or both;</em></blockquote><p>We can conclude that here are the supported key exchange modes:</p><ul><li>(EC)DHE, i.e: Elliptic-Curve Diffie-Hellman ephemeral</li><li>PSK-only</li><li>PSK with (EC)DHE</li></ul><h3>What is (EC)DHE?</h3><p>Before we jump into (EC)DHE, we have to know what is Diffie-Hellman(DH). In short, Diffie-Hellman is a key exchange method. The RFC mentions 3 types of Diffie-Hellman:</p><ul><li>DH: static Diffie-Hellman (removed)</li><li>DHE: Diffie-Hellman ephemeral</li><li>ECDHE: Elliptic-Curve Diffie-Hellman ephemeral</li></ul><p>For the first type, <strong>DH is removed in TLS 1.3 due to the lack of perfect forward secrecy</strong>, which means all the messages will be leaked if attacker gets the private key.</p><p>For the second and third types, unlike static DH which uses the same private key, Ephemeral Diffie-Hellman <strong>creates new private key for each new connection</strong>, based on the parameters exchanged during the handshake. For ECDHE, the main difference with DHE is to apply algebraic structure of elliptic curves, for decreasing the key size while keeping a similar security level.</p><h3>Diffie-Hellman parameters</h3><p>Compared to TLS 1.2, TLS 1.3 still applies some types of Diffie-Hellman, but it limits the Diffie-Hellman parameters since not all of them are secure.</p><h3>What is PSK (Pre-Shared Key)?</h3><p>Different from TLS 1.2, TLS 1.3 enables client and server create a new connection based on the their previous connection, by using PSK.</p><p>The top is the full handshake and the bottom is the shortened handshake with using PSK:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p_D0Jd0xSd3sXMHB.png" /></figure><pre>+  Indicates noteworthy extensions sent in the previously noted message.<br>*  Indicates optional or situation-dependent messages/extensions that are not always sent.<br>() Indicates messages protected using keys derived from a client_early_traffic_secret.<br>{} Indicates messages protected using keys derived from a [sender]_handshake_traffic_secret.<br>[] Indicates messages protected using keys derived from [sender]_application_traffic_secret_N.</pre><p>How does it work?</p><ol><li>After client and server finishes the first full handshake process, the server sends a PSK to the client.</li><li>When the client comes back and connect to the server again, the client sends the PSK to the server for negotiating to use PSK for the connection.</li><li>If the server accepts, the connection references the previous cryptographical context, instead of going through the full handshake again. For example, the server doesn’t need to send the certificate to client for verification.</li></ol><p>This <a href="https://www.ibm.com/docs/en/sdk-java-technology/8?topic=handshake-session-resumption-pre-shared-key">IBM documentation</a> makes a simple step by step explanation on it. <strong>Be aware that resumption is not a totally new feature. In TLS 1.2, this is achieved by using session ids and tickets.</strong></p><h3>How does PSK bring in 0-RTT?</h3><p>If client and server have PSK, client can send data on the first flight (“early data”). Client uses PSK to encrypt early data and authenticate server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/0*u6zNxn-2BL_MRNcZ.png" /></figure><p>Yet, this handshake comes with different security concerns, as mentioned in the <a href="https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/">Cloudflare article</a></p><h3>2. Server Hello</h3><p>Let’s get back to the step 2 of the full handshake. Now the server has received Client Hello message, it sends back a Server Hello message.</p><p>The Server Hello message is similar to TLS 1.2. Server checks if the parameter sent from client is acceptable or not. It also sends its certificate to client for verification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/0*Oz3BNaKNCoiMwC8s.png" /></figure><h3>3. Finished message</h3><p>After all, the client sends a finished message to server. Then client and server starts exchanging encrypted data.</p><h3>TLS 1.3 improvement summary</h3><ul><li>Improve speed performance TLS 1.3 only takes 1-RRT (or even 0-RRT!), but TLS 1.2 takes 2-RRT.</li><li>Remove insecure cryptography RSA key exchange due to its secure vulnerability. Only Diffie-Hellman key exchange mechanism is used for key exchange in TLS.</li></ul><p>For more details about the improvments, Cloudflare’s <a href="https://blog.cloudflare.com/rfc-8446-aka-tls-1-3/">article</a> make a good explanation.</p><h3>Recap of the difference between TLS 1.2 and TLS 1.3</h3><p>The left is TLS 1.2 using RSA based cipher suite, the right is TLS 1.3 using DHE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YVnBHhh5MkeTc7hk.png" /></figure><h3>Summary</h3><p>I summarise my learning about TCP 3-way handshake and TLS handshake (1.2 and 1.3 version) in this article, including these key concepts:</p><ul><li>When browser starts connecting with server, it has to go through TCP 3-way handshake, TLS handshake and then sends a HTTP request to the server</li><li>TCP 3-way handshake includes SYN, ACK-SYN, ACK, with an incremental sequence number</li><li>After TCP 3-way handshake, if the website is HTTPS, browser and server starts TLS handshake</li><li>The main purposes of TLS handshake are to negotiate the cryptography used to encrypt data and to verify the server based on SSL certificate</li><li>RSA and DH are the common cryptography used in TLS handshake. The former relies on pre-master key and master key, while the latter relies on public DH key and private DH key</li><li>TLS 1.3 major improvements include reducing RTT (from 2-RTT to 1-RTT) and enhancing security by removing insecure cryptography</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=243862438cd9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What happens from typing in a URL to displaying a website? (Part 1 — DNS cache and DNS lookup)]]></title>
            <link>https://medium.com/@alysachan830/what-happens-from-typing-in-a-url-to-displaying-a-website-part-1-dns-cache-and-dns-lookup-86441848ea59?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/86441848ea59</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sun, 20 Feb 2022 16:55:20 GMT</pubDate>
            <atom:updated>2022-04-23T14:31:56.219Z</atom:updated>
            <content:encoded><![CDATA[<h3>DNS cache and DNS lookup: What happens from typing in a URL to displaying a website? (Part 1)</h3><p>This question is a popular interview question which can involve lots of in-depth concepts, which can be separated into few articles</p><ol><li>DNS cache, DNS lookup (This article)</li><li>TCP/IP connection</li><li>SSL, TLS, HTTP protocol</li><li>Render files</li></ol><p>This article focuses on the first part, DNS cache and DNS lookup:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TEWzcbRYWIwotTcb.png" /></figure><h3>General concept</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*twSnFd0spJu9CTKh.jpg" /></figure><p><em>Image from </em><a href="https://twitter.com/manekinekko/status/1281704000572858375/photo/1"><em>Wassim Chegham Twitter</em></a></p><p>Before we start, let’s have a general concept of what happens when we type in https://www.example.com/ in an address bar in a browser.</p><p>The short answer is:</p><ol><li>Build a connection between my browser and the website’s server.</li><li>Get HTML, CSS and JavaScript files from the server.</li><li>Render the files.</li></ol><p>That’s the general process we need for showing website’s website. But for sure there are lots of detail under the hood. Let’s start with step 1, <strong>finding out IP address of website’s server.</strong></p><h3>Find out IP address of website’s server</h3><p>After we type in the URL of website, the browser starts finding out the IP address of the domain name, we cannot build a connection with website’s server by only using the domain name <a href="https://www.example.com/">https://www.example.com/.</a></p><p>Domain name is easy to be read by human, but under the hood, it has a corresponding IP address. The browser needs its IP address, such as 192.158.1.38., and build a connection with it.</p><h3>DNS lookup</h3><p>To find out the IP address of https://www.example.com/, there are 2 main skips:</p><ol><li>Check the cache in browser, OS, router and ISP for finding out the IP address of <a href="https://www.example.com/">https://www.example.com/</a></li><li>If we cannot find the IP address in the cache above, continue the DNS lookup.</li></ol><p>The purpose of looking at the local cache first is to speed up the time we need for DNS resolution.</p><p>Let’s start understanding what is DNS cache.</p><h3>DNS cache</h3><p>We begins from searching the website’s IP address from <strong>browser cache</strong> to <strong>ISP cache</strong>. The searching process stops when the IP address is found successfully.</p><ol><li>Browser cache <br>The browser looks at the browser cache for finding the domain name’s IP address. The browser caches the IP address we have visited recently, with setting a TTL (Time to live).</li><li>OS cache <br>Checks the cache in OS. Same as the browser cache, the DNS cache in OS stores IP address we visited recently with TTL.</li><li>Router cache <br>Check the DNS cache in router. The router may have a DNS server by using ISP’s DNS server.</li><li>ISP cache <br>If the browser cannot find the website’s IP address in the process above, it continues to looks the cache in the DNS server of your ISP (Internet Service Provider) that provides Internet service to you.</li></ol><p>If the DNS server ISP doesn’t have the cache of the IP address, the ISP will start a DNS lookup for you.</p><h3>DNS lookup</h3><p>Now, my browser sends a request to the ISP. The ISP reissues the request to other DNS servers, which is the DNS Root nameserver, for asking the IP address of <a href="https://www.example.com/">https://www.example.com/</a></p><h3>DNS query</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AVDKxcX-6IIzV2Ka.png" /></figure><p>To continue the process, we start from step 4 in the illustration above.</p><ol><li>The root nameserver receives the request from ISP DNS server. It looks at the top level domain (TLD) of https://www.example.com/, which is .com. Then, it responds to ISP DNS server with the IP address of the nameserver of .com.</li><li>ISP DNS server reissues the request to .com nameserver. The .com nameserver responds with the IP address of the nameserver of example.com.</li><li>ISP DNS server reissues the request to example.com nameserver. The server responses with the IP address of the www.example.com. DONE! Finally we get the IP address of the domain!</li></ol><p>Besides, after getting the IP address of https://www.example.com/, <strong>ISP DNS server keeps the IP address of it in cache</strong>. Hence, next time the browser may directly get the IP address from the cache, instead of going through the recursive query again.</p><p>Although this helps saving time and reducing traffic, one <strong>potential problem is that the cache may be inaccurate if the IP address of </strong><strong>example.com is updated.</strong> But that&#39;s another topic that won&#39;t be discussed in this article.</p><h3>What does “the nameserver of example.com&quot; mean?</h3><p>You may wonder what does “the nameserver of example.com&quot; mean? In short, this nameserver connects your website&#39;s domain name (https://www.example.com/) with the IP address of the server. In this example, it is the last nameserver that the ISP DNS server talks to, which is also called <strong>authoritative nameserver</strong>.</p><h3>Situation 1: Register domain name and use its default nameserver</h3><p>For example, I register the domain https://www.example.com/ and host the website in some famous registrars such as GoDaddy or Namecheap. Take GoDaddy as an example, by default, GoDaddy will provide a nameserver, which connects the IP address, <a href="http://exmaple.com/">exmaple.com</a> website&#39;s server and the domain name <a href="https://www.example.com/">https://www.example.com/</a></p><h3>Situation 2: Register domain name and use custom nameserver</h3><p>How about I register the domain on GoDaddy, but I host my website’s on Vercel?</p><p>In this case, I can use the nameserver provided by GoDaddy when I purchased a domain name on GoDaddy. It connects my domain name https://www.example.com/ and the domain name&#39;s IP address provided by GoDaddy. The most basic type of IP address is called &quot;A record&quot;.</p><p>After that, I have to visit my Vercel account and update the “A record”, for example 192.0.2.1, of my website hosted in Vercel. Now, the IP address is pointing to the server of my website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qX6A8FOrAtVg5wEs.png" /></figure><p>Let’s get back to the query process in TLD server mentioned above.</p><ol><li>The .com TLD server finds out the IP address of the nameserver provided by GoDaddy. Return it to the ISP DNS server.</li><li>ISP DNS server makes a request to the nameserver provided by GoDaddy.</li><li>The nameserver provided by GoDaddy has the IP address of www.example.com and responds to ISP DNS server with it.</li><li>ISP DNS server receives the response and returns the IP address to my browser.</li><li>DONE!</li></ol><p>An article in Cloudflare, <a href="https://www.cloudflare.com/en-gb/learning/dns/dns-server-types/">DNS server types</a>, is recommended for understanding more details about different types of DNS server, such as TLD server, root server.</p><h3>DNS recursive query and iterative query</h3><p>Let’s get back to whole picture of DNS lookup. We have 2 types of DNS query in the whole process.</p><p>The request and response between your browser and ISP DNS server is <strong>recursive query</strong>. On the contrary, the communication between ISP DNS server and different nameservers is <strong>iterative query</strong>.</p><p>To know the difference, we can take reference from <a href="https://www.cloudflare.com/en-gb/learning/dns/what-is-recursive-dns/">an article of Cloudflare</a>:</p><blockquote><em>…Put another way, the client does a form of delegation in a recursive DNS query. It tells the DNS resolver, “Hey, I need the IP address for this domain, please hunt it down and don’t get back to me until you have it.” Meanwhile, in an iterative query, the client tells the DNS resolver, “Hey, I need the IP address for this domain. Please let me know the address of the next DNS server in the lookup process so I can look it up myself.”</em></blockquote><ul><li><strong>Recursive DNS query</strong> <br>After the client sends a request to the ISP DNS server, it <strong>must respond with the answer that the client is asking for, i.e: the IP address of </strong><strong>www.example.com.</strong> If the ISP DNS server doesn&#39;t have the answer, it must keeps finding out the answer by itself until it has the answer. Then it responds back to the client.</li><li><strong>Iterative DNS query</strong> <br>After the ISP DNS server sends a request to a remote DNS server, the remote DNS server <strong>does not necessary to respond with the answer</strong>, it can respond an IP address of another DNS server to make a referral. The ISP DNS server looks at the response and reissue a new request to another server. It keeps repeating this process until it finds out the answer.</li></ul><h3>What is DNS (Domain Name System)?</h3><p>The DNS is a hierarchical naming system. The hierarchy starts from right to left. That’s why DNS query starts from root server to subdomain. Each layer is separated by a dot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*15SDBYv0ma1ef8pl.png" /></figure><p>The DNS hierarchy starts from right to left. In the DNS lookup process mentioned above, we start from root server, then TLD (Top level domain) server, and finally second level domain server.</p><p>You may wonder what is the meaning of host? In this example, www is the hostname of https://www.example.com, which is a very common host name. Host name means the name of the device which is connected to the network. In lots of case, the hostname of web server is www. Another example is ftp for an FTP server. Besides, www is also a subdomain of example.com, as mentioned in the <a href="https://en.wikipedia.org/wiki/World_Wide_Web#WWW_prefix">Wikipedia</a> about World Wide Web (www).</p><p>The hierarchy works like a tree diagram, from top to bottom. In <a href="https://en.wikipedia.org/wiki/Domain_name">Wikipedia</a> about domain name, it mentions the DNS query for en.wikipedia.org works as described below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tzdIovR2AqdfNKrm.png" /></figure><ul><li>. is root nameserver</li><li>com, org ... are TLS nameserver</li><li>wikipedia is the second level domain nameserver</li></ul><h3>DNS zone</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fw7nozUjYHN7XsNI.png" /></figure><p>The image here illustrates our example of querying the IP address of https://www.example.com. The layers are also called DNS zones. The DNS resolution starts from the DNS root zone (top) to domain zone (bottom).</p><h3>Zone file with resource record (RR)</h3><p>Example of a zone file for example.com</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g5UxI_2f4Q_9_C_W.png" /></figure><p><em>Screenshot from </em><a href="https://en.wikipedia.org/wiki/Zone_file"><em>Wikipedia</em></a></p><p>In short, a zone file is a text file that describes a DNS zone. It has a SOA that describes the information of that DNS zone. Also, the zone file also collects resource records (RR), for mapping between resources such as domain names and IP address.</p><p>In the image above, we have some important resource records. For example, “A” stands for address, which means the IP address (IPv4). Also, “CNAME” records , which points to a domain, i.e <a href="http://www.example.com./">www.example.com.</a></p><p>For understanding the basic concepts about zone file, this article, <a href="https://help.dyn.com/how-to-format-a-zone-file/">How To Format A Zone File</a>, may be helpful.</p><h3>Summary</h3><p>To summarize, for finding out the IP address of the website, the browser starts with looking at the caches locally, and then reaches out to ISP DNS server. Finally, if no result is found, it needs to reach out remote DNS servers.</p><p>This article is a summary of my learning in DNS lookup and servers. Feel free to share your opinions or advice if I make any mistakes!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86441848ea59" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Basics of OAuth 2.0: Authorization code flow, Implicit flow, state and PKCE]]></title>
            <link>https://medium.com/@alysachan830/the-basics-of-oauth-2-0-authorization-code-implicit-flow-state-and-pkce-ed95d3478e1c?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed95d3478e1c</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sun, 16 Jan 2022 10:10:03 GMT</pubDate>
            <atom:updated>2022-01-17T00:15:24.097Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgPt1BLKrFbMji5by0pleQ.png" /></figure><p>As a beginner learning authentication in back-end development, I come across the topic of OAuth 2.0. In this article, I try to summarize my learning on:</p><ul><li>OAuth 2.0 concept</li><li>Authorization code flow and implicit flow with Google OAuth 2.0 API example</li><li>Common CSRF attack, state parameter and PKCE</li></ul><p>Here is another beginner-friendly <a href="https://aaronparecki.com/oauth-2-simplified/#client-credentials">article</a> about the topics I cover in this article.</p><h3>Core concept of OAuth 2.0</h3><p>OAuth is about the process of a user authorizing an application to have a limited access to his data in a third-party application, like Google.</p><p>Here is an OAuth 2.0 example that you may have already experienced:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/0*I7NviAlHsXu7t47D.png" /></figure><p><em>Image from </em><a href="https://developers.google.com/identity/protocols/oauth2/openid-connect"><em>Google Open ID Connect documentation</em></a></p><p>This situation happens when you are visiting a website, which requires your data from Google. The website needs your permission to get your Google Drive data from Google. It exactly shows the core concept of OAuth, suggested in the official <a href="https://datatracker.ietf.org/doc/html/rfc6749">RFC6749</a>:</p><blockquote><em>The OAuth 2.0 authorization framework enables a third-party application to obtain limited access to an HTTP service, either on behalf of a resource owner by orchestrating an approval interaction between the resource owner and the HTTP service, or by allowing the third-party application to obtain access on its own behalf.</em></blockquote><p>Back to our Google example, with the OAuth concept, we can see few parties involved in this process:</p><ul><li>Resource owner (you)</li><li>Client (the website you are visiting)</li><li>Resource server / Authorization Server (Google)</li></ul><p>Details about the roles are mentioned in <a href="https://datatracker.ietf.org/doc/html/rfc6749">RFC6749</a></p><h3>Google Drive example</h3><p>Here is a simple example of these 3 parties.</p><ol><li>You are Tom, who visits a website that needs to get access to your Google Drive data</li><li>The website requests Google to get your Google Drive’s data</li><li>Google asks for your consent</li><li>If you agree to grant permission for the website to get access to your Google Drive’s data, Google returns your data to the website</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEKcdyNLye4e_OYaUd5u9w@2x.png" /></figure><h3>Concept 1: OAuth is about authorization, not authentication</h3><p>Some people may relate authentication with OAuth. However, <strong>OAuth is about authorization, which stands for open authorization.</strong> It is not originally designed for authentication.</p><p>If we have to implement authentication with OAuth, we can use <strong>open ID Connect</strong> , an industry standard built on top of OAuth 2.0, for authenticating users. This topic is not covered in this article.</p><h3>Concept 2: Website won’t obtain your confidential information</h3><p>The advantage of using OAuth 2.0 is to protect user’s confidential information in the third party service. In our previous Google Drive example, the website won’t obtain the user’s confidential information in his Google account, such as his Google password.</p><h3>Grant types</h3><p>Here are some common grant types in OAuth 2.0:</p><ul><li>Authorization code flow</li><li>Implicit flow</li><li>Client credentials flow (It will not be covered in this article)</li></ul><p>Imagine your website has to display the user’s Google Drive data. There are two grant types in Google OAuth 2.0 API which you can choose:</p><ol><li>Authorization code flow (for server-side web app)</li><li>Implicit flow (for client-side web app)</li></ol><h3>Prerequisites: register your app</h3><p>Before we start, no matter which grant types we choose to implement, we have to register our web applications in that third-party service. It will return the credentials we need for the next step, such as client ID, client secret etc. Details can be found in third-party service, such as Google OAuth 2.0 API documentation.</p><p>After we have register our apps, we can start implementing the OAuth 2.0 framework.</p><h3>Authorization code flow</h3><h3>When do you use authorization code flow?</h3><p><strong>You are able to store client secret in your server to avoid exposing it to the public.</strong> If you can only work for the front-end side, such as single-page applications (SPA) or mobile apps, which will leak the client secret to the public, it is strongly recommended to use authorization code with PKCE, which will be discussed later in this article.</p><h3>Steps</h3><p>To focus on implementing authorization with OAuth, <strong>I simplify the explanation by skipping the login process, i.e authentication, here.</strong> Imagine we have to display user’s Google Drive data once he gets into our website, the process works as described below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DbRcjr8MXVxGHpW0.png" /></figure><p>In short, after user agrees to grant permission, the website’s server:</p><ol><li>Receives an authorization code in the callback URL</li><li>Exchanges the authorization code for access token and refresh token</li><li>Calls Google Drive API to get user’s Google Drive data with the access token (client id and client secret are also required)</li></ol><p><strong>Authorization code</strong></p><ul><li>Short-lived and for single use</li><li>If authorization server receives multiple attempts to exchange authorization code for access token, the server should revoke all access tokens that have been granted already</li><li>Token for exchanging access token and refresh token. After the user agrees to grant permission, the authorization server (e.g Google’s server) will direct the user to a URL which includes the authorization code as query:</li></ul><pre>https://oauth2.example.com/auth?code=4/P7q7W91a-oMsCeLvIaQm6bTrgtp7</pre><p>Then the website’s server can exchange it for access token and refresh token, from the authorization server (e.g Google’s server)</p><p><strong>Access token</strong></p><ul><li>Token required for getting user’s Google resource</li><li>Short-lived and periodically expires</li></ul><p><strong>Refresh token</strong></p><ul><li>Token for getting a new access token, when the previous access token expires. User’s permission is not required during this process</li></ul><p>For Google OAuth 2.0 API:</p><ul><li>In Google OAuth API, normally refresh token never expires, unless it is revoked. However, as mentioned in <a href="https://developers.google.com/identity/protocols/oauth2">Google’s documentation</a>, refresh tokens for Google Cloud platform project with status ‘testing’ works differently:</li></ul><blockquote><em>A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of “Testing” is issued a refresh token expiring in 7 days.</em></blockquote><ul><li>Also, in Google OAuth API, there’s a <a href="https://developers.google.com/identity/protocols/oauth2">limit</a> for issuing refresh token for Google account for per OAuth 2.0 client ID. It is recommended to store the refresh token permanently.</li></ul><h3>But why can’t the server directly return access token and refresh token?</h3><p>You may wonder <strong>why cannot the server return access token and refresh token directly?</strong> What’s the point of using an authorization code? Like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JU50oRwIGOv6aeQc.png" /></figure><p>Indeed, the idea suggested above does workable. This flow is called implicit flow, which is another grant type. The third-party directly returns access token and refresh token, without issuing an authorization token. <strong>However, this flow is insecure because access token and refresh token may be exposed in the frond-end side, which can be stolen easily.</strong></p><p>Implicit flow mentioned in <a href="https://www.rfc-editor.org/rfc/rfc6749#section-1.3">RFC6749</a>:</p><blockquote><em>The implicit grant is a simplified authorization code flow optimized for clients implemented in a browser using a scripting language such as JavaScript. In the implicit flow, instead of issuing the client an authorization code, the client is issued an access token directly</em></blockquote><p>When the user agrees to grant permission, the third-party service (e.g Google) will make a GET request with the callback URL we have set in Google console before. Like this:</p><pre>http://localhost/callback?code={authorization code here}</pre><p>Imagine the server directly returns access token and refresh token in the URL, like this:</p><pre>http://localhost/callback?access_token={...}&amp;refresh_token={...}</pre><p>The attacker can steal the tokens here and directly call Google Drive API to obtain the user’s Google Drive data. For example, now the tokens are exposed to all JavaScript in the user’s browser, including JavaScript from third-party libraries. Any common XSS attacks can easily steal the tokens here.</p><blockquote><em>Although implicit flow is relatively insure compared to authorization code flow, Google does support implicit flow for JavaScript Web Apps. In this case, Google suggests developer to use Google’s JavaScript client library for the implementation for to have better security. Details can be founded </em><a href="https://developers.google.com/identity/protocols/oauth2/javascript-implicit-flow"><em>here</em></a><em>.</em></blockquote><h3>Implicit flow</h3><p>As mentioned above, implicit flow is another grant type. <strong>It is a ‘simplified’ version of authorization code flow</strong>, which skips the authorization code part and directly returns an access token. If the hacker steals the access token, he can call Google API, such as Google Drive API, to get the user’s Google Drive data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rtNv6KywMbNd4EJh.png" /></figure><p>Take Google OAuth API as an example, Google provides implicit flow for client-side web applications. There are 2 approaches to implement the flow:</p><ul><li>Use gapi.auth2 and gapi.client objects, provided by Google JavaScript API Library to handle user&#39;s authorization and authorization request (Officially recommended approach)</li><li>Directly call OAuth 2.0 endpoint without using any library</li></ul><p>No matter you use JavaScript API Library or directly using OAuth 2.0 endpoint approach, <strong>the access token will be exposed in the front-channel, which may be stolen in XSS attack.</strong></p><h3>Does it mean that authorization code flow is secure enough?</h3><p>Compare to implicit flow, <strong>authorization code flow is more secure</strong> since we exchange tokens in <strong>back-channel</strong>. In the example above, the website’s server exchange tokens with Google’s server. This process happens in back-channel and tokens are not exposed to the front-end side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UPTMG7xje773tc-O.png" /></figure><p>However, that’s not secure enough when it comes to preventing cross site request forgery (CSRF) attack.</p><h3>Common CSRF attack</h3><p>The attack can cheat the user to send an unintentional request. Consider the example below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G_l6It5-Wg8y309C.png" /></figure><p>In the example above, given that:</p><ul><li>My website has a feature that create files in user’s Google Drive</li><li>After user signs in my website, a JWT token is stored in the user’s cookie. User can connect to his Google Drive after signs in my website, by using OAuth 2.0 flow</li></ul><p>The CSRF attack starts with the following steps:</p><ol><li>Hacker signs in my website and connects his Google Drive</li><li>Hacker gets the redirect URL which contains authorization code returned by Google</li><li>Hacker stops running the OAuth 2.0 here</li><li>Hacker develops a malicious website / link. Later a user, Tom visits it. Tom is a user of my website and has signed in already before</li><li>He clicks on that malicious website / link, it triggers a GET request to <strong>my website</strong>. (The “cross-site” part in CSRF), with Tom’s JWT in his browser’s cookie</li><li>My server validates Tom’s cookie and sends a request to Google, for exchanging access token and refresh token</li><li>Now Tom’s account in my website is connected to hacker’s Google Drive. When Tom visits my website and use my website’s feature to create file in Google Drive, he create files in hacker’s Google Drive</li></ol><h3>But can CORS setting block cross-site attack?</h3><p>I thought of this question when I was learning CSRF:</p><p>If my website has only allow domain https://myexamplesite.com to access my resource by setting CORS, wouldn&#39;t my website block the GET request in step 7 in the diagram? Because the hacker&#39;s website, e.g https://evilhackersite.com) does not have the same origin as my website (e.g https://myexmaplesite.com).</p><p>However, as mentioned in <a href="https://security.stackexchange.com/questions/157061/how-does-csrf-correlate-with-same-origin-policy">this discussion</a>, we can’t prevent CSRF attack by setting CORS:</p><blockquote><em>… This means that if you are performing a CSRF attack on a vulnerable site which results in some server side state change (e.g. user creation, document deletion etc), the attack will be successful but you would not be able to read the response.</em></blockquote><p>With CORS setting, the attacker can still send a request to my server and my server will react to it. The limitation is that he cannot read the responses of his request.</p><h3>Solution: Request authorization code with ‘state’</h3><p>The cause of the security issue above is that <strong>we cannot verify whether the authorization code we are using is requested by the user or the hacker.</strong> To solve the problem, we can add state parameter, which is an <strong>anti-forgery state token</strong>, for verification.</p><p>For example, Google recommends developer to generate and add state parameter in the URL for requesting the authorization code, at the beginning of the OAuth 2.0 flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bu38AacJQD1-P-LM.png" /></figure><p><em>Screenshot from Google OAuth server-side Web Apps documentation</em></p><h3>Steps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QAMOs4-537wKJzrf.png" /></figure><ol><li>At the beginning of the OAuth 2.0 flow, my server requests authorization from Google, by generating a URL, the process here is same as the authorization code flow mentioned before.</li><li>My server creates token, which is an anti-forgery state token and store it in the server. Also, add a state parameter in the URL we have generated. Here&#39;s an example mentioned in Google doc:</li></ol><pre>https://accounts.google.com/o/oauth2/v2/auth?<br> scope=https%3A//www.googleapis.com/auth/drive.metadata.readonly&amp;<br> access_type=offline&amp;<br> include_granted_scopes=true&amp;<br> response_type=code&amp;<br> state=state_parameter_passthrough_value&amp;<br> redirect_uri=https%3A//oauth2.example.com/code&amp;<br> client_id=client_id</pre><p>The token can be a random string with 30 or so characters. Then my server sends the request to Google’s server.</p><p>Finally, Google returns a callback URL with authorization code and state parameter. My server checks if the state token in the URL is same as the state token stored in my server. If yes, we can confirm that the authorization code is really requested by the user and can be used to exchange for an access token and refresh token.</p><h3>Authorization Code Interception Attack</h3><p>Another security risk is that the authorization code may be intercepted by the hacker. Then he can get the access token and the user’s data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p9SNYJXDzp_5fh_y.png" /></figure><p><em>Image from </em><a href="https://datatracker.ietf.org/doc/html/rfc7636#section-1.1"><em>RFC7636</em></a></p><p>As suggested in RFC7636, the security issue happens if:</p><ul><li>A malicious app is registered in the client’s device</li><li>The malicious app registers the same custom URI scheme as the legitimate OAuth 2.0 App’s custom URL, which can handle the response which contains an authorization code</li><li>The hacker knows the client secret. For example, SPA or mobile app will expose client secret to the public</li></ul><p>In step 2 and 3, the communication is protected by TLS and cannot be intercepted by the hacker.</p><h3>Solution: PKCE (Proof Key for Code Exchange)</h3><p>From the issue above, we now know that authorization code is not reliable. To solve the issue, we use PKCE.</p><p>Official definition of PKCE:</p><blockquote><em>PKCE (RFC 7636) is an extension to the Authorization Code flow to prevent CSRF and authorization code injection attacks.</em></blockquote><p>In short, my application can dynamically create 3 things when requesting for an authorization code:</p><ul><li>code_verifier</li><li>code_challenge</li><li>code_challenge_method</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*6r5rSLFJMrcXzhWj.png" /></figure><p><em>Image from </em><a href="https://datatracker.ietf.org/doc/html/rfc7636#section-1.1"><em>RFC7636</em></a></p><ul><li>t(code_verifier) = code_challenge</li><li>t_m = code_challenge_method</li></ul><p>Although PKCE is originally designed for improving authorization code flow, PKCE is can also be used in implicit grant type.</p><h3>Steps</h3><ol><li>My app generates a code_verifier, high-entropy cryptographic random string.</li><li>My app generates a code_challenge, by using one of the following transformations on the code_verifier:<br>- plain (strongly discouraged!) <br>code_challenge = code_verifer<br>- S256 <br>code_challenge = BASE64URL-ENCODE(SHA256(ASCII(code_verifier)))</li><li>My app generates a code_challenge_method, which is S256 here.</li><li>My app sends a request with these 3 values to the authorization server. The server returns an authorization code.</li><li>My app requests an access token from the server, with:<br>- Authorization code<br>-code_verifier</li><li>The authorization server transforms code_verifier with the code_challenge_method I sent to it in step 4. It compares the result with the code_challenge it received before in step 4.</li></ol><p>As a result, if the hacker intercepts the authorization code, he fails to get access token since he cannot intercept the code_verifier I dynamically created in the application.</p><h3>Google OAuth 2.0 API with PKCE</h3><p>In <a href="https://developers.google.com/identity/protocols/oauth2/native-app">Google OAuth 2.0 API</a>, Google supports PKCE in Mobile &amp; Desktop Apps. However, at the time I am writing this article, <strong>it seems that Google OAuth 2.0 API does not support PKCE in JavaScript web apps, such as SPA,</strong> even though implicit flow with PKCE is a recommended practice for SPA. The <a href="https://stackoverflow.com/questions/60724690/using-google-oidc-with-code-flow-and-pkce">stackoverflow</a> discussion here addresses the same idea.</p><h3>PKCE VS state?</h3><p>Now we understand the OAuth 2.0 security issues that handled by state parameter and PKCE. Be aware that <strong>PKCE and state solves different problems, that means they cannot replace each other.</strong></p><ul><li><strong>PKCE: Prevent authorization interception</strong> <br>If my authorization code is stolen by the hacker, he still cannot exchange it for an access token since he cannot steal the code_verifier my app created dynamically</li><li><strong>state parameter: Prevent to use hacker&#39;s authorization code</strong> <br>To ensure the authorization code is really requested by the user, not the hacker, my server creates a random string as &#39;state&#39;. It will be stored in my server and sent to the authorization server. When the authorization returns the callback URL with authorization code to my server, it has to also contain a state parameter with the same value I stored in my server.</li></ul><p>In other words, we can have <strong>both</strong> PKCE and state parameter in our OAuth 2.0 implementation for better security. One example is the Google OAuth 2.0 API for mobile &amp; desktop apps API mentioned above.</p><h3>Summary</h3><p>I summurize what I have learnt about OAuth 2.0 with Google OAuth 2.0 API in this article. It introduces the basics of OAuth 2.0 concept, grant types including Authorization code flow and Implicit flow, the purpose of state and PKCE. Feel free to share your opinions on this topic with me!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed95d3478e1c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cookie and Session (III) — simple express-session authentication example]]></title>
            <link>https://medium.com/@alysachan830/cookie-and-session-iii-simple-express-session-authentication-example-6dedf9d2288b?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dedf9d2288b</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[nodejs]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Wed, 22 Dec 2021 16:34:53 GMT</pubDate>
            <atom:updated>2021-12-23T01:07:28.820Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cookie and Session (III): Simple express-session authentication example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjeVqfEfx9Jz6onXl7PKYQ.png" /></figure><p>My previous two articles about session and cookies:</p><p><a href="https://medium.com/@alysachan830/cookie-and-session-i-the-basic-concept-fea3d52bc8d3">Cookie and Session (I) — The basic concept</a><br><a href="https://medium.com/@alysachan830/cookie-and-session-ii-how-session-works-in-express-session-7e08d102deb8">Cookie and Session (II) — How session works in express-session</a></p><h3>Complete example code</h3><p>Here is <a href="https://github.com/alysachan830/simple-express-session-authentication">my GitHub Repo</a> for this example.</p><h3>Simple express-session authentication example</h3><p>This article focuses on implementation of authentication with express-session, connect-flash, redis and ejs. Here are the features I build in my example:</p><ul><li>User login</li><li>User can visit his profile page which shows his previous login time records</li><li>If user has logged in before, skip the login page and redirect to his profile page</li><li>Inside profile page, user can choose to logout. Then we clear his previous login time records. Redirect him to the login page.</li></ul><p>Before we start, let’s setup the environment.</p><h3>Files</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/0*uCa3QpBv-qFEeciA.png" /></figure><ul><li>app.js: for all server code</li><li>views: all html pages</li><li>.env: for storing express session secret</li><li>.prettierrc (optional): prettier setting for formatting our code</li></ul><h3>Pages we need</h3><p>In this practice, we have to show two pages:</p><ul><li>index.ejs: login page</li><li>profile.ejs: profile page</li></ul><p>ejs is a JavaScript template language for generating a HTML file. You can write JavaScript inside ejs file.</p><h3>Project setup</h3><pre>require(&#39;dotenv&#39;).config()<br>const express = require(&#39;express&#39;)<br>const flash = require(&#39;connect-flash&#39;)<br>const app = express()<br>const port = 3000<br>app.set(&#39;view engine&#39;, &#39;ejs&#39;)</pre><pre>const redis = require(&#39;redis&#39;)<br>var session = require(&#39;express-session&#39;)</pre><pre>let RedisStore = require(&#39;connect-redis&#39;)(session)<br>let redisClient = redis.createClient()</pre><pre>redisClient.on(&#39;error&#39;, (err) =&gt;<br>  console.log(`Fail to connect to redis. ${err}`)<br>)<br>redisClient.on(&#39;connect&#39;, () =&gt; console.log(&#39;Successfully connect to redis&#39;))</pre><pre>const users = [<br>  { id: 111, username: &#39;tom&#39;, password: &#39;123&#39; },<br>  { id: 222, username: &#39;chris&#39;, password: &#39;123&#39; },<br>  { id: 333, username: &#39;mary&#39;, password: &#39;123&#39; },<br>]</pre><pre>app.use(<br>  session({<br>    store: new RedisStore({ client: redisClient }),<br>    secret: process.env.SECRET,<br>    resave: false,<br>    saveUninitialized: false,<br>  })<br>)</pre><pre>app.use(flash())</pre><p><strong>Important note: we should not expose the </strong><strong>secret in session.</strong> It is the key for signing session ID stored in user&#39;s browser. Best practice is mentioned in the <a href="https://github.com/expressjs/session">express-session documentation</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y_yjNQf8RLIMWyBM.png" /></figure><pre>app.use(<br>  session({<br>    store: new RedisStore({ client: redisClient }),<br>    secret: process.env.SECRET,<br>    resave: false,<br>    saveUninitialized: false,<br>  })<br>)</pre><p>To create an en environment variable, install dotenv and create a file named .env:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hh3HltoAOIvNL_XL.png" /></figure><h3>Fake user data for demonstration</h3><pre>const users = [<br>  { id: 111, username: &#39;tom&#39;, password: &#39;123&#39; },<br>  { id: 222, username: &#39;chris&#39;, password: &#39;123&#39; },<br>  { id: 333, username: &#39;mary&#39;, password: &#39;123&#39; },<br>]</pre><p>Be aware that in production level, these information should be stored in database and the <strong>password should be hashed</strong>. It is <strong>strongly not recommended</strong> to store password in plain text, i.e: 123, for security reason.</p><p>Since this practice focuses on express and session, I simply create a few fake data with JavaScript object instead of storing them in a database.</p><p><strong>Let’s start our project!</strong></p><h3>User login page (homepage)</h3><p>The homepage is the login page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/0*Waj83vZ5u28YsCAt.png" /></figure><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  if (req.session.isAuth) {<br>    return res.redirect(&#39;/profile&#39;)<br>  }<br>  const showInputError = req.flash(&#39;showInputError&#39;)[0] || false<br>  res.render(&#39;index&#39;, { showInputError })<br>})</pre><h3>Authentication based on isAuth</h3><p>Firstly, we have to decide whether this user has logged in before, by checking <strong>does the session have </strong><strong>isAuth key and value</strong>.</p><p>Before we continue, you may ask why can we access his session if he has not logged in yet?</p><p>No matter the user has logged in or not, <strong>we can access his session because we set a global session middleware before entering every route</strong>:</p><pre>// session middleware, for creating a session<br>app.use(<br>  session({<br>    store: new RedisStore({ client: redisClient }),<br>    secret: process.env.SECRET,<br>    resave: false,<br>    saveUninitialized: false,<br>  })<br>)</pre><pre>// login page<br>app.get(&#39;/&#39;, (req, res) =&gt; {<br>    ...<br>})</pre><p>As mentioned in my previous article, this middleware creates a new session or get the existing session data from Redis store if the client’s cookie has session ID. Hence, every user has session before entering any routes in our application.</p><p><strong>Then, we decide whether the user has logged in or not before, by checking the </strong><strong>isAuth key and value in his session object.</strong></p><pre>if (req.session.isAuth) {<br>  return res.redirect(&#39;/profile&#39;)<br>}</pre><p>How or when should we add isAuth to the session? The answer will be shown in app.post(&#39;/login&#39;) {...} and explained later. Right now, we only have to understand that isAuth is the property we used for authentication.</p><p>At the end, <strong>if the user has </strong><strong>isAuth in his session, that means he has logged in already.</strong> He can <strong>skip the login page and be redirected to </strong><strong>/profile.</strong> Otherwise, we render the homepage for asking the user to login.</p><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  // ...<br>  const showInputError = req.flash(&#39;showInputError&#39;)[0] || false<br>  res.render(&#39;index&#39;, { showInputError })<br>})</pre><h3>Validate input</h3><p>After the user clicks on the ‘Sign in’ button, the HTML form sends a POST request to our server. We validate the user&#39;s input, by comparing the username and password in the request object sent by the client with the fake data in our server.</p><p>Fake data:</p><pre>const users = [<br>  { id: 111, username: &#39;tom&#39;, password: &#39;123&#39; },<br>  { id: 222, username: &#39;chris&#39;, password: &#39;123&#39; },<br>  { id: 333, username: &#39;mary&#39;, password: &#39;123&#39; },<br>]</pre><p>POST request for login</p><pre>app.post(&#39;/&#39;, (req, res) =&gt; {<br>  const { username, password } = req.body<br><br>  // Prevent empty input<br>  if (username.trim() === &#39;&#39; || password.trim() === &#39;&#39;) {<br>    req.flash(&#39;showInputError&#39;, true)<br>    return res.redirect(&#39;/&#39;)<br>  }<br><br>  // Validate input<br>  const targetUser = users.find(<br>    (user) =&gt; user.username === username &amp;&amp; user.password === password<br>  )<br><br>  // Wrong username or password<br>  if (!targetUser) {<br>    req.flash(&#39;showInputError&#39;, true)<br>    return res.redirect(&#39;/&#39;)<br>  }<br><br>  req.session.isAuth = true<br>  req.session.username = targetUser.username<br>  req.session.timestamps = []<br>  res.redirect(&#39;/profile&#39;)<br>})</pre><p>If the user’s username and password match one of our fake data, he is a valid user and signs in successfully. <strong>As mentioned at the beginning of this article, we check on </strong><strong>isAuth for authentication.</strong> After we validate the user, we <strong>add </strong><strong>isAuth to the user&#39;s session for showing that this user has logged in successfully.</strong></p><p>Meanwhile, <strong>we add </strong><strong>username and </strong><strong>timestamps in the user&#39;s session</strong>, for later to be displayed in the user&#39;s /profile page.</p><p>Let’s try login with the correct username and password</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*eR-BhKvwljxikmSO.png" /></figure><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*Vk5tS8k9cpIpB5jf.png" /></figure><h3>Show error message with connect-flash</h3><p>However, if the users inputs the information incorrectly, we have to:</p><ol><li>Redirect him back to / (i.e login page)</li><li>Render the login page with <strong>an error message.</strong></li></ol><p>To display the error message in the homepage, we have to decide whether we have to show the error, by using connect-flash.</p><h3>connect-flash</h3><p>connect-flash is a library for passing data when redirecting user to a page. The data passed by flash will be removed <strong>after the data is rendered on the page.</strong></p><p>Important notes:</p><ul><li>Flash message data is stored in session. You have to add session middleware before adding flash middleware.</li></ul><pre>app.use(<br>  session({<br>    store: new RedisStore({ client: redisClient }),<br>    secret: process.env.SECRET,<br>    resave: false,<br>    saveUninitialized: false,<br>  })<br>)</pre><pre>app.use(flash())</pre><ul><li>If you call a render function without including flash message data, the data <strong>will still be kept</strong> and will not be removed.</li></ul><h3>Implementation</h3><p>Back to our example, if we receive empty input or incorrect input, we add a flash message to the user’s request object:</p><pre>// Prevent empty input<br>if (username.trim() === &#39;&#39; || password.trim() === &#39;&#39;) {<br>  req.flash(&#39;showInputError&#39;, true)<br>  return res.redirect(&#39;/&#39;)<br>}</pre><pre>//...</pre><pre>// Wrong username or password<br>if (!targetUser) {<br> req.flash(&#39;showInputError&#39;, true)<br> return res.redirect(&#39;/&#39;)<br>}</pre><p>After the user is redirected to the homepage, we check showInputError for deciding whether we have to display error message on the page:</p><pre>app.get(&#39;/&#39;, (req, res) =&gt; {</pre><pre>  if (req.session.isAuth) {<br>    return res.redirect(&#39;/profile&#39;)<br>  }</pre><pre>  // flash uses array to store data<br>  const showInputError = req.flash(&#39;showInputError&#39;)[0] || false<br>  res.render(&#39;index&#39;, { showInputError })</pre><pre>})</pre><p>Let’s try to have empty input.</p><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*C21oUQJFDqkpGwU7.png" /></figure><h3>Can we have other approaches?</h3><p><em>Feel free to skip this part if you are not interested to know the problems of other approaches.</em></p><p>Before using flash message, I thought of two possible approaches for passing showInputError after res.redirect():</p><ul><li>Approach 1: Directly store the value of showErrorInput in session without using flash</li><li>Approach 2: Store the value of showErrorInput in URL query. Instead of redirecting the user to localhost:3000/, redirect him to <a href="http://example.org/?showInputError=true">http://localhost:3000/?showInputError=true</a></li></ul><h3>Approach 1: Store showErrorInput in session directly</h3><p>Before I worked with flash message, I tried another approach, only storing error message in session. Like this:</p><pre>app.post(&#39;/&#39;, (req, res) =&gt; {<br>  const { username, password } = req.body</pre><pre>  // Prevent empty input<br>  if (username.trim() === &#39;&#39; || password.trim() === &#39;&#39;) {<br>    // Directly store showInputError in session<br>    req.session.showInputError = true<br>    return res.redirect(&#39;/&#39;)<br>  }</pre><pre>  // Validate input<br>  const targetUser = users.find(<br>    (user) =&gt; user.username === username &amp;&amp; user.password === password<br>  )</pre><pre>  // Wrong username or password<br>  if (!targetUser) {<br>    // Directly store showInputError in session<br>    req.session.showInputError = true<br>    return res.redirect(&#39;/&#39;)<br>  }</pre><pre>  req.session.showInputError = false<br>  req.session.isAuth = true<br>  req.session.username = targetUser.username<br>  req.session.timestamps = []<br>  res.redirect(&#39;/profile&#39;)<br>})</pre><p>After that, the display of error message will be shown based on the showInputError inside stored session:</p><pre>app.get(&#39;/&#39;, (req, res) =&gt; {</pre><pre>  if (req.session.isAuth) {<br>    return res.redirect(&#39;/profile&#39;)<br>  }</pre><pre>  const showInputError = req.session.showInputError || false</pre><pre>  // Clear session. Otherwise showInputError may still be true all the time<br>  req.session.destroy((err) =&gt; {<br>    if (err) {<br>      return res.redirect(&#39;/&#39;)<br>    }<br>  })</pre><pre>  res.clearCookie(&#39;connect.sid&#39;)<br>  res.render(&#39;index&#39;, { showInputError })<br>})</pre><p>From the code above, we can spot some problems:</p><ul><li>I have to deliberately destroy the session data. Otherwise showInputError will always be true and stored in the user&#39;s session. <strong>When the user refreshes page after login fails for his first time, the page will keep showing </strong><strong>&quot;Invalid username and password!&quot;, instead of clearing the message.</strong></li><li>Normally we use session to store use’s data, not the data for deciding the content of the page (i.e display of error message). Therefore, this approach may look confusing to other developers.</li></ul><h3>Approach 2: Encode showErrorInput in URL</h3><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  if (req.session.isAuth) {<br>    return res.redirect(&#39;/profile&#39;)<br>  }<br>  const showInputError = req.query.showInputError || false<br>  res.render(&#39;index&#39;, { showInputError })<br>})</pre><pre>app.post(&#39;/&#39;, (req, res) =&gt; {<br>  const { username, password } = req.body</pre><pre>  // Prevent empty input<br>  if (username.trim() === &#39;&#39; || password.trim() === &#39;&#39;) {<br>    const query = new URLSearchParams({ showInputError: true }).toString()<br>    return res.redirect(`${req.headers.origin}/?${query}`)<br>  }</pre><pre>  // ...</pre><pre>  // Wrong username or password<br>  if (!targetUser) {<br>    const query = new URLSearchParams({ showInputError: true }).toString()<br>    return res.redirect(`${req.headers.origin}/?${query}`)<br>  }</pre><pre>  // ...<br>})</pre><p>In this approach, if the user’s input is incorrect, we redirect him to an URL with query showInputError=true. Based on the query, we check if we should render the error message.</p><p>This solution is problematic because <strong>if the user stays at </strong><strong>localhost:3000/?showInputError=true and refreshes page, he will still stay at </strong><strong>localhost:3000/?showInputError=true and the page will keep showing the error message.</strong></p><h3>Profile page</h3><p>Back to our example, the user is now redirected to /profile page after executing the following code:</p><pre>app.get(&#39;/profile&#39;, checkIsAuthAndAddTimestamp, (req, res) =&gt; {<br>  const { username, timestamps } = req.session<br>  res.render(&#39;profile&#39;, { username, timestamps })<br>})</pre><p>Before we render /profile, we need to execute a middleware function for authentication. That’s because user may directly access /profile, instead of coming from the homepage.</p><pre>const checkIsAuthAndAddTimestamp = (req, res, next) =&gt; {<br>  if (req.session.isAuth) {<br>    req.session.timestamps.push(new Date().getTime())<br>    next()<br>  } else {<br>    res.redirect(&#39;/&#39;)<br>  }<br>}</pre><h3>Render profile page with EJS variables</h3><p>After going through the middleware, we can render the profile page. It shows the user’s username and his login records, which are stored in his session object before.</p><pre>app.get(&#39;/profile&#39;, checkIsAuthAndAddTimestamp, (req, res) =&gt; {<br>  const { username, timestamps } = req.session<br>  res.render(&#39;profile&#39;, { username, timestamps })<br>})</pre><p>We pass the variables to &#39;profile&#39; EJS template and render them:</p><p>profile.ejs</p><pre>&lt;body&gt;<br>    &lt;h2&gt;&lt;%= username %&gt;&lt;/h2&gt;<br>    &lt;p&gt;Login time records:&lt;/p&gt;<br>    &lt;ul&gt;<br>      &lt;% for (const timestamp of timestamps) { %&gt;<br>      &lt;li&gt;&lt;%= new Date(timestamp) %&gt;&lt;/li&gt;<br>      &lt;% } %&gt;<br>    &lt;/ul&gt;<br>    &lt;form action=&quot;/logout&quot; method=&quot;post&quot;&gt;<br>      &lt;button type=&quot;submit&quot;&gt;Sign out&lt;/button&gt;<br>    &lt;/form&gt;<br>  &lt;/body&gt;</pre><h3>Get session data from Redis</h3><p>Imagine we access &#39;/profile&#39; again now. We can get into the &#39;/profile&#39; page without being asked to login. That&#39;s because <strong>we have stored a session ID in the user&#39;s browser. Its session object has the key and value </strong><strong>isAuth: true</strong></p><p>The process works as mentioned below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xsJjfT411ooq0UPr.png" /></figure><p>In terminal, we can use redis-cli for checking the existing data in Redis. For example, after I login successfully for the first time, Redis should have store a session data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZinPrPq7Poz_Qq2F.png" /></figure><p>The session ID can also be found in my browser’s cookie now:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xQutj49TLXf7LZJX.png" /></figure><p>If I access &#39;/profile&#39; again, the login time record accumulates and all will be displayed on the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*bJGoDLuedwqDNADu.png" /></figure><h3>Log out</h3><p><strong>We are almost done!</strong></p><p>The last feature is to log out when the user clicks on the ‘Sign out’ button. The code here is quite straightforward:</p><pre>app.post(&#39;/logout&#39;, (req, res) =&gt; {<br>  req.session.destroy((err) =&gt; {<br>    if (err) {<br>      return res.redirect(&#39;/profile&#39;)<br>    }<br>  })<br>  res.clearCookie(&#39;connect.sid&#39;)<br>  res.redirect(&#39;/&#39;)<br>})</pre><p>req.session.destroy is a method in express-session for destroying session data and res.clearCookie is a function to clear a specific cookie in the client&#39;s browser.</p><h3>Complete logic of this example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ofZSTtHTeDFn3FtY.png" /></figure><h3>Summary</h3><p>This is a basic example showing how to implement session-based authentication with express-session, connect-redis and ejs. In the next article, I will share my learning on how to connect with mySQL database for storing username and password.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dedf9d2288b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cookie and Session (II) — How session works in express-session]]></title>
            <link>https://medium.com/@alysachan830/cookie-and-session-ii-how-session-works-in-express-session-7e08d102deb8?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e08d102deb8</guid>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Sun, 12 Dec 2021 10:01:46 GMT</pubDate>
            <atom:updated>2021-12-23T01:07:44.728Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cookie and Session (II): How session works in express-session</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-D6Ids2z9ebtz0_m9qeBBA.png" /></figure><p>My previous article, <a href="https://medium.com/@alysachan830/cookie-and-session-i-the-basic-concept-fea3d52bc8d3">Cookie and Session (I) — The basic concept</a>, introduces the basic concept of session, session ID, cookie and how they work for authentication. For beginners, I believe the most difficult part of session and cookie is not about how to issue a session ID, but the whole picture of how session works, such as how should we store and set session ID.</p><p>To have a whole picture of how session works. This article discusses about two questions, by taking express-session as an example:</p><ul><li>How session is created?</li><li>How session is stored?</li></ul><h3>Why do we need session?</h3><p>Before we start, we have to ask why do we need session. The reason is that HTTP protocol is stateless, which means it cannot keep track of user. To solve this problem, the server can create a session for tracking the user, including the data in the requests and responses between the client and server.</p><p>Let’s start with the first question: how to create a session?</p><h3>Question 1: How session is created?</h3><p>Session is created in server when the client send a request to it for the first time. There are different ways to create a session, depends on the language you use. This article focuses on express-session .</p><p>For express-session , we create a session in a middleware.</p><pre>var app = express()<br>var session = require(&#39;express-session&#39;)<br><br>app.use(session({<br>  secret: &#39;keyboardcat&#39;,<br>  resave: false,<br>  saveUninitialized: true,<br>  cookie: { secure: true }<br>}))<br><br>// Access the session as req.session<br>app.get(&#39;/&#39;, function(req, res, next) {<br>	console.log(req.session)<br>  res.send(&#39;Hello World!&#39;)<br>})</pre><p>To make sure we can the incoming request has a session, we should put the session middleware before all of the routes. The middleware will:</p><ul><li>Check if this request has cookie with valid session ID</li><li>If yes, search for the matching session ID in session store and retrieve the data</li><li>If no, create a new session and add it to the request object. Also create a session ID cookie based on the cookie options you set in the middleware</li></ul><p>More details about this process is mentioned in this <a href="https://stackoverflow.com/questions/28789857/how-is-the-express-req-session-object-persisted">stackoverflow</a> discussion.</p><h3>Session options</h3><p>There are options for manipulating the session. Details of each common options such as httpOnly , secret, secure can be found on the <a href="https://github.com/expressjs/session">official documentation</a>.</p><p>Take resave option as an example,express-session will store the session automatically if the session has been altered, because we set resave: false in the session middleware.</p><p><strong>Session.save(callback)</strong></p><blockquote>…This method is automatically called at the end of the HTTP response if the session data has been altered (though this behavior can be altered with various options in the middleware constructor). Because of this, typically this method does not need to be called.</blockquote><p>If the session is not altered, the session ID will not be set in the cookie.</p><h3>Session ID</h3><p>Besides the session object, a session ID is necessary for searching a particular session that stored in the session store. The process works like:</p><p>client → GET request (with session ID in cookie) → server → search for this user’s session with his session ID</p><p>When a client visits the website for the first time, he does not have a session ID in his cookie. Thus, after he sends his first GET request to the server, the server has to set a session ID in his cookie.</p><h4>So how do we set session ID in cookie?</h4><p>Session ID will be set in cookie by a session middleware. Be careful of theresave option. As mentioned above, with resave: false option, <strong>the session ID will not be set in cookie if the session has not been altered.</strong></p><p>Also, <strong>be aware that if you use local server, you should not follow the official example that sets</strong> cookie: { secure: true } . This options will not allow setting cookie in HTTP website. For example, you can just delete cookie: { secure: true } :</p><pre>app.use(session({<br>  secret: &#39;keyboard cat&#39;,<br>  resave: false,<br>  saveUninitialized: false,<br>}))</pre><p>After accessing to localhost 3000 (I use port 3000 for example), the session ID is set. We can check it in the browser by:</p><p>Right click &gt; Inspect &gt; Application &gt; Storage (on the left side) &gt; Cookies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jhz7yhP2e6KXRKvagUksCg.png" /></figure><p>By default the key is connect.sid . We can modify the name option for customising the name of the key:</p><pre>app.use(session({<br>  secret: &#39;keyboard cat&#39;,<br>  name: &#39;test&#39;, // Customise the name to &#39;test&#39;<br>  resave: false,<br>  saveUninitialized: false,<br>}))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74WpD_BLwmHBkR-HjTK41A.png" /></figure><p>Now the session ID is saved in the user’s cookie.</p><h4>But how session ID is generated?</h4><p>The session ID is generated by uid-safe library and stored in session store by default. You can assign another function for generating session ID in genid option.</p><p>However, the session ID stored in server looks different from the id we store in our user’s cookie.</p><p>Let’s take a look at the session ID in server’s memory store:</p><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>    req.session.foo = &#39;foo&#39;<br>    console.log(req)<br>    res.send(&#39;Hello World!&#39;)<br>})</pre><p>The session ID is oyd0L2r... :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UTouY5ds7ChDUgDwyy-OA.png" /></figure><p>Compared to the value in my browser’s cookie:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lpFvP2a5-9garkcc8n_IKA.png" /></figure><p>s%3A is added at the beginning and .V2... is added at the end. The reason is that express-session uses cookie-signature library and your secret option to generate the value and sign the cookie. The mechanism is shown in the source code.</p><p><em>Feel free to skip this part if you are not interested in reading source code.</em></p><p>After reading this <a href="https://github.com/aszx87410/blog/issues/46">article</a> (in Chinese) about how session ID is generated, I have a brief idea on this question. There are mainly two steps for creating a session ID:</p><ul><li>Generate a UID by using uid-safe</li><li>Create a signature with the UID and your secret by using cookie-signature</li></ul><p>express-session / index.js</p><pre>function session(options) {<br>  // get the session id generate function<br>  var generateId = opts.genid || generateSessionId,<br>}</pre><p>Generate a UID</p><pre>function generateSessionId(sess) {<br>  return uid(24);<br>}</pre><p>Create a signature with that UID and your secret</p><pre>/**<br> * Set cookie on response.<br> *<br> * @private<br> */</pre><pre>function setcookie(res, name, val, secret, options) {<br>  var signed = &#39;s:&#39; + signature.sign(val, secret);<br>  var data = cookie.serialize(name, signed, options);</pre><pre>  debug(&#39;set-cookie %s&#39;, data);</pre><pre>  var prev = res.getHeader(&#39;Set-Cookie&#39;) || []<br>  var header = Array.isArray(prev) ? prev.concat(data) : [prev, data];</pre><pre>  res.setHeader(&#39;Set-Cookie&#39;, header)<br>}</pre><p>node-cookie-signture / index.js</p><pre>var crypto = require(&#39;crypto&#39;);</pre><pre>/**<br> * Sign the given `val` with `secret`.<br> *<br> * @param {String} val<br> * @param {String} secret<br> * @return {String}<br> * @api private<br> */</pre><pre>exports.sign = function(val, secret){<br>  if (&#39;string&#39; != typeof val) throw new TypeError(&quot;Cookie value must be provided as a string.&quot;);<br>  if (&#39;string&#39; != typeof secret) throw new TypeError(&quot;Secret string must be provided.&quot;);<br>  return val + &#39;.&#39; + crypto<br>    .createHmac(&#39;sha256&#39;, secret)<br>    .update(val)<br>    .digest(&#39;base64&#39;)<br>    .replace(/\\=+$/, &#39;&#39;);<br>};</pre><p>The result will be s: +UID + . + crypto.createHmac(...) . Colon is a special character, which equals to s%3A (percent % is an escape character). That&#39;s why the value stored in cookie starts withs%3A .</p><p>Additionally, in the source code above, we need a secret for creating a signature. That’s the reason why we <strong>should not</strong> reveal our secret in production. We should hide it with using <strong>environment variable</strong>:</p><pre>app.use(session({<br>  secret: process.env.SECRET,<br>  ...<br>}))</pre><p>Since the signature depends on the secret, be careful <strong>if you change your </strong><strong>secret , all the cookies that were set previously will be invalid.</strong></p><p>More details about secret can be found in the express-session official document.</p><h3>Question 2: How session is stored?</h3><p>We now know how session and session ID are created. But how do we store them, such that we can validate the session ID sent by the client? The common options are in-memory store or database.</p><p>By default, express-session creates a new memory store instance for storing session data in server.</p><p>A simple example:</p><pre>const express = require(&#39;express&#39;)<br>const app = express()<br>const port = 3000</pre><pre>var session = require(&#39;express-session&#39;)</pre><pre>app.use(session<br>  secret: &#39;keyboard cat&#39;,<br>  resave: false,<br>  saveUninitialized: false,<br>})) </pre><pre>// No store option is provided,<br>// so express-session will create a MemoryStore instance for storing session here</pre><pre>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  req.session.foo = &#39;some text here&#39;<br>  res.send(&#39;Hello World!&#39;)<br>  console.log(req);<br>})</pre><p>Console log message in server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0pngzIyqHCyY6aph_3HSA.png" /></figure><p>In the request object, there is a property called sessionStore , with a MemoryStore instance as a value. That’s where we store our session by default.</p><p>Session ID I set in cookie:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kl8l-vKW1lyR55Fe6mYXMA.png" /></figure><h4>So what is session store?</h4><p>Session store is a place where we store session data on server.</p><p>By default, express-session creates a new MemoryStore instance for storing session data in server.</p><p>Source code from express-session :</p><pre>function session(options) {  <br>  // get the session store<br>  var store = opts.store || new MemoryStore()<br>}</pre><p><strong>However, in production, it is not recommended to use the default memory store, as mentioned in the official documentation. </strong>We should use other modules, such as connect-redis , a Redis-based session store.</p><p>To store session, it is common to use a in-memory store or database. Brief comparison here:</p><p><strong>Pros of using in-memory store</strong></p><ul><li>Fast read and write speed since in-memory store stores and retrieves data from RAM</li><li>Database may have disk I/O issue if the server keeps writing in and reading session data from the database, which results in database performance issue</li></ul><p><strong>Cons of using in-memory store</strong></p><ul><li>Risk of losing data. When we relaunch the app or the app is crashed, we may lose the data in in-memory store</li></ul><p>This <a href="https://www.youtube.com/watch?v=p3q5zWCw8J4&amp;ab_channel=TED-Ed">TED</a> video clearly explains how memory works in computer.</p><h4>connect-redis implemtation</h4><p>To start with connect-redis , make sure you have installed Redis already.</p><p>After that, install all required packages in your project, as mentioned in the connect-redis documentation:</p><pre>npm install redis@v3 connect-redis express-session</pre><p>app.js</p><pre>const express = require(&#39;express&#39;)<br>const app = express()<br>const port = 3000<br><br>const redis = require(&#39;redis&#39;)<br>var session = require(&#39;express-session&#39;)<br><br>let RedisStore = require(&#39;connect-redis&#39;)(session)<br>let redisClient = redis.createClient()<br><br>redisClient.on(&#39;error&#39;, (err) =&gt; console.log(`Fail to connect with redis. ${err}`));<br>redisClient.on(&#39;connect&#39;, () =&gt; console.log(&#39;Successful to connect with redis&#39;));<br><br>app.use(session({<br>  store: new RedisStore({ client: redisClient }),<br>  secret: &#39;keyboard cat&#39;,<br>  resave: false,<br>  saveUninitialized: false,<br>}))<br><br>app.get(&#39;/&#39;, (req, res) =&gt; {<br>  req.session.foo = &#39;some text here&#39; // To imitate alter the session <br>  res.send(&#39;Hello World!&#39;)<br>})<br><br>app.listen(port, () =&gt; {<br>  console.log(`Example app listening at &lt;http://localhost&gt;:${port}`)<br>})</pre><p>Few important notes here:</p><ul><li>Because resave is false , new session without any modification will not be stored. That&#39;s why req.session.foo = &#39;some text here&#39; is required</li><li>The default expire date of data in session store is one day (86400 seconds). It can be customised by ttl option</li></ul><p>Let’s run the program by typing node app.js in terminal.</p><ol><li>Access localhost:3000</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_4HOgUIByJXE-brvhcixdg.png" /></figure><p>2. Check if the session ID is set in cookie or not.</p><p>Inspect &gt; Application &gt; Storage (on the left side) &gt; Cookies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ulAOqHJvoDY_-wZ8L_pZw.png" /></figure><p>3. Check if the session data is stored in Redis by accessing redis-cli in terminal</p><pre>redis-cli</pre><p>Check all available keys and get the session data</p><pre>KEYS *</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*jABS2qOrK8DJ7xurDOmQIQ.png" /></figure><pre>GET &quot;sess:vUj5V4Du8OmK2jc067-SOtC9j-0UQPjU&quot;</pre><p>Result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHZoxJRHSOxNaSHe_hzMAg.png" /></figure><h3>Summary</h3><p>To understand the basic of how session works, we go through two questions in this article, by taking express-session as an example:</p><ul><li>How session is create?</li></ul><p>Session is created in a session middleware.</p><ul><li>How session is stored?</li></ul><p>Session is stored in MemoryStore, an instance created by the middleware by default. This implementation is not suggested for production. We can use external in-memory store such as Redis for storing session.</p><p>In the next article, we will focus on login and logout example using express-session and connect-redis.</p><h3>Reference</h3><p><a href="https://gabrieleromanato.name/creating-and-managing-sessions-in-expressjs">Creating and managing sessions in ExpressJS</a></p><p><a href="https://medium.com/swlh/session-management-in-nodejs-using-redis-as-session-store-64186112aa9">Session Management in Nodejs Using Redis as Session Store</a></p><p><a href="https://www.section.io/engineering-education/session-management-in-nodejs-using-expressjs-and-express-session/">Session Management in Node.js using ExpressJS and Express Session</a></p><p><a href="https://github.com/aszx87410/blog/issues/46">深入 Session 與 Cookie：Express、PHP 與 Rails 的實作</a></p><p><a href="https://medium.com/johnny%E7%9A%84%E8%BD%89%E8%81%B7%E5%B7%A5%E7%A8%8B%E5%B8%AB%E7%AD%86%E8%A8%98/node-js-cookie-session%E9%A9%97%E8%AD%89%E5%8E%9F%E7%90%86%E4%BB%A5%E5%8F%8Aexpress-session%E5%A5%97%E4%BB%B6%E4%BD%BF%E7%94%A8-aeafa386837e">[Node.js] cookie-session驗證原理以及express-session套件使用</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e08d102deb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cookie and Session (I) — The basic concept]]></title>
            <link>https://medium.com/@alysachan830/cookie-and-session-i-the-basic-concept-fea3d52bc8d3?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/fea3d52bc8d3</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 14:00:58 GMT</pubDate>
            <atom:updated>2021-12-23T01:08:00.610Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cookie and Session (I): The basic concept</h3><h3>Why do we need cookie?</h3><p>Cookie solves the problem of stateless HTTP request.</p><p>Every HTTP request is a new request. When you visit a website. The server cannot recognise who you are even though you may have visited it before. It is because <strong>every request from the client (also called browser or user agent) to the server is treated as a new request.</strong></p><p><strong>That is why we describe HTTP request as‘stateless’.</strong></p><p>This feature causes difficulty for developing web applications, such as e-commerce website that keeps track of customers’ shopping carts. Yet, due to the stateless feature, the customer will lose their shopping cart data after closing the browser.</p><p>Therefore, the RFC about cookie is named <strong>HTTP State Management Mechanism</strong>, aiming to solve the problem of stateless HTTP request. Moreover, the RFC introduced the term <strong>‘session’</strong> and two new headers in HTTP request and response, <strong>Cookie and Set-cookie.</strong></p><p><a href="https://datatracker.ietf.org/doc/html/rfc2109">RFC 2109</a> — HTTP State Management Mechanism — 1997 Feb</p><blockquote><em>This document describes a way to create stateful sessions with HTTP requests and responses. Currently, HTTP servers respond to each client request without relating that request to previous or subsequent requests; the technique allows clients and servers that wish to exchange state information to place HTTP requests and responses within a larger context, which we term a “session”.</em></blockquote><p>In other words, ‘session’ means a context, which involves an interaction between the server and client. It can maintain and exchange a user’s state information.</p><p>Key attributes of session mentioned in <a href="https://datatracker.ietf.org/doc/html/rfc2109">RFC 2109</a>:</p><blockquote><em>Each session has a beginning and an end.Each session is relatively short-lived.Either the user agent or the origin server may terminate a session.The session is implicit in the exchange of state information.</em></blockquote><p>In short, session allows server and client exchange state information within a particular period of time. It can also be terminated by the client or server.</p><h3>How does cookie work?</h3><p>Cookie can be set by:</p><ul><li>Server with using Set-cookie header.</li><li>Using JavaScript document.cookie.</li></ul><p>If a cookie is set with HttpOnly, it cannot be edited by scripting languages.</p><h3>The whole picture of how cookie and session are used</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bejurH3uZoTrZita.png" /></figure><p>The picture above describes implementing authentication by using cookie and session. It is called <strong>session-based authentication.</strong></p><h3>The first time to login</h3><ol><li>Send POST request to the server, with your username and password</li><li>Server receives request. <strong>Create session ID ‘abc123’ as your unique identifier</strong></li><li>Store the sessionId in database</li><li>Return response with Set-cookie: sessionId=abc123, for setting the session ID in your browser</li></ol><h3>The next time you login</h3><ol><li>Browser sends a POST request to the server, with session ID stored in your browser’s cookie</li><li>Server searches for your information based on your session ID</li><li>Server checks if the session ID is valid or not</li><li>Server returns your information through a response</li></ol><p>The main difference is that the server will create a session ID if you login for the first time.</p><p>The purpose of session ID will be discussed later in this article.</p><h3>Details</h3><p>After understanding the basic concepts, we can take a look at a more detailed steps of how server and client exchange session (state information of the user), including:</p><ol><li>How does server set cookie? (Server → Client)</li><li>How does client send cookie to server? (Client → Server)</li></ol><p>The example below takes reference from <a href="https://en.wikipedia.org/wiki/HTTP_cookie#Setting_a_cookie">Wikipedia</a>. Compared to the previous example, it demonstrates setting more than one cookie in the response.</p><h3>How does server sets cookie? (Server → client)</h3><ol><li>Client sends a HTTP request to the homepage of <a href="http://www.example.org/">www.example.org</a></li></ol><pre>GET/index.htmlHTTP/1.1<br>Host: www.example.org<br>...</pre><p>2. Server responses with two Set-cookie headers</p><pre>HTTP/1.0 200OKContent-type: text/html<br>Set-Cookie: theme=light<br>Set-Cookie: sessionID=abc123; Expires=Wed, 09 Jun 2021 10:18:14 GMT<br>...</pre><p>The server uses Set-Cookie HTTP-header in the response for setting cookie.</p><p>The two cookies are different. theme=light and sessionToken=abc123 are called session cookie and persistent cookie respectively. The only difference here is whether the Expires attribute is set.</p><ul><li>Session cookie</li></ul><p>No Expires attribute is set. When the browser is closed, the cookie will be deleted.</p><ul><li>Persistent cookie</li></ul><p>A cookie that will be deleted by the browser based on its Expires attribute. When the browser is closed, it will not be deleted.</p><h4>Server creates session ID for each user</h4><p>In the above response, server creates a session ID abc123 and attaches it in the response. Session ID works as an &#39;unique identifier&#39; for identitying each user. The server will store the user&#39;s information with his unique session ID in database.</p><p>3. Client visits spec.html page on www.example.org website</p><pre>GET/spec.htmlHTTP/1.1<br>Host: www.example.org<br>Cookie: theme=light; sessionID=abc123<br>…</pre><p>When the user visits another page <a href="http://www.example.org/">s</a>pec.html in the website, server can retrieve his information based on the session ID stored in the user’s cookie.</p><p>Now we know how server sets cookies in response. The following explanation is about how client sends cookie through a HTTP request.</p><h3>How does client sends cookie to server? (Client → Server)</h3><h3>Steps</h3><p>Steps of sending a request with Cookie request header:</p><ol><li>Client sends a HTTP request to the server</li><li>Browser checks with all stored cookies’ attributes, such as Domain, Path, Expires and HttpOnly attributes. For example, domain and path attribute show what website does the cookie belong to. <strong>Be aware that the browser does not include these attributes in the request, they are only used by browser for deciding whether the cookie will be included in the request</strong></li><li>Browser finds out the cookie(s) that fulfil the requirement, such as having an unexpired, correct domain, path and HttpOnly setting</li><li>Browser sends the correct cookie(s) in the Cookie header, only with their names and values</li></ol><p>Cookie header is sent with the following key-value pair format, with semicolon and space ; as the separator:</p><p>Cookie: name=value; name2=value2; name3=value3</p><p><strong>Be aware that you cannot set cookie that is different from your domain.</strong></p><p><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie">MDN</a> mentions:</p><blockquote><em>A cookie for a domain that does not include the server that set it should be rejected by the user agent</em></blockquote><p>User agent means client or browser. For example, a server hosted on example.com cannot set cookie with anotherweb.com or foo.example.com.</p><p>Additionally, cookie header is <strong>optional</strong> and <strong>can be omitted</strong>. For example, it can be done by the browser’s privacy setting block cookies.</p><h3>Cookie is not the only solution</h3><p>Other than cookie, there are alternatives for maintaining a user’s state .</p><p>In <a href="https://datatracker.ietf.org/doc/html/rfc6265">RFC 6265</a> (the latest RFC), it mentions a few solutions other than cookie and points out the disadvantages of them:</p><h4>Embed state information in URLs</h4><ul><li>It is unreliable. If user clicks the back button, the user will roll back to the previous state. It would be problematic for a shopping website if it adds or removes shopping cart items by changing the URL. If a user clicks the back button, he may accidentally add or remove an item.</li><li>Embedding state information in URL is unfriendly to web caches.</li><li>It may causes unexpected result if the client can edit the URL, which is sent to the server.</li></ul><h4>Use hidden fields in HTML forms</h4><ul><li>Same concern here. It may causes unexpected result if the client can edit the information which is sent to the server.</li></ul><h4>Use client’s Internet Protocol (IP) address</h4><ul><li>It is unreliable to identity user or computer based on its IP address. For example, if you use an HTTP Proxy, the server only knows the proxy’s IP address, instead of your computer’s. All of the proxy users will be treated as one user to a server. In addition, if you use a temporary IP address every time given by an ISP when you connect with it every time, you will appear to be a different user.</li></ul><p>To avoid these disadvantages, cookie is a more friendly option for exchanging state information between server and client.</p><h3>First-party cookie &amp; third-party cookie</h3><h3>First-party cookie</h3><p>Cookie created by the host domain. Normally first-party cookie helps improving user experience of the website. For example, first-party cookie helps tracking your shopping cart items or skip login pages if you have logged in before.</p><h3>Third-party cookie</h3><p>Third party cookie means cookie that is sent by a server which is not the connected by the user. For example, when you visit example.com, it may contain a JavaScript file from a different domain, ad.example.com . It sets cookie in its response to your browser, for tracking your actions on example.com , mainly for advertising purpose. By analysing your actions recorded by the cookie, advertising company can easily post Ads based on your interest.</p><p><strong>Recent RFC suggests letting browsers to decide their own third-party cookie policy.</strong> By default, Google Chrome, Firefox allows setting third-party cookies but newer version Safari does not.</p><h3>Be aware of the size of cookie</h3><p>If the size of cookie is too large, it will slow down the connection. For example, if a website connects with multiple servers and each request contains a cookie. The total size of the cookies may slow down the connection.</p><h3>Limitations of cookie</h3><p>A browser can:</p><ul><li>Support cookies with 4KB at maximum.</li><li>Support at least 50 cookies per domain.</li><li>Support at least 3000 cookies in total.</li></ul><h3>Summary</h3><ul><li>With session, client and server exchange user’s state information with cookie within a particular period of time.</li><li>Session-based authentication means authenticating a user by his session ID stored in cookie in his browser. The server retrieves his information by finding out the matching session ID stored in database.</li><li>Session ID is a unique id for each user. It is created in server and used for identifying user.</li><li>Server uses Set-cookie HTTP response header for setting cookie. Browser users Cookie header in HTTP request to send cookie to the server.</li><li>Browser bans cookie with domain attribute that does not match the domain of the host server.</li></ul><p>This article summarises the basic concept of cookie, session and sessionID . The next article will discuss the security problems of cookie and session-based authentication.</p><h3>References</h3><p><a href="https://cookie-script.com/all-you-need-to-know-about-third-party-cookies.html">All you need to know about Third-Party Cookies</a></p><p><a href="https://arxiv.org/abs/cs/0105018">HTTP Cookies: Standards, Privacy, and Politics</a></p><p><a href="https://en.wikipedia.org/wiki/HTTP_cookie#Setting_a_cookie">HTTP cookie</a></p><p><a href="https://blog.huli.tw/2019/08/09/session-and-cookie-part2/">淺談 Session 與 Cookie：一起來讀 RFC</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fea3d52bc8d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[非本科轉職香港軟體工程師 — 求職與面試心得]]></title>
            <link>https://medium.com/@alysachan830/%E9%9D%9E%E6%9C%AC%E7%A7%91%E8%BD%89%E8%81%B7%E9%A6%99%E6%B8%AF%E8%BB%9F%E9%AB%94%E5%B7%A5%E7%A8%8B%E5%B8%AB-%E6%B1%82%E8%81%B7%E8%88%87%E9%9D%A2%E8%A9%A6%E5%BF%83%E5%BE%97-24104ffe02fb?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/24104ffe02fb</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[front-end-development]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 05:03:37 GMT</pubDate>
            <atom:updated>2021-11-09T12:13:31.085Z</atom:updated>
            <content:encoded><![CDATA[<h3>非本科轉職香港軟體工程師 — 求職與面試心得</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONBCtmOoJPBinEXmhqqjBw.png" /><figcaption>個人 <a href="https://www.linkedin.com/in/alysa-chan/">LinkedIn</a></figcaption></figure><p>最近很幸運獲得成為全職軟體工程師的機會，因此在這段期間自己想整理一些求職心得，並分享給志同道合的人。因為過去我很幸運受到一些幫助才可以踏上程式開發的旅程。所以當我有所成長和學習時，我也想積極幫助曾經跟我經歷相似的人，也算是一種報恩吧。</p><p>但要留意的是，這篇不會談及面試題目。而且我目前只是面試了 4 間公司，投寄了大概 10 封以內的求職信，所以求職經驗非常淺薄，以下只會按個人經歷來分享一些心得。</p><h3>個人背景</h3><p>2019 年從香港本地大學畢業，科系是視覺藝術，主攻設計領域。畢業後成為設計師大概一年，工作上開始處理網頁設計，並寫了一點 CSS，開始對寫程式和開發感興趣，覺得寫邏輯和開發功能，比起只把畫面做好更有成功感和難度更高。在疫情剛爆發下，我多了時間學習和思考其他發展方向，於是下班後再學習前端開發。直到去年 8 月決心辭去工作，往程式開發方面發展，投放大量時間在線上課程及bootcamp 學習，並與社群裏的老師和同學持續交流，開始了全職學習寫程式的日子。</p><p>從零開始由 HTML 學到 JavaScript，並上手 Vue.js 和 Nuxt.js，甚至接觸了一點點後端 Node.js 和資料庫。因緣際會下在今年年中開始接下第一份遠端的前端專案，與台灣的後端工程師合作開發 LINE Chatbot，直到 10 月尾才開始在本地求職，目標是找全職 junior 前端工程師或軟體工程師工作。</p><h3>求職準備</h3><p>在正式寄履歷之前，我主要在台灣的線上社群六角學院學習了一年前端開發，期間再透過遠端接案來提升實戰力。在技術學習以外，我做了以下的求職準備：</p><ul><li>開發個人網站顯示個人作品，並整理相關 GitHub Repo 的 Readme。</li><li>辦好 LinkedIn 帳戶和在裏面尋找職缺。</li><li>把履歷和個人網站，一併在 Bootcamp 裏的大群裏公開分享，讓大家一同檢閱。並有有按校長建議修改。</li><li>諮詢了 3 位本地工程師，當中包括資深工程師，檢視自己的求職作品和能力程度。除了本地，當然我也有諮詢台灣那邊的工程師朋友們和線上 Bootcamp 的校長。</li><li>參加本地技術交流會，認識本地行業和市場，並主動打聽職缺。</li></ul><p>在台灣的話，我在 Bootcamp 的社群上觀察到很多同學只需要準備好大概 2 份作品，並弄好 CakeResume，就可以開始寄履歷找工作了，不需要開發個人網站和使用 LinkedIn。但因為我是在香港求職，所以做法跟台灣的同學有點不一樣，例如香港好像不太會用 CakeResume，主要用 LinkedIn。</p><blockquote><em>你可能會問：為什麼一開始不找香港本地的 Bootcamp？</em></blockquote><p>因為我負擔不起。本地有名氣的 Bootcamp 課程至少需要港幣一萬元以上。但台灣的 Bootcamp 相對便宜，而且社群對新手很友善。另外本身自己性格比較積極進取，除了英語，也能流暢地用國語溝通，所以很快就主動認識到一些志同道合的學習伙伴，例如在 Bootcamp 期間我會自發每週辦線上技術分享會，在 Discord 邀請同組組員分享技術和學習困難等，更誇張是我當時講國語的時間比講母語廣東話還多。</p><p>回到重點，我覺得以上提到每一點的準備都是<strong>同等重要</strong>。以下再詳細分享一下。</p><h3>開發個人網站和整理作品</h3><p>這是我的個人網站，用來一次過展示個人作品和自己的技術文章：<br><a href="https://alysachan.com/">https://alysachan.com/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDFi59wDsGY2gwM-Z6mT7A.png" /><figcaption>作品列表</figcaption></figure><h3>個人網站能展示求職的誠意</h3><p>一開始我有點猶豫是否要花時間開發個人網站，但當時我認識的一些台灣工程師朋友都認同這項目會加分，因此我也花時間完成了。的確，我之後在 LinkedIn 觀察到沒有很多香港工程師有開發個人網站，如果有個人網站能即時顯示個人作品的重點，也能突顯個人誠意和態度。尤其對於非本科轉職工程師的人來說，可能會因為不是本科背景而被刷掉，所以建議盡力做好能加分的項目，爭取面試機會。</p><p><strong>你可能會問，建議使用什麼工具來開發？</strong></p><p>在一次的 Bootcamp 的直播分享裏，我主動請教 Kuro 老師（台灣一位很知名的前端大師），建議使用什麼框架來開發個人作品網站。他當時的答案是<strong>使用自己最熟悉的工具</strong>，不要用不熟的。因為個人網站是火力展示，如果面試官覺得你的網站好像卡卡的，那麼就可能有反效果。最後我決定使用自己比較熟的 Nuxt.js （Nuxt 2）來開發個人網站。</p><h3>面試官很忙，沒時間逐一打開你的 GitHub Repo</h3><p>常見台灣同學的做法是直接在 CakeResume 加入自己作品的連結，但香港的開發者好像沒有習慣在履歷附上個人作品連結。可是，如果只把整個 GitHub 給面試官，除非你本身很厲害，有參與開發有名的項目（建議你不用再花時間看完這篇文章了），否則面試官不會有時間逐一打開你的 Repo 來了解你過去的作品。因此，<strong>在個人網站裏就可以迅速向面試官展示你的作品重點，並附圖解釋功能、開發流程、甚至如何克服當中的挑戰</strong>。這都是面試官會感興趣的地方。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3OxmTKeDS1GCIlPz.png" /></figure><p><em>附上流程圖解釋自己在開發時如何解決問題，內容</em><a href="https://alysachan.com/works/flash-ticketing"><em>在此</em></a></p><p>當然，個人網站只是其中一種做法，我也有聽聞只把 GitHub 整理好，把重點都寫好在 Readme 裏就行，這也是看個人喜好。對於台灣的求職者，個人網站可能只是附帶的加分項目，所以建議自己考慮是否需要開發。</p><h3>公開分享自己的履歷和作品</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4ybBM9PREh_QHmSX.png" /><figcaption>在 Slack 上的大群分享自己的網站與 LinkedIn，希望獲得更多意見</figcaption></figure><p>在剛剛準備好履歷後，我就在 Slack 的六角學院社群，以及個人臉書上主動公開自己的<a href="https://alysachan.com/">個人網站</a>和 <a href="https://www.linkedin.com/in/alysa-chan/">LinkedIn</a>，好處是：</p><ul><li>收集更多意見，讓群組裏一共 4000 多人一同檢視你的作品和履歷。</li><li>尋求潛在的合作機會。</li><li>主動邀請別人加 LinkedIn，建立更多工作關係。</li></ul><p>以上三點都是我的親身收穫。<strong>最大收穫當然是讓閱歷無數的校長對我的內容提出改善意見，我大概按他意見修改兩次才敢投履歷。</strong>與此同時，我也有把內容分享給合作過的資深工程師作檢視，確保內容無問題。當時那位工程師，以及校長也同時提到，依他們的觀察，認為如果以台灣來說我大概已經是有一年前端工程師的經驗了，這大大提升了我投履歷時的信心。我之後也在臉書上公開了這些內容，並邀請別人 connect 自己的 <a href="https://www.linkedin.com/in/alysa-chan/">LinkedIn</a>，也意外收到私訊邀請合作。</p><p>我也有聽聞有同學在自學期間已主動公開履歷和作品，並因此更早找到工作機會！詳細可看這位同學的<a href="https://vocus.cc/article/615660c5fd89780001769c6c">求職心得</a>。</p><h3>主動邀請本地在職工程師，檢視自己的學習和求職內容</h3><p>因為我是在香港求職，所以不能單靠在台灣社群的資源來準備求職。很幸運地我發現自己身邊的確有朋友認識一些本地工程師，因此可以幫助聯絡。<strong>為了確保我的學習 roadmap，以及開發所用的工具和知識都是合乎香港本地市場的需求</strong>，我需要主動請教他們在香港求職的問題。問題例如香港偏用 React.js，但目前我是學 Vue.js 為主，會不會影響求職？透過 Zoom 或電話方式，分享自己目前所開發過的專案，以及學習的程度，邀請他們檢視我的做法是否合乎一個本地 junior 前端工程師的要求，同時也請教很多找公司和面試的問題。</p><p><strong>如果身邊沒有工程師朋友，建議從社群或參加實體技術交流會來認識一些在職工程師</strong>，我曾經在一些實體技術交流會上，完場時主動與講者和參加者互動，幸好很多人對資淺的開發者都很友善，於是我主動請教了很多問題，收穫絕對不少。六角的線上社群本身也有很多樂於分享的工程師，因為我在社群上主動問問題和分享，才認識到一些較資深的工程師。在自學過程中，我一直請教他們很多問題，真的非常感激他們！</p><h3>參加本地技術交流會，甚至主動即場面試</h3><p><strong>有求職需求的話，可以把技術交流會當作是尋找職缺的機會，甚至可以即場展示個人作品和履歷。</strong></p><p>在一次本地的技術交流會中，剛好有一些公司的負責人和開發者到場分享，並表示自己公司在徵才。因此在完場時，我再主動前去詢問有關徵才的事，並多了解他們的公司，繼而主動詢問可否即場展示自己的作品和工作經驗給他們，最後更意外開始第一場的面試。我還記得當時技術交流會的內容我很多都還沒學會，但幸好當時我去了，所以意外獲得一些求職機會。<strong>所以建議有同樣求職需求的人，把握每一次實體交流會的機會，除了學習技術之外，也可主動留意是否有徵才資訊。即使沒有，也建議主動與場內的開發者交流，說不定有新的合作機會。</strong></p><p>有些人可能擔心自己還沒準備好，或者怕被拒絕。當時我也有這樣的想法，<strong>但後來覺得機會就在現在，錯過就沒了，而且我永遠不會覺得自己是 100% 準備好</strong>，加上即使當場被拒絕了，別人都很忙碌，過幾天後也不會記得你，所以我最後還是決定即場試試看。</p><h3>面試就是不斷碰壁和調整方向的過程</h3><p><strong>先講結論，面試過程不累，但準備面試的期間挺累。特別是心理上，要不斷克服自我懷疑的壓力。</strong> 但面試多了，遇上各種狀況，例如試過即場 dead air 答不上問題、要即場示範白板題、或筆試被考到完全不懂的技術等等。即使明知會有問題答不上，但因為經歷多了，明知會失敗也會表現得愈來愈淡定。</p><p>目前為止，我大概學到以下的事：</p><ul><li>有些面試官看重你的想法和分析問題的思路，不是要單一答案。</li><li>不用等面試官問你，也可以主動找機會展示你最熟的知識。</li><li>多強調自己的非本科背景，比本科生的優勢在哪裏。</li><li>每間公司看重的技能都不同，不能一概而論什麼題目是必考。</li><li>可以與別人賽後檢討，但要學會趕快收拾心情，以平常心面對。因為錄取與否不一定與你的表現有關，可能是該公司有其他考慮。</li><li>較有興趣的公司，放到較後才去應徵，因為剛面試的狀態不會是最好，建議先用幾場面試來熱熱身。</li></ul><p>以上建議相信大家有聽過，但有些點我會再解釋。</p><h3>不用等面試官問你，也可以主動找機會展示你最熟的知識</h3><p>對於新手開發者來說，我們的知識可能很不夠，但總有一些知識你可能是比較熟，例如你很熟悉 SPA 和 SSR 的分別，以及它們運作的原理，或者你曾經在過去的專案中解決過一些印象深刻的問題，對該問題頗有心得。但面試官不是你，他根本不知道這些，更可能由頭到尾都沒問過這些問題。</p><p>因此，<strong>可嘗試鋪排一些機會來主動展示你最熟的知識，而不是被動地等他問你。例如在展示個人作品時，刻意多放時間解釋一些你很熟的知識點，展示你對某些技術的理解，或者強調自己在開發上很深刻的難題等等。刻意拋出一些你很有信心解釋的 keyword，吸引面試官就著這些 keyword 來問問題</strong>，而不是一直在等為什麼面試官沒有問我最熟的領域，這也能讓面試官更快知道你的長處是在哪裏，再考慮你是否一個適合此職缺，這對雙方來說都是一件好事。</p><h3>強調自己的非本科背景，比本科的優勢在哪裏</h3><p>非本科轉職不代表我們完全放棄了過往的背景，就連大神 Vue.js 的作者尤雨溪本身也是唸藝術和設計科系，我更曾經在 twitter 上見識過他厲害的畫功。在面試中，如果你能用具體例子指出你的背景如何幫助你開發，反而能夠加分。以我自己為例，<strong>因為我是設計背景出身，所以比本科生的優勢當然在於對畫面的敏感度明顯高很多</strong>。本科生可能比較集中開發功能而忽略了版面的要求，但設計出身的工程師，會很自然對版面和使用者體驗有一定要求。這樣絕對有助於與 UI/UX 設計師合作，有助平衡使用者體驗與功能的開發。更佳的答法是舉出實際例子，例如<strong>我在 XXX 專案裏，曾經因為我這樣的技能，因此解決了 XXX 問題。</strong></p><p>當然，對於很看重本科學歷的公司，這賣點可能無法吸引面試官，甚至會有反效果也不一定，因此也是看狀況而定。</p><h3>面試是互相了解，不是單方面的考核</h3><p>從面試官的語氣、問題深度和範疇，就會知道這間公司的工作氣氛和重視什麼。有時候表面看上去職缺要求很合適自己，但面試過程中才發現面試的問題很無趣，過程中也沒有聊到自己感興趣的部分，或者對團隊合作的風格不太滿意。最後才知道自己並不喜歡這公司，或者某一類的產業。<strong>所以，面試除了是讓面試官了解你的能力，也是讓你理解自己是否適合這公司。</strong> 我甚至有聽過較資深一點的工程師，會反問面試官問題來測試公司的技術能力，不再是單方面被考核。</p><h3>刷面試題不只是為了面試</h3><p>除了一般的技術問題，也有遇過演算法及底層觀念的題目。因此，我也要再花時間了解一些演算法、資料結構等概念。<strong>過程中我開始明白，刷面試題不止只為了面試，更是為了在面對一個問題時，我能夠不止在工具層面來思考問題，而是在底層運作上找出問題。</strong>例如：</p><ul><li>使用時間複雜度和空間複雜度的計算方法，更能清楚判斷哪種寫法會比較可取。</li><li>程式一開始能跑，但跑久了竟然會崩潰，後來發現是因為沒理解好瀏覽器的垃圾回收機制，導致 memory leak 問題。</li></ul><p>這些概念問題可能是常見面試題，但我們不只是因為面試才去理解，而是因為<strong>工作上確實有機會遇上涉及這些概念的問題，所以我們需要對此有深入理解。</strong></p><h3>心得總結</h3><p>跟台灣的同學不同，因為我在香港求職，所以我要另外花時間了解本地市場的要求，這也是我在求職時非常擔心的部分。</p><p>以上曾經說過，面試過程不累，準備面試的過程才累。原因是每次面試後，我就會馬上檢討，並調整我接下來的準備方向。即使結果令人失望，我也要趕快平伏心情來面對下一場面試，有點像以前考大學時的備戰心態一樣，我甚至有預想過如果沒有一間公司願意給我機會，我之後要怎麼繼續自學。但也多得一些失敗的經歷，我可以快速修正自己的準備方向，<strong>用 fail fast learn fast 的方式，往更正確的方向快速成長</strong>。</p><blockquote>Patient with results. Impatient with your actions.</blockquote><p>最後跟大家分享我最近很喜歡的座右銘，練習對結果保持耐性，但不猶豫地主動採取行動。不論求職還是工作，要往前走的話，就要學會主動爭取機會和資源，並且持續學習。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24104ffe02fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[【JSDC 2021 筆記】Memory Leak 與你的距離]]></title>
            <link>https://medium.com/@alysachan830/jsdc-2021-%E7%AD%86%E8%A8%98-memory-leak-%E8%88%87%E4%BD%A0%E7%9A%84%E8%B7%9D%E9%9B%A2-d5b0c3bec649?source=rss-fb1b2b8c75------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5b0c3bec649</guid>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Alysa Chan]]></dc:creator>
            <pubDate>Tue, 09 Nov 2021 04:27:53 GMT</pubDate>
            <atom:updated>2021-11-09T04:27:53.287Z</atom:updated>
            <content:encoded><![CDATA[<p>JSDC.tw 2021 活動 <br>題目：Memory Leak 與你的距離<br>講者：Kuan</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fvnrrV4DQ2rVIU8FFNumIA.png" /><figcaption>圖片來源：<a href="https://2021.jsdc.tw/">https://2021.jsdc.tw/</a></figcaption></figure><p>作者文章：<a href="https://vocus.cc/article/61176c17fd89780001942f1c">https://vocus.cc/article/61176c17fd89780001942f1c</a></p><p>這次分享的重點：</p><ul><li>理解垃圾回收機制（garbage collection）</li><li>解決 memory leak</li></ul><p>作者的遭遇：一開始程式可以順利執行，但過了一段時間後程式就會崩潰，後來發現在 Node.js SSR server 裏的資料沒有被正常清理，導致記憶體被消耗。</p><h3>什麼是記憶體（memory）？</h3><p>在程式中的變數、函式、被編譯後的程式碼等，都存在記憶體裏。記憶體可分為：</p><ul><li>stack</li><li>heap</li></ul><p>當程式在 runtime 被執行時，就會使用 heap 裏的記憶體空間來存放新的資料。在執行的過程中，有些放在 heap 裏的資料已經不再被需要，例如在函式裏的宣告一個物件，當函式被執行完後，這物件就沒用了，而這物件理應是要被清除掉，否則浪費記憶體空間。</p><p>如果把 heap 的記憶體塞爆，就會發生錯誤：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h0QUCp7DoY2-GPHO.png" /></figure><h3>Stack 與 Heap</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f1PM5Bw6piN3Z8ig.png" /></figure><h3>Stack</h3><p>後進先出，存取速度快。</p><ul><li>存放靜態資料，即是 Primitive 資料（number, string, boolean…）</li><li>存放動態資料的 reference（ Reference of array, object, function…）。但這裏只存 reference，原資料會存在 heap 裏。然後在 stack 裏的 reference 就會指向該筆在 heap 裏的原資料。</li><li>Stack 又稱為 executaion stack。當呼叫函式時，就會把一個 function frame 推進 stack 裏。所有與這個函式有關的資料，例如變數、參數、回傳值等都會記錄在此 frame 裏。當此函式執行完後，此 frame 和其相關的資料就會被清除掉。</li></ul><h3>Heap</h3><ul><li>存放全域物件（Node.js 是 global，瀏覽器是 window）</li><li>Large object space。用來放資料量比較大的 object。這裏的資料不會被 Garbage collection 進行 fragment。</li><li>Code space 存放編譯後的程式碼。</li></ul><h3>程式碼示範</h3><p>用以下程式碼作例子，看看 stack 和 heap 的運作：</p><pre>function makeWizard(name) {<br>    const wizard = {<br>        name,<br>        spell: [&#39;teleport&#39;],<br>        level: 1<br>    }</pre><pre>    return wizard<br>}</pre><pre>const novice = &#39;kuan&#39;<br>makeWizard(novice)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JLQyiIXe8P94hmQ_.png" /></figure><p>以上程式碼中，當呼叫完 makeWizard(novice) 後，在 stack 裏的 makeWizard 這個 function frame 會被清除。剩下在 heap 裏的 { name: &#39;kuan&#39;, spell:...} 這物件。<strong>而此物件最後會被 garbage collection 機制清除掉。</strong></p><h3>Garbage collection in V8</h3><ol><li>標記：區分哪些記憶體是有用和沒用</li><li>清理：釋放沒用的記憶體</li><li>整理：清理完記憶體後，heap 裏會有很多不連續的碎片：</li></ol><p>清理前：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YF79RqCZsUQwVq56.png" /></figure><p>清理後，出現不連續的記憶體，阻礙使用新空間：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZHmSV6YkRAM10Jh_.png" /></figure><p>所以需要整理：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jbRxTc9uI6YcnKnr.png" /></figure><p>V8 引擎有兩個 garbage collector。包括：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bl5yOg2H43kzanAj.png" /></figure><p>Young Generation 是指剛分配的記憶體空間的垃圾。Young generation 會分為 nursery、intermediate。每次記憶體熬過一次 garbage collection，沒有被清除掉，就會升級：</p><pre>(Start) Nursery --&gt; Intermediate --&gt; Old generation</pre><p>我們把記憶體分為不同階級的做法，與 the generational hypothesis 有關。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QYcpdQuuFCtcE0wz.png" /></figure><p>上圖表達的重點是，大多情況下，記憶體在還沒成為 old generation 的記憶體之前，也就是在 young generation 的階段時，就會熬不過 Garbage collection 而被回收。很少資料可以一直存活，熬過一次次的 garbage collection，變成 old generation 的記憶體。例如執行完一個函式，該函式裏面的區域變數資料都會被回收掉。</p><h3>Minor GC，針對回收剛進來的資料</h3><p>從以上 generational hypothesis 的圖表現象得知，設計一個針對收集 young generation 資料的演算法，可以提高清除垃圾的效率。做法是把 Young generation 的資料存放分為 From Space 和 To Space：</p><p>當有新資料進來時：</p><ol><li>先看看 From space 有沒有空間，並進行 minor GC，遍歷 From space 的資料，看看是不是有用（reachable），即是有沒有被引用（reference）。</li><li>如果該筆資料是有用，就把它<strong>複製</strong>到 To space 裏</li></ol><p>例如現在有筆新資料 #6 進來。我們先針對 From Space 裏的資料做檢查，發現 #1, #3, #5 都是有用，於是把它們複製到 To Space 裏。之後，#1, #3, #5 就會由原本指向 From Space 裏的 reference，更新為指向在 To Space 裏的 reference。</p><p>把#1,#3,#5 複製去 To Space：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oL03wbfBiwNrbCUT.jpg" /></figure><p>之後再把 #6 放進 To Space，並清空整個 From Space：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6-P3JZjEFWWwiZKY.png" /></figure><p>接著，把原本的 To Space 變為 From Space。現在 To Space 再次變空：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kx88aNWaTyWp1MwO.png" /></figure><p>關於效能的考量：</p><ul><li>因為我們知道會在 Young generation 裏會存活的資料其實很少，所以不用擔心複製資料和更新指向這些動作所造成的效能問題。</li><li>另外，如果資料經歷了兩次 GC 都沒有被回收掉，就會升級移動到 Old generation，以致不會再次對這些資料進行同樣的檢查。</li><li>機制會把有用的資料，以像是連續區塊的方式，一整包複雜到 To Space 裏，因此複雜的動作，順便完成了 Major GC 裏「整理」的步驟。</li></ul><h3>Major GC （負責整個 heap 垃圾回收）</h3><p>首先，V8 會定義一些起點，起點可以是 global object、call stack。從起點（root） 出發，標記該筆資料是否 reachable。以下圖為例，G、H、I 資料不是 reachable。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*miSNAWYEtPuoUO5V.png" /></figure><p>這些不 reachable 的記憶體位置就會被儲放到 free list 裏。未來如果需要記憶體空間，就會從 free list 裏取出可用空間。這些在 free list 裏的資料，會指向空的記憶體。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WfLk5GqAOaFtvsBE.png" /></figure><p>接著，為了整合騰空出來的零碎記憶體，Major GC 會進行整理，包括把在 GC 下存活的資料，整理為一個連續的區塊，並更新 reference。但為了效能考慮，V8 的 major GC 只會針對整合碎片化太嚴重的資料。</p><h3>小總結</h3><p>V8 裏的 GC，不論是 major GC 還是 minor GC，也是看<strong>該資料是否 reachable</strong> ，來決定是否要回收資料。</p><h3>造成 Memory leak 的原因</h3><p><strong>根據以上原則，memory leak 出現的原因是我們有些資料是 reachable，但其實我們沒有用到。換言之，我們意外地把某些資料放到 reachable 的空間。</strong> 例如，有一個陣列是 reachable，然後我們把某筆資料推進這個陣列裏，導致這筆資料是 reachable，並不會被回收。</p><h3>在程式中尋找 memory leak 的部分</h3><p>我們可以用一些工具，在大量程式碼中找出 memory leak 的部分。以下先舉一個會造成 memory leak 的例子，只要 server 一接到 request，就會推進一個 global 的陣列裏，以致這個陣列一直變大，塞滿 memory：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ID_UoGd0pKDqoNQz.jpg" /></figure><p>之後使用 node 的 debugger 來查看 memory 在進行 k6 壓力測試前後的變化，作者在<a href="https://vocus.cc/article/61176c17fd89780001942f1c">文章</a>裏已寫了具體步驟，以下的內容會有點重複。</p><p>在終端機輸入 node --inspect app.js 開啟 debugger。並在 Chrome 輸入 chrome://inspect/#devices，出現以下畫面。按 app.js 下的 inspect 按鈕</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pPI-Th8vRq9BqpWq.png" /></figure><p>之後為 heap 進行一次 Snapshot：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QnYdX1RPqatLxqjt.png" /></figure><ul><li>Constructor：把所有資料分類</li><li>Distance：從 root 開始往下遍歷時的深度</li><li>Retained size：這筆資料被釋放後，會騰空出多少空間。例如這筆資料所 reference 的資寫都會一併被釋放。以下示範也會集中觀察這裏的變化。</li></ul><p>作者的情況是，他的專案裏有一個 API，經過多次呼叫這 API 就會出現 memory leak，因此他使用 k6 進行壓力測試，不斷呼叫該 API 來抓出 memory leak 的現象。</p><p><strong>在使用 k6 進行壓力測試前，記得先拍下測試前的 snapshot，用來與測試後的結果進行對比。</strong></p><p>開啟一個新檔案 script.js，寫簡單的 k6 測試：</p><pre>import http from &quot;k6/http&quot;;<br>import { sleep } from &quot;k6&quot;;</pre><pre>export default function() {<br>  http.get(&quot;&lt;http://localhost:3000&gt;&quot;);<br>  sleep(1);<br>}</pre><p><a href="https://www.notion.so/37ccd268d0578f8759be37cefa38729b">code 來源</a></p><p>之後在終端機啟動執行測試：</p><pre>k6 run --duration 5m --vus 100 script.js</pre><ul><li>duration：測試時間</li><li>vus：測試的虛擬使用者數量（virtual users）</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7nEzE6BeEAp3TcFb.png" /></figure><p>測試前的 snapshot：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Lwl8hGTATv6AS8mS.png" /></figure><p>測試後的 snapshot：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xLOIHTkwIsaUvmXz.png" /></figure><p>打開 Closure，會發現 requests 這個 Array 的 retained size 很大：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dKPpabdcFycD4FNT.png" /></figure><p>仔細查看這 array 的內容，都是一些 incoming message。它是一個 HTTP request 的資料型態。因為在示範中，每次 server 收到 request，就會把該 request messag 推進這個 array 裏。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t8UxpYpFRfALXajH.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p0K1xX2LM9gjPPH0.png" /></figure><h3>總結</h3><p>透過理解 memory 和 garbage collection 的運作機制，我們得知如果有資料是 reachable 的話，他們不會被 garbage collection 回收掉，如果資料量變大，長久下會塞滿記憶體空間，造成 memory leak 問題。透過使用 inspect 工具、snapshot 和 k6 測試，我們可以知道哪些資料佔了很多記憶體，但沒有被釋放。有些情況是可能這些資料不是由工程師自己本身造成，而是因為某些插件而造成。但利用以上工具和方法來測試，就更易找出 memory leak 的部分。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5b0c3bec649" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>