<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SleeperShark on Medium]]></title>
        <description><![CDATA[Stories by SleeperShark on Medium]]></description>
        <link>https://medium.com/@40243105s?source=rss-4a420509a6d3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GCWS6uFn5LjQ7eQP_kHLLg.jpeg</url>
            <title>Stories by SleeperShark on Medium</title>
            <link>https://medium.com/@40243105s?source=rss-4a420509a6d3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:16:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@40243105s/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Python structures data set?]]></title>
            <link>https://medium.com/@40243105s/how-python-structures-data-set-12dbc86f83ea?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/12dbc86f83ea</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[intermediate-python]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Sun, 29 Jun 2025 11:53:03 GMT</pubDate>
            <atom:updated>2025-06-29T11:53:03.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Python structures data?</h3><h4>TypedDict, namedtuple, Dataclass</h4><h4>TL;DR</h4><ul><li><strong>TypedDict</strong><br>- 給 dictionary 的 Type Annotation (not data container)<br>- 搭配 Type Checker 檢查<br>- 提供 value type info 與 Type Hint 效果。</li><li><strong>namedtuple</strong><br>- 透過 named attributes access 的方式取得資料的 tuple<br>- 最輕量的 data container，空間優化首選</li><li><strong>dataclass <br></strong>- 快速搭建 <strong>class</strong> schema blueprint<br>- 彈性、擴充性最好的 data container</li><li>其他提供 <strong>schema + validation + serialization</strong> 功能的 third-party packages：<br><a href="https://github.com/python-attrs/attrs">attrs</a>, <a href="https://github.com/jcrist/msgspec">msgspec</a>, <a href="https://github.com/marshmallow-code/marshmallow">marshmallow</a>, <a href="https://github.com/pydantic/pydantic">pydantic</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7pUq5oFRJD3vC7KX" /><figcaption>Photo by <a href="https://unsplash.com/@rubaitulazad?utm_source=medium&amp;utm_medium=referral">Rubaitul Azad</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>在很多開發情境中，常常會拿到ㄧ個序列的 “資料組 data set”，而資料組的各個欄位則是由資料組的 order 來決定。</p><p>比如說，透過 csv.reader 取得的 row data，或是 sqlite3 的 cursor.execute 取得的資料列，都是用 iterable 的資料結構來儲存，而具體的欄位對應的資料，則是根據 csv header / SELECT 語句的內容來決定。</p><p>如果直接以這些原始結構進行操作，將<strong>難以理解實際上 row 資料各個順序的意涵</strong>；而解方則是透過一些 data structure 的包裝，透過 field name 的方式取代原本的 iterable order，明文定義 field 與 value 之間的關係。</p><p>以下快速介紹 3 個 Python 原生的 module，讓開發者可以<strong>快速的包裝 data row</strong>，除了增加資料的透明性，且會在 IDE 上提供友善的 Type Hint，大大優化開發上的體驗。</p><h3>TypedDict</h3><p>首先先從約束力最低的 TypedDict 開始說。</p><p>從 module 就可以知道，<a href="https://typing.python.org/en/latest/spec/typeddict.html">typing.TypedDict</a> 其實連 data container 都不是，僅僅是一個對 dictionary 的<strong>型別注釋</strong> (Type Annotation) 而已。</p><p>因此，一如其他 Python 的 typing，即使傳入的參數不符合 TypedDict 定義，執行上也<strong>不會報錯</strong>。</p><pre>from typing import TypedDict<br><br>class User(TypedDict):<br>    name: str<br>    age: int<br><br>def print_user(user: User):<br>    print(user)<br><br>user: User = {&quot;name&quot;: &quot;Alex&quot;}<br><br>print_user(user) <br># {&#39;name&#39;: &#39;Alex&#39;}<br><br># 即使 user 變數不符合 User 定義，依然可以運行</pre><p>TypedDict 基本上的應用情境如下：</p><ul><li>搭配 Python Type Checker (如 MyPy, Pylance) 做型別檢查。<br>(其實 TypedDict 本來就源自於 MyPy 套件型別)</li><li>增加 dictionary data 的可讀性。</li><li>在編輯器中提供 Type Hint，減少 hard-coded key 的錯誤。</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*5wdQWGvEkNauw6nhYccCiA.png" /></figure><p>整理<a href="https://typing.python.org/en/latest/spec/typeddict.html">官方文件</a>中實用的情境：</p><pre>from typing import TypedDict<br><br># ===== Basic usage =====<br>class Movie(TypedDict):<br>    name: str<br>    year: int<br><br># Used as dictionary typing<br>movie_1: Movie = {&quot;name&quot;: &quot;Titanic&quot;, &quot;year&quot;: 1997}<br><br># Or use it as a dictionary constructor<br>movie_2 = Movie(name=&quot;Blade runner&quot;, year=1982)<br># type of movie_2 is still dict<br>print(type(movie_2)) # &lt;class &#39;dict&#39;&gt;<br><br><br># ===== Inheritance =====<br>class BookBaseMovie(Movie):<br>    based_on: str<br>    # This class posses 3 keys: name, year, based_on<br><br><br># ===== Totality =====<br>class Movie(TypedDict, total=False):    # `total` default to True<br>    name: str<br>    year: int<br><br>m = Movie(name=&quot;Zootopia&quot;)  # No type check error<br><br><br># ===== Required and NotRequired =====<br>from typing import Required, NotRequired<br><br># Use `Required` when total is False<br>class Movie(TypedDict, total=False):<br>    title: Required[str]<br>    year: int<br><br># Use `NotRequired` when total is True<br>class Movie(TypedDict):<br>    title: str<br>    year: NotRequired[int]<br><br><br># ===== Interaction with Annotated[] =====<br>from typing import Annotated<br><br>class Movie(TypedDict):<br>    title: str<br>    year: NotRequired[Annotated[int, lambda value: value &gt; 1800]]<br><br># OR<br>class Movie(TypedDict):<br>    title: str<br>    year: Annotated[NotRequired[int], lambda value: value &gt; 1800]<br><br><br># ===== Read-only Items =====<br># NOTE: `ReadOnly` qualifier was added in 3.13<br>from typing import ReadOnly<br><br>class Band(TypedDict):<br>    name: str<br>    members: ReadOnly[list[str]]<br><br>blur: Band = {&quot;name&quot;: &quot;blur&quot;, &quot;members&quot;: []}<br>blur[&quot;members&quot;] = [&quot;Damon Albarn&quot;]  # Type check error: &quot;members&quot; is read-only<br><br># Can be implemented with Required / NotRequired<br>class OptionalName(TypedDict):<br>    name: ReadOnly[NotRequired[str]]</pre><h3>namedtuple</h3><p><a href="https://docs.python.org/3/library/collections.html#collections.namedtuple">collections.namedtuple</a> 可以理解成 tuple 的加強版，除了原本的 immutable, indexing 的特性外，named attribute 的功能讓開發者可以透過 dot 的方式取得資料。<br>當然，IDE 也會貼心的提供 type hint 做參考。</p><pre>from collections import namedtuple<br><br>Point = namedtuple(&#39;Point&#39;, [&#39;x&#39;, &#39;y&#39;])<br><br>def print_point(p: Point):<br>    print(p)                      # Point(x=1, y=1)<br>    print(type(p))                # &lt;class &#39;__main__.Point&#39;&gt;<br>    print(isinstance(p, tuple))   # True<br>    print(isinstance(p, Point))   # True<br>    print(p[0])                   # 1<br>    print(p[1])                   # 2<br>    print(p.x)                    # 1<br>    print(p.y)                    # 2<br><br>print_point(Point(x=1, y=2))</pre><p>namedtuple 的最大優點是<strong>空間效能</strong>。</p><p>在每一個 Python 物件中，會有存有一個負責管理 attributes 資料的 <strong>instance dictionaries。<br></strong>除了預先在 class 中定義好的 attributes 以外，可以隨性地透過 dot 或是 setattr 的方式儲存任意 attribute。</p><p>可以透過 instance.__dict__ 來查看當前物件的 attributes:</p><pre>class ClassPoint:<br>    def __init__(self, x, y):<br>        self.x = x<br>        self.y = y<br><br>point = ClassPoint(x=1, y=2)<br>point.z = 3<br>print(point.__dict__)<br># {&#39;x&#39;: 1, &#39;y&#39;: 2, &#39;z&#39;: 3}</pre><p>而 tuple (以及namedtuple) 透過定義 __slots__ property 來規範物件的 attributes。<br>雖然不能再任意增加 attribute，但因為無需生成 instance dictionary，大幅減少 memory 的使用。</p><p>（參考：<a href="https://medium.com/@apps.merkurev/dont-forget-about-slots-in-python-c397f414c490">https://medium.com/@apps.merkurev/dont-forget-about-slots-in-python-c397f414c490</a>）</p><pre>from collections import namedtuple<br><br>TuplePoint = namedtuple(&#39;Point&#39;, [&#39;x&#39;, &#39;y&#39;])<br>point = TuplePoint(x=1, y=2)<br><br>print(point.__dict__)<br># raise AttributeError: &#39;tuple&#39; object has no attribute &#39;__dict__&#39;.</pre><p>用 <a href="https://github.com/pympler/pympler">pympler</a> 簡單比較用 list, dictionary, tuple, namedtuple 與 class instance 來儲存 x, y, z 三點座標的 memory usage:</p><pre>from pympler import asizeof<br>from collections import namedtuple<br><br>class ClassPoint:<br>    def __init__(self, x, y, z):<br>        self.x = x<br>        self.y = y<br>        self.z = z<br><br>TuplePoint = namedtuple(&#39;Point&#39;, [&#39;x&#39;, &#39;y&#39;, &#39;z&#39;])<br><br>print(f&#39;tuple: {asizeof.asizeof((1, 2, 3))} bytes&#39;)<br>print(f&#39;list: {asizeof.asizeof([1, 2, 3])} bytes&#39;)<br>print(f&#39;dict: {asizeof.asizeof(dict(x=1, y=2, z=3))} bytes&#39;)<br>print(f&#39;namedtuple: {asizeof.asizeof(TuplePoint(x=1, y=2, z=3))} bytes&#39;)<br>print(f&#39;instance: {asizeof.asizeof(ClassPoint(x=1, y=2, z=3))} bytes&#39;)<br><br># tuple: 160 bytes<br># list: 184 bytes<br># dict: 424 bytes<br># namedtuple: 160 bytes<br># instance: 584 bytes</pre><p>整理<a href="https://docs.python.org/3/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields">官方文件</a>中實用的情境：</p><pre>from collections import namedtuple<br><br># ===== Basic usage =====<br>Point = namedtuple(&#39;Point&#39;, [&#39;x&#39;, &#39;y&#39;])<br><br># Or, construct with class, which allows you to add or change functionality<br>class Point(namedtuple(&#39;Point&#39;, [&#39;x&#39;, &#39;y&#39;])):<br>    __slots__ = ()  # prevent the creation of instance dictionaries<br><br>    @property<br>    def hypot(self):<br>        return (self.x ** 2 + self.y ** 2) ** 0.5<br><br>    def __str__(self):<br>        return &#39;Point: x=%6.3f  y=%6.3f  hypot=%6.3f&#39; % (self.x, self.y, self.hypot)<br><br>p = Point(x=1, y=2)<br><br># provide concise default __repr__<br>print(p)<br># Point(x=1, y=2)<br><br># provide concise default dcoumentation<br>print(Point.__doc__)<br># Point(x, y)<br><br># yet can customize doc by making direct assignments:<br>Point.__doc__ += &#39;: A 2D-point coordinate.&#39;<br>Point.x.__doc__ = &#39;Point X (integer)&#39;<br><br><br># ===== default values =====<br># `defaults` will take the values as kwargs<br>Account = namedtuple(<br>    &#39;Account&#39;,<br>    [&#39;type&#39;, &#39;user&#39;, &#39;balance&#39;],<br>    defaults=[&#39;Anonymous&#39;, 0],<br>)<br><br>a1 = Account(type=&#39;premium&#39;, user=&#39;Terry&#39;)<br>a2 = Account(type=&#39;ordinary&#39;, balance=200)<br><br>print(a1)  # Account(type=&#39;premium&#39;, user=&#39;Terry&#39;, balance=0)<br>print(a2)  # Account(type=&#39;ordinary&#39;, user=&#39;Anonymous&#39;, balance=200)<br><br><br># ===== namedtuple._make =====<br>p_data = (<br>    (1, 2),<br>    (3, 5),<br>    (9, 1),<br>)<br>points = list(map(Point._make, p_data))<br># list of Point obj<br><br><br># ===== namedtuple._asdict =====<br># useful for serialization<br>print(Point(x=1, y=2)._asdict())  # {&#39;x&#39;: 1, &#39;y&#39;: 2}<br><br><br># ===== namedtuple._replace =====<br># Return a new instance of the named tuple replacing specified fields<br># with new values<br>print(Point(x=1, y=1)._replace(x=1000)) # Point(x=1000, y=1)<br><br><br># ===== namedtuple._fields =====<br># Useful for introspection and for creating new named tuple types<br># from existing named tuples<br>Point3D = namedtuple(&#39;Point3D&#39;, Point._fields + (&#39;z&#39;,))<br>print(Point3D(x=1, y=2, z=3)) # Point3D(x=1, y=2, z=3)</pre><h3>Dataclass</h3><p>dataclasses.dataclass decorator 是一個在定義 class schema 的語法糖，從文件中可以得知，在 class definition 上加上 @dataclass 後，Python 會自動幫你定義好幾個 <a href="https://docs.python.org/3/glossary.html#term-special-method">special methods</a> (或稱 dunder methods)：</p><blockquote>The @dataclass decorator will add various “<strong>dunder</strong>” methods to the class…<br>and returns the same class that it is called on.</blockquote><p>因此，本質上<strong> dataclass 與 custom Class 是一樣的效果</strong>，只是省略了一些繁瑣的 function definition，同時 dataclass 也提供了一些額外的功能。</p><p>以大家最熟悉的例子：__init__ 來說，加上@dataclass 後，就不用手動一個一個在 function 定義 params、又要再一個一個 assign 到 self 上：</p><pre>from dataclasses import dataclass<br><br>@dataclass<br>class InventoryItem:<br>    name: str<br>    unit_price: float<br>    quantity_on_hand: int<br><br><br># Above is equivalent to:<br>class NormalInventoryItem:<br>    name: str<br>    unit_price: float<br>    quantity_on_hand: int<br>    <br>    # dataclass will take care of the passed arguments<br>    # and assignments in __init__ method<br>    def __init__(self, name, unit_price, quantity_on_hand):<br>        self.name = name<br>        self.unit_price = unit_price<br>        self.quantity_on_hand = quantity_on_hand</pre><p>除了基本用法，以下快速帶過幾個 dataclass 的進階應用：</p><h4><a href="https://docs.python.org/3/library/dataclasses.html#dataclasses.dataclass">dataclass parameters</a></h4><p>先附上 dataclass parameters 的 default values，再挑幾個比較常用的來介紹：</p><pre>@dataclass(<br>    init=True,<br>    repr=True,<br>    eq=True,<br>    order=True,<br>    unsafe_hash=False,<br>    frozen=False,<br>    kw_only=False,<br>    slots=False,<br>    weakref_slot=False,<br>)<br>class Demo:<br>    number: int<br>    text: str</pre><ul><li><strong>init</strong>: 要不要生成 __init__() method</li><li><strong>repr</strong>: 要不要生成 __repr__() method</li></ul><p>預設的 repr 效果如下：</p><pre>print(Demo(number=1, text=&quot;Python&quot;))<br># Demo(number=1, text=&#39;Python&#39;)</pre><ul><li><strong>eq</strong>: 要不要生成 __eq__() method<br>(比較邏輯為將 fields value 打包成 tuple 後，跟對象 class 作比較)</li><li><strong>order</strong>: 要不要生成下列 comparison methods，比較方法同 eq<br>( __lt__(), __le__(), __gt__(), __ge__() )</li><li><strong>frozen</strong>: 是否要讓 instance 有 frozen 效果<br>(即 ”<strong>類似</strong>” immutable 的)。<br>底層機制是透過 __setattr__() 和 __delattr__() 被呼叫時拋出 FrozenInstanceError 來阻止 instance 被變動，但有微小的 perfomance penalty，具體說明可看<a href="https://docs.python.org/3/library/dataclasses.html#frozen-instances">這裡</a>。</li><li><strong>kw_only</strong>: 是否在建構 instance 只允許透過 keyword argument 的方式傳入。我個人是偏好這個選項的，所謂 <strong><em>Explicit is better than implicit</em></strong>.</li><li><strong>slots</strong>: 是否要建構 __slots__ attributes 來提升 space efficiency。<br>這個 namedtuple 有介紹到，不贅述，底下附個證明。</li></ul><pre>from dataclasses import dataclass<br>from pympler.asizeof import asizeof<br><br>@dataclass<br>class DemoOne:<br>    number: int<br>    text: str<br><br>@dataclass(slots=True)<br>class DemoTwo:<br>    number: int<br>    text: str<br><br>print(asizeof(DemoOne(number=1, text=&quot;2&quot;)))  # 520<br>print(asizeof(DemoTwo(number=1, text=&quot;2&quot;)))  # 128</pre><h4>default field value</h4><p>像普通的 class field definition 一樣，可以在 field type annotation 旁邊加上 default values:</p><pre>@dataclass<br>class Demo:<br>    number: int = 0<br><br>print(Demo().number)  # 0</pre><h4><a href="https://docs.python.org/3/library/dataclasses.html#dataclasses.field">dataclasses.<strong>field</strong></a></h4><p>dataclass 內部 Field 物件的 constructor function，可以對 attributes 做更細節的設定：</p><pre>from dataclasses import dataclass, field<br><br>@dataclass<br>class C:<br>    my_string: str = field(default=&quot;default&quot;)<br>    my_list: list[int] = field(default_factory=list)<br><br># list all field function parameters and their default values below:<br>field(<br>  default=MISSING,<br>  default_factory=MISSING,<br>  init=True,<br>  repr=True,<br>  hash=None,<br>  compare=True,<br>  metadata=None,<br>  kw_only=MISSING,<br>)</pre><ul><li><strong><em>init</em></strong>: field 是否作為 dataclass init function 中的其中一個參數。<br>（可搭配下面介紹的 post-init 操作）</li><li><strong><em>repr</em></strong>: fielf 是否要顯示於 repr 字串中。<br>（eg. password 等機敏資料就可以設定為 False）</li><li><strong><em>default</em></strong>/ <strong>default_factory</strong>:該 field 的預設值 / 預設值的 factory function<br>- 兩者只能設定其一<br>- default_factory 需為 zero-argument callable</li><li><strong><em>compare</em></strong><em>: </em>field 是否要出現在 equality / comparison method 中。<br>（__eq__ / __gt__, etc）</li><li><strong><em>kw_only</em></strong>: field 是否只能以 kw 的方式傳入 constructor。</li></ul><h4><a href="https://docs.python.org/3/library/dataclasses.html#post-init-processing">Post-init processing</a></h4><p>dataclass 提供了一個 __post__init__ function，會在 __init__ function 結束後呼叫。__post__init__ 可以搭配以下兩種情境來應用：</p><ul><li>field(init=False)，在 __post__init__ 中才做處理：</li></ul><pre>@dataclass<br>class C:<br>    a: float<br>    b: float<br>    c: float = field(init=False)<br>    # c variable won&#39;t be passed int __init__func<br><br>    def __post_init__(self):<br>        self.c = self.a + self.b<br>        # assign c in __post_init__<br><br>print(C(a=1, b=2))  # C(a=1, b=2, c=3)</pre><ul><li>搭配 dataclass 特殊的 Type Annotation <strong><em>InitVar</em></strong>，表示該 field 只會在 init 階段傳入、並不會成為 dataclass 的 attribute。<br>dataclass 會在 __post_init__ 中傳入 InitVar，方便開發者利用該參數：</li></ul><pre>@dataclass<br>class C:<br>    i: int<br>    j: int | None = None<br>    database: InitVar[DatabaseType | None] = None<br><br>    def __post_init__(self, database):<br>        if self.j is None and database is not None:<br>            self.j = database.lookup(&#39;j&#39;)<br><br>c = C(10, database=my_database)</pre><h4><a href="https://docs.python.org/3/library/dataclasses.html#class-variables">Class variables</a></h4><p>由於 dataclass 的 schema definition 方式取代了原本定義 class variable 的邏輯，如果要給 dataclass 定義 class variable 的話，需要使用 typing.ClassVar 來達成，讓 dataclass 在 init 階段忽略此欄位：</p><pre>from dataclasses import dataclass<br>from typing import Any, ClassVar<br><br>@dataclass<br>class Person:<br>    name: str<br>    cnt: ClassVar[int] = 0<br><br>    def __post_init__(self):<br>        self.__class__.cnt += 1<br><br><br>p1 = Person(name=&quot;P1&quot;)  # no need to pass cnt parameter<br>p2 = Person(name=&quot;P2&quot;)<br>print(p1)          # Person(name=&#39;P1&#39;)<br>print(Person.cnt)  # 2</pre><h3>Conclusion, and more solutions</h3><p>以上介紹的三個 Python 原生模組，可以讓開發者快速的搭建 data schema structure，增加可讀性的同時也減少 typo 的機率。</p><p>不過要強調的是，TypedDict、namedtuple 和 dataclass 都沒有<strong>驗證欄位 (validation) </strong>的效果；<br>雖然在 TypedDict,dataclass 可以透過 type hint 對 field type 進行標註，但即使傳入與 type hint 不合的 data type，在執行時也不會出限任何錯誤。</p><p>如果想要有 <strong>data schema / field validation / serialization</strong> 的功能，可參考以下列舉的 third-party packages：</p><ul><li><a href="https://github.com/python-attrs/attrs">attrs</a></li><li><a href="https://github.com/jcrist/msgspec">msgspec</a></li><li><a href="https://github.com/marshmallow-code/marshmallow">marshmallow</a></li><li><a href="https://github.com/pydantic/pydantic">pydantic</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12dbc86f83ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[如何在 Github Action 取得當前 PR 的變更行數？ — 關於 git, PR, checkout action 的眉眉角角]]></title>
            <link>https://medium.com/@40243105s/%E5%A6%82%E4%BD%95%E5%9C%A8-github-action-%E5%8F%96%E5%BE%97%E7%95%B6%E5%89%8D-pr-%E7%9A%84%E8%AE%8A%E6%9B%B4%E8%A1%8C%E6%95%B8-%E9%97%9C%E6%96%BC-git-pr-checkout-action-%E7%9A%84%E7%9C%89%E7%9C%89%E8%A7%92%E8%A7%92-78ecb8d35036?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/78ecb8d35036</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[git-checkout]]></category>
            <category><![CDATA[github-actions]]></category>
            <category><![CDATA[git-diff]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Thu, 05 Jun 2025 02:12:55 GMT</pubDate>
            <atom:updated>2025-06-05T02:12:55.666Z</atom:updated>
            <content:encoded><![CDATA[<h3>如何在 Github Action 取得當前 PR 的變更行數？ — 關於 git, PR, checkout action 的眉眉角角</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*xQdcz3_tWJtQfb12.png" /></figure><p>這幾天在公司遇到的一個需求，需要在 PR ready 時，根據 PR code changed 的行數，來定義這個 PR 的 size (Large/Medium/Small)，並把 size 屬性 assign 給這個 PR 的 project Item。</p><p>在研究的過程中遇到不少有趣的問題，也對 git 和 Github Action 的一些細節有了更深的認識，特此記錄。</p><h3>Solution 1. gh command</h3><p>gh 是專屬於 github 的 CLI，如果你當前的 action runner 有權限可以 access 當前 repo 的資料，可直接透過 <a href="https://cli.github.com/manual/gh_pr_diff">gh pr diff</a> 指令來取得 PR 的改變內容：</p><pre>- name: Get pull request changes<br>  env:<br>    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}<br>    PR_NUM: ${{ github.event.pull_request.number }}  # 這個參數留待下面說明<br>  run: |<br>     PR_DIFF=$(gh pr diff $PR_NUM --patch)<br>     # 接著就可以看到裡面有 &quot;x file changed, xxx insertions(+), x deletions(-)&quot; 的字段  <br>     # 隨後就可以用 grep 來處理 (下面會有 parsing 方式說明)</pre><p>（可參考文件：<a href="https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/using-github-cli-in-workflows">Using GitHub CLI in workflows</a>）</p><p>可惜的是，當前的 Action 執行者是沒有 repo 權限的 Github App，因此不能用這個方式來達成。</p><h3>Solution 2. Using git and checkout</h3><p>第二個方法則是用我們熟悉的 checkout action 與 git 指令來達成。</p><h4>Step 1. Get the context of the PR</h4><p>首先，要取得當前啟動這個 action 的 PR 資料，可以從 <strong>context</strong> 這個特數的參數入手。</p><p><strong>Context</strong> 是儲存各種關於 action 資訊的變數，根據<a href="https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/accessing-contextual-information-about-workflow-runs#about-contexts">官方文件</a>：</p><blockquote>Contexts are a way to access information about workflow runs, variables, runner environments, jobs, and steps.<br>Each context is an <strong>object</strong> that contains properties, which can be strings or other objects.</blockquote><p>而在這次的情境中，我需要的是與 “pull_request” 相關的資料，此時可以從 github.event.pull_request context 取的 pull_request 物件。</p><p>想確認 pull_request 物件的具體內容，你可以配合 <a href="https://docs.github.com/en/enterprise-cloud@latest/actions/writing-workflows/choosing-what-your-workflow-does/evaluate-expressions-in-workflows-and-actions#tojson">action evaluation</a> 的 toJson function，將 pull_request 倒出來後用 echo 印出：</p><pre>- name: Print pull_request objects<br>  env:<br>    PR_OBJ: ${{ toJSON(github.event.pull_request) }}<br>  run: |<br>    echo $PR_OBJ</pre><p>列舉幾個變數：</p><ul><li><strong>number</strong><br> 當前 PR 號碼</li><li><strong>commits</strong><br>該 PR 內包含的 PR 數量</li><li><strong>head </strong>object<strong><br></strong>Pull Request 發起的 branch 的 HEAD commit 資訊。比對 head.sha，就是你這隻 PR 裡最後一個 commit 的 sha 碼。</li><li><strong>base </strong>object<br>Pull Request 要 merge 的對象 branch (以 master 為例) 的 HEAD commit 資訊。比對 base.sha ，就是當前 master branch 的最新的 commit sha。</li></ul><p>對於這次的任務來說，關鍵是取得 github.event.pull_request.base.sha，來作為稍後比對的 commit 標準。</p><h4>Step 2. Checkout the repo</h4><p>接下來，就是用我們熟悉的 <a href="https://github.com/actions/checkout">checkout action</a>，把 repo clone 到目前的 runner 環境中。</p><p>這邊可以考考大家：請問在預設沒有設定 checkout ref 參數的情況下， checkout 的對象到底會是哪一個 commit？</p><p>先來看看官方文件對於 ref 的說明：</p><blockquote><strong>ref:</strong><br>The branch, tag or SHA to checkout. When checking out the repository that<br> triggered a workflow, this defaults to the reference or SHA for that event.<br> Otherwise, uses the default branch.</blockquote><p>如果看到這裡，你也以為 PR-triggered 的 default ref 是我們 PR branch 的 HEAD — 也就是github.event.pull_request.head.sha 的話，那就錯啦！</p><p>讓我們仔細觀察 checkout action 的 output log：</p><pre><strong>Fetching the repository</strong><br>  /usr/bin/git -c protocol.version=2 fetch --no-tags --prune --no-recurse-submodules --depth=3 origin +25ec8c52c525e3bbdf1d1d0fc0290b3ad325fd56:refs/remotes/pull/14716/merge<br>  From https://github.com/{my-account}/{my-repo}<br>   <strong>* [new ref]         25ec8c52c525e3bbdf1d1d0fc0290b3ad325fd56</strong> -&gt; pull/14716/merge<br><br><strong>Determining the checkout info</strong><br>/usr/bin/git sparse-checkout disable<br>/usr/bin/git config --local --unset-all extensions.worktreeConfig<br>Checking out the ref<br>  /usr/bin/git checkout --progress --force refs/remotes/pull/14716/merge<br>  Note: switching to &#39;refs/remotes/pull/14716/merge&#39;.<br>  You are in &#39;detached HEAD&#39; state. You can look around, make experimental<br>  changes and commit them, and you can discard any commits you make in this<br>  state without impacting any branches by switching back to a branch.<br>  If you want to create a new branch to retain commits you create, you may<br>  do so (now or later) by using -c with the switch command. Example:<br>    git switch -c &lt;new-branch-name&gt;<br>  Or undo this operation with:<br>    git switch -<br>  Turn off this advice by setting config variable advice.detachedHead to false<br><strong>  HEAD is now at 25ec8c5 Merge ab4e5c32efe9c227b6bde6462301555d4020cace into a5e8968ed1f38a912f86b671f526fbb0c45093c6<br></strong>/usr/bin/git log -1 --format=%H<br>25ec8c52c525e3bbdf1d1d0fc0290b3ad325fd56</pre><p>在我的 PR 中，ab4e5c32 是 head.sha，而 a5e8968 則是 base.sha，但最後 checkout 的對象是 <strong>25ec8c5</strong> — 從上面的 log 中應該不難推斷，他其實是將 head merge 到 base 後的 new merge commit！<br>也就是說，<strong>25ec8c5</strong> commit 是在這次 checkout action 時才產生的、也只會存活在這次的 action 中，因此你在 repo 的 log 中是不會看到這隻 commit 的。</p><p>其實想想也合理，畢竟 PR 就是要針對 commit merge 後的結果表現是否與預期相符，如果只 checkout head.sha 的進度來跑 action，是不足以保證 merge 後 code 的表現的。</p><p>這也是為什麼如果你的 PR 對 merge branch 有衝突的話，action 就不會執行的原因 — 因為在 checkout 階段時，無法產生這隻 merge commit。</p><p>順帶一提，在官方文件中也提供了 “只 checkout 當前 PR HEAD” 的方法，就是將 github.event.pull_request.head.sha 帶入 ref 就是囉！</p><pre># Checkout pull request HEAD commit instead of merge commit<br>- uses: actions/checkout@v4<br>  with:<br>    ref: ${{ github.event.pull_request.head.sha }}</pre><p>再來第二個問題：checkout 預設行為是只會提取<strong>最後一個 commit 紀錄</strong>（也就是剛剛新鮮的 merge commit），但我們稍後要透過 git diff 指令觀察 PR commit 與當前的 merge branch HEAD 的進度差異，該如何提取其他 commit？又應該提取多少 commit 歷史紀錄，才能找到對應要比對的 commit ？</p><p>答案是利用 checkout 的 fetch-depth 參數：</p><blockquote><strong>fetch-depth:</strong><br>Number of commits to fetch. 0 indicates all history for all branches and tags.<br>Default: 1</blockquote><p>官方文件明確說明了，預設的 fetch-depth 為 1，也就是只提取 ref 參數指定的 commit。<br>如果數字 N &gt; 1，則代表提取 N-1 個 ref commit 之前的紀錄。<br>（標示為 0 則會把所有 commit 都拉回來）</p><p>我們要比對的是 action merge commit 以及 merge branch 的 HEAD commit，也因爲 ref 是 PR action 時產生的 merge 到 target branch 所產生的 commit，因此只要往前走一格 — 即fetch-depth = 2 — 就是我們要比對的 merge branch HEAD 了。</p><p>最後簡單記錄一下我一開始的錯誤思路。</p><p>我原本是想拿 PR Head 與 merge Branch Head 來比對，因此將 fetch-depth 的對象設為 PR commit 的數量 + 2，但這個 solution 的前提是 <strong>“進行 PR 時，merge branch 沒有新增的 PR”</strong>。</p><p>原因是，雖然我們很確定 action merge commit 與 merge Branch HEAD 的距離絕對是 1 ，但 action merge commit 與 PR Head 的距離<strong>取決於 PR HEAD 與目前 merge Branch HEAD 的進度差距</strong>，而非 PR commits 的數量。<br>這個 commit 距離差距是無法從 action 去取得的，與其為了取得 PR Head 而把 fetch-depth 設為 0 (也就是拉回<strong>“所有”</strong> commit)，還是單純的比對 action merge commit 和 merge Branch HEAD 比較合理。</p><p>以我們公司的 repo 為例，根據 action log，fetch-depth = 2 耗時不到一秒，但 fetch-depth = 0 (共 3.4 W 個 commits) 則需耗時 9 秒，相當不划算。</p><h4>Step 2. Parse the git diff output</h4><p>取得兩隻目標 commit 以後，接下來就是用 git diff 指令來比對進度差距了。</p><pre>- name: Calculating PR changed lines<br>  shell: bash {0}<br>  env:<br>    BASE_SHA: ${{ github.event.pull_request.base.sha }}<br>  run: |<br>    GIT_DIFF=$(git diff $BASE_SHA...HEAD --shortstat)<br>    echo $GIT_DIFF<br>    # n file changed, n insertions(+), n deletions(-)<br><br>    INSERTIONS=$(<br>      echo $GIT_DIFF | \<br>      grep --only-matching --extended-regexp --ignore-case &quot;([0-9]+) insertions&quot; | \<br>      grep --only-matching --extended-regexp &quot;([0-9]+)&quot;<br>    )<br>    echo &quot;insertions: $INSERTIONS&quot;<br>    if [ -z &quot;$INSERTIONS&quot; ]; then INSERTIONS=0; fi<br><br>    DELETIONS=$(<br>      echo $GIT_DIFF | \<br>      grep --only-matching --extended-regexp --ignore-case &quot;([0-9]+) deletions&quot; | \<br>      grep --only-matching --extended-regexp &quot;([0-9]+)&quot;<br>    )<br>    echo &quot;deletions: $DELETIONS&quot;<br>    if [ -z &quot;$DELETIONS&quot; ]; then DELETIONS=0; fi<br><br>    LINE_OF_CODE=$( expr $INSERTIONS + $DELETIONS )<br>    echo &quot;Total lines modified: $LINE_OF_CODE&quot;</pre><p>快速帶過重點：</p><ul><li><a href="https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-code--shortstatcode">--shortstat option</a><br>只印出最後改變數量的統計，不要印出詳細的比對內容。<br>（也就是只有 n file changed, n insertions(+), n deletions(-) output）</li><li><a href="https://git-scm.com/docs/git-diff#Documentation/git-diff.txt-Comparingbranches">Comparing branch</a><br>git diff $BASE_SHA HEAD 比對的是兩者<strong>各自改變的內容</strong>。<br>但我們要比對的是 “action merge 的結果，比起 Base HEAD 多了什麼東西”，因此要多加三個 dot: git diff $BASE_SHA...HEAD，這個比對的效果就會是 “HEAD 相對於 BASE_SHA，新增的進度”。<br>Quote 文件的説明：</li></ul><blockquote>git diff topic…master<br>Changes that occurred on the master branch since when the topic branch was started off it.</blockquote><ul><li>shell: bash {0}<br>這也是我沒預期會遇到的坑。<br>由於 diff 指令的特性是「有 difference 則 exit code 為 1，沒有 difference 則 exit code 為 0」，因此我們這邊 git diff 的 exit code 結果必然是 1。<br>但是在<a href="https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsshell">預設</a>中，action 會以 bash -e {0} 的設定來 run script，而 -e 是<strong>Exit immediately if a command exits with a non-zero status</strong> — 你應該也猜到會發生什麼事了 — git diff一旦比對完成、exit code 結果 1，被 shell 捕捉到後會視為錯誤並退出整個 action，導致你無法繼續剩餘的部分。<br>因此這裡要 customize action shell 設定，移除 -e，才能順利運作。</li></ul><p>剩下的就是透過 grep regex 比對來拿到對應的 insertions, deletions 數字，最後相加取得變更總行數。</p><h3>結語</h3><p>經過這次的任務，讓我更近一步熟悉 Github Action 中</p><ul><li>Context</li><li>Checkout action</li><li>Git command</li></ul><p>的應用等概念與細節。</p><p>隨著經驗增加，我也真的越來越能體會「認真看文件」的重要性：其實很多時候遇到的問題，都能在官方文件中獲得解答，關鍵是你能否耐著性子，慢慢根據問題、log message 去爬梳文字、找到解開困難的關鍵。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78ecb8d35036" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“Symbol not found: _mysql_affected_rows ERROR” when running mysqlclient package in Mac M1]]></title>
            <link>https://medium.com/@40243105s/symbol-not-found-mysql-affected-rows-error-when-running-mysqlclient-package-in-mac-m1-a1a44398951d?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1a44398951d</guid>
            <category><![CDATA[m1-mac]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mysqlclient]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Sat, 15 Feb 2025 10:51:18 GMT</pubDate>
            <atom:updated>2025-06-19T13:05:16.178Z</atom:updated>
            <content:encoded><![CDATA[<p>想快速寫一個使用 SQLAlchemy + mysqlclient 的 Python script 時，在執行時出現</p><p>ImportError: dlopen(/Users/{usernaame}/{package}/env3/lib/python3.{n}/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so, 2): Symbol not found: _mysql_affected_rows</p><p>的錯誤訊息，一開始還以為是小問題，結果花了 2 個多小時才搞定。</p><p>一言以蔽之，這是 M1 晶片的 arm 架構在運行時發生不相容的狀況，詳情請參考這個 <a href="https://github.com/PyMySQL/mysqlclient/issues/496#issuecomment-1038011859">github issues thread</a>。<br>解決辦法則是改使用 x86_64 的架構版本來運行整個 Python 環境，包含 brew, brew installed pkg 與 Python interpreter。</p><p>以下給想快速解決這個 issue 的懶人 script:<br>(NOTE: 底下請將 PY_VERSION 變數換成你自己要的 python 版本)</p><pre># Install homebrew(x86_64)<br>softwareupdate --install-rosetta<br>arch --x86_64 /bin/bash -c &quot;$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)&quot;<br><br># Alias the name of x86_64 command<br>alias brew86=&quot;arch -x86_64 /usr/local/bin/brew&quot;<br>alias pyenv86=&quot;arch -x86_64 pyenv&quot;<br><br># Install python relatedpkgs on x86_64<br>PY_VERSION=3.11<br>brew86 install openssl readline sqlite3 xz zlib<br>CFLAGS=&quot;-I$(brew86 --prefix openssl)/include -I$(brew86 --prefix bzip2)/include -I$(brew86 --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include&quot; LDFLAGS=&quot;-L$(brew86 --prefix openssl)/lib -L$(brew86 --prefix readline)/lib -L$(brew86 --prefix zlib)/lib -L$(brew86 --prefix bzip2)/lib&quot; pyenv86 install $PY_VERSION -v<br># NOTE: don&#39;t miss the `-v` option in pyenv installation,<br># since it would take quite a while to complete<br><br># After installation, you shoud see the prompt message as below:<br># ...Installed Python-&lt;ver&gt; to /Users/&lt;user&gt;/.pyenv/versions/3.11.8<br><br># Alias the python interpreter of x86<br>alias python3_x86=/Users/&lt;user&gt;/.pyenv/versions/3.11.8/bin/python<br><br># Use x86 python to create venv<br>python3_x86 -m pip install virtualenv<br>python3_x86 -m venv .venv<br><br># Source venv and check your python arch is right<br>source .venv/bin/activate<br>echo &quot;import platform;print(platform.architecture()[0]);print(platform.machine())&quot; | python<br># Sould see results below:<br>64bit<br>x86_64<br><br># Follow official doc to install mysqlclient<br># See: https://pypi.org/project/mysqlclient/<br>brew86 install mysql-client pkg-config<br>export PKG_CONFIG_PATH=&quot;$(brew86 --prefix)/opt/mysql-client/lib/pkgconfig&quot;<br>pip install mysqlclient</pre><p>以上是我事後憑記憶大概寫的一個 quick note，如有任合問題歡迎留言討論。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1a44398951d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[跟著專案一起成長的後端工程師筆記]]></title>
            <link>https://medium.com/@40243105s/%E8%B7%9F%E8%91%97%E5%B0%88%E6%A1%88%E4%B8%80%E8%B5%B7%E6%88%90%E9%95%B7%E7%9A%84%E5%BE%8C%E7%AB%AF%E5%B7%A5%E7%A8%8B%E5%B8%AB%E7%AD%86%E8%A8%98-a0f4640a31b4?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0f4640a31b4</guid>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[code-refactoring]]></category>
            <category><![CDATA[app-optimization]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[refactor]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Tue, 14 Jan 2025 13:37:29 GMT</pubDate>
            <atom:updated>2025-01-14T13:37:29.192Z</atom:updated>
            <content:encoded><![CDATA[<h3>TL;DR</h3><ul><li>這是一份 Jr. 後端工程師歷時兩年的後端 app 優化筆記。</li><li>內容將包涵<br>- Backend App Development<br>- Unit test<br>- DB management<br>- CICD<br>- Cloud infra<br>- Documentation<br>的開發建議。</li><li>適合 Jr. 後端工程師服用，提供一些<strong>小型專案優化</strong>方向的思考。</li><li>想略過前情提要者可以直接拉到最後一個段落: Optimization overview 的部分。</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xEMQWYAaYI3mfhCM" /><figcaption>Photo by <a href="https://unsplash.com/@blakeconnally?utm_source=medium&amp;utm_medium=referral">Blake Connally</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>2022 年 10 月，在前輩的邀請下，我正式成為 <a href="https://www.ccclub.io/">ccClub 官網</a>後端開發成員。</p><p>當時官網已歷經約一年半的開發後，已部署上線提供 ccClub 基礎班的課程資平台與後台，但由於團隊組成其實都是志願者、也沒有人是正式的 Django Developer，對於後端程式架構都沒有完備的概念，因此當時的 Repo 可以說是處於「可以 work，不會壞」，但很多細節都還有待打磨的狀態。</p><p>這邊先簡單提供當時 Repo 結構與我個人能力的概覽：</p><h4>Backend Service</h4><ul><li>使用 Python 與 pip 進行開發</li><li>Django + DRF，提供 Json API service</li><li>Pytest + pytest-django 執行 unit test</li><li>Docker-compose 部署 App + DB (local + prod)</li><li>Github + Github flow 作 Version control (local / dev/ prod)</li><li>Github Action 執行 CI &amp; CD，將 App Image 儲存於 ECR</li><li>部署於 AWS single EC2，靜態檔案部分存於 S3</li><li>使用 Route 53 + CloudFront + ACM 管理網域 &amp; SSL</li></ul><h4>Personal Experience</h4><ul><li>剛從 AppWorks School 畢業上岸、轉職兩個月的 Jr Backend Engineer。</li><li>轉職前學過 2 個月的 Django，轉職期間學了 6 個月的 Express.js，入職後用了兩個月的 Laravel。<br>(實際上進來團隊時已經把 Django 忘的差不多了)</li><li>轉職專案架構：GoDaddy Domain + EC2 instance + S3 Storage + MongoDB</li><li>基本上就是大概有 CRUD 概念的 API 工具人</li></ul><p>在隨後的兩年裡，我利用業餘時間重構了這份專案，從 App、Test、CICD、DB Backup、到 AWS Infra 層面的調整，讓 ccClub 的後端 App 逐步往我心目中「合格的、穩定的、可靠的」的方向前進。</p><p>老實說，有這種可以「無所顧忌」來練習重構的機會其實非常難得。儘管一路上因為經驗不足而嗑嗑絆絆、但我也扎實的在這趟旅途中獲得了豐厚的成長。</p><p>接下來，我將詳實的紀錄我在這趟重構的經歷，希望可以幫助到那些「想優化專案、卻不知從何開始下手」的工程師，加減從我的經驗中看到可以參考的部分，並應用於自己的專案上。</p><p>持續優化，持續成長。</p><h3>Optimization Overview</h3><p>這裡先透過 one-liner 的形式簡介我做的重構內容，未來再慢慢透過文章的形式補齊細節。</p><h4>Mindset and Management</h4><ul><li>如何熟悉一套新專案？</li><li><strong>Documentation</strong><br>兩個原則： Simple &amp; Explicit，以「新人導向」的思維撰寫文件，以及各種文件類型。</li></ul><h4><strong>Application Development</strong></h4><ul><li>DRF<strong>: 善用 Serializer</strong><br>使用 DRF Serializer 進行 Request 驗證 + Response 封裝</li><li>drf-yasg: <strong>優化 Swagger 文件撰寫</strong><br> drf-yasg 套件應用的小撇步: 用 serializer 來 render form、re-usable template</li><li><strong>Design Response Patten<br></strong>Instance / List / Paginated response 的設計，以及 FP vs OOP 應用上的反思</li><li>Python<strong>: 井井有條的 Packages</strong><br>使用 __init__.py + __all__ 管理自定義的 modules</li><li>Django <strong>ORM: 後端效能的敵人：N+1 queries</strong><br>用 prefetch / selecte_related 取得 relation data</li><li><strong>Design Service: extract code from controller</strong><br>利用 function extraction 減少 controller code 的認知負荷</li><li>Django<strong>: Exception handler middleware</strong><br>Customized middleware to render error response</li><li>Django<strong>: API log middleware</strong></li><li><strong>DB backup</strong><br>使用 cronjob + shell script 來備份資料，以及思考其他備份方式的可能性</li><li><strong>DB restore: “stateless” and “stateful”</strong><br>還原 DB 時，可以選擇還原整個 DB 狀態 (包含 Table creation)，或者是單純匯入資料 (只有 Row insertion)，根據 App 操作有不同的考量</li></ul><h4>CI / CD</h4><ul><li>Docker<strong>: base image &amp; app image<br></strong>將建立環境的 container 與複製 app code 的 Image 拆成兩部分來執行，優化部署速度</li><li>Docker<strong>: base image choice</strong><br>如果專案有數十個 install package 的話，建議不要使用 alpine 作為 base image</li><li>Github Action: <strong>caching downloadable resources</strong><br>用 cache 加速 CI pip installation 的步驟</li><li>Docker: <strong>multi-platform image</strong><br>用 manual trigger Github Action 來 build 多平台的 Image</li></ul><h4>Cloud Infrastructure</h4><ul><li><strong>Network Security</strong><br>CloudFront 負責 SSL 與 forward Request 到 EC2，因此 EC2 SG (Security Group) 只要開 Port 80 給 CloudFront 即可。額外探討其他幾種 Firewall alternatives solution</li><li><strong>Security: IAM role on instance</strong><br>任何在 EC2 內進行的 AWS resources access，都可以藉由 IAM role 達成，避免使用 User access keys，徒增管理負擔</li></ul><p>(未完待續，有時間會陸陸續續更新)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0f4640a31b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Certified Cloud Practitioner (CLF-C02) 考前關鍵字衝刺筆記]]></title>
            <link>https://medium.com/@40243105s/aws-certified-cloud-practitioner-clf-c02-%E8%AD%89%E7%85%A7%E8%80%83%E5%89%8D%E9%97%9C%E9%8D%B5%E5%AD%97%E8%A1%9D%E5%88%BA%E7%AD%86%E8%A8%98-ba751c400675?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba751c400675</guid>
            <category><![CDATA[aws-cloud-practitioner]]></category>
            <category><![CDATA[aws-clf-c02]]></category>
            <category><![CDATA[aws-cloud]]></category>
            <category><![CDATA[aws-certification]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Tue, 24 Dec 2024 03:51:31 GMT</pubDate>
            <atom:updated>2025-01-02T02:43:05.677Z</atom:updated>
            <content:encoded><![CDATA[<p>本篇筆記為以下兩堂 Udemy 課程的重點整理：</p><ul><li><a href="https://www.udemy.com/course/aws-certified-cloud-practitioner-new">Ultimate AWS Certified Cloud Practitioner CLF-C02</a> (課程)</li><li><a href="https://www.udemy.com/course/practice-exams-aws-certified-cloud-practitioner">6 Practice Exams | AWS Certified Cloud Practitioner CLF-C02</a> (考卷)</li></ul><p><strong>粗體字</strong>為在測驗中頻繁出現的概念；<br>AWS 服務名稱一律使用 quote 做標註。</p><p>服用建議：</p><ol><li>先把課程 run 過一遍，對 AWS 個服務有個概念</li><li>寫測驗卷遇到不熟的 service 時，用本篇的關鍵字搜尋一下</li><li>考前迅速瀏覽各服務與概念關鍵字，加強腦內 K-V mapping 關係</li></ol><p>(如果覺得本篇有幫助到你，麻煩幫我按個 👏 支持一下)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/0*zEyNLCC6faPsFoqr" /></figure><h3>Table Of Content</h3><ul><li><strong>Basic AWS Cloud Concept</strong></li><li>IAM (Credential Report, Access Advisor)</li><li>EC2 (SG)</li><li>EC2 <strong>Instance Storage</strong><br>(EBS, AMI, Image Builder, Instance Store, EFS, FSx)</li><li>ELB &amp; ASG</li><li>S3 (Snowcone, Snowball Edge, OpsHub, Storage Gateway)</li><li><strong>Database</strong><br>(RDS, Aurora, ElasticCache, DynamoDB&amp;DAX, Redshift, EMR, Athena, QuickSight, DocumentDB, Neptune, Timestream, QLDB, Amazon Managed Block, Glue, DMS)</li><li><strong>Container &amp; other Compute Services</strong><br>(ECS, Fargate, ECR, EKS, Lambda, API Gateway, Batch, Lightsail)</li><li><strong>Deploying and Managing Infra at Scale</strong><br>(CloudFormation, CDK, Elastic Beanstalk, CodeDeploy, CodeBuild, CodePipeline, CodeArtifact, SSM, SSM Session Manager, SSM Parameter Store, Cloud9, CodeCommit, CodeStar)</li><li><strong>Global Infrastructure</strong><br>(Regions, AZ, PoP, Edge Location, Route 53, CloudFront, S3 Transfer Accelerator, AWS Global Accelerator, Outposts, WaveLength, Local Zones)</li><li><strong>Cloud Integration</strong> (SQS, SNS, Kinesis, Amazon MQ)</li><li><strong>Cloud Monitoring<br></strong>(CloudWatch (Alarm/Logs), EventBridge, CloudTrail, CodeGuru, <br>AWS Health Dashboard)</li><li><strong>Virtual Private Cloud</strong><br>(VPC, NACL, VPC Flow Logs, VPC Peering, VPC Endpoint, VPC PrivateLink , Site-to-Site VPN, Direct Connect, AWS ClientVPN, Transit GW)</li><li><strong>Security &amp; Compliance</strong><br>(Shield, WAF, Network Firewall, Firewall Manager, Penetration Test, KMS, CloudHSM, ACM, Secret Manager, Artifact, GuardDuty, Inspector, Config, Macie, Security Hub, Detective, AWS Abuse, <br>IAM Access Analyzer, AWS OpsWorks)</li><li><strong>Machine Learning</strong><br>(Rekognition, Transcribe, Polly, Translate, Lex, Connect, Comprehend, SageMaker, Forecast, Kendra, Personalize, Textract)</li><li><strong>Account Management, Billing &amp; Support<br></strong>(AWS Organizations, AWS Control Tower, AWS RAM, AWS Service Catalog, Saving Plans, AWS Compute Optimizer, AWS Pricing Calculator, Cost Allocation Tags, Cost &amp; Usage Report, Cost Explorer, Billing Alarms, AWS Budgets, AWS Cost Anomaly Detection, AWS Service Quotas, Trusted Advisor, AWS Support Plans)</li><li><strong>Advanced Identity</strong><br>(AWS STS, AWS Cognito, AWS Directory Service, <br>AWS IAM Identity Center)</li><li><strong>Backup, Recovery &amp; Migration<br></strong>(AWS Backup, AWS DRS, AWS DataSync, AWS Application Discovery Service, AWS MGN, AWS Migration Evaluator, AWS Migration Hub)</li><li><strong>Other Services<br></strong>(Amazon Workspace, Amazon Appstream 2.0, AWS IoT Core, AWS Elastic Transcoder, AWS AppSync, AWS Amplify, AWS Application Composer, AWS Service Farm, AWS FIS, AWS Step Functions, AWS Ground Station, AWS Pinpoint)</li><li><strong>AWS Architecting Ecosystem<br></strong>(Well Architected Framework, Cloud Adoption Framework, AWS Marketplace, AWS Data Exchange,AWS IQ, AMS)</li></ul><h4>🤩 Advantages of Cloud Computing:</h4><ul><li><strong>Trade capital expense</strong> (CAPEX) for operational expense (OPEX)</li><li>Benefit from massive economies of scale</li><li><strong>❌ NO MORE guessing capacity</strong></li><li>Speed ⬆ + ️Agility ⬆</li><li>❌ STOP putting money and effort in maintaining data center</li><li>Go global in minutes</li></ul><h4>🤖 Types of Cloud Computing</h4><ul><li>Infrastructure as a Service ( Iaas )<br>eg. EC2</li><li>Platform as a Service ( PaaS )<br>eg. Elastic Beanstalk , WordPress</li><li>Function as a Service (FaaS)<br>eg. Lambda</li><li>Software as a Service ( SaaS )<br>eg. Gmail , Rekognition, RDS</li></ul><h4>💰 Pricing models of the Cloud</h4><ul><li>Compute</li><li>Storage</li><li>Data transfer <strong>OUT</strong><br>( IN is free)</li></ul><p><strong>NOTE</strong>: data transfer between AWS services within same region is not charged.</p><h4>🗺️ AWS Regions</h4><ul><li>cluster of data centers</li><li>choose region over: Compliance / Proximity / Available services / Pricing</li><li><strong>{3,6} AZ</strong> per region</li><li><strong>1+ data</strong> center perAZ</li><li>Points of Presence</li></ul><h3>IAM — Identity Access Management</h3><h4>✏️ <strong>IAM constitution</strong></h4><ul><li><strong>Users</strong>: operation entity</li><li><strong>Groups</strong>: <br>- Grouping user to manage permission<br>- Not mandatory<br>- <strong>CANT</strong> <strong>be nested</strong></li><li><strong>Roles</strong>: attached to AWS services ( EC2 / Lambda / CloudFormation )</li><li><strong>Policy</strong>: JSON Document (🟢 = Optional)<br>- Version<br>- Id 🟢<br>- Statement <br>- Sid 🟢<br>- Effect ( Allow / Deny )<br>- Principle (account / user / role)<br>- Action<br>- Resource<br>- Condition 🟢</li></ul><h4>📟 <strong>MFA devices in AWS</strong></h4><ul><li>Virtual MFA device (app on smartphone)</li><li>Universal 2nd Factor (U2F) Security Key (looks like USB)</li><li>Hardware key Fob MFA Device</li></ul><h4>🚪 <strong>Ways to access AWS</strong></h4><ul><li>AWS Management Console<br>(password + MFA)</li><li>AWS Command Line Interface (CLI)<br>(access key)</li><li>AWS Software Developer Kit (SDK)<br>(access key)</li></ul><h4>🛠️ <strong>IAM Security Tools</strong></h4><ul><li>IAM Credentials Report (account-level)<br>- List all account’s users &amp; credential status</li><li>IAM Access Advisor (user-level)<br>- Renamed as “Last Access” now<br>- Shows (1) permissions granted to user (2) service last access time<br>- Used to revise policies</li></ul><h4>⚠️ <strong>Best practices for security</strong></h4><ul><li>Use IAM user, <strong>instead of ROOT account</strong>, to manage resources</li><li>Enable Multi Factor Authentication (MFA)</li><li>Create policies based on Principle of least privilege (PoLP)</li><li>Use access key to for programmatic access</li><li>Rotate key regularly</li><li>Audit permissions using Credential Report &amp; IAM Access Advisor</li><li>❌ <strong>DON’T</strong> share access key</li></ul><h4>👤 Tasks require ROOT account:</h4><ul><li>Change account settings</li><li>Close account</li><li>Change support plan</li><li>Register as a seller in Marketplace</li></ul><h3>EC2 — Elastic Compute Cloud</h3><ul><li>IaaS</li><li>config: OS (Linux / Windows / MacOS), CPU, RAM</li><li>Storage<br>- network-attached (EBS/EFS)<br>- hardware (Instance Store)</li><li>Networking: private IP / public IP / Elastic IP / SG</li><li><strong>User Data</strong>: <br>- bootstrap script (only run once at first launch)<br>- runs with root user</li></ul><h4>🖥️ Instance Naming Convention</h4><p>eg: m5.2xlarge</p><ul><li>m = instance class</li><li>5 = generation</li><li>2xlarge = size</li></ul><h4>🖥️ <strong>Instance Types</strong></h4><ul><li>General Purpose</li><li>Compute Optimized <br>(eg.media Transcoding, ML modeling, Gaming)</li><li>Memory Optimized<br>(eg. in-mem DB, real-time processing of unstructured data)</li><li>Storage Optimized<br>(OLTP, DB, Data ware house, )</li></ul><h4>👮 SG — Security Group</h4><ul><li>EC2<strong>-level, </strong>belongs to VPC</li><li><strong>ALLOW</strong> rules only</li><li>stateful</li><li>Regulation on: <br>- IN / OUT<br>- Protocol<br>- Port<br>- Source (by CIDR / other SG / CloudFront)</li><li>Scenario:<br>- app not accessible (Time out) -&gt; denied by SG<br>- connection refused -&gt; denied by App</li></ul><h4>💰 Instances Purchasing Options</h4><ul><li><strong>On-Demand<br></strong>- Pay <strong>per sec<br>- Minimum charge of 60s</strong></li><li><strong>Spot Instances<br></strong>- Up to 90% discount<strong><br></strong>- For<strong> interruptible workload<br></strong>(eg. batch jobs, image processing, distributed workload)</li><li><strong>Reserved Instances </strong>(1 / 3 years)<br>- Up to 72% discount<br>- Fixed instance attributes (instance type, region, tenancy, OS)<br>- Scope: regional or zonal (<strong>reserve capacity in an AZ</strong>)<br>- Get more discount according to the <strong>upfront</strong> <strong>payment</strong> percentage<br>- Can sell unused quota in Marketplace</li><li><strong>Convertible Reserved Instances </strong>(1 / 3 y)<br>- Up to 66% discount<br>- Can change instance type, family, OS, tenancy</li><li><strong>Savings Plans </strong>(1 / 3 y)<strong><br></strong>- Commit to a certain type of usage<br>- Usage beyond plans is billed at on-demand</li><li><strong>Dedicated Instances </strong>(1 / 3 y)<strong><br></strong>- Run on hardware dedicated to you<br>- May share hardware with accounts in same org</li><li><strong>Dedicated Hosts<br>- A</strong>ddress <strong>compliance</strong> requirements and use existing server-bound software <strong>licenses</strong></li><li><strong>Capacity Reservation<br></strong>- Reserve on-demand instances capacity in a specific AZ for duration<br> (location-sensitive tasks)<br>- No time commitment, no discount</li></ul><h3>EC2 Instance Storage</h3><h4>💾 EBS Volume — Elastic Block Store</h4><ul><li><strong>Network drive</strong>, not physically attach → <strong>latency exist!</strong></li><li><strong>Zonal resource </strong>— use snapshot to transfer across zone/region</li><li>Mounted to <strong>one instance at a time</strong></li><li>Billed for provisioned capacity:<br>- <strong>Volume type</strong><br>- S<strong>torage volume</strong> in GB<br>- Number of <strong>IPOS</strong></li></ul><p><strong>EBS SnapShot: </strong>backup of EBS</p><ul><li>Used to copy EBS across AZ / region</li><li><strong>Stored in</strong> S3</li><li><strong>SnapShot Archive</strong>: store in archive tier, cheaper but take days to restore</li><li><strong>Recycle Bin</strong>: can recover deleted snapshot within specified retention</li></ul><h4>📸 AMI — Amazon Machine Image</h4><ul><li>Customization of an EC2 instance</li><li>Image of instance’s os / software / config / storage / monitoring</li><li><strong>Regional resources </strong>— can be copied across regions</li><li>Need to stop instance before building AMI</li><li>Public / Private / Marketplace AMI</li></ul><h4>📹 EC2 Image Builder</h4><ul><li>Automate the create, maintain, validate, test AMI</li><li>Can be scheduled</li><li>Free (only pay for underlying resources)</li></ul><h4>📼 EC2 Instance Store</h4><ul><li>Hardware disk are <strong>physically</strong> <strong>attached</strong> to the host computer<br>(EBS: network attached)</li><li><strong>Better I/O</strong></li><li>Ideal for <strong>temporary storage that changes frequently</strong>, such as <strong>buffer</strong>, <strong>cache</strong>, scratch data.</li><li>Risk of data loss, client needs to take care backup on their own</li></ul><h4>🗂️ EFS — Elastic File System</h4><ul><li>Managed NFS (Network file system) for use <strong>with AWS or on-premise</strong></li><li><strong>Can be mounted on 100s of </strong>EC2<strong> </strong>in<strong> multi-AZ</strong></li><li>high available and <strong>scalable</strong> → no capacity planning!</li></ul><p><strong>EFS Infrequent Access (EFS-IA)</strong></p><ul><li>Cost-optimized (up to 92% discount)</li><li>Define Lifecycle Policy to move files to EFS-IA</li></ul><h4>🗃️ Amazon FSx</h4><ul><li>3rd party high-performance file systems</li><li>FSx For Windows: Network File System for Windows servers</li><li>FSx For Lusture: Linux + cluster, for High Performance<br>Computing</li></ul><h3>Elastic Load Balancing (ELB) &amp; Auto Scaling Groups (ASG)</h3><ul><li>Vertical Scaling (scale up / scale down)</li><li>Horizontal Scaling ( scale out / scale in )</li><li><strong>High Availability — </strong>run service across multi-AZ / region</li><li><strong>Scalability — </strong>ability to accommodate larger load</li><li><strong>Elasticity — </strong>ability to scale according to demand</li><li><strong>Agility — </strong>easy for developer to scale</li></ul><h4>⚖️ ELB — Elastic Load Balancer</h4><ul><li>Managed load balancer</li><li>Support health checks</li></ul><p>ELB types:</p><ul><li>ALB — Application Load Balancer<br>- HTTP / HTTPs ( <strong>Layer 7 </strong>)</li><li>NLB<strong> </strong>— Network Load Balancer, <br>- TCP ( <strong>Layer 4</strong> )<br>- High performance</li><li>Gateway Load Balancer<br>- GENEVE protocol on IP packet ( <strong>Layer 3</strong> )<br>- Intrusion detection: route traffic to firewalls</li><li>Classic Load Balancer (deprecated)</li></ul><h4>⚙️ ASG — Auto Scaling Group</h4><ul><li>Scale out / scale in based<strong> on demand</strong></li><li><strong>Free</strong> — only pay for underlying resources</li><li>min / desired / max number</li><li>Replace unhealthy instances</li></ul><p>Scaling Strategies:</p><ul><li><strong>Manual Scaling</strong></li><li><strong>Dynamic Scaling</strong><br>- Simple Scaling<br> (eg. trigger according to CPU usage)<br>- Target Tracking Scaling<br> (eg. avg CPU stay at a specific level)<br>- Scheduled Scaling</li><li><strong>Predictive Scaling<br></strong>- Use ML to predict traffic pattern</li></ul><h3>Amazon S3 — Simple Storage Service</h3><p>Usage:</p><ul><li>Backup and storage</li><li>Disaster recovery</li><li>Archive</li><li>Application hosting</li><li><strong>Static website hosting </strong>(need to be enabled)</li><li>Data lakes &amp; big data analytics</li><li>Software delivery</li></ul><p>🚨 S3 don’t charge on:<br>- <strong>Data transfer in</strong> S3 from internet or AWS<br>- Data <strong>transfer out</strong> to AWS resource that is <strong>in the same region as S3 bucket</strong></p><h4>🪣 S3 Bucket</h4><ul><li><strong>Regional</strong> resource, with globally-unique name</li><li>Store “<strong>Object</strong> (files)” in “Buckets (directories)”</li><li>Object:<br>- key = full path = prefix + object name<br>- value = content of the body<br>- Metadata<br>- tags<br>- version Id</li></ul><h4>👮 S3 Security</h4><ul><li>User-Based: IAM policies</li><li>Resource-Based:<br>- Bucket Policies (for public access, cross account access)<br>- Object ACL<br>- Bucket ACL</li><li>Encryption<br>- Server-Side Encryption (Amazon S3-Managed Keys, SSE-S3)<br>- Client-Side Encryption (AWS encryption SDK)</li><li>IAM Access Analyzer</li></ul><h4>📚 S3 Versioning</h4><ul><li>Enabled at <strong>bucket level</strong></li><li>Protect against accidental deletes</li></ul><h4>⛓️ S3 Replication</h4><ul><li>MUST <strong>enable versioning</strong> <strong>for source &amp; destination buckets</strong></li><li>MUST assign proper IAM permission to buckets</li><li>SRR = Same-Region Replication</li><li>CRR = Cross-Region Replication</li></ul><h4>📀 S3 Storage Class</h4><ul><li>All charged for <strong>Data egress</strong></li><li>Can move object between classes manually OR using S3 Lifecycle config</li><li>All with ≥ 3 AZ (except One Zone-IA)</li></ul><p>💸 = additional charge for retrieval<br>📦 = with min billable size<br>🗓️ = min storage duration</p><ul><li><strong>Standard</strong></li><li><strong>Standard-IA</strong> 💸, 📦, 🗓 = 30D<br>- less frequently accessed but requires <strong>rapid access<br></strong>- eg: disaster Recovery, backups</li><li><strong>One Zone-IA </strong>💸, 📦, 🗓 = 30D<strong><br></strong>- cheaper but risk data loss compared to Standard-IA</li><li><strong>Glacier Instant Retrieval </strong>💸, 📦, ️️ 🗓 = 90D<strong><br></strong>- ms retrieval, for data accessed once / quarter</li><li><strong>Glacier Flexible Retrieval </strong>💸, 📦, 🗓 = 90D<strong><br>- </strong>Expedited (1 to 5 minutes)<br>- Standard (3 to 5 hours)<br>- Bulk (5 to 12 hours)</li><li><strong>Glacier Deep Archive </strong>💸, 📦, 🗓 = 90D<strong><br></strong>- Standard (12 hours)<br>- Bulk (48 hours)</li><li><strong>Intelligent Tiering<br></strong>- monitoring + auto-tiering fee</li></ul><h4>❄️ Snow Family</h4><ul><li>Highly-secure, portable <strong>offline devices</strong> to collect / migrate / process data<br>(eg. <strong>edge computing</strong>, <strong>data migration</strong> from slow-internet-connected on-premise)</li><li>Use <strong>AWS </strong>OpsHub <strong>to manage devices</strong> at client side</li><li><strong>Run </strong>EC2 / Lambda<strong> service at the edge</strong></li><li>Snowcone<br>- terabytes (Exabyte), portable solution</li><li>Snowball Edge<br>- petabytes, compute / storage optimized</li></ul><h4>⛅ AWS Storage Gateway</h4><ul><li><strong>Hybrid cloud</strong> solution to extend on-premises storage to S3</li><li>Long migration / compliance / security / strategy</li><li><strong>Data encrypted by default</strong>!</li><li>Types:<br>- <strong>File</strong> Gateway<br>- <strong>Volume</strong> Gateway<br>- <strong>Tape</strong> Gateway</li></ul><h3>Database</h3><h4>⛁ Amazon RDS — Relational Database Service</h4><ul><li><strong>Managed</strong> SQL DB service:<br>- Automated provisioning &amp; OS patching<br>- <strong>Point in Time Restore</strong>: backups + restore to specific timestamp<br>- Monitoring dashboard<br>- Multi-AZ setup for DR (Disaster Recovery)<br>- Scaling capability (vertical / horizontal)</li><li>Postgre / MySQL / MariaDB / Oracle / MSSQL / IBM DB2</li><li>Support reservation to optimize cost</li><li>Scenarios:<br>- want to set up DB and access OS config -&gt; use EC2<br>- want to scale / backup without manual effort -&gt; use RDS</li></ul><h4>⛃ Amazon Aurora</h4><ul><li><strong>AWS cloud-optimized SQL DB </strong>👍</li><li>Better performance yet charge more</li><li>Support PostgreSQL / MySQL</li><li>Storage grows automatically (up to 128 TB)</li><li>Aurora Serverless : auto-scaling, no capacity planning needed</li></ul><h4>RDS Deployment Strategies</h4><ul><li><strong>Read Replicas<br></strong>- data only written in <strong>main DB</strong><br>- scale out read replicas (up to 15 replicas)</li><li><strong>Multi-AZ<br>- </strong>read / write to main DB<br>- <strong>Failover DB </strong>in other AZ<strong> </strong>in case of AZ outage (HA)</li><li><strong>Multi-Region<br>- Read replicas</strong> in multi-regions<br>- Disaster recovery in case of region issue<br>- Local performance for global reads</li></ul><h4>🗃️ ElasticCache</h4><ul><li>Managed <strong>Redis</strong> / <strong>Memcached </strong>(in-mem DB)</li><li>Reduce read load of DB</li></ul><h4>🎯 DynamoDB</h4><ul><li>Managed <strong>NoSQL DB </strong>(Key — Value)</li><li>HA — replication across 3 AZ</li><li>Low latency, auto-scaling, distributed <strong>serverless</strong> DB</li><li>Support reservation to optimize cost</li><li>DAX — DynamoDB Accelerator<br>- Managed in-mem <strong>cache</strong> for DynamoDB <br>- 10x performance</li><li>DynamoDB Global Tables:<br>- Make DynamoDB table accessible in multi-regions<br>- <strong>Active-Active</strong> replication (r/w in any regions!)</li></ul><h4>🧮 RedShift</h4><ul><li><strong>OLAP</strong> solution<br>(online analytical processing)</li><li><strong>Columnar</strong> storage of data</li><li>Use <strong>SQL interface</strong> to perform queries</li><li>Analytics and <strong>data warehousing</strong></li><li>Integrated with BI tools (QuickSight / Tableau)</li><li>Redshift Serverless: auto provisions + scaling</li></ul><h4>🗺 EMR — Elastic MapReduce</h4><ul><li>Creating <strong>Hadoop clusters </strong>(big data analytics)</li><li>Clusters = many EC2 instances</li><li>Support Apache Spark / HBase / Presto / Flink</li></ul><h4>📊 Amazon Athena</h4><ul><li><strong>Serverless</strong> query service to <strong>analyze data in</strong> S3</li><li>Use <strong>SQL</strong> language to query files</li><li>Use case: BI / analytics / report, analyze, query log</li></ul><h4>🧿 Amazon QuickSight</h4><ul><li>Serverless ML <strong>business intelligence (BI) service</strong></li><li>Create interactive <strong>dashboards</strong></li><li>Integrated with RDS, Aurora, Athena, Redshift, S3</li></ul><h4>📃 DocumentDB</h4><ul><li>AWS-implementation of <strong>MongoDB </strong>(NoSQL DB)</li><li>Fully-managed, HA (3 AZ), storage grows automatically</li></ul><h4>🔵 Amazon Neptune</h4><ul><li>Fully-managed <strong>graph DB</strong></li><li>HA (3 AZ), read-replicas available</li></ul><h4>🕰️ Amazon Timestream</h4><ul><li>Fully-managed, serverless <strong>time series DB</strong></li><li>Identify patterns in data in near real-time</li></ul><h4>📚 Amazon QLDB — Quantum Ledger DB</h4><ul><li>ledger (帳) = book recording <strong>financial transactions</strong></li><li>Fully-managed, serverless, HA</li><li><strong>Immutable</strong>: data is <strong>cryptographically</strong> verifiable<br>(no decentralization component)</li></ul><h4>🔗 Amazon Managed Blockchain</h4><ul><li>Managed blockchian service</li><li>Compatible with <br>- Hyperledger Fabric<br>- Ethereum</li></ul><h4>🧴 AWS Glue</h4><ul><li>Managed, serverless <strong>ETL</strong> service<br>(extract — transform — load)</li><li>[S3 / RDS ] <br>— extract →<br>[Glue (transform) ]<br> — load →<br>[Redshift]</li></ul><h4>🚛 DMS — Database Migration Service</h4><ul><li>Securely <strong>migrate DB to AWS</strong></li><li>Source DB → Target DB</li><li>Homogeneous migrations (eg. Oracle → Oracle)</li><li>Heterogeneous migrations (eg. MSSQL → Aurora)</li></ul><h3>Container &amp; other Compute Services</h3><h4>📦 ECS — Elastic Container Service</h4><ul><li>Help launch/terminate Docker container on “provisioned” AWS</li><li><strong>Client must provision + maintain infra (EC2)</strong></li><li><strong>Free</strong>, only pay for underlying resources</li></ul><h4>⛩ Fargate</h4><ul><li>Launch Docker container on AWS</li><li><strong>Serverless</strong> — don’t need to provision infra</li></ul><h4>🗃️ ECR — Elastic Container Registry</h4><ul><li>Private <strong>Docker Registry</strong> on AWS</li><li>ECS / Fargate / EKS can pull images from ECR</li></ul><h4>🕸️ EKS — Elastic Kubernets Service</h4><ul><li>Managed K8s cluster on AWS</li><li>Containers can be hosted on EC2 / Fargate</li></ul><h4>&lt;/&gt; AWS Lambda</h4><ul><li><strong>Serverless</strong>, <strong>FaaS</strong> (Function as a Service)</li><li>Limited by time (short executions)</li><li>Scaling automatically</li><li>Pay for (1) num of request (2) compute time</li><li>Support Js, Python, Java, C#, Ruby, Custom Runtime API (Rust / Go)</li></ul><h4>🚪 Amazon API Gateway</h4><ul><li>Fully-managed, <strong>scalable</strong>, <strong>serverless</strong> service to easily<br>create / <strong>publish / </strong>maintain / monitor<strong> API</strong></li><li>support<br>(1) <strong>RESTful APIs<br></strong>(2) WebSocket APIs</li><li>Support security / auth / throttling / API keys</li></ul><h4>📚 AWS Batch</h4><ul><li>Fully-managed <strong>batch processing</strong> at any scale</li><li>Dynamically launch EC2 or Spot Instances</li><li>Batch Jobs are defined as Docker image, run on ECS</li><li>Compare to Lambda :<br>- No time limit<br>- <strong>Any runtime</strong> in the form of docker image<br>- Relies on EBS / instance store<br>- Relies on EC2</li></ul><h4>⛵ Amazon Lightsail</h4><ul><li>Simple application solution consisting of server, object storage, DB, networking…<br> → great for user <strong>with little cloud experience</strong>!</li><li>HA but no auto-scaling, limited AWS integrations</li></ul><h3>Deploying and Managing Infrastructure at Scale</h3><h4>💭 CloudFormation</h4><ul><li><strong>IaC</strong> — Infrastructure as Code, <strong>declarative</strong> programming</li><li><strong>Free</strong> — only pay for underlying resources</li><li>Easily manage cost with <strong>tags</strong> and CloudFormation template</li><li>Integrated with Application Composer : visualize components and their relations</li></ul><h4>🧰 CDK — AWS Cloud Development Kit</h4><ul><li>Define IaC using a familiar language<br>(Js, Ts, Python, Java, .Net)</li><li>CDK then compiled code into CloudFormation template<br>(Json / Yaml)</li></ul><h4>🫛 Elastic Beanstalk</h4><ul><li><strong>PaaS</strong></li><li><strong>Free</strong>, pay for underlying resources</li><li>Enable user to focus on app code development</li><li>Managed services:<br>- instances’ provisioning, config, OS<br>- Deployment strategy (configurable)<br>- Load balancing &amp; auto-scaling<br>- <strong>health-monitoring</strong></li><li>Architecture models:<br>- single instance<br>- LB + ASG<br>- ASG only (eg. workers)</li><li><strong>Still retail FULL control over resources &amp; can access underlying resources</strong></li></ul><h4>🚀 CodeDeploy</h4><ul><li>Automatically deploy app with CodeDeploy Agent</li><li><strong>Hybrid</strong> solution: for EC2 / on-premise server<br>( <strong>Server provisioning relies on User! </strong>)</li></ul><h4>🧱 CodeBuild</h4><ul><li>Fully-managed, serverless</li><li>compile src code, run test, produce packages/artifact that are ready to deploy</li><li>pay-as-you-go</li></ul><h4>🗜️ CodePipeline</h4><ul><li><strong>Orchestrate</strong> the different steps of deployment (<strong>CICD</strong>)</li><li>Code → Build → Test → Provision → Deploy</li><li>eg. CodeCommit <br> → CodeBuild <br> → CodeDeploy<br> → Beanstalk</li></ul><h4>📦 CodeArtifact</h4><ul><li>Secure, scalable, and cost-effective <strong>artifact</strong> management</li><li>Storing &amp; retrieving <strong>dependencies</strong></li><li>Works with common dependency management tools<br>(eg. Maven, Gradle, npm, yarn, twine, pip, NuGet)</li></ul><h4>👨🏻‍💼 SSM — AWS Systems Manager</h4><ul><li><strong>Hybrid</strong> solution — manage EC2 / on-premise systems at scale</li><li>Group resources &amp; visualize aggregated data</li><li>Get <strong>operational</strong> <strong>insights about the state of infrastructure</strong></li><li>Features:<br>- <strong>Patching</strong> automation<br>- <strong>Run commands</strong> across servers<br>- SSM Parameter Store &amp; SSM Session Manager</li><li>Works through SSM agent installed at client side</li></ul><h4>⌨️ SSM Session Manager</h4><ul><li>Allow user to <strong>init a secure shell </strong>on a SSM-managed server</li><li><strong>NO SSH / bastion hosts required!</strong></li><li><strong>NO port 22 required!</strong></li></ul><h4>🔐 SSM Parameter Store</h4><ul><li>Secure storage for configs &amp; secrets</li><li>Serverless, scalable, easy SDK</li><li>Control access using IAM</li><li>Version tracking &amp; encryption (optional)</li></ul><h4>🗑️ Deprecated Services</h4><ul><li>Cloud9<br>- An integrated IDE<br>- No longer available to new customers since 2024</li><li>CodeCommit<br>- A version control service that enables user to host private Git repositories in AWS<br>- No longer available to new customers since 2024</li><li>CodeStar<br>- A central platform that coordinates the various CICD tools<br>- Integrated with Cloud9, CodeCommit, CodeBuild, CodeDeploy, CodePipeline, CodeArtifact<br>- No longer available to new customers since 2024</li></ul><h3>Global Infrastructure</h3><h4>🏗️ Global Infrastructure</h4><ul><li><strong>Regions</strong></li><li>Availability Zone</li><li>Edge Location / PoP — Points of Prescence<br>for content delivery as close as possible to users</li></ul><h4>🧩 Global Application</h4><ul><li>Route 53 : Global DNS</li><li>CloudFront : Global CDN</li><li>S3 Transfer Acceleration</li><li>AWS Global Accelerator : improve networking through AWS global network</li></ul><h4>🔀 Route 53</h4><ul><li>Managed <strong>DNS</strong></li><li>Record types:<br>- A = IPv4<br>- AAAA = IPv6<br>- CNAME = to another hostname<br>- Alias = to AWS resource</li><li><strong>Routing policies:<br></strong>- Simple routing policy<br>- Weighted routing policy<br>- Latency routing policy<br>- Failover routing policy</li></ul><h4>🌐 CloudFront</h4><ul><li>AWS <strong>CDN </strong>service, improve performance through <strong>caching</strong></li><li>PoP globally</li><li><strong>DDoS protection </strong>(integrated with Shield, WAF)</li><li>Origins:<br>- S3 bucket<br>- Custom Origin (HTTP)</li></ul><h4>⏩ S3 Transfer Acceleration</h4><ul><li>Increase transfer speed through <strong>AWS</strong> <strong>edge location </strong>(to S3 through AWS internal network)</li></ul><h4>🏎️ AWS Global Accelerator</h4><ul><li>Improve availability &amp; performance of App <strong>through AWS global network</strong></li><li>Leverage <strong>AWS internal network</strong> &amp; <strong>Edge location</strong> to <strong>optimize routing</strong></li><li><strong>No caching</strong>, proxying <strong>packets</strong> at the edge<br>→ Good for HTTP and other non-HTTP protocols networking<br>(eg. Gaming with UDP, IoT with MQTT)</li></ul><h4>📬 AWS Outposts</h4><ul><li><strong>Hybrid solution</strong></li><li><strong>Physical server racks </strong>that <strong>offer AWS service at on-premise system</strong><br>(Clients are responsible for Outpost Rack physical security)</li><li>Low latency, data processing, managed services <strong>locally</strong></li><li>Services work on Outposts: EC2, EBS, S3, EKS, ECS, RDS, EMR</li></ul><h4>📱 AWS WaveLength</h4><ul><li>Bring AWS services to the edge of <strong>5G</strong> networks through <strong>telecommunications</strong> providers’ data centers</li><li>Ultra-low latency</li><li>High bandwidth &amp; secure connection to AWS region</li><li>Use cases:<br>- Smart Cities<br>- ML assisted diagnostics<br>- Connected vehicle<br>- Live streaming<br>- Real-time gaming</li></ul><h4>⚲ AWS Local Zones</h4><ul><li><strong>Extension of an AWS Region</strong></li><li>Place compute, storage, database services closer to end users for <strong>latency-sensitive applications</strong></li></ul><h4>🎨 Cloud Application Architecture Pattern</h4><ul><li>Single Region, Single AZ<br>- simple</li><li>Single Region, Multi AZ<br>- HA</li><li>Multi Region, Active-Passive<br>- improve global read’s latency</li><li>Multi Region, Active-Active<br>- improve global read &amp; write latency</li></ul><h3>Cloud Integration</h3><h4>🚥 SQS — Simple Queue Service</h4><ul><li>Fully-managed, serverless, highly scalable</li><li>Use to <strong>decouple</strong> applications</li><li>Messages are processed in order (FIFO)</li></ul><h4>🌊 Amazon Kinesis</h4><ul><li>Real-time big <strong>data streaming</strong></li><li>Managed service to collect, process, analyze streaming data</li></ul><h4>🚨 SNS — Simple Notification Service</h4><ul><li><strong>pub/sub</strong> model<br>( — event subscriber)</li><li>Publish = event publisher send message to SNS <strong>Topic;<br></strong>Subscribe = event subscriber listen to <strong>Topic</strong></li><li>Subscribers: SQS, Lambda, Kinesis, Email, SMS, HTTP endpoint</li></ul><h4>🐰 Amazon MQ</h4><ul><li>Managed message broker service for <strong>RabbitMQ</strong> / <strong>ActiveMQ</strong></li><li>Solution for migration to cloud <strong>without re-engineering application</strong></li><li>MQTT / AMQP protocols</li></ul><h3>Cloud Monitoring</h3><h4>⌚ CloudWatch</h4><ul><li><strong>Metrics</strong>: variable to monitor</li><li>Monitor service metrics with dashboard</li><li>Important Metrics:<br>- EC2: CPU Utils, Network, Status check<br>- EBS: Disk IO<br>- S3: BucektSizeBytes, NumberOfObjects, AllRequests<br>- Billing<br>- Service limits</li></ul><h4>⏰ CloudWatch Alarm</h4><ul><li>Used to trigger notification for any metrics</li><li>Actions: Auto scaling, EC2 actions, SNS nitifications</li><li>eg. billing alarm</li></ul><h4>📋 CloudWatch Logs</h4><ul><li>Collect logs from a variety of services: <br>EC2, Elastic Beanstalk, Lambda, CloudTrail, Route 53 etc.</li><li>Enables real-time monitoring of logs</li><li>Assign proper IAM <strong>Role</strong> to AWS services to send logs to CloudWatch</li></ul><h4>🌉 EventBridge</h4><ul><li>Trigger actions according to <strong>schedule (Cron Jobs) </strong>or <strong>event pattern</strong></li><li>Targets: Lambda, SQS, SNS, EC2 Actions, Batch, CodeBuild etc.</li></ul><h4>👣 CloudTrail</h4><ul><li>Provide governance, <strong>compliance</strong> and <strong>audit</strong> for AWS account</li><li>Collect <strong>logs of events / API calls</strong> made in one account by:<br>- console<br>- SDK<br>- CLI<br>- AWS services</li><li>Scenario: resource is modified → investigate CloudTrail first!</li></ul><h4>🩻 AWS X-Ray</h4><ul><li><strong>Visual analysis</strong> of <strong>distributed</strong> apps</li><li><strong>Debug / Troubleshooting</strong> performance</li><li>Understand dependencies in a <strong>micro-service architecture</strong></li></ul><h4>👳🏼‍♂️ Amazon CodeGuru</h4><ul><li>ML-powered service</li><li>CodeGuru<strong> </strong>Reviewer<strong>: automated code reviews</strong></li><li>CodeGuru Profiler: app performance recommendations</li></ul><h4>🏥 AWS Health Dashboard</h4><ul><li>Service History: shows all regions, <strong>all services health</strong></li><li>Offer <strong>subscription to an RSS feed</strong> for interruption notification</li><li><strong>Account Health Dashboard<br>- </strong>provide <strong>alerts</strong> &amp; remediation guidance when AWS is experiencing <strong>events that may impact user</strong><br>- <strong>personalized view</strong> into the performance and availability of user’s AWS resources</li></ul><h3>VPC — Virtual Private Cloud</h3><h4>🕸 VPC</h4><ul><li>Subnet: partition network inside VPC (AZ resource)</li><li>Public Subnet: accessible from internet through IGW (Internet Gateway)</li><li>Private Subnet: not accessible from internet, but can <strong>access to internet through</strong> NAT GW(Net Address Translation, AWS-managed)</li><li>Route Table: define access between subnets and internet</li></ul><h4>✋ NACL — Network Access Control List</h4><ul><li>Firewall attached at <strong>“</strong>Subnet<strong> level”</strong></li><li>ALLOW / DENY</li><li><strong>Stateless</strong></li><li>Rules on CIDR</li></ul><h4>🪵 VPC Flow Logs</h4><ul><li>Capture IP traffic into interfaces:<br>- VPC flow logs<br>- Subnet flow logs<br>- ENI flow logs (Elastic Network interface)</li><li>Monitor and troubleshoot connectivity issues</li></ul><h4>🤝 VPC Peering</h4><ul><li>Connect 2 VPC privately using AWS’s network</li><li><strong>CANT have overlapping CIDR!</strong></li><li><strong>NOT transitive!</strong></li></ul><h4>📍 VPC Endpoint</h4><ul><li>Connect AWS services using <strong>private network</strong></li><li>Enhanced security</li><li>VPC Endpoint GW: S3 &amp; DynamoDB</li><li>VPC Endpoint interface: other services</li></ul><h4>🔐 VPC Private Link</h4><ul><li>Mose <strong>secure</strong> &amp; <strong>scalable</strong> way to expose service in 1000s VPC</li><li><strong>DONT</strong> require VPC peering, IGW, NAT, route tables</li><li>Require NLB on Service VPC</li><li>Require ENI on Customer VPC</li></ul><h4>🔌 Site-to-Site VPN &amp; Direct Connect</h4><p><strong>Site-to-Site VPN</strong></p><ul><li>Connect to on-premise VPN to AWS</li><li>Encrypted connection over <strong>public internet</strong></li><li>On-premise: CGW (Customer GW)<br>AWS: VGW (Virtual Private GW)</li></ul><p><strong>DX — Direct Connect</strong></p><ul><li>A “<strong>physical connection”</strong> between on-premise and AWS</li><li>Connection is secure &amp; <strong>fast over a private network</strong></li><li>Takes at least 1 month to establish</li></ul><h4>🔒 AWS ClientVPN</h4><ul><li>Connect from user’s computer using <strong>OpenVPN </strong>to VPC and on-premise</li><li>Allow user to connect to EC2 <strong>over a private IP</strong></li><li>Goes over <strong>public internet</strong></li></ul><h4>⛩️ Transit Gateway</h4><ul><li>One single gateway to <strong>transitively peering</strong> 1000+ VPCs</li><li>Works with DX, VPN connections</li></ul><h3>Security &amp; Compliance</h3><ul><li>AWS responsibility = Security <strong>OF</strong> the Cloud</li><li>Customer responsibility = Security IN the Cloud</li></ul><h4>🚫 DDoS — Distributed Denial of Service</h4><p>Protection against DDoS on AWS:</p><ul><li>AWS Shield Standard</li><li>AWS Shield Advanced</li><li>AWS WAF</li><li>CloudFront &amp; Route 53: mitigation against attack at the edge</li></ul><h4>🔰 AWS Shield</h4><ul><li>AWS Shield Standard<br>- <strong>Free service activated on default</strong><br>- Protections against SYN/UDP Floods, Reflection attacks, L3/L4 attacks</li><li>AWS Shield Advanced<br>- DDoS mitigation service with <strong>additional charge<br></strong>- Protection against more sophisticated attack on EC2, ELB, CloudFront, AWS Global Accelerator, Route 53<br>- 24/7 access to AWS DRP (DDoS Response Team)</li></ul><h4>🟫 WAF — Web Application Firewall</h4><ul><li>Protects web app <strong>on L7 (HTTP)</strong></li><li>Deploy on ALB, API Gateway, CloudFront</li><li>Define Web ACL:<br>- rules on IP / <strong>Headers</strong> / <strong>Body</strong> / URI<br>- Protects from <strong>SQL injection / Cross-Site Scripting (XSS)<br></strong>- Geo-match (<strong>block countries</strong>)<br>- Rate-based rules</li></ul><h4>📛 AWS Network Firewall</h4><ul><li>Protect <strong>entire</strong> VPC from L3 to L7</li><li>Guard flow from any directions:<br>- VPC to VPC<br>- VPC to internet<br>- internet to VPC<br>- to/from DX &amp; Site-to-Site VPN</li></ul><h4>👔 AWS Firewall Manager</h4><ul><li><strong>Centrally manage security rules in all accounts of</strong> AWS Organization</li><li>Security policy:<br>- VPC SG for EC2, ALB…<br>- WAF<br>- AWS Shield Advanced<br>- AWS Network Firewall</li></ul><h4>🧪 Penetration Testing on AWS Cloud</h4><ul><li><strong>Penetration tests</strong> are allowed <strong>without prior approval</strong> over:<br>- EC2, NAT GW, ELB<br>- RDS, Aurora<br>- CloudFront<br>- API Gateway, Lambda<br>- Lightsail<br>- Elasitc Beanstalk</li><li>Prohibited activities:<br>- DNS zone walking on Route 53<br>- DoS, DDoS<br>- Port flooding, Protocol flooding, Request flooding</li></ul><h4>🔑 AWS KMS — Key Management Service</h4><ul><li>Most “encryption” service in AWS</li><li>KMS = AWS-managed encryption keys</li><li><strong>Encryption automatically enabled</strong>:<br>- CloudTrail logs<br>- S3 Glacier<br>- Storage Gateway</li><li>Encryption opt-in:<br>- EBS volume, EFS drives<br>- S3 bucket server-side encryption<br>- Redshift, RDS</li></ul><h4>🔒 CloudHSM — Hardware Security Module</h4><ul><li>Dedicated encryption <strong>“hardware”</strong> provisioned by AWS</li><li>Tamper-resistent</li><li>Encryption keys <strong>managed by customer </strong>(not AWS)</li></ul><h4>⚷ Types of KMS keys</h4><ul><li>Customer Managed Key (CMK)</li><li>AWS Managed Key</li><li>AWS Owned Key<br>- collection of CMKs that an AWS service owns and manage</li><li>CloudHSM Keys<br>- generated from CloudHSM device</li></ul><h4>🏅 ACM — AWS Certificate Manager</h4><ul><li>Provision, manage, deploy SSL/TLS Certificates</li><li>Provide in-flight encryption for app (HTTPS)</li><li>Integration with:<br>- ELB<br>- CloudFront Distribution<br>- APIs / API Gateway</li></ul><h4>👔 AWS Secret Manager</h4><ul><li>Store, manage, rotate secrets</li><li>Integrated with RDS</li><li>Secrets are encrypted using KMS</li></ul><h4>📑 AWS Artifact</h4><ul><li><strong>NOT a service</strong></li><li>On-demand access to AWS <strong>compliance documents </strong>and<strong> AWS agreements</strong></li><li>Used to support internal <strong>audit or compliance</strong></li></ul><h4>💂🏻‍♂️ Amazon GuardDuty</h4><ul><li>ML-driven service to <strong>detect malicious access</strong> from logs:<br>- CloudTrail event logs<br>- VPC Flow Logs<br>- DNS Logs<br>- Logs from RDS, EKS, EBS, Lambda, S3…</li><li>Can protect against <strong>CryptoCurrency</strong> attacks</li></ul><h4>🔎 Amazon Inspector</h4><ul><li>Automated <strong>security assessments</strong> <strong>on </strong>EC2<strong>, Container Images, </strong>Lambda</li><li>For EC2 instances:<br>- against unintended network accessibility<br>- identify <strong>OS known vulnerability</strong></li><li>For <strong>container image assessment</strong> in ECR</li><li>For Lambda functions<br>- identify code vulnerability and package dependencies</li></ul><h4>⚙️ AWS Config</h4><ul><li>Help <strong>auditing &amp; recording compliance</strong></li><li>Record <strong>config change history</strong></li><li>Receive alert for changes</li></ul><h4>𖡎 AWS Macie</h4><ul><li>Data security &amp; data privacy service</li><li>Use ML &amp; patten matching to <strong>identify PII</strong> (Personal Identifiable Info) <strong>in </strong>S3</li></ul><h4>🕍 AWS Security Hub</h4><ul><li>Central security tool to manage security <strong>across AWS accounts</strong></li><li>Must enable the AWS Config first</li><li>Aggregate alerts in: Config, GuardDuty, Inspector, Macie, IAM Analyzer, SSM, Firewall Manager, AWS Health, AWS Partner Network Solution</li></ul><h4>🕵 Amazon Detective</h4><ul><li>Use ML and graphs to analyze, investigate, and <strong>identify root cause</strong> of security issues <strong>in depth</strong></li><li>Collect and process events from VPC Flow Logs, CloudTrail, GuardDuty</li></ul><h4>😡 AWS Abuse</h4><ul><li><strong>Report suspected AWS resources used for abusive or illegal purposes</strong></li><li>Abusive &amp; prohibited behaviors:<br>- Spam<br>- Port scanning<br>- DoS / DDoS<br>- Intrusion attempts<br>- Host objectionable or copyrighted content<br>- Distributing malware</li></ul><h4>📊 IAM Access Analyzer</h4><ul><li>Identify resources shared <strong>externally</strong><br>(eg. S3, IAM, KMS, Lambda, SQS, Secret Manager)</li><li>Define <strong>Zone of Trust</strong></li></ul><h4>🐶 AWS OpsWorks</h4><ul><li>A configuration management service that helps user <strong>configure and operate applications</strong> in a cloud <strong>using Puppet or Chef</strong></li></ul><h3>Machine Learning</h3><h4>🧐 Amazon Rekognition</h4><ul><li>Use ML to find <strong>objects / people / text / scenes</strong> in images or videos</li><li><strong>Facial analysis / facial search</strong></li><li>Use cases:<br>- l<strong>abeling<br>- </strong>text detection / face detection<br>- face analysis (gender, age, range, emotions….)<br>- celebrity recognition<br>- <strong>pathing</strong> (for sports game analysis)</li></ul><h4>📝 Amazon Transcribe</h4><ul><li><strong>Speech → text</strong></li><li>ASR (Automatic Speech Recognition)<br>= <strong>deep-learning</strong>-powered technique</li><li>Generate metadata for media to create searchable archive</li></ul><h4>🗣 Amazon Polly</h4><ul><li><strong>Text → speech</strong></li><li>DL-powered technique to create applications that talk</li></ul><h4>🔄 Amazon Translate</h4><ul><li>Natural &amp; accurate <strong>language translation</strong></li></ul><h4>🤖 Amazon Lex</h4><ul><li>Same technology that powered <strong>Alexa</strong></li><li>Use ASR to convert speech to txt</li><li>Natural Language Understanding to recognize intent of txt, callers</li><li>Help build <strong>chatbots, call center bot</strong></li></ul><h4>⫘ Amazon Connect</h4><ul><li>Cloud-based <strong>virtual contact center</strong></li><li>Receive call than transform it into stream data</li><li>Often integrated with <strong>CRM </strong>(Company Resource Management)</li><li>eg. a phone call to schedule<br>→ Connect<br>→ (stream) → Lex<br> → Lambda<br>→ CRM</li></ul><h4>🤓 Amazon Comprehend</h4><ul><li><strong>NLP</strong> — Natural Language Processing</li><li>Fully managed and serverless</li><li>Use ML to find <strong>insights in txt</strong>:<br>- language<br>- place, people, brands, events<br>- understand positive / negative note<br>- Analyze txt using <strong>tokenization</strong></li></ul><h4>🛕 Amazon SageMaker</h4><ul><li>Fully managed service to <strong>build ML models</strong></li><li>Get discount through <strong>Savings Plan</strong></li></ul><h4>📈 Amazon Forecast</h4><ul><li>Fully managed service</li><li>Use ML to deliver highly accurate<strong> forecasts </strong>based on historical time-series data</li><li>Use cases: Product <strong>Demand Planning</strong>, Financial Planning, Resource Planning</li></ul><h4>👀 Amazon Kendra</h4><ul><li>Fully managed <strong>document search </strong>ML service</li><li>document: txt, pdf, HTML, PPT, Word…</li></ul><h4>🤩 Amazon Personalize</h4><ul><li>Fully managed ML-service</li><li>Build apps with real-time <strong>personalized recommendations</strong></li></ul><h4>🔬 Amazon Textract</h4><ul><li>Automatically extracts text, handwriting, and data from any scanned documents using AI and ML</li></ul><h3>Account Management, Billing &amp; Support</h3><h4>🏢 AWS Organization</h4><ul><li>Global service, allow to <strong>manage multiple AWS accounts</strong></li><li>Cost benefits:<br>- <strong>Consolidated Billing — 1 bill across all accounts<br>- </strong>Pricing benefits from <strong>aggregated usage</strong> (EC2, S3)<br>- Pooling <strong>Reserved EC2 </strong>for optimal savings</li><li><strong>Automate AWS account creation</strong> through API</li><li><strong>SCP — Service Control Policies</strong>: restrict account privileges</li><li>Use <strong>tagging</strong> for billing purpose</li><li>Remove an AWS account <strong>only if it operates as a standalone account</strong>.</li></ul><h4>🗼 AWS Control Tower</h4><ul><li>Runs on top of AWS Organizations: an easy way to set up and govern multi-accounts and SCPs</li></ul><h4>👔 AWS RAM — Resource Access Manager</h4><ul><li>Pooling usage with:<br>- other AWS account<br>- account within Organization</li></ul><h4>🗂 AWS Service Catalog</h4><ul><li><strong>Service template</strong> in Organizations pre-defined by admins, can be quickly launched</li></ul><h4>🏷️ Pricing Model in AWS</h4><ul><li>Pay-as-you-go</li><li>Save when you reserve</li><li>Pay less by using more</li><li>Pay less as AWS grows</li></ul><h4>📋 Savings Plan</h4><ul><li>Commit <strong>certain usage</strong> per hour for 1 / 3 years</li><li><strong>EC2 Savings Plan<br></strong>- Up to 72% discount<br>- Commit to usage of individual instance families <strong>in a region<br></strong>- All upfront <strong>(save most)</strong> / partial upfront / no upfront</li><li><strong>Compute Savings Plan</strong> — pooling computing usage<strong><br></strong>- Up to 66% discount<br>- <strong>Regardless of Region, Family, compute options<br></strong>- Options: EC2, Fargate, Lambda</li><li>Machine Learning Savings Plan: SageMaker</li></ul><h4>⚖️ AWS Compute Optimizer</h4><ul><li>Use ML to analyze resources’ configs &amp; utilization CloudWatch metrics</li><li>Provide recommendations to <strong>reduce costs &amp; improve performance</strong></li><li><strong>Support resources</strong>: Ec2, ASG, EBS, Lambda</li></ul><h4>📑 Billing &amp; Costing Management tools</h4><ul><li>Estimate costs in the cloud:<br>- Pricing Calculator</li><li>Tracking costs in the cloud:<br>- Billing Dashboard<br>- Cost Allocation Tags<br>- Cost &amp; Usage Report<br>- Cost Explorer</li><li>Monitoring against costs plans:<br>- Billing Alarms<br>- Budgets</li></ul><h4>🧮 AWS Pricing Calculator</h4><ul><li>Estimate the cost for your solution architecture</li></ul><h4>🏷 Cost Allocation Tags</h4><ul><li>Used to track AWS costs on a detailed level</li><li>For each resource: <br>- key must be unique<br>- each tag key can only have 1 value</li><li>Must activate <strong>AWS generated tags &amp; User-defined tags separately </strong>before they can appear in Cost Explorer or on a cost allocation report</li><li>Tags can be used to create <strong>Resource Groups</strong></li></ul><h4>📝 Cost &amp; Usage Report</h4><ul><li><strong>The most comprehensive set of AWS cost and usage data</strong></li><li>List all usage for each service used by account / IAM users <strong>in hourly / daily line items</strong></li><li><strong>Activate tags</strong> for cost allocation features</li><li>Integrated with Athena, Redshift, QuickSight</li></ul><h4>🧭 Cost Explorer</h4><ul><li><strong>Forecast usage based on previous usage</strong></li><li>On monthly / hourly / resource level granularity</li></ul><h4>🚨 Billing Alarms</h4><ul><li>Billing data metric is <strong>stored in </strong>CloudWatch<strong> us-east-1</strong></li><li>A simple alarm on worldwide cost</li></ul><h4>💵 AWS Budgets</h4><ul><li>Create budget and send alarms when costs exceeds<br>(powerful alarm tool)</li><li>4 types of budges:<br>- Usage<br>- Cost<br>- <strong>Reservation<br></strong>- Savings Plan</li><li>Can filter by service, account, tag, purchase option, instance type, region, AZ…</li></ul><h4>🕵🏽‍♂️ AWS Cost Anomaly Detection</h4><ul><li>Use ML to monitor and <strong>detect unusual spending</strong></li><li>Sends anomaly detection report and root-cause analysis using SNS</li></ul><h4>🎟️ AWS Service Quotas</h4><ul><li>Notify user when close to a service quota value threshold</li><li>Manage Service Quotas on CloudWatch console</li><li>Triggered actions:<br>- <strong>Request a quota increase</strong><br>- Shut down resources</li></ul><h4>👩🏻‍🏫 Trusted Advisor</h4><ul><li><strong>🚨🚨🚨 Frequently appeared in the exam!</strong></li><li><strong>No pre-installation required</strong>, high level AWS account assessment</li><li>Provide suggestions on 6 categories:<br>- <strong>Cost Optimization</strong><br>- Performance<br>- <strong>Security</strong><br>- Fault tolerance<br>- <strong>Service limits<br></strong>- Operational excellence</li><li>In <strong>Business+</strong> Support plans:<br>- <strong>Full set of checks</strong><br>- Programmatic access using AWS Support API</li></ul><h4>🙌 AWS Support Plans</h4><ul><li><strong>Basic</strong><br>- free<br>- 24 * 7 access to customer service, doc, whitepaper, support forum<br>- <strong>7 core </strong>Trusted Advisor<strong> checks<br>- </strong>Personal Health Dashboard</li><li><strong>Developer</strong>: Basic +<br>- <strong>Business hours email</strong> access to Cloud Support Associates<br>- Unlimited cases / unlimited contacts</li><li><strong>Business: </strong>Developer +<br>- <strong>Full set of</strong> Trusted Advisor<br>- <strong>24 * 7 phone, email, chat access to Cloud Support Engineer<br></strong>- Access to <strong>Infrastructure Event Management</strong> (with extra fee)<br>- Prod system down response <strong>in &lt; 1 hour</strong></li><li><strong>Enterprise On-Ramp</strong>: Business +<br>- Access to<strong> pool of TAM</strong> (Technical Account Managers)<br>- <strong>Concierge Support Team </strong>(billing &amp; account consultants)<br>- Infrastructure Event Management,Well-Architected &amp; Operations Reviews<br>- Business-critical system down respond <strong>in &lt; 30 min</strong></li><li><strong>Enterprise: </strong>On-Ramp +<br>- Access to<strong> designated TAM</strong> (Technical Account Managers)<br>- Access to AWS Incident Detection and Response (with extra fee)<br>- Business-critical system down respond <strong>in &lt; 15 min</strong></li></ul><h3>Advanced Identity</h3><h4>🪙 AWS STS — Security Token Service</h4><ul><li>Create <strong>temporary, limited-privileges credential</strong> to access AWS</li><li>Use cases:<br>- <strong>Identity federation</strong><br>- IAM roles for cross account access</li></ul><h4>🧙 Amazon Cognito</h4><ul><li><strong>Database storing identity credential for Web / Mobile app users</strong></li></ul><h4>📂 AWS Directory Service</h4><ul><li>Integrate Microsoft AD (Active Directory) in AWS</li><li>AWS Managed Microsoft AD</li><li>AD Connector</li><li>Simple AD</li></ul><h4>🪪 AWS IAM Identity Center</h4><ul><li><strong>AWS SSO service (Single Sign-On)</strong></li><li>One login for multiple AWS accounts &amp; apps</li></ul><h3>Backup, Recovery &amp; Migration</h3><h4>🔁 AWS Backup</h4><ul><li>Fully-managed service to <strong>centrally manage &amp; automate backups </strong>across AWS services</li><li><strong>Support PITR (Point-in-Time Recovery)</strong></li><li>Cross Region Backup / Cross Account Backup</li></ul><h4>📖 Disaster Recovery Strategies</h4><ul><li>Backup &amp; Restore</li><li>Pilot Light</li><li>Warm Standby</li><li>Multi-site / Hot-Site</li></ul><h4>❤️‍🩹 AWS DRS — Elastic Disaster Recovery</h4><ul><li>Quickly &amp; easily recover physical, virtual, cloud-based servers into AWS</li><li><strong>Minimize downtime</strong> when service crash</li><li><strong>Continuous</strong> <strong>block-level replication</strong> for servers</li></ul><h4>💽 AWS DataSync</h4><ul><li>Move large amount of data from on-premise to AWS</li><li><strong>Incremental replication</strong> after first full load</li><li>Can sync to S3, EFS, FSx</li></ul><h4>📜 Cloud Migration Strategies</h4><ul><li><strong>Retire</strong></li><li><strong>Retain</strong></li><li><strong>Relocate</strong></li><li><strong>Rehost </strong>— “lift and shift”<br>- No cloud optimization<br>- eg. move a running mysql server from on-premise to EC2</li><li><strong>Replatform</strong> — ”lift and reshape”<br>- Not changing core, but leverage Cloud Optimization<br>- eg. on-premise MySQL to RDS</li><li><strong>Repurchase</strong> — “drop and shop”<br>- eg. move to a SaaS</li><li><strong>Refactor</strong> / <strong>Rearchitect<br>- </strong>eg. decouple a monolithic app into microservices</li></ul><h4>💡 AWS Application Discovery Service</h4><ul><li>Gather on-premise data center’s info then make migration plan</li><li>AWS Agent-less Dicvoery Connector<br>- Agent-less discovery</li><li>AWS Application Discovery Agent<br>- Agent-based discovery<br>- Gather more details</li><li>Integrate data in AWS Migration Hub</li></ul><h4>🚛 AWS MGN — Application Migration Service</h4><ul><li><strong>Re-host solution</strong> to migrate apps to AWS</li></ul><h4>💾 AWS Migration Evaluator</h4><ul><li>Help build <strong>data-driven business case for migration</strong></li><li>Install Agent-less Connector<br> → Take a snapshot of on-premise environment configs<br> → Analyze current state, define target, develop migration plan</li></ul><h4>🕋 AWS Migration Hub</h4><ul><li>Centrally manage migration-related data:<br>- inventory data for assessment<br>- tracking of migrations</li><li>AWS Migration Hub Orchestrator<br>- template for enterprise app migration (SAP, MSSQL)</li><li>Support migration status updates from MGN &amp; DMS</li></ul><h3>Other Services</h3><h4>🖥 Amazon WorkSpace</h4><ul><li><strong>DaaS</strong>: Desktop as a Service to provision <strong>Windows / Linux</strong> Desktop</li><li>Eliminate management of on-premise VDI (Virtual Desktop Infrastructure)</li></ul><h4>🎬 Amazon AppStream 2.0</h4><ul><li>Stream a desktop application using a <strong>web browser</strong></li><li>NO provisioned infrastructure required</li></ul><h4>🫀 AWS IoT Core</h4><ul><li><strong>Connect IoT devices to AWS</strong></li><li>Serverless, secure, scalable</li></ul><h4>📻 Amazon Elastic Transcoder</h4><ul><li>Convert media files in S3 into required format</li><li>Fully managed, secure, scalable, cost-effective</li></ul><h4>📲 AWS AppSync</h4><ul><li><strong>Store &amp; sync data across mobile and web apps</strong> in real time</li><li>Make use of GraphQL</li><li>Real-time subscription, Offline data sync</li><li>Integrated in AWS Amplify</li></ul><h4>🔊 AWS Amplify</h4><ul><li>Help user to <strong>quickly develop &amp; deploy full-stack web / mobile app</strong></li><li>Authentication, Storage, API, CICD, PubSub, Analytics, AI/ML, Monitoring…</li></ul><h4>🎶 AWS Application Composer</h4><ul><li><strong>Build serverless apps quickly with GUI</strong></li><li>Generate IaC using CloudFormation</li><li>Can import CloudFormation / SAM templates</li></ul><h4>🛖 AWS Device Farm</h4><ul><li>Run tests concurrently on <strong>multiple devices </strong>(mobiles, tablets)</li></ul><h4>💉 AWS FIS — Fault Injection Simulator</h4><ul><li>Fully-managed service to <strong>run fault injection experiments</strong></li><li>Based on <strong>Chaos Engineering</strong>, stressing app by <strong>creating disruptive events</strong></li></ul><h4>𓊍 AWS Step Functions</h4><ul><li>Build <strong>serverless visual workflow</strong> <strong>to orchestrate lambda</strong> functions</li><li>Features: sequence, parallel, conditions, timeouts, error handling</li></ul><h4>🛰️ AWS Ground Station</h4><ul><li>Fully-managed service to <strong>utilize satellite communication</strong></li></ul><h4>🎯 AWS Pinpoint</h4><ul><li>Scalable 2-way <strong>marketing</strong> communication service</li><li>Support email / <strong>SMS</strong> / push / voice / in-app message</li><li><strong>Segment &amp; personalize</strong> content of messages</li><li>Use case: <strong>run campaign</strong> by sending SMS messages</li></ul><h3>AWS Architecting Ecosystem</h3><h4>🏛️ WAF — Well Architected Framework</h4><p>6 Pillars:</p><ul><li><strong>Operational Excellence</strong><br>- Performance <strong>operation as code </strong>(eg. IaC, Config)<strong><br></strong>- <strong>Frequent, small, reversible changes<br></strong>- <strong>Anticipate failure &amp; </strong>learn from failure<br>- Use managed / serverless service<br>- Implement <strong>observability</strong> for actionable insights</li><li><strong>Security</strong><br>- Protect data in transit and at rest<br>- Keep people away from data<br>- Shared Responsibility Model</li><li><strong>Reliability<br></strong>- Automatic recovery<br>- Scale easily to match demand<br>- Stop guessing capacity</li><li><strong>Performance<br></strong>- Democratize advanced tech<br>- Go global in minutes<br>- Use serverless architectures</li><li><strong>Cost Optimization<br></strong>- Stop spending on data centers<br>- Use managed and app-level services to reduce cost of ownership</li><li><strong>Sustainability<br></strong>- Understand your impact<br>- Establish sustainability goals<br>- Maximize utilization</li></ul><h4>🛠️ AWS Well-Architected Tool</h4><ul><li>Free tool in the AWS Management Console</li><li>Review the state of architectures against 6 WAF pillars</li><li>Define workload and answer a set of questions, then it will provides a plan on adopting architectural best practices</li></ul><h4>📙 AWS CAF — Cloud Adoption Framework</h4><p>6 perspectives:</p><ul><li>3 Business capabilities:<br>- <strong>Business</strong><br>- <strong>People</strong><br>- <strong>Governance</strong></li><li>3 Technical Capabilities:<br>- <strong>Platform</strong><br>- <strong>Security</strong><br>- <strong>Operations</strong></li></ul><p>CAF Transformation Phase:</p><ul><li><strong>Envision</strong></li><li><strong>Align</strong></li><li><strong>Lauch</strong></li><li><strong>Scale</strong></li></ul><p><strong>Cloud Foundations:</strong></p><ul><li>Provides a guided path to help customers deploy, configure, and secure their new workloads while ensuring they are ready for on-going operations in the cloud</li><li>Help customers navigate through curated AWS Services, AWS Solutions, Partner Solutions, and Guidance.</li></ul><h4>🛒 AWS Marketplace</h4><ul><li>Digital catalog with plenty of software listings from 3rd party</li><li>eg.<br>- Custom AMI<br>- CloudFormation template<br>- SaaS<br>- Containers</li></ul><h4>💾 AWS Data Exchange</h4><ul><li>Easily find, subscribe to, and use <strong>third-party data</strong> in the cloud</li></ul><h4>🤝🏻 APN — AWS Partner Network</h4><ul><li><strong>APN Technology Partners<br>- </strong>provide hardware / connectivity / software</li><li><strong>APN Consulting Partners<br></strong>- professional services firm to help build on AWS</li><li><strong>APN Training Partners<br></strong>- help you learn AWS</li></ul><h4>💡 AWS IQ</h4><ul><li>Quickly find professional help for AWS projects</li><li>Engage and pay AWS-Certified 3rd party experts</li><li>As Customer / As Experts</li></ul><h4>👷🏻‍♂️ AMS — AWS Managed Services</h4><ul><li>AWS offers a team of experts to manage and operate all resources for clients</li></ul><p>待整理部分…</p><p><strong>AWS Architecture Center: <br></strong>provides reference architecture diagrams, vetted architecture solutions, Well-Architected best practices, patterns, icons, etc.</p><p><strong>Amazon OpenSearch</strong> <br>designed to set up, manage, and scale search and analytics solutions, providing the necessary functionalities to efficiently find articles based on different criteria.</p><p>Amazon MSK (Managed Streaming for Kafka)</p><p>AWS Total Cost of Ownership (TCO) Calculator allow you to estimate the cost savings when using AWS, compared to on-premises, and provide a detailed set of reports that can be used in executive presentations.</p><p><strong>AWS Billing Conductor</strong> is a customizable billing service that allows the organization to define billing groups, set pricing rules, create custom line items, and generate a unique Cost and Usage Report (CUR) for each billing group.</p><p><strong>AWS Activate for Startups</strong> program is specially designed to provide startups with the low-cost, easy-to-use infrastructure needed to scale and grow.</p><p>Amazon Elasticsearch Service is involved with operational analytics such as application monitoring, log analytics and clickstream analytics. Amazon Elasticsearch Service allows you to search, explore, filter, aggregate, and visualize your data in near real-time. <br><strong>→ Renamed as OpenSearch</strong></p><p>Amazon Simple Workflow Service (SWF) is a web service that makes it easy to coordinate work across distributed application components. SWF enables applications for a range of use cases, including media processing, web application back-ends, business process workflows, and analytics pipelines, to be designed as a coordination of tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba751c400675" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VSCode 新手必學的技巧錦囊]]></title>
            <link>https://medium.com/@40243105s/vscode-%E6%96%B0%E6%89%8B%E5%BF%85%E5%AD%B8%E7%9A%84%E6%8A%80%E5%B7%A7%E9%8C%A6%E5%9B%8A-a051e8352996?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a051e8352996</guid>
            <category><![CDATA[vscode-tips]]></category>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[vscode-shortcut]]></category>
            <category><![CDATA[vscode-keyboard-shortcuts]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Fri, 22 Nov 2024 16:17:07 GMT</pubDate>
            <atom:updated>2024-11-26T04:04:12.565Z</atom:updated>
            <content:encoded><![CDATA[<p>本篇目錄：</p><p><strong>UI 操作</strong></p><ul><li>查看快捷鍵列表：cmd + k + cmd + s</li><li>放大 / 縮小視窗：cmd + + / cmd + -</li><li>開啟 / 收合 Side bar：cmd + b</li><li>收合展開的目錄列表</li><li>開啟 / 收合終端機：cmd + `</li><li>根據檔名搜尋檔案：cmd + p</li><li>輸入 VSCode 指令：shift + cmd + p</li><li>全域關鍵字搜尋：shift + cmd + f</li><li>關閉當前分頁：cmd + w / 關閉全部分頁： cmd + k + w</li><li>比較兩份檔案內容差異</li></ul><p><strong>檔案編輯操作</strong></p><ul><li>游標移動至句首 / 句尾：cmd + 右鍵 / cmd + 左鍵</li><li>向右 / 向左 跨字移動：option + 右鍵 / option+ 左鍵</li><li>存檔： cmd + s</li><li>檔案關鍵字搜尋： cmd + f</li><li>往下新增一行：cmd + enter</li><li>註解 N 行程式碼：cmd + /</li><li>上下移動 N 行程式碼：option + 上 / option + 下</li><li>複製 N 行程式碼： shift + option + 下</li><li>產生多個 cursor 來同步編輯：option + 左鍵</li><li>根據指定字串，在下個相符字串的位置生成 cursor： cmd + d</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_t8xlIRbj2Epf5_E.jpeg" /></figure><h3>前言</h3><p>剛入門 coding 的初學者們，使用 VSCode 時往往還是習慣使用滑鼠進行 UI 操作。<br>此時，如果學會一些常用的快捷鍵等小技巧的話，將可省下手部在滑鼠、鍵盤間移動的時間，大幅提升 coding 的效率！</p><p>這篇文章簡單介紹幾個我 <strong>個人 </strong>在 VSCode 寫程式時最常用的幾個功能，如果你有額外想分享的小技巧也歡迎在留言區討論。</p><blockquote>PS 1.<br>以下這篇文章會以 MacOS 作為範例，如果是 Windows 的使用者，請在快捷鍵列表中搜尋我提供對應指令的「正式名稱」來查找。<br>(原則上 cmd -&gt; ctrl，option -&gt; alt)</blockquote><blockquote>PS 2.<br>我的操作介面會以英文顯示為主，如果你是想要持續學習 programming 的話，我會建議你也習慣操作英文介面、熟悉英文對應的關鍵字，未來在查找操作方式也比較容易對照。</blockquote><h3>UI 操作</h3><h4>✏️ 查看快捷鍵列表</h4><p>指令：cmd + k + cmd + s<br>（cmd 跟 k , cmd 跟 s 輸入時要連起來按）</p><p>正式名稱：Open Keyboard Shortcut</p><p>就像學國字前要先學如何查辭典一樣，如果找特定功能的快捷鍵，最好的方式當然是從快捷鍵列表來查詢。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rwvFkw4etzr61wqvJKsP4w.png" /></figure><p>在上面還有一個篩選表，可以根據關鍵字找你要的功能。</p><p>例如，輸入 keyboard 關鍵字，就可以看到這個功能：<br><strong>Open Keyboard Shortcuts</strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*98pWYhW9Ng-tQ5h0wRNaDA.gif" /></figure><p>連擊兩次 highlight 的欄位，可以自定成自己想要的快捷鍵設定。<br>(但原則上我不建議，還是以官方預設的快捷鍵為主。)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qIg_E6gDbwT7D_xmWU8ZFg.gif" /></figure><h4>✏️ 放大 / 縮小視窗</h4><p>指令：<br>- 放大：cmd + +<br>- 縮小：cmd + -</p><p>正式名稱：Zoom In / Zoom Out</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QhqbHX0hh7Sy1mWNzVdPw.gif" /></figure><h4>✏️️ 開啟 / 收合 Side bar</h4><p>指令：cmd + b</p><p>正式名稱：Toggle Primary Side Bar Visibility</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlTJetrstMYdVxckVkD-Cg.gif" /></figure><h4>✏️ 收合展開的目錄列表</h4><p>目錄列 (Explorer) 旁邊有一個小按鈕，可以幫你快速把展開的路徑收合<br>(Collapse Folder)。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*gIa8KEUo866DdTO7u3kGdQ.png" /><figcaption>Collapse Folder</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cXTQFD3yqoeXJQAdSqr7mQ.gif" /><figcaption>一鍵收合所有展開的路徑，只留下當前目錄下的物件</figcaption></figure><h4>✏️ 開啟 / 收合終端機</h4><p>指令：ctrl + ` <br>(`是跟波浪符 ~ 的同一個按鍵)</p><p>正式名稱：Toggle Terminal</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bL2CG-kKsfjTbkK_Y6sbPw.gif" /></figure><h4>✏️ 根據路徑/檔名搜尋檔案</h4><p>指令：cmd + p，然後輸入要找的檔案名稱。<br>不用完全一樣，只要近似的關鍵字即可。</p><p>正式名稱：Go to File</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HYb10Y-hNkm1eFxmJLkwfw.gif" /><figcaption>範例：先搜尋 sec- 關鍵字，再搜尋 ec2 關鍵字</figcaption></figure><h4>✏️ 輸入 VSCode 指令</h4><p>指令：shift + cmd + p，然後輸入關鍵字。<br>( 或者先按 cmd + p 以後，再輸入一個 &gt; 也行 )</p><p>正式名稱：Show All Command</p><p>在 VSCode 裡，語言、環境、套件都有相對應的功能可以透過指令進行操作。<br>這裡簡單示範「選擇執行Python 編譯器版本」，指定編譯時使用的 Python 版本，可以看到兩次執行後的結果因編譯器而不同：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6UA8UoMMG79gcoSaNgbnQ.gif" /></figure><h4>✏️ 全域關鍵字搜尋</h4><p>指令：shift + cmd + f，然後輸入關鍵字。<br>(或是直接點擊左邊 Sidebar 的放大鏡圖案即可)</p><p>正式名稱：Find in Files</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I6iDjo7ZOAL5TUxzRG1uWg.gif" /></figure><p>除了基本的搜尋以外，在輸入欄的右邊還有三個調整選項，由左至右分別是</p><ol><li><strong>Match Case</strong>: 大小寫要一致 (case-sensitive)</li><li><strong>Match Whole Word</strong>: 比對的字串是完整的一個字，而不是片段</li><li><strong>Use Regular Expression</strong>: 使用正規表達式搜尋字串</li></ol><p>（正規表達式 Regex 不在本篇的教學範圍，有興趣的同學可以自己學一下）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*CwD3Me6EGbyVia3YJhrriA.png" /><figcaption>1: <strong>Match Case</strong> 2: <strong>Match Whole Word</strong> 3: <strong>Use Regular Expression</strong></figcaption></figure><p>此外，點擊選項下面的 … 按鈕，還可以使用 Regex 來限定：</p><ul><li><strong>files to include</strong>: 要包含的檔案範圍，如：輸入 courses/ ，代表<strong>只搜尋</strong> courses資料夾底下的檔案。</li><li><strong>files to exclude</strong>: 要排除的檔案範圍，如：輸入 courses/，代表搜尋 courses <strong>資料夾以外</strong>的檔案。</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*OMxNlwkq3MIptpebXkKcRQ.png" /><figcaption>點擊三個選項下方的 <strong>…</strong> 按鈕，展開<strong>搜尋範圍選項</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aE5ompRR1uxPVyjXbNrXmw.gif" /></figure><h4>✏️ 關閉分頁</h4><p>關閉<strong>當前</strong>分頁： cmd + w</p><p>正式名稱： Close Window</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MuKg_lpH8bYHE8reWYg4CQ.gif" /></figure><p>關閉<strong>所有</strong>分頁： cmd + k + w</p><p>正式名稱： Close All Editors in Group</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8xF-rzlty6Gxj_368khog.gif" /></figure><h4>✏️ 比較檔案差異</h4><p>假如我有兩個檔案: test.py 與 test-copy.py，要比較兩個檔案內容上有沒有差異：</p><ol><li>先在 Explorer 中，在 test.py 上按右鍵，選擇 <strong>Select for Compare</strong></li><li>接著在 test-copy.py 上按右鍵，選擇 <strong>Compare with Selected</strong></li><li>就可以看到一個比較兩個檔案差異的 Editor 畫面，差異部分會以紅/綠色來顯示。</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2KyjgguIk3VhL39pJXdC2g.gif" /></figure><h3>程式編輯操作</h3><p>接下來的部分是與程式編輯相關的快捷鍵，在開啟的 Editor 頁面進行操作。</p><h4>✏️ 快速移動游標 (Cursor)</h4><ul><li>cmd + 右鍵 -&gt; 移動到該行最右</li><li>cmd + 左鍵 -&gt; 移動到該行最左</li><li>option + 右鍵 -&gt; 向右跨字移動</li><li>option + 左鍵 -&gt; 向左跨字移動</li></ul><h4>✏️ 存檔</h4><p>指令： cmd + s</p><p>正式名稱：File: Save</p><p>在對檔案進行編輯後，會看到檔案名稱旁邊有一個小白色圓點，代表編輯的內還沒有存檔。</p><p>按下 cmd + s 後，可以看到原點消失，代表檔案已經儲存。</p><p>在運行任何 Python 檔案之前，都請務必記得先存檔。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pkc3lB3dysPE-9Y20pSkHA.gif" /></figure><h4>✏️ 檔案關鍵字搜尋</h4><p>指令：cmd + f，然後輸入要查找的關鍵字。</p><p>正式名稱：Find</p><p>搜尋欄旁邊也有三個跟全域搜尋時一樣的搜尋設定，這邊就不重複了。</p><p>搜尋欄旁邊的上下箭頭，可以找 上一個/下一個 符合的字段。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxi10xJRQOMpOYgDHVFVeQ.gif" /></figure><p>按一下搜尋欄做邊的三角形按鈕，可以展開一個「<strong>取代字串</strong>」功能，輸入你像把目標字串轉換成的取代字串，然後按旁邊的 Replace / Replace All 選項，就可以快速取代特定關鍵字。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*fARAtZDv0voUxRHs9JnMgw.png" /><figcaption>按鍵 1 : Replace，取代當前比對到的字串 / 按鍵 2: Replace All，取代這個檔案裡所有比對到的字串</figcaption></figure><p>在下面這個示範，我搜尋了檔案中的 <strong>number</strong>，並使用 Replace All 功能，將 number 取代為 <strong>num。</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W0pmAK1YszxqPzwB7TBPqw.gif" /></figure><h4>✏️ 往下新增一行</h4><p>以前我們如果想在下方插入一行空白行的話，習慣上都會先將鼠標移動到尾部，然後按下 Enter 來插入，同時 cursor 也會移動到新的一行上。</p><p>在 VSCode 裡，只要按下 cmd + enter，就可以馬上往下插入一行。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GWQ2FxhC_kKIa2KRMtvfdg.gif" /><figcaption>即使不移動到行尾，只要按下 cmd + enter 就可以向下插入一行。</figcaption></figure><h4>✏️ 註解 N 行程式碼</h4><p>將游標 (cursor) 移動到要註解的程式碼上，然後按 cmd + / 。<br>對註解的程式碼再按一次cmd + / 則可以取消註解。</p><p>正式名稱：Toggle Line Commant</p><p>這個快捷鍵適用於<strong>所有程式語言的註解</strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55O3eYZ62Xydg1_C-ZAgsw.gif" /><figcaption>按下 cmd + / 註解程式碼；對註解的程式碼再按一次 cmd + / 來取消註解</figcaption></figure><p>如果想要多行註解，只要<strong>按住</strong> shift、按上下鍵移動，<strong>反白要 comment 的行數後</strong>，再按下 cmd + / 後就可以一次註解這一段 code block 了。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rSreGhiL0wpSU4rcdwoszQ.gif" /></figure><h4>✏️ 上下移動 N 行程式碼</h4><p>將 cursor 移動到你要搬移的行數上，在<strong>按住</strong> option 後再按 上/下 鍵，就可以將該行程式碼往上 / 往下移動。</p><p>正式名稱：Move Line Down / Move Line Up</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ul1w-p2wS-kSPjCvtQ90Cg.gif" /></figure><p>同理，反白程式碼段落就可以一次搬移多行程式碼。</p><h4>✏️ 複製 N 行程式碼</h4><p>將 cursor 移動到你要複製的行數上，按下 shift + option + 下，就可以複製該行程式碼</p><p>正式名稱：Copy Line Down</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qIPGw4_uCc-5qr5j1QH92w.gif" /></figure><p>同理，反白程式碼段落就可以一次複製多行程式碼。</p><h4>✏️ 產生多個 Cursor 來同步編輯</h4><p><strong>按住</strong> option 後，按下左鍵，可以在鼠標處產生一個 cursor。繼續在下一個地方按一次左鍵，就可以再產生一個 cursor，達到<strong>多處同步編輯</strong>的效果。</p><p>按下 esc 來回復到一個 cursor 的狀態。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZANxG9kjN16Rt_ScPXKww.gif" /></figure><h4>✏️ 根據指定字串，在下個相符字串的位置生成 cursor</h4><p>移動到你要調整的字串，按下 cmd + d，VSCode 會自動在下一個符合字串旁邊加上一個 cursor，進行多處同步編輯。</p><p>正式名稱：Add Selection to Next Find Match</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uk24HzqGIDRFSfxKl1dPjQ.gif" /></figure><p>除了完整的單字，你也可以只反白一段你要比對字串，然後按 cmd + d 產生多個 cursor。</p><p>在下面的例子裡，雖然變數名稱不一樣( test_a ~ test_e )，但我可以透過反白同樣的前綴 “test_” ，然後產生 cursor 進行同步編輯。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8QcdElty9r_JyKK11m4ilg.gif" /></figure><p>multi-cursor 還有一個很猛的功能：如果你在 multi-cursor 狀態下進行複製，可以一次複製多行內容，再到對應的地方貼上去。</p><p>比如說，在下面這個例子，我透過 “num_” 前綴複製了 num_a ~ num_c 的變數名稱，再將 multi-cursor 移動到下面 “new_num_” 的變數段落按貼上，可以看到 new_num_a 段落貼上的內容是 num_a，new_num_b 段落貼上的內容是 num_b，以此類推。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l98GPKrn5l43Fv9Yb_Z0hA.gif" /></figure><h3>結尾</h3><p>這篇整理了我個人覺得最實用的 VSCode 功能，實際上還有很多沒收錄的遺珠之憾，如果你有興趣可以自己再搜尋官網或 YT 教學影片研究。</p><p>想當然的，一口氣認識這麼多快捷鍵和小功能肯定無法馬上記住，不過只要是常用的功能，時間久了肌肉記憶會自然而然的熟能生巧，需要的話也可以把這篇文章收藏起來，有需要的話就隨時參考一下。</p><p>善用 VSCode 的這些小功能，相信一定能大幅提升你的開發效率，一步步往程式高手的方向邁進，加油💪！</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a051e8352996" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Py立得 — 高手都該會的串列函示: filter, map, reduce]]></title>
            <link>https://medium.com/ccclub/py-instant-tips-filter-map-reduce-a37f942a486f?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a37f942a486f</guid>
            <category><![CDATA[functional-programming]]></category>
            <category><![CDATA[py-立得]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[python進階]]></category>
            <category><![CDATA[lists-functions]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 07:46:13 GMT</pubDate>
            <atom:updated>2024-11-06T07:46:13.785Z</atom:updated>
            <content:encoded><![CDATA[<h3>Py立得 — 高手都該會的串列函示：filter、map、reduce</h3><h4>向高手學習串列函示的用法，優化雜亂的程式碼！</h4><p>在這篇文章中，你將會學到：</p><ul><li>如何用 <strong>filter</strong> function 篩選出符合情境的 list element。</li><li>如何用 <strong>map</strong> function 將 list element 轉化成特定的資料。</li><li>如何用 <strong>reduce</strong> function 歸納 list element 資料。</li><li>理解如何使用 list function 來提升程式碼的品質。</li></ul><p>（感謝夥伴 <a href="https://medium.com/@yuehchloe">Chloe Yueh</a> 的美編支援）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*NgKmeT_0DwaWKV753xjjTw.png" /></figure><p>當我們在 Python 開發中遇到「逐筆取得 list element」的操作情境時，通常會用以下幾種 for 迴圈的寫法來完成：</p><pre>my_list = [0, 1, 2, 3]<br><br># 1. 用 in operator 逐一取出 element<br>for element in my_list:<br>    print(element)<br><br><br># 2. 用 range 迭代 list 中的 index，再用 index 取出 element<br>for index in range(len(my_list)):<br>    element = my_list[index]<br>    print(f&quot;{index}: {elemen}&quot;)<br><br><br># 3. 利用 enumerate function<br># 可以同時迭代 index 與對應的 element 變數<br>for index, element in enumerate(my_list):<br>    print(f&quot;{index}: {elemen}&quot;)</pre><p>在這篇文章中，要帶大家認識三個與 list 迭代相關進階的 function:</p><ul><li>filter</li><li>map</li><li>reduce</li></ul><p>讓我們配合情境理解，一起認識這三個 function 的機制與用法吧！</p><h3>情境：Py 嬸的考試成績統計</h3><p>數學考試結束了，數學老師 Py 嬸 為了方便統計成績數據，設計了一個 class: Student，裏面紀錄了學生的姓名、性別、考試成績。</p><p>再來，Py 嬸把課程中的六位學生資料，透過 Student class 物件來儲存，然後統一集合到 students list 中</p><pre>class Student:<br>    name: str<br>    gender: str<br>    score: int<br><br>    def __init__(self, name, gender, score) -&gt; None:<br>        self.name = name<br>        self.gender = gender<br>        self.score = score<br><br># 3 個男同學<br>alex = Student(&quot;Alex&quot;, &quot;Male&quot;, 75)<br>mike = Student(&quot;Mike&quot;, &quot;Male&quot;, 82)<br>hugo = Student(&quot;Hugo&quot;, &quot;Male&quot;, 52)<br><br># 3 個女同學<br>sharon = Student(&quot;Sharon&quot;, &quot;Female&quot;, 69)<br>ann = Student(&quot;Ann&quot;, &quot;Female&quot;, 90)<br>jasmine = Student(&quot;Jasmine&quot;, &quot;Female&quot;, 58)<br><br>students = [<br>    alex,<br>    mike,<br>    hugo,<br>    sharon,<br>    ann,<br>    jasmine,<br>]</pre><p>接下來，就看我們如何用 list function，幫 Py 嬸整理出所需要的數據吧！</p><h3>filter</h3><p>Py 嬸身為一名老師，最關心的第一件事當然是這次有哪幾位學生考得不及格。<br>她希望能快速找到這些不及格學生的名單，並為他們安排課後輔導。</p><p>想要快速的從 list 中<strong>篩選出符合特定條件的 element</strong>，我們可以用 <strong>filter() </strong>function！</p><p><a href="https://docs.python.org/3/library/functions.html#filter"><strong>filter</strong></a> 的使用方式如下：</p><blockquote>filter(<em>function</em>, <em>iterable</em>)</blockquote><p>第一個參數 function 為「判斷 element 是否符合條件」的函式；<br>第二個參數 iterable ，則是要篩選的 list 變數。</p><p>（不熟悉 function as parameter 用法？趕快看 <a href="https://medium.com/ccclub/py-instant-tips-lambda-function-c71e0a1499d8">這篇文章</a> 複習一下📖）</p><p>當我們設計 <strong>filter</strong> 的第一個參數 function（也就是我們拿來篩選 element 判斷函式）時，需要注意：</p><ol><li>function 的參數為 list 中的 element</li><li>function 將回傳 boolean 作為篩選 element 的依據<br>(True -&gt; 搜集 element，False -&gt; 篩除 element)</li></ol><p>最終，<strong>filter</strong> 會將篩選後的 elements 以 filter object 的形式回傳。<br>最後再將 filter object 代入 list()function 完成計算。</p><p>讓我們配合圖示，看看 <strong>filter</strong> 是怎麼運作的：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*cPnFBIruOsaMMcrSzz14Zg.png" /></figure><p>(1) 呼叫 <strong>filter</strong> ，帶入 my_func 和 my_list</p><p>(2) <strong>filter</strong> 會逐一取出 my_list 中的 element，並帶入 my_func 中</p><p>(3) 如果 my_func(element) 的結果為 True ，則把該 element 放入 filter object 中；最後在遍歷所有 element 後回傳 filter object</p><p>讓我們再用正常的 for 迴圈，模擬 <strong>filter</strong> 的功能，加強大家的理解：</p><pre>def mock_filter(func, target_list):<br>    result = list()<br><br>    for element in target_list:<br>        if func(element):<br>            # 如果 check_func(element) 結果為 True，則搜集起來<br>            result.append(element)<br>    <br>    # 完成迴圈後，回傳搜集後的結果<br>    return result<br><br># ！注意！：這裡的示範回傳的是 list<br># 實際上 filter 是回傳 filter object</pre><p>有了 <strong>filter</strong> 這項工具，接下來 Py 嬸只要定義 <strong>filter</strong> 的參數 function 為 「score attribute 小於 60 分」，就可以快速篩選出不及格的學生囉！</p><pre># 篩選條件為 score &lt; 60 分的學生<br>failed_students = filter(<br>    lambda student: student.score &lt; 60,<br>    students<br>)<br><br># 記得將 filter object 轉換成 list<br>failed_students = list(failed_students)<br><br>for student in failed_students:<br>    print(f&quot;{student.name} 不及格，成績為 {student.score}&quot;)<br><br># Hugo 不及格，成績為 52<br># Jasmine 不及格，成績為 58</pre><p>在上面的程式中，我們成功透過 <strong>filter</strong> function，找到 Hugo 和 Jasmine 兩個即將留校察看倒霉蛋😅。</p><p>如果 Py 嬸再進一步分別登記「男生不及格人數」與「女生不及格人數」，我們只要稍稍的調整前面示範中判斷 function 就可以囉！</p><pre># 篩選不及格男生<br>failed_boys = filter(<br>    lambda exam: exam.score &lt; 60 and exam.gender == &quot;Male&quot;,<br>    students,<br>)<br>failed_boys = list(failed_boys)<br><br># 篩選不及格女生<br>failed_girls = filter(<br>    lambda exam: exam.score &lt; 60 and exam.gender == &quot;Female&quot;,<br>    students,<br>)<br>failed_girls = list(failed_girls)</pre><blockquote><strong>🔋 知識充電站 🔋</strong></blockquote><blockquote>如果將 None 作為 <strong>filter</strong> 的第一個參數，則 <strong>filter</strong> 會使用 bool() function 作為 <strong>filter</strong> 的判斷函示，透過這樣的寫法可以篩除掉 list 中的<br>- None<br>- 0<br>- 空字串<br>- 空 list<br>- 空 dict<br>等等「帶入 bool() 後結果為 False」的變數。</blockquote><pre>some_list = [<br>    99,<br>    0,<br>    None,<br>    list(),             # 空 list<br>    dict(),             # 空 dictionary<br>    &quot;Test the result&quot;,<br>    &quot;&quot;,                 # 空字串<br>    [1, 2, 3],<br>]<br><br>filtered_list = list(filter(None, some_list))<br><br>print(filtered_list)<br># [99, &#39;Test the result&#39;, [1, 2, 3]]</pre><h3>map</h3><p>接下來，Py 嬸想要根據各個學生的成績高低，印出相對應的成績評語：</p><blockquote>如果成績 ≥ 80，評語是 “{學生姓名}: {分數}, 超級優秀！” ；<br>如果成績 ≥ 60，評語是 “{學生姓名}: {分數}, 繼續努力！” ；<br>如果成績 &lt; 60，評語是 “{學生姓名}: {分數}, 需要加油！” ；</blockquote><p>Py 嬸該怎麼做，才能把學生 list 資料，轉換成對應的評語 list 儲存呢？🤔</p><p>遇到這種「將 list 中的每一個 element，<strong>經過同樣的處理步驟</strong>，轉換成對應的結果」的情境，我們可以使用 <strong>map</strong> function!</p><p><strong>map</strong> (映射) 一詞在程式語言的世界中有「對應關係」的意涵，比如說 Python 的 dictionary 就可以說是一個 key-value mapping 的儲存形式。</p><p>而 Python 中的 <strong>map()</strong> 也有「對應」的概念：<strong>map</strong> 會將 list 中的 element 逐一取出，根據傳入的 function 轉換成對應的 element’，最後用新的 list 儲存轉換後的結果。</p><p><a href="https://docs.python.org/3/library/functions.html#map"><strong>map</strong></a> 的使用方式如下：</p><blockquote><strong>map</strong>(<em>function</em>, <em>iterable)</em></blockquote><p>第一個參數 function 為「將 element 轉換成 element’」的函式；<br>第二個參數 iterable ，則是要篩選的 list 變數。</p><p>把 function 和 list 傳入 <strong>map</strong> 後，<strong>map</strong> 會將轉換後的結果以 map object 的型別回傳。<br>跟 filter 操作一樣，我們一樣把 map object 代入 list() function，完成 map 的運算。</p><p>讓我們配合圖示，看看 <strong>map</strong> 是怎麼運作的：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*YZ7IIOWwJMormbxfNRj8kg.png" /></figure><p>(1) 呼叫 <strong>map</strong> ，帶入 my_func 和 my_list</p><p>(2) <strong>map</strong> 會逐一取出 my_list 中的 element，並帶入 my_func 中</p><p>(3) 每個 element 在經過 my_func 後，取得一個對應的 element’ 結果，將所有轉換後的結果儲存在 map object 後回傳。</p><p>讓我們嘗試用 for 迴圈的形式模擬 <strong>map</strong> 的效果：</p><pre>def mock_map(func, target_list):<br>    results = list()<br><br>    for elem in target_list:<br>        # 將 elem 帶入 func<br>        # 取得轉換後的結果 transformed_elem<br>        transformed_elem = func(elem)<br><br>        # 將 transformed_elem 儲存到 results list 中<br>        results.append(transformed_elem)<br>        <br>    return results<br>    <br># ！注意！：這裡的示範回傳的是 list<br># 實際上 map 是回傳 map object</pre><p>學會了 <strong>map</strong> 後，Py 嬸就可以設計一個</p><blockquote>將 Student 物件，根據 score 和 name，轉換成特定的 string 型別的評語</blockquote><p>的 function，配合 <strong>map</strong> 的效果，取得轉換後的評語 list 囉！</p><pre>def transform_student_to_remark(student: Student):<br>    if student.score &gt;= 80:<br>        remark = &quot;非常優秀！&quot;<br>    elif student.score &gt;= 60:<br>        remark = &quot;繼續努力！&quot;<br>    else:<br>        remark = &quot;需要加油！&quot;<br>    <br>    return f&quot;{student.name}:{student.score}, {remark}&quot;<br><br><br># map function 回傳 map object<br>remarks = map(transform_student_to_remark, students)<br><br># 把 map object 轉換成 list<br>remarks = list(remarks)<br><br>for remark in remarks:<br>    print(remark)<br><br># Alex:75, 繼續努力！<br># Mike:82, 非常優秀！<br># Hugo:52, 需要加油！<br># Sharon:69, 繼續努力！<br># Ann:90, 非常優秀！<br># Jasmine:58, 需要加油！</pre><p>我們再來嘗試一個例子：如果 Py 嬸想要幫大家的成績上調 10%，一樣可以透過 <strong>map</strong> 來完成：</p><pre>import copy<br><br># 將 Student.score 上調 10% 的函數<br>def adjust_ten_percent(student: Student):<br>    # 先複製一份先的物件<br>    copy_student = copy.deepcopy(student)<br>    # 上調備份 Student 物件的分數<br>    copy_student.score = int(student.score * 1.1)<br>    <br>    # 最後記得把調整後的 copy_student 回傳<br>    return copy_student<br><br>adjusted_students = map(<br>    adjust_ten_percent,<br>    students,<br>)<br><br># 將 map object 轉換成 list<br>adjusted_students = list(adjusted_students)<br><br>for student in adjusted_students:<br>    print(f&quot;{student.name}: {student.score}&quot;)<br><br># Alex: 82<br># Mike: 90<br># Hugo: 57<br># Sharon: 75<br># Ann: 99<br># Jasmine: 63</pre><blockquote><strong>🔋 知識充電站 🔋</strong></blockquote><blockquote>如果有學過 list comprehension 的同學，可能會發現 <strong>filter</strong> 和 <strong>map</strong> 的效果其實也可以用 list comprehension 的方式來達成！</blockquote><blockquote>下面就示範一組簡單的情境，讓我們用 list comprehension 與新學的 <strong>filter</strong>/<strong>map</strong> function，來完成「從數列中取得偶數」與「將數列的各個數字變成兩倍大」的任務吧！</blockquote><pre>nums = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br><br># 取得偶數<br>even_by_filter        = list(filter(lambda num: num % 2 == 0, nums))<br>even_by_comprehension = [num for num in nums if num % 2 == 0]<br><br>print(even_by_filter)<br># [2, 4, 6, 8, 10]<br>print(even_by_comprehension)<br># [2, 4, 6, 8, 10]<br><br># 將各個數字乘以2<br>two_times_by_map           = list(map(lambda num: num * 2, nums))<br>two_times_by_comprehension = [num*2 for num in nums]<br><br>print(two_times_by_map)<br># [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]<br>print(two_times_by_comprehension)<br># [2, 4, 6, 8, 10, 12, 14, 16, 18, 20]</pre><blockquote><strong>🔋 知識充電站 🔋 — 為什麼不直接回傳 list ?<br><br></strong>有人可能會好奇，為什麼在 <strong>filter</strong> 與 <strong>map</strong> 中， Python 不直接回傳 list 就好，反而是給了我們一個 filter/map object，還要使用者自己在包一層 list 再使用呢？</blockquote><blockquote>這個問題牽涉到一個 Python 的一個特性：惰性求值 <strong>Lazy Evaluation</strong>，Python 會在<strong>你調用物件時才去計算變數的值</strong>，以節省CPU與記憶體的資源。</blockquote><blockquote>有興趣深入理解 <strong>Lazy Evaluation </strong>機制的同學可以看下列兩篇文章：<br>- <a href="https://realpython.com/python-lazy-evaluation/">What’s Lazy Evaluation in Python?</a> (Python 中不同的惰性求值情境)<br>- <a href="https://medium.com/@yuhanschen/python-iterator-iterable-e599ff3ab5d0">Python — Iterator &amp; Iterable(Basic)</a> (Python 如何實作惰性求值效果)</blockquote><blockquote>最後讓我們用 code 來看看 <strong>Lazy Evaluation </strong>的效果吧！</blockquote><pre>def map_func(item):<br>    print(f&quot;Evaluating item {item}...&quot;)<br>    return item * 2<br><br>nums = [1, 2, 3]<br><br>map_obj = map(map_func, nums)  # 這時候還沒開始進行 map 運算<br><br>print(&quot;Ready to evaluate the map_obj...&quot;)<br>print()<br><br>for result in map_obj:   <br>    # 逐一取出 item 的時候<br>    # 才會把 item 代入 map_func 來計算<br>    print(f&quot;Getting map result {result}&quot;)<br>    print()<br><br><br># output 運算結果<br>Ready to evaluate the map_obj...<br><br>Evaluating item 1...<br>Getting map result 2<br><br>Evaluating item 2...<br>Getting map result 4<br><br>Evaluating item 3...<br>Getting map result 6</pre><h3>reduce</h3><p>在登記完所有學生的考試成績後，接下來 Py 嬸準備做一些數據分析，比如說找到最高分的同學，還有統計這次考試的人次、總分，以計算平均分數等等。</p><p>如果想要將 list 的各個 element 結果，<strong>歸納、整理為一筆單一的資料</strong> (英文術語叫做 aggregate )，可以使用 <strong>reduce</strong> function！</p><p>reduce 在英文語意中有「減少、縮減規模」的意涵，而在程式應用中，則是將一個 list -- 由數個 element 組合而成的資料結構 -- 整理成一筆資料。</p><p>Python 的 <strong>reduce</strong> 是來自 functools module 的 function，<a href="https://docs.python.org/3/library/functools.html#functools.reduce"><strong>reduce</strong></a> 的使用方法如下：</p><blockquote>reduce(function, iterable, carry)</blockquote><p>除了 function、list 參數輸入外，還額外多了一種參數：carry。</p><p>reduce 在遍歷 element 的過程，<strong>會持續把 carry 傳遞給下一個 function 參數</strong>；而使用者可以在 function 參數中決定要怎麼將資料搜集到 carry 中，並在最後的將 carry return，以傳遞給下一個 function。</p><p>讓我們配合圖示，看看 reduce 是怎麼運作的：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*_oeC1UKLped0sx6L0mO93A.png" /></figure><p>(1) 呼叫 <strong>reduce</strong> ，帶入 my_func 和 my_list，以及初始的 carry_0</p><p>(2) 遍歷到 elem_0 時，my_func 接收 carry_0 與 elem_0，經過計算後，回傳一個 carry 值 (圖示中的 carry_1)；接下來就不斷重複一樣的操作。</p><p>(3) 在遍歷完所有的 elem 後，最後 return 的 carry_n ，就是最終 reduce 回傳的結果。</p><p>如果把 <strong>reduce</strong> 寫成 for 迴圈形式，則看起來會是這樣：</p><pre>def mock_reduce(func, target_list, carry):<br>    for elem in target_list:<br>        # 每遍歷一次，把 carry 與 elem 都帶入 func<br>        # 回傳的結果就是新的 carry<br>        carry = func(carry, elem)<br>    <br>    # 遍歷完成後，去得的結果就是 carry<br>    return carry</pre><p>有了 <strong>reduce</strong> 的概念後，我們來看看 Py 嬸要怎麼利用 reduce，統計應試人數與成績總分，來取得考試平均分數吧！</p><pre>from functools import reduce<br><br># 要提供在 reduce 參數的 func<br># 第一個參數是 carry<br># 第二個參數是遍歷的 element<br>def collect_statistics(carry: dict, student: Student):<br>    # 計算總人數<br>    carry[&#39;total_student&#39;] += 1<br>    # 計算總分<br>    carry[&#39;total_score&#39;] += student.score<br>    # 回傳 carry，讓 carry 傳遞到下一個 func<br>    return carry<br><br>statistics = reduce(<br>    collect_statistics,<br>    students,<br>    {<br>        &quot;total_student&quot;: 0,     # 初始的 carry_0:<br>        &quot;total_score&quot;: 0,       # 總分與人數數據都是 0 的 dict<br>    },<br>)<br><br>print(statistics)<br># {&#39;total_student&#39;: 6, &#39;total_score&#39;: 426}</pre><p>經過 <strong>reduce</strong> 的整理後，我們成功的從 students list 中統計到總人數與總成績，接著只要兩者相除，就可以得到這次考試的平均分數囉！</p><p>我們再來練習一下，如何用 reduce 找到這次考試的第一名學生：</p><pre>from functools import reduce<br><br>top_student = reduce(<br>    lambda carry, student: carry if carry.score &gt; student.score else student,<br>    students,<br>)<br><br>print(f&quot;top student: {top_student.name}, score: {top_student.score}&quot;)<br># top student: Ann, score: 90</pre><p>在上面的程式中，有兩個地方需要說明：</p><ol><li>如果 <strong>reduce</strong> 沒有帶入初始 carry 值，那 <strong>reduce</strong> <strong>會以第一個 element: elem_0 作為 carry</strong>，並跳過 elem_0 的遍歷，從 func(carry, elem_1) 開始。</li><li>帶入的 lambda function，接收兩個參數：carry 和 student。<br>回傳的結果則是一個三元運算子 (<a href="https://docs.python.org/3/reference/expressions.html#conditional-expressions">ternary operator</a>)：如果 carry.score 大於 student.score，則回傳 carry；反之，則回傳 student。</li></ol><p>透過這樣的寫法，我們等於是在每一次的遍歷中，比較<br>「目前為止看到最高分的 student 成績」<br>與<br>「當前 student的成績」<br>看誰的成績更高，就以他作為下一次的 carry 結果。</p><p>相對於 <strong>filter</strong> 和 <strong>map</strong>，<strong>reduce</strong> 在運用上會稍微複雜一點，新手務必在練習時注意以下兩個重點：</p><ol><li>帶入 <strong>reduce</strong> 的 function，接收的第一個參數是 carry，第二個參數是遍歷到的 element。<br>你可以按情境調整變數名稱(ex. 你可以寫成 lambda ca, stu)，<strong>但順序不可以弄錯</strong>！</li><li>很多新手在設計 <strong>reduce</strong> 的 func 時，只顧著對 carry 進行操作，忘記在最後 return carry，這樣在下一次遍歷的 func 執行時，<strong>carry 的變數值會變成 </strong>None ！</li></ol><pre>from functools import reduce<br><br>def add_up(carry, num):<br>    if carry is None:<br>        # 如果 carry 是 none 的話，拋出 exception<br>        print(&quot;current num is &quot; + str(num))<br>        raise Exception(&quot;Carry is None!!!&quot;)<br><br>    carry = carry + num<br>    # 假設這裡忘記 return<br><br>nums = [1,2,3,4]<br>total_amount = reduce(add_up, nums, 0)<br>print(total_amount)<br><br># ---------------------------------------------<br># current num is 2<br># Traceback (most recent call last):<br># .....<br>#    total_amount = reduce(add_up, nums, 0)<br># .....  <br># Exception: Carry is None!!!</pre><h3>Not Bad…But Why?</h3><p>認識完三個 list function 後，相信不少同學會在心中默默浮現一個疑問：</p><blockquote>既然上面程式範例的情境需求，實際上都可以用 for 迴圈的形式達成，那為什需要多學這幾種 list function，多背這些 function 的用法，這不是自找麻煩嗎🤔？</blockquote><p>的確，以功能面而言，我們可以用新手更熟悉的 for loop 形式，完成 filter / map / reduce 的效果，但我還是鼓勵大家多嘗試使用 list function，原因如下：</p><blockquote>💡 <strong>1. 增加程式碼可讀性，減少認知負荷</strong></blockquote><p>除了順暢運行外，一份好的程式碼，還應該要「易於理解」，讓閱讀者在瀏覽程式碼時，<strong>能夠用最短的時間，正確的理解這段程式碼的意圖</strong>，也就是我們常聽到的「<strong>可讀性 Readibility</strong>」。</p><p>以下面這段程式碼為例：</p><pre>from datetime import date<br><br>results = []<br><br>for order in orders:<br>    if order.dispatch_date &gt; date.today():<br>        continue<br>    <br>    buyer = order.buyer<br>    if not buyer.account.verified:<br>        continue<br>    # ...code...<br>    results.append(order)</pre><p>這段程式碼，如果只給你三秒鐘的時間觀察，大概只能出解讀 2 個訊息:</p><ul><li>在 for 迴圈中逐步檢視訂單 order</li><li>order 有可能在某些情況下被放入 results list 中</li></ul><p>至於在 for 迴圈中，order 有沒有什麼調整異動？ results 在中間有沒有機會加入其他非 order 的資料？在 for 迴圈中還有什麼變數操作？抱歉，這些你都只能透過逐行檢視程式碼，才能確認更多的細節。</p><p>然而，如果我把上面的程式碼，改寫成以下的樣子：</p><pre>def is_order_verified_to_be_deliver(order):<br>    if order.dispatch_date &gt; date.today():<br>        return False<br><br>    buyer = order.buyer<br>    if not buyer.account.verified:<br>        return False<br>    # ...code...<br>    return True<br><br><br>results = list(filter(<br>    is_order_verified_to_be_deliver,<br>    orders,<br>))</pre><p>透過 <strong>filter</strong> function 的運用，我可以在極短的時間內，得知</p><ul><li>results 結果是某些符合篩選條件的 order</li><li>“is_order_verified_to_be_deliver” 這個 function 是用於判斷 order 是否符合條件</li></ul><p>使用 list function，讓閱讀者可以馬上領略程式碼的意圖，快速確認下一步行動。</p><p>比如說，如果我要對 order 的合格條件新增一個「必須是國內運送」，我可以放心的將注意力移動到 “is_order_verified_to_be_deliver” 這個判斷function，微幅調整 function 定義，不用花大把的時間爬梳 for 迴圈的內容，思考要如何將新的邏輯嵌入既有的程式中，而不影響既有的功能。</p><blockquote>💡 <strong>2. 限縮變數範圍，專注於關鍵變數</strong></blockquote><p>一樣來看剛剛 order 範例的程式碼:</p><pre># ...code...<br>my_package_numbers = 200<br><br>results = []<br><br>for order in orders:<br>    if order.dispatch_date &gt; date.today():<br>        continue<br>    <br>    buyer = order.buyer<br>    if not buyer.account.verified:<br>        continue<br>    # ...code...<br>    results.append(order)</pre><p>在 for 迴圈這段篩選邏輯的程式碼中，你可以操作到以下變數:</p><ul><li>results</li><li>order (當前的遞迴 element)</li><li>orders，以及透過 index 操作所有在 orders 內的 element</li><li>my_package_numbers、以及所有在 for 迴圈之前宣告的變數</li></ul><p>再來，讓我們看看用 filter 的程式碼：</p><pre>def is_order_verified_to_be_deliver(order):<br>    if order.dispatch_date &gt; date.today():<br>        return False<br><br>    buyer = order.buyer<br>    if not buyer.account.verified:<br>        return False<br>    # ...code...<br>    return True<br><br><br># ...code...<br>my_package_numbers = 200<br><br>results = list(filter(<br>    is_order_verified_to_be_deliver,<br>    orders,<br>))</pre><p>在抽取出來的 “is_order_verified_to_be_deliver” function 中，你可以操作到的變數為</p><ul><li>order</li></ul><p>嗯，對，<strong>只有 order</strong>。</p><p>相比於 for 迴圈，使用 list function 能夠縮小可操作的變數範圍，讓你在開發時，<strong>專注在「操作 element」的相關邏輯</strong>，而不用顧及其他的變數的管理；<br>而在閱讀程式時，也無需額外觀察 element 是否有與其他重要變數有交互關係，也是一個瀏覽程式碼時減少認知負荷的關鍵。</p><blockquote><strong>🔋 知識充電站 🔋</strong></blockquote><blockquote>在 function 中可以取得的變數，取決於定義 function 的變數範圍 (<a href="https://www.geeksforgeeks.org/python-scope-of-variables/">variable scope</a>)，如上面 filter的例子裡， is_order_verified_to_be_deliver 確實只能使用到傳入的變數 order，但如果我們把 function 的宣告順序往下移到 my_package_numbers 後，它就可以調用到這個變數囉！</blockquote><blockquote>根據你的需求情境，來決定應該在什麼位置定義你的 list function。</blockquote><pre># ...code...<br>my_package_numbers = 200<br><br>def is_order_verified_to_be_deliver(order):<br>    # 你可以在調用 my_package_numbers 這個變數<br>    if my_package_numbers &lt; 200:<br>        return False<br><br>    if order.dispatch_date &gt; date.today():<br>        return False<br><br>    buyer = order.buyer<br>    if not buyer.account.verified:<br>        return False<br>    # ...code...<br>    return True<br><br>results = list(filter(<br>    is_order_verified_to_be_deliver,<br>    orders,<br>))</pre><blockquote><strong>💡 3. 建立易於檢視的程式流程，逐步取得目標</strong></blockquote><p>我們來看下面一個例子：</p><blockquote>假設有一個 integer list: nums，求從這些數字中，取得 “<strong>偶數的平方和</strong>”。</blockquote><p>用 for 迴圈的話，可以透過以下程式碼完成：</p><pre>nums = [1, 5, 2, 3, 7, 6, 19, 22]<br><br>results = 0<br><br>for num in nums:<br>    if nums % 2 == 0:<br>        num_square = num ** 2<br>        results += num_square<br><br>print(results)</pre><p>這項簡單的任務，雖然只用了不到 5 行的 for 迴圈就完成了，但你會發現，這短短幾行的 for loop code block 裡，就牽涉到三個邏輯判斷：</p><ul><li>判斷是否為偶數</li><li>建立平方數</li><li>加總平方和</li></ul><p>可以想像的是，如果我們要寫的功能，複雜度遠超過這三個操作，那實際上在撰寫數百行的 for loop 程式碼時，光是要小心翼翼的調整目標程式碼、避免動到其他程式邏輯、用 print 逐行確認對應調整是否正確，就夠讓人心煩的了……</p><p>然而，如果我們今天改用 list function 來完成這項任務的話：</p><pre>nums = [1, 5, 2, 3, 7, 6, 19, 22]<br><br>even_nums = list(<br>    filter(<br>        lambda num: num % 2 == 0,<br>        nums,<br>    )<br>)<br><br>even_squares = list(<br>    map(<br>        lambda num: num ** 2,<br>        even_squares,<br>    )<br>)<br><br>square_sum = reduce(<br>    lambda carry, num: carry + num,<br>    num_squares,<br>)<br><br>print(square_sum)</pre><p>用 list function 改寫程式碼後，會發現我們是以「階段性」的概念逐步取得最終的結果：</p><ol><li>用 <strong>filter</strong> 取得偶數數列 even_nums</li><li>用 <strong>map</strong> 取得偶數平方數列 even_squares</li><li>用 <strong>reduce</strong> 取得最終的 list 的和 square_sum</li></ol><p>由於 <strong>filter</strong>, <strong>map</strong> function 都是<strong>建立新的物件來儲存最後的結果</strong>，不會改動既有的 list 元素，我們甚至可以在最後同時用 print 檢視 even_nums, even_squares, square_sum 三個參數，來檢查每個步驟後的結果，都如我們預期的進行。</p><p>利用 list function ，將原本的 for 迴圈操作拆解成一個一個的小步驟，而且每個小步驟的操作都不影響原本的變數，如此一來我們可以更有效率的檢視我們程式碼的正確性。</p><p>如果需要對其中某個操作進行調整，如：從「偶數平方和」改成「奇數平方和」，可以直接針對 even_nums 的部分進行調整，不用擔心會異動到其他程式碼流程。</p><blockquote><strong>🔋 知識充電站 🔋</strong></blockquote><blockquote>list function 這種「每一次操作都是建立一個新物件」的概念，源自於 <strong>functional programming</strong> 的開發思維，有利於<strong>提升整體程式碼的可讀性與維護性</strong>。</blockquote><blockquote>如果想要進一步理解 <strong>functional programming</strong>，推薦大家參考 「<a href="https://youtu.be/qpOcRG3e9Q8?si=5CI1lv2PkVB84su8">走歪的工程師James：初探 Functional Programming</a>」 這部影片歐！</blockquote><p>在這篇文章中，我們學到了三個好用的 list function: <strong>filter</strong>, <strong>map</strong> 和 <strong>reduce</strong>，以及如何透過 list function 來大大提升程式碼的品質 💪💪💪</p><p>如果你對這個主題有任合問題、或是其他想學習的 Python 知識，都歡迎你在留言區跟我們分享歐！</p><blockquote><em>我們是 ccClub 團隊，致力於讓 Python 成為大家的第二外語，希望能用淺顯易懂、循序漸進的方式，帶領新手一步步跨入程式設計的世界。</em></blockquote><blockquote><em>如果你喜歡這篇文章，請給我們 1~10 個掌聲👏。<br>如果你喜歡「</em><strong><em>Py立得</em></strong><em>」的教學系列文，請給我們 31 個以上的掌聲👏。</em></blockquote><blockquote><em>Facebook: </em><a href="https://www.facebook.com/ccclub.io"><em>ccClub Python讀書會</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a37f942a486f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ccclub/py-instant-tips-filter-map-reduce-a37f942a486f">Py立得 — 高手都該會的串列函示: filter, map, reduce</a> was originally published in <a href="https://medium.com/ccclub">ccClub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[給新手的套件升版指南]]></title>
            <link>https://medium.com/@40243105s/%E7%B5%A6%E6%96%B0%E6%89%8B%E7%9A%84%E5%A5%97%E4%BB%B6%E5%8D%87%E7%89%88%E6%8C%87%E5%8D%97-0141a8c8e947?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/0141a8c8e947</guid>
            <category><![CDATA[code]]></category>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[new-code]]></category>
            <category><![CDATA[package-management]]></category>
            <category><![CDATA[package-update]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Tue, 22 Oct 2024 10:08:53 GMT</pubDate>
            <atom:updated>2024-10-22T10:11:42.956Z</atom:updated>
            <content:encoded><![CDATA[<p>當我們還是程式新手練習設計小專案時，下載模組 (Package/Module, 以下都以 Package 代稱) 都是使用當前最新版本，新手小專案也通常只有幾個月的生命週期，完全無需考慮產品迭代；</p><p>然而在真實的產品情境中，往往需要定期更新套件版本以維護產品的穩定性，但是嘛升版這件事情好像多多少少變成有點「經驗談」了，沒什麼教學或是有系統的教學可以參考，搞得我第一次升版時戰戰兢兢，生怕一不小心就要把專案給炸了😅</p><p>雖然主流框架在官方文件中都有一份 Upgrade Guide，但都是一些老生常談的建議，沒啥具體細節。<br>對於第一次嘗試升版的菜雞來說，幫助不能說很多，只能說幾乎沒有 🥲。</p><p>這次趁著把 ccClub 後端 API 升版的過程，詳細的把各種升版所需要的眉眉角角細節與過程整理出來，給後進一些參考。</p><p>需要聲明的是，雖然示範裡升版對象是 Django + 各種 Python Package，但我寫的 Guideline 屬於軟體升版的大原則，不限於 pip 管理的 Package Upgrade，即使你使用的是不同的語言/框架/套件管理系統，這些概念大致都能套用其中。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*V1Vqq5NJ7jdj5Qnz" /></figure><h3>升版前需要知道的概念</h3><h4><strong>Package Manager</strong></h4><p>當代的 語言/環境 基本上都有對應的 <strong>Package Manager (套件管理系統) </strong>來張羅各個套件的 Dependencies，如 Python 的 pip/poetry，Js 的 NPM，PHP 的 Composer 等等。</p><p>在開始調整版號前，可以先確認 Package Manager 的相關指令，至少要知道「如何下載特定版號的 Package」，在 pip 的話就是 <br>pip install --upgrade &lt;PACKAGE&gt;==x.x.x <br>來升級某套件至指定版好。</p><p>另外比較常見的情境是，<strong>如果 Package 來源 (Repository) 不是官方 Publication</strong> (e.g. 沒有在 PyPI 註冊的 Package)，那要記得設定 Package Manager 的 Sourcing URL，也就是它搜尋 available packages 的地方。<br>(各個 PM 的名稱不一樣，可以打相關的關鍵字來查)</p><p>以 pip 為例，輸入 pip install --help ，會看到幾個與 index-url 相關的選項：</p><pre>Package Index Options:<br>  -i, --index-url &lt;url&gt;       Base URL of the Python Package Index (default https://pypi.org/simple). This should point to a repository compliant with PEP 503 (the simple<br>                              repository API) or a local directory laid out in the same format.<br>  --extra-index-url &lt;url&gt;     Extra URLs of package indexes to use in addition to --index-url. Should follow the same rules as --index-url.</pre><p>從說明中可以看到，pip 預設會從 https://pypi.org/simple 中搜尋你輸入的 Package 名稱，再把對應的 Package 下載下來。<br>前往 <a href="https://pip.pypa.io/en/stable/cli/pip_install/#finding-packages">pip install 的 Document</a> 有更詳細的 Configuration 設定說明，我這邊就不詳細展開了，重點是<strong>要知道自己安裝的 Package 來自哪一個 Repo</strong>。</p><h4>Version Number</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ImaSasR8l6FX9oE_.png" /></figure><p>版號 (Version number) 的概念網路上已經很多相關教學介紹了，我這邊就提大原則：</p><ul><li><strong>小版號</strong>：基本上沒有什麼 concern，大多數時候可安心直升。</li><li><strong>中版號</strong>：推出新 feature，原則上只要大版號一致，則仍與當前 Package 相容 (Compatible)。</li><li><strong>大版號</strong>：Breaking Change，通常套件作者會在提升大版號時順便 Upgrade 環境語言，條件允許的話你也可以考慮在升大版號時順便把環境升一升。</li></ul><h4>Release Notes / Changelogs</h4><p>在查看新版本更新內容時，當然不可能到 Package 的 Repo 裡一個一個 commit 檢查做了哪些異動。這時候你可以找一下 Package 有沒有 <strong>Release Notes</strong> 供使用者查閱。</p><p><strong>Release Notes (</strong>或叫<strong> Changelogs)</strong> 是作者給各版號調整的內容綱要，查閱時可以留意以下幾個關鍵字，代表 Package 開始/停止 支援的 環境 or Dependencies 版本。</p><ul><li><strong>Support</strong></li><li><strong>Drop / no longer support</strong></li><li><strong>Deprecated</strong></li></ul><p>比如說在 <a href="https://docs.djangoproject.com/en/5.1/releases/5.0/">Django 5.0.0 的 Release Nots</a> 中，就說明 Django 在這個版本支援了 3 種 Python 版號，而 3.8 與 3.9 兩個比較舊的版本則只能應用於 Django 4 以前的版本。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUIIN1N2scui8VQxzIuCkQ.png" /></figure><p>注意 Deprecated 則是要檢視 <strong>Package 的接口有無異動或棄用</strong>，比如 <a href="https://www.django-rest-framework.org/community/release-notes/#390">Django rest_framework 的 3.9.0 Release Notes</a>，就標示有幾個 function 的參數被調整命名，需要使用者調整。<br>後面的 # 數字是對應的 PR 紀錄，供使用者檢視該調整的詳細內容。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*ejiHpd4x2xB-gwyHQNZl4g.png" /></figure><h3><strong>升版前置作業：盤點專案細節</strong></h3><p>看到 Package 有新版本時千萬別直接無腦輸入 pip install --upgrade ，在正式開始升版前，務必完成以下步驟，以降低升版後 app crash 的風險。</p><h4>補齊 Unit Test</h4><p>Testing 是檢查你寫的程式是否能正確執行預期功能的最好手段，因此在開發階段時，就應盡量為每一個 feature 寫好預期的 Test。</p><p>Testing 也是一個大話題，這邊簡單提醒各位新手，如果還沒為自己的專案寫過測試的話，趁這個機會好好補齊 Testing 的知識與經驗，在升版前至少把 App 最核心的功能補上測試。</p><p>在接手 cc 後端專案前，我也耗費近一年半的時間把 API Test 補齊 🫠，不過為了產品的可靠性，這些努力絕對是值得的～</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOMRJ9mSbT3pPzB6LuF-qw.png" /><figcaption>Pytest 單元測試成功畫面 🎉🎉🎉</figcaption></figure><h4>盤點 Packages &amp; Environment</h4><p>一個專案的 Package 組成，（以我自己的標準）大致可分為三類，以下同樣以 ccClub 的 Django App 作例子：</p><ul><li>核心套件：當然是 Django</li><li>衛星套件：圍繞著 Django 套件所開發的輔助 Packages，如<br>django-rest-framework<br>drf-yasg<br>pytest-django<br>django-storages<br>django-debug-toolbar<br>等等，這些小套件的升版都依附於 Django 本身的版本迭代進度。</li><li>獨立套件，如 jsonschema，這個套件是拿來在 Testing 檢查 API JSON 結構的，本身與 Django 沒有任何干係。</li></ul><p>除此之外，也請快速確認一下語言環境設定，以我們的 app 為例，我們使用 Docker 的 python:3.7-alpine 作爲我們的 base image，在升大版號時，也要根據 Release Notes 說明調整對應的 Python 版本。</p><p>以下是 ccclub-api 所使用的環境與套件概覽：</p><pre># 語言<br>Python: 3.7<br><br># 核心套件<br>Django==3.0.6<br><br># 衛星套件<br>djangorestframework==3.11.0<br>django-polymorphic==2.1.2<br>django-storages==1.11.1<br>pytest-django==3.9.0<br>django-debug-toolbar==3.0<br><br># 獨立套件<br>jsonschema==4.17.3<br>boto3==1.18.49</pre><h3>升版步驟建議</h3><p>每個開發者都有各自的開發流程考量，在這裡我只提供一個我個人偏好的升版步驟建議供你參考，至於實際上要如何進行由各位自行決定🤗。</p><p>基本的升版流程，就是</p><p>下指令升版<br>-&gt; 跑測試，看結果<br>-&gt; 如果測試有 error，查看 log 看是哪一個功能有問題，作對應的調整，再跑測試，直到 pass 為止<br>-&gt; 然後繼續升下一版 (lopping)…</p><p>升版的起點從你的核心套件開始，以下就以我們的 Django 專案來做例子。</p><h4>從小版號開始升</h4><p>如前所述，小版號通常只是一些 bugfix 的修改，原則上不影響整個 App 的結構，因此不太需要一版一版的升，大概可以抓一次跳 3 ~6 個以上進行升版。</p><p>升小版號有一個小技巧，就是可以看一下 Release 的日期，如果有兩個相鄰不到一個禮拜的，可以升比較新的那個版本，舊的那個可以略過。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8LCP_7W2Lhl9L4WidMhjiA.png" /><figcaption>這兩個小版號天數只差不到兩天，可以直接考慮升到 3.13.1</figcaption></figure><p>在我的 case 裡，我把 Django 從 3.0.6 升到 3.0.10 -&gt; 3.0.14 (3.0.x 的最後一個版本)，跑完 test 確定沒有問題後，可以在這裡打一個 commit，甚至 Release 一個版本，發佈一個對應 Django 3.0.x 的最終穩定版。</p><h4>升中版號</h4><p>中版號引入的新功能有時會影響其他衛星 Package 的功能，因此升版後一樣先跑測試檢查。</p><p>我在升版 Django 3.0.14 -&gt; Django 3.1.1 後，執行測試時就有發生 error:</p><pre>  File &quot;/usr/local/lib/python3.7/site-packages/polymorphic/models.py&quot;, line 17, in &lt;module&gt;<br>    from .base import PolymorphicModelBase<br>  File &quot;/usr/local/lib/python3.7/site-packages/polymorphic/base.py&quot;, line 18, in &lt;module&gt;<br>    from .managers import PolymorphicManager<br>  File &quot;/usr/local/lib/python3.7/site-packages/polymorphic/managers.py&quot;, line 10, in &lt;module&gt;<br>    from polymorphic.query import PolymorphicQuerySet<br>  File &quot;/usr/local/lib/python3.7/site-packages/polymorphic/query.py&quot;, line 11, in &lt;module&gt;<br>    from django.db.models import FieldDoesNotExist<br>ImportError: cannot import name &#39;FieldDoesNotExist&#39; from &#39;django.db.models&#39; (/usr/local/lib/python3.7/site-packages/django/db/models/__init__.py)<br>make: *** [Makefile:40: test] Error 1</pre><p>這裡看到 polymorphic 這個 Package 在 <br>from django.db.models import FieldDoesNotExis<br>時發生問題。</p><p>這時可以到 django-polymorphic 的 Release Notes，找到一個支援 <a href="https://django-polymorphic.readthedocs.io/en/stable/changelog.html#changes-in-3-0-0-2020-08-21">Django 3.X 版本的升版</a>，因此我就把這個套件從 2.1.2 升到 3.0.0 :</p><pre>pip install --upgrade django-polymorphic==3.0.0</pre><p>跑完測試後，雖然 pass 了，但又看到一個 Warning:</p><pre>api_project/tests/test_swagger.py::test_swagger<br>  /usr/local/lib/python3.7/site-packages/rest_framework/authentication.py:135: RemovedInDjango40Warning: Passing None for the middleware get_response argument is deprecated.<br>    check = CSRFCheck()<br><br>-- Docs: https://docs.pytest.org/en/latest/warnings.html</pre><p>Warning message 顯示 rest_framework 有一個即將在 Django 4 被棄用的部分。<br>不過因為現在距離 Django 4 還有一段距離，因此可以在升到 3 的最終版以後再來處理這件事，不用太緊張🤪。</p><p>在升中版號時，除了 Testing 抓出的 error、warning，你也可以加減海巡一下其他衛星套件有沒有對應的升版可以處理。<br>我個人的底線只有 Testing 至少要過，其他 Package 等真的出現 error 跟著調整版本。</p><p>把中版號的小版號一樣升到底以後，一樣打個 commit、Release 一個 final 穩定版，就可以邁入大版號了。</p><h4>升大版號</h4><p>升大版號前，<strong>請先仔細閱讀 Release notes 對環境的要求</strong>，看目前的環境是否滿足 Package 要求。</p><p>以 Django 為例， <a href="https://docs.djangoproject.com/en/5.1/releases/4.0/">Django 4.0</a> 不支援 Python 3.8 以下的版本，所以在這個情境裡我就要準備調整 Docker image 的 Python 版本了。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63xfcccVb5IRfOgjwNX7eA.png" /></figure><p>升版後一樣跑測試、排除套件衝突，不過這時我會建議巡一下各個衛星套件有沒有對應大版號的調整。<br>以升到 Django 4 為例，你可以在 Release Nots 頁面上搜尋「Django 4」，或是根據 Django 4.0.0 Release 的日期往後爬爬看，如果有對應可以調整的版本可以一起往上升。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*XWY__V9xZJ2Q2GaBOLy0vw.png" /><figcaption>Django 4.0 在 2021.12.07 發佈後，rest_framework 緊接著在 2021.12.13 釋出可以兼容的版本</figcaption></figure><p>剛剛提到在大版號調整前可以先升級環境，此時也是個檢查獨立套件是否需要跟著升版的好時機，基本流程跟前ㄧ致就不再多贅述。</p><h4>升大版號別搶頭香</h4><p>如果你已經把核心套件升到當前大、中版號的最高時，看到核心套件在近期剛釋出了最新的大版號時，其實可以不用急著馬上升上去，可以等他的一、二個新的中版號都發布後再考慮升版。</p><p>原因相信你也多少可以想到，當核心套件大版號剛釋出時，一來可能還有 Bug，二來他周遭的衛星套件可能還沒有開發完相容的版本，此時貿然升版可能會遇到不少問題，不妨等幾個相關的 Packages 陸陸續續穩定以後再慢慢升上去。</p><h4>升版過程中的除錯建議</h4><p>在 Upgrade 的過程中，出現錯誤在所難免。此時如果 Console 上的 Error Message 仍不足以幫助你解決錯誤，除了在 Google / Stackoverflow 上查找關鍵字外，另一個尋找問題的來源是各 Package 的 <strong>Github Issues</strong> 頁面，無論是 Open 還是 Closed Issues，都可以查一下有沒有相關的討論串可以參考，追蹤當前各種 fix 的進度。</p><p>如果真的運氣不好，遇到 Package 作者停止維運、但又發生衝突的狀況，就只能<br>(1) 找找看有沒有其他功能類似、且還有持續維護的 Packages<br>(2) 考慮把 Package fork 到自己的 Repo 裡，再自己看要怎麼改寫 Package 才能解決 Conflict 了🥲</p><h3>升版後的 Follow-up</h3><p>完成升版後，對於上線的產品，除了密切觀察是否有錯誤訊息外，另一個很重要的回家作業是<strong>研究 Release Notes 中新發佈的功能</strong>，看看是否有新的 feature 可以應用在專案上。</p><p>「能夠正確執行」是每個專案的基本要求。身為持續成長的開發者，當然要力求自己維護的專案也要持續迭代，讓這份專案的 pattern/coding style 能夠向 Package 的方向靠攏，在日後的開發與維運中，即使遇到問題，也比較找得到對應的 Solution 來解決。</p><h3>結論</h3><p>在閱讀本篇後，請記得三件個專案升版過程中不可或缺的關鍵：</p><ul><li>閱讀套件的 <strong>Upgrade Guide</strong> 和 <strong>Release Notes</strong></li><li>覆蓋核心功能的 <strong>Unit Test</strong></li><li>Patience, Patience, <strong>Patience</strong>！</li></ul><p>最後祝大家的能順利升版，有任何建議或疑問歡迎在留言區指教 😇😇😇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0141a8c8e947" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Py立得 — mutable 與 immutable 物件]]></title>
            <link>https://medium.com/ccclub/py-instant-tips-mutable-immutable-e128f641eb6e?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e128f641eb6e</guid>
            <category><![CDATA[py-立得]]></category>
            <category><![CDATA[immutable-objects]]></category>
            <category><![CDATA[mutable-objects]]></category>
            <category><![CDATA[python進階]]></category>
            <category><![CDATA[ccclub]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 02:13:35 GMT</pubDate>
            <atom:updated>2024-09-16T02:28:24.846Z</atom:updated>
            <content:encoded><![CDATA[<h3>Py立得 — mutable 與 immutable 物件</h3><h4>理解可變與不可變物件，寫出更 Pythonic 的程式碼</h4><p>從這篇文章中，你將會學到：</p><ul><li>immutable object 與 mutable object 的機制與差異</li><li>用 + 串接 string 和用 join 串接 string 在效率上的差異</li><li>設計 function 的傳入參數時要注意是否為 mutable object</li><li>設計 function 的參數的預設值時要注意是否為 mutable object</li></ul><p>（感謝夥伴 <a href="https://medium.com/@yuehchloe">Chloe Yueh</a> 的美編支援）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*stYxE_q-ICK8Bz6zEGCMRQ.png" /></figure><p>在上一篇文章中談到了 Python 的所有資料都是由 object 構成，以及 variable 透過 address 找到儲存在 memory 的資料。<br>這次就延續這些基礎知識，來認識一個 Python 語言中很重要的概念：<strong>Mutable object</strong> 與 I<strong>mmutable object</strong>。</p><p>(沒聽過 object 跟address？快到 <a href="https://medium.com/ccclub/py-instant-tips-python-is-64e82970719b"><strong>這裡</strong></a><strong> </strong>去複習一下吧！)</p><h3>Immutable Object</h3><p><a href="https://docs.python.org/3/glossary.html#term-immutable">Immutable</a> Object 指的是那些一旦在 memory 中建立後，其 <strong>value 就無法改變</strong>的資料類別，比如下列資料型別：</p><ul><li>int</li><li>float</li><li>string</li><li>tuple</li><li>frozenset</li></ul><p>我們以最熟悉的 int 為例，當我們輸入 num = 1000 時，這個 num 變數所記錄的 int object value 就 1000，即使我們對 num 進行 加 / 減 / 乘 / 除 / 取餘數 任何操作，Python interpreter <strong>不是對既有的 int object value 進行變更，而是建立一個新的 int object</strong>，並把計算後的結果儲存到新的 object 中。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*_OLobIWGvxzQsQEzvdrzrg.png" /></figure><p>讓我們來實際跑一下程式，用 id()確認變數的 memory address，證明 int 是 immutable object，以及計算後的結果，<strong>都是用新的物件來儲存</strong>：</p><pre>&gt;&gt; num = 1000<br><br>&gt;&gt; print(id(num))<br>134623788426288<br><br># 對 num 進行加法，會產生一個新物件<br>&gt;&gt; print(num + 1000)<br>2000<br>&gt;&gt; print(id(num + 1000))<br>134623788426192<br># 原本的 num 變數的 value, address 不變<br>&gt;&gt; print(num)<br>1000<br>&gt;&gt; print(id(num))<br>134623788426288<br><br># 再試試看對 num 進行乘法<br>&gt;&gt; print(num * 3)<br>3000<br>&gt;&gt; print(id(num * 3))<br>134623788426256<br>&gt;&gt; print(id(num))<br>134623788426288</pre><p>如果我們要儲存 immutable object 計算後的結果，就必須將結果 assign (賦值)給新的變數：</p><p>new_num = num + 1000</p><p>或是把結果 reassign 給原本的變數名稱：</p><p>num = num + 1000(或是 num += 1000)</p><p>你才能利用計算後的 int 數值。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*ELoKPZGjpze3PkiZQmU77g.png" /></figure><p>我們再來看看對另一種 immutable 型別: <strong>string</strong> 進行操作的例子🌰：</p><pre>&gt;&gt; name = &quot;Alex&quot;<br><br>&gt;&gt; print(id(name))<br>136059618945264<br><br># 用 + 串接 string<br>&gt;&gt; name_shout = name + &quot;!!!&quot;<br><br># name_shout 是一個新物件<br>&gt;&gt; print(name_shout)<br>Alex!!!<br>&gt;&gt; print(id(name_shout))<br>136059618945328<br><br># 原有的 name 物件 value, address 不變<br>&gt;&gt; print(name)<br>Alex<br>&gt;&gt; print(id(name))<br>136059618945264<br><br># 用 upper() 將string 改為大寫<br>upper_name = name.upper()<br><br># upper_name 是一個新物件<br>&gt;&gt; print(upper_name)<br>ALEX<br>&gt;&gt; print(id(upper_name))<br>136059618945328<br><br># 原有的 name 物件 value, address 不變<br>&gt;&gt; print(name)<br>Alex<br>&gt;&gt; print(id(name))<br>136059618945264</pre><p>從上面的運行結果可以發現，無論是用 + 串接 string，或是執行 upper function 得到大寫的 string，都不影響原有的 name object！</p><p>如果嘗試對 immutable object 的 value 進行變動的話，則會跳出 Error:</p><pre>&gt;&gt; club    = &quot;ccClub&quot;<br>&gt;&gt; club[3] = &quot;c&quot;<br><br># Traceback (most recent call last):<br>#   File &quot;/Users/leadbest/repo/pyTest/test_2.py&quot;, line 3, in &lt;module&gt;<br>#   club[3] = &quot;c&quot;<br>#    ~~~~^^^<br># TypeError: &#39;str&#39; object does not support item assignment</pre><p>小結一下，關於 I<strong>mmutable Object</strong>：</p><ul><li>object 在建立後，value <strong>無法</strong>被改變。</li><li>透過操作子/函式對 immutable object 進行任何計算操作，interpreter 都會建立一個新的 object 來儲存計算後的結果，不影響原有的 object。</li><li>如果想要儲存計算後的結果，必須透過 assign 的方式，將計算後產生的新 object 的 address 綁定給變數。</li></ul><h3>Mutable Object</h3><p>顧名思義，mutable object 就是那些在建立後，value 可以被改變的資料型別，如 list、dict、set，以及我們自定義的 class 等。</p><p>我們來以 list 為例🌰，透過 append、extend、pop、remove 進行操作時，改變的對象都是原有的 list object，裡面的 element 組成會根據調用的函數而改變：</p><pre>&gt;&gt; course = [&quot;Alex&quot;, &quot;Mark&quot;, &quot;Ethan&quot;]<br><br>&gt;&gt; print(course)<br>[&#39;Alex&#39;, &#39;Mark&#39;, &#39;Ethan&#39;]<br><br>&gt;&gt; print(id(list))<br>132232573294592<br><br># 透過 function 增加 list element<br>&gt;&gt; course.append(&quot;Judy&quot;)<br>&gt;&gt; course.extend([&quot;May&quot;, &quot;Steven&quot;])<br><br># 原有的 course object 的 value 改變<br>&gt;&gt; print(course)<br>[&#39;Alex&#39;, &#39;Mark&#39;, &#39;Ethan&#39;, &#39;Judy&#39;, &#39;May&#39;, &#39;Steven&#39;]<br>&gt;&gt; print(id(course))<br>132232573294592<br><br># 透過 function 移除 list element<br>&gt;&gt; course.pop()<br>&gt;&gt; course.remove(&quot;Alex&quot;)<br><br># 原有的 course object 的 value 改變<br>&gt;&gt; print(course)<br>[&#39;Mark&#39;, &#39;Ethan&#39;, &#39;Judy&#39;, &#39;May&#39;<br>&gt;&gt; print(id(course))<br>132232573294592</pre><p>從結果可以發現，不管是透過 append, extend 來新增 list 內的 element，或是用 pop, remove 移除 element，都沒有改變 course list 的 address，也就是說我們一直是<strong>對同一個 list object 進行操作</strong>。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*iHU_35kv4B8vqyAberu0NA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*hx7Tk6yuPfLKp3KyrYGFPg.png" /></figure><blockquote><strong>🔋 知識充電站 🔋</strong></blockquote><blockquote>在 <a href="https://docs.python.org/3/tutorial/datastructures.html">Python 官方文件</a>中，如果你在函數說明中看到 <strong>in place</strong> (原地) 這個關鍵字，指的就是對 mutable object 本身進行操作，比如說 list.sort() 和 list.reverse()，都是直接對原有的 list object 的 value 進行排序處理，而不是建立一個新的 list object 再排序。</blockquote><p>要注意的是，雖然 list 是 mutable object，但不代表對 list 的每一種操作都是對原有的 object 操作，<strong>有些函式或操作子是透過建立新的 list、對新的 list 進行操作</strong>，回傳給變數的結果也是新的 list 的 address，比如以下幾個例子🌰：</p><pre>&gt;&gt; fruits_a = [&quot;pipeapple&quot;, &quot;apple&quot;]<br>&gt;&gt; fruits_b = [&quot;tomato&quot;, &quot;papaya&quot;]<br><br><br># 實驗一<br># 用 + 連接兩個 list<br>&gt;&gt; fruits_all = fruits_a + fruits_b<br><br># fruits_all 是一個新建立的物件<br>&gt;&gt; print(fruits_all)<br>[&#39;pipeapple&#39;, &#39;apple&#39;, &#39;tomato&#39;, &#39;papaya&#39;]<br>&gt;&gt; print(id(fruits_all))<br>135691873054848<br><br># 原本的 fruits_a 和 fruits_b 物件沒有影響<br>&gt;&gt; print(fruits_a)<br>[&#39;pipeapple&#39;, &#39;apple&#39;]<br>&gt;&gt; print(id(fruits_a))<br>135691871142400<br><br>&gt;&gt; print(fruits_b)<br>[&#39;tomato&#39;, &#39;papaya&#39;]<br>&gt;&gt; print(id(fruits_b))<br>135691873089216<br><br># 實驗二<br># 用 sorted() 對 list 進行排序<br>&gt;&gt; sorted_fruits_a = sorted(fruits_a)<br><br># sorted 會複製原本的 list，然後對新的 list 排序<br>&gt;&gt; print(sorted_fruits_a)<br>[&#39;apple&#39;, &#39;pipeapple&#39;]<br>&gt;&gt; print(id(sorted_fruits_a))<br>135691873109120<br><br># 原本的 fruits_a 物件沒有影響<br>&gt;&gt; print(fruits_a)<br>[&#39;pipeapple&#39;, &#39;apple&#39;]<br>&gt;&gt; print(id(fruits_a))<br>135691871142400<br><br># 實驗三<br># list slicing 操作也是建立新物件來儲存分割後的結果<br>&gt;&gt; sliced_fruits_a = fruits_a[1:]<br>&gt;&gt; print(sliced_fruits_a)<br>[&#39;apple&#39;]<br>&gt;&gt; print(id(sliced_fruits_a))<br>135691873111808<br><br># 原本的 fruits_a 物件沒有影響<br>&gt;&gt; print(fruits_a)<br>[&#39;pipeapple&#39;, &#39;apple&#39;]<br>&gt;&gt; print(id(fruits_a))<br>135691871142400</pre><p>因此，在對 mutable object 進行操作時，要注意呼叫的操作方式是針對 mutable 物件本身進行異動、還是建立新的物件來儲存異動結果，根據自己程式的需求選擇適合的函式歐！</p><p>小結一下，關於 mutable object：</p><ul><li>object 在建立後，value <strong>可以</strong>被改變。</li><li>以 list 為例，如果呼叫 <br>- list.append()<br>- list.extend()<br>- list.sort()<br>- list.reverse() <br>這些函式的話，是對原有的 list object 的 value 進行更動。</li><li>不是所有對 mutable object 的操作都是 in place 的，比如說<br>- sorted(list)<br>- list.copy()<br>- 用 + 連接 list<br>- 對 list 做 slicing<br>都會建立一個新的 list 物件，並回傳新物件的 address 給變數，原有的 list 物件的 value 則維持不變。</li></ul><p>在認識了 mutable 與 immutable object 的機制後，就能解釋一些過去對你來說「知其然，不知其所以然」的 Python 規則，比如：</p><h3>Concatenate String V.S. Join String</h3><p>若想將 N 個字串組合成一個大字串，我們可以</p><ol><li>str_a + str_b + … 將一個一個字串相加</li><li>建立一個 list: str_list = [str_a, str_b, str_c …]，再用 &quot;&quot;.join(str_list) 得到組合後的結果</li></ol><p>我們來簡單用 <strong>timeit</strong> 模組測試一下，這兩個不同的字串結合機制，在效率上的差異：</p><pre>from timeit import Timer<br><br># 用 + 連接 string<br>def generate_by_concat(times):<br>    def _inner():<br>        target = &quot;&quot;<br>        <br>        for i in range(times):<br>            target += &quot;Hello!&quot;<br>    return _inner<br><br># 用 join 把 list 中的 string 串接在一起<br>def generate_by_join(times):<br>    def _inner():<br>        hello_list = list()<br>        <br>        for i in range(times):<br>            hello_list.append(&quot;Hello!&quot;)<br>        <br>        target = &quot;&quot;.join(hello_list)<br>    return _inner<br><br># 連接 10 個字串<br>print(Timer(generate_by_concat(10)).timeit(1))<br># 1.8749997252598405e-06<br>print(Timer(generate_by_join(10)).timeit(1))<br># 2.0410006982274354e-06<br><br># 連接 10,000 個字串<br>print(Timer(generate_by_concat(10000)).timeit(1))<br># 0.0005040830001235008<br>print(Timer(generate_by_join(10000)).timeit(1))<br># 0.0002974579983856529<br><br># 連接 10,000,000 個字串<br>print(Timer(generate_by_concat(10000000)).timeit(1))<br># 0.524733666999964<br>print(Timer(generate_by_join(10000000)).timeit(1))<br># 0.2807453749992419</pre><p>從結果可以發現，當組合 10 個 “Hello!” 字串時，用 + 來建立會稍快一點；<br>但隨著組合字串的次數增加，反而是用 join 的方式會比較快，在次數達到 一千萬次時，用 + 連接 string 所花費的時間將近是 join 的兩倍了！</p><p>兩個方法在速度上的差異，關鍵就在計算對象是 mutable 還是 immutable object:</p><ul><li>在 generate_by_concat 函式中，由於 string 是 immutable object，每多接一次字串: target += “Hello!”，就等於要新建立一個 string object 來儲存。<br>在這個函式中，如要連接 N 個 “Hello!” 字串，就會有建立 N 個 string object 的過程。</li></ul><pre>target = &quot;Hello!&quot;<br><br>target += &quot;Hello!&quot;<br># target 建立一個新的 string 物件： &quot;Hello!Hello!&quot;<br><br>target += &quot;Hello!&quot;<br># target 建立一個新的 string 物件： &quot;Hello!Hello!Hello!&quot;</pre><ul><li>在 generate_by_join 函式中，用 append 將字串加到既有的 hello_list 中，這個過程不用建立新物件；一直等到最後 join 的階段，才會將結合後的最終字串用新建立的 string object 儲存。<br>在這個函式中，無論要連接幾個 “Hello!” 字串，都只需要建立 2 個物件：儲存 string 的 list object，以及最後 join 結果的 string object。</li></ul><pre>target = [&quot;Hello!&quot;]<br><br>target.append(&quot;Hello!&quot;)<br># target 是原本的 list 物件，value 是 [&quot;Hello!&quot;, &quot;Hello!&quot;]<br><br>target.append(&quot;Hello!&quot;)<br># target 是原本的 list 物件，value 是 [&quot;Hello!&quot;, &quot;Hello!&quot;, &quot;Hellow!&quot;]<br><br>final_string = &quot;&quot;.join(target)<br># 建立一個新的 string 物件: final_string，value 是 &quot;Hello!Hello!Hello!&quot;</pre><p>當字串組合次數 N 較少時，由於初始建立 list 物件的成本比 string 物件高，因此 generate_by_concat 的速度要稍快一點；</p><p>然而，一旦 N 的數量上升，generate_by_concat 需要不斷建立 string object 的效能負擔就開始累積，所花費的時間會超過 generate_by_join 。</p><h3>誰動了我的資料？！</h3><p>當我第一次學到 global scope 與 local scope 的概念後，以為 function 內的變數與外面的變數是「彼此獨立的存在」，因此傳入 function 的 int, string 等資料，無論進行什麼操作，都不會影響到外面的參數，近比如下面的這個例子：</p><pre>def do_something_to_the_num(num):<br>    num **= 2<br>    num += 100<br>    num *= 10<br>    num -= 1<br>    return num<br><br>num = 10<br>new_num = do_something_to_the_num(num)<br><br>print(num)<br>#10<br>print(new_num)<br># 1999</pre><p>然而，當我傳入 list 物件，並在 function 中對 list 物件進行操作時，卻驚訝的發現「我外面的 list 也發生變化了」！</p><pre>from random import randint<br><br>def append_1_to_10(num_list):<br>    for i in range(1, 11):<br>        num_list.append(i)<br>    return num_list<br>    <br>num_list = []<br>new_list = append_1_to_10(num_list)<br><br>print(num_list)<br># [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<br># !!! 我的 num_list 被偷加東西了！</pre><p>在學會 address, immutable object, mutable object 觀念後，相信你已經猜到了這個現象的原因：</p><p>當我們將變數 num_list 傳入到 function 進行操作時，傳入的對象其實不是「一個 list copy」，而是<strong>傳入了 num_list 所綁定的 address</strong>，讓 function 中的 local 變數 num_list 也綁定了這個 address。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*LOA6S706fZswlcWsTfbITA.png" /></figure><p>因此，在 append_1_to_10 中進行的 append 操作，實際上是依據 address 找到對應的 list object，把數字 1–10 加到 list 中。</p><p>至於為什麼範例一中的 num 變數沒有受到 function 的影響，則是因為 int 是 immutable object，在每一行計算過程中，都是用新產生的物件來儲存計算結果，因此 function 內的任何操作都不會影響到外面 num 物件所綁定的 int object address。</p><h3>難以捉摸的 function default value</h3><p>假設我們要設計一個 function：record_my_journey，來紀錄旅遊日誌。<br>這個函數需要有以下功能：</p><ul><li>可輸入兩個參數：旅遊地點 place string，以及記錄日誌 note dictionary</li><li>輸入參數後，record_my_journey 會以 place 作為 key、當天的日期 string 作為 value，記錄在 note 中，並在最後回傳 note 變數</li><li>使用者也可以選擇不傳入日誌，只帶入 place 參數，讓 function 自動產生一個全新的 note 回傳給使用者，作為記錄日誌的首章</li></ul><p>根據上面的需求，我們可以這樣設計我們的 record_my_journey：</p><pre>from datetime import date<br><br>def record_my_journey(place, note=dict()):<br>    note[place] = date.today().strftime(&#39;%Y-%m-%d&#39;)<br>    return note</pre><p>接下來，我們實際使用這段 function 看看:</p><pre># 第一本 note 由預設值建立<br>&gt;&gt; my_note = record_my_journey(&#39;Taiwan&#39;)<br>&gt;&gt; print(my_note)<br>{&#39;Taiwan&#39;: &#39;2024-06-26&#39;}<br><br># 用第一本 note 寫下第二個旅遊紀錄<br>&gt;&gt; record_my_journey(&#39;Australia&#39;, my_note)<br>&gt;&gt; print(my_note)<br>{&#39;Taiwan&#39;: &#39;2024-06-26&#39;, &#39;Australia&#39;: &#39;2024-06-26&#39;}<br><br># 第二本 note 也由預設值建立<br>&gt;&gt; my_note_2 = record_my_journey(&#39;India&#39;)<br>&gt;&gt; print(my_note_2)<br>{&#39;Taiwan&#39;: &#39;2024-06-26&#39;, &#39;Australia&#39;: &#39;2024-06-26&#39;, &#39;India&#39;: &#39;2024-06-26&#39;}<br># !!! 為什麼 my_note_2 會用到 my_note ???<br><br># 確認一下 兩本 note 的物件<br>&gt;&gt; print(id(my_note))<br>138350073948160<br>&gt;&gt; print(id(my_note_2))<br>138350073948160<br>&gt;&gt; print(my_note is my_note_2)<br>True<br># !!! 兩本 note 居然原來是一個物件 !!!</pre><p>上面的程式碼中，原本我們預期在沒有帶任何 note 參數的情況下，my_note_2 應該要是一個由 record_my_journey 函式預設提供的新的 dict；</p><p>而實際的情況是，由 function 預設值產生的 my_note 與 my_note_2，兩個指向的是同一個 dict 物件 address: 138350073948160，後續對 my_note 與 my_note_2 進行的任何操作，都是對這個 dictionary 進行調整。</p><p>造成這個「<strong>預設物件複用</strong>」現象的原因，在於 <strong>“function” 本身也是一個 immutable object！</strong></p><p>當 Python 編譯執行到 function 部分時，在 assign 預設值的 value 時，會在 memory 建立一個相對應的物件，並把物件的 address 指向給參數的預設值。<br>且因為 function 是 immutable object 的特性，預設值物件的 address value 無法被改變。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*VcZQOQ5WzMeCc5vM2HsDBw.png" /></figure><p>呼叫 function 時，如果對應參數沒有放入相對應的變數的話，function 就會使用在上述過程時建立的 default 物件 address，代入 function code block 中執行。</p><p>在上面的例子裡，dict 是 mutable object，而且對 dict 存入 key-value 是一個 in-place 的操作，不會生成新的 dict，所以上面 function 的寫法，無論在什麼地方呼叫，使用的預設值對象會一直是同一個 dict object。</p><p>因此，在設計帶有 mutable type 的參數時，務必要注意預設值的處理。<br>如果你想要有「<strong>若沒有提供對應的變數，則建立一個新的物件</strong>」的效果，應該要將程式改寫如下：</p><pre>def record_my_journey(place, note=None):<br>    if note is None:<br>        note = dict()<br>        # function 的 code block 與參數宣告不同，會重新執行<br>        # 因此這裡的 dict() 會確實地建立一個新的 dict object<br><br>    note[place] = date.today().strftime(&#39;%Y-%m-%d&#39;)<br><br>    return note<br><br>my_note = record_my_journey(&#39;Taiwan&#39;)<br>print(my_note)<br># {&#39;Taiwan&#39;: &#39;2024-06-26&#39;}<br><br>my_note_2 = record_my_journey(&#39;India&#39;)<br>print(my_note_2)<br># {&#39;India&#39;: &#39;2024-06-26&#39;}</pre><p>在這篇文章中，我們深入探討了immutable object 與 mutable object 的機制，以及這些特性會如何影響我們設計的 Python 程式碼。</p><p>如果你對這個主題有任何問題、或是其他想學習的 Python 知識，都歡迎你在留言區跟我們分享歐！</p><blockquote><em>我們是 ccClub 團隊，致力於讓 Python 成為大家的第二外語，希望能用淺顯易懂、循序漸進的方式，帶領新手一步步跨入程式設計的世界。</em></blockquote><blockquote><em>如果你喜歡這篇文章，請給我們 1~20 個掌聲👏。<br>如果你喜歡「</em><strong><em>Py立得</em></strong><em>」的教學系列文，請給我們 31 個以上的掌聲👏。</em></blockquote><blockquote><em>Facebook: </em><a href="https://www.facebook.com/ccclub.io"><em>ccClub Python讀書會</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e128f641eb6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ccclub/py-instant-tips-mutable-immutable-e128f641eb6e">Py立得 — mutable 與 immutable 物件</a> was originally published in <a href="https://medium.com/ccclub">ccClub</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bash 隨筆記記 — -x tracing mode]]></title>
            <link>https://medium.com/@40243105s/bash-%E9%9A%A8%E7%AD%86%E8%A8%98%E8%A8%98-x-tracing-mode-6dd584178190?source=rss-4a420509a6d3------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dd584178190</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[shell-script]]></category>
            <category><![CDATA[bash-script]]></category>
            <dc:creator><![CDATA[SleeperShark]]></dc:creator>
            <pubDate>Sun, 01 Sep 2024 05:40:56 GMT</pubDate>
            <atom:updated>2024-09-01T05:45:52.771Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bash 隨筆記記 — -x tracing mode</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28RwShBanUoMf3ZVkiB8QA.png" /></figure><p>本文為 <a href="https://linuxcommand.org/tlcl.php"><em>The Linux Command Line</em></a>: Ch30 之筆記。</p><h3>-x tracing mode</h3><p>可以印出 shell script 執行過的行數，可用於 debug 追蹤。</p><p><strong>-x</strong> option description from <a href="https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html">reference</a>:</p><blockquote><strong>-x</strong></blockquote><blockquote>Print a trace of simple commands, for commands, case commands, select commands, and arithmetic for commands and their arguments or associated word lists after they are expanded and before they are executed. <br>The value of the <strong>PS4</strong> variable is expanded and the resultant value is printed before the command and its expanded arguments.</blockquote><h4>Example</h4><pre>#!/bin/bash -x<br><br># even-or-odd<br><br>clear<br><br>read -p &quot;Enter a number &gt; &quot;<br><br>if [[ ! &quot;$REPLY&quot; =~ ^[[:digit:]]*$ ]]; then<br>    echo &quot;Invalid number input.&quot; &gt;&amp;2s<br>    exit 1<br>fi<br><br>if [ $(($REPLY % 2)) -eq 0 ]; then<br>    echo &quot;${REPLY} is an even number&quot;<br>else<br>    echo &quot;${REPLY} is an odd number&quot;<br>fi</pre><p>執行指令 ./even-or-odd.sh ，輸入 1：</p><pre>+ read -p &#39;Enter a number &gt; &#39;<br>Enter a number &gt; 1<br>+ [[ ! 1 =~ ^[[:digit:]]$ ]]<br>+ &#39;[&#39; 1 -eq 0 &#39;]&#39;<br>+ echo &#39;1 is an odd number&#39;<br>1 is an odd number</pre><p>執行指令 ./even-or-odd.sh ，輸入 abc：</p><pre>+ read -p &#39;Enter a number &gt; &#39;<br>Enter a number &gt; abc<br>+ [[ ! sss =~ ^[[:digit:]]$ ]]<br>+ echo &#39;Invalid number input.&#39;<br>+ exit 1</pre><h3>調整 PS4 以顯示執行行數</h3><p>-x 印出 tracing output 時會以 PS4 (prompt string 4) 作為開頭，因此可以調整 PS4 shell variable 來客製化印出格式。</p><p>當前的 PS4:</p><pre>echo $PS4<br>+%N:%i&gt;</pre><p>在 Shell Script 中，將 PS4 帶上 $LINENO shell varaible：</p><pre>#!/bin/bash -x<br><br># even-or-odd-checker<br><br>clear<br><br>PS4=&#39;$LINENO +&#39;<br><br>read -p &quot;Enter a number &gt; &quot;<br><br>if [[ ! &quot;$REPLY&quot; =~ ^[[:digit:]]*$ ]]; then<br>    echo &quot;Invalid number input.&quot; &gt;&amp;2s<br>    exit 1<br>fi<br><br>if [ $(($REPLY % 2)) -eq 0 ]; then<br>    echo &quot;${REPLY} is an even number&quot;<br>else<br>    echo &quot;${REPLY} is an odd number&quot;<br>fi</pre><p>執行指令 ./even-or-odd.sh ，輸入 666：</p><pre>+ PS4=&#39;$LINENO + &#39;<br>9 + read -p &#39;Enter a number &gt; &#39;<br>Enter a number &gt; 666<br>11 + [[ ! 666 =~ ^[[:digit:]]*$ ]]<br>16 + &#39;[&#39; 0 -eq 0 &#39;]&#39;<br>17 + echo &#39;666 is an even number&#39;<br>666 is an even number</pre><h3>局部 Tracing Mode</h3><p>如果不想要全部的 script 都有 tracing output，可改用 set -x / set +x 來局部的啟動 tracing mode:</p><pre>#!/bin/bash<br><br># even-or-odd-checker<br><br>clear<br><br>PS4=&#39;$LINENO + &#39;<br><br>read -p &quot;Enter a number &gt; &quot;<br><br>if [[ ! &quot;$REPLY&quot; =~ ^[[:digit:]]*$ ]]; then<br>    echo &quot;Invalid number input.&quot; &gt;&amp;2<br>    exit 1<br>fi<br><br>if [ $(($REPLY % 2)) -eq 0 ]; then<br>    set -x     # tracing mode starts<br>    echo &quot;${REPLY} is an even number&quot;<br>    set +x     # tracing mode starts<br>else<br>    echo &quot;${REPLY} is an odd number&quot;<br>fi</pre><p>執行指令 ./even-or-odd.sh ，輸入 10：</p><pre>Enter a number &gt; 10<br>18 + echo &#39;10 is an even number&#39;<br>10 is an even number<br>19 + set +x</pre><p>執行指令 ./even-or-odd.sh ，輸入 abc：</p><pre>Enter a number &gt; abc<br>Invalid number input.</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dd584178190" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>