<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tony Yang on Medium]]></title>
        <description><![CDATA[Stories by Tony Yang on Medium]]></description>
        <link>https://medium.com/@tonyyoung3?source=rss-1309abeb9842------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rFCMNSw2w-a98vcxLeFFoQ.jpeg</url>
            <title>Stories by Tony Yang on Medium</title>
            <link>https://medium.com/@tonyyoung3?source=rss-1309abeb9842------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:52:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tonyyoung3/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Improve Data Reliability with Data Contracts]]></title>
            <link>https://tonyyoung3.medium.com/improve-data-reliability-with-data-contracts-42ad32ae3702?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/42ad32ae3702</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Fri, 02 Aug 2024 04:00:05 GMT</pubDate>
            <atom:updated>2024-08-02T04:12:45.957Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s data-driven world, ensuring the reliability and consistency of data is crucial for making informed decisions. One powerful tool for achieving this is the implementation of data contracts. This article explores why data contracts are important and demonstrates how to create them using Python, with a focus on stock price data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RnGewwqWQVDrR9Re" /><figcaption>Photo by <a href="https://unsplash.com/@kommumikation?utm_source=medium&amp;utm_medium=referral">Mika Baumeister</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Why Data Contracts Are Important</h3><p>Data contracts are formal agreements that specify the structure, format, and rules for data exchanged between different parts of a system or between systems. They are essential for several reasons:</p><ol><li><strong>Data Consistency</strong>: As data flows through various stages of processing and analysis, it’s crucial to maintain consistency. Data contracts ensure that the data structure remains uniform across different components of your data pipeline.</li><li><strong>Type Safety</strong>: Data types can change unexpectedly, leading to errors in downstream processes. Data contracts enforce type checking, preventing issues caused by mismatched data types.</li><li><strong>Schema Evolution</strong>: As business needs evolve, data schemas may need to change. Data contracts provide a clear way to version and manage these changes, ensuring backward compatibility when necessary.</li><li><strong>Documentation</strong>: Data contracts serve as living documentation, clearly defining the expected data structure for all stakeholders.</li><li><strong>Error Prevention</strong>: By catching data inconsistencies early, data contracts help prevent cascading errors that can be costly and time-consuming to fix later in the data pipeline.</li></ol><h3>Implementing Data Contracts in Python</h3><p>Let’s explore three ways to implement data contracts in Python, using stock price data as an example.</p><h3>1. Using Classes</h3><p>One way to implement a data contract is by defining a class that represents the structure of your data:</p><pre>from datetime import date<br><br>class StockPriceData:<br>    def __init__(self, <br>                 symbol: str,<br>                 date: date,<br>                 open_price: float,<br>                 high_price: float,<br>                 low_price: float,<br>                 close_price: float,<br>                 volume: int,<br>                 adjusted_close: float ):<br>        self.symbol = symbol<br>        self.date = date<br>        self.open_price = open_price<br>        self.high_price = high_price<br>        self.low_price = low_price<br>        self.close_price = close_price<br>        self.volume = volume<br>        self.adjusted_close = adjusted_close<br>        <br>        self.validate()  # Validate immediately after setting attributes<br><br>    def validate(self):<br>        if not isinstance(self.symbol, str):<br>            raise ValueError(&quot;Symbol must be a string&quot;)<br>        if not isinstance(self.date, date):<br>            raise ValueError(&quot;Date must be a date object&quot;)<br>        if self.open_price &lt;= 0:<br>            raise ValueError(&quot;Open price must be positive&quot;)<br>        if self.high_price &lt; self.open_price:<br>            raise ValueError(&quot;High price must be &gt;= open price&quot;)<br>        if self.low_price &gt; self.open_price:<br>            raise ValueError(&quot;Low price must be &lt;= open price&quot;)<br>        if self.close_price &lt;= 0:<br>            raise ValueError(&quot;Close price must be positive&quot;)<br>        if self.volume &lt; 0:<br>            raise ValueError(&quot;Volume must be non-negative&quot;)<br>        if self.adjusted_close is not None and self.adjusted_close &lt;= 0:<br>            raise ValueError(&quot;Adjusted close must be positive&quot;)<br><br># Usage<br>try:<br>    stock_data = StockPriceData(&quot;AAPL&quot;, date(2023, 8, 1), 150.0, 155.0, 149.0, 152.0, 1000000)<br>    print(&quot;Data is valid&quot;)<br>except ValueError as e:<br>    print(f&quot;Data contract violation: {str(e)}&quot;)<br><br># Example of invalid data<br>try:<br>    invalid_stock_data = StockPriceData(&quot;AAPL&quot;, date(2023, 8, 1), 150.0, 145.0, 149.0, 152.0, 1000000)<br>except ValueError as e:<br>    print(f&quot;Data contract violation: {str(e)}&quot;)</pre><h3><strong>2. Simple Error Checking</strong></h3><p>For a lighter-weight approach, you can use simple error checking with assertions:</p><pre>def validate_stock_data(data: dict):<br>    assert &#39;symbol&#39; in data and isinstance(data[&#39;symbol&#39;], str), &quot;Symbol must be a string&quot;<br>    assert &#39;date&#39; in data and isinstance(data[&#39;date&#39;], str), &quot;Date must be a string&quot;<br>    assert &#39;open&#39; in data and isinstance(data[&#39;open&#39;], (int, float)) and data[&#39;open&#39;] &gt; 0, &quot;Open price must be a positive number&quot;<br>    assert &#39;high&#39; in data and isinstance(data[&#39;high&#39;], (int, float)) and data[&#39;high&#39;] &gt;= data[&#39;open&#39;], &quot;High price must be &gt;= open price&quot;<br>    assert &#39;low&#39; in data and isinstance(data[&#39;low&#39;], (int, float)) and data[&#39;low&#39;] &lt;= data[&#39;open&#39;], &quot;Low price must be &lt;= open price&quot;<br>    assert &#39;close&#39; in data and isinstance(data[&#39;close&#39;], (int, float)) and data[&#39;close&#39;] &gt; 0, &quot;Close price must be a positive number&quot;<br>    assert &#39;volume&#39; in data and isinstance(data[&#39;volume&#39;], int) and data[&#39;volume&#39;] &gt;= 0, &quot;Volume must be a non-negative integer&quot;<br><br># Usage<br>stock_data = {<br>    &#39;symbol&#39;: &#39;AAPL&#39;,<br>    &#39;date&#39;: &#39;2023-08-01&#39;,<br>    &#39;open&#39;: 150.0,<br>    &#39;high&#39;: 155.0,<br>    &#39;low&#39;: 149.0,<br>    &#39;close&#39;: 152.0,<br>    &#39;volume&#39;: 1000000<br>}<br><br>try:<br>    validate_stock_data(stock_data)<br>    print(&quot;Data is valid&quot;)<br>except AssertionError as e:<br>    print(f&quot;Data contract violation: {str(e)}&quot;)</pre><h3>3. Using YAML for Schema Definition and Validation</h3><p>For more complex data structures, using YAML to define your schema and a validation library can provide a powerful and flexible solution. Here’s how you can implement this approach:</p><pre># stock_price_schema.yaml<br>symbol:<br>  type: string<br>  required: true<br>date:<br>  type: date<br>  required: true<br>open_price:<br>  type: float<br>  min: 0<br>  required: true<br>high_price:<br>  type: float<br>  min: 0<br>  required: true<br>low_price:<br>  type: float<br>  min: 0<br>  required: true<br>close_price:<br>  type: float<br>  min: 0<br>  required: true<br>volume:<br>  type: integer<br>  min: 0<br>  required: true<br>adjusted_close:<br>  type: float<br>  min: 0<br>  nullable: true</pre><pre>import yaml<br>from cerberus import Validator<br>from datetime import date<br><br># Load the schema from the YAML file<br>with open(&#39;stock_price_schema.yaml&#39;, &#39;r&#39;) as file:<br>    schema = yaml.safe_load(file)<br><br># Create a validator<br>validator = Validator(schema)<br><br><br><br># Function to validate stock data<br>def validate_stock_data(data):<br>    if not validator.validate(data):<br>        raise ValueError(validator.errors)<br>    <br>    # Additional custom validations<br>    if data[&#39;high_price&#39;] &lt; data[&#39;open_price&#39;]:<br>        raise ValueError(&quot;High price must be &gt;= open price&quot;)<br>    if data[&#39;low_price&#39;] &gt; data[&#39;open_price&#39;]:<br>        raise ValueError(&quot;Low price must be &lt;= open price&quot;)<br><br># Usage<br>stock_data = {<br>    &#39;symbol&#39;: &#39;AAPL&#39;,<br>    &#39;date&#39;: date(2023, 8, 1),<br>    &#39;open_price&#39;: 150.0,<br>    &#39;high_price&#39;: 155.0,<br>    &#39;low_price&#39;: 149.0,<br>    &#39;close_price&#39;: 152.0,<br>    &#39;volume&#39;: 1000000<br>}<br><br>try:<br>    validate_stock_data(stock_data)<br>    print(&quot;Data is valid:&quot;, stock_data)<br>except ValueError as e:<br>    print(f&quot;Data contract violation: {e}&quot;)<br><br># Example of invalid data<br>invalid_stock_data = {<br>    &#39;symbol&#39;: &#39;AAPL&#39;,<br>    &#39;date&#39;: date(2023, 8, 1),<br>    &#39;open_price&#39;: 150.0,<br>    &#39;high_price&#39;: 145.0,  # Invalid: high_price &lt; open_price<br>    &#39;low_price&#39;: 149.0,<br>    &#39;close_price&#39;: 152.0,<br>    &#39;volume&#39;: 1000000<br>}<br><br>try:<br>    validate_stock_data(invalid_stock_data)<br>except ValueError as e:<br>    print(f&quot;Data contract violation: {e}&quot;)</pre><p>Implementing data contracts is crucial for ensuring data reliability and consistency in modern data-driven systems. We’ve explored three approaches to creating data contracts for stock price data: Python classes, simple error checking, and YAML-based schema validation. The YAML-based approach stands out for its readability, maintainability, and separation of concerns, offering a clear, language-agnostic specification that’s easily understood by both technical and non-technical team members. By combining a YAML schema with Python-based custom validations, we create a powerful and flexible solution for enforcing data integrity. As data continues to grow in importance, adopting such practices becomes increasingly vital for maintaining data quality, preventing errors, and supporting confident decision-making in any data-driven organization. Choosing the right approach for your needs can significantly enhance the reliability and trustworthiness of your data pipelines, paving the way for more robust and dependable data systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42ad32ae3702" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming Real-Time Stock Data with Python and Kafka]]></title>
            <link>https://tonyyoung3.medium.com/streaming-real-time-stock-data-with-python-and-kafka-1cf208121053?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cf208121053</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 06:33:01 GMT</pubDate>
            <atom:updated>2023-10-09T06:33:01.107Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VwZ8Z1g5XJRiD3hF" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In today’s fast-paced financial markets, having access to real-time stock data is crucial for traders, analysts, and investors. Streaming this data efficiently and reliably can be challenging. However, with the power of Python and Apache Kafka, you can easily set up a data streaming pipeline to capture, process, and analyze real-time stock data.</p><p>Why Use Kafka for Real-Time Data:</p><p>Apache Kafka is a distributed streaming platform that is widely used for building real-time data pipelines. It offers several advantages for streaming real-time stock data:</p><ol><li>Scalability: Kafka can handle massive amounts of data and scale horizontally to meet growing demands.</li><li>Durability: Kafka retains data for a specified period, making it suitable for historical analysis.</li><li>Reliability: It ensures data integrity, even in the face of hardware failures or network issues.</li><li>Publish-Subscribe Model: Kafka’s pub-sub architecture allows multiple consumers to subscribe to real-time data streams simultaneously.</li></ol><h3><strong>Getting Started</strong></h3><p>To begin streaming real-time stock data using Python and Kafka, follow these steps:</p><p>1. Install Kafka:</p><p>Ensure that you have Apache Kafka installed and running. You can download it from the <a href="https://kafka.apache.org/downloads">official Apache Kafka website</a>.</p><p>2. Set Up Kafka Topics and run producer:</p><pre>#create topic<br>bin/kafka-topics.sh --create --topic conn-events --bootstrap-server localhost:9092</pre><pre>#run producer<br>bin/kafka-console-producer.sh --topic conn-events --bootstrap-server localhost:9092</pre><p>3. Python Libraries:</p><p>Install Python libraries for Kafka integration, such as confluent-kafka-python or kafka-python, depending on your preference.</p><p>4. Producer Application:</p><p>Write a Python producer application to fetch real-time stock data from a data provider (e.g., an API) and publish it to the Kafka topics. You can use libraries like requests to fetch data.</p><pre>from confluent_kafka import Producer<br>import yfinance as yf<br>import time<br>import requests<br>import json<br><br># Kafka producer configuration<br>headers = {<br> &#39;User-Agent&#39;: &#39;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3&#39;,<br> &#39;Content-Type&#39;: &#39;application/json&#39;,<br> &#39;Authorization&#39;: &#39;Bearer &lt;token&gt;&#39;<br>}<br><br>conf = {<br> &#39;bootstrap.servers&#39;: &#39;localhost:9092&#39;, # Replace with your Kafka broker address<br> &#39;client.id&#39;: &#39;stock-price-producer&#39;<br>}<br><br># Create a Kafka producer instance<br>producer = Producer(conf)</pre><pre># Kafka topic to send stock price data<br>topic = ‘conn-events’<br># Ticker symbol of the stock (e.g., Apple Inc.)<br>ticker_symbol = &#39;BTC-USD&#39;<br><br>def fetch_and_send_stock_price():<br> while True:<br>   try:<br>     url = &#39;https://query2.finance.yahoo.com/v8/finance/chart/btc-usd&#39;<br>     response = requests.get(url, headers=headers)<br>     data = json.loads(response.text)<br>     price = data[&quot;chart&quot;][&quot;result&quot;][0][&quot;meta&quot;][&quot;regularMarketPrice&quot;]<br>     #Function to fetch stock price and send to Kafka<br>     # Produce the stock price to the Kafka topic<br>     producer.produce(topic, key=ticker_symbol, value=str(price))<br>     producer.flush()<br>     print(f&quot;Sent {ticker_symbol} price to Kafka: {price}&quot;)<br>   except Exception as e:<br>   print(f&quot;Error fetching/sending stock price: {e}&quot;)<br><br># Sleep for a specified interval (e.g., 5 seconds) before fetching the next price<br>   time.sleep(30)<br><br># Start sending stock price data<br>fetch_and_send_stock_price()</pre><p>The price must now be visible on the consumer’s console too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*foNsFIP-VgIZfQmDmc4yoA.png" /></figure><p>5. Consumer Application:</p><p>Write a Python consumer application to store data in csv.</p><pre>from confluent_kafka import Consumer, KafkaError<br>import csv<br>import os<br>from datetime import datetime</pre><pre>#tettt<br># Define the CSV file name<br>csv_file = “data.csv”<br><br># Kafka consumer configuration<br>conf = {<br> &#39;bootstrap.servers&#39;: &#39;localhost:9092&#39;, # Replace with your Kafka broker address<br> &#39;group.id&#39;: &#39;my-group&#39;,<br> &#39;auto.offset.reset&#39;: &#39;earliest&#39;<br>}<br><br># Create a Kafka consumer instance<br>consumer = Consumer(conf)<br><br># Subscribe to a Kafka topic<br>topic = &#39;conn-events&#39; # Replace with your topic name<br>consumer.subscribe([topic])</pre><pre># Consume messages<br>while True:<br> msg = consumer.poll(1.0)<br> <br> if msg is None:<br>   continue<br> if msg.error():<br>   if msg.error().code() == KafkaError._PARTITION_EOF:<br>     print(‘Reached end of partition’)<br>   else:<br>     print(f’Error: {msg.error()}’)<br> else:<br>   print(f’received {msg.value().decode(“utf-8”)}’)<br>   data_price = [(datetime.now().strftime(“%Y-%m-%d %H:%M:%S”), msg.value().decode(“utf-8”))]<br>   if not os.path.exists(csv_file):<br> # If it doesn’t exist, create a new file and write header and data<br>     with open(csv_file, mode=”w”, newline=””) as file:<br>       writer = csv.writer(file)<br>       writer.writerow([“time”, “price”]) # Header<br>       writer.writerows(data_price)<br>   else:<br>   # If it exists, open it in append mode and add data<br>     with open(csv_file, mode=”a”, newline=””) as file:<br>       writer = csv.writer(file)<br>       writer.writerows(data_price)</pre><p>File ‘data.csv’ is created or updated by above scripts</p><h3><strong>Conclusion</strong></h3><p>By using Python and Apache Kafka, you can build a robust and scalable real-time stock data streaming solution. This technology stack empowers you to stay updated with the latest market changes, make informed decisions, and gain valuable insights into financial markets.</p><p>In the ever-evolving world of finance, real-time data is the key to staying ahead of the curve, and Python and Kafka provide the tools you need to harness the power of real-time stock data.</p><p><a href="https://github.com/tonyyoung3/yfinance_kafka/blob/main/stock_price_Consumer.py">complete code</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cf208121053" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Data Pipelines in Airflow]]></title>
            <link>https://tonyyoung3.medium.com/building-data-pipelines-in-airflow-68668b38320?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/68668b38320</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Sun, 27 Aug 2023 20:44:13 GMT</pubDate>
            <atom:updated>2023-08-27T20:44:13.789Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s data-driven world, organizations rely on efficient and robust data pipelines to process, transform, and move data between various sources and destinations. Apache Airflow, an open-source platform, has emerged as a go-to solution for orchestrating and automating complex data workflows. Whether you’re a data engineer, analyst, or scientist, understanding how to build a data pipeline using Airflow can significantly enhance your data processing capabilities.</p><h3><strong>What Is a Data Pipeline and Why Airflow?</strong></h3><p>Before diving into the specifics, we’ll explore the foundational concepts of data pipelines. A data pipeline is a series of processes that extract, transform, and load (ETL) data from its source to a destination. Airflow, known for its programmable workflow automation, brings several advantages to the table. It offers flexibility, scalability, and fault tolerance, making it an ideal choice for orchestrating even the most intricate data workflows.</p><h3>Setting Up Your Airflow Environment</h3><p>We kick-start our journey by guiding you through the setup of an Airflow environment. From installation to configuration, you’ll learn how to prepare the groundwork for building effective data pipelines.</p><pre>#Create a virtual environment<br>pip install virtualenv<br>virtualenv airflow-env<br>source airflow-env/bin/activate</pre><pre>#install and initialize airflow<br>pip install apache-airflow<br>airflow db init<br>airflow webserver — port 8080<br>airflow scheduler</pre><h3><strong>Create DAG</strong></h3><p>On this occasion, I will be setting up a straightforward pipeline. The initial task involves extracting data from a CSV file and then transferring it into an SQLite3 database. Subsequently, the second task employs PySpark to store the data into a CSV file.</p><pre>#Task 1 : extract data from csv and loads it into sqlight3<br>def loading():<br> conn = sqlite3.connect(os.path.dirname(__file__) + “/../test_credit.db”)<br> cursor = conn.cursor()<br> csv_file_path = os.path.dirname(__file__) + “/../price_n.csv”<br> with open(csv_file_path, “r”) as csv_file:<br> csv_reader = csv.reader(csv_file)<br> next(csv_reader) # Skip the header row<br> insert_query = “INSERT INTO stock (Date, Open, High, Low, Close, Volume, id) VALUES (?, ?, ?, ?, ?, ?, ?)”<br> for row in csv_reader:<br> cursor.execute(insert_query, row)<br> conn.commit()<br> conn.close()</pre><pre>#Task 2 use pyspark to extract the data from sqlight3<br>def spark_conn():<br>    spark = SparkSession.builder.appName(&quot;SQLExample&quot;).getOrCreate()<br><br>    db_path= &quot;../test_credit.db&quot;<br><br>    conn = sqlite3.connect(os.path.dirname(__file__) + &quot;/../test_credit.db&quot;)<br>    cursor = conn.cursor()<br><br>    query = &quot;SELECT * FROM stock limit 10 &quot;<br>    result = cursor.execute(query).fetchall()<br><br>    # Convert results to a list of Row objects<br>    row_objects = [Row(*row) for row in result]<br><br>    # Create a DataFrame from the Row objects<br>    df = spark.createDataFrame(row_objects)<br>    <br>    df_single_partition = df.coalesce(1)<br>    out_csv_file_path = os.path.dirname(__file__) + &quot;/../output&quot;<br>    df_single_partition.write.csv(out_csv_file_path, header=True, mode=&quot;overwrite&quot;)<br><br>    # Show the DataFrame<br>    #df.show()<br>    conn.close()<br>    spark.stop()    </pre><pre>#create a DAG which runs abote 2 tasks<br>dag = DAG(<br>    &#39;my_dag&#39;,<br>    default_args=default_args,<br>    description=&#39;A simple DAG&#39;,<br>    schedule_interval=timedelta(days=1),<br>)<br><br>start_task = DummyOperator(task_id=&#39;start&#39;, dag=dag)<br><br>python_task = PythonOperator(<br>    task_id=&#39;loading&#39;,<br>    python_callable=loading,<br>    dag=dag,<br>)<br><br>python_task_2 = PythonOperator(<br>    task_id=&#39;spark_conn&#39;,<br>    python_callable=spark_conn,<br>    dag=dag,<br>)<br><br>end_task = DummyOperator(task_id=&#39;end&#39;, dag=dag)<br><br>start_task &gt;&gt; python_task &gt;&gt; python_task_2 &gt;&gt; end_task</pre><p>Don’t forget to change the path in airflow config file (airflow.cfg). Otherwise the DAG will not be shown on airflow UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*5g4KzIoR4F-9EIYtqW_OAA.png" /></figure><p>Now the DAG is on airflowUI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bE6p11tdiEAu4zZaPosGaw.png" /></figure><p>In conclusion, this article has provided a primer on creating your initial data pipeline using Apache Airflow. Through DAGs, operators, and scheduling, you’ve embarked on an efficient journey to orchestrate data workflows. Our example pipeline — CSV data extraction to SQLite3 and CSV storage via PySpark — lays the groundwork for your data engineering pursuits. With Airflow’s adaptability for complex tasks and dynamic scheduling, you’re poised to automate ETL, ensure data quality, and scale operations. Armed with this foundation, you’re primed to explore advanced data pipeline design and execution, enhancing your ability to drive data-centric projects forward.</p><p>Repo <a href="https://github.com/tonyyoung3/data_pipeline">https://github.com/tonyyoung3/data_pipeline</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68668b38320" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B test for beginner]]></title>
            <link>https://tonyyoung3.medium.com/a-b-test-for-beginner-76632ab0a49b?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/76632ab0a49b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Wed, 12 Apr 2023 07:05:02 GMT</pubDate>
            <atom:updated>2023-04-12T07:05:02.713Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>What is A/B test</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0kvfodwGQyzNrejM" /><figcaption>Photo by <a href="https://unsplash.com/es/@isaacmsmith?utm_source=medium&amp;utm_medium=referral">Isaac Smith</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A/B testing (also known as split testing or bucket testing) is a method of comparing two versions of a webpage or app feature to determine which one performs better. In an A/B test, two variants (A and B) are compared by randomly assigning users to either variant A or B and measuring their respective responses to each version.</p><p>The goal of A/B testing is to identify which variant performs better in terms of a specific metric, such as conversion rate, click-through rate, or engagement. By using this method, businesses can make data-driven decisions about which version to use moving forward.</p><p>A/B testing can be used for a wide range of applications, including optimizing website design, testing new product features, and improving email marketing campaigns. The key to a successful A/B test is to ensure that the variants being tested differ only in the specific element being tested, so that any observed differences in performance can be attributed to that element. Additionally, A/B tests should be carefully designed and executed to ensure statistical significance and avoid biases.</p><h3>Code example</h3><pre>import numpy as np<br>from statsmodels.stats.proportion import proportions_ztest<br><br># Define the control and treatment groups<br>control_group = np.array([0, 1, 1, 0, 0, 1, 0, 1, 0, 1])<br>treatment_group = np.array([1, 1, 0, 1, 1, 1, 0, 1, 0, 1])<br><br># Calculate the number of successes and trials in each group<br>control_successes = sum(control_group)<br>treatment_successes = sum(treatment_group)<br>control_trials = len(control_group)<br>treatment_trials = len(treatment_group)<br><br># Calculate the estimated success rates for each group<br>control_rate = control_successes / control_trials<br>treatment_rate = treatment_successes / treatment_trials<br><br># Conduct a two-sample proportions z-test<br>z_stat, p_value = proportions_ztest([control_successes, treatment_successes], [control_trials, treatment_trials])<br><br># Print the results<br>print(f&quot;Control rate: {control_rate:.3f}&quot;)<br>print(f&quot;Treatment rate: {treatment_rate:.3f}&quot;)<br>print(f&quot;z-statistic: {z_stat:.3f}&quot;)<br>print(f&quot;p-value: {p_value:.3f}&quot;)</pre><p>In this example, we have two groups: a control group and a treatment group. We want to compare the success rates of the two groups (i.e., the proportion of successes out of the total number of trials). We use a two-sample proportions z-test to determine whether the difference in success rates between the two groups is statistically significant. The proportions_ztest function from statsmodels calculates the z-statistic and p-value for us.</p><p>The output of this code will be something like:</p><pre>The p-value in this case is 0.061, which is greater than the typical significance level of 0.05. This means we cannot reject the null hypothesis that there is no difference in success rates between the two groups. However, if the p-value had been less than 0.05, we would have concluded that there is a statistically significant difference between the two groups.</pre><p>The p-value in this case is 0.061, which is greater than the typical significance level of 0.05. This means we cannot reject the null hypothesis that there is no difference in success rates between the two groups. However, if the p-value had been less than 0.05, we would have concluded that there is a statistically significant difference between the two groups.</p><p>Here are some important things to consider when using A/B testing:</p><ol><li>Set clear objectives: It’s important to define clear objectives and key performance indicators (KPIs) for your A/B test. This will help you determine which variant performs better and whether the difference is statistically significant.</li><li>Select a large enough sample size: Your sample size should be large enough to ensure statistical significance. If your sample size is too small, the results may not be reliable. There are various online calculators and tools available to help you determine the required sample size.</li><li>Randomly assign users: To ensure that your results are not biased, users should be randomly assigned to the different variants of your test. This can be done using various methods, such as cookies or IP addresses.</li><li>Run the test for a sufficient duration: A/B tests should run for a sufficient duration to ensure that you capture different user behaviors and patterns. Running the test for too short a period may not provide enough data to make an informed decision.</li><li>Test only one variable at a time: To ensure that you can attribute any differences in performance to the variable being tested, you should only test one variable at a time. Testing multiple variables at once may make it difficult to determine which variable is driving any observed differences.</li><li>Monitor the test carefully: It’s important to monitor the test regularly to ensure that it is running smoothly and that there are no technical issues. You should also check for any unexpected results or anomalies that may impact the test.</li><li>Consider the ethical implications: While A/B testing can be a powerful tool, it’s important to consider the ethical implications of testing on your users. You should ensure that your tests do not negatively impact your users’ experience or violate their privacy.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76632ab0a49b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto apply jobs on LinkedIn in Python]]></title>
            <link>https://tonyyoung3.medium.com/auto-apply-on-linkedin-in-python-349c317cc10d?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/349c317cc10d</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[selenium]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Sat, 25 Feb 2023 07:55:34 GMT</pubDate>
            <atom:updated>2023-02-25T08:07:09.242Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is a step by step instruction to run python to auto apply jobs on LinkedIn. Note that it only applies for the jobs with tag ‘East Apply’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FQ2NDveLB6896HB1" /><figcaption>Photo by <a href="https://unsplash.com/es/@gregbulla?utm_source=medium&amp;utm_medium=referral">Greg Bulla</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Import Selenium and other modules</strong></h3><pre>from selenium import webdriver<br>from selenium.webdriver.common.keys import Keys<br>from selenium.common.exceptions import NoSuchElementException<br>import time<br>from selenium.webdriver.chrome.options import Options<br>chrome_options = Options()<br>chrome_options.add_argument(&#39;--no-sandbox&#39;)<br>chrome_options.add_argument(&#39;--disable-dev-shm-usage&#39;)</pre><p>Be sure to use time.sleep between actions as the browaser needs some time to load data.</p><h3><strong>Observe the application process</strong></h3><p>First, observe the process to know what actions need to be done by Selenium. Below are the steps:</p><ol><li>Log in</li><li>Go to jobs and put the keywords for tile and location in search section</li><li>Select easy apply</li><li>Go through each job in left panel and click Easy apply in right panel</li><li>select resume and other details required</li><li>click next page at left panel if the jobs in current page are all processed</li></ol><p><strong>Steps 1–3</strong></p><p>By observing the url, we can tell that the keywords for title and password are in it. Hence, we can put them into request url directly. Such as :</p><pre>driver.get(&#39;https://www.linkedin.com/jobs/search/?f_AL=true&amp;keywords=product%20Analyst&amp;location=taiwan&amp;geoId=&amp;trk=public_jobs_jobs-search-bar_search-submit&amp;position=1&amp;pageNum=0&#39;)</pre><p>the the code is run by selenium, LinkedIn will ask to login. To put account and password in right place, we need to check the location. First, find the sign in button by chcking link text = ‘Sign In’.</p><pre>time.sleep(2)<br>sign_in_button = driver.find_element(&#39;link text&#39;, &quot;Sign in&quot;)<br>sign_in_button.click()</pre><p>Then, put the password and account, select login</p><pre>time.sleep(5)<br>email_field = driver.find_element(&#39;id&#39;, &quot;username&quot;)<br>email_field.send_keys(ACCOUNT_EMAIL)<br>password_field = driver.find_element(&quot;id&quot;, &quot;password&quot;)<br>password_field.send_keys(ACCOUNT_PASSWORD)<br>password_field.send_keys(Keys.ENTER)</pre><p><strong>Step 4</strong></p><p>Once log in, check the location of job cards. it should have 25 jobs each page. However, if selenium does not scroll down the left panel, the code will only get 7 jobs. In order to mock the scroll function, we need to use javascript to view the last job found. It has to be run for around 3 times to get all 25 jobs.</p><pre>while i &lt; 4:<br>    jobs_list = driver.find_elements(&quot;css selector&quot;, &quot;.job-card-container--clickable&quot;)<br>    print(len(jobs_list))<br>    driver.execute_script(&quot;arguments[0].scrollIntoView();&quot;, jobs_list[-1])<br>    time.sleep(2)<br>    i += 1</pre><p><strong>Step 5</strong></p><p>After getting all job information, we can start to apply it. As before, find the location of the Easy Apply button and click it. Finally the application process starts.</p><pre>apply_button = driver.find_element(&quot;css selector&quot;,&quot;.jobs-apply-button--top-card&quot;)<br>if apply_button.text == &#39;Easy Apply&#39;:<br>    apply_button.click()</pre><p>There are few paths of application process. The first type can be done in one page only. However, the second type has 2–3 pages. Moreover, the thrid type needs more personal information. The code this time will save it rather than trying to finish application. Hence, use if statement to tell selenium what to do</p><pre>try:<br><br>    apply_button = driver.find_element(&quot;css selector&quot;,&quot;.jobs-apply-button--top-card&quot;)<br>    if apply_button.text == &#39;Easy Apply&#39;:<br>        apply_button.click()<br>        time.sleep(5)<br>        submit_button = driver.find_element(&quot;class name&quot;, &quot;artdeco-button--primary&quot;)<br>        # first page<br>        if submit_button.text == &#39;Next&#39;:<br>            time.sleep(5)<br>            # resume_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Choose Resume&#39;]&quot;)<br>            submit_button.click()<br>            # Second page<br>            resume_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Choose Resume&#39;]&quot;)<br>            resume_button.click()<br>            time.sleep(3)<br>            # Third page<br>            review_button = driver.find_element(&quot;class name&quot;, &quot;artdeco-button--primary&quot;)<br>            print(review_button)<br>            if review_button.text == &#39;Review&#39;:<br>                print(&#39;review section&#39;)<br>                review_button.click()<br>                time.sleep(3)<br>                fin_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Submit application&#39;]&quot;)<br>                fin_button.click()<br>                time.sleep(3)<br>            else:<br>                print(&#39;continue section&#39;)<br>                review_button.click()<br>                time.sleep(5)<br><br>                # Forth page<br>                qCheck = driver.find_element(&quot;class name&quot;, &quot;t-16&quot;)<br>                if qCheck.text == &#39;Additional Questions&#39;:<br>                    time.sleep(3)<br>                    close_button = driver.find_element(&quot;class name&quot;, &quot;artdeco-modal__dismiss&quot;)<br>                    close_button.click()<br>                    time.sleep(5)<br>                    # save application which needs more info<br><br>                    save_button = driver.find_elements(&quot;class name&quot;, &quot;artdeco-modal__confirm-dialog-btn&quot;)[1]<br>                    save_button.click()<br>                    time.sleep(5)<br>                else:<br>                    next_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Review your application&#39;]&quot;)<br>                    next_button.click()<br>                    time.sleep(3)<br>                    fin_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Submit application&#39;]&quot;)<br>                    fin_button.click()<br>                    time.sleep(3)<br><br>        elif submit_button.text == &#39;Submit application&#39;:<br>            resume_button = driver.find_element(&quot;css selector&quot;, &quot;[aria-label=&#39;Choose Resume&#39;]&quot;)<br>            resume_button.click()<br>            submit_button.click()<br>            time.sleep(3)<br>        else:<br>            close_button = driver.find_element(&quot;class name&quot;, &quot;artdeco-modal__dismiss&quot;)<br>            close_button.click()<br>            time.sleep(5)</pre><p>Lastly, after 25 jobs are all completed, go to next page.</p><pre>driver.find_elements(&quot;xpath&quot;, f&quot;//button[@aria-label=&#39;Page {page}&#39;]&quot;)[0].click()</pre><p>That’s it. Thanks for reading. Please let me know if anything is not clear and clap if you think it is helpful !</p><h3><strong>Full code</strong></h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c480c2dfebe72a58259c08e83911614b/href">https://medium.com/media/c480c2dfebe72a58259c08e83911614b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=349c317cc10d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get customized dataset from Finviz in Python]]></title>
            <link>https://tonyyoung3.medium.com/get-customized-dataset-from-finviz-in-python-4f6b39347881?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f6b39347881</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[parsing]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Sat, 02 Jul 2022 03:26:43 GMT</pubDate>
            <atom:updated>2022-07-02T03:26:43.485Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xc6LWb4Xv76By76p" /><figcaption>Photo by <a href="https://unsplash.com/@anniespratt?utm_source=medium&amp;utm_medium=referral">Annie Spratt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://finviz.com/screener.ashx">Finviz</a> is a website with a powerful stock screening tool. However, it has some constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IEvKLyXj1D9Es0ph_m25Sg.png" /></figure><ol><li>User can only use one screener set at one time, so it is not easy for user to collect stocks filtered by different screener sets.</li><li>The data falls in different tabs, so user can’t see the data in same window.</li></ol><p>Therefore, I wrote a python script to solve above issues.</p><h3>Website structure</h3><p>First, take a look at the url. Take below url for example, we can see the pattern for parsing</p><figure><img alt="https://finviz.com/screener.ashx?v=111&amp;f=cap_midunder,fa_pe_profitable,geo_usa,ta_highlow52w_nh,ta_sma20_pa,ta_sma200_pa&amp;ft=3&amp;o=industry&amp;r=" src="https://cdn-images-1.medium.com/max/1024/1*2ldbzlJI7gsINn5ucMbrwg.png" /></figure><p>The red part is for different tabs, the purple part is for screener setting, the green part is the rows of data.</p><p>Then check the html structure to see under which nodes the data we want is stored. The columns is stored under node with <strong>class</strong> <strong>table-light </strong>and the data is stored under node with <strong>valign</strong> <strong>top.</strong></p><p><strong>Important : </strong>The node feature must be <strong>unique, </strong>otherwise the parser will get some unnecessary data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hLSH7T1LTm-Wm3C4fMfvDw.png" /><figcaption>column names</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqsMzcltwnM1eZepTpcG2Q.png" /><figcaption>data</figcaption></figure><h3><strong>Coding</strong></h3><p>After finding out where the data is, we can start coding !<br>First, import the tools needed</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9211cc6cae1b55c951da6944121b4349/href">https://medium.com/media/9211cc6cae1b55c951da6944121b4349/href</a></iframe><p>After that, create the parse function. In this function, use bs4 to extract data from the website by observing html structure. Save the column names and data separately and merge them into a dataframe</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be3c34362a2496aa355652b0750aba95/href">https://medium.com/media/be3c34362a2496aa355652b0750aba95/href</a></iframe><p>The second part is file generate function. The function use the function we just created to parse the different tabs and merge them into the final dataset. If the function is triggered more than one time, the data will be saved in same file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c9469b0697f920c2e9049654705a703/href">https://medium.com/media/6c9469b0697f920c2e9049654705a703/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/746dfb997196f46e739b10d2ac59a37c/href">https://medium.com/media/746dfb997196f46e739b10d2ac59a37c/href</a></iframe><p><a href="https://github.com/tonyyoung3/finviz_screener">Code</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f6b39347881" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Python to save Earning Call transcript from Motley Fool]]></title>
            <link>https://tonyyoung3.medium.com/use-python-to-save-earning-call-transcript-from-motley-fool-2116544f5c09?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/2116544f5c09</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[scraping]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Mon, 20 Sep 2021 01:47:40 GMT</pubDate>
            <atom:updated>2021-09-20T02:30:50.491Z</atom:updated>
            <content:encoded><![CDATA[<p>This is for those who wants to get Earning Call(EC) transcript plain text data from Motley fool.</p><p><strong>Table of Contents:</strong></p><ul><li>Observe HTML structure (Motley Fool)</li><li>Use BS4 and re to find nodes where we can get target data</li><li>Parse data and save to txt</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l129XeE-QSTipg0r" /><figcaption>Photo by <a href="https://unsplash.com/@impatrickt?utm_source=medium&amp;utm_medium=referral">Patrick Tomasso</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Observe HTML Structure</h3><p>URL - <a href="https://www.fool.com/author/20032/?page=1">https://www.fool.com/author/20032/?page=1</a></p><p>We want to get the URL embedded in highlighted texts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*IzwZQd4yW-j6vWXmaYvwoQ.png" /></figure><p>By inspecting html code, we can see the link is within class “list-content”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*2413PVfsdpNK6REdiPEccw.png" /></figure><p>Start writing code</p><pre>#import required tools<br>!pip install bs4</pre><pre>from bs4 import BeautifulSoup</pre><pre>import requests</pre><pre>import os</pre><pre>import time</pre><pre>import re</pre><p>Create a function for getting url. Use time.sleep() to avoid requesting too frequently. For each url, run the parsing script we will do later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bec46fdd7043dc53392bce45bff521e5/href">https://medium.com/media/bec46fdd7043dc53392bce45bff521e5/href</a></iframe><p>Next, we need to know which nodes transcript within. the HTML tells us that the transcript is text of tag &lt;p&gt; under Class ‘article-content’</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fL76hXzjc2Z757DE0fQc5g.png" /></figure><p>Based on the result, create function for transcript. And we can use the second paragraph to get year / quarter / stock quote for creating txt file. Note that we have to test RE rule for different pattern of texts. In this function, we also create folder for each stocks and save transcripts to them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0029dbc9826c2044a6503f50c391577a/href">https://medium.com/media/0029dbc9826c2044a6503f50c391577a/href</a></iframe><p>Last, run the file_gen function and create log file</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1ac9777c3f4522ff8f1bfba66b425b9/href">https://medium.com/media/e1ac9777c3f4522ff8f1bfba66b425b9/href</a></iframe><p>Output</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*PvVi_iYoZEd4q7oPuL8dFw.png" /></figure><p><a href="https://github.com/tonyyoung3/python_stock_analysis/blob/main/EC_transcript.ipynb">Code</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2116544f5c09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data exploration on stocks listed on TWSE through Python]]></title>
            <link>https://tonyyoung3.medium.com/data-exploration-on-stocks-listed-on-twse-through-python-ee5a83c9ef0b?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee5a83c9ef0b</guid>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Sun, 27 Jun 2021 17:24:40 GMT</pubDate>
            <atom:updated>2021-06-28T03:19:45.149Z</atom:updated>
            <content:encoded><![CDATA[<p>This is only for beginners who wants to analyze stocks through Python</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5ocN0sBtyoeGiwYv" /><figcaption>Photo by <a href="https://unsplash.com/@kmuza?utm_source=medium&amp;utm_medium=referral">Carlos Muza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>Table of Contents:</strong></p><ul><li>Pull Data with Yfinance Api and TWSE website</li><li>Data cleaning</li><li>Dataframe manipulation</li><li>Visualize</li><li>Calculate ratios</li></ul><h3><strong>Pull Data with Yfinance API and TWSE website</strong></h3><pre>#import modules we need<br>import requests<br>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import scipy.optimize as solver<br>from functools import reduce</pre><pre>!pip3 install yfinance #install yfinance<br>import yfinance as yf</pre><p>First, get stock codes and industry from TWSE websites</p><pre>url = &quot;https://isin.twse.com.tw/isin/C_public.jsp?strMode=2&quot;</pre><pre>html_text = requests.get(url)<br>html_text.encoding =  html_text.apparent_encoding<br>industry_table = pd.read_html(html_text.text)[0]<br>industry_table</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KK8r70W-JLoNKLl1THLQfQ.png" /></figure><p>we just want to keep normal stocks and separate codes and name</p><pre>industry_table.columns = industry_table.iloc[0]</pre><pre>industry_table = industry_table[2:][industry_table[&quot;市場別&quot;] == &#39;上市&#39;][industry_table[&quot;產業別&quot;].isnull() == False]</pre><pre>industry_table[&#39;code&#39;], industry_table[&#39;name&#39;] = industry_table[&#39;有價證券代號及名稱&#39;].str.split().str #split code and name</pre><pre>industry_table = industry_table[industry_table[&#39;code&#39;].str.len() == 4]</pre><pre>industry_table</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2VoxRPCm3Wp2XPHIkbXORA.png" /></figure><pre>for i in industry_table[&#39;code&#39;] :</pre><pre>stock = i + &#39;.TW&#39;</pre><pre>hisdata = yf.Ticker(stock)</pre><pre>df = hisdata.history(period=&quot;5y&quot;)</pre><pre>df[&#39;STOCK_ID&#39;] = i</pre><pre>historical_data = pd.concat([historical_data, df])</pre><pre># time.sleep(0.8)</pre><pre>historical_data_f = historical_data[[&#39;STOCK_ID&#39;,&#39;Close&#39;]][historical_data[&#39;STOCK_ID&#39;].str.len()==4]</pre><pre>historical_data_f</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*pMwnD0wyK6CoPCuLqt5yLQ.png" /></figure><p>we need to set stock codes as columns and close price as values for further exploration</p><pre>df_pivot = pd.pivot(historical_data_f.reset_index(), index=&quot;Date&quot;, columns=&quot;STOCK_ID&quot;, values=&quot;Close&quot;)</pre><pre>df_pivot</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKp2E_OmUj3qsoy5Prh5bw.png" /></figure><p>check columns with missing data</p><pre>df_pivot.columns[df_pivot.isnull().any()]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZB8TtRsqmU2E6ZfKY6hlVQ.png" /></figure><p>every stocks are with missing data, here we use ffill method to use price of previous day</p><pre>df_pivot.fillna(method = &#39;ffill&#39;,inplace=True) #use prev price if missing</pre><p>check data again</p><pre>df_pivot.columns[df_pivot.isna().any()]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEv-aM6PjJqX5sx7t7tWnw.png" /></figure><p>we still can see around 70 stocks with missing data. Let’s check dataset again</p><pre>df_pivot[df_pivot.columns[df_pivot.isna().any()]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FRwRWSdDsgsDdjOaaIgKSw.png" /></figure><p>oh ! looks like some of them are missing price since first day. It might be source issue or they are new after start date. we can drop them this time.</p><pre>df_pivot = df_pivot[ df_pivot.columns[~df_pivot.isnull().any()]]</pre><pre>print(df_pivot[df_pivot.isnull().T.any()])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*Uf1IivLiSpVLC5siv37gOA.png" /></figure><p>now we don’t have missing data</p><h3>Calculation</h3><p>finally we can start calculating values we want to know.</p><p>before that, let’s reset name of column and index</p><pre>df_pivot.index.name = None</pre><pre>df_pivot = df_pivot.rename_axis(columns=None)</pre><p>as we want to use month data to calc return, so we only need price of month end.</p><pre>df_period = df_pivot.groupby(df_pivot.index.strftime(&#39;%Y-%m&#39;)).tail(1)</pre><pre>returns_m = df_period.pct_change()</pre><pre>returns_m = returns_m[1:]</pre><pre>returns_m</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ra-K5kwY5UNc0JPvllT0Jw.png" /></figure><p>we also want to see returns of each industry type, the way is to sum the returns of stocks under same industry and divide it by numbers of stocks</p><pre>for i in industry_table[&#39;code&#39;]:</pre><pre>industry = industry_table[&#39;產業別&#39;][industry_table[&#39;code&#39;]== i].values[0]</pre><pre>if industry not in  returns_m.columns:</pre><pre>if i in returns_m.columns:</pre><pre>returns_m[industry] = returns_m.loc[:,i]</pre><pre>print(returns_m[industry].iloc[0] )</pre><pre>else :</pre><pre>if i in returns_m.columns:</pre><pre>returns_m[industry] = returns_m[industry] + returns_m.loc[:,i]<br></pre><pre>for i in industry_table[&#39;產業別&#39;].drop_duplicates():</pre><pre>n = industry_table.groupby(&#39;產業別&#39;).size()[i]</pre><pre>if i in returns_m.columns :</pre><pre>returns_m[i] = returns_m[i]/n<br></pre><pre>return_industry = returns_m.iloc[:,891:]</pre><pre>return_industry</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTfJo17Jq9VWgAinLjUnww.png" /></figure><p>Let’s go back to stocks to see how data is scattered.</p><pre>return_stock = returns_m.iloc[:,:892]</pre><pre>y = np.array(return_stock.mean()*12 ).tolist()</pre><pre>x = np.array(np.sqrt(return_stock.var()) *12).tolist()</pre><pre>sharp  = return_stock.mean()*12 / np.sqrt(return_stock.var()*12).tolist()</pre><pre>label_s = return_stock.columns.tolist()</pre><pre>import matplotlib.pyplot as plt</pre><pre>fig = plt.figure(figsize = (12,6))</pre><pre>fig.subplots_adjust(top=0.85)</pre><pre>ax = fig.add_subplot()</pre><pre>fig.subplots_adjust(top=0.85)</pre><pre>ax0 = ax.scatter(x=x,y=y,</pre><pre>c=sharp,</pre><pre>marker = &#39;o&#39;)</pre><pre>ax.spines[&#39;right&#39;].set_visible(False)</pre><pre>ax.spines[&#39;top&#39;].set_visible(False)</pre><pre>ax.set_title(&#39;stocks sharp ratio&#39;, fontsize=22, fontweight=&#39;bold&#39;)</pre><pre>ax.set_xlabel(&#39;Risk&#39;)</pre><pre>ax.set_ylabel(&#39;Ann Return&#39;)</pre><pre>fig.colorbar(ax0,ax=ax, label = &#39;Sharpe Ratio&#39;)</pre><pre>for i in range(len(x)) :</pre><pre>if y[i] &gt; 0.3 and x[i] &gt; 4:</pre><pre>plt.annotate(label_s[i], (x[i], y[i]))</pre><pre>plt.savefig(&#39;Efficient_Frontier.png&#39;,dpi=300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qj-hHGw151BgZi-cZ_M3Iw.png" /></figure><p>Some outliers are there. let’s keep them this time. Now we check the relations between industries</p><pre>cov = return_industry.cov()*12 #ANNUALIZE<br>exp_return = return_industry.mean()*12 #ANNUALIZE<br>exp_return.sort_values(ascending=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*hoizzj_EMkDz8fEpEzPIIw.png" /></figure><p>the best performers since 2016 is 航運業、造紙、半導體、鋼鐵. So we can think of building portfolios based on them. Next we check covariance matrix</p><pre>(cov.style</pre><pre>.format(&#39;{:.4f}&#39;)</pre><pre>.background_gradient(&#39;Greens&#39;,)</pre><pre>.applymap(lambda x: &#39;background-color : red&#39; if x &gt; 0.05  else (&#39;background-color : yellow&#39; if x &lt; 0 else &#39;&#39;))</pre><pre>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sW9Ot-qVj-znD2PhFLzygw.png" /></figure><p>Seems top performers are not highly related. So let’s try to build portfolio based on those 4 industries.</p><pre>return_industry_t = return_industry[[‘航運業’,’造紙工業’,’半導體業’,’鋼鐵工業’]]</pre><pre>return_industry_t</pre><p>draw scatter of different weights</p><pre>industry_expected_return =  return_industry_t.mean()*12</pre><pre>covariance_matrix = return_industry_t.cov()*12</pre><pre>risk_list = []</pre><pre>return_list = []</pre><pre>industry = len(return_industry_t.columns)</pre><pre>stop = 0</pre><pre>while stop &lt; 1000000:</pre><pre>try:</pre><pre>stop += 1</pre><pre>weight = np.random.rand(industry)</pre><pre>weight = weight / sum(weight)</pre><pre>return_list.append(sum(industry_expected_return * weight))</pre><pre>risk_list.append(np.sqrt(reduce(np.dot, [weight, covariance_matrix, weight.T])))</pre><pre>except:</pre><pre>pass</pre><pre>fig = plt.figure(figsize = (10,6))</pre><pre>fig.suptitle(&#39;Stochastic simulation results&#39;, fontsize=18, fontweight=&#39;bold&#39;)</pre><pre>ax = fig.add_subplot()</pre><pre>ax.plot(risk_list, return_list, &#39;o&#39;)</pre><pre>ax.set_title(&#39;n=1,000,000&#39;, fontsize=16)</pre><pre>fig.savefig(&#39;result.png&#39;,dpi=300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rQbSMLiknmtvS8EyEAU9eg.png" /></figure><p>get best solution (minimize risk) using constraint</p><pre>x0 = stocks_weights</pre><pre>bounds = tuple((0, 1) for x in range(industry))</pre><pre>constraints = [{&#39;type&#39;: &#39;eq&#39;, &#39;fun&#39;: lambda x: sum(x) - 1}]</pre><pre>minimize_variance = solver.minimize(standard_deviation, x0=x0, constraints=constraints, bounds=bounds)</pre><pre>mvp_risk = minimize_variance.fun</pre><pre>mvp_return = sum(minimize_variance.x * industry_expected_return)</pre><pre>print(&#39;return of minimized portfolio:&#39; + str(round(mvp_return,2)))</pre><pre>print(&#39;risk of minimized portfolio:&#39; + str(round(mvp_risk,2)))</pre><pre>Output: return of minimized portfolio :0.32 <br>         risk of minimized portfolio :0.23</pre><p>check weights</p><pre>for i in range(industry):</pre><pre>print(str(return_industry_t.columns[i])+&#39; % of portfolio : &#39; + str(format(minimize_variance.x[i], &#39;.4f&#39;)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*1-VcCvrDA8nmmZamPpxLIw.png" /></figure><p>store values on efficient frontier</p><pre>x0 = stocks_weights</pre><pre>bounds = tuple((0, 1) for x in range(industry))</pre><pre>efficient_fronter_return_range = np.arange(0.315, 0.340, .0025)</pre><pre>efficient_fronter_risk_list = []</pre><pre>for i in efficient_fronter_return_range:</pre><pre>constraints = [{&#39;type&#39;: &#39;eq&#39;, &#39;fun&#39;: lambda x: sum(x) - 1},</pre><pre>{&#39;type&#39;: &#39;eq&#39;, &#39;fun&#39;: lambda x: sum(x * industry_expected_return) - i}]</pre><pre>efficient_fronter = solver.minimize(standard_deviation, x0=x0, constraints=constraints, bounds=bounds)</pre><pre>efficient_fronter_risk_list.append(efficient_fronter.fun)</pre><p>draw efficient forntier</p><pre>risk_free = 0.01</pre><pre>fig = plt.figure(figsize = (12,6))</pre><pre>fig.subplots_adjust(top=0.85)</pre><pre>ax = fig.add_subplot()</pre><pre>fig.subplots_adjust(top=0.85)</pre><pre>ax0 = ax.scatter(risk_list, return_list,</pre><pre>c=(np.array(return_list)-risk_free)/np.array(risk_list),</pre><pre>marker = &#39;o&#39;)</pre><pre>ax.plot(efficient_fronter_risk_list, efficient_fronter_return_range, linewidth=1, color=&#39;#251f6b&#39;, marker=&#39;o&#39;,</pre><pre>markerfacecolor=&#39;#251f6b&#39;, markersize=5)</pre><pre>ax.plot(mvp_risk, mvp_return,&#39;*&#39;,color=&#39;r&#39;, markerfacecolor=&#39;#ed1313&#39;,  markersize=25)</pre><pre>ax.spines[&#39;right&#39;].set_visible(False)</pre><pre>ax.spines[&#39;top&#39;].set_visible(False)</pre><pre>ax.set_title(&#39;Efficient Frontier&#39;, fontsize=22, fontweight=&#39;bold&#39;)</pre><pre>ax.set_xlabel(&#39;Risk&#39;)</pre><pre>ax.set_ylabel(&#39;Return&#39;)</pre><pre>fig.colorbar(ax0, ax=ax, label = &#39;Sharpe Ratio&#39;)</pre><pre>plt.savefig(&#39;Efficient_Frontier.png&#39;,dpi=300)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZlyoeNEhLFgCYMuqsZ5_aQ.png" /></figure><h3>Conclusion</h3><p>This is just an example of how to construct and evaluate a portfolio. As you can see, the optimized weights are very imbalanced, due to the high Volatility of 航運 and 造紙. The most important thing is to learn how to manipulate dataframe using .iloc and .loc and .style to visualize data, which are very useful.</p><p>Hope this helps.</p><p><a href="https://github.com/tonyyoung3/python_stock.git">Codes</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee5a83c9ef0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[2020 investment year end review]]></title>
            <link>https://tonyyoung3.medium.com/2020-investment-year-end-review-679e866f2500?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/679e866f2500</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[stock-market]]></category>
            <category><![CDATA[美股]]></category>
            <category><![CDATA[stocks]]></category>
            <category><![CDATA[投資]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Fri, 01 Jan 2021 11:17:57 GMT</pubDate>
            <atom:updated>2021-01-03T16:06:27.197Z</atom:updated>
            <content:encoded><![CDATA[<h3>3 stocks to watch in 2021</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QGtng9S7hHzcEtno" /><figcaption>Photo by <a href="https://unsplash.com/@mrkfn?utm_source=medium&amp;utm_medium=referral">Mark Finn</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To summarize 2020, I would say : It is the best times. It is the worst times - as long as you are investor and didn’t short the market. It is my first time to invest in US stock market. Since 16/03/2020, my portfolio performance is around 118% (3/16 ~ 12/31)，NASDAQ100 was around 70%。 In this article, I will share thoughts on <strong>retrospectives of 2020</strong>, <strong>stocks to watch in 2021</strong>, and <strong>market risk in 2021</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*KJ1p9jAG7H2P3riKmv-hkA.png" /></figure><p>In 2020, Some moves can be done better :</p><ol><li>Sold Tesla around 750 (before split) , then bought back around 1400 (before split). I just sold them again around 690 (after split)… we’ll see if it is another big miss.</li><li>Sold Shopify around 700, then bought back around 1,100.</li><li>Sold Zoom around 160, then bought it back around 300.</li><li>Sold Bitcoin around 10,000, then bought it back around 17,000.</li></ol><p>It is smarter to hold the position tight, as long as the reason to buy is still existing. At least , don’t sell it if it is not going below 60MA.</p><p>Looking forward to 2021, I will focus on 3 stocks, as I think the game is changing in these fields.</p><p><strong>Gamestop (GME)</strong></p><p>GME is an American video game, consumer electronics and gaming merchandise retailer. The reasons it would be special in 2021 :</p><ol><li>PS5 is coming ! it is a new super cycle in video game industry</li><li><a href="https://news.microsoft.com/2020/10/08/gamestop-announces-multiyear-strategic-partnership-with-microsoft/">Strategic partnership with Microsoft</a> - in short, GME can benefit from xbox users’ life time revenue. It is a signal that we might see a new paradigm … maybe Sony will do the same thing ?</li><li>Digital transformation - ecommerce revenue is +257% in 2020 (~Q3), closed 800 stores since 2019.</li><li>Ryan Cohen, the former CEO of Chewy, is very <a href="https://www.cnbc.com/2020/11/20/former-chewy-ceo-ryan-cohen-urges-gamestop-to-become-the-amazon-of-video-games.html">bullish</a> on it.</li></ol><p>Not all retailers are defeated by Amazon, like Best Buy. I believe GME is another one. Apart from that, it is still cheap.</p><p><strong>Crwdstrike (CRWD)</strong></p><p>CRWD is an American cybersecurity technology company. It provides endpoint security, threat intelligence, and cyberattack response services. SAAS has been super star in market for a long time. why would CRWD be outstanding ?</p><ol><li>COVID-19 accelerates the adaption of cloud services. Zoom provided the emergency solution, that’s why it soared in 2020. But if companies want to migrate the systems, security is the top concern, and CRWD provides the solution.</li><li><a href="https://www.crn.com/news/security/solarwinds-deploys-crowdstrike-to-secure-systems-after-hack">Solarwinds deploys CRWD’s solution after attack</a>. What does it mean when a solution is adapted by competitor !</li></ol><p>Not like GME, CRWD has never been cheap, as it has a steady 80% YOY even before pandemic. As long as it is the <a href="https://www.crowdstrike.com/resources/reports/gartner-magic-quadrant-endpoint-protection-platforms-2019/">market leader</a> in security industry, it will surprise investors in next few years.</p><p><strong>CRISPR Therapeutics AG (CRISP)</strong></p><p>CRISPR Therapeutics AG, a gene editing company, focuses on developing transformative gene-based medicines for serious human diseases. Famous fund manager Cathie Wood is saying that <a href="https://markets.businessinsider.com/news/stocks/editas-stock-price-ark-cathie-wood-genomic-stocks-driving-returns-2020-12-1029911942">the next FANG will be gene editing</a>.</p><ol><li>If you are not familiar with gene editing, <a href="https://www.netflix.com/watch/80243752?trackId=13752289&amp;tctx=0%2C0%2Cdeac289298b9cff49f9c2a0a132fa29c7aca2e09%3Aae886372f33d99bd2483922c3ee34863b0d65095%2Cdeac289298b9cff49f9c2a0a132fa29c7aca2e09%3Aae886372f33d99bd2483922c3ee34863b0d65095%2Cunknown%2C">this</a> is a good start.</li><li>Emmanuelle Charpentier, Founder of CRISP is <a href="https://www.nobelprize.org/prizes/chemistry/2020/charpentier/facts/">Noble Prize winner in 2020</a>.</li><li>According to <a href="https://research.ark-invest.com/thank-you-deep-learning-wp-0">Ark research,</a> the cost can still be lower.</li></ol><p>There is a lot of potential in gene editing, while market cap of CRISP is only 10B. It would be interesting to see how the story is going.</p><p>Speaking of macro environment, FED’s decision matters. If FED decides to take the money back ( stopping buying bond or raise interest rate) , I will lower my exposure in market. COVID-19 impact will not be as huge as we saw in 2020, as we are getting used to it.</p><p>Anyways, the market is always changing. All we can do is make the plan and change it if needed. Hope you all can beat market again in 2021 !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=679e866f2500" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CS50 結業心得]]></title>
            <link>https://tonyyoung3.medium.com/cs50-%E7%B5%90%E6%A5%AD%E5%BF%83%E5%BE%97-e81f2511693d?source=rss-1309abeb9842------2</link>
            <guid isPermaLink="false">https://medium.com/p/e81f2511693d</guid>
            <category><![CDATA[程式]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[cs50]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[初學]]></category>
            <dc:creator><![CDATA[Tony Yang]]></dc:creator>
            <pubDate>Wed, 16 Dec 2020 14:55:29 GMT</pubDate>
            <atom:updated>2020-12-16T15:03:11.470Z</atom:updated>
            <content:encoded><![CDATA[<p>自從畢業後，不知是終於發現自己的喜好，還是因為社會的現實(?)，開始對寫程式產生興趣，但也是斷斷續續，到處買課程來東看看西看看，就連本文的主角- CS50，早在兩年前就把課程做完，卻遲遲不願面對作業; 直到現在因為工作考核的關係，將完成CS50設定成個人目標，才下定決心把它完成。</p><p>一句話心得 ：作業好花時間R…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RIsqPKqpEULTI0Rj" /><figcaption>Photo by <a href="https://unsplash.com/@goshua13?utm_source=medium&amp;utm_medium=referral">Joshua Aragon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>先簡單介紹我的背景，商學學院畢業，之後後因為工作的關係，對SQL還算熟悉，html / javascript 的部分有上過AlphaCamp的課程，python則是有自己好玩上網學了些爬蟲/資料分析的技巧，但除了SQL都還是很初級的程度。</p><p>我完成的是CS50 2020，每年的課程內容都會有所不同，而且有些作業如果到了下一年是不能重複使用的，所以想完成的話盡量在同一年寫完！以下是CS50 2020的大綱:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*GVAg9R-rzc52ZSE1gksyQQ.png" /></figure><p>課程部份其實都還算好懂，只有memory的部分需要花點時間掌握。作業的話從第四週以後應該都非常花時間！幾乎每個作業我都花了兩小時以上（除了SQL ，因為還算熟悉一小時內就結束了。），期末專案會根據你在第八週的選擇決定內容（Android/iOS/Web/Games)，內容自由度很高，相對的也很花時間構思，大概花了10+小時才完成下面這些東西…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fr0T0b6gXqkc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dr0T0b6gXqkc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fr0T0b6gXqkc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bb1ca6b4e59c60698d98c3db75d606e8/href">https://medium.com/media/bb1ca6b4e59c60698d98c3db75d606e8/href</a></iframe><p>上完這堂課的心得是：課程內容不用說當然精彩，令人驚豔的是作業設計得恰到好處，讓學生可以完全專注在應該要學習的東西上。一般網路課程的作業，不是複製貼上就可以完成，就是會被很多環境設定的東西搞到崩潰。作業也有分難易度，可以依照個人的程度選擇，非常適合像我一樣沒有深入接觸過程式的人。如果對程式有興趣，非常適合拿來當做第一課喔！</p><p>最後附上結業證書，希望有上課的人都可以順利結業！</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7tCJOiR2MZV463xdS03HzA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e81f2511693d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>