<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Suci Lin on Medium]]></title>
        <description><![CDATA[Stories by Suci Lin on Medium]]></description>
        <link>https://medium.com/@suci?source=rss-c069d7b66f91------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Vt5ERXVyqvfkeoTMCyD64w.png</url>
            <title>Stories by Suci Lin on Medium</title>
            <link>https://medium.com/@suci?source=rss-c069d7b66f91------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:42:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@suci/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Snowflake in 20 Minutes via Web-UI]]></title>
            <link>https://medium.com/@suci/snowflake-in-20-minutes-via-web-ui-75a20afd056?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/75a20afd056</guid>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[toturial]]></category>
            <category><![CDATA[snowflake-computing]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Sat, 13 Jul 2019 03:23:47 GMT</pubDate>
            <atom:updated>2019-07-13T03:23:47.270Z</atom:updated>
            <content:encoded><![CDATA[<p>Snowflake provided a great <a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial.html">tutorial</a> to introduce some key concepts and tasks as the followings:</p><p><strong>Create objects -&gt;Load data to table -&gt; Query data</strong></p><ul><li>How to create required Snowflake objects (databases, tables, etc.) for storing and querying data.</li><li>Loading a small amount of sample data from CSV files into a table.</li><li>Querying data from the table.</li></ul><p>However, the tutorial is presented by the <em>Snowflake command line client</em>, <strong>SnowSQL</strong>. Of course, it is a very convenient tool if you are familiar with command line interface. Otherwise, you can try the web interface. Snowflake Web UI is very easy to use and quite powerful.</p><p>Therefore, I will show you the same steps via <strong>Web-UI </strong>to complete the tutorial. Hope it will help you to begin your journal in Snowflake <strong>().</strong></p><h3><strong>Before You Begin</strong></h3><ul><li>An activated Snowflake user to access Snowflake (sign up for a <a href="https://trial.snowflake.com/">free trial</a> if needed) and granted to create databases, tables, and virtual warehouses</li><li>(Nice to have) Basic information about Snowflake</li><li>(Nice to have) Basic knowledge about SQL</li><li>Download sample data files from Snowflake (<a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial-prerequisites.html#sample-data-files-for-loading">link</a>)</li></ul><h3>Getting Started</h3><h4>1. Connect and login into Snowflake web interface</h4><p>To log into the Snowflake web interface:</p><ul><li>Go to the URL provided by Snowflake for your account.</li><li>Login with your credentials</li></ul><p>If you login via SnowSQL, you will need to know what if you Full Account Name related to your Snowflake region. (<a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial-log-in.html">link</a>)</p><h4>2.Create Snowflake Objects</h4><p>After you login, you need to create a database, a table and virtual warehouse.</p><ul><li>Create the sf_tuts database using the <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-database.html">CREATE DATABASE</a> command</li><li>Create a table named emp_basic in sf_tuts.public using the <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-table.html">CREATE TABLE</a> command:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7kmn-RvxG1TNhtnROW0CQ.gif" /><figcaption>Fig 1. Create a database and a table</figcaption></figure><ul><li>Create a table named emp_basic in sf_tuts.public using the <a href="https://docs.snowflake.net/manuals/sql-reference/sql/create-table.html">CREATE TABLE</a> command.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3_40ogGT2gVh3JleVrJ_Q.gif" /><figcaption>Fig 2. Create a data warehouse</figcaption></figure><h4>3.Load data the Target Table</h4><p>There are many ways to import data into Snowflake. For example, to execute <a href="https://docs.snowflake.net/manuals/sql-reference/sql/put.html">PUT</a> or <a href="https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html">COPY</a> to upload local data files to the table via SnowSQL. The alternative way is loading data by <a href="https://docs.snowflake.net/manuals/user-guide/data-load-web-ui.html"><strong>Snowflake Wizard</strong></a> of the Web UI. Simple, effective, but with some limitations (file size limit: 50 MB).</p><p>This step is combined step 3 and step 4 of the tutorial at official site. Upladed data will be load into target.</p><p>If you have downloaded and checked the sample files (<a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial-prerequisites.html#sample-data-files-for-loading">link</a>), they include dummy employee data in CSV format with five records each with comma (,) character delimiter.</p><pre>#An example record<br>Lem,Boissier,lboissier@sf_tuts.com,3002 Ruskin Trail,Shikārpur,8/25/2017</pre><p>In Snowflake, when loading CSV data with a default setting, <strong>no blank spaces </strong>is expected to exist before or after the commas separating the fields in each record.</p><p>In addition, if the entire field contain delimiter (ex, comma characters in the following employees 03.csv), they must be must be escaped or enclosed (double quotes (&quot; &quot;)).</p><pre># employees03.csv of sample data<br>Althea,Featherstone,afeatherstona@sf_tuts.com,&quot;8172 Browning Street, Apt B&quot;,Calatrava,7/12/2017</pre><p>When you in “<strong>Create File Format</strong>” step, you need to choose a correct enclosed characters (Fig 3–6).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z906S5zDD5Q8HU8E9z8KQA.gif" /><figcaption>Fig 3. upload data</figcaption></figure><p>Note: The uploaded files will be removed after the data loading to table is finish. You can not see the list of the uploaded files in stage by executing a <a href="https://docs.snowflake.net/manuals/sql-reference/sql/list.html">LIST</a> command.</p><pre># Corresponding SQL statement<br># internal stage name was created automatically by Snowflake</pre><pre>PUT file://&lt;file_path&gt;/employees02.csv <a href="http://twitter.com/EMP_BASIC/ui1562857962069">@EMP_BASIC/ui1562857962069</a><br>PUT file://&lt;file_path&gt;/employees04.csv <a href="http://twitter.com/EMP_BASIC/ui1562857962069">@EMP_BASIC/ui1562857962069</a><br>PUT file://&lt;file_path&gt;/employees01.csv <a href="http://twitter.com/EMP_BASIC/ui1562857962069">@EMP_BASIC/ui1562857962069</a><br>PUT file://&lt;file_path&gt;/employees03.csv <a href="http://twitter.com/EMP_BASIC/ui1562857962069">@EMP_BASIC/ui1562857962069</a><br>PUT file://&lt;file_path&gt;/employees05.csv <a href="http://twitter.com/EMP_BASIC/ui1562857962069">@EMP_BASIC/ui1562857962069</a></pre><pre>COPY INTO “SF_TUTS”.”PUBLIC”.”EMP_BASIC” FROM @/ui1562857962069 FILE_FORMAT = ‘“SF_TUTS”.”PUBLIC”.”EMPLOYEE”’ ON_ERROR = ‘ABORT_STATEMENT’ PURGE = TRUE;</pre><p>Please see more details in<a href="https://docs.snowflake.net/manuals/user-guide/data-load-web-ui.html"> Loading Using the Web UI</a></p><h4>Step 4. Query the Loaded Data</h4><p>After you loaded data into the target table “em_basic”, you can use SQL statement to query data.</p><p>Four examples introduced in the <a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial-query.html">tutorial</a> as the followings.</p><ul><li>Return all rows and columns from the table</li><li>Insert two additional rows into the table</li><li>Return a list of email addresses with United Kingdom domain names using the <a href="https://docs.snowflake.net/manuals/sql-reference/functions/like.html">LIKE</a> function</li><li>Filter the list by employees whose start date occurred earlier than “January 1, 2017&quot;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8LqbVCsXM6kQ3FuxW64I1A.gif" /><figcaption>Fig 4. Query the Loaded Data</figcaption></figure><pre># Query the Loaded Data</pre><pre>SELECT count(*) FROM emp_basic;</pre><pre>SELECT * FROM emp_basic;</pre><pre>INSERT INTO emp_basic VALUES<br>  (&#39;Clementine&#39;,&#39;Adamou&#39;,&#39;cadamou@sf_tuts.com&#39;,&#39;10510 Sachs Road&#39;,&#39;Klenak&#39;,&#39;2017-9-22&#39;) ,<br>  (&#39;Marlowe&#39;,&#39;De Anesy&#39;,&#39;madamouc@sf_tuts.co.uk&#39;,&#39;36768 Northfield Plaza&#39;,&#39;Fangshan&#39;,&#39;2017-1-26&#39;);</pre><pre>SELECT email FROM emp_basic WHERE email LIKE &#39;%.uk&#39;;</pre><pre>SELECT first_name, last_name, DATEADD(&#39;day&#39;,90,start_date) FROM emp_basic WHERE start_date &lt;= &#39;2017-01-01&#39;;</pre><h4>Step 5. Clean Up</h4><p>It is not necessary to drop database and warehouse. You should make sure what your are going to do before you execute it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IQ5KxGU9YC6znQpxUYD1A.gif" /><figcaption>Fig 5. Clean up</figcaption></figure><pre># Clean Up DB and Warehouse</pre><pre>DROP DATABASE IF EXISTS sf_tuts;<br><br>DROP WAREHOUSE IF EXISTS sf_tuts_wh;</pre><h3>Summary</h3><p>In this article, I introduced how to use very basic functions of Snowflake step by step. Most commands and steps are from the tutorials, <a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial.html">Snowflake in 20 Minutes</a>.</p><p>Some concepts are as the followings.</p><ul><li>Data loading is performed in two steps in Snowflake: Stage data and Copy data into a target table.</li><li>A file format for loading data is important and should be aligned with your input data. Web UI provides a simple interface to create it.</li><li>There are no much difference between SnowSql and WebUI, but user interface, uploading file limitation, and <a href="https://docs.snowflake.net/manuals/user-guide/ui-query-profile.html">query profiling</a> (I didn’t show here). I use Web UI all the time, but loading data mechanism is more systematical and programmed.</li></ul><p>There are more resources you can learn about Snowflake. You can refer the reference in the bottom.</p><p>本文分享了如何透過Web UI進行基本的Snowflake 應用與操作：</p><p>主要步驟為：</p><ol><li>登入</li><li>建立Snowflake Objects</li><li>Upload data</li><li>Query data</li><li>清理database 和 warehouse</li></ol><p>內容參考自Snowflake 官方網站的教學 <a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial.html">Snowflake in 20 Minutes</a>，官網的教學是以<em>Snowflake command line client</em>, <strong>SnowSQL</strong>, 作為示範，但是本文完全以Web UI來操作。</p><p>請務必參考延伸參考文件來學習更進階的Snowflake的操作和資料庫應用邏輯。</p><h3>Reference</h3><ol><li><a href="https://docs.snowflake.net/manuals/user-guide-intro.html">Introduction to Snowflake</a></li><li><a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial.html">Snowflake in 20 Minutes</a></li><li><a href="https://docs.snowflake.net/manuals/other-resources.html">Getting Started and more Tutorials, Videos &amp; Other Resources</a></li><li><a href="https://docs.snowflake.net/manuals/user-guide/ui-using.html">Introduction of Snowflake Web Interface</a></li><li><a href="https://www.snowflake.com/blog/tech-tip-getting-data-snowflake/">Tech Tip: Quick Start for Getting Your First Data into Snowflake</a></li><li><a href="https://docs.snowflake.net/manuals/sql-reference/constructs.html">Query Syntax</a></li><li><a href="https://docs.snowflake.net/manuals/sql-reference/sql-dml.html">DML Commands</a></li><li><a href="https://community.snowflake.com/s/">Snowflake Community and education resources</a> (Snowflake Lodge)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75a20afd056" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hello World! Snowflake Data Warehouse]]></title>
            <link>https://medium.com/@suci/hello-world-snowflake-data-warehouse-199c74f9f2fb?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/199c74f9f2fb</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[hello-world]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Sun, 02 Jun 2019 16:50:20 GMT</pubDate>
            <atom:updated>2019-06-09T14:50:38.797Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PZiMdZqkB2mvlbLIpRxP4g.jpeg" /><figcaption><a href="https://unsplash.com/photos/ePwEGur8gIw">https://unsplash.com/photos/ePwEGur8gIw</a></figcaption></figure><p>If you google “<strong><em>Snowflake</em></strong>”, you may still get some results talking about a small piece of snow. However, in a data engineering area, especially for data warehouse, the <strong><em>Snowflake</em></strong><em> </em>is gaining popularity very quickly.</p><p>I have been trying get familiar with Snowflake since the few days ago. There are some impressive features I found from Snowflake, including</p><ul><li>Great Web UI</li><li>Cloud solution (Elastic computing)</li><li>CLONE command to copy data object</li><li>Account-to-account sharing of data (data sharing)</li><li>Third party connections</li></ul><p>To bring a little more about Snowflake, I am going to introduce how to show a Hello-World example on Snowflake.</p><h3>Before You Begin</h3><h4>Prerequisites</h4><p>It should be easy to show “Hello World” by following the steps in this article. However, it’s better to have the following knowledge.</p><ul><li>(Nice to have) Basic information about Snowflake Web UI</li><li>(Nice to have) Basic information about SQL and SQL UDTFs（User-Defined Table Function）</li></ul><h3>Getting Started</h3><h4>1. Get an account or Snowflake user role (sign up for a <a href="https://trial.snowflake.com/">free trial</a> if you do not have already one)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E4NLk-3zjXukWw8lRzjuJg.gif" /><figcaption>Sign up a Snowflake Account</figcaption></figure><h4>2. Connect and login into Snowflake web interface</h4><p>To log into the Snowflake web interface:</p><ul><li>Go to the URL provided by Snowflake for your account.</li><li>Login with your credentials</li></ul><h4>3. (Optional) Getting to know Snowflake</h4><p>If you closed it before going through this simple tutorial. You can show it by clicking <strong>show Help panel</strong> from <strong>Help</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xe5a_5WJD17tBb4RtAO8yA.gif" /></figure><p>Know more about the UI of <a href="https://docs.snowflake.net/manuals/user-guide/snowflake-manager.html#worksheet-page">Worksheet Page</a> by the following instruction picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*edJcKqjcysYz2ijx.png" /><figcaption>Worksheet Page</figcaption></figure><h4>4. Run SQL statement to get Hello World</h4><ul><li>Copy and pasta the following simple example SQL statement in SQL editor. Replace <strong><em>DEMO_DB</em></strong> by your database name.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e5acc8443666b33973848b807e6bd4a/href">https://medium.com/media/6e5acc8443666b33973848b807e6bd4a/href</a></iframe><ul><li>Run the statement to get the result<strong>: Hello</strong> and <strong>World</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B9TIsd9jhWFJ7ZsxiAKAlQ.png" /><figcaption><strong>Hello World</strong></figcaption></figure><h3>Conclusion</h3><p>This is a <strong>SQL Hello World </strong>example to help you to get more about <strong>Snowflake</strong>. There are many data warehouse companies. Several of them provide cloud computing service. <strong>Snowflake</strong> is worth to study if you are looking for a new data warehousing solution.</p><p>I didn’t cover any deeper topics about <strong>Snowflake. </strong>Actually, you can find many <a href="https://docs.snowflake.net/manuals/index.html">excellent documentation</a> from the official site. Getting started with Snowflake from <a href="https://support.snowflake.net/s/education-services">Snowflake Education Services</a> is also a good option.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5Jk0pyO9HpnYHDWupWcGA.png" /><figcaption><a href="https://support.snowflake.net/s/education-services">Snowflake Education Services</a></figcaption></figure><p>本文分享了❶一個近期熱門的雲端data warehouse <strong>服務 Snowflake </strong>，並❷透過一個簡單SQL Hello World 範例，介紹<strong>Snowflake</strong>的worksheet介面．</p><p>主要步驟為：</p><ol><li>申請並活化帳號</li><li>登入並連結</li><li>瀏覽Web使用者介面</li><li>在Web 介面的worksheet上，執行SQL User-Defined Table Function，取得結果( Hello World)</li></ol><p>請參考官方文件，更多延伸參考文件和<a href="https://support.snowflake.net/s/education-services">Snowflake Education Services</a> 來學習<strong>Snowflake</strong>更進階的使用方法。</p><h3>Reference</h3><ol><li><a href="https://docs.snowflake.net/manuals/user-guide/getting-started-tutorial.html">Snowflake in 20 Minutes</a> (Good tutorial)</li><li><a href="https://docs.snowflake.net/manuals/user-guide/snowflake-manager.html">Quick Tour of the Web Interface</a></li><li><a href="https://docs.snowflake.net/manuals/sql-reference/udf-table-functions.html#basic-examples">Sample SQL UDTFs (User-Defined Table Functions)</a></li><li><a href="https://support.snowflake.net/s/education-services">Snowflake Education Services</a></li><li>Chinese articles about Snowflake (still not many for now)</li></ol><ul><li><a href="https://medium.com/%E3%84%9A%E5%8C%97%E7%9A%84%E6%89%80%E8%A6%8B%E6%89%80%E8%81%9E/create-snowflake-trial-account-c52d0ec330ca">Create Snowflake trial account</a></li><li><a href="https://medium.com/%E3%84%9A%E5%8C%97%E7%9A%84%E6%89%80%E8%A6%8B%E6%89%80%E8%81%9E/snowflake-credit-%E8%A8%88%E8%B2%BB%E6%96%B9%E5%BC%8F-506cd8693092">Snowflake credit 計費方式</a></li><li><a href="https://medium.com/%E3%84%9A%E5%8C%97%E7%9A%84%E6%89%80%E8%A6%8B%E6%89%80%E8%81%9E/snowflake-time-travel-85e36273ea27">Snowflake Time Travel</a></li><li><a href="https://medium.com/%E3%84%9A%E5%8C%97%E7%9A%84%E6%89%80%E8%A6%8B%E6%89%80%E8%81%9E/snowflake-ddl-%E6%85%A2%E6%85%A2%E6%9B%B4%E6%96%B0-10dc375ca330">Snowflake DDL</a></li><li><a href="https://medium.com/%E3%84%9A%E5%8C%97%E7%9A%84%E6%89%80%E8%A6%8B%E6%89%80%E8%81%9E/snowflake-connector-for-python-ceba57631897">Snowflake Connector for Python</a> (added in 6/9)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=199c74f9f2fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python with VS code]]></title>
            <link>https://medium.com/@suci/python-with-vs-code-bc2285c9088d?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc2285c9088d</guid>
            <category><![CDATA[vscode]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[workflow]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Wed, 01 May 2019 10:07:11 GMT</pubDate>
            <atom:updated>2019-05-09T00:33:07.506Z</atom:updated>
            <content:encoded><![CDATA[<p>After working with several different source code editors, started with Vim, Atom, and Pycharm/IntelliJ, and the recent testing is<strong> Visual Studio Code. </strong>My favorite is still <strong>Vim</strong>. However, I still spent much time to switch between <strong>Pycharm/IntelliJ</strong> and <strong>VS Code </strong>for some testing 😳.</p><p>I really like VS Code’s <strong>light-weight</strong>, highly customizable with many extensions and it is open source. I keep VS Code as my favorite and default editor in GUI-based OS (eg. MacOS). But for larger code bases, I sometimes consider switching to Pycharm.</p><p>Recently, I watched and learned some things new from a great <a href="http://https%3A%2F%2Fep2018.europython.eu%2Fconference%2Ftalks%2Fget-productive-with-python-in-visual-studio-code">sharing</a> of <strong>EuroPython</strong> <strong>2018</strong> for Python and VS Code. It is inspiring me to write down how I use VS Code with Python (<strong>general development workflow, configuration </strong>with some extensions) and share it. You may find somethings helpful.</p><h3>PRE-REQUISITES</h3><ul><li>Basic knowledge of Python</li><li>Install <a href="https://www.python.org/downloads/">Python 3 </a>in your OS (I use pyenv to switch different versions of Python)</li><li>(Optional) Go through <a href="https://code.visualstudio.com/docs">VS Code Getting Started Doc</a> to know the basic usages</li><li><a href="https://code.visualstudio.com/Download">Download and install the VS Code</a></li><li>Install the <a href="https://marketplace.visualstudio.com/items?itemName=ms-python.python">Python extension for VS Code</a></li><li>(Optional) Install more extensions for newly installation or backup by <a href="https://marketplace.visualstudio.com/items?itemName=Shan.code-settings-sync">Settings Sync</a> or well-defined <a href="https://code.visualstudio.com/docs/editor/extension-gallery#_workspace-recommended-extensions">recommended extensions</a></li></ul><h3>Getting Started</h3><h3>1. General Development Workflow</h3><p>My development flow for python projects with a well-configured VS Code is as the followings:</p><h4>a. Preparation Stage</h4><ul><li>(prerequisite) <strong>Create a new virtual environment</strong> for Python interpreter (Optional, if not exist for this project)</li></ul><p><strong>(i) </strong>Start VS Code (in a project folder as the below). The folder will be considered as “workspace” in VS Code.</p><pre>mkdir py_project<br>cd py_project<br>code </pre><p>Or<strong> </strong>open my working Workspace/Project/Folder/File (<strong>File &gt; Open Folder</strong>)</p><pre>* Open a Folder / Project — <strong>⌘Cmd + o<br></strong>* Create a new file — <strong>⌘Cmd + n<br></strong>* Open the terminal pane — <strong>Ctrl + `</strong></pre><p><strong>(ii) Select a Python interpreter</strong> (if need to change)</p><ul><li>open<strong> Command Palette</strong> (⇧⌘P) =&gt; <strong>Python: Select Interpreter</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZCTlAGyxTzcsEjIqB72cpw.gif" /><figcaption>Change Python interpreter</figcaption></figure><p><strong>(iii) Create a terminal for the active interpreter of the current virtual environment</strong></p><pre><strong>⌘Cmd + shift + p =&gt; Search and choose &quot;Python:Create Terminal</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6035AVK8dau0DoHNW16IkQ.png" /><figcaption>Create a terminal for the current Python interpreter</figcaption></figure><h4>b. Developing Stage</h4><p><strong>(i) Coding</strong> (after defining purposes and necessary system design)</p><p><strong><em>Editing code</em></strong></p><ul><li>Working with <strong>Autocomplete</strong> and <a href="https://code.visualstudio.com/docs/editor/intellisense"><strong>IntelliSense</strong></a></li><li>Make code more readable and conventional with<strong> Formatting (autopep8) and Linting (Pylint)</strong></li><li>Refactoring with <strong>Extract Variable, Extract Method and Sort Imports</strong></li><li>See more details on editing, formatting, and refactoring in <a href="https://code.visualstudio.com/docs/python/editing">Editing code</a> (VS Code official doc).</li></ul><p><strong><em>Select code and run</em></strong></p><ul><li>After finishing a part of coding, you may want to try it to see if it works as you expected.</li><li>Select one or more lines, then press <strong>Shift+Enter</strong> or right-click (Python: Run Selection/Line in Python Terminal). The part of the code will be executed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X29iP2ESfNpMDsBZw2ueLw.png" /></figure><p><strong><em>Save the code</em></strong></p><ul><li>Save (⌘S)</li></ul><p><strong>(ii) Debugging</strong></p><pre><strong>General debugging flow</strong></pre><pre>1.Add debugging breaking points<br>2.Switch the debugging mode by &quot;Starting Debugging&quot; (F5)<br>3.Check code, variables and behaviors <br>4.Debug Console<br>5.With debugging actions to check code, variables and behaviors:<br> * &quot;Step Into&quot; (F11) &quot;Step Out&quot; (Shift F11)<br> * &quot;Step Over&quot; (F11) <br> * &quot;Run to Cursor&quot;, one time breaking point<br> * Continue debugging (F5):jump to the next breaking points</pre><p><a href="https://code.visualstudio.com/docs/python/python-tutorial#_configure-and-run-the-debugger">https://code.visualstudio.com/docs/python/python-tutorial#_configure-and-run-the-debugger</a></p><p><strong>(iii) Testing</strong></p><ul><li>Discover Unit Tests</li><li>Enable and configure test framework</li><li>Select test folder to run the test</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b5hXi9URgUPn9uxCaP1tFA.png" /></figure><h3>2. Configuration</h3><p>Some configurations I may change</p><h4>Edit Settings (<strong>Ctrl/⌘cmd +,)</strong></h4><p><strong>(i) </strong>Configure the Python extension (Setup the Python interpreter path)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4mPL9U8PVfFO7bpQTPZUig.png" /></figure><p><strong>(ii) </strong>Use environment variables</p><p><strong>(iii)</strong> Code Linting</p><ul><li>Customize pylint (.pylintrc) (<a href="https://code.visualstudio.com/docs/python/linting">ref</a>)</li></ul><p><strong>(iv) </strong>Code Formatting</p><p><strong>(v) </strong>Unit tests</p><h3>3. Know More Extension</h3><p>Some extensions I used to make my code editing easier. (changed time to time)</p><p>Linter</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker">Code Spell Checker</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=shardulm94.trailing-spaces">Trailing Spaces</a></li></ul><p>Formatters</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=aaron-bond.better-comments">Better Comments</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=spywhere.guides">Guides</a></li></ul><p>Snippets</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=njpwerner.autodocstring">autoDocstring</a></li></ul><p>Theme</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=Equinusocio.vsc-material-theme">Material Theme</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=vscode-icons-team.vscode-icons">vscode-icons</a></li></ul><p>Other</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=auiworks.amvim">amVim</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=CoenraadS.bracket-pair-colorizer">Bracket Pair Colorizer</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens">GitLens — Git supercharged</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one">Markdown All in One</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=alefragnani.project-manager">Project Manager</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=Shan.code-settings-sync">Settings Sync</a></li><li>TODO <a href="https://marketplace.visualstudio.com/items?itemName=wayou.vscode-todo-highlight">Highlight</a></li></ul><h3>4. Know More Tips</h3><ul><li>Familiar some Key Shortcut (<a href="https://github.com/Microsoft/vscode-tips-and-tricks">VS Code Tips and Tricks</a>)</li><li><a href="https://code.visualstudio.com/docs/getstarted/keybindings">Customized key binding</a></li><li><a href="https://code.visualstudio.com/docs/editor/userdefinedsnippets">Creating Your Own Snippets</a></li><li><a href="https://code.visualstudio.com/docs/editor/integrated-terminal">Use built–in terminal emulator</a></li><li><a href="https://code.visualstudio.com/docs/azure/docker">Working with Docker</a></li><li><a href="https://code.visualstudio.com/docs/editor/versioncontrol">Git/ VCS integration</a> (just convenient, may not fully functional if compared with source tree or other GUI)</li><li><a href="https://code.visualstudio.com/docs/python/jupyter-support">Working with Jupyter Notebooks</a></li></ul><h3>Conclusion</h3><p>In this article, I showed how I use VS code generally (basic development flow). I don’t write too much about my configuration, refactoring, and some other tips and my habits. I may update it time to time. However, the official website of VS code actually contains a lot of useful articles and tutorial. There are more detailed functions and tips to save you time in your everyday development lifecycle.</p><p>Using more and taking more advantage of all of Visual Studio Code’s capabilities will maximize my productivity. I believe my working flow with VS Code will be changed and improved time to time.</p><p>本文分享我使用VS code 進行python 程式的開發流程 （但我實際上最喜歡使用Vim）</p><p>❶開發環境的預先準備</p><ul><li>相關程式的安裝 (Python, VS Code和 extension)</li></ul><p>❷開發流程</p><p><strong>(i) Coding</strong></p><ul><li><strong>開</strong> VS code -&gt;開working Workspace/Project/Folder/File</li><li><strong>新的專案要建立新的</strong> virtual environment for Python interpreter</li><li><strong>確認是否需要切換虛擬環境或是 </strong>interpreter</li><li><strong>edit</strong></li><li><strong>(select code) and run</strong></li><li><strong>save</strong></li></ul><p><strong>(ii) Debugging</strong></p><ul><li>加breaking point,</li><li>‘F5’ run debugging</li><li>Stepping並同時 Inspecting 變數 和利用Debug</li><li>也可執行Run to Cursor</li></ul><p><strong>(iii) Testing</strong></p><ul><li>Discover Unit Tests</li><li>Enable and configure test framework</li><li>Select test folder to run the test</li></ul><p>官方網站其實有很多參考文件，可以協助學習如何建置更進階的VS Code環境和許多進階用法。</p><h3>Reference:</h3><ol><li><a href="https://code.visualstudio.com/docs">https://code.visualstudio.com/docs</a> (VS code official site)</li><li><a href="https://code.visualstudio.com/docs#python-articles">https://code.visualstudio.com/docs#python-articles</a></li><li><a href="https://devblogs.microsoft.com/python/">https://devblogs.microsoft.com/python/</a> (latest update about python in Microsoft)</li><li><a href="https://code.visualstudio.com/docs/python/python-tutorial">Getting Started with Python in VS Code</a> (Great tutorial)</li><li><a href="https://twitter.com/pythonvscode">https://twitter.com/pythonvscode</a></li><li><a href="https://code.visualstudio.com/docs/python/environments">Configuring Python environments</a> (from the official site)</li><li><a href="https://code.visualstudio.com/docs/getstarted/keybindings#_default-keyboard-shortcuts">keyboard-shortcuts</a> (from the official site)</li><li><a href="https://ep2018.europython.eu/conference/talks/get-productive-with-python-in-visual-studio-code">Get Productive with Python and VS Code</a> (EuroPython 2018, recommend to watch for beginners)</li><li><a href="https://www.barbarianmeetscoding.com/blog/2019/02/08/boost-your-coding-fu-with-vscode-and-vim">Boost Your Coding Fu With Visual Studio Code and Vim</a> (If you like vim as well)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc2285c9088d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Classes of Systems Thinking with DevOps]]></title>
            <link>https://medium.com/@suci/4-classes-of-systems-thinking-with-devops-becdfe4a84f0?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/becdfe4a84f0</guid>
            <category><![CDATA[系統思考的四堂課]]></category>
            <category><![CDATA[devopsdays-taipei-2018]]></category>
            <category><![CDATA[systems-thinking]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Mon, 03 Sep 2018 14:28:47 GMT</pubDate>
            <atom:updated>2018-09-03T16:43:10.632Z</atom:updated>
            <content:encoded><![CDATA[<p>I took a <a href="https://devopsdays.tw/workshop.html#workshop0831">Systems Thinking workshop </a>delivered by <a href="https://www.slideshare.net/williamyeh">William Yeh</a>, who is a Server Director at Gogolook. This is the first workshop of a series <a href="https://devopsdays.tw/">DevOpsDays Taipei 2018 Workshop.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iUdRtIKtnE00_aUk5s0nAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJE2NydrTiNB9iTRr348XA.jpeg" /></figure><p>William is an experienced architect and lecturer. He is one of the reasons to drive me to spend money and one working day to take this course. Another reason is that he was going to use Systems Thinking approach to <strong>organization archetypes</strong> with <strong>Conway’s Law.</strong></p><p>I am going to share some feedbacks and brief summaries of this workshop, but not details. Because you are recommended to take his class if you cannot apply system thinkings well. :)</p><h3>Two Pre-Class Homework</h3><p>We were assigned to read two materials:</p><ul><li>Chapter 7 of <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=1&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiw4Zb-7Z7dAhXCP48KHWS6C4QQFjAAegQIBxAB&amp;url=https%3A%2F%2Fwww.amazon.com%2FDevOps-Handbook-World-Class-Reliability-Organizations%2Fdp%2F1942788002&amp;usg=AOvVaw3qtpSPnp65HJj6t6s-je8R">The DevOps Handbook</a>: <strong>How to design our Organization and Architecture with Conway’s Law in mind</strong></li><li>Two articles about <a href="https://en.wikipedia.org/wiki/Middle_income_trap"><strong>Middle income trap;</strong></a><strong> </strong><a href="http://www.naipo.com/Portals/1/web_tw/Knowledge_Center/mainland/IPNC_161214_0803.htm"><strong>1</strong></a><strong> and </strong><a href="http://bit.ly/2rJcqQA"><strong>2</strong></a><strong> (in Chinese)</strong></li></ul><p>Then we were supposed to extract cause-and-effect logics from these articles and answer a few questions. William would explain it and analyze two systems by Systems Thinks approach in the class.</p><p>This assignment actually was quite harder if you haven’t had related training and don’t spend time on it.</p><h3>4 Classes</h3><h4>The First Class: Introduction/Causal Analysis</h4><p>The first class focused on how to think and what is Thinking Model. He taught fundamental parts about <strong>Thinking Model </strong>by extracting cause-and-effect logics from a speech of Charles T. Munger at USC in 1994. The term,<strong> System, </strong>was also introduced. In the final part, he introduced SLR-CPI approach to develop a thinking process</p><p>SLR-CPI: developed by William himself, including 6 steps: Status statement, Label, Relation, Check, Pattern, Integration.</p><h4><strong>The Second Class: System Dynamics</strong></h4><h4>The Third Class: Systems Archetypes</h4><p>The 2nd and 3rd classes talked about CLD and 9 System Archetypes with many examples and practices. It really needs to take a lot of practices to apply System Archetypes well. I can only use a few simple archetypes into the CLD.</p><h4>The Fourth Class: Applications</h4><p>All the students were split into 3 teams. We started tp discuss examples and to adopt what we have learned before in this class.</p><p>In the final part of this class, William shared the CLD for our assigments and explian how to use it to think and to solve problems. The CLDs made by William are really sophisticated. They were definitely refined again and again.</p><h3>Feedbacks</h3><p>During this class, to know more about <strong>how to think</strong> and <strong>how to accumulate thinking models</strong> are what I consider most useful.</p><p>Systems thinking is a great approache to help you to solve the problem. As I mentioned, it really takes time to be proficient. I will start adpot it to train my ability and accumulate thinks models to help me make a decision.</p><p>William also provided many good reference books for Systems Thinking, Thinking Models, Growth Mindset, and self-growth. I will start with Thinking Models to learn experts’ experience and models.</p><p>I really recommend this class, especially if you are interseted in <strong>organization in software department.</strong></p><h3><strong>Feedbacks in </strong>Chinese</h3><p>對於我這個想要入門<strong>系統思考的人，</strong>很幸運能參加系統思考的四堂課 ／ DevOps特別版。在這場workshop中，葉秉哲 (William Yeh)，從入門到練習都幫我們準備好了。</p><p>課程內容包含下面：</p><ul><li>課前作業</li><li>第一堂／因果分析</li><li>第二堂／系統動力學</li><li>第三堂／系統基模</li><li>第四堂／應用：DevOps 與商業模式分析</li></ul><p>第一堂課，就讓我們獲益良多，從一篇文章討論，並推導什麼是思維模型，為什麼重要，再解釋系統是什麼，陸續延伸到 William 自己研發的SLR-CPI方法。後續的三堂課就是理論和練習交替，讓我們更熟習系統思考的使用。最後的二十到三十分鐘，把課前作業的康威定律和 The DevOps Handbook 第七章的內容拿來出來分析和討論。</p><p>William 講義後的推薦書單也是很實用，我打算從思維模型的部分開始來進行增強，包含了<a href="https://www.books.com.tw/products/0010643983">窮查理的普通知識</a>，<a href="https://www.books.com.tw/products/0010457479">培養商業腦的7種組織力</a>，<a href="https://search.books.com.tw/redirect/move/key/%E5%B7%A6%E6%80%9D%E5%8F%B3%E6%83%B3/area/mid/item/0010721292/page/1/idx/1/cat/001/pdf/1">左思右想</a>，和<a href="https://www.books.com.tw/products/0010762830">一流的人讀書，都在哪裡畫線</a>。</p><p>唯一的遺憾就是真的時間不夠用，太多內容想要練習和討論了。想要短時間就能靈活運用系統基模也是不太容易的。要累積更多思維模型，還是得多多刻意練習。這個workshop推薦對系統思考有興趣的朋友參加。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=becdfe4a84f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build Your MQTT Broker in CloudMQTT]]></title>
            <link>https://medium.com/@suci/build-your-mqtt-broker-in-cloudmqtt-f95ba04270c3?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/f95ba04270c3</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[mqtt]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Sun, 22 Apr 2018 15:55:34 GMT</pubDate>
            <atom:updated>2018-04-22T17:23:10.257Z</atom:updated>
            <content:encoded><![CDATA[<p>Ihave been working with IoT and data applications based on machine-to-machine communication for years. Message Queuing Telemetry Transport (MQTT) is one of popular and mature protocols in efficient transport for two-way messaging between “things” and services.</p><p>There are many ways to implement MQTT message broker and clients. Most famous is <a href="https://mosquitto.org/"><strong>mosquitto</strong></a> project. However, sometimes, I just need a simple MQTT broker to focus on testing with MQTT pub/sub messaging. Similar with my previous <a href="https://medium.com/@suci/connect-apache-kafka-instances-in-cloudkarafka-14e0dc85e734">article</a> for Apache Kafka, I just Goolge it and found <a href="https://www.cloudmqtt.com/"><strong>CloudMQTT</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*qFKZ2B9ARcdy6O4AUn5Ygg.png" /><figcaption><strong>Managed Mosquitto servers in the cloud</strong></figcaption></figure><p>CloudMQTT is a globally distributed MQTT broker, designed for “Internet of Things” messaging between low power sensors or mobile devices. Just like CloudKafka, there are different animal plans (paid plans) to create your own message brokers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FyNZoFjwVY-4ZMMK8YwdRg.png" /><figcaption>Plan and pricing: your own broker on a dedicated server.</figcaption></figure><p>For simple testing, I chose “Cute Cat” (FREE). I will show how to apply this free plan and connect by a python client modified by <a href="http://(paho.mqtt.python)/">a sample app</a> from CloudMQTT. <strong>This example will only simply subscribes and publish message to with a topic via CloudMQTT with Python client</strong>.</p><h3>PRE-REQUISITES</h3><ul><li>Basic *nix OS skills</li><li>Basic knowledge on any programming language (prefer Python)</li><li>Experiences on MQTT or sub/pub services may be also helpful.</li></ul><h3>1.Login/Register</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xv204SGCeYYogQl_SKM_2w.png" /><figcaption><a href="https://www.cloudmqtt.com/">https://www.cloudmqtt.com/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zz-zY8tc6zq_HRcCx0WaFA.png" /><figcaption><a href="https://customer.cloudmqtt.com/login">https://customer.cloudmqtt.com/login</a></figcaption></figure><h3>2. Create new instance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dlHq-1ejBumFhHf7HllzAQ.png" /><figcaption>Create a new instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HEy0JThoudyKBI8ptUbTaQ.png" /><figcaption>instance name and plan</figcaption></figure><p>Two regions of AWS data center (US-East-1 and EU-West-1) areoffered now in free plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jt1lctbA47ZSCWT7wpjzbg.png" /><figcaption>List all instances</figcaption></figure><h3>3. Details of the instance</h3><p>CloudMQTT provides dashboard to manange brokers (more screen shots in the bottom of this article) . However, in free plan, there are only few features we can access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zIVUITz0stbX_FZuY6xfIg.png" /><figcaption>Details</figcaption></figure><h3>4. Connect the broker by python client</h3><p>The official documentation has provided <a href="http://(paho.mqtt.python)/">a sample app</a> to show how to connect to CloudMQTT broker by python and many other interfaces.</p><p>However, there is only a small part which may make beginners feel confused about<strong> environment variables </strong>for<strong> connection information. </strong>You can get those information in the instance “Details” page.</p><p>I slightly modified the sample code and upload in <a href="https://github.com/sucitw/python_snippets/tree/master/mqtt">github</a>. Download the source code. You just need to fill the connection information in the <strong>config.ini </strong>(rename from config.tmp). The sample app will subscribe a topic in the config.ini and just publish ONE message. (You may need to install <a href="https://github.com/eclipse/paho.mqtt.python">client library</a>, paho-mqtt, to run this python script.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*NcJhDIXwvpf0uBb5RpBjeA.png" /><figcaption>config.ini</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frbOG01iOsUVFuyQMKKG7Q.png" /></figure><p>You can also use Websocket UI to see the message content and send a message.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H0WeZ6dFf38h4YVyIefxDw.png" /></figure><p>Ref:</p><ol><li><a href="https://www.cloudmqtt.com/docs-python.html">Python client for CloudMQTT</a></li><li><a href="https://github.com/eclipse/paho.mqtt.python">MQTT python client library (paho.mqtt.python</a>)</li></ol><p>如果你想親手測試MQTT subscribe 和 publish message的功能，但是不想（不會）建立自己的MQTT Broker，可以試看看 CloudMQTT 。CloudMQTT提供了不同付費方案提供使用者建立MQTT broker instances。</p><p>在此我介紹如何申請和透過Python client (Eclipse Paho™ MQTT Python Client) 來連結這個免費共享的雲端MQTT broker。其中特別說明，如何透過設定config.ini 來取代環境變數，以取得CloudMQTT 提供的連線資訊，連結到 brokers的方式。這個簡單的MQTT 客戶端程式，只會訂閱所設定的主題(topic)，和發布一封訊息做為測試。請進一步參考本文和<a href="https://github.com/sucitw/python_snippets/tree/master/mqtt">sample code</a>。</p><h3>Screenshots for the dashboard of CloudMQTT</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-s5hZNM6jB2FoOehR-qufw.png" /><figcaption>users</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HZxJ6JLqViU450PsaspCmw.png" /><figcaption>Bridges</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0vigQluvRh0mcnO5-4VbA.png" /><figcaption>Connect to AWS Kinesis</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sBMMErZaUYGdSWD2TdPeYg.png" /><figcaption>Server log</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ul6KK5mvltn7jkwZAmTp5w.png" /><figcaption>Connections</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f95ba04270c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hello-World in Apache NiFi]]></title>
            <link>https://medium.com/@suci/hello-world-nifi-dcafcba0fdb0?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcafcba0fdb0</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[nifi]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Sat, 25 Nov 2017 04:14:26 GMT</pubDate>
            <atom:updated>2020-06-27T23:00:44.355Z</atom:updated>
            <content:encoded><![CDATA[<p>I am using Apache <strong>NiFi </strong>Processors to ingest data from various purposes. NiFi allows users to collect and process data by using <strong>flow based programming</strong> in Web UI. Ingesting data via Nifi is very convenient. The latest version is 1.4.</p><p>I will introduce how to enable NiFi via Docker and Homebrew in <strong>Mac</strong> and a Hello-World sample to run NiFi.</p><h3>PRE-REQUISITES</h3><ul><li>Successfully running Docker in your machine (for running NiFi in docker)</li><li>(Better to have) Basic concepts of NiFi (data flow, processor, and connection. See more in <a href="https://nifi.apache.org/docs/nifi-docs/html/user-guide.html">user guide of Apache NiFi)</a></li></ul><h3>Getting Started</h3><h4>1. Run NiFi</h4><p>You can run NiFi via a.) Docker, b.) installing it in Mac or c.) installing it in Windows</p><p><strong>a.) Run a NiFi container</strong></p><p>I usually use docker to run everything in my Mac. If you can run docker well in your local machine, you may use the following command to download and run NiFi 1.4 or latest version.</p><pre>docker run -d -P -h nifi -p 38080:8080 -p 38181:8181 --name nifi_1.4.0 --memory=4g -v /docker/apache/nifi apache/nifi:1.4.0</pre><pre># latest version<br>docker run -d -P -h nifi -p 38080:8080 -p 38181:8181 --name nifi_latest --memory=4g -v /docker/apache/nifi apache/nifi:latest</pre><p><strong>b.) Download, install and run NiFi in Mac</strong></p><p>brew install nifi</p><p>Just follow <a href="https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#downloading-and-installing-nifi">the official doc</a> from Apache Nifi</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBDwBu-AnfhCvuUQZd4DIw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8fWQuuiQaFmHKkbrhmQhKQ.png" /></figure><p>Find the folder where NiFi was installed. Ex. /usr/local/Cellar/nifi/1.4.0</p><p>bin/nifi.sh run, run in the foreground,</p><p>bin/nifi.sh start, run in the background</p><p>bin/nifi.sh status, check the status</p><p>bin/nifi.sh stop, stop NiFi</p><p><strong>c.) Download, install and run NiFi in Windows</strong></p><ol><li>Download from <a href="https://nifi.apache.org/download.html">Nifi official website </a>.</li><li>Choose nifi-x.y.z-bin.zip (ex, <a href="https://www.apache.org/dyn/closer.lua?path=/nifi/1.4.0/nifi-1.4.0-bin.zip">nifi-1.4.0-bin.zip</a>).</li><li>Extract to a specific folder (ex. c:\Users\username\nifi).</li><li>Find and execute <strong>run-nifi.bat</strong> in “your nifi folder\bin\”</li></ol><h4>2. Verify if NiFi is running</h4><p>Run Nifi instance locally, then connect NiFi by <a href="http://localhost:38080/nifi">http://localhost:38080/nifi </a>(or <a href="http://localhost:38080/nifi">http://localhost:8080/nifi</a> if you run in Mac or Windows) to see NiFi user interface in the browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Udq_hgep9GPdM7yg17tkJQ.png" /></figure><h4><strong>3.Building your first Apache Nifi dataflow</strong></h4><p>I try to introduce the simplest use case as an example here. <strong>(Warning: many screen-shot pictures! You may zoom out your browser.)</strong></p><p>Only two processors are used in this dataflow:</p><ul><li><strong>GenerateFlowFile</strong> as input and <strong>PutFile</strong> as output</li></ul><p>GenerateFlowFlies will generate random texts as flowfiles. PutFile will write the content from GenerateFlowFiles into the disk as many individual files.</p><p><strong>a). ​Add a Processor</strong></p><p>To start, drag the processor icon near NiFi logo in the menu bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xThWNDhJ1dwnqiVYPduH5w.png" /><figcaption>drag the processor icon</figcaption></figure><p>Search in Filter, choose <strong>GenerateFlowFile </strong>to add to NiFi canvas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*0z3k2IIAVtun6lSFDXW_TQ.gif" /></figure><p>Configure the settings according to the Warning message and the behavior you expect. The detail configurations of processors are not introduced here. Basically, you can find the detail usage when you right click at processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*J0fy3v6C6YD2RtLz9fZ6Qg.gif" /></figure><p>This processor will generate FlowFiles with random data.</p><p>Then, add another processor:<strong>PutFile. </strong>Need to add corresponding values of configuration for PutFile.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q9lVuSi8jC0WWTVmdXocKA.gif" /></figure><p><strong>b). Connect processor</strong></p><p>Drag from a processor to connect to another processor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*zHlP4dG-J5qB8oBQmn_Vdw.png" /></figure><p><strong>c). Run two processors as your first NiFi data flow</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oo1tSc2Bj8ILQ7sQTHp4QA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*zsaadXzBcIglWcKvAXC_Cw.png" /></figure><p><strong>d) Verify the data flow</strong></p><p>I ran NiFi in docker. So I entered the NiFi docker container to check if data were written into /tmp/ folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFz5V01Jm-kLMcOTmXP-7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DZ7NG-WgXPxRc-rPL9rYFQ.png" /></figure><p>Then, stop processors to not continue writing files.</p><h3>Conclusion</h3><p>I introduced a VERY simple example to show <strong>what is NiFi </strong>with an input processor and an output processor, <strong>GenerateFlowFile</strong> and <strong>PutFile</strong>. The basic steps are adding appropriate processors, connecting them, then run it.</p><p>Apache NiFi is a powerful tool with scalable directed graphs for pulling data from external sources; routing, transforming, and aggregating it; and finally delivering it to its final destinations.</p><p>More NiFi usages and use cases will be explored as it keep growing. However, as many data ingestion pipelines tasks, you still should firstly design a reasonable and appropriate ingestion logic based on your purpose and data infrastructure environment.</p><p>本文分享了當❶如何透過docker，以及在Mac/Windows 建立NiFi 使用環境❷透過一個簡單的dataflow介紹NiFi的基本概念．</p><p>主要步驟為：</p><ol><li>選用適合的Processor (在此為 <strong>GenerateFlowFile</strong> and <strong>PutFile)，GenerateFlowFile</strong> 隨機產生文字，以flowfile的形式傳給<strong>PutFile，PutFile </strong>將這些內容寫進指定的資料夾中。</li><li>設定這兩個 processors</li><li>連接這兩個 processors</li><li>執行並驗證這一個dataflow</li></ol><p>值得注意的是，在使用NiFi 之前，應該預先思考並準備好相關資訊（data source, target and etc.），設計處理data 每個步驟和邏輯，接著篩選適合的processor來完成data ingestion的工作。</p><p>本方法是以一個簡單的NiFi dataflow的範例，包含input 和output，提供簡單的測試，請參考延伸參考文件來建置更進階的NiF環境和設計進階的data 處理邏輯。</p><p>Ref:</p><ol><li><a href="https://nifi.apache.org/docs/nifi-docs/html/getting-started.html">Getting-started with NiFi</a> from official docs of Apache NiFi</li><li>Apache NiFi How to Build a Flow — <a href="https://www.youtube.com/watch?v=bzqP-Loyq-Y">Part 1</a> and <a href="https://www.youtube.com/watch?v=YVMUnfjMGVk">Part2</a> (youtube)</li><li><a href="https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html">nifi in depth</a></li><li><a href="http://www.cnblogs.com/neillee/p/6841320.html">『NiFi 学习之路』入门</a> (Chinese)</li><li><a href="https://medium.com/@yjhyjhyjh0/apache-nifi-overview-90294747dbed">Apache NiFi Overview</a> and <a href="https://medium.com/@yjhyjhyjh0/apache-nifi-user-guide-6fc2d35fbb4f">Apache NiFi User Guide</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcafcba0fdb0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running PySpark on Jupyter Notebook with Docker]]></title>
            <link>https://medium.com/@suci/running-pyspark-on-jupyter-notebook-with-docker-602b18ac4494?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/602b18ac4494</guid>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Tue, 12 Sep 2017 11:50:46 GMT</pubDate>
            <atom:updated>2020-09-13T10:30:21.322Z</atom:updated>
            <content:encoded><![CDATA[<pre>2020/09/13 add a docker command with volumes function</pre><p>It is much much easier to run <a href="https://spark.apache.org/docs/latest/api/python/"><strong>PySpark</strong></a> with docker now, especially using an image from <a href="http://jupyter/docker-stacks">the repository of Jupyter</a>.</p><p>When you just want to try or learn Python. it is very convenient to use <a href="https://jupyter.org/">Jupyter Notebook</a> for an interactive developing environment. The same reason makes me want to run Spark through PySpark in Jupyter Notenook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*eb-jKl6gEteYlv4SKojx-g.png" /><figcaption>Spark + Python + Jupyter Notebook + Docker</figcaption></figure><p>In this article (Yes, another one <strong>“Running xxx on/with Docker”</strong>), I will introduce you <strong>how to create an environment to run PySpark on Jupyter notebook easily</strong> — Read on!</p><h3>PRE-REQUISITES</h3><ul><li>Successfully running Docker in your machine</li><li>Basic knowledge for Jupyter notebook and Docker</li><li>Read README.md in <a href="https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook"><strong>jupyter/docker-stacks/pyspark-notebook/</strong></a></li></ul><h3>Getting Started</h3><h4>1.Run a container</h4><pre>docker run -it --rm -p 8888:8888 jupyter/pyspark-notebook</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z0LFKBOuJcvGKgl9x0Jluw.png" /><figcaption>Run a container to start a Jypyter notebook server</figcaption></figure><p>You can also use <strong><em>-v</em></strong> to persist data generated in notebook of Docker container. I mounted my local host folder with the default folder of notebook in container is <strong><em>“/home/jovyan/work”</em></strong>.</p><p>“/Users/shuhsi/github” &lt;-&gt;“/home/jovyan/work”</p><pre>docker run -it --rm -p 8888:8888 -v /Users/shuhsi/github:/home/jovyan/work jupyter/pyspark-notebook</pre><h4>2. Connect to a Jupyter notebook</h4><pre><a href="http://localhost:8888/?token=e144d004f6652ae6406a78adf894621e62fdeb1fc57d02e8"><em># </em></a><em>Copy/paste this URL into your browser (if the first)</em><br><a href="http://localhost:8888/?token=e144d004f6652ae6406a78adf894621e62fdeb1fc57d02e8">http://localhost:8888/?token=e144d004f6652ae6406a78adf894621e62fdeb1fc57d02e8</a></pre><h4>3.Try to run a sample code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uo-kW0EuONI5-MnOuu6yUg.png" /><figcaption>Run sample code</figcaption></figure><pre>import pyspark <br>sc = pyspark.SparkContext(&#39;local[*]&#39;)</pre><pre># do something to prove it works<br>rdd = sc.parallelize(range(1000))<br>rdd.takeSample(False, 5)</pre><h3>Conclusion</h3><p>As the above shown, it is VERY easy to create an environment to run PySpark on Jupyter notebook by the following steps:</p><ol><li>Check PRE-REQUISITES firstly, especially the ability to run docker.</li><li>Excuse “docker run” command to pull the image from docker pull <strong>jupyter/pyspark-notebook </strong>and<strong> </strong>run a container to start Jupyter notebook and Spark environment.</li><li>Connect to the Jupyter notebook server and run sample code to verify the environment.</li></ol><p>However, this environment is just to provide a Spark local mode to test some simple spark code. If you need cluster mode, you may check the reference article for more advanced ways to run Spark.</p><p>本文分享了當❶你想要用PySpark跑Spark job ❷且想在Jupyter notebook 環境進行調試時，可以採取的安裝與設定步驟來建置該環境。</p><p>主要步驟為：</p><ol><li>先確認該主機可以run docker</li><li>Run “Docker run” 的指令和參數去pull image 且啟動container &quot;docker run -it --rm -p 8888:8888 jupyter/pyspark-notebook&quot;</li><li>連接到該 Jupyter notebook server 並測試 PySpark sample code 來確認是否安裝與設定順利</li></ol><p>要注意，本方法是提供Spark local mode 提供簡單的測試，可以參考延伸參考文件來建置更進階的Spark cluster環境。</p><h3>Reference reading:</h3><ol><li><a href="https://spark.apache.org/docs/latest/running-on-mesos.html">https://spark.apache.org/docs/latest/running-on-mesos.html</a></li><li><a href="https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook">Jupyter/docker-stacks/pyspark-notebook</a></li></ol><p><a href="https://github.com/jupyter/docker-stacks">jupyter/docker-stacks</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=602b18ac4494" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Docker on Windows 7 behind a proxy]]></title>
            <link>https://medium.com/@suci/running-docker-on-windows-7-behind-a-proxy-b363d3966d7?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/b363d3966d7</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[proxy]]></category>
            <category><![CDATA[windows-7]]></category>
            <category><![CDATA[git-bash]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Tue, 08 Aug 2017 10:18:50 GMT</pubDate>
            <atom:updated>2018-09-24T08:19:44.617Z</atom:updated>
            <content:encoded><![CDATA[<p>Running docker on Windows and Mac now is much easier than few years ago. “<a href="https://docs.docker.com/docker-for-windows/install/#download-docker-for-windows"><strong>Docker for Windows</strong></a>”, one of <a href="https://www.docker.com/community-edition">Docker CE </a>app, will enable Hyper-V and has supported Windows 10 and Windows Server 2016.</p><p>However, for some reasons, your system may not meet the requirements to run ”Docker for Windows“. You will need use <a href="https://docs.docker.com/toolbox/overview/"><strong>Docker Toolbox</strong></a> to install and run docker. And for some reasons, you are working in a <em>restrictive network environment with</em> <strong>proxy server</strong>s. How should we deal with this condition?</p><p>I am going to share how I run Docker on <strong>Windows 7</strong> <strong>behind a proxy</strong>. Yes, two points in this article: ❶ Docker on Win 7 and ❷ behind a proxy.</p><h4><strong>PRE-REQUISITES</strong></h4><ul><li>Older <strong>64 bit</strong> Windows systems that do not meet the requirements of <a href="https://docs.docker.com/docker-for-windows/">Docker for Windows</a>. There are some different procedures between Windows 7 and 8, especially on enabling <strong>virtualization</strong>. I use <strong>windows 7</strong> in this article.</li><li>Confirm virtualization is enabled on your machine. For Windows 8, you can just check <strong>Performance</strong> tab from <strong>Task Manager</strong>. For Windows 7, you can run the <a href="http://www.microsoft.com/en-us/download/details.aspx?id=592">Microsoft® Hardware-Assisted Virtualization Detection Tool</a> and follow the on-screen instructions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*m-Szzops9IN5Ync26p0s0w.png" /></figure><p>This is very easy. However, if you didn’t enable hardware virtualization, you may need <strong>enable VT-x or AMD-v </strong>from <strong>BIOS</strong> settings.</p><h3><strong>Install Docker</strong></h3><h4>Download and install <a href="https://docs.docker.com/toolbox/toolbox_install_windows/">Docker Toolbox</a> on Windows</h4><p>The installer will install the following software:</p><ul><li>Docker Client for Windows</li><li>Docker Toolbox management tool and ISO</li><li>Oracle VM VirtualBox</li><li>Git MSYS-git UNIX tools</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*3LJLQKYEQwkIYHkLwGOr6g.png" /></figure><h3>Setup Docker</h3><h4>Start docker toolbox terminal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*CwyA7aEZXPe_zkhTvzQC1w.png" /></figure><p>Launch a Docker Toolbox terminal by clicking “Docker Quickstart Terminal” icon. A shell script will check VirtualBox setting and create a new VM for docker by git-bash. Then Docker-machine can leverages VirtualBox’s VBoxManage to initialise, start, stop and delete the VM by <a href="https://github.com/boot2docker/boot2docker"><strong>Boot2Docker</strong></a>.</p><h4><strong>Setup a Proxy in git-bash</strong></h4><p>However, you may still need to setup a <strong>proxy in git-bash</strong> to get Boot2Docker iso file to create a VM for Docker-machine. Otherwise, you may see the error message as the below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WLSOsrG-VguxK7iYZ1n0GA.png" /></figure><p>To setup and save the environment variable, you need to edit a “<strong>.bash_profile</strong>” for git-bash. Usually, the file in located in your Git home folder. Add the following code in your .bash_profile file. For NO_PROXY, you may also add ip address of docker-machine.</p><pre>export HTTP_PROXY=<em>http://account:password@proxy_Domain:port</em><br>export HTTPS_PROXY=<em>https://account:password@proxy_Domain:port</em><br>export NO_PROXY=<em>localhost,127.0.0.1,$docker-machine_ip</em></pre><p>Restart “Docker Quickstart Terminal”, then it should look like the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZYHi_H0P84I8whCQ2CPKkA.png" /></figure><p>Now Docker is up and running!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nFzEOXTWK0SLoDyX35TI1Q.png" /></figure><h4>Error when connect to docker machine</h4><p>If the IP address of the default docker machine was <strong>not</strong> added into the NO_PROXY variable list, you may occur the follow error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nBACgyAAzcQsRMl9RuQycg.png" /></figure><p>Check ip of docker-machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EAfa4FpcsEfK744jhyitIQ.png" /><figcaption>check docker-machine information</figcaption></figure><p>Add ip of docker-machine by editing “.bash_profile”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*SH4uHpxFFVbGOja5M53_2w.png" /><figcaption>Modified .bash_profile</figcaption></figure><h3>Verify your installation</h3><p>Type the following command to pull a image and run a container for “hello-world”</p><pre>$ docker run hello-world</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*t6TqFxCpsA7QUZqtqK45Vg.png" /></figure><h4>Another way to setup a proxy by connecting the default docker-machine</h4><p>This above describes how to add a proxy settings in git-bash and initiate a docker terminal via git-bash. We can also setup the proxy in a docker-machine by the following commands.</p><pre>#connect docker-machine<br>$ docker-machine ssh default<br>$ sudo -s</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qxnApZOgzYsfI1eONezM6Q.png" /></figure><p>Write a proxy to the configuration profile by the followings commands:</p><pre>$ echo &quot;export HTTP_PROXY=http://account:password@proxy_Domain:port&quot; &gt;&gt; /var/lib/boot2docker/profile</pre><pre>$ echo <br>&quot;export  HTTPS_PROXY=https://account:password@proxy_Domain:port&quot; &gt;&gt; /var/lib/boot2docker/profile</pre><pre># See the content of profile to confirm the configuration<br>$ cat /var/lib/boot2docker/profile</pre><pre>#Restart docker machine<br>$ docker-machine restart default</pre><h3><strong>Conclusion</strong></h3><p>Install Docker on win 7 with a proxy by the following steps:</p><ol><li>Install docker by <a href="https://docs.docker.com/toolbox/toolbox_install_windows/">Docker Toolbox</a></li><li>Setup a proxy by adding settings ❶ in .bash_profile of git-bash or ❷ in default VM of docker-machine</li><li>Try “hello-world” to verify the installation and settings</li></ol><p>本文分享了當❶有需要在舊有的windows （win 7 or 8）且❷需要透過proxy連到外網的環境下使用docker時，所採取的安裝與設定步驟。</p><p>主要步驟為：</p><ol><li>透過 <a href="https://docs.docker.com/toolbox/toolbox_install_windows/">Docker Toolbox</a> 安裝docker</li><li>提供兩種方式增加proxy設定，❶ 可在git-bash 的.bash_profile 或是❷ 可透過ssh 連到default docker-machine 做環境變數修改</li><li>測試 “hello-world” 來確認是否安裝與設定順利</li></ol><h4>Reference reading:</h4><ol><li><a href="https://docs.docker.com/toolbox/toolbox_install_windows/">Docker Toolbox</a></li><li><a href="https://github.com/boot2docker/boot2docker">boot2docker</a></li><li><a href="https://stackoverflow.com/questions/34169721/set-an-environment-variable-in-git-bash">https://stackoverflow.com/questions/34169721/set-an-environment-variable-in-git-bash</a></li><li><a href="http://www.netinstructions.com/how-to-install-docker-on-windows-behind-a-proxy/">http://www.netinstructions.com/how-to-install-docker-on-windows-behind-a-proxy/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b363d3966d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Connect Apache Kafka instances in CloudKarafka]]></title>
            <link>https://medium.com/@suci/connect-apache-kafka-instances-in-cloudkarafka-14e0dc85e734?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/14e0dc85e734</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ssl-certificate]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Tue, 27 Jun 2017 18:35:56 GMT</pubDate>
            <atom:updated>2017-06-27T18:43:47.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/239/1*Dq5FCjg6BzeLpsmIdaN74w.png" /><figcaption>Message streaming as a Service</figcaption></figure><p>When I was preparing my <a href="https://tw.pycon.org/2017/en-us/events/talk/323492357779488854/">talk</a> in <a href="https://tw.pycon.org/2017/en-us/">Pycon tw 2017</a>, I tried different ways to build my own Kafka broker, including my previous post (<a href="https://medium.com/@suci/running-apache-kafka-on-docker-c312ebd70413">Running Apache kafka on docker</a>). Then, I found <a href="https://www.cloudkarafka.com/">CloudKarafka</a>.</p><p>Cloudkarafka is a streaming platform in the public cloud, designed for Apache Kafka workloads. There are different animal plans (paid plans) to create your own Kafka instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YobdYy7D94qHSTStMB27Q.png" /><figcaption>Dedicated Instances</figcaption></figure><p>If you just want to try Kafka basic producer and consumer API (like me), you can choose DEVELOP DUCK (FREE PLAN). I will show how to apply this Free instance and connect by the python client (Kafka-python). You can also check the <a href="https://www.cloudkarafka.com/docs.html">documentation</a> from their website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*RTcAN9TkrOOp3yTIyeOvgA.png" /><figcaption>Free Shared Plan</figcaption></figure><h3>1.Login/Register</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Cg7xe-ZGpI7k-ZQNgRmNQ.png" /><figcaption><a href="https://www.cloudkarafka.com/">https://www.cloudkarafka.com/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*znZL0GFhqNh1mHRmBTvKEg.png" /><figcaption>login/sigup</figcaption></figure><h3>2. Create new instance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYP7adA0bN1zIJDjunc7OQ.png" /><figcaption>new instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nwsou3ySbMHNvRSU_90oQ.png" /><figcaption>instance name and plan</figcaption></figure><p>Only one region of AWS data center (US-East-1) is offered now in free plan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UUiNMT0g4MiCYIHyNXnaPw.png" /></figure><h3>3. Details of the instance</h3><p>Cloudkarafka provides dashboard to monitor brokers and management (more screen shots in the bottom of this article) . However, in free plan, there are only few features we can access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yc0OJcUEELY0_8l6erL83g.png" /><figcaption>Details</figcaption></figure><h3>4. Connect the broker by python client</h3><p>The official documentation has showed how to connect to CloudKarafka broker by kafka-python. (<a href="https://www.cloudkarafka.com/docs-python.html">https://www.cloudkarafka.com/docs-python.html</a>)</p><p>However, there is only a small part which may make users feel confused- <strong>certification key.</strong></p><p><strong>Where you can get certification files to connect brokers?</strong></p><p>The sample code from Cloudkarafka use <strong>many environment variables (shown as the below) </strong>to setup values of client certification ( for a <a href="https://docs.python.org/3/library/ssl.html#ssl.SSLContext"><strong>ssl_context</strong></a><strong> </strong>object) and broker.</p><pre><strong>&#39;CLOUDKARAFKA_CA&#39;</strong>: <br>value for <em>cadata</em> object, usually an ASCII string of one or more PEM-encoded certificates or a <a href="https://docs.python.org/3/glossary.html#term-bytes-like-object">bytes-like object</a> of DER-encoded certificates.</pre><pre><strong>&#39;CLOUDKARAFKA_CERT&#39;</strong>: <br><em>certfile, </em>the path to a single file in <strong>PEM</strong> <strong>format</strong> containing the certificate as well as any number of CA certificates needed to establish the certificate&#39;s authenticity.</pre><pre><strong>&#39;CLOUDKARAFKA_PRIVATE_KEY&#39;</strong>: <br><em>keyfile,</em>the path which point to a file containing the private key in.</pre><pre><strong>&#39;CLOUDKARAFKA_BROKERS&#39;</strong>: broker address</pre><pre><strong>&#39;CLOUDKARAFKA_TOPIC_PREFIX&#39;</strong>: <br>prefix for topic #this is necessary unless you create a new topics</pre><p>It is a reasonable and proper way to access configuration variables. However, if you are not familiar with writing configuration into your <strong>client environment variables </strong>or don’t want to follow this way by some reasons, I provide another way to setup configuration and connect Cloudkarafka brokers.</p><p>Firstly, you need to download a certificates file (.env file). If you open it with text editor, you will see the configuration values for CLOUDKARAFKA_CA, CLOUDKARAFKA_CERT, CLOUDKARAFKA_PRIVATE_KEY, and broker address. All you need are in this file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHWJidHwlCUgnwnHOAfL8w.png" /></figure><p>I saved those certificate information in three files, then created settings.py to read them when I need to connect brokers.</p><pre>CLOUDKARAFKA_CA: “~/.ssh/cloud_kafka/ca_cert.pem”<br>CLOUDKARAFKA_CERT: “~/.ssh/cloud_kafka/signed_cert.pem” <br>CLOUDKARAFKA_PRIVATE_KEY: “~/.ssh/cloud_kafka/private_key.pem”</pre><p>Please check my <a href="https://github.com/sucitw/benchmark-python-client-for-kafka/tree/master/client/kafka-python/cloudkarafka">sample client code</a>.</p><p>Ref:</p><p>1. <a href="https://www.cloudkarafka.com/docs-python.html">Python client for Cloudkarafka</a><br>2. <a href="https://github.com/dpkp/kafka-python">kafka-python</a>]maintained by Parse.ly</p><p>如果你想親手測試Apache Kafka Producer和 Consumer API的使用方式，但是不想（不會）建立自己的Kafka Broker， CloudKarafka 是一個好選擇。CloudKarafka 提供了不同付費方案提供使用者建立Kafka instances。</p><p>在此我介紹如何申請和透過Python client (Kafka-python) 來連結這個免費共享的雲端Kafka broker。其中特別說明，如何不透過設定環境變數來取得cloudkarafka 提供的SSL 憑證內容，以連結到 brokers的方式。請參考本文和<a href="https://github.com/sucitw/benchmark-python-client-for-kafka/tree/master/client/kafka-python/cloudkarafka">sample code</a>。</p><h3>Screenshots for the dashboard of Cloudkarafka</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMWBo1d5NFrsa1CIJq8sGw.png" /><figcaption>Topics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q5MOPv0MQnIkZrHkJVyucQ.png" /><figcaption>Nodes information</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i3ZzU5TX6Igvxu3HgRX8hw.png" /><figcaption>Server metrics</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyzBRlbHIMz1LIcv6absdg.png" /><figcaption>Logs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vtLWvjvyIQYKiZO8PXa-tQ.png" /><figcaption>Certificate</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*Wa-dEtbP-OUBe-01HxNPNQ.png" /><figcaption>Management</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdI7yhW1AQYNZvvnfliVsQ.png" /><figcaption>New management dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fl_MgjrW86dmFOUSDj6MBA.png" /><figcaption>Integration with other services</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OPZ9hUUYcoickRGMsvhUTg.png" /><figcaption>VPC</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14e0dc85e734" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Apache Kafka on Docker]]></title>
            <link>https://medium.com/@suci/running-apache-kafka-on-docker-c312ebd70413?source=rss-c069d7b66f91------2</link>
            <guid isPermaLink="false">https://medium.com/p/c312ebd70413</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Suci Lin]]></dc:creator>
            <pubDate>Thu, 01 Jun 2017 07:21:09 GMT</pubDate>
            <atom:updated>2017-09-25T03:16:28.485Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*G35Pogo3vM4K-66s7KNhnw.jpeg" /></figure><p>I am going to give a talk in pycontw 2017 about Apache Kafka. In <a href="https://tw.pycon.org/2017/en-us/events/talk/323492357779488854/">the talk</a>, I plan to arrange a live demonstration on my MacBook Pro. However, the environment is always the first item on my to-do list.</p><p>There are many ways to install and run Apache Kafka. It is getting more popular to deploy applications and application infrastructure by Docker. I’d like to share how I use docker to run Apache Kafka via Docker.</p><p>OUTLINE</p><ul><li><strong>Overview</strong></li><li><strong>Pre-requisites</strong></li><li><strong>Two docker compose YML files</strong></li><li><strong>Running Apache Kafka on Docker</strong></li><li><strong>Verify whether Apache Kafka started successfully</strong></li></ul><h3><strong>Overview</strong></h3><p>Usually, we need two containers at least : one for Kafka, the other for Zookeeper. So far (2017/05/26), there is no official image for Kafka yet in Docker Hub. I used two different Apache Kafka images as my test environments.</p><h3>Pre-requisites</h3><ul><li><strong>Docker</strong> (Docker machine, docker compose)</li><li>(opt) <strong>Python</strong> for verifying Kafka broker</li></ul><h3><strong>Two docker compose YML files</strong></h3><p>The docker compose YML files can be found in <a href="https://github.com/sucitw/benchmark-python-client-for-kafka/blob/master/docker-compose_ap-kafka.yml">docker-compose_ap-kafka.yml</a> and <a href="https://github.com/sucitw/benchmark-python-client-for-kafka/blob/master/docker-compose_cp-kafka.yml">docker-compose_cp-kafka.yml</a>.</p><p>There are several variables should be reviewed and modified based on the environment.</p><p>❶ <strong>Images source</strong>: wurstmeister/zookeeper and wurstmeister/kafka:0.10.1.0–1; confluent/zookeeper and confluent/kafka:0.10.0.0-cp1</p><p>❷ <strong>Environment variables:</strong></p><p><strong>KAFKA_ADVERTISED_HOST_NAME :</strong> the hostname/ip</p><p><strong>KAFKA_ZOOKEEPER_CONNECT:</strong> URL to Zookeeper</p><p>❸ <strong>Ports:</strong> 2181 for Zookeeper ; 9092 for Kafka</p><h3><strong>Running Apache Kafka on Docker</strong></h3><h4><strong>Create a docker-machine for kafka image from wurstmeister</strong></h4><p>After modifying the compose yml file ( host ip), please type the follow command to build docker containers for one node Apache Kafka.</p><pre>docker-machine create —-driver virtualbox —-virtualbox-memory 6000 ap-kafka</pre><pre>eval $(docker-machine env ap-kafka)</pre><pre>docker-compose -f docker-compose_ap-kafka.yml up -d</pre><p><strong>Verify whether it started successfully</strong></p><p>Now a Kafka broker is up and running, we should verify whether it started successfully. Although there are few third-party apps for monitoring and managing Kafka, we can simply use the following two ways to check it.</p><ul><li>via Apache Kafka shell script (from <a href="https://kafka.apache.org/downloads">the official website</a> )</li><li>via <a href="https://github.com/sucitw/benchmark-python-client-for-kafka">benchmark.py</a> (need to install python and necessary libraries)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7mkcTpniWi7_CAiPZW_rTA.png" /><figcaption>benchmark.py for testing performance of different python client libraries for Apache Kafka</figcaption></figure><h4><strong>Create a docker-machine for confluent kafka</strong></h4><p>The procedure for confluent kafka image is similar with the above.</p><pre>docker-machine create —-driver virtualbox —-virtualbox-memory 6000 cp-kafka</pre><pre>eval <strong>$(</strong>docker-machine env confluent<strong>)</strong></pre><pre>docker-compose -f docker-compose_cp-kafka.yml up -d</pre><p>This small scale of Apache Kafka would be useful for Kafka API and client testing.</p><p>本文介紹了我如何在Mac上透過Docker compose 建立Apache Kafka 測試環境，依照docker images 的來源，有兩個獨立的Docker compose YML可以用來建立single node 的 Kafka Broker。</p><p>Ref:</p><ol><li><a href="https://kafka.apache.org/">Apache Kafka official website</a></li><li><a href="https://github.com/sucitw/benchmark-python-client-for-kafka">Python kafka benchmark</a></li><li><a href="https://hub.docker.com/r/wurstmeister/kafka/">wurstmeister Docker Kafka image</a></li><li><a href="http://docs.confluent.io/current/cp-docker-images/docs/quickstart.html">Docker image installation from Confluent</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c312ebd70413" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>