<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Martin Yung on Medium]]></title>
        <description><![CDATA[Stories by Martin Yung on Medium]]></description>
        <link>https://medium.com/@martinyce?source=rss-9f2c4bc7ed45------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NvA-UBXLdt8znkbyWzHdKg.jpeg</url>
            <title>Stories by Martin Yung on Medium</title>
            <link>https://medium.com/@martinyce?source=rss-9f2c4bc7ed45------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:42:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@martinyce/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using python script for data ETL]]></title>
            <link>https://codeburst.io/using-python-script-for-data-etl-53138c567906?source=rss-9f2c4bc7ed45------2</link>
            <guid isPermaLink="false">https://medium.com/p/53138c567906</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Martin Yung]]></dc:creator>
            <pubDate>Sun, 28 Jan 2018 04:19:54 GMT</pubDate>
            <atom:updated>2018-01-29T12:42:13.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*rnzQXZQ446QvlVcYr_ocmw.gif" /><figcaption><a href="https://dribbble.com/shots/2309065-Connecting-Networks">Data Flow</a></figcaption></figure><p>So you need to perform simple Extract Transform Load (ETL) from different databases to a data warehouse to perform some data aggregation for business intelligence. There are alot of ETL products out there which you felt is overkilled for your simple use case.</p><p>Here I am going to walk you through on how to <em>Extract</em> data from mysql, sql-server and firebird, <em>Transform</em> the data and <em>Load</em> them into sql-server (data warehouse) using <strong>python 3.6</strong>.</p><p>Before we begin, let’s setup our project directory:</p><pre>python_etl<br>    |__main.py<br>    |__db_credentials.py<br>    |__variables.py<br>    |__sql_queries.py<br>    |__etl.py</pre><p>Aside from having sql-server, mysql and firebird installed, we also need the below 3 python modules, run pip install [module_name] and we are ready to go.<br> <br>- <a href="https://dev.mysql.com/doc/connector-python/en/connector-python-introduction.html">mysql-connector-python</a>: connecting to mysql<br>- <a href="https://github.com/mkleehammer/pyodbc/wiki">pyodbc</a>: connecting to sql-server<br>- <a href="https://firebirdsql.org/file/documentation/drivers_documentation/python/fdb/getting-started.html">fdb</a>: connecting to firebird</p><h4>Setup Database Credentials and Variables</h4><p>Setup a variable to store the data warehouse database name in variables.py</p><pre>datawarehouse_name = &#39;your_datawarehouse_name&#39;</pre><p>Setup all your source databases and target database connection strings and credentials in your db_credentials.py as shown below. Store the config as list so we can iterate through many databases later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a85917cb5d035a8bfa1b668562ebe51/href">https://medium.com/media/1a85917cb5d035a8bfa1b668562ebe51/href</a></iframe><h4>SQL Queries</h4><p>In your sql_queries.py, here is where you will store all your sql queries for extracting from source databases and loading into your target database (data warehouse)</p><p>As we are dealing with different data platforms, we can use different syntax for each data platform by separate the queries according to the database type.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/130ead0c11ff6619a5fe76ee0d0b90c7/href">https://medium.com/media/130ead0c11ff6619a5fe76ee0d0b90c7/href</a></iframe><h4>Extract Transform Load</h4><p>In your etl.py import the following python modules and variables to get started.</p><pre># python modules<br>import mysql.connector<br>import pyodbc<br>import fdb</pre><pre># variables<br>from variables import datawarehouse_name</pre><p>Here we will have two methods, etl() and etl_process() . etl_process() is the method to establish database source connection according to the database platform, and call the etl() method.</p><p>in etl() method, first it will run the extract query, store the sql data in the variable data, and insert it into target database which is your data warehouse. Transformation of data can be done by manipulating the data variable which is of type tuple.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a6d0b9ebe4f28b3a277df1293b13e21/href">https://medium.com/media/2a6d0b9ebe4f28b3a277df1293b13e21/href</a></iframe><h4>Putting it All Together</h4><p>In our main.py we can loop through all credentials and perform the etl for all databases.<br>Import all relevant variables and methods:</p><pre># variables<br>from db_credentials import datawarehouse_db_config, sqlserver_db_config, mysql_db_config, fbd_db_config<br>from sql_queries import fbd_queries, sqlserver_queries, mysql_queries<br>from variables import *</pre><pre># methods<br>from etl import etl_process</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c855073621e2d9dae312c978a7b630c/href">https://medium.com/media/3c855073621e2d9dae312c978a7b630c/href</a></iframe><p>In your terminal hit python main.py and voila, you have just build a etl using pure python script.</p><p>Do hit me up with any questions or best practices by leaving a comment or on Twitter <a href="https://twitter.com/martin_yce">@martin_yce</a>, Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53138c567906" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/using-python-script-for-data-etl-53138c567906">Using python script for data ETL</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Upload files to S3 using vuejs and grails]]></title>
            <link>https://codeburst.io/upload-files-to-s3-using-vuejs-and-grails-783594cecb8a?source=rss-9f2c4bc7ed45------2</link>
            <guid isPermaLink="false">https://medium.com/p/783594cecb8a</guid>
            <category><![CDATA[vuejs]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[grails]]></category>
            <category><![CDATA[groovy]]></category>
            <category><![CDATA[amazon-s3]]></category>
            <dc:creator><![CDATA[Martin Yung]]></dc:creator>
            <pubDate>Sat, 09 Sep 2017 06:07:59 GMT</pubDate>
            <atom:updated>2017-09-09T14:35:42.607Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently have the opportunity to work on a big data project using groovy on grails as backend framework and serving the frontend using vuejs. Here is a simple tutorial on how to upload a file to S3 using vuejs and groovy on grails.</p><h4>Disclaimer:</h4><p>Basic understanding of vuejs and grails is required as I only include the code snippet of related components instead of a full functional code base.</p><p>I am using <em>grails 3.2.7, groovy 2.4.11, java 1.8 </em>and<em> vuejs 2.0.</em></p><h4>Getting Started — Frontend</h4><p>Let’s start by building the vuejs upload component. Create a Upload.vue component in the directory [your_project]/src/components/</p><p>Wire up the basic html template by using bootstrap to beautify the upload. Notice that the default html <em>select file button</em> is hidden and replaced with bootstrap button with the click event handler using method selectFile</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d64f6540f002351cdf5184db6ff3d86/href">https://medium.com/media/6d64f6540f002351cdf5184db6ff3d86/href</a></iframe><p>Let’s add required http service and methods to complete the upload component. I am using <a href="https://github.com/pagekit/vue-resource">vue-resource</a> for the http post request and take note that there is no authorisation involved to keep things simple.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47e220550ba2aed13015f270463f5209/href">https://medium.com/media/47e220550ba2aed13015f270463f5209/href</a></iframe><h4>Installing Grails AWS SDK S3 Plugin</h4><p>Now our frontend component has completed, let’s install <a href="https://github.com/agorapulse/grails-aws-sdk/tree/master/grails-aws-sdk-s3">Grails AWS SDK S3 Plugin</a>. From the documentation, add plugin dependency to your build.gradle:</p><pre>repositories {<br>    ...<br>    maven { url &#39;http://dl.bintray.com/agorapulse/libs&#39; }<br>    ...<br>}<br><br>dependencies {<br>  ...<br>  compile &#39;org.grails.plugins:aws-sdk-s3:2.1.5&#39;<br>  ...</pre><p>Proceed to setup your aws credentials in your ~/.aws/credentials The easiest way is install aws cli and follow the aws documentation <a href="http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html">here</a>.</p><h4>Wire Up Grails UploadController</h4><p>Finally, create UploadController.groovy and integrate the AWS s3 plugin to push uploaded files to your S3 bucket.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c826d84bfba64809ae74864411e09a79/href">https://medium.com/media/c826d84bfba64809ae74864411e09a79/href</a></iframe><p>That’s it, we have successfully build a S3 file upload feature using grails as backend and vuejs as the frontend framework.</p><p>Feel free to hit me up with any questions or best practices by leaving a comment or on Twitter <a href="https://twitter.com/martin_yce">@martin_yce</a>, Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=783594cecb8a" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/upload-files-to-s3-using-vuejs-and-grails-783594cecb8a">Upload files to S3 using vuejs and grails</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implement conversational-form in React.js]]></title>
            <link>https://codeburst.io/https-medium-com-martinyce-implement-conversational-form-in-react-js-d6cb5a8f1e34?source=rss-9f2c4bc7ed45------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6cb5a8f1e34</guid>
            <category><![CDATA[conversational-ui]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[jsx]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Martin Yung]]></dc:creator>
            <pubDate>Mon, 26 Jun 2017 09:19:05 GMT</pubDate>
            <atom:updated>2017-06-26T13:30:43.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JAkBDQL-LAMD2KstduFkbA.png" /><figcaption>conversational-form by space10</figcaption></figure><p><a href="https://github.com/space10-community/conversational-form">Conversational-form</a> by <a href="https://space10.io/">space10</a> is a simple way to make your website more interactive. Recently I have encountered a few problems implementing conversational-form into project with react.js, so here’s a little guide on implementation with react.js from the help of conversational-form contributor <a href="https://github.com/felixnielsen">Felix Nielsen</a>.</p><h4>Getting started</h4><p>First, include conversational form CDN into your <em>index.html</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2295ad14e7ae2ab7ed11525fa1a44062/href">https://medium.com/media/2295ad14e7ae2ab7ed11525fa1a44062/href</a></iframe><h4>Adding to your react component</h4><p>Start by adding the conversational-form reference this.cf = null to your react component.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b53664a945dfdb32655fe19f4b7c89d/href">https://medium.com/media/3b53664a945dfdb32655fe19f4b7c89d/href</a></iframe><h4>Adding questions and form submit action</h4><p>After that, customise the questions and actions for the submit button.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/989b62c693839473efbfc88d3c183869/href">https://medium.com/media/989b62c693839473efbfc88d3c183869/href</a></iframe><h4>Time to render()</h4><p>in the render() section, wrap the conversational form to an element by giving it id=&quot;cf-context&quot; , if not the conversational-form will occupy the whole html body. In the form, just call the ref to the questions you have set up previously in the componentDidMount() and viola, the conversational form has been successfully integrated with react.js.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/97ebfb8b3b552333deb6cf2c3cd363b9/href">https://medium.com/media/97ebfb8b3b552333deb6cf2c3cd363b9/href</a></iframe><p>Conversational form is a great way to bring simple interactive to websites. The integration with react by <a href="https://github.com/felixnielsen">Felix Nielsen</a> can be found <a href="https://github.com/space10-community/conversational-form/commit/d55564d1317b7bcb19db1aab6113b7869c3f195a">here</a>.</p><p>Feel free to hit me up with any questions or best practices by leaving a comment or on Twitter <a href="https://twitter.com/martin_yce">@martin_yce</a></p><p>Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6cb5a8f1e34" width="1" height="1" alt=""><hr><p><a href="https://codeburst.io/https-medium-com-martinyce-implement-conversational-form-in-react-js-d6cb5a8f1e34">Implement conversational-form in React.js</a> was originally published in <a href="https://codeburst.io/">codeburst</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>