<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by I-Feng Lin on Medium]]></title>
        <description><![CDATA[Stories by I-Feng Lin on Medium]]></description>
        <link>https://medium.com/@ifengleelin?source=rss-45c813b931e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bGkJEzpbHuCcrvmMYdax6A.jpeg</url>
            <title>Stories by I-Feng Lin on Medium</title>
            <link>https://medium.com/@ifengleelin?source=rss-45c813b931e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:14:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ifengleelin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[dbt has become the data build tool for many companies of different sizes.]]></title>
            <link>https://medium.com/@ifengleelin/dbt-has-become-the-data-build-tool-for-many-companies-of-different-sizes-b69fc76b4a73?source=rss-45c813b931e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/b69fc76b4a73</guid>
            <dc:creator><![CDATA[I-Feng Lin]]></dc:creator>
            <pubDate>Wed, 16 Mar 2022 21:43:52 GMT</pubDate>
            <atom:updated>2022-03-16T21:44:46.982Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.getdbt.com/">dbt</a> has become <em>the</em> data build tool for many companies of different sizes. The tool allows SQL developers to work in a more “software engineering” way to modularize, reuse, and generate SQL code with ease, and on top of that, to run the same code against different databases by using different adapters. It’s exciting to see an open-source project taking the prominent spot in the modern data stack. Thanks to its open sourceness, we are seeing more projects built around it to enrich its features. Among them, fal opens up the door for dbt projects to Python. It seamlessly runs Python scripts from dbt projects to read from or write to data models defined in your project, so that you can use Python code in addition to SQL for your data transformation. For the data workers, that means you have the goodness of both worlds.</p><p>After using dbt + fal for a couple of tasks, I was very glad to join a user feedback interview with <a href="https://www.linkedin.com/in/burkaygur/">Burkay Gur</a>, the co-founder of <a href="https://www.linkedin.com/company/features-and-labels/">Features &amp; Labels</a>, the team behind fal. The talk was designed to provide feedback for his team, but I took away more than what I had expected. I feel fortunate that I could discuss my use cases with another experienced engineer who is working on his venture specifically in the domain of data technology. It is rewarding and satisfactory to learn that my experience and skills allow me to connect with people working much deeper in the field and to share my opinions with them. Burkay also shared with me some of the exciting coming features in fal and its design concept. In addition, he also generously shared (or sneakily asked by me) some of his thoughts behind starting developing fal. It’s very intriguing for me to learn how a former engineer/manager founded a team that develops a new software technology, which fills in a unique spot in the market.</p><p>If you are interested in fal, please check out their GitHub:</p><p><a href="https://github.com/fal-ai/fal">https://github.com/fal-ai/fal</a></p><p>Here is the icing on the cake: with their monetary reward on my feedback session, a small amount of donation will be made to Community Organized Relief Effort, which is on the ground in Poland to support the immediate needs of people fleeing Ukraine due to the happening war. Read how they are helping here: <a href="https://donate.coreresponse.org/">https://donate.coreresponse.org/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b69fc76b4a73" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data quality is often neglected in the early stage of product development because a minimal viable…]]></title>
            <link>https://medium.com/@ifengleelin/data-quality-is-often-neglected-in-the-early-stage-of-product-development-because-a-minimal-viable-11b8b90d8c5?source=rss-45c813b931e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/11b8b90d8c5</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[great-expectations]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[I-Feng Lin]]></dc:creator>
            <pubDate>Mon, 07 Feb 2022 08:06:51 GMT</pubDate>
            <atom:updated>2022-02-07T08:21:48.325Z</atom:updated>
            <content:encoded><![CDATA[<p>Data quality is often neglected in the early stage of product development because a minimal viable or pilot product usually doesn’t require thorough data validation to justify its functionalities. Even in a mature and well-tested software product, it is still often overlooked because the code test coverage doesn’t reflect the coverage of possible input to the system, and the output resulting from most of the input combinations is left unverified. That would not be an issue only in an ideal world, in which the data we collect (by humans and machines) or our clients provide are clean with only expected values, but real life is full of human errors and signal noises. People don’t always get the spelling right and systems glitch or have its limit of countering random factors.</p><p>Data wrangling is often called to the rescue, which processes data before they are fed to the system. The dirtier your data are, the more human-eye inspection is needed at this step, and because human involvement is nearly indispensable, data wrangling is usually in the preliminary stage before the data go into the data pipeline. On the other end of the pipeline, as an addition to the software quality assurance, which conventionally focuses on the functioning of the software under different circumstances, but not on the logic behind the data derived by the system, data quality assessment, or data tests, can be helpful. For example, tools like <a href="https://greatexpectations.io/">Great Expectations</a> put data quality under the microscope with data profiling, testing, and reporting. <a href="https://docs.getdbt.com/docs/building-a-dbt-project/tests">dbt</a> also provides testing besides data modeling. While we are glad that these tools are catching the data errors for us, we still need to fix them by spotting the causes in the data pipeline. That is where things get tricky. The more complex your data processing is, the more difficult for you to spot the errors. After all, you don’t have a compiler to tell you which line of the code causes the wrong resulting data.</p><p><a href="https://www.linkedin.com/in/spbail/">Sam Bail</a> shows in the <a href="https://www.getdbt.com/coalesce-2020/building-a-robust-data-pipeline-with-dbt-airflow-and-great-expectations/">link</a> how you can test your data every time you hand off the data between teams or systems with data pipeline tooling such as Airflow. I would go even one step further to test my data after every logical data processing step. For example, my data are from multiple sources, and they may duplicate or conflict with each other, and my de-duplicating and merging logic are complex. What I do is to add a testing step after de-duplication to verify resulting data are as expected, before the pipeline goes to the merging step and another testing. Saying I find duplicate data after the merging step, I can then assume the logic error to lie in the merging step and not in any earlier ones, given my previous testing is solid. The beauty of it is, data quality assurance is tightly integrated into the data pipeline by running “data unit testing” after each step. If you put data wrangling into the picture, the first step of the data pipeline should be testing the input, i.e. the outcome of wrangled data. This model even allows test-driven development at the data level (if you like it.)</p><p>Does this sound interesting to you? <a href="https://alpas.ai/">Alpas AI</a> is looking for a data analyst to work with us to enhance data quality in our global supplier search platform. This role is in close collaboration with our business domain and data engineering, and we see it highly crucial to our mission to make the supply chain transparent for our clients. Feel free to reach me/us to know more. :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11b8b90d8c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How BigQuery GIS scales up your geospatial projects]]></title>
            <link>https://medium.com/@ifengleelin/how-bigquery-gis-scales-up-your-geospatial-projects-5edce84a63ee?source=rss-45c813b931e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5edce84a63ee</guid>
            <category><![CDATA[beacons]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[google]]></category>
            <dc:creator><![CDATA[I-Feng Lin]]></dc:creator>
            <pubDate>Sun, 14 Oct 2018 20:56:54 GMT</pubDate>
            <atom:updated>2018-10-14T20:56:54.411Z</atom:updated>
            <content:encoded><![CDATA[<p>Google released a new feature in open alpha in BigQuery, the data warehouse that enables users to manipulate and query data with SQL commands. The new feature is named BigQuery GIS, which empowers the data platform to support geometry data types and spatial operations. The introduced commands look a lot like those in PostGIS, which is a great benefit for PostgreSQL users. What makes BigQueryGIS outstanding is to apply these commands to data retrieved from BigQuery in fast response times.</p><h3>What is BigQuery</h3><p>BigQuery is one of Google’s big data solutions, which offers a RESTful web service to perform the analysis of large datasets in conjunction with Google Storage. It offers SQL-like languages, namely Standard SQL or Legacy SQL, to retrieve and manipulate data in TBs within seconds. With the recent integration of machine learning and geospatial information systems, this data warehouse is virtually transformed into a data analytics platform in the cloud.</p><h3>What is BigQuery GIS</h3><p>Big Query Geospatial Information Systems, or BQ GIS for short, is the enhancement of BigQuery with features to support geospatial data types and functions. You can store points, line strings, multi-polygons, and collections in the tables in BQ GIS with a special data type named Geophghy. Geophghy objects can be directly passed to geospatial functions in Standard SQL to retrieve the results or store them in a new table. The supported functions including constructions, transformations, predicates, or measures of geospatial objects.</p><h3>Why BigQuery GIS</h3><ul><li>Reduce complexity — everything in one tool</li><li>Require only one step to export data from the warehouse</li><li>Increase speed — geometric operations in BigQuery is fast with large amounts data</li><li>Scale up easily with BigQuery</li></ul><h3>Our Use Case and Implementation</h3><p>We have users across Germany and other European countries. Let’s say we want to create a heat map of the number of users divided by zip regions. Given the reported coordinates of user requests and the German zip region files, we can map every request to a zip region and count the number of users in each region. We can color the map of Germany like this, where red areas have the highest numbers of requests:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/0*unoe1-hLYuYD-kaW" /></figure><p>And this requires just four steps:</p><p>Step 1: Import shapefiles into BigQuery</p><p>Download <a href="https://www.suche-postleitzahl.org/">German zip region shapefiles</a> and convert .shape into WKT (well-known text) with GDAL with the following command:</p><p><em>&gt; ogr2ogr -f csv -lco GEOMETRY=AS_WKT plz-gebiete.csv plz-gebiete.shp</em></p><p>Then import the created csv file into a table BigQuery (<a href="https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv">read here</a> to load with BigQuery UI, command line, or scripts)</p><p>Step 2: Convert regions into polygons with Standard SQL. Here we use a geospatial construction command <em>ST_GeogFromText</em> to convert WKT in the table loaded from the csv file into polygons and store it in another table named<em> geodata.de_zip_regions_geo.</em></p><pre>CREATE TABLE geodata.de_zip_regions_geo</pre><pre>AS (</pre><pre>SELECT</pre><pre>ST_GeogFromText(WKT) polygon,</pre><pre>zip_code</pre><pre>FROM</pre><pre>`beaconinside-analytics-test.geodata.de_zip_regions`<br>)</pre><p>Step 3: Convert coordinates into points with Standard SQL. Here we use a geospatial construction command <em>ST_GeogPoint</em> to convert pairs of float values longitude and latitude in a table into Geography objects Points and store them in another table named<em> demodata.user_requests_geo.</em></p><pre>Standard SQL command:</pre><pre>CREATE TABLE demodata.user_requests_geo</pre><pre>AS (</pre><pre>SELECT</pre><pre>ST_GeogPoint(lon, lat) point,</pre><pre>user_id_sdk,</pre><pre>created_at</pre><pre>FROM</pre><pre>`beaconinside-analytics-test.demodata.user_requests`</pre><pre>)</pre><p>Step 4: Count users in each region with Standard SQL. We select from both <em>regions</em> and <em>requests tables</em> and we need a geospatial predicator <em>ST_Covers </em>to be the condition of the join of two tables. The requests whose coordinates are covered by the regions are joined in the new table. It would be equivalent to use <em>ST_Intersects</em> or <em>ST_Contains</em> here. The rows are grouped by their zip code and we count the unique number of <em>user_id_sdk</em>, which is the identifier of users, to get the number of users in regions. The polygon is selected by <em>ANY_VALUE </em>because regions of the same zip code have the same polygon.</p><pre>SELECT</pre><pre>  zip_code,</pre><pre>  ANY_VALUE(regions.polygon) polygon,</pre><pre>  COUNT(DISTINCT requests.user_id_sdk) num_users</pre><pre>FROM</pre><pre>  `beaconinside-analytics-test.geodata.de_zip_regions_geo` regions,</pre><pre>  `beaconinside-analytics-test.demodata.user_requests_geo` requests</pre><pre>WHERE</pre><pre>  ST_Covers( regions.polygon, requests.point )</pre><pre>GROUP BY</pre><pre>  zip_code</pre><pre>ORDER BY</pre><pre>  num_users DESC</pre><p>The rows of the results consists of the zip code of a region, its Geography (polygon,) and the number of users in the region. The results can also be displayed in a tool named BQ Geo Vis, which is released together with BQ GIS as more like a preview tool because of the number of rows is limited to 2000 and the size of data to 10 MB. Therefore in the following screenshot, you can see only some regions colored. The results are sorted descending in order to plot the top regions. Red indicates a high number of users. Nonetheless, the table can be stored downloaded and plotted by other tools, such as the Germany map showed in the beginning plotted by matplotlib.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vJhAd9VmLrrncXfr" /></figure><h3>Why we like BQ GIS</h3><p>With all the advantages mentioned earlier, you’d probably still wonder if you want to migrate your project to BQ GIS to have the same outcome. After all, GIS-support is nothing new among databases. What really makes a difference is not what you get but <em>how long </em>it takes when the amount of data scale up. As an active API provider in proximity marketing, Beaconinside receives a huge amount of user requests per day. Mapping requests of one day to regions can take up to seconds. How about mapping requests of one month? We found that BQ GIS is exceptionally successful when the amount of data is large and when really <em>large</em>. The following chart shows that the spatial join and aggregation done with BQ GIS takes slightly more time when the number of requests triples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Md1rPYXpebWXaYVh" /></figure><p>We are very happy to see such a technology helps us analyze geospatial data on a large scale without demanding high computation power on our side. Moreover, this simplifies the pipeline of data if you are already using BigQuery as your data warehouse. We look forward to seeing that BQ GIS becomes a standard feature and provides more functionalities, such as coordinate system conversions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5edce84a63ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>