<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Allen Fang on Medium]]></title>
        <description><![CDATA[Stories by Allen Fang on Medium]]></description>
        <link>https://medium.com/@AllenFangTW?source=rss-f0510ed38e8c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8awliWwny3nPYqDlK11rOQ.jpeg</url>
            <title>Stories by Allen Fang on Medium</title>
            <link>https://medium.com/@AllenFangTW?source=rss-f0510ed38e8c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 19:38:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@AllenFangTW/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Troubleshooting Google BigQuery Vector Search with LangChain]]></title>
            <link>https://medium.com/@AllenFangTW/troubleshooting-google-bigquery-vector-search-with-langchain-c35dc70115b1?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c35dc70115b1</guid>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[bigquery-vector-search]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Tue, 18 Feb 2025 04:08:44 GMT</pubDate>
            <atom:updated>2025-02-18T04:08:44.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_jdZYi37KsTGtMdv" /><figcaption>Photo by <a href="https://unsplash.com/@disruptxn?utm_source=medium&amp;utm_medium=referral">Desola Lanre-Ologun</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It’s just my troubleshooting journey when I first time use langchain to do the similarity search via google big query. I’m also a noob to Python, Langchain and GCP, so please forget me those noob issue 😅 or if you already know the answer.</p><p>The story is start from a simple python script, it almost follow the <a href="https://python.langchain.com/docs/integrations/vectorstores/google_bigquery_vector_search">langchain doc</a>.</p><pre># main.py<br>from langchain_google_vertexai import VertexAIEmbeddings<br>from langchain_google_community import BigQueryVectorStore<br><br>def main():<br>    PROJECT_ID = &quot;my-project&quot;<br>    REGION = &quot;us-central1&quot;<br>    DATASET = &quot;my-dataset&quot;  <br>    TABLE = &quot;ntn_514580605659&quot;<br><br>    embedding = VertexAIEmbeddings(<br>        model_name=&quot;textembedding-gecko@latest&quot;, project=PROJECT_ID<br>    )<br><br>    store = BigQueryVectorStore(<br>        embedding=embedding,<br>        project_id=PROJECT_ID,<br>        dataset_name=DATASET,<br>        table_name=TABLE,<br>        location=REGION<br>    )<br><br>    query = &quot;....&quot;<br>    docs = store.similarity_search(query)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>At beginning, I already have big query vector table with data on GCP which created by my colleague from long time ago. Subsequently, I encounter first error when run the script.</p><h3><a href="https://stackoverflow.com/questions/74831594/cannot-import-name-wkbwriter-from-shapely-geos-when-import-google-cloud-ai-p">cannot import name ‘WKBWriter’ from ‘shapely.geos’</a></h3><p>It’s likely a missing lib or a lib version mismatch. I just try to install shapely directly but it doesn’t help, but following solve the problem.</p><pre>pip install -U google-cloud-aiplatform &quot;shapely&lt;2&quot;</pre><h3>503 Getting metadata from plugin failed with error: (‘Unable to acquire impersonated credentials’, …)</h3><p>After resolve the library issue, now it’s auth problem. The way you need is to create a service account then grant the proper roles to it:</p><p>Go to IAM &amp; Admin in GCP console, switch to the service accounts tab, then create a service account with following granted roles:</p><ul><li>Vertex AI User</li><li>BigQuery User</li><li>BigQuery Data Owner</li></ul><p>After you complete the creation of a service account, you are able to get a email like [service-account-name]@[project-id].iam.gserviceaccount.com</p><p>So right now you can do the login in your console via</p><pre>gcloud auth application-default login --impersonate-service-account [service-account-email]</pre><p>In case you stay in wrong project, following command could switch the default project in local</p><pre>gcloud config set project [proejct_id]</pre><h3>google.api_core.exceptions.NotFound: 404 POST …, Not found: Dataset [project-id]:[dataset] was not found in location</h3><p>This error comes up when I solve the authentication problem and the issue is actually just I give the wrong location. Even our entire service or whatever things on GCP is in us-central1. But for biy query, it support the Multi-regions and unfortunately our data set is in the multi-region (US). So this could be solved by change the location to ‘US’</p><pre>store = BigQueryVectorStore(<br>        embedding=embedding,<br>        project_id=PROJECT_ID,<br>        dataset_name=DATASET,<br>        table_name=TABLE,<br>        location=REGION  ## change to &quot;US&quot;<br>    )</pre><p>You can refer to <a href="https://cloud.google.com/bigquery/docs/locations#multi-regions">https://cloud.google.com/bigquery/docs/locations#multi-regions</a> for more details</p><h3>AttributeError: module ‘google.cloud.bigquery.enums’ has no attribute ‘QueryApiMethod’</h3><p>After fixed above problem, I saw this error immediately, it’s again like a lib problem. Maybe the root cause is my local default python lib are mess. So I decide to look some python package/lib management solution, which similar to yarn/npm in Node.js, it’s for ensuing the right libs are installed for my work. Without spending too much time on comparison, I quickly choose poetry , you can use below command to install it into your mac system</p><pre>brew install poetry</pre><p>Prepare a new project and set it up</p><pre>## create a new project<br>poetry new langchainpy-bigquery-vector-search<br><br>## install the lib we need<br>poetry add langchain langchain-google-vertexai &quot;langchain-google-community[featurestore]&quot;<br><br>## enable virtual env for project<br>poetry config virtualenvs.in-project true</pre><p>Rightnow, you can go to the VSCode to switch the python interpreter via “shift+ctrl+p” then input Python:Select interpreter and choose the venv your just created by poetry. After all of it, you can run the python script with poetry:</p><pre>poetry run python3 main.py</pre><h3>google.api_core.exceptions.BadRequest: 400 POST … embedding should exist in 1st argument of VECTOR_SEARCH</h3><p>It’s the final issue 😈. After trace the langchain source code and analyze the SQL generated by langchain, I notice that the embedding column name could be wrong:</p><pre>SELECT *<br>        FROM (<br>            with embeddings as (<br>SELECT 0 as row_num, @emb_0 AS content_embedding<br>)<br><br>        SELECT<br>            base.*,<br>            query.row_num,<br>            distance AS score<br>            <br>        FROM VECTOR_SEARCH(<br>            (SELECT * FROM `[project-id].[dataset].[tablename]` WHERE TRUE),<br>            &quot;embedding&quot;,<br>            (SELECT row_num, embedding FROM embeddings),<br>            distance_type =&gt; &quot;EUCLIDEAN&quot;,<br>            top_k =&gt; 5<br>        )<br>        <br>        ) AS result<br>        ORDER BY row_num, score</pre><p>By default, langchain use “embedding” as the default embedding column name, then I just figure out the embedding column in our table is “content_embedding”. After trace the code a bit, we can assign the embedding column when construct the BigQueryVectorStore</p><pre>store = BigQueryVectorStore(<br>        embedding=embedding,<br>        project_id=PROJECT_ID,<br>        dataset_name=DATASET,<br>        table_name=TABLE,<br>        embedding_field=&quot;content_embedding&quot;,<br>        location=&quot;US&quot;<br>    )</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c35dc70115b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[_luaL_checkversion_ error when start OpenResty with lua-resty-aws]]></title>
            <link>https://medium.com/@AllenFangTW/lual-checkversion-error-when-start-openresty-with-lua-resty-aws-15e5d37c5b23?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/15e5d37c5b23</guid>
            <category><![CDATA[openresty]]></category>
            <category><![CDATA[lua-resty-aws]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 07:25:47 GMT</pubDate>
            <atom:updated>2023-11-03T07:25:47.787Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>Recently, I started to study <a href="https://apisix.apache.org/">APISIX</a> which is a Cloud-Native API Gateway. Along the way, I also learned something about <a href="https://openresty.org/">OpenResty</a> which is a Web Platform by Extending <a href="https://www.nginx.com/">NGINX</a> with <a href="https://luajit.org/">LuaJIT</a>. Due to some reasons, I start to learn and work on integrating the lua-resty-aws in OpenResty.</p><p>This is just a note for a stupid problem which struggling me over couple hours 😂 (I’m a newbie for Lua, OpenResty etc)</p><p>My environment:</p><ul><li>MacOS (M2 chip)</li><li>OpenResty: openresty/1.21.4.2 (installed by brew)</li><li>Lua: 5.4 (installed by brew)</li><li>Luarocks: 3.9.1 (installed by brew)</li></ul><p>If you got the problem to install the lua-resty-aws in your MacOS(M2), you can also refer to my <a href="https://medium.com/@AllenFangTW/luarocks-install-lua-resty-aws-failed-on-mac-os-0804812f1536">previous post</a>.</p><p>I will provide another post for an entire project setup for lua-resty-aws in OpenResty. Here will not focus too much basic setup detail.</p><h3>Problem</h3><p>Anyway, entire story was begun from a simplenginx.conf</p><pre>worker_processes  1;<br>error_log logs/error.log info;<br>events {<br>    worker_connections 1024;<br>}<br>http {<br>    lua_package_path &quot;...&quot;;<br>    lua_package_cpath &quot;...&quot;;<br>    init_by_lua_block {<br>        local _ = require(&quot;resty.aws.config&quot;).global # trying to init resty.aws<br>    }<br>    server {<br>        listen 8080;<br>        location / {<br>            default_type text/plain;<br>            content_by_lua_block {<br>                omit ... <br>            }<br>        }<br>    }<br>}</pre><p>Then I got following error consistently when I start nginx :</p><pre>2023/11/02 15:42:40 [error] 51403#10225587: *12 lua entry thread aborted: runtime error: ...w/Cellar/openresty/1.21.4.2_1/lualib/resty/core/base.lua:80: loop or previous error loading module &#39;hello&#39;<br>stack traceback:<br>coroutine 0:<br> [C]: in function &#39;error&#39;<br> ...w/Cellar/openresty/1.21.4.2_1/lualib/resty/core/base.lua:80: in function &#39;require&#39;<br> content_by_lua(nginx.conf:12):2: in main chunk, client: 127.0.0.1, server: , request: &quot;GET /favicon.ico HTTP/1.1&quot;, host: &quot;127.0.0.1:8080&quot;, referrer: &quot;http://127.0.0.1:8080/&quot;<br>2023/11/02 15:47:07 [error] 52226#10230933: *13 lua entry thread aborted: runtime error: error loading module &#39;lfs&#39; from file &#39;/opt/homebrew/lib/lua/5.4/lfs.so&#39;:<br> dlopen(/opt/homebrew/lib/lua/5.4/lfs.so, 0x0006): symbol not found in flat namespace &#39;_luaL_checkversion_&#39;<br>stack traceback:<br>coroutine 0:<br> [C]: in function &#39;require&#39;<br> content_by_lua(nginx.conf:13):2: in main chunk, client: 127.0.0.1, server: , request: &quot;GET / HTTP/1.1&quot;, host: &quot;127.0.0.1:8080&quot;</pre><p>Above error is not regarding to the lua module not found or something wrong in lfs library in direct, but it look like lua-resty-aws leverage lfs but lfs somehow can’t be loaded or imported successfully.</p><p>I also tried to only load lfs directly in thenginx.conf or other lua script rather than loading lua-resty-aws, this error also came out again 😤, ex:</p><pre>init_by_lua_block {<br>        local fs = require(&quot;lfs&quot;)<br>}</pre><h3>Root Cause</h3><p>After deeply dig and some learns from APISIX source code, I just noticed that APISIX also leverage OpenResty on version 1.21.4.2, however it can work successfully in my local (APISIX also leverage lfs).</p><p>After trace bunch of source codes of APISIX, I notice that APISIX sticky in Lua@5.1 which is different to my project (5.4), so I try to install lfs by Lua@5.1:</p><pre>luarocks install luafilesystem --lua-version 5.1</pre><p>Because the default Lua version for Luarocks is 5.4 in my local, so it resulted the Luarocks to install lfs in user dictory, my case is: <strong>/Users/allen/.luarocks/lib/lua/5.1, </strong>so you will see the lfs.so got installed in this dir.</p><p>The next is adjust the lua_package_cpath in nginx.conf</p><pre>lua_package_cpath &quot;...;/Users/allen/.luarocks/lib/lua/5.1?.so;&quot;</pre><p>Subsequently, the error disappear after nginx reload. But I got other minor issues so I literally install all the dependencies like cjson, lua-resty-core , lua-resty-aws etc packages under Lua@5.1.</p><p>I didn’t study if this issue only happen in MacOS(M2 chip). Anyway, just a share for helping.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15e5d37c5b23" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[luarocks install lua-resty-aws failed on Mac OS]]></title>
            <link>https://medium.com/@AllenFangTW/luarocks-install-lua-resty-aws-failed-on-mac-os-0804812f1536?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0804812f1536</guid>
            <category><![CDATA[luaexpat]]></category>
            <category><![CDATA[lua-resty-aws]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Wed, 01 Nov 2023 02:43:07 GMT</pubDate>
            <atom:updated>2023-11-01T02:43:07.906Z</atom:updated>
            <content:encoded><![CDATA[<p>I got following error when I was trying to install the lua-resty-aws in MacOS(M2 chip)</p><pre>luaexpat 1.5.1-1 depends on lua &gt;= 5.1 (5.4-1 provided by VM)<br>env MACOSX_DEPLOYMENT_TARGET=11.0 gcc -O2 -fPIC -I/opt/homebrew/opt/lua/include/lua5.4 -c src/lxplib.c -o src/lxplib.o -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -Isrc/<br>src/lxplib.c:12:10: fatal error: &#39;expat_config.h&#39; file not found<br>#include &quot;expat_config.h&quot;<br>         ^~~~~~~~~~~~~~~~<br>1 error generated.<br><br>Error: Failed installing dependency: https://luarocks.org/luaexpat-1.5.1-1.src.rock - Build error: Failed compiling object src/lxplib.o</pre><p>How to solve:</p><ol><li><strong>Check if you have installed the </strong><strong>expat in your local. If yes, skip this step, </strong>otherwise<strong> </strong>install via homebrew for instance.</li></ol><pre>$ HOMEBREW_NO_AUTO_UPDATE=1 brew install expat</pre><p>2. Get the <strong>expat</strong> installation path</p><pre>$ brew info expat <br><br># My case is /opt/homebrew/Cellar/expat/2.5.0 </pre><p>3. Install again</p><pre>$ luarocks install lua-resty-aws EXPAT_DIR=/opt/homebrew/Cellar/expat/2.5.0</pre><p>Hope this will be helpful to you 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0804812f1536" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Things You might be Interested in Your Flow Config]]></title>
            <link>https://medium.com/shopback-tech-blog/5-things-you-might-be-interested-in-your-flow-config-1ad35b023e93?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ad35b023e93</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[flow]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Sat, 16 Mar 2019 07:15:47 GMT</pubDate>
            <atom:updated>2019-03-17T07:49:38.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xlS5hnpMjbdHLxSQR38dTw.png" /></figure><p>In ShopBack, we use <a href="https://flow.org/">FaceBook Flow</a> as our static type checker in a new web application project. Flow is much lightweight than TypeScript and easier to learn for developer. Before we started to adopt Flow, we already knew the community is not stronger than TypeScript and if you face a problem that will be a bit hard to fix properly. So in this article, I would like to share common Q&amp;A you probably will encounter when you adopt Flow.</p><h3>Should I ignore node_module in my .flowconfig?</h3><p>No. when you start to adopt flow in your existing project, you don’t need to ignore node_module. If you ignore it like that in .flowconfig:</p><pre>[ignore]<br>.*/node_modules/.*</pre><p>Flow check will blame about it’s unable to resolve dependencies like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*BM9dMtj_mXij1BD6p_qvcA.png" /></figure><p>If you don’t ignore it already but still see abnormal check errors from node_modules, you can consider two solutions:</p><p>1. Do not ignore node_module and let flow to check the node_modules. But you can consider to ignore the specified packages like:</p><pre>[ignore]<br>.*/node_modules/jsonlint/.*</pre><p>2. Use <a href="https://flow.org/en/docs/config/untyped/">untyped</a> to make flow makes node_module un-typechecked but can still be require()‘d.</p><pre>[untyped]<br>.*/node_modules/*</pre><h3>How to fix module resolve error?</h3><p>Developer usually like a clean module resolve path, for example:</p><p>import something from &#39;utils/calculator&#39;</p><p>instead of</p><p>import something from &#39;../../../../../utils/calculator&#39;</p><p>But if you use the favor one, flow check must show the error about it’s unable to resolve the modules. Don’t worry, <a href="https://flow.org/en/docs/config/options/#toc-module-name-mapper-regex-string">module.name_mapper</a> can solve this:</p><pre>[options]<br>module.name_mapper=&#39;^utils\/\(.*\)$&#39; -&gt; &#39;&lt;PROJECT_ROOT&gt;/src/utils/\1&#39;</pre><pre>module.name_mapper=&#39;^components\/\(.*\)$&#39; -&gt; &#39;&lt;PROJECT_ROOT&gt;/src/components/\1&#39;</pre><pre>module.name_mapper=&#39;^containers\/\(.*\)$&#39; -&gt; &#39;&lt;PROJECT_ROOT&gt;/src/containers/\1&#39;</pre><h3>How to resolve a CSS/SCSS module?</h3><p>When you try to import a CSS or SCSS file into a module, flow check must complain about it can not resolve the this module due to file extension issue:<br><strong>“Required module not found”</strong></p><p>So you have to tell flow that what extension should be resolved, for example:</p><pre>[options]<br>module.file_ext=.scss</pre><p>However, it’s still not enough due to Flow can not recognize the CSS or SCSS syntax, so one more configs you have to add:</p><pre>[options]<br>module.file_ext=.scss<br>module.name_mapper.extension=&#39;scss&#39; -&gt; &#39;empty/object&#39;</pre><p>Unfortunately, after you configure above settings, Flow checks on all the JavaScript file will be ignored... so you have to add JavaScript file to another extension:</p><pre>[options]<br>module.file_ext=.js<br>// Remember to add any file extension you need to check module.file_ext=.scss<br>module.name_mapper.extension=&#39;scss&#39; -&gt; &#39;empty/object&#39;</pre><h3>Handle third party code</h3><p>Sometimes, you may just cope a third code into your code base but you want those code won’t parsed or parsed but doesn’t preserve types. Again, it’s a little similar issue and it can be easy to solved by <a href="https://flow.org/en/docs/config/untyped/">untyped</a>. Following is official example:</p><pre>[untyped]<br>.*/third_party/.*<br>.*/src/\(foo\|bar\)/.*<br>.*\.untype\.js</pre><h3>What&#39;s the difference between ignore and untyped?</h3><p>[ignore] tell Flow <strong>DO NOT PARSE</strong> the specified files, default Flow will parse everything!! In most cases, You should not use this to ignore <strong>your code</strong> because that will cause checking error in somewhere that import/require this code.</p><p>[untyped] instead causes a file to be ignored by the type checker as if it had noflow in it, resolve modules as any type. If you don’t want Flow to do the type check on your code but still need it to be resolved, you are supposed to use untyped instead of ignore .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ad35b023e93" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/5-things-you-might-be-interested-in-your-flow-config-1ad35b023e93">5 Things You might be Interested in Your Flow Config</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Design and Implement a Table Component based on React.js - Part 1]]></title>
            <link>https://medium.com/shopback-tech-blog/design-and-implement-a-table-component-based-on-react-js-part-1-b3d93f12d812?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3d93f12d812</guid>
            <category><![CDATA[front-end-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[react-components]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Tue, 25 Dec 2018 09:52:47 GMT</pubDate>
            <atom:updated>2019-01-02T02:36:01.325Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WzxWZoW1arorHD5GJATGXw.png" /></figure><p>I’m creator of <a href="https://github.com/AllenFang/react-bootstrap-table#readme"><strong>react-bootstrap-table</strong></a> and <a href="https://github.com/react-bootstrap-table/react-bootstrap-table2"><strong>react-bootstrap-table2</strong></a>. That is about three years journey for creating this two React UI library. Currently, I already deprecated react-bootstrap-table due to bad maintainability. It was published at three years ago when I was a Big Data Engineer, I made so many mistakes due to poor knowledges for React, Javascript and Frontend etc. That’s why I decided to create a new react-bootstrap-table2 since from the end of 2017.</p><p>This is a series posts and I will divide it as four parts.</p><ul><li><strong><em>Part1 - Big picture and design thought for a React table component</em></strong></li><li>Part2 - Internal Design and Implementation</li><li>Part3 - React Component Customizability</li><li>Part4 - Monorepo Sharing</li></ul><p>In this post, I would like to share some concepts and high level thought about what kind of table features we should provide to developer or user.</p><h3>Rich Functional Table</h3><p>If the table component you develop is just for internal use, I think probably you don’t need to think too much. However, if you want to open source, you are better to consider more rich functions. I’m <a href="http://trirand.com/blog/jqgrid/jqgrid.html"><strong>jqGrid</strong></a> big fans cause of jqGrid support a lots of fantastic features as well as good flexibility for customization on each part of table.</p><p>In react-bootstrap-table we have a lots of cool features like: row selection, Excel-like keyboard navigation, row expand, pagination, cell edit, filter/search etc. These are key things for developer they may consider to use your library because you give them awesome built-in functionalities.</p><p>If you support a lots awesome feature, <a href="https://github.com/storybooks/storybook">storybook</a> is best choice to perform your feature online. Like react-bootstrap-table, we have a <a href="http://react-bootstrap-table.github.io/react-bootstrap-table2/storybook/index.html">online demo</a> built by storybook</p><h3>Developer Friendly Library</h3><p>Firstly, easier for developer to use your UI library is most important thing when you design whatever component or API. In React perspective, I got three principles:</p><h4>Custom Data Management</h4><p>It’s an important perspective when you design a table component based on React. In react-bootstrap-table we have a lots of features which involved data management, like column filter, table search, sort, pagination, cell editing etc. Most of cases, developer expect you to help them complete these stuffs but sometime is not. For example, they may want to perform their search logic or implement their pagination logic through server side.</p><p>In react-bootstrap-table I call it remote mode. Developer can enable remote mode either on all functionalities or specific function. Following is a real case :</p><pre>const RemoteSort = props =&gt; (<br>  &lt;BootstrapTable<br>    ...<br>    remote={ { sort: true } } // only enable remote for sort<br>    onTableChange={ props.onTableChange }<br>  /&gt;<br>);</pre><pre>class Example extends React.Component {<br>  state = { data: products }</pre><pre>  handleTableChange = (type, { sortField, sortOrder, data }) =&gt; {<br>    // perform you sorting logic here<br>    const result = doMySort();<br>    this.setState(() =&gt; ({ data: result}));<br>  }</pre><pre>  render() {<br>    return (<br>      &lt;RemoteSort<br>        data={ this.state.data }<br>        onTableChange={ this.handleTableChange }<br>      /&gt;<br>    );<br>  }<br>}</pre><blockquote>The key point here is you are not only support custom data management but also allow developer to specify which part of data management they want to customize.</blockquote><h4>Controlled and Uncontrolled Management</h4><p>This idea is almost same as above. I would like to use a dropdown component as an example:</p><p>That’s said I want to <strong>open source</strong> a dropdown react component, did you think following example a good idea to force user to use your component like that?!</p><pre>import MyDropDown from &#39;my-dropdown&#39;;  // My open source component</pre><pre>class Example extends React.component {<br>  state = { currentOption: null }</pre><pre>  handleOptionChange = (newOption) =&gt; {<br>    this.setState(() =&gt; ({ currentOption: newOption }));<br>  }</pre><pre>  render() {<br>    return (<br>      &lt;MyDropDown<br>        options={ ... }<br>        currentValue={ this.state.currentOption }<br>        onChange={ this.handleOptionChange }<br>      /&gt;<br>    )<br>  }<br>}</pre><p>Absolutely No. The only benefit for you is MyDropDown component is a stateless(pure) component, but the drawback is bad usage for your user, because they have to handle the dropdown state by themself. In this case, MyDropDown is just an <strong>Uncontrolled Component</strong>.</p><p>However, developer hope you can help them to handle the logic of dropdown change internally:</p><pre>const Example = () =&gt; &lt;MyDropDown options={...} /&gt;;</pre><p>It’s simple apparently for developer, the dropdown component is self controlled which is a <strong>Controlled Component</strong>. Becasue developer is not necessary to handle the option changes in their code, if they need to know the current select value, you can consider to provide following ways for them:</p><ul><li>Use a callback, like onChange and pass the newest value as argument.</li><li>Use <a href="https://reactjs.org/docs/refs-and-the-dom.html">React Refs</a>.</li></ul><blockquote>In my opinion, if you want to open source a React component, you have to implement your component as <strong>Controlled</strong> and <strong>Uncontrolled</strong> <strong>both!!</strong></blockquote><p><a href="https://react-bootstrap-table.github.io/react-bootstrap-table2/storybook/index.html?selectedKind=Row%20Selection&amp;selectedStory=Selection%20Management&amp;full=0&amp;addons=1&amp;stories=1&amp;panelRight=0&amp;addonPanel=storybook%2Factions%2Factions-panel">Here</a> is an example about how react-bootstrap-table support developer an <strong>Uncontrolled</strong> way to manage the row selection externally.</p><h3>Exposed API</h3><p>This is useful for developer when they want to:</p><ul><li>Get internal state programmatically</li><li>Set or Control internal state programmatically</li></ul><p>For example, in react-bootstrap-table we have column filter, but sometime developer want to do the filter programmatically or trigger the filter from their component or something. In addition, they may want to know what filter value is applied on which column. Hence, react-bootstrap-table give developer <a href="https://react-bootstrap-table.github.io/react-bootstrap-table2/docs/exposed-api.html">some way</a> to access internal state or function.</p><p>For the developer, they can easy to do it like:</p><pre>this.table.filter(...); // Call a filter function to do the filter<br>this.table.state.data;  // Access the table state</pre><pre>&lt;BootstrapTable ref={ n =&gt; this.table = n } ... /&gt;</pre><blockquote>When you are designing the react component, it’s better to think more about how developer they access your state or function. Design the function or state more flexible is good for exposed API definitely.</blockquote><h3>Conclusion</h3><p>In this post, just share few high level thought when you design a table component. Actually, you can adopt these ideas to any React component as well. In the part 2, I will share more detail about How I design and implement a React table component.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3d93f12d812" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/design-and-implement-a-table-component-based-on-react-js-part-1-b3d93f12d812">Design and Implement a Table Component based on React.js - Part 1</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 Tips about lerna]]></title>
            <link>https://medium.com/shopback-tech-blog/5-tips-about-lerna-4186840093f2?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4186840093f2</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[monorepo]]></category>
            <category><![CDATA[lerna]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Fri, 21 Dec 2018 07:19:31 GMT</pubDate>
            <atom:updated>2018-12-25T02:47:06.464Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fon-bwJagsEm1Be4Dehk0A.png" /></figure><p>In this article, I will introduce some common questions about using lerna to create a javascript <a href="https://github.com/babel/babel/blob/master/doc/design/monorepo.md">monorepos</a> project.</p><blockquote>When writing this article, the version of lerna is 3.6.0</blockquote><p>If you want to have an independent workspace to play lerna, I will recommend to visit <a href="https://github.com/AllenFang/lerna-tutorial-example">lerna-playbook</a> which is a docker environment for lerna I created before. In addition, you can also read <a href="https://medium.com/shopback-engineering/lerna-tutorial-series-brief-f77f40c5777f">this post</a> as well.</p><h3>How to get started?</h3><p>When you clone the project first time, you have to install all the dependencies and link any cross-dependencies. Do not execute npm install in each packages when initialize, <a href="https://github.com/lerna/lerna/blob/master/commands/bootstrap#readme">lerna bootstrap</a> will help you.</p><p>You can consider to add lerna bootstrap in postinstall command, that will be helpful in some cases, for instances: <a href="https://github.com/react-bootstrap-table/react-bootstrap-table2/blob/master/package.json#L8">https://github.com/react-bootstrap-table/react-bootstrap-table2/blob/master/package.json#L8</a></p><p>If you project is not initialized yet, <a href="https://github.com/lerna/lerna/blob/master/commands/init#readme">lerna init</a> can help you to create new lerna project!!</p><p>If you project is initialized, but you have create a new package, please check <a href="https://github.com/lerna/lerna/tree/master/commands/create#readme">lerna create</a>!!</p><h3>How to install a dependency?</h3><p>This is most common question when develop use lerna in their project first time. You still can use yarn or npm to install dependencies in each package but it’s a little bit inefficiency. Lerna give you <a href="https://github.com/lerna/lerna/tree/master/commands/add#readme">lerna add</a> command so that you can install a dependency to either all packages or specific packages.</p><p>For example:</p><pre>$ lerna add react</pre><p>This command will install react to all the packages. If the dependency is just for development, you can add --dev option.</p><p>In addition, if the dependency is just for some specific packages, you can use --scope option:</p><pre>$ lerna add react --scope=module1</pre><p>However, there will be another question is about can I install react in multiple packages? <a href="https://github.com/lerna/lerna/issues/1608">The answer seems negative</a>. You have to execute lerna add one by one or use prefix:</p><pre>$ lerna add react packages/react-bootstrap-*</pre><h3>How to run a command in each package?</h3><p>Lerna provide <a href="https://github.com/lerna/lerna/tree/master/commands/exec#readme">lerna exec</a> for developer to run any command in all packages. For example:</p><pre>$ lerna exec -- rm -rf ./node_modules</pre><p>Be carefully, the double-dash is necessary. In addition, just like lerna add or other lerna command, lerna exec respect --scope option so that you can specify what package you want to execute.</p><p>There’s one more thing that if you are trying to execute some npm command ex: npm test, lerna give you another good way to execute: <a href="https://github.com/lerna/lerna/tree/master/commands/run#readme">lerna run</a></p><pre>$ lerna run test  # Same as execute npm test in each package</pre><h3>How to publish packages?</h3><p>Publish your packages to npm registery is most difficult and complicated things in monorepo management. <a href="https://github.com/lerna/lerna/tree/master/commands/publish#readme">lerna publish</a> command can help you to achieve that and it is worth to introduce lerna publish in another article.</p><p>lerna publish will try to publish the packages that have changed since the last release. Following is the things this command does:</p><ol><li>Add a new commit to change the version in package.json</li><li>Publish to npm</li><li>Create a git tag and push to remote git repository.</li></ol><p>Anyway, there are some interesting things can be introduced, I will share more in another article.</p><h3>How to publish to private npm registry?</h3><p>The most quick way is add --registry &lt;url&gt; option when do the publish. Sometime this is useful when you don’t want to configure the registry configuration in all package.json file.</p><p>However, the best practice is configure the <a href="https://docs.npmjs.com/files/package.json#publishconfig">publishConfig</a> in your package.json:</p><pre>&quot;publishConfig&quot;: {<br>   &quot;registry&quot;: YOUR_REGISTRY_URL_HERE<br>}</pre><p>In addition, make sure you configure your private registry url in .npmrc file:</p><pre>registry = YOUR_REGISTRY_URL_HERE</pre><h3>Conclusion</h3><p>In the next article, I will introduce more about lerna publish and give a long introduction about how to integrate react, webpack and lerna. Anyway, hope this article can help you and any feedback is welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4186840093f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/5-tips-about-lerna-4186840093f2">5 Tips about lerna</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why My Browser Extension was Taken Down?]]></title>
            <link>https://medium.com/shopback-tech-blog/why-my-browser-extension-was-taken-down-cfa5df922bff?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfa5df922bff</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[webextensions]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <category><![CDATA[troubleshooting]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Sun, 07 Oct 2018 16:31:24 GMT</pubDate>
            <atom:updated>2018-10-07T16:35:19.512Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why my browser extension was taken down?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*VTPbvaj6Aql9jf_OQ1oA7Q.jpeg" /></figure><p>ShopBack’s browser extension is a fantastic tool that offers shopper a quicker and faster way to earn cashback. You can visit our <a href="https://www.shopback.sg/cashback-button/how-it-works">tutorial page</a> if you are interested on it. However, since we launched our new browser extension to Chrome and Firefox, we faced three times taken down just within one year. So in below section, I will share what we experienced and how we tackled it!!</p><p>If you are a browser extension developer, you are supposed to read the policies of Chrome extension and Firefox add-ons.</p><ul><li><a href="https://developer.chrome.com/webstore/program_policies">Google Chrome WebStore Policies</a></li><li><a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/AMO/Policy/Reviews">Mozilla FireFox Add-ons Policies</a></li></ul><p>Google and Mozilla have similar policies especially on handling user privacy data. So I believe if you are following those common policies carefully, your extension will be safe on both platforms.</p><p>Before sharing our experiences, let’s introduce how Chrome/Firefox inform developer if your extension is taken down:</p><h4>Google Chrome</h4><p>Unfortunately, Google do not inform you if your extension is taken down. The ways to find out your extension is taken down is via <a href="https://chrome.google.com/webstore/developer/dashboard">Chrome WebStore Dashboard</a> or the extension install page.</p><h4>Mozilla Firefox</h4><p>Mozilla is much friendly on this part, they will send email to you and explain a roughly reason. So you just keep watch your email and don’t let it go into the spam.</p><p>In the next, I’m going to share three times take down experiences from Google Chrome and Mozilla Firefox.</p><h3>Obfuscate code</h3><p>In Chrome policy, they describe it as <strong>Code Readability Requirements</strong>. Actually, they allow you to minify the source code in the following forms:</p><ul><li>Removal of whitespace, newlines, code comments, and block delimiters</li><li>Shortening of variable and function names</li><li>Collapsing files together</li></ul><p>However, this policy is not only apply on your source code but also all the external code or resource fetch by extension.</p><p>Last time, the reason of our extension was taken down is because we were accidentally include a Font Awesome library via Font Awesome’s Free CDN. Here is how we use Font Awesome in extension popup:</p><pre>&lt;script defer src=&quot;https://use.fontawesome.com/releases/v5.0.6/js/all.js&quot;&gt;&lt;/script&gt;</pre><p>This external resource is minified and uglified. Seems like extension reviewer doesn’t like it and we fixed this issue via a directly reference to Font Awesome source file. Anyway, if there’s a third party library is bundled in one file, that is fine, for example: we use react and use webpack to bundle the source code. So just be careful about those external resources.</p><h3>User Data Privacy</h3><p>Google and Mozilla they emphasize the user’s privacy in the their policy. In fact, your extension is able to collect the user’s data or behavior absolutely. However, user also have right to disable this mechanism if they don’t like. Because ShopBack’s extension is very care the user’s experiences and behaviors when they are shopping, so we did collect a lots of shopper’s information but we didn’t give them a way to disable this mechanism. Hence, we faced the second time take down by Google.</p><p>So.. How to fix it? It’s simple, just implement a <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/user_interface/Options_pages"><strong>Options page</strong></a> to let user have ability to enable/disable the mechanism of collecting user data.</p><h3>Package Extension</h3><p>The third time take down is from Firefox and it was a little bit tricky. The reason is there are something different between our uploads and their builds. Actually, Firefox will ask you to upload the source code and add-on when you publish. The third time taken down is they found the result of building is different from the upload one. Yes.. it’s a mistake from engineer. However, it illustrate that Mozilla’s review is much strict on the final build files. BTW, We also faced an issue that their reviewer can not build our extension cause of some environment problem.</p><p>So the lesson learn is we provide a docker in the source code and write a README to instruct them to build the our extension in a right way. Anyway, just make sure the builds are always the same in a release.</p><h3>Conclusion</h3><p>In the past of year, I think the most difficult thing is figure out why your extension is taken down by <strong>Google</strong>. Because they don’t say anything usually. So we just learn a lesson when we face a taken down every time. Anyway, developer should pay more attentions on the policies when you develop a browser extension.</p><p>It’s welcome to share any taken down experiences in below comments :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfa5df922bff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/why-my-browser-extension-was-taken-down-cfa5df922bff">Why My Browser Extension was Taken Down?</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Selenium & WebExtension - Installation]]></title>
            <link>https://medium.com/shopback-tech-blog/selenium-webextension-installation-1a0e83694bbe?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a0e83694bbe</guid>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[browsers]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Mon, 03 Sep 2018 03:25:44 GMT</pubDate>
            <atom:updated>2018-09-03T06:35:35.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*U3JLcJ7OgkLlI3CQFhB7ew.png" /></figure><p>Recently, we built a E2E test environment with selenium@4.0.0-alpha.1 and wrote some test suites for testing ShopBack’s <a href="https://chrome.google.com/webstore/detail/shopback-cashback-button/hjngckebbndpdeeakdgohmcdnecidcjk">browser extension</a>. In the following section, we will go through how we install a local browser extension into Google Chrome and Mozilla Firefox which initialized by Selenium.</p><h3>Google Chrome</h3><ul><li>selenium@4.0.0-alpha.1</li><li>chromedriver@2.41.0</li></ul><p>Google Chrome is much simpler than Firefox, just prepare your local extension as well then write few selenium code like below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1cb9a71f659fea88819f2b96890daa14/href">https://medium.com/media/1cb9a71f659fea88819f2b96890daa14/href</a></iframe><p>However, How will you do if you want to install extension after chrome start up? Google Chrome doesn’t allow you to load a local extension after it started, instead a .crx file is acceptable. So what is crx file? Actually, it’s just kind of archive file like .zip. It’s almost the same thing for Firefox’s .xpi file.</p><p>You can get your .crx file through <a href="/service/chrome://extensions">chrome://extensions</a>:</p><ol><li>Enable the developer mode on the right top corner.</li><li>Click the Pack extension button. A dialog appears.</li><li>Choose the path of your local extension.</li><li>Done, Chrome will generate a .pem and .crx file the parent path of step three.</li></ol><p>Now, you can get this thing done by this .crx :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edc2ad0dc38e2d2d9d1a5658f4406818/href">https://medium.com/media/edc2ad0dc38e2d2d9d1a5658f4406818/href</a></iframe><h3>Mozilla Firefox</h3><ul><li>selenium@4.0.0-alpha.1</li><li>geckodriver@1.12.1</li></ul><p>Firefox is a bit tricky when install browser extension through Selenium. You are not allow to install local browser extension into Firefox even you disable the xpinstall.signatures.required like below:</p><pre>const firefoxOption = new firefox.Options();</pre><pre>firefoxOption.setPreference(&#39;xpinstall.signatures.required&#39;, false);</pre><p>Fortunately, you still can achieve it in Firefox Developer Edition<strong><em>(a.k.a AURORA)</em></strong>. Firstly, you have to finish is install Developer Edition from <a href="https://www.mozilla.org/firefox/developer/">https://www.mozilla.org/firefox/developer</a></p><p>Secondly, it is supposed to be a <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/manifest.json/applications">application property in manifest.json</a>. For example:</p><pre>&quot;applications&quot;: {<br>    &quot;gecko&quot;: {<br>        &quot;id&quot;: &quot;ANY_ID_HERE&quot;,<br>        &quot;strict_min_version&quot;: &quot;52.0&quot;<br>    }<br>}</pre><p>After finish above two steps, when you install a local extension, you can not give a extension folder just like case of chrome. You are supposed to assign a .xpi file. It is almost same as .zip, you can just archive your extension to .zip file then change to .xpi file.</p><blockquote>When you archive the extension, please do not archive the root folder of local extension, instead, you should select all the extension assets then archive. ref: <a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Package_your_extension_">https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Package_your_extension_</a></blockquote><p>The last step is initializing a Developer Edition through Selenium then install extension:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68500dfa8049d7f9b88a89f311f2e867/href">https://medium.com/media/68500dfa8049d7f9b88a89f311f2e867/href</a></iframe><p>The error message shown below appears when running, it means selenium can not locate Firefox:</p><p><strong><em>Error: Could not locate Firefox on the current system</em></strong></p><p>You can config the firefox bin path manually via following command:</p><pre>firefox.Channel.AURORA.darwin_ = PATH_TO_FIREFOX_BIN;</pre><p>In the Mac OSX, the default path should be:</p><pre>/Applications/Firefox Developer Edition.app/Contents/MacOS/firefox-bin</pre><p>Anyway, as I know there still are couple ways to install the local extension to firefox. However, some of them are tricky workaround or only work on Selenium 3. I will share more in this post if I get some updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a0e83694bbe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/selenium-webextension-installation-1a0e83694bbe">Selenium &amp; WebExtension - Installation</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case Study: A Monolithic Software Refactoring]]></title>
            <link>https://medium.com/shopback-tech-blog/case-study-a-monolithic-software-refactoring-787b8314a2e0?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/787b8314a2e0</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Tue, 24 Jul 2018 02:48:28 GMT</pubDate>
            <atom:updated>2018-09-01T11:11:16.132Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*qcjF3XT7E-wOI0hw4jb4uQ.jpeg" /></figure><p>In this article, I just want to share a short journey about refactoring a monolithic eCommerce software. I will not too focus on how to use CQRS or DDD, because a huge software refactoring is not just about engineering but also people, decision, organization and culture. Just share my thoughts.</p><h3>Background</h3><p>I was an engineer in this team and responsibility is maintain a large Checkout system. It’s a B2C platform and already exists for more than five years and developed on Ruby on Rails. We were a small team, except me, the other engineers we were all new graduate and all of us involved Ruby development less than one year. This system was handover from the other team about six months ago.</p><p>This software is just a monolithic MVC structure, it means</p><ul><li>6000+ lines in one controller file and have other controllers…</li><li>4000+ lines in one model file and have other models…</li></ul><p><strong><em>There’re still a lots of bad smells that I don’t want to write them out</em></strong>. In fact, this situation already introduced a lots of bugs and previous engineers just try to pile up their codes for fixing bugs or features everyday. Overall, the software quality is pretty low. However, this system still have huge orders per day!!</p><h3>What’s Wrong?</h3><p>After I took over this system, I started feel there are something bad:</p><ul><li><strong>No Tiers</strong>: Actually, we have some ActiveResource or some other rails modules, but overall it is still a bloated MVC.</li><li><strong>No ACLs</strong>: Our system rely on other APIs to interact on data. We don’t have to connect the database. However, all the API responses or external data are spread to everywhere without any wrapped.</li><li><strong>Weakness Domain Modeling:</strong> Only few business logic is in the right place. Actually, you can find a lots of important logic everywhere.</li><li><strong>No boundary/Context</strong>: Just like ACLs. We have to avoid the core software is corrupted by any irrelevant things. We should consider this in any tiny thing. For example: You can find Rails Params or Routes everywhere.</li><li><strong>Bad Flexibility</strong>: Functions and Class have heavy dependencies. Hard to extend and lack of abstraction design. There’re too many fragment functions, some are over used some are useless.</li></ul><h3>Available Solutions</h3><p>I think nobody want to contribute on this codes. The risk is pretty high when you do any change even the change is minor. So I started to think how we can overcome this bad situation?!</p><p>At that moment, I had a strong feeling that we probably can drop current system and rewrite a new one:</p><blockquote><strong>Keep maintain the legacy one and develop a new one at the same time</strong></blockquote><p>However, this solution is nearly impossible because below concerns:</p><ul><li><strong>Resources</strong>: Maintain and develop new one is inefficient sometime</li><li><strong>Organization</strong>: Not all the managers agree this solution.</li><li><strong>Culture</strong>: Not everyone embrace the big changes.</li></ul><p>There’re too many restrictions in the real world. So the question now will be: “How we refactoring on a monolithic software?“</p><blockquote><strong>Domain Driven Design, DDD</strong></blockquote><p><a href="https://en.wikipedia.org/wiki/Domain-driven_design">DDD</a> is just a way to help you build a flexible software. Which not mean you can always get success when you adopt it. Before adopt DDD, I had to look around our team and current project status.</p><ul><li><strong>Project Status</strong>: Only one month handover, all the engineers and product manager still can’t understand all the Checkout logic. This is because we don’t have a completed handover and lack of documents.</li><li><strong>Engineer Level</strong>: If you team member is not familiar with object-oriented programming, it will be a high risk when you adopt DDD. In our team, they don’t have enough object-oriented experiences on programming, design and thoughts.</li></ul><p>Anyway, we can give our member a completed learning path to let them can master on OOP and DDD. However, it’s also too inefficient. In addition, DDD is not very easy to adopt it because it’s not just programming.</p><p>In fact, DDD is best solution except for rebuilding the whole codebase. But this path is too long and also we definitely need more domain and DDD experts to join.</p><h3>A Progressive Solution with CQS</h3><p>There were too many perspectives to be concerned when you refactor this software. Finally, the most concern is people and how you start. Following is just my step by step plan:</p><h4>Add Boundary</h4><p>In this monolithic software, the first thing we need to do is establish the boundary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJw5G9G6vhwrwz23IccJFw.png" /></figure><p>There’re two bounded layer being add in above diagram: <strong><em>ACL</em></strong> and <strong><em>Facade</em></strong>.</p><h4><strong><em>ACL</em></strong></h4><p>We have a lots of APIs which provided by other teams. However, after we get the response of APIs, we always return a plain Ruby object or a JSON data to the caller directly. It’s really a bad practice if there’re any changes on property name or type, because these response data already spread to everywhere.</p><p>It’s important to establish this layer for avoiding core service/domain being corrupted by other contexts. ACL have to restrict the API’s changes in this layer and return meaningful data(Wrapped class or Adapter) to the caller, for example: OrderDetail or ShippingMethods.</p><h4><strong><em>Facade</em></strong></h4><p>Facade is just a kind of context interface for Controller, for instance: OrderFacade, ShippingFacade. This layer is intended for avoiding the core domain models to know any details about Rails’s Controller or Param, Route etc. In this layer, every payload should be described as a <a href="https://en.wikipedia.org/wiki/Data_transfer_object"><em>DTO</em></a> and the return value should be a <em>VO, View Object</em>.</p><p>Following is our mainly purposes for Facade:</p><ul><li>A boundary between controller and core model</li><li>Integrate or construct multiple different processes in the Facade.</li><li>Core service should be unawareness of any Rails frameworks.</li></ul><h4>Progressive Refactoring with CQS</h4><p><a href="https://en.wikipedia.org/wiki/Command%E2%80%93query_separation">CQS</a> is a pretty simple pattern:</p><blockquote>It states that every <a href="https://en.wikipedia.org/wiki/Method_(computer_science)">method</a> should either be a <em>command</em> that performs an action, or a <em>query</em> that returns data to the caller</blockquote><p>In our system, more than 95% actions is about query. For example, get shipping &amp; payment methods, amount calculation, invoice, orders etc. This could be a reason to leverage on CQS. But mainly reason for our teams are:</p><ul><li>Easy to understand and implement</li><li>Flow/Process division is easier</li><li>Easy to become a standard for every engineers.</li></ul><p>In the beginning, we choose one case to refactor(ex: Get shipping methods) and start to define the component and glossary as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*hU_GfenjTXzleg44rgzIkA.png" /></figure><blockquote>In each refactoring case, we extract the domain models and made it more flexible as possible as we can.</blockquote><blockquote>This is a most easiest way to let every members to involve a case and also understand the business logic deeply.</blockquote><p>Every time we refactoring a case, we are closer to with this system and it’s good also for designing the domain models. Even our member didn’t have much experiences on OOP but they still could learn OOP skills in each refactoring case. <strong><em>Not only level up the software but also the engineers.</em></strong></p><h4>Model Design with OOAD</h4><p>This is the most hardest thing when we leveraged this solution, because we separated refactoring by each single case. It’s high risky to design the model in this situation. However, we still had a lots knowhow so that we can avoid to design wrong models at the beginning. Anyway, be careful about domain modeling, flexibility is first thing you have to concern.</p><p><a href="https://en.wikipedia.org/wiki/Single_responsibility_principle"><strong>Single Responsibility Principle, SRP</strong></a> is a important concept for me when design a class and <a href="https://www.google.com.tw/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=2&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwi2lOzKgLXcAhWM7mEKHWO2BaYQFgg2MAE&amp;url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FSoftware_design_pattern&amp;usg=AOvVaw350ZxXDF6fq0moCQsGF9Wb"><strong>Design Pattern</strong></a> can help you to plan a flexible design.</p><blockquote>Extract a common logic to a single class or function is not real modeling. It will make your code unmaintainable and unreadable sometime.</blockquote><p>Above just a feeling when I refactoring or modeling domain. Think following question twice or more when you extract a common logic. Because it maybe an important logic.</p><blockquote>Where this logic should be?</blockquote><p>In addition, keep in mind:</p><blockquote>* Make thing simple instead complex</blockquote><blockquote>* Loose coupling and loose dependency are king rules</blockquote><h3>Conclusion</h3><p>Software refactoring is not only a kind of art but also a hard skill. For me, the hardest thing are people and the first step, sometime you should pay more attention on it. Anyway, enjoy the journey and you will find the interesting parts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=787b8314a2e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/case-study-a-monolithic-software-refactoring-787b8314a2e0">Case Study: A Monolithic Software Refactoring</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How ShopBack Build and Package a Browser Extension]]></title>
            <link>https://medium.com/shopback-tech-blog/how-shopback-build-and-package-a-browser-extension-3b228f0c10ba?source=rss-f0510ed38e8c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b228f0c10ba</guid>
            <category><![CDATA[webextensions]]></category>
            <category><![CDATA[ecommerce-software]]></category>
            <category><![CDATA[ecommerce-web-development]]></category>
            <category><![CDATA[front-end-development]]></category>
            <dc:creator><![CDATA[Allen Fang]]></dc:creator>
            <pubDate>Tue, 17 Jul 2018 09:54:17 GMT</pubDate>
            <atom:updated>2018-09-01T11:12:06.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*UwjStvQLMArNsnqfX-6DmQ.png" /></figure><p>ShopBack leverages on <a href="https://developer.mozilla.org/en-US/Add-ons/WebExtensions"><strong>WebExtensions</strong></a> to build a cross-browser extension in one source code. In addition, we also use <a href="https://reactjs.org/"><strong>React.js</strong></a> to implement the extension UI. In this sharing, we are going to introduce how we build and package a eCommerce browser extension for Google Chrome and Mozilla Firefox.</p><h3>Build &amp; Package Extension</h3><p>In ShopBack browser extension team, we use <strong>webpack@2</strong> for bundling the source code and use <strong>gulp@4</strong> to establish the build/package flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rvsv-11uNZip1UfQKxK6EA.png" /></figure><p>In the following, we divide above flow into five main topics:</p><ul><li>Obfuscate and uglify code</li><li>Manage Webpack chunks with WebExtension</li><li>Manage extension assets</li><li>Attach source code</li><li>Work on locales(Special case)</li></ul><h4>1. Obfuscate and uglify code</h4><p>ShopBack’s browser extension is for customer to easily get cashback rewards when they are shopping. It’s kind of a eCommerce extension, sometimes we have to protect our source code to avoid to be inspected too easier by our competitors.</p><p>uglifyjs-webpack-plugin satisfies for most of cases. However in ShopBack, we also use uglifyOptions.mangle.properties to do more advanced uglified for some functions or variables name:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17d1d580c1e3d1c04edd181d91a0c929/href">https://medium.com/media/17d1d580c1e3d1c04edd181d91a0c929/href</a></iframe><p>However, the value of constant variable which can not be uglified. So there’s one more gulp task for dealing with this by gulp-replace to replace some sensitive value, then in the bundling stage, we use NormalModuleReplacementPlugin to replace the modules dynamically.</p><p>For example:</p><p>If we have a constant.js which have some sensitive content, this gulp task will generate a constant.obfuscated.js in the same folder then webpack will select this new source file via NormalModuleReplacementPlugin .</p><p>For developers, we just write code as usual but config which content should be obfuscated when building extension.</p><h4>2. Manage Webpack chunks with WebExtensions</h4><p>In our practices, the <strong>background</strong>, <strong>popup</strong> and <strong>option scripts</strong> have their one chunk only. For the <strong>content scripts</strong>, we have several content scripts so it’s straightforward to divide them into different chunks.</p><p>About webpack production bundling, we usually output the final chunk name with a SHA hash to avoid the caching. In this situation, we have to patch the manifest.json to meet the final chunk name.</p><p>So firstly, we need to collect every chunk file names after webpack bundling:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e83fab32ce056f903b670403143ce4c/href">https://medium.com/media/4e83fab32ce056f903b670403143ce4c/href</a></iframe><p>Then we patch the manifest.json with correct chunk file name via another gulp task:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef8c05ba2a38ff03489cbdc467bb6b6c/href">https://medium.com/media/ef8c05ba2a38ff03489cbdc467bb6b6c/href</a></iframe><p>So final result may look like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/83ef1430684ccae3b6d8e6cfe144ca10/href">https://medium.com/media/83ef1430684ccae3b6d8e6cfe144ca10/href</a></iframe><h4>3. Manage extension assets</h4><p>An extension usually includes some of assets, like html, css, image etc. Basically, you can put those static assets in your extension folder then config in the <a href="https://developer.mozilla.org/en-US/Add-ons/WebExtensions/manifest.json/web_accessible_resources"><strong>web_accessible_resources</strong></a>.</p><p>In ShopBack extension, we treat these static assets as kind of module so that we can use these images though resolved by webpack.</p><p>So there’s only manifest.json and few extension icon in our extension base folder. Following is how we handle the static assets</p><ul><li>Resolve HTML via html-webpack-plugin</li><li>Resolve image in Javascript via url-loader</li><li>Resolve image in SCSS resolve by resolve-url-loader</li></ul><p>The benefit is we don’t config <a href="https://developer.mozilla.org/en-US/Add-ons/WebExtensions/manifest.json/web_accessible_resources">web_accessible_resources</a> every time when there’s new asset is added. Everything resolved by webpack.</p><h4>4. Attach Source Code</h4><p>It is important thing if you have to publish your extension to Google Chrome. Actually, most of browser extension policy tell you <strong><em>NOT minify you source code</em></strong>. But if you did, you had better to upload your source code when publish or not they can taken down your extension.</p><blockquote>Mozilla Firefox did much good on this part because they have a field to ask you to upload your source code in the 7 day in publish page.</blockquote><blockquote>Google Chrome doesn’t ask you to upload when publish.</blockquote><p>If you really don’t want to attach the source code into the package, you probably can give Google reviewer a private download link when you are publishing.</p><h4>5. Work on locales</h4><p>This is special case in ShopBack, because Chrome extension allow you to specify a region to publish. So in our build process, we will build several chrome extension packages per country. You can easy to implement a distributed task via gulp.series or gulp.parallel.</p><p>For different countries, just few different parts as below:</p><ol><li>The homepage_url in manifest.json</li><li>The default_locale in manifest.json</li><li>What kind of locales need to be supported</li></ol><p>So in our special Chrome build process, we will patch the manifest.json according to corresponding information. All of these gulp tasks are generated dynamically. So if ShopBack have a new marketplace to launch, we just setup above config, that’s it!! It’s not necessary to write other gulp task.</p><p>If you are interesting on how to generate a gulp.series tasks dynamically, you can refer our previous <a href="https://medium.com/shopback-engineering/https-medium-com-shopback-engineering-construct-gulp-series-tasks-dynamically-db744b923229"><strong>post</strong></a>.</p><h3>Conclusion</h3><p>In this article, we share a build flow that we done in the past six months. There’s sill have few things we can improve, we will share it in near future. Any feedback is welcome and in the next posts, we will share some experiences after we publish our extension.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b228f0c10ba" width="1" height="1" alt=""><hr><p><a href="https://medium.com/shopback-tech-blog/how-shopback-build-and-package-a-browser-extension-3b228f0c10ba">How ShopBack Build and Package a Browser Extension</a> was originally published in <a href="https://medium.com/shopback-tech-blog">ShopBack Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>