<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hiroki Gota on Medium]]></title>
        <description><![CDATA[Stories by Hiroki Gota on Medium]]></description>
        <link>https://medium.com/@hiroki-gota?source=rss-a878d1a269c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iwCPovRjelKwMbo9_WLt6g.jpeg</url>
            <title>Stories by Hiroki Gota on Medium</title>
            <link>https://medium.com/@hiroki-gota?source=rss-a878d1a269c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:05:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hiroki-gota/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Here’s Why You Should Start Your Career with Lapis]]></title>
            <link>https://medium.com/lapis/heres-why-you-should-start-your-career-with-lapis-31272c642a64?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/31272c642a64</guid>
            <category><![CDATA[sofware-development]]></category>
            <category><![CDATA[australia]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[graduate-students]]></category>
            <category><![CDATA[news]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Mon, 16 Aug 2021 08:18:30 GMT</pubDate>
            <atom:updated>2022-06-21T05:53:18.545Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*t1P0Ild4ECP2-e3VUNK8Gw.png" /></figure><h3>Are you a fresh grad looking for a job?</h3><p>Hey 🌞! Are you a fresh grad looking for your first job in Software Development in Australia? You might:</p><ul><li>Enjoy problem-solving</li><li>Have hands-on experience in building apps using web and cloud technology</li><li>Be an excellent learner and enjoy learning by coding or talking</li><li>Want to master Software Service end to end</li><li>Be passionate about building apps</li><li>Have a GitHub account and use GitHub Actions</li></ul><p><strong>Spatial Vision’s 2022 Graduate Cadet Program now has a Software Development</strong> <strong>stream</strong>, working with our <strong>Lapis brand</strong>. The program offers A 12-month <strong>paid work placement</strong> for graduates with the opportunity to design and deliver your own project.</p><h3>1. Mentoring and Coaching</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*EdylmQj--tVQ6eJJBU0N4g.png" /></figure><p>At Lapis, you will learn everything about Software Development through mentoring and coaching. You will be a part of our experienced development team and you will be paring with a senior team member from day 1.</p><h4>Share your knowledge and grow</h4><p>Mentoring and coaching are key methods of our knowledge sharing across the team, not only for grads but also for any member who is willing to step up and take on a new challenge.</p><h3>2. Balanced, Cross-Functional Team</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*dYTU2yiy8tjFg_uTqc2jxA.png" /></figure><p>Our cross-functional team is made up of specialists in the fields including Product Owners, Business Analysts and Specialists in QA, Security, Mobile and DevSecOps.</p><h4>You’ll never be alone</h4><p>You will be with your cross-functional team every step of the way through for a range of service delivery projects, working with leaders, seniors and juniors alike so you can learn from a diverse and enriched learning experience.</p><h4>BdDevSecOps (you sell it, you build it and you run it)</h4><p>Thanks to the fixed service delivery team, you will also learn about Business Development and Operational pipeline end to end. We share it all within the team and encourage you to bring your idea to further help our clients.</p><h3>3. Diversity and Inclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iE0IEZs7qJ9bXALBxuONiQ.png" /><figcaption>Lapis Team</figcaption></figure><p>Our team is culturally diverse and t<em>rust</em> is incredibly important to us. We do what we can to ensure everyone feels <em>safe</em> and supported in their opinions and ask for help when needed. We support the <a href="https://sdgs.un.org/goals/goal5">Sustainable Development Goals</a>, with a strong focus on <em>Diversity and Inclusion</em>. In pursuit of this goal, we’re always looking to support more <a href="https://wit.org.au/"><strong><em>Women in Tech</em></strong></a><strong><em> </em></strong>and develop a better gender balance internally. We encourage all people of all genders and cultural backgrounds to join us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O2dAUtZyLSGDejEe3Rj8Cg.jpeg" /></figure><h3>4. Work-Life Balance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*I0AISxGA9MEbGytXjzUM6A.png" /></figure><p>Work-life balance is very important to us, and we believe it’s a key factor in developing a successful career. Your career journey is a marathon and not a sprint. We are strong proponents of the idea that can achieve your professional goals and still have a life outside of work.</p><h4>The 10,000-hour rules</h4><p>Software development can be a complex business and let’s be honest - it takes time to become a <em>Ninja</em>, along with deliberate, consistent practice and strong mentoring and coaches.</p><p>This does not mean you should be working late or on the weekend. At Lapis, we encourage you to focus on one task at a time and finish on time so you can come back the next day with a fresh mind.</p><p>Some people might spend their time for further learning in tech and others spent their time for recreational or with their family. You can learn so many other things outside your professional job, and we encourage you to do so!</p><h4>Taco, Ping-Pong and Social Events</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*rPdzFL2v8PGxQRYCFqfVKA.png" /></figure><p>We have an honest and direct reward system using <a href="https://www.heytaco.chat/">Taco</a> in our slack. You might receive a monthly staff award if you have achieved something amazing.</p><p>We love playing Ping-Pong at our office in the CBD and have a Friday drink and fun whenever we get together. Our leader (your boss), <a href="https://www.linkedin.com/in/imranqazi/">Imran</a> often arranges a quarterly team dinner with a fun social event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YZh3tK9Sin9MMxOB0Y6S-A.gif" /></figure><h3>5. Career Opportunities</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/128/1*R0XbFFnm67RgBytfScoa1A.png" /></figure><p>So what’s it really like to be a graduate at Lapis? Ask our most recent grad developer Ti:</p><ul><li>Ti Zou: <a href="https://www.linkedin.com/in/ti-zhou/?originalSubdomain=au">Senior Software Engineer at Lapis</a></li></ul><p>You may be wondering, what opportunities can come my way from being a graduate at Lapis? Here are some people who started their careers with Spatial Vision. Reach out to them for more 😸!</p><ul><li>Srishti Chaudhary: <a href="https://www.linkedin.com/in/srishti-chaudhary-88058224/">People, Transformation &amp; Consulting Expert at ANZ</a></li><li>Anastasia Foulidis: <a href="https://www.linkedin.com/in/anastasia-foulidis-5a690165/">QA Lead at Lais</a></li><li>Linda Warui: <a href="https://www.linkedin.com/in/linda-warui/?originalSubdomain=ke">Head of Shared Technology at Interswitch Group</a></li><li>Ryan Xiao: <a href="https://www.linkedin.com/in/ryan-haoruo-xiao-796221a0/?originalSubdomain=au">Software Engineer at Atlassian</a></li></ul><p>Apply to be a 2022 Graduate today 🙇!</p><p><a href="https://spatialvision.com.au/graduate-cadet-program/">Graduate Cadet Program | Software, Geospatial and Consulting Streams Available</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31272c642a64" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/heres-why-you-should-start-your-career-with-lapis-31272c642a64">Here’s Why You Should Start Your Career with Lapis</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate Kotlin backend APIs into Nx workspace]]></title>
            <link>https://medium.com/lapis/migrate-kotlin-backend-apis-into-nx-workspace-2b170c292ab8?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b170c292ab8</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[nx]]></category>
            <category><![CDATA[monorepo]]></category>
            <category><![CDATA[kotlin]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Fri, 11 Jun 2021 01:58:19 GMT</pubDate>
            <atom:updated>2021-06-11T01:58:19.419Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hFwTXRnfLjy-1NFxEYUbJg.png" /></figure><p><a href="https://nx.dev/">Nx</a>, a TypeScript based monorepo tool, is not only a great tool for Node.js based technology such as Angular, Ionic or NestJS but also works well with Kotlin based backend APIs with multiple modules.</p><h4>Modular Monolithic Kotlin Backend APIs</h4><p>At <a href="https://lapisit.com.au/">Lapis</a>, we are busy migrating existing <strong><em>manyrepo</em></strong> projects into the Nx workspace, in order to <a href="http://manyrepo/">simplify and streamline the development life cycle</a>.</p><p>We have recently built and released <a href="https://www.dpi.nsw.gov.au/__data/assets/pdf_file/0006/871737/AquaOnline-Stakeholder-Update-February-2019.pdf">AquaOnline</a>, a Commercial Aquaculture Management System for <a href="https://www.dpi.nsw.gov.au/">NSW DPI</a>. AquaOnline manages government administrative information for commercial aquaculture farms (land-based facilities) and leases (sea and rivers) with regulated permissions. The following diagram shows an initial domain model identified prior to the development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qgnHxe5iQd_YxRclQZk75Q.png" /><figcaption>Domain Model</figcaption></figure><p>We use a <a href="https://www.infoq.com/news/2020/01/monolith-architectural-drivers/">Modular Monolithic architecture</a> approach, with a single SQLServer database and multiple schemas (each schema contains feature specific entities) and run on <a href="https://aws.amazon.com/fargate/">AWS Fargate</a>, a managed Docker Container Service.</p><h3>Manyrepos by feature modules</h3><p>The following diagram shows the Kotlin-based Backend-For-Frontend APIs and feature modules, which use to have their own GIT repository and CI processes before migrating to the Nx workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rk2DxNu8D22ViH9sU_nsBw.png" /><figcaption>AquaAssist Backend API Modules</figcaption></figure><h3>Apps and Libs</h3><p>Nx workspace provides a standard folder structure to organise project artefacts. <strong>The apps</strong> <strong>folder</strong> contains frontend and backend applications, such as Angular or NestJS. <strong>The libs folder</strong> contains any reusable feature modules, which could be shared between the apps, both the frontend and backend. We can simply follow the same convention and place our Kotlin based modules in apps and libs folders.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*66whcpJS5C9oBUZUiCaDZQ.png" /><figcaption><a href="https://nx.dev/latest/angular/getting-started/nx-setup">https://nx.dev/latest/angular/getting-started/nx-setup</a></figcaption></figure><h3>Generators to create apps and libs</h3><p>Nx comes with a set of generators to scaffold an application or a library and configure it in nx.json, workspace.json, Typescript, ESLint and Jest configuration files. We could even create <a href="https://nx.dev/latest/angular/nx-plugin/overview">a custom generator plugin</a> for Kotlin. (The <a href="https://nx.dev/nx-community">Nx community plugins</a> have a generator for SprintBoot).</p><h4>Generate Kotlin based Feature Library</h4><p>Since we’re migrating existing repositories, we use a Node generator to scaffold a library instead of building a new generator.</p><p><strong>Naming Conventions</strong></p><p>We decided to prefix Kotlin based modules with kotlin-, so we can differentiate them from other Typescript based modules. The following command will generate a new folder structure in the libs folder and configure it in the configuration files. We delete the generated files in libs/kotlin-lib-core and copy Kotlin resources from the existing repository.</p><pre>nx generate @nrwl/node:library --name=kotlin-lib-core --no-interactive --dry-run</pre><pre>CREATE libs/kotlin-lib-core/README.md<br>CREATE libs/kotlin-lib-core/.babelrc<br>CREATE libs/kotlin-lib-core/src/index.ts<br>CREATE libs/kotlin-lib-core/src/lib/kotlin-lib-core.spec.ts<br>CREATE libs/kotlin-lib-core/src/lib/kotlin-lib-core.ts<br>CREATE libs/kotlin-lib-core/tsconfig.json<br>CREATE libs/kotlin-lib-core/tsconfig.lib.json<br>UPDATE tsconfig.base.json<br>UPDATE workspace.json<br>UPDATE nx.json<br>CREATE libs/kotlin-lib-core/.eslintrc.json<br>CREATE libs/kotlin-lib-core/jest.config.js</pre><h3>nx.json</h3><p>nx.json defines projects and the generator adds the new project in it. The project specifies 2 attributes, implicitDependencies and tags. implicitDependencies specifies a list of dependant modules. <a href="https://nx.dev/latest/angular/cli/affected">Nx affected</a> command will build the module when the dependant module has been changed in the build pipeline. tags will limit the usage of the library for apps and libs having the same tag.</p><pre>“kotlin-lib-core”: {<br> “implicitDependencies”: [“kotlin-aquaonline-pom”],<br> “tags”: [“scope:kotlin”]<br>},<br>&quot;kotlin-lib-transaction&quot;: {<br>  &quot;implicitDependencies&quot;: [&quot;kotlin-lib-core&quot;],<br>  &quot;tags&quot;: [&quot;scope:kotlin&quot;]<br>},</pre><p>nx dep-grapth uses implicitDependencies to visualise your dependency graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lrPCXmP-EmXvkiY06O_nSg.png" /><figcaption>nx dep-grapth</figcaption></figure><h3>workspace.json</h3><p>workspace.json defines the project configuration such as the root folder (libs/kotlin-lib-core) and the builders, a set of tasks executed via Nx. Nx provides a set of builders for known architecture (Angular or NestJS), including build, serve, lint and test. We’ll follow similar DSLs in order to build a CI pipeline seamlessly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGaK9qjgdn69LIXNOKO0Vg.png" /><figcaption>The project configuration in workspace.json</figcaption></figure><p>The above configuration shows 4 tasks, lint, build, test and deploy. These can be executed as follows:</p><pre>nx lint kotlin-lib-core<br>nx build kotlin-lib-core<br>nx test kotlin-lib-core<br>nx deploy kotlin-lib-core</pre><h4>Run Commands Builder</h4><p>We use <strong><em>run-commands builder</em></strong>, which allows you to run any CLI. The following shows the build task details. The builder simply executes the defined <em>commands</em>. install.sh runs maven install within a Docker container; change the owner of the target folder and copy the result into the dist folder. This is mimicking how Nx builds a typescript based project and the transpiled files are stored in the dist folder.</p><pre>&quot;build&quot;: {<br>          &quot;builder&quot;: &quot;<a href="http://twitter.com/nrwl/workspace">@nrwl/workspace</a>:run-commands&quot;,<br>          &quot;options&quot;: {<br>            &quot;commands&quot;: [<br>              &quot;./install.sh&quot;,<br>              &quot;./chown.sh&quot;,<br>              &quot;mkdir -p ../../dist/kotlin-lib-core&quot;,<br>              &quot;mv ./target ../../dist/kotlin-lib-core&quot;<br>            ],<br>            &quot;cwd&quot;: &quot;libs/kotlin-lib-core&quot;,<br>            &quot;parallel&quot;: false<br>          }<br>        }</pre><p>You can define any tasks for a project or set of projects with the same architecture. For Kotlin APIs, we also want to build a Docker image and push it to a Container Registry.</p><p>The following shows slightly different tasks for a Kotlin API, including prepare-docker and build-push-docker tasks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5TvhQFNV6E5kS11DJKjsw.png" /><figcaption>The project configuration for the Kotlin application</figcaption></figure><h4>Force build order with strictlyOrderedTargets</h4><p>When you define your custom tasks, it’s important to specify the build order to ensure they run sequentially, as Nx, by default, executes tasks in parallel to minimise the total execution time. You can specify it using strictlyOrderedTargets in nx.json.</p><pre>&quot;tasksRunnerOptions&quot;: {<br>    &quot;default&quot;: {<br>      &quot;runner&quot;: &quot;<a href="http://twitter.com/nrwl/nx-cloud">@nrwl/nx-cloud</a>&quot;,<br>      &quot;options&quot;: {<br>        &quot;cacheableOperations&quot;: [&quot;build&quot;, &quot;lint&quot;, &quot;test&quot;, &quot;e2e&quot;],<br>        &quot;accessToken&quot;: &quot;...&quot;,<br>        &quot;canTrackAnalytics&quot;: false,<br>        &quot;showUsageWarnings&quot;: true,<br>        &quot;strictlyOrderedTargets&quot;: [<br>          &quot;build&quot;,<br>          &quot;prepare-test&quot;,<br>          &quot;test&quot;,<br>          &quot;lint&quot;,<br>          &quot;e2e&quot;,<br>          &quot;prepare-docker&quot;,<br>          &quot;build-push-docker&quot;,<br>          &quot;deploy&quot;,<br>          &quot;deploy-ec2&quot;<br>        ]<br>      }<br>    }</pre><h3>CI/CD Pipeline</h3><p>We now have a set of tasks that can be executed in the CI pipeline. We use the GitLab CI and execute them, by mapping these tasks with <strong><em>GitLab CI stages and jobs</em></strong>.</p><pre>stages:<br>    - init<br>    - init-test-db<br>    - build<br>    - test<br>    - build-push-docker<br>    - deploy</pre><p>You can simply execute nx affected command in each stage and jobs. Nx looks up any changes in the repository and only run tasks for projects which have changed or dependant on the changed projects.</p><pre>nx affected:build --base=HEAD~1 --head=HEAD --with-deps<br>nx affected:test --base=HEAD~1 --head=HEAD --with-deps<br>nx affected:build-push-docker --base=HEAD~1 --head=HEAD --with-deps<br>nx affected:deploy --base=HEAD~1 --head=HEAD --with-deps</pre><p>nx affected greatly simplifies the pipeline configuration for multi-module Kotlin-based APIs as you no longer need to worry about which repo should be pushed first or how to trigger another pipeline that is dependant on a particular feature module.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IDQavVf4Ur9F_icaxqRMbQ.png" /><figcaption>CI Pipeline</figcaption></figure><h3>Summary</h3><ul><li>Nx workspace can manage apps and libs using any technology, including Kotlin-based APIs</li><li>Nx Generators help you to configure a new project, even you don’t need scaffolded resources</li><li>Having a good naming convention will help to organise related apps and libs</li><li>implicitDependenciesspecifies Kotlin module dependencies and let you leverage nx affected command</li><li>run-commands builder lets you run any CLI via Nx on any platform (Linux, Mac and Windows)</li><li>strictlyOrderedTargets specified how the task should be executed sequentially</li><li>nx affected simplifies multi-module project pipeline by automatically discover changes and only run the pipeline that is affected</li></ul><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b170c292ab8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/migrate-kotlin-backend-apis-into-nx-workspace-2b170c292ab8">Migrate Kotlin backend APIs into Nx workspace</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adapting Monorepo with Nx + Ionic + Nest + AWS Serverless + Gitlab CI]]></title>
            <link>https://medium.com/lapis/adapting-monorepo-with-nx-ionic-nest-aws-serverless-gitlab-ci-a7d7a34f9070?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7d7a34f9070</guid>
            <category><![CDATA[ionic]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[monorepo]]></category>
            <category><![CDATA[nx]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Wed, 07 Oct 2020 20:26:33 GMT</pubDate>
            <atom:updated>2020-10-21T04:29:05.193Z</atom:updated>
            <content:encoded><![CDATA[<h3>Simplify the development life cycle with Nx, a monorepo tool</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3jG7ytfUkkLFW13Y.png" /><figcaption>Nx: a monorepo tool</figcaption></figure><p>We’d like to share our recent experience with <a href="https://github.com/nrwl/nx">Nx</a>. Nx is a Typescript based monorepo tool, primarily supporting Node.js based frontend, backend apps, and libraries. The topics we are covering today:</p><ul><li>Why monorepo?</li><li>Nx: A monorepo tool</li><li>Create Nx workspace for a project</li><li>Build, lint, test, serve and run</li><li>Single package.json policy</li><li>Affected apps by dep-graph and git diff</li><li>Computation caching and Nx Cloud</li><li>API Gateway + Lambda ready package</li><li>Gitlab CI Pipeline for a monorepo</li><li>Lessons learned</li></ul><h3>Why monorepo?</h3><p><a href="https://www.lapisit.com.au/">Lapis (our company)</a> builds and manages custom end-to-end solutions for our clients and we use AWS and various cloud-based services.</p><p>Our architecture is a more <a href="https://www.infoq.com/news/2020/01/monolith-architectural-drivers/">modular monolithic approach</a> than fully decoupled microservices, as our solutions are often used for a small number of focused business users (100s ~ 10,000s) and scalability in both systems and team size is not our main concern.</p><h4>Our projects</h4><ul><li>10 ~ 20 active development projects per year</li><li>Intellectual property (IP) of the project artifacts belong to clients and we often need to hand over code repositories</li><li>Systems are typically in operations for 5 ~ 20 years and evolve over a period of time</li></ul><h4>Typical Application Components</h4><ul><li>Angular webapps (Role-based administrative business workflow)</li><li>Ionic Mobile apps (In the field offline-capable, native or PWA)</li><li>Express/NestJS/Kotlin APIs for webapps (BFF)</li><li>Common APIs and internal libraries</li><li>System integrations (Authentication and service integrations)</li><li>Database(s) including DDLs, SQLs and scripts for migrate/backup/restore</li><li>AWS resource configuration (CloudFormation templates), credentials, and documentation</li></ul><h4>Manyrepo, Angular workspace, and Meta</h4><p>We’ve been using <strong><em>manyrepo</em></strong> approach for a project like above, by creating a repository for each component. The project structure looks like below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VcRFPNNLrkUeFvFH" /></figure><h4>Manyrepo cons</h4><ul><li>Maintain configurations and scripts for <strong><em>each archetype</em></strong> such as building, testing, and deployment</li><li>Coordinating a release process can become complicated as the knowledge of what components must be released, relies on the development team</li><li>Setting up an entire project locally might take a long time as each component has its own README.md and has a variation (No standard process)</li></ul><h3>Nx: A mono repo tool</h3><p>Nx is a Typescript based mono repo tool, built on top of Angular DevKit (<a href="https://cli.angular.io/">CLI</a> and <a href="https://blog.angular.io/schematics-an-introduction-dc1dfbc2a2b2">Schematics</a>) and provides a workspace, CLI, a cloud-based computation caching, and a great language level IDE support. Nx was created by Angular team at Google and then the core members decided to start a company (<a href="https://nrwl.io/">Nrwl</a>) providing Nx toolchain, consulting and education.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/995/1*rXkDaVDIOUEP3bDkJE3DyA.png" /><figcaption><a href="https://nrwl.io/">https://nrwl.io/</a></figcaption></figure><h4>Why Nx?</h4><p>We looked at several<a href="https://github.com/korfuri/awesome-monorepo"> tools available</a> and found Nx supported most of our typical application components, Angular webapps, Ionic and Node.js based back-end (Express or Nest).</p><p>Nx website says ‘Develop like Google’. Apparently, Google manages their entire codebase in <a href="https://research.google/pubs/pub45424/"><strong><em>1 repository</em></strong></a> and this copy <em>sounds like</em> encouraging every other company should follow.</p><h4>One monorepo to rule them all vs One monorepo per project</h4><p>We look at a life-cycle of each project and how it should evolve over a period of time. The following are a few reasons we’ve decided to take 1 monorepo per project approach:</p><ul><li>We want to use the latest technologies for a new project and not constrained by existing modules and breaking changes before starting a new project</li><li>Some projects (systems) do not simply evolve. They are <strong><em>Fit for Purpose</em></strong> and users are happy for a long time.</li><li>Typescript, frameworks and libraries continuously evolve and you often get breaking changes (not backward compatible), which might require a substantial effort/cost to re-write</li><li>Having said that, it’d be <strong><em>ideal</em></strong> if you can <strong><em>afford</em></strong> to upgrade entire codebase for every dependency update so your team only need to manage 1 set of dependencies/platform. (It must be a <strong><em>dream</em></strong> of every developer who’s worked a little longer than 1 year in the Javascript development space!)</li></ul><h3>Create Nx workspace for a project</h3><p><strong>create-nx-workspace</strong> generates an Nx workspace, which is a directory containing your apps, libs, tests, configuration etc:</p><pre>npx create-nx-workspace@latest<br>npx: installed 193 in 30.053s<br>? Workspace name (e.g., org name)     <br>      blueberry<br>? What to create in the new workspace angular-nest<br>      [a workspace with a full stack application (Angular + Nest)]<br>? Application name<br>      blueberry<br>? Default stylesheet format   <br>      SASS(.scss)  [<a href="http://sass-lang.com/">http://sass-lang.com</a>]<br>? Use the free tier of the distributed cache provided by Nx Cloud?       Yes [Faster command execution, faster CI. Learn more at <a href="https://nx.app/">https://nx.app</a>]</pre><p>It generates a set of configuration files and apps, libs and tools subdirectories containing application artifacts. The Angular and NestJS apps are ready to serve locally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*2JMJC5YekgwYDuLpPCrytA.png" /><figcaption>Nx workspace</figcaption></figure><h4>Schematics: Generator</h4><p>Nx leverages <a href="https://www.npmjs.com/package/@angular-devkit/schematics">Schematics</a>, a tool from Angular DevKit, to generate boilerplate code for artifacts (apps, module, service, components etc.). The create-nx-workspace is essentially running a set of schematics within the workspace:</p><pre># Generate Angular app, called blueberry<br>nx g @nrwl/angular:app blueberry</pre><pre># Generate Nest app, called api<br>nx g @nrwl/nest:app api --frontendProject=blueberry</pre><pre># Generate a Node module, shared by Angular and Nest<br>nx g @nrwl/node:library api-interface</pre><h3>Build, lint, test, serve and run</h3><p>Nx uses git to determine what artifacts are <strong><em>affected</em></strong> by looking at git diff. Our typical development loop looks like the following using <strong><em>Nx tasks</em></strong>:</p><pre># Create a feature branch <br>git checkout -b feature/hello-world</pre><pre># Generate a set of components using nx generate + schematics<br>nx g &lt;schematics&gt;</pre><pre># Implement, test, serve (end 2 end test via browser)<br>nx test api # test api<br>nx serve api # run api locally</pre><pre># Commit, push (lint, test) and create a MR to develop branch<br>git commit and push...</pre><pre># Run a CI pipeline<br>nx affected:build<br>nx affected:test<br>nx affected:deploy</pre><p>You can create custom tasks using a pre-defined builder or create your own <a href="https://nx.dev/latest/angular/workspace/builders/creating-custom-builders">custom builder</a>.</p><p>This is a powerful concept as you can define any arbitrary tasks for your apps and run via Nx. For instance, creating a deploy task which deploys an API to AWS Lambda. There are many ‘builders’ developed by Nx community and we’ll be using it later to build a lambda ready zip file containing a NestJS app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*fNrXdii27wokNwY6ASYHuQ.png" /><figcaption>NestJS API tasks defined in angular.json</figcaption></figure><h3>Single package.json policy</h3><p>Nx manages all of your npm dependencies in a <strong><em>single root package.json. </em></strong>This decision is driven by a single version policy of Google and you can read it in detail in this <a href="https://github.com/nrwl/nx/issues/1777">GitHub issue</a> and <a href="https://dl.acm.org/doi/fullHtml/10.1145/2854146#body-5">this article</a>.</p><h4>Dependency hell?</h4><p>Our first impression was this could become an issue in a long term. You wanted to upgrade one of the frameworks, which used a library that was also used in another framework and upgrade of one introduce other breaking changes.</p><p>Fortunately, npm had already solved this, called <strong><em>peer dependency</em></strong>. You could learn it <a href="https://medium.com/@imdongchen/how-does-npm-handle-conflicting-package-versions-44f90950cca5">here</a>.</p><h3>Affected apps by dep-graph and git diff</h3><p>Another powerful command is called <a href="https://nx.dev/angular/guides/monorepo-dependency-diagrams"><strong><em>dep-graph</em></strong></a>, which shows the dependency graph of a workspace so you can visualise what libraries are used for an app or what apps are dependant on a library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSjKVhRBmhtqQ528KOqGhQ.png" /><figcaption>A dependency graph for a project, showing what apps used what dependencies</figcaption></figure><p>Nx uses a combination of <strong><em>projects</em></strong> and<strong><em>implicitDependencies</em></strong> in nx.json to identify dependencies and used for a companion CLI command called <strong><em>affected</em></strong>.</p><p>The goal of affected command is to <strong><em>only</em></strong> build, lint, test and deploy apps affected by changes in the repository. Nx simply uses git diff to identify the changes, find the affected apps by looking at the dep-graph.</p><p>We feel this affected feature alone worth using Nx as it is hard to track the changes and re-build only affected in a manyrepo strategy.</p><h3>Computation caching</h3><p>Nx supports a powerful concept called <strong><em>computation caching</em></strong>. It is a form of caching to avoid re-build, re-lint, re-test, re-deploy if it has been done once.</p><p><a href="https://nx.dev/angular/workspace/computation-caching">Nx documentation</a> says they need to know two things to avoid re-computing:</p><blockquote>1. We need to store the results of the computation. 2. We need to know when we are about to compute something we already computed before. There are three things that tell Nx if something has been computed before:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*xOARGCpLSD1NMqUSKnz8Jw.png" /><figcaption>Source code, Runtime and Args Cache Inputs</figcaption></figure><h4>Execute once output anywhere</h4><p>This concept is similar to <strong><em>Build once run anywhere</em></strong>, often mentioned in the Docker community. Nx computation cache is focusing on minimising the development loop for a monorepo environment, in which the codebase can be evolved significantly in size and you will want to save every second that can affect on the team’s productivity.</p><h4>Nx Cloud</h4><p>Nx computation cache naturally happens in a local environment and you can <strong><em>feel</em></strong> the difference every time you run build, lint or test. Nx takes a step forward and try to save <strong><em>computation across the development machines and CI environment</em></strong>, by caching the outputs of your computations on Nx Cloud.</p><p>The following screenshot shows the Nx cloud console, providing a summary of times spent on each build. This is a snapshot of a project which has been running for 4 sprints (10 days per sprint) with 2 ~3 full-time developers.</p><p>Unfortunately, we’re not saving too much time and the tool does not currently provide any more details, such as how many machines are used, their platforms, the reason they couldn’t save time etc. however it’s a great metric to capture/visualise across our development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*_r7GUhT5WwdoiDJh_gdAOw.png" /><figcaption>Trend summary in Nx Console</figcaption></figure><h3>API Gateway + Lambda ready package</h3><p>We often use AWS API Gateway and Lambda (serverless) as it is cost-effective and the development-deployment life cycle is simple and easy to maintain.</p><p>This repository, <a href="https://github.com/benMain/aws-serverless-fastify"><strong><em>AWS Serverless Express for Fastify</em></strong></a> provides a boilerplate code for Fastify and the author is kindly providing an <a href="https://github.com/benMain/aws-serverless-fastify-nest-example">example for Nest</a>JS.</p><p>The example provides 3 essential configurations to successfully deploy Node.js based backend API to Lambda:</p><ul><li><a href="https://github.com/awslabs/aws-serverless-express/blob/master/examples/basic-starter/cloudformation.yaml"><strong><em>CloudFormation template</em></strong></a> for creating API Gateway, IAM Role and Lambda</li><li><a href="https://github.com/awslabs/aws-serverless-express/blob/master/examples/basic-starter/simple-proxy-api.yaml"><strong><em>Open API (aka Swagger) specification</em></strong></a> for <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html">API Gateway Lambda proxy integration</a></li><li><a href="https://github.com/benMain/aws-serverless-fastify-nest-example/blob/master/src/lambda-entry-point.ts"><strong><em>Lambda entry point</em></strong></a> for passing HTTP Request to NestJS (Fastify)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*7uFeH_Q6TFWjYiC11NIzMw.png" /><figcaption>Minimum configuration files for API Gateway + Lambda</figcaption></figure><h4>Build API with Lambda entry point</h4><p>Nx builds NestJS using @nrwl/node:build, which produces main.js containing all of your implementations, but it does not include dependencies (node_modules). You can run it by node dist/apps/api/main from the Nx root directory.</p><p>The issue is the root node_modules contains modules for the whole project, including Nx, TypeScript and Angular, which are not required by NestJS.</p><p>There are several discussions in the Nx community for how to solve this issue. We recommend reading ‘<a href="https://github.com/nrwl/nx/issues/1777">package.json per app</a>’, which tells you why Nx does not support multiple package.json and how people solve this.</p><p>We’ve used <a href="https://github.com/Apployees/apployees-nx/#enhanced-node-builder----apployees-nxnode">Apployees-Nx</a>, @apployees-nx/node searches your code and dynamically generate package.json, which only contains the NestJS dependencies.</p><p>The following scripts (in package.json) builds NestJS; install node_modules in dist dir; zip resources; so you can deploy it to Lambda:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b589eea253fc6cbef9f1ef83d7b899dc/href">https://medium.com/media/b589eea253fc6cbef9f1ef83d7b899dc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*q1lJhgHXbwPR7tops16zxQ.png" /><figcaption>The dist directory for NestJS app with package.json and node_modules</figcaption></figure><h3>GitLab CI and Nx</h3><p>The CI pipeline can leverage Nx <strong><em>affected</em></strong> capability so the CI only build, lint, test and deploy <strong><em>affected artifacts</em></strong>. You use <strong><em>base</em></strong> and <strong><em>head </em></strong>options to find what affected and Nx finds them by git diff. We create these scripts in package.json so we can call them in CI:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccf483112ab5a5813cf96fa2643aa7f0/href">https://medium.com/media/ccf483112ab5a5813cf96fa2643aa7f0/href</a></iframe><p><strong>The challenge</strong></p><p>In manyrepo, you can make each pipeline simple and straight forward and use triggers to start another pipeline easily and flexibly.</p><p>In monorepo, it is harder as each component might require a specific platform or environment and the pipeline can become very complicated quickly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qzmRudaOjclEHtCTr6n7Ug.png" /></figure><h3>Lessons learned</h3><ul><li><strong>Less configuration/boilerplate</strong>: Monorepo reduces a lot of redundant configuration and scripts found in manyrepo strategy as every configuration/scripts are available across the apps and you can easily parameterise them as the code structure is predictable</li><li><strong>Promote modular architecture</strong>: Thanks to a generator, you can effortlessly make a shared library and it is instantly available for all apps</li><li><strong>Shared components can be naturally evolved</strong>: You can first create a component in an app and easily promote to a library, which includes models, interfaces, validations and DSL specific business logic</li><li><strong>Nx works well for any technology</strong>: You can add any artifacts into Nx workspace including database migration scripts, Python-based Lambda or even Kotlin based API (a legacy Maven project) and leverage Nx tooling such as affected, computation cache and standardised build, lint, test deploy cycle</li><li><strong>Better DSL for CI Pipeline</strong>: We feel CI tool needs to be refined to fully utilise a tool like Nx, we believe GitLab CI is working hard to better support Monorepo pipeline and look forward to seeing improvements</li></ul><p>Thanks for reading!</p><h3>References</h3><ul><li><strong>Cons of monorepo</strong>: <a href="https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b">https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b</a></li><li><strong>Monorepo for a service project</strong>: <a href="https://itnext.io/building-scalable-multi-platform-projects-with-angular-and-nx-a13b40885e54">https://itnext.io/building-scalable-multi-platform-projects-with-angular-and-nx-a13b40885e54</a></li></ul><ul><li><a href="https://indepth.dev/the-shell-library-patterns-with-nx-and-monorepo-architectures/">Variations of the Shell Library pattern with Nx &amp; Angular Monorepos</a></li><li><a href="https://www.npmjs.com/package/nx-electron">nx-electron</a></li><li><a href="https://github.com/angular-architects/nx-ddd-plugin/blob/master/libs/ddd/README.md">angular-architects/nx-ddd-plugin</a></li><li><a href="https://blog.bitsrc.io/selecting-the-right-tool-for-your-monorepo-fafe409134b3">Selecting the Right Tool for Your Monorepo</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7d7a34f9070" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/adapting-monorepo-with-nx-ionic-nest-aws-serverless-gitlab-ci-a7d7a34f9070">Adapting Monorepo with Nx + Ionic + Nest + AWS Serverless + Gitlab CI</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating from Jenkins to GitLab CI : DSL side by side]]></title>
            <link>https://medium.com/lapis/migrating-from-jenkins-to-gitlab-ci-dsl-side-by-side-38933983fbe5?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/38933983fbe5</guid>
            <category><![CDATA[jenkins]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[gitlab-ci]]></category>
            <category><![CDATA[migrate]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Fri, 03 Jan 2020 02:25:12 GMT</pubDate>
            <atom:updated>2020-07-21T19:51:59.500Z</atom:updated>
            <content:encoded><![CDATA[<h3>Migrating from Jenkins to GitLab CI : DSL side by side</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vXdW87YkixpLJT57" /><figcaption>Good bye Jenkins and hello GitLab CI</figcaption></figure><p>At Spatial Vision, we have been using <a href="https://jenkins.io/">Jenkins</a> for Continuous Integration, building, testing and deploying web apps, native mobile apps, APIs and AWS resources to target environments.</p><p>Jenkins manages over 50 projects and nearly 500 jobs including active development builds and on-going maintenance tasks.</p><p>Today, we’d like to share our story on migrating from Jenkins to GitLab CI, by comparing the <a href="https://martinfowler.com/books/dsl.html">DSL</a> of each platform so you can understand the differences between them and it gives you a quick introduction of GitLab CI for Jenkins users.</p><ul><li><a href="#b4a3">Our Jenkins</a></li><li><a href="#42a0">Jenkins cons (why we looked for an alternative)</a></li><li><a href="#e5e2">GitLab CI</a></li><li><a href="#7c87">Convert Jenkins DSL to GitLab CI DSL</a></li><li>Lessons Learned</li></ul><h3>Our Jenkins</h3><p>We have a total of 4 Jenkins instances, running on an internal <a href="https://linuxcontainers.org/">Ubuntu LXC containers</a>, and automating CI processes using a number of build machines.</p><p>The reason we run multiple Jenkins instances is that Jenkins could become slower in listing, viewing and editing of jobs when you add a lot of jobs.</p><p>We provide managed services for our clients using AWS based infrastructure and each project typically has a webapp/native mobile apps and APIs (Development/QA/UAT/Production environments).</p><p>The backed services are created and updated by AWS CloudFormation and AWS CLI.</p><h4>Jenkins Jobs</h4><p>The following are common Jenkins Jobs and they are executed based on the code change or periodic schedule.</p><ul><li>Build HTML 5 web applications using Node.js and Angular</li><li>Build native mobile applications (Ionic, NativeScript, ReactNative) in a Mac mini build machine</li><li>Deploy to Google Play and iTunes using Fastlane/MS App Centre</li><li>Build and test APIs (Node.js, Kotlin/Java) on a Dockernized build machine</li><li>Build and push docker images to Docker Registry (GitLab built-in registry and Docker Hub)</li><li>Deploy AWS resources including S3, CloudFront, Lambda, EC2/ECS, Fargate, RDS etc.</li><li>Deploy Docker containers for Docker compose based APIs</li><li>Create/update database instances including AWS RDS (PostgreSQL and MS SQLServer)</li></ul><h4>Jenkins agents</h4><p>These jobs are executed on an internal build machines (Ubuntu, Mac mini and Windows) and AWS EC2 instances in which Jenkins Agents are installed for deployment jobs.</p><h3>Jenkins cons (why we looked for an alternative)</h3><p>Jenkins has been around for many years and we see it as one of the best model of open source based platform. It has many plugins that are ready to perform a particular job. However we’ve found some issues over a period of time and have been looking for an alternative tools for a quite while.</p><ul><li>Although Jenkins project is still <a href="https://github.com/jenkinsci/jenkins">active</a>, we feel the community is shrinking and some issues never get resolved and there are many plugins that are outdated</li><li>Jenkins Configurations, including Global and project level, are not easily reusable as these are managed within the Jenkins UI</li><li>Jenkins introduced <a href="https://jenkins.io/doc/book/pipeline">pipeline</a> (Groovy script based ) which could be very complicated for a very simple job and you need a JVM/Groovy for a project even you might only want to build a NPM package</li><li>There is <a href="https://github.com/jenkinsci/configuration-as-code-plugin">a configuration plugin</a> that lets you create ‘Configuration as Code’ and it offers yaml syntax but it only supports a small set of plugins and you might need to overwrite the global configuration for a project, which might be harder to manage when you have a large number of global configuration established</li><li>We feel <strong><em>Jenkins web application user experience</em></strong> is quite outdated and make you click so many links to view what you’re looking for such as Console output for a particular build</li><li><a href="https://jenkins.io/">Jenkins</a> now offer CDF, Jenkins X, Tekton and Spinnaker, that are promoted as new generation Continuous Delivery platforms (we rather want a CI/CD platform that works well)</li></ul><h3>GitLab CI</h3><p>We’ve been using <a href="https://about.gitlab.com/free-trial/self-managed/">Self-Managed GitLab</a> for our code repository. GitLab provides much more than just a code repository, such as a docker registry, error tracking, wiki, issue tracking, CI etc.</p><p><a href="https://about.gitlab.com/product/continuous-integration/">GitLab CI</a> has been around for a couple of years and it has become one of the most popular CI tools in the community. GitLab products have a great documentation and their feature development is so fast and we get new features so frequent and they are generally very useful.</p><p>We’ve tried on GitLab CI by migrating some existing Jenkins jobs including building and deploying API, webapp and native app.</p><p>We’ll walk through the migration process by comparing the DSLs used in both platforms.</p><h3>Convert Jenkins DSL to GitLab CI DSL</h3><h4>Our user story</h4><blockquote>“Build and Push Kotlin based API Docker image and deploy to a development environment”</blockquote><h4><strong>Node/Agent/Slave</strong> vs GitLab <strong>Runner</strong></h4><p>First of all, we need a machine to run a job, which can be any platforms including Linux, Mac and Windows.</p><h4>Jenkins Agent</h4><p>Jenkins Agent must be installed on a host machine and you specify the work space (remote directory), labels and availability. You can decide the number of executors in order to run jobs concurrently.</p><h4>GitLab Runner</h4><p>For GitLab, you need to <em>install</em> a GitLab Runner on a host machine and then <em>register</em> runners. You can register any number of Runners (Executors) on a host machine in order to run jobs in parallel.</p><p>GitLab Runner has a set of arch types, including SSH, Shell, Parallels, VitualBox, Docker or even Kubernetes. We like this feature as we often use a Docker container for a build and it is naturally supported:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*E31cgB2EoqJBhE3jnbuKzQ.png" /><figcaption>GitLab Runner can use a Docker image</figcaption></figure><p>For building an API image, we use <a href="https://docs.gitlab.com/runner/executors/shell.html"><em>Shell executor</em></a> to build a Docker image and push it to GitLab registry.</p><h4>Manage node vs Config.toml</h4><p>Jenkins only provides the Agent configuration via UI, called ‘Manage Node’. GitLab Runners can be configured via UI or using a config.toml file installed on the host of Runners.</p><h4>Labels vs Tags</h4><p>Labels in Jenkins describes the capabilities of the Agent so when you run a job, you can specify ‘what capability is required’ instead of ‘what machine should be used’. GitLab CI calls it Tags, which seems more natural DSL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AgAkZ998BWqzIIz1" /><figcaption>Jenkins Agent vs GitLab CI Runner</figcaption></figure><h4>Workspace vs builds &amp; cache</h4><p>We normally create a Jenkins user for a Jenkins Agent and allocate a home directory, which is the same in GitLab Runners.</p><p>Jenkins creates a directory called workspace. When you create a new job, a sub directory will be created in workspace in where the code is cloned and built.</p><p>GitLab Runner creates ‘builds’ directory and each Runner has its own sub directory inside it. Runners cache any artefacts (specified in gitlab-ci.yml) in ‘cache’ directory.</p><p>GitLab has a concept of group/sub groups to bundle related to repositories and Runners replicate the group/sub group structure inside the builds directory. e.g. builds/my-runner/my-project-group/my-api</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNz0cZQk5gM-xEmX" /><figcaption>Jenkins workspace vs GitLab runner builds directory</figcaption></figure><h4>Create a new job</h4><p>When a Runner has been installed and registered, we can then create a CI job.</p><h4>New Item vs The repository</h4><p>In Jenkins, you need to create a ‘New Item’ to create a job <em>AFTER you’ve created a repository somewhere (GitLab, GitHub, Bitbucket etc)</em>.</p><p>GitLab CI is immediately available for <em>EVERY repository</em> you create on GitLab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-5kf5n8F-ofEYWtj" /><figcaption>New Item Page vs CI/CD menu</figcaption></figure><h4>Configure vs .gitlab-ci.yml</h4><p>In Jenkins, you specify the build details in ‘Configure’ menu, which has a total of 6 tabs including ‘General’, ‘Source Code Management’, ‘Build Triggers’, ‘Build Environment’, ‘Build’ and ‘Post-build Actions’.</p><p>We normally use ‘Copy from’ feature to copy an existing job that is similar to a new project, then modify the content.</p><p>GitLab provides a configuration file called ‘.gitlab-ci.yml’. It needs to be placed in the root of your repository and GitLab then automatically detects the file and run a job described in the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qlGoHpqhU5GRvcYn" /><figcaption>Configure page vs .gitlab-ci.yml</figcaption></figure><h4>Jobs vs Pipeline + Jobs</h4><p>In Jenkins, you will create a set of jobs (e.g. build, test and deploy) and connect them by triggering another job or use pipeline feature. For a typical API CI, we need build test and 4 deploy jobs (dev/qa/uat/prod) and glue one job to another.</p><p>GitLab provides a pipeline concept with ‘Stages’ (e.g. build, test and deploy). Each stage can have a multiple jobs and each job can have its own set of configuration including environmental variables or what branch to be executed.</p><h4>Source Code Management vs The repository</h4><p>Jenkins (still) supports CVS, Subversion and GIT and you can clone multiple repositories in one job. This is so generic and flexible you can fetch any number of repositories for build, test and deploy.</p><p>However, our CVS/Subversion were retired and migrated into GIT long time ago and this flexibility rather forces you to configure it every time.</p><p>You also need to somehow specify the branches to build, which does not work well when you follow <a href="https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow">Git Flow</a>.</p><p>GitLab CI is available for each repository and you don’t need to tell any other system for how to fetch your repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6rc1hZLNwMTnQK0e" /><figcaption>Source Code Management menu vs GitLab repository</figcaption></figure><h4>Build Triggers vs When and Only/Except</h4><p>In Jenkins, you need to specify how you want to fetch your code, either periodically or poll the repository.</p><p>If you want to run a job as soon as the code has been pushed for a branch, you need to frequently poll it or use Jenkins webhook and call it on a GIT repository.</p><p>GitLab CI pipeline runs automatically as soon as a change has been pushed for a branch. It also offers 3 simple and clear DSL, ‘When’ and ‘Only/Except’ to limit jobs running within the pipeline.</p><p>When: You can set manual to only trigger a job manually or make it dependent on another job using on_success/on_failure.</p><p>Only/Except can limit what branches or tags to run a particular job. We use this to ‘deploy production’ only on a master branch type of scenarios.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AbL2S8GfpgMSkv1M" /><figcaption>Build Triggers vs when/only/except</figcaption></figure><h4>Build vs Stages</h4><p>Jenkins Build supports a number of scripts (shell, batch, ssh) and Java build tool such as Ant/Maven by default (you can add more builds using plugins).</p><p>We use custom shell scripts to fetch some configuration items; run test; build a docker image; push to the Docker registry.</p><p>We could run pretty much the same script in build stage of the pipeline in GitLab CI. The scripts are identical and run on the same host machine and worked great.</p><p>At this stage, we encountered a problem. We need to initialise a test database prior to running a test. In Jenkins, we have a job for it and trigger it prior to the test job.</p><p>This database initialisation was only available on Jenkins so we had to call Jenkins webhook from a GitLab CI.</p><p>In GitLab CI, the only solution we found was to use curl to call Jenkins web hook trigger in ‘before_script’ section and ‘sleep for a period of time’ to finish the database initialisation. (You could make a better script and use Jenkins CLI/API to check whether a job has been completed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t5NXGMMAp-vQJ2J4" /><figcaption>Build vs build stage in .gitlab-ci.yml</figcaption></figure><h4>Post-build Actions vs Stages and Triggers</h4><p>In Jenkins, we use Post-build Actions to run a deploy job when the build job has been successfully completed.</p><p>In GitLab, you can create triggers to run another pipeline. A trigger can be either called from .gitlab-ci.yml (using curl) or from a webhook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q65jmXC-0cwGs5hO" /><figcaption>Post-build Actions vs stages</figcaption></figure><h4>Jobs for each environment vs Pipeline</h4><p>In Jenkins, we normally create a deployment job for each environment, development/QA/UAT/Production. This is due to the fact we need our client’s approval (UAT) prior to a release to a production and we cannot fully automate for a Continuous Delivery.</p><p>We use GitFlow to achieve the deployment process with a specific branch as follows:</p><ul><li>Development: Automatic deployment using <strong><em>develop</em></strong> branch</li><li>QA (Internal): Nightly or on demand using <strong><em>develop</em></strong> branch</li><li>UAT/Production: Manual using <strong><em>release/master</em></strong> branches</li></ul><p>In Jenkins, it is not easy to make related jobs bundled and you need a very good naming conventions to make jobs somehow related each other. (Unless you use pipeline feature)</p><p>In GitLab CI, we can use stages and bundle jobs with DSLs such as environments/when/only/except so the pipeline is visualised clearly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JC2KeaCszg-cpMRf" /></figure><h3>Lessons learned</h3><p>We’re happy with our trial on GitLab CI and feel our CI processes are more seamless with our development activities thanks to a clear DSL with modern development practice.</p><p>We’ve since then created many CI processes using GitLab CI and there are a number of .gitlab-ci.yml templates we can reuse across projects.</p><p>The following are a few lessons learned on GitLab CI</p><ul><li>As of Jan 2020, <a href="https://about.gitlab.com/direction/monorepos/">Monorepo is not fully supported</a> and we’re simply unable to use some Monorepo tool chain, which makes Jenkins little more appealing due to its flexibility</li><li>Always create a project specific Runners instead of sharing by the capability of the Runner to avoid a conflict for Runner’s capabilities between projects</li><li>GitLab is very active and GitLab Docs is comprehensive and easy to follow. Their issue tracker is also active and many issues are solved quickly, however there are bugs shipped from time to time and it might affect on your development activities when you are fully dependent on GitLab tool chain</li><li>Late last year, GitLab shipped <a href="https://gitlab.com/gitlab-org/gitlab-runner/issues/4805">a critical bug</a> in GitLab CI and we’re unable to build or deploy anything for a couple of days. We had to abandon all of Docker based builds for work around and wasted a lot of times</li><li>‘Multi-project pipeline’ is a feature currently only available for Premium users (we’ve been using a community free edition) and there is no other reliable way to visualise and manage multi-project pipelines (which makes hard to create a sophisticated pipelines for a complex project having many components)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38933983fbe5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/migrating-from-jenkins-to-gitlab-ci-dsl-side-by-side-38933983fbe5">Migrating from Jenkins to GitLab CI : DSL side by side</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running AngularJS 1.6 in Angular 5 (side by side) Part 3: Build, deploy and beyond]]></title>
            <link>https://medium.com/lapis/running-angularjs-1-6-in-angular-5-side-by-side-part-3-build-deploy-and-beyond-2729c4a032ac?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2729c4a032ac</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[angularjs]]></category>
            <category><![CDATA[angular-cli]]></category>
            <category><![CDATA[angular-5]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 03:18:17 GMT</pubDate>
            <atom:updated>2019-02-13T00:39:30.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*q83JOe1hPVTBgw9tk8aZRg.png" /></figure><h3>Release it!</h3><p>This is a part 3 of <em>Running AngularJS 1.6 in Angular 5 (side by side)</em>, you can read <a href="https://medium.com/sv-blog/running-angularjs-1-6-in-angular-5-side-by-side-d2ed771e2a8f">part 1</a> and <a href="https://medium.com/sv-blog/running-angularjs-1-6-in-angular-5-side-by-side-part-2-build-new-features-in-angular-5-514156b12dd3">part 2</a>.</p><p>We’ve completed the new features and it’s now time to release to the production.</p><p>Today, we’d like to share our experience with the release process including</p><ul><li>Building a release bundle</li><li>Deploy to S3 bucket</li><li>Lessons learned</li></ul><h3>Building a release bundle</h3><h4>AngularJS</h4><p>In AugularJS, we used <a href="https://gruntjs.com/">Grunt</a> (generated by <a href="http://yeoman.io/">Yeomen</a>) to create a release bundle. Grunt build performs:</p><ul><li>Build a css file from scss</li><li>Convert html templates into a single js file</li><li>Bundle js files including the libraries into a single js file</li><li>Uglify js and css</li><li>Revisionize the file names</li><li>Update/minify index.html</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*3hLiS39mxBF1qh5sAt9fcA.png" /><figcaption>Gruntfile.js build task</figcaption></figure><p>The release bundle contains a set of js/css/image/font files with index.html.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*fV2i0sSCBhoi2GVHm00uBA.png" /><figcaption>The dist directory contents</figcaption></figure><h4>Angular</h4><p>In Angular, we use Angular CLI, which uses Webpack internally. There are 3 main steps for building a release bundle:</p><ol><li>build with new angular cli</li><li>minify/uglify with a generic minifier/uglifier</li><li>Revisionize (Force loading the new release)</li></ol><p>The reason we need to minify/uglify by ourselves is our AngularJS app does not work when using Angular CLI uglify process! In another words, Angular CLI build is not compatible with <strong>@angular/upgrade</strong> we use to run AngularJS side by side. It is a big surprise given Angular team is providing comprehensive tools and documentation for the upgrade module. There are many issues Angular CLI repository complaining about <a href="https://github.com/angular/angular-cli/issues/11910">it</a>.</p><p><strong>Build with Angular CLI 1.7.4</strong></p><p>We need to firstly make sure AngularJS app is uglify friendly for a generic minifier/uglifier by converting any implicit dependency injection to explicit dependency injection:</p><pre>someModule.controller(&#39;MyController&#39;, function($scope, greeter) {<br>    // ... <br>});</pre><p>to</p><pre>someModule.controller(&#39;MyController&#39;, [&#39;$scope&#39;,&#39;greeter&#39;, function($scope, greeter) {<br>    // ...<br>}]);</pre><p>When AngularJS injection has been fixed, we can run ng build (Please see the <a href="https://github.com/angular/angular-cli/wiki/build">official doc for options</a>).</p><pre>ng build --aot true --build-optimizer true --sourcemaps false --extract-css true --named-chunks false</pre><p>The following is the contents of dist directory. There are four JS bundle files including inline.bundle.js, main.bundle.js, polyfills.bundle.js and scripts.bundle.js and one CSS bundle file, styles.bundle.css. At this stage, JS files are so large, a total of 20MB.</p><p>It’s also worth to mention that ng build does not bundle the partial html files as scripts unlike AngularJs Grunt ngTemplate task. This will force us to copy ng1 directory to serve partial html files at the time of page loading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*qGShGDvs9DQ9sr-8ULFayg.png" /></figure><p><strong>Minify/Uglify</strong></p><p>We have used ‘uglify-js’ version 3.4.2 and ‘uglifycss’ version 0.0.29:</p><pre>echo &quot;run js uglifier&quot;<br>  node --stack_size=100000 uglifyjs ./dist/inline.bundle.js &gt; ./dist/inline.01.bundle.js<br>  node --stack_size=100000 uglifyjs ./dist/polyfills.bundle.js &gt; ./dist/polyfills.01.bundle.js<br>  node --stack_size=100000 uglifyjs ./dist/scripts.bundle.js &gt; ./dist/scripts.01.bundle.js<br>  node --stack_size=100000 uglifyjs ./dist/main.bundle.js &gt; ./dist/main.01.bundle.js</pre><pre>echo &quot;run css uglifier&quot;<br>  node uglifycss ./dist/styles.bundle.css &gt; ./dist/styles.01.bundle.css</pre><p>The process reduced the file sizes to almost half, which is still over 10MB. This is 4 times larger than the previous AngularJS app (2.5MB including all partial html files).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*NfPHbe_Q6iPfEPTbWOkYNg.png" /></figure><p><strong>Revisionize (Force loading the new release)</strong></p><p>Lastly, we need to revisionize the files in order to force reloading the application bundles. There are a range of tools available for a rev task including <a href="https://github.com/sebdeckers/grunt-rev#readme">grunt</a> and <a href="https://github.com/TNT-Likely/webpack-rev-replace-plugin#readme">webpack</a>. (we used a custom rev task which is already existed in our tool chain).</p><h3>Deploy to S3</h3><p><a href="https://maps.datafarming.com.au/">DataFarming</a> is hosted in AWS S3 as a static content and served using AWS CloudFront. We use AWS CLI to sync the dist directory with the S3 bucket and then invalidate the CloudFront cache. There is no change in this part and we now have a new release!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZBs3tY808GxEnB4979kng.png" /><figcaption>A new feature released in production</figcaption></figure><h3>Lessons learned</h3><p>Finally we’d like to share our lessons learned on this migration strategy.</p><h4>Good parts</h4><ul><li><strong>No Glitches</strong>: Overall, the migration is smooth and there is no technical issues in building a new feature using Angular while leveraging the existing features implemented in AngularJS</li><li><strong>Brave new world</strong>: We can now fully take advantages of Angular Component Architecture conjunction with NGRX as well as use UI components available in Angular Material</li><li><strong>Align with Angular eco system</strong>: Angular is now 7 and providing more features that help us improve our application development/features (<a href="https://blog.angular.io/schematics-an-introduction-dc1dfbc2a2b2">Schematics</a>, <a href="https://angular.io/guide/universal">Univeersal</a>, <a href="https://material.angular.io/cdk/categories">Angular CDK </a>etc.)</li></ul><h4>Bad parts</h4><ul><li><strong>Build issues</strong>: ng build does not understand that we’re including AngularJS using the upgrade module and there is no discussion or clear guide for how you can solve it. The Angular CLI team also clearly says they don’t want us to touch any of the webpack tasks directly. So the issue we face had to be solved by ourselves</li><li><strong>The size of application</strong>: The application has become significantly larger compare to the previous release (4 times larger than AngularJS app). We’ve built a number of applications using Angular 2, 4, 5, 6 and 7 and the size of a release bundle can be 500KB ~ 5MB depending on the libraries used for a project. We also experienced earlier versions of ng build with eager optmizations often broke the app for no reason and we were forced to release (quietly) a version without optimising the release bundle.</li><li><strong>Benefit for the end users</strong>: As a developer, this project is successful in a way we can leverage the newer technology for the app, it is not beneficial for the end users at all due to the increase in size. This is especially a problem for people who access our application in a field with a mobile device and they could easily stop using it because of the loading time.</li></ul><h4>Do we use this architecture for another AngularJS app?</h4><p>The biggest motivation of this migration strategy is to add new features using the latest technologies while reusing the custom implementation (services, directives and all the reusable parts) of the AngularJS app.</p><p>DataFarming has definitely achieved the goal and substantially improved a new feature development from a development perspective. It is also economical as we are able to use all of the existing implementation.</p><p>So we’ve decided to try on another larger project.</p><p><strong>Case: Angular UI Bootstrap based project</strong></p><p>Many of our AngularJS apps use AngularJS UI Bootstrap (which uses Bootstrap 3). The issue we faced quickly was Angular powered Bootstrap (UI Bootstrap for Angular 2 ~ 7) use Bootstrap 4 and you can’t simply use these 2 versions of Bootstrap side by side. The end of the story.</p><p>Our solution for the UI Bootstrap based apps are to run 2 single apps (AngularJS and Angular) side by side by having 2 application entries (i.e. index-1.html and index-2.html). Any services/directives implemented in AngularJS required in Angular are simply ported (duplicated) in Angular by rewriting (mostly copied and pasted with a small modification) in TypeScript. The routing is simply loading index-1.html or index-2.html depending on the route.</p><h3>Conclusion</h3><p>We’re initially very excited with the upgrade module provided by the Angular team. They provided <a href="https://angular.io/guide/upgrade">a comprehensive in depth technical document</a> on this topic, however it was not practical and it did not help us to make what strategy we should take in terms of the user experience.</p><p>In our experience, the development process has been great and we’ve built our new feature quickly and nicely using Angular echo system. However it is not really friendly as a end user as the application is significantly larger to load.</p><p>We can certainly spend more time to further optimise but it is not that straight forward since we’re not be able to ‘customise’ Angular CLI build operation and there is no guarantee if a custom optimisation would work in the future version such as Angular 6 and 7. For instance, Angular 6 uses angular.json instead of .angular-cli.json and it is radically different structure and you need to deal with all internal process changes yourself.</p><p>This is definitely a major drawback of using Angular CLI centric development compared to using a traditional webpack as you cannot hack your build process easily for a special requirement (e.g. embedding AngularJS app or dealing with another embedded libraries).</p><p>In conclusion, if you don’t have too many 3rd party libraries in AngularJS app and the size of your app is not important this strategy will work for you. If you use UI Bootstrap, 2 single page apps side by side is the way to go.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2729c4a032ac" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/running-angularjs-1-6-in-angular-5-side-by-side-part-3-build-deploy-and-beyond-2729c4a032ac">Running AngularJS 1.6 in Angular 5 (side by side) Part 3: Build, deploy and beyond</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identify Fish using Machine Learning]]></title>
            <link>https://medium.com/lapis/identify-fish-using-machine-learning-13399e62e9bf?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/13399e62e9bf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[research-and-development]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Thu, 22 Nov 2018 05:45:17 GMT</pubDate>
            <atom:updated>2018-11-23T02:20:44.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*Y8rX6YyZlEjQApH-3Oz5eA.png" /></figure><h3>The Hackathon</h3><p>Recently, Spatial Vision was invited to an open tender process which featured a hackathon to assess our capabilities/offering. It was the first tender process that we had been involved in which had this component so we were very excited to showcase our development capability.</p><p>One of the hackathon challenges was to create an app for recreational fishers which would identify the species of fish they caught in order to determine whether take it home or release it.</p><h4><strong>Other Requirements</strong></h4><ul><li>We only had 5 days with 2 developers to build all challenges (2 apps)</li><li>After an initial estimate, we could only spend 2 days for the app</li><li>The app must be an native and runs on both Android and iOS</li></ul><h3><strong>Beginning to Build</strong></h3><p><strong><em>The user story we were given was:</em></strong><em> Given I have caught a fish in the beach, I want to know the species so I know whether I can keep it or release it.</em></p><p>We decided to build a feature based on the user story which first captures a photo of the fish then shows a list of species sorted based on the score (%).</p><h3><strong>Choosing Machine Learning Platform</strong></h3><p>First of all, our method of identifying fish was to use ‘machine learning’, more specifically image recognition/classification. It’s the hottest topic in IT industry and we had played around with some of them, such as Google Dialogflow and Vision API.</p><p>There were many services available including Amazon Rekognition, MS Computer Vision etc, however their APIs are more broad and focusing on ‘to recognise objects in a photo’ (object detection) or ‘if a person is happy or sad’ (face/sentiment detection) and not ‘what kind of fish is this?’.</p><p>We wanted this API done without having to implement it ourselves so we could focus on the user experience. We searched around and found a service called Vize.ai. It was still in beta at the time but provided exactly what we needed, which was to classify, train and call it as API.</p><h3>Vize.ai</h3><p><a href="https://vize.ai/">Custom Image Recognition API - Vize.ai</a></p><h3>Tasks and labels</h3><p>Vize.ai defines an image recognition/classification as tasks and labels. In our case, a task is ‘identify fish’ and labels are ‘Barramundi’, ‘Flathead’, ‘Snapper’ etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikzQFIeFjRO2Snx_nDon1w.png" /><figcaption>Tasks in Vize.ai</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBkDf2MhHl1UMPDsZl9nxg.png" /><figcaption>Labels in Vize.ai</figcaption></figure><h3>Creating Labels (the fish species)</h3><p>We created a total of 8 labels, covering the most common fish caught in Australia. (in the context of fishing in a bay in Queensland).</p><ul><li>Bartailed flathead</li><li>Dusky flathead</li><li>Other flathead</li><li>Sand whiting</li><li>Snapper</li><li>Tarwhine</li><li>Trumpeter Whiting</li><li>Yellowfin Brim</li></ul><h3>Collecting data (photos)</h3><p>The next step was to collect sample photos of each species as a training data set.</p><p>Since we had a strict time constraint, we used Google image search and simply searched by species names. We then picked up a variety of photos including:</p><ul><li>Taken by recreational fishers, who are holding the fish that are just caught</li><li>Showing the fin or head that are unique to the species</li><li>Showing the colour of fish clearly</li></ul><p>The reason we included the recreational fishing photos was we wanted to see if we could classify well with photos that contain not only the fish but also surroundings such as a person, fishing gear, ocean, clouds and water.</p><p>We collected 20 ~ 30 photos per species and uploaded them to Vize.ai.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*On-k5mw4wIQbD31TnI_0fw.png" /><figcaption>Google Image Search by Species name</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*wWRqpE8xUJ3WAKQPT9TzXA.png" /><figcaption>Photos uploaded in Vize.ai “Snapper Label”</figcaption></figure><h3>Training</h3><p>The next step was to train our model with the the photos uploaded.</p><p>Vize.ai splits your images into training images and testing images. The majority of the images are used for training, while the remainder are used to test the accuracy of the model.</p><p>This process took just over half hour and we eventually got 72% accuracy as the result of training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*LsTo0Wx5YZtFPRLyWRCJ2Q.png" /><figcaption>Vize.ai Task with the training status and accuracy</figcaption></figure><h3>Classify (Test the model)</h3><p>Vize.ai provides ‘Classify Image’ feature, which allows you to test your model (label + photos) easily. We simply collected some test data and uploaded to classify them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*kt7c5X3HqRCPmgW54wjRKw.png" /><figcaption>Classify Image in Vize.ai</figcaption></figure><h4>The test result — Good</h4><p>The 1st photo to test was actually a photo of a snapper taken by a guy and the relevance (score) was 98.83%! It was astonishing to see how we could easily build an image classification API with so little number of photos!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*MUQqraf4Gg4LPKA8agPzVw.png" /><figcaption>The test result with JSON data</figcaption></figure><h4>The test result — Not Good</h4><p>The 2nd test was little harder. We picked up a sand whiting and see if our model could distinguish between sand whiting and trumpeter whiting.</p><p>They looked very similar in shape and only noticeable difference was trumpeter whiting had a unique line pattern in the body, which we were hoping our model could recognise, given all of Trumpeter Whiting photos did have the pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*XDTYJSUICROktUYmXTBUXQ.png" /><figcaption>Sand Whiting training photos</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*prGskVzaLUoStfp77hlrZA.png" /><figcaption>Trumpeter Whiting training photos</figcaption></figure><p>Unfortunately our model thought it was Trumpeter Whiting. Even worse, the second score was given to Snapper!</p><p>Unfortunately we couldn’t decide why this went wrong and how we could improve as we didn’t have any clue on what ML algorithms Vize.ai used and what kind of errors they had in their test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*HTU0x8H5Lrnk7Z7h02Fgvw.png" /></figure><h4>The test result — Bad</h4><p>The next test was to classify fish that was not in our model. We tested a photo of tuna, which did not look anything like the ones in our labels.</p><p>We imagined it’d give very low scores for all of labeled fish but our model thought it was a snapper… (with a fairly confident score of 96.54%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*LHu2yFrd3yviguQDoYFliQ.png" /></figure><h4>The test result — Ugly</h4><p>Given the result of tuna test, we decided to test if a photo was not even a fish. our model thought this koala was “Other flathead” with a score of 67.4%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*oUyKzBpnUOdSgQzKpZEoCg.png" /></figure><h3>Findings so far</h3><ul><li>We didn’t need too many photos to train a model if a photo to classify was obvious or distinguishable</li><li>The API still gives a higher score (over 95%) even the fish does not look like any of the existing labels</li><li>The API gives lower scores (60%) if it is not fish (could be used to determine whether there is a fish in the photo)</li><li>It was not easy to re-train our model based on the test result as we didn’t have any knowledge on how the model made the decision (Algorithms) in the first place and how we could refine them</li><li>Nonetheless, it was good enough for us to create an app using this API so we could showcase a prototype</li></ul><h3>Building a prototype APP</h3><p>We didn’t have any more time to find alternative ML service so we built a native app that takes a photo of fish; call Vize.ai API; List the result by relevance. The following are the screenshots of the application prototype.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*N6y8CLw4LBNubyh0Lv3sKg.png" /><figcaption>Prototype Take a photo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*LYaIEkJ2LdGpwkhlpTGLWQ.png" /><figcaption>Prototype calling Vize.ai API</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*UHFsbz3ONB1ECYmFCTQuqQ.png" /><figcaption>Prototype Showing the result with the score</figcaption></figure><h3>Lessons learned</h3><p>We created 2 apps successfully within the time and we’re happy with our outcome.</p><p>We also internally discussed a possibility of further refining our prototype and potentially release it as a product. The following are some of our thoughts on how to refine:</p><ul><li>Combine with another classification API such as Vision API and identify whether there is fish in a photo, prior to our model so we can avoid classifying a photo that is not a fish</li><li>Further research alternative recognition APIs/Services and compare the implementation and insights</li><li>To distinguish a family of the same kind (dusky flathead, trumpeter whitings), we might require photos that only contain fish by extracting the fish from a photo</li><li>We need experts to classify appropriately when introducing a new label so our training data set has good characteristics of the fish</li><li>Automate retraining</li><li>Collect GPS location and time (another meta data) as meta data to help identifying the fish on top of image recognition</li></ul><p>Overall, it was a great exercise to attend this hackathon as we learned that using a ML service (API) in an application was very easy and we could think of many features that could utilise it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13399e62e9bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/identify-fish-using-machine-learning-13399e62e9bf">Identify Fish using Machine Learning</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running AngularJS 1.6 in Angular 5 (side by side)]]></title>
            <link>https://medium.com/lapis/running-angularjs-1-6-in-angular-5-side-by-side-d2ed771e2a8f?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2ed771e2a8f</guid>
            <category><![CDATA[angularjs]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[angular-5]]></category>
            <category><![CDATA[angular-cli]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Wed, 28 Mar 2018 06:34:31 GMT</pubDate>
            <atom:updated>2020-12-16T19:31:43.909Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q83JOe1hPVTBgw9tk8aZRg.png" /><figcaption>Side by side</figcaption></figure><p>We’d like to share one of the migration strategies from AngularJS to Angular. This blog is part 1 of 3, please also read <a href="https://medium.com/sv-blog/running-angularjs-1-6-in-angular-5-side-by-side-part-2-build-new-features-in-angular-5-514156b12dd3">part 2: Building new features in Angular 5</a> and <a href="https://medium.com/sv-blog/running-angularjs-1-6-in-angular-5-side-by-side-part-2-build-new-features-in-angular-5-514156b12dd3">part 3: Build, deploy and beyond</a>.</p><p>At Lapis, We run a number of AngularJS (1.2.x ~ 1.6.x) apps in production which were built around 2012 ~ 2015. Their main tech stack components are:</p><ol><li>Angular 1.x</li><li>Bootstrap 3.x</li><li>Leaflet</li><li>OpenLayer</li><li>LocalStorage (ngstorage)</li><li>Font Awesome</li><li>Moment JS (Date/time)</li><li>Grunt for build</li><li>Docker for the build process</li></ol><p>Many of the apps are stable and there is no reason to migrate/upgrade (commercially) to Angular 5 (6 is coming soon). However, some of the apps have on-going enhancement work and require new libraries for new features.</p><h3>Problem</h3><p>When you need a new library or external capabilities, most libraries (Angular wrapper etc.) are now only targeting Angular 2+.</p><p>Data Farming Mapping Website is a good example. The client wanted to add a shopping cart capability including eWay integration. While it is entirely possible to stay in 1.6, there was a risk of adopting external capabilities that are no longer maintained.</p><h3>ngUpgrade: Run AngularJS and Angular side by side</h3><p>This topic is not new and a lot of materials have been published over the last couple of years (Since Angular 2). We learned a lot through a number of blogs and official documentation, which were kind of outdated as Angular was evolving so quickly especially since early 2017 and after March 2018.</p><h3>Must read</h3><p>It is still a good idea to go through the history of the migration process as well as reading the official documentation:</p><ol><li>One of the recent blogs is about this topic: <a href="https://medium.com/code-divoire/a-story-of-ngupgrade-bringing-an-angularjs-application-from-1-6-to-angular-4-84eae4434010">https://medium.com/code-divoire/a-story-of-ngupgrade-bringing-an-angularjs-application-from-1-6-to-angular-4-84eae4434010</a></li><li>Insight of ngUpgrade, very helpful to understand the architectural part of ngUpgrade: <a href="https://blog.nrwl.io/ngupgrade-in-depth-436a52298a00">https://blog.nrwl.io/ngupgrade-in-depth-436a52298a00</a></li><li>A bit outdated but it provides the mechanics in detail: <a href="https://angular.io/guide/upgrade">https://angular.io/guide/upgrade</a></li><li>The only working implementation found (via the 1st blog): <a href="https://github.com/manfredsteyer/ngUpgrade-without-preparation">https://github.com/manfredsteyer/ngUpgrade-without-preparation</a></li></ol><h3>Case Study: Data Farming Mapping Website</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41QGe1jGq2Fps0sms6eiRQ.png" /><figcaption><a href="https://maps.datafarming.com.au/#">https://maps.datafarming.com.au/#</a></figcaption></figure><h4>AngularJS app overview</h4><p>We selected the Data Farming app as a migration candidate as we were about to implement some new features including an e-commerce capability using eWay and a custom shopping cart for downloadable shapefiles.</p><p>The app was originally created in order to demonstrate an API capability, however, it quickly evolved to an app directly used by farmers and consultants. Technically speaking, it has the following characteristics:</p><ol><li>AngularJS 1.6 + Angular Material, Grunt base and a few libraries including Auth0, OpenLayer 3, jsPDF and introJS</li><li>The app is relatively small (Login; List Farms; Draw paddocks in the map; Upload KML/Shapefiles; View NDVI images for paddock and print PDF)</li><li>Building a new feature which requires a new library (eWay) and new UX (shopping cart, purchase history etc.)</li><li>The existing code base needs to be reviewed prior to extending as it was built for a simple prototype</li></ol><h4>Migration step summary</h4><p>The following are the steps we took after a lot of reading. There are different migration strategies available in the official documentation, however, our preference is running side by side with Angular 5 being the main application.</p><ol><li>Create Angular 5 application using Angular CLI (1.7.3)</li><li>Copy an existing Angular JS app (should be 1.6.x and refactored so the structure of the code is feature-based and each file has a single impl)</li><li>Migrate JS dependencies (Switch npm or copy bower_components)</li><li>Fix template paths in both JS and HTML</li><li>Migrate scss files and fix import paths</li><li>Fix API proxy path from Grunt to Angular CLI</li><li>Fix minor CSS issues caused by an introduction of view-container and view-frame</li><li>Regression test (It seems all working at this stage, have a good test by QA)</li><li>Add Angular Routing and introduce a new page</li><li>Switch between AngularJS page and Angular page</li></ol><h3>Migration steps</h3><h4>Create Angular 5 application using Angular CLI (1.7.3)</h4><p>Firstly, create an Angular 5 app using Angular CLI. (Used Node 8.10.0 LTS)</p><pre>npm install -g @angular/cli</pre><pre>ng new ng52 --style=scss<br>cd ng52<br>ng serve</pre><h4>Copy an existing Angular JS app into src/ng1</h4><p>The app is s Grunt based Angular JS 1.6 app using Bower for web dependencies, NodeJS for the build process and SCSS for css processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*lWWdDjHtMhGPSKqny9vk_Q.png" /></figure><p>We firstly copied the app directory into src of Angular 5 app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*zvCOoXRketb4a2TphOGF5Q.png" /></figure><h4>Migrate JS dependencies (copy bower_components)</h4><p>Most of our JS dependencies are from bower, which essentially copies GitHub repositories into bower_components. We also have some dependencies fetched via CDN as well as embedded JS library.</p><p>Whilst it is possible to switch all dependencies into npm, our approach is to simply copy the required dependencies as a part of embedded scripts in Angular 5.</p><p>This decision is based on the time we want to spend as well as these dependencies will not be updated during the migration. (Of course, all of these can be migrated over a period of time when needed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*T5SWijX9MXRls8-1HPkJ8w.png" /><figcaption>Bower components copied into src/ng1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oW6ogcox-3U-e5qLl7uNlA.png" /><figcaption>Some libs are still using CDNs</figcaption></figure><p>When all dependencies are copied, simply list them in scripts of .angular.cli.json, in the same order as the original index.html. (followed by the application code)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*4phOljPotthlg9kf2OlKYg.png" /><figcaption>.angular.cli.json</figcaption></figure><h4>Bootstrap AngularJS in Angular</h4><p>At this point, Angular JS can be bootstrapped and show a page as all JS runtime is available. First, install the upgrade module and imports the module:</p><pre>npm install <strong>@angular/upgrade --save</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*6hejj5AylTKVbljETDrzgA.png" /><figcaption>app.module.ts</figcaption></figure><p>Then bootstrap AngularJS app in app.component.ts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*9LrWQOSUJGVJN9yhXExuPA.png" /><figcaption>app.component.ts</figcaption></figure><p>Lastly, create a template for AngularJS app in app.component.html</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*CxBWICrWM2X_hhaCP9jYGg.png" /><figcaption>app.component.html</figcaption></figure><p>Run the application by ng serve and we see AngularJS app running with a few errors including no style and 404 errors for HTML templates.</p><h4>Fix template paths in both JS and HTML</h4><p>At this point, AngularJS app is served in src/ng1/app directory and breaks the paths for partial htmls:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/1*zNv1Co0_pkKyiL7R9MQIKQ.png" /><figcaption>404 error for partial html templates</figcaption></figure><p>To fix 404 errors, add ng1 in .angular-cli.json</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*G2X2vYQbf9rlkkWVp-MTTA.png" /><figcaption>.angular-cli.json</figcaption></figure><p>Paths in HTML (ng-include) and js (routing, directives etc.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*iPsCsPHvQ9bIXRDAPA6b3w.png" /><figcaption>before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*GTWl_9dk0cAF6uzmqQuE9w.png" /><figcaption>after</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/812/1*LJ02m_ExddWTK6UkZNcXkQ.png" /><figcaption>before</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*mVeM-BjkG9w6vvjxUypFkQ.png" /><figcaption>after</figcaption></figure><p>After fixing the templates, we now see the page without style</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*sCVH9AZgpggHdDXr-zCfyg.png" /><figcaption>Home page without styles</figcaption></figure><h4>Migrate scss files and fix import paths</h4><p>AngularJS app uses scss, which is supported by Angular CLI. You can specify it when you create an app (or update in .angular-cli.json):</p><pre><strong>&quot;defaults&quot;</strong>: {<br>  <strong>&quot;styleExt&quot;</strong>: <strong>&quot;scss&quot;</strong>,<br>  <strong>&quot;component&quot;</strong>: {}<br>}</pre><p>Once scss is setup, then specify all scss files in styles of .angular-cli.json. We took this approach instead of fixing @import of scss so these dependencies are visible in .angular-cli.json and we can reuse them in Angular 5 app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*67sUSFnjWUCg0-RwTmYGeg.png" /></figure><p>Then fix variable references and path errors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*D58Q0JahCs1Y674NQUNq1Q.png" /></figure><h4>Fix API proxy path from Grunt to Angular CLI</h4><p>AngularJS app uses Grunt proxy, which we can simply replace with proxy.conf.json:</p><pre>{<br>  <strong>&quot;/ndvi&quot;</strong>: {<br>    <strong>&quot;target&quot;</strong>: <strong>&quot;http://localhost:3010&quot;</strong>,<br>    <strong>&quot;secure&quot;</strong>: <strong>false<br>  </strong>}<br>}</pre><p>Start app with</p><pre><strong>&quot;start&quot;</strong>: <strong>&quot;ng serve --proxy-config proxy.conf.json&quot;</strong>,</pre><h4>Fix minor CSS issues by view-container and view-frame</h4><p>The example GitHub repo has the following app.component.html. AngularJS app is rendered in ng-view and Angular 5 app is rendered in router-outlet.</p><pre>&lt;<strong>div class=&quot;view-container&quot;</strong>&gt;<br>  &lt;<strong>div ng-view class=&quot;view-frame&quot;</strong>&gt;&lt;/<strong>div</strong>&gt;<br>  <em>&lt;!-- router-outlet&gt;&lt;/router-outlet --&gt;<br></em>&lt;/<strong>div</strong>&gt;</pre><p>AngularJS app does not expect this extra divs and we have to fix css accordingly.</p><h4>Regression test</h4><p>AngularJS app seems all working at this stage, so we pass it to QA and make sure every part of the app is working.</p><h4>Add Angular Routing and introduce a new page</h4><p>AngularJS app is now successfully running in Angular 5 app, it is time to introduce a brand new page in Angular 5. We can just run a few Angular CLI command and create a component and a module.</p><pre>ng g component home</pre><h4>Switch between AngularJS page and Angular 5 page</h4><p>The last part is to switch the routing between AngularJS and Angular 5. We took the following implementation from the example GitHub repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*wGwmcsVi7J7MNkc844uz7A.png" /><figcaption>app.module.ts</figcaption></figure><p>CustomHandlingStrategy checks the URL and decides whether the URL passed should be handled by Angular 5, otherwise, simply forward to the AngularJS app.</p><h4>The result</h4><p>We’ve successfully run 2 apps side by side!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*RyppjC5wXYKdlpVuU_Kc0A.png" /><figcaption>AngularJS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*KKdrffRZj4olbrb3kypwPA.png" /><figcaption>Angular 5</figcaption></figure><h3>Next step</h3><p>We’re now ready to start a new feature in Angular 5 including how to share the common resources such as header/footer, Local storage etc.</p><p>We’ll share our findings in the next post.</p><p>Part 2 is now available:</p><p><a href="https://medium.com/sv-blog/running-angularjs-1-6-in-angular-5-side-by-side-part-2-build-new-features-in-angular-5-514156b12dd3">Running AngularJS 1.6 in Angular 5 (side by side) Part 2: Build new features in Angular 5</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2ed771e2a8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/running-angularjs-1-6-in-angular-5-side-by-side-d2ed771e2a8f">Running AngularJS 1.6 in Angular 5 (side by side)</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Team R&D 2018 — Cloud Vision API]]></title>
            <link>https://medium.com/lapis/bla-2f9681b9e880?source=rss-a878d1a269c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f9681b9e880</guid>
            <category><![CDATA[research-and-development]]></category>
            <dc:creator><![CDATA[Hiroki Gota]]></dc:creator>
            <pubDate>Fri, 16 Mar 2018 05:06:31 GMT</pubDate>
            <atom:updated>2018-03-27T23:49:53.738Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*eIMgK4vNuB5vWsB3EQIYIA.png" /></figure><h3>Overview</h3><p>At Spatial Vision, we’re trying a new approach for R&amp;D, Team R&amp;D (or simply hack day). Traditionally our R&amp;D process looks like:</p><ol><li>There is a piece of technology/topic we’d like to understand based on our potential work</li><li>Creates a set of criteria/goal for R&amp;D</li><li>A senior developer research/reads some documentations, creates a prototype</li><li>A senior developer presents the outcome to the management and team</li></ol><p>While this approach most likely produces a better outcome from a financial point of view, it does not scale well and there are a number of people who are interested in working on R&amp;D works.</p><p>So we’ve decided to switch it to a team based approach and and produce a working result like a hackday.</p><p>Our basic rule is to form a team (3 ~5 people) and each person has a role and work collaboratively.</p><h3>Problem: Find appropriateness of uploaded photos</h3><p>We’ve built a number in-field mobile applications used by general public, as a part of citizen science programs (<a href="http://www.spatialvision.com.au/citizenscience/">http://www.spatialvision.com.au/citizenscience/</a>).</p><p>In the last DevFest Melbourne 2017 (<a href="http://gdgmelbourne.com/">http://gdgmelbourne.com/</a>), we’ve discovered Google Cloud Vision API (<a href="https://cloud.google.com/vision/">https://cloud.google.com/vision/</a>) has evolved so much and it seems very easy to ‘Detect inappropriate content’.</p><h3>Epic user story</h3><pre>As a citizen science program administrator, I want to see how many of the photos uploaded are actually appropriate for scientific purposes so I can filter inappropriate photos</pre><p>An app user can report ‘a bird’ found in a field with a selection of species and some photos.</p><h3>Solution model</h3><p>We have a number of photos with some meta data, such as species name, category (Bird, Frog, Fish etc.). We can create an API that fetches the photos and metadata from a data source; use Cloud Vision API and annotate them; and finally visualise the annotation results to see the quality of reported photos.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*PyZfs6MirZcLw_iqFrqLbA.png" /><figcaption>Photo annotation API and visualisation</figcaption></figure><p>The Cloud vision API provides annotate API, which provides an annotation of a given photo. (You can test this online: <a href="https://cloud.google.com/vision/">https://cloud.google.com/vision/</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*bOkER3Js9LHPuFtggS_a-Q.png" /></figure><h3>The team</h3><p>Given we’ve got a solution the next step is decide who does what. We’ve decided to use a role lotto instead of the skills set so people could work on an area they never worked (or maybe you might get what you are good at).</p><ol><li>Elnaz: API Developer, building a Node API to get an annotation for each API</li><li>Parham: Data specialist, preparing the test data including the jpg files and meta data</li><li>Ryan: Quality Assurance, documenting user acceptance criteria and performing exploring test</li><li>Craig: Web Developer, visualising the annotation data</li><li>Hiroki: Journalist, presenting the outcome on our blog</li></ol><h3>The result</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f9681b9e880" width="1" height="1" alt=""><hr><p><a href="https://medium.com/lapis/bla-2f9681b9e880">Team R&amp;D 2018 — Cloud Vision API</a> was originally published in <a href="https://medium.com/lapis">Code道</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>