<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moreno Feltscher on Medium]]></title>
        <description><![CDATA[Stories by Moreno Feltscher on Medium]]></description>
        <link>https://medium.com/@morenofeltscher?source=rss-6c3aa195d9bc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*dkbYYO_vwj7Ug1YdUpXJ2g.jpeg</url>
            <title>Stories by Moreno Feltscher on Medium</title>
            <link>https://medium.com/@morenofeltscher?source=rss-6c3aa195d9bc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 17:00:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@morenofeltscher/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Replacing Google Site Search and Google Search Appliance with a modern Open Source Solution]]></title>
            <link>https://medium.com/smartive/replacing-google-site-search-and-google-search-appliance-with-a-modern-open-source-solution-bf5370bfd18c?source=rss-6c3aa195d9bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf5370bfd18c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Moreno Feltscher]]></dc:creator>
            <pubDate>Wed, 30 May 2018 07:35:51 GMT</pubDate>
            <atom:updated>2021-02-18T23:29:09.337Z</atom:updated>
            <content:encoded><![CDATA[<h3>Replacing Google Site Search and Google Search Appliance with a Modern, Customizable Open Source Solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7jwGNhA7yNNY3n6-xhq62A.jpeg" /><figcaption>Stockholm City Public Library</figcaption></figure><p>As you might have heard Google <a href="https://www.wordstream.com/blog/ws/2017/04/14/google-site-search-retirement">announced</a> that they’re discontinuing their out-of-the box search product Google Site Search (GSS). Furthermore Google also <a href="https://www.searchtechnologies.com/blog/gsa-discontinued-frequently-asked-questions">stopped</a> selling new licences of their Enterprise product Google Search Appliance (GSA) and is going to stop supporting it by March 2019. Therefore many customers were forced to look for a new product in order to replace those solutions or, regarding GSA, are still looking into this.</p><p>One of our customers was facing this exact problem since they relied heavily on those two products for their overall search experience throughout hundreds of pages and domains. Their solution back then was based on a central API endpoint which aggregated search results from various backends such as custom search engines to GSS and GSA. Since we are responsible for that API implementation we got the opportunity to replace those deprecated systems, namely GSS and GSA.</p><p>Our deep knowledge of <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a>, which fits the customer’s stack very well, led us to the decision of using it as our search index system after evaluating a couple of other solutions. The main challenge remained in finding an open source software to crawl dozens of sites in an efficient way. We looked into several products and ended up going for <a href="http://nutch.apache.org/">Apache Nutch</a>. If you want to know more about this decision make sure to check out <a href="https://blog.smartive.ch/replace-google-gsa-with-a-custom-search-engine-and-crawler-813838691a2">Thilo’s blog post</a> which he wrote on this topic.</p><h3>Multilingual Content</h3><p>One of the big challenges we were facing was the fact that the customer’s various web sites were written in three different languages; German, French and Italian. Apache Nutch provides a <a href="https://wiki.apache.org/nutch/LanguageIdentifierPlugin">language detection plugin</a> which works really well by adding a new meta field containing the language identifier to a Nutch document.</p><p>In order to search for documents across multiple languages efficiently we had to make sure the correct stemming methods and search analyzers for different languages are applied in the search index. We chose to store documents in different indices per language as recommended by the <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/one-lang-docs.html">Elasticsearch documentation</a>. Unfortunately this was not a built-in feature by Nutch so we decided to implement that feature ourselves and contribute it back to the project. My co-worker <a href="https://blog.smartive.ch/@nmaro">Nicola</a> took over this work and submitted a <a href="https://github.com/apache/nutch/pull/257">pull request</a> which got merged into Nutch.</p><p>As a result we have four different indices now, three for the main languages and one so-called sink index. All documents with unrecognised languages (other than German, French or Italian) are kept in this index. This solution allows us to only search documents in the user’s main language. As an example, a German user would retrieve documents only from the German and, with a lower scoring applied to them, the sink index. This guarantees that matching documents written in German are ranked higher than any other documents and therefore appear first in the search results, resulting in a much better user experience.</p><h3>Configuration Management</h3><p>Since both GSS and GSA offered a web admin interface or so called “admin console” to configure both the crawler and the search experience we aimed to do so as well. We went for something in between those two products, offering both the simplicity of GSS, but also keep the solution as extendable as possible in order to aim towards the goals of offering the administrative users a large set of configuration settings just as GSA does. As agreed with the customer’s power users, we started offering the two most used settings; editing seed lists (the list of URLs to crawl) as well as black- / whitelisting of URLs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8OedIhWtLsRWLu1anKsU0A.png" /><figcaption>Search Admin Console built with React and Material design</figcaption></figure><p>On the technical side we decided to go for a modern set of technologies and frameworks. For the frontend we’re using React, as well as Material design for the UI. This allows us to provide the user with a shiny interface that makes it easy and fun to configure our search solution. The frontend talks to an API which is written in TypeScript and makes use of our very own routing framework <a href="http://giuseppe.smartive.ch/">Giuseppe</a>. One of the customer’s requirements was the ability to log into the admin console using their Active Directory accounts. Using <a href="https://jwt.io/">JSON Web Tokens</a> together with a LDAP connector we’re able to fulfil this requirement and allow the client to give access to new people by applying the correct permissions in their Active Directory.</p><h3>Monitoring &amp; Analysis</h3><p>One of the key features our customer demanded was good monitoring over what’s going on with the crawler. We’re using Filebeat in order to parse Nutch’s log files and log anything that indicates problems with the crawling process into Elasticsearch in order to allow the administrator to look into it using Kibana. The same Filebeat instance also parses the admin console’s log files and ships them to Elasticsearch, allowing us to detect any authentication attacks.</p><p>Kibana is also used for Search Index Analysis, meaning we’re giving an administrator the possibility to review what’s in the search index at any moment. This helps with finding possible problems or missing documents and allows those users to visualize the data that got indexed.</p><h3>Architecture &amp; Deployment</h3><p>Since we strongly believe in container-based microservice architectures and their advantages we set up our services using <a href="https://www.docker.com/">Docker</a> images right from the beginning. Some parts of the architecture were already there such as the Elasticsearch cluster as a standalone system or the Search API which runs on a Cloud Foundry stack. Everything else, from crawler to the admin console, we implemented using customized Docker images and split them logically in a way that makes sense in order to guarantee good maintainability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_ZVC5_EujRnq-vMnqQbkA.png" /><figcaption>System Architecture (simplified)</figcaption></figure><p>This approach using containers also allows us to easily develop the whole system locally using the exact same images as on production together with docker-compose.</p><p>During deployment we build all images, test them and if tests are being passed we publish them in the customer’s internal <a href="https://about.gitlab.com/2016/05/23/gitlab-container-registry/">GitLab Docker registry</a>. Once the images are published using Semver we first stop all running crawling jobs before safely deploying them onto our production (there are also testing and development environments of course) Docker system. The crawler gets started and picks up new crawling jobs from its internal crawl database. There are no downtimes for users or any administrators.</p><h3>Testing</h3><p>Testing a complex system, such as a search solution consisting of multiple sub-systems written in multiple languages, can be a tricky thing. That’s why in order to do full integration testing we decided to split the overall solution into two sections; the crawler and the search index on one side and the search admin console on the other. Again we’re using the exact same Docker images as described in the architecture section above which ensures their functionality when going into production.</p><p>In order to properly test crawling and indexing data from actual web sites we decided to set up <a href="http://www.mock-server.com/">MockServer</a>, a simple Docker container that allows us to programatically serve static fixture files. These mocked files are getting crawled by Nutch before being indexed into Elasticsearch. We chose <a href="https://facebook.github.io/jest/">Jest</a> as our testing framework, mainly because of its ability to implement simple snapshot based tests which work great in combination with Elasticsearch since it’s all JSON. The search admin console also is being tested using Jest which is a lot easier in this case since everything is written in React and JavaScript.</p><p>Overall we now have a testing setup which makes us feel more secure if we do adjustments to any part of the system and let’s us sleep well at night even before major production releases.</p><h3>What’s next?</h3><p>As one might assume this whole project, even though already running very well in production, is still in the process of being improved in order to offer our customers the best possible solution there is. We got a couple of things on our to-do list, such as:</p><h4>Overall search experience</h4><ul><li>Search ranking based on popularity algorithms</li><li>Optimization of the ranking algorithm based on analytics feedback</li><li>Add “Did-You-Mean” features and spelling correction</li><li>Personalization of search results based upon user preferences and analytics</li><li>A/B testing of different ranking algorithms</li><li>Individualized searches for different platforms on the same data source</li></ul><h4>Search Admin-Console</h4><ul><li>Allow administrators to adjust field boosting / scoring for document search</li><li>Add possibility to edit synonyms per language (synonyms are currently being handled in the search index templates)</li><li>Add a dashboard which visualizes the key components of the system (crawler and index)</li><li>Allow boosting of certain documents for marketing purposes</li></ul><h4>Analytics / Reports</h4><ul><li>User experience analytics in terms of analyzing which results are being clicked on the most and their actual ranking among other results</li><li>Feed usage data back into the search engine for better result ranking</li><li>Track search queries and clicked search results for evaluation of <a href="https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Precision_at_K">Precision@K</a> and <a href="https://en.wikipedia.org/wiki/Mean_reciprocal_rank">MRR</a> to improve the search algorithm with statistical information</li><li>Perform an in-depth analysis using analytics data to provide a foundation for further optimization of the Precision@K and to improve the overall search experience</li></ul><h3>Interested?</h3><p>If you’re looking for a replacement of either Google Site Search or Google Search Appliance (or any other search solution of course) we’re happy to help. We believe that by using only open source software we’re able to adjust to the market’s need very quickly and build highly customised, sustainable and user-friendly search solutions. Do you have any requirement standardised solutions cannot solve? We’re happy to provide you with a solution that fits your needs and exceeds your expectations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9aMLLJppZZ6j-OpEta5r6g.png" /></figure><p>Feel free to contact us at any time: <a href="/service/mailto:hello@smartive.ch">hello@smartive.ch</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf5370bfd18c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartive/replacing-google-site-search-and-google-search-appliance-with-a-modern-open-source-solution-bf5370bfd18c">Replacing Google Site Search and Google Search Appliance with a modern Open Source Solution</a> was originally published in <a href="https://medium.com/smartive">smartive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[smartive.ch goes Gatsby.js]]></title>
            <link>https://medium.com/smartive/smartive-ch-goes-gatsby-js-27a056b3b817?source=rss-6c3aa195d9bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/27a056b3b817</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[gatsbyjs]]></category>
            <category><![CDATA[website]]></category>
            <category><![CDATA[react]]></category>
            <dc:creator><![CDATA[Moreno Feltscher]]></dc:creator>
            <pubDate>Wed, 27 Sep 2017 11:07:53 GMT</pubDate>
            <atom:updated>2017-09-27T12:19:32.529Z</atom:updated>
            <content:encoded><![CDATA[<p>At smartive we always saw the potential of static site generators. After using Wintersmith and Next.js we recently made the switch to Gatsby.js for our company site. A decision we don’t regret. Here’s why.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqyA7LbmCqUecrqCNfcgUg.png" /></figure><p>As mentioned we already built our company website using static site generators early on. Last year, we finally made the switch from our good old custom PHP application with little to no logic, which served us for about two years, to <a href="http://wintersmith.io/">Wintersmith</a>. At that time this was one of the leading static site generators based on Node.js. Since most of our applications at that time were already JavaScript based it seemed to be the perfect fit. The fact that our company was undergoing a complete rebranding in terms of our corporate identity came in quite handy as well.</p><p>After running Wintersmith for almost a year we encountered its limits. Some of the major drawbacks at that time were:</p><ul><li>No code splitting, resulting in the client having to load a bunch of unnecessary JavaScript and CSS files</li><li>Pulling in external resources, such as blog posts, was quite unhandy and sometimes even impossible</li><li>Build process optimization was almost impossible and in our case resulted in a Webpack setup on top of Wintersmith which was not really maintainable</li></ul><h3>React to the Rescue!</h3><p>Since by the time we encountered the problems described above we already had deep knowledge of React we started looking for an alternative based on that hot new thing.</p><p>The first thing that caught our attention was <a href="https://github.com/zeit/next.js/">Next.js</a>, as seemingly everyone going for a server-side rendered React app was using it. After some days hacking on our app we encountered some issues, especially when it came to frontend rendering. We chose <a href="https://prismic.io/">prismic.io</a> for our backend system which served all the content. Although this felt right at first but, as all of us are developers, working around the constraints of it just didn’t feel right.</p><p>Luckily at that time <a href="https://www.gatsbyjs.org/">Gatsby.j</a>s version 1.0 <a href="https://www.gatsbyjs.org/blog/gatsby-v1/">just got released</a> and we decided to give it a try during one of our so-called Hackdays. We instantly fell in love with the simplicity of the system. Our first approach was to just use all the components which we already had created for Next.js and backed it by simple JSON files containing the content we wanted to serve using the amazing yet simple GraphQL-based pull-in mechanism Gatsby.js provides. This was accomplished by using the <a href="https://www.npmjs.com/package/gatsby-transformer-json">gatsby-transformer-json plugin</a> internally. Keep in mind that our content rarely changes, so this was always the way we wanted it to be (without knowing for some time as we had to admit to ourselves).</p><p>One of our main goals all along was to show our latest blog posts on Medium. Unfortunately at that time there was no plugin to achieve this so we decided to write one ourselves. By the time of writing I’m proud to say we successfully did so and even <a href="https://github.com/gatsbyjs/gatsby/pull/1907">contributed it back to the community</a>. Make sure to check it out if you’re interested in a similar solution.<br>The only thing left was to actually rebuild and deploy our site once a new blog post gets released on Medium. We chose <a href="https://ifttt.com/">IFTTT</a> for this task, mainly because of its simplicity. Every time IFTTT picks up a newly published blog post it triggers a GitLab CI pipeline using a webhook, which then rebuilds and deploys our application onto our Docker Cloud infrastructure.</p><p>The result of our work using Gatsby.js is an outstanding <a href="https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fsmartive.ch&amp;tab=desktop">Google PageSpeed score</a> thanks to the built-in code splitting and cache handling mechanisms. If you’re interested in how <a href="https://smartive.ch/">smartive.ch</a> is built you can have a look at our code <a href="https://github.com/smartive/smartive.ch">on GitHub</a>.</p><p>We are really looking forward to build other cool stuff for our customers using Gatsby.js!</p><p>Special thanks goes to <a href="https://medium.com/u/c10858e435f0">Robert Vogt</a> who did most of the work on our website and contributed the Medium plugin mentioned above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27a056b3b817" width="1" height="1" alt=""><hr><p><a href="https://medium.com/smartive/smartive-ch-goes-gatsby-js-27a056b3b817">smartive.ch goes Gatsby.js</a> was originally published in <a href="https://medium.com/smartive">smartive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>