<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sam Dutton on Medium]]></title>
        <description><![CDATA[Stories by Sam Dutton on Medium]]></description>
        <link>https://medium.com/@samdutton?source=rss-84c8e3357532------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*yksgkVDrU0iLi-uI1aus0A.jpeg</url>
            <title>Stories by Sam Dutton on Medium</title>
            <link>https://medium.com/@samdutton?source=rss-84c8e3357532------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:45:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samdutton/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How do you use images?]]></title>
            <link>https://medium.com/@samdutton/how-do-you-use-images-5ffa9130c6db?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ffa9130c6db</guid>
            <category><![CDATA[responsive-design]]></category>
            <category><![CDATA[responsive-images]]></category>
            <category><![CDATA[web-images]]></category>
            <category><![CDATA[pictures]]></category>
            <category><![CDATA[srcset]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Thu, 18 Jul 2019 16:43:06 GMT</pubDate>
            <atom:updated>2019-07-19T14:39:14.775Z</atom:updated>
            <content:encoded><![CDATA[<h3>Survey: how do web developers use images?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/0*C4JY933QgT05e0Hu" /><figcaption>Watson</figcaption></figure><p>We want to find out more about the ways you use images.</p><p>Do you use responsive images, lazy loading, or other optimizations?</p><p>If not, why not? What affects your decisions?</p><p>It would really help if you could answer a few simple questions with our online survey: <a href="https://forms.gle/vF9SW95simXKoZrKA">forms.gle/vF9SW95simXKoZrKA</a>. We’ll collect responses for two weeks, then publish a summary of the results.</p><p>If you don’t have time for the survey, you can just reply to this post.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ffa9130c6db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitHub Pages + Cloudflare: custom domain checklist]]></title>
            <link>https://medium.com/@samdutton/github-pages-cloudflare-custom-domain-checklist-e86c786194a4?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/e86c786194a4</guid>
            <category><![CDATA[cloudflare]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[cloudflare-hosting]]></category>
            <category><![CDATA[github-pages]]></category>
            <category><![CDATA[website-hosting]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Wed, 20 Mar 2019 15:16:52 GMT</pubDate>
            <atom:updated>2019-09-09T16:38:07.791Z</atom:updated>
            <content:encoded><![CDATA[<p>GitHub Pages + CloudFlare is one of the easiest ways to set up a website.</p><p>I’ve done this a few times recently, so I put together a checklist below of things not to forget.</p><p>This is not intended as a complete guide, but please let me know if I missed something.</p><h3>Domain host</h3><ul><li>Add Cloudflare nameservers aida.ns.cloudflare.com and plato.ns.cloudflare.com:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*d3nl9Vaevkpodt2l9iHHhw.png" /><figcaption>Set nameserver values for Cloudflare</figcaption></figure><h3>GitHub settings for your repo</h3><ul><li>Set the branch and directory you want to use for the GitHub Pages site. Note that <strong>you don’t have to use a <em>gh-pages</em> branch</strong>:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1OwahQ-fAtKhFiWLIXhkXA.png" /><figcaption>From the settings page for your repo on GitHub, set the custom GitHub Pages</figcaption></figure><ul><li>Set the custom domain for the site:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGyBllpynNLgkQfgyf9qPw.png" /></figure><h3>Repo</h3><ul><li>Add a plain text file named CNAME to the GitHub Pages branch/directory you specified in the repo settings (see above). This file should only contain your custom domain:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*yag_RTsGBBGHuyKLX0oEwA.png" /><figcaption>Add a CNAME file with your custom domain to the branch/directory for your GitHub Pages site</figcaption></figure><h3>Cloudflare settings</h3><ul><li>From the Cloudflare DNS settings for your site, add A records pointing to GitHub (185.199.108.153, 185.199.109.153, 185.199.110.153 and 185.199.111.153). This works for any apex domain (such as example.com):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*hll0q-wksoAtOZAF81lDWA.png" /><figcaption>Add A records to the Cloudflare DNS settings for your site</figcaption></figure><ul><li>You might also want to specify that www URLs redirect:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*B1fy5JLOAbWfNwuWHirQEw.png" /></figure><ul><li>From the Crypto page, change settings to always use HTTPS:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*6Mp5Ph64ixSdH97Yhiz19A.png" /><figcaption>Set your site to always use HTTPS</figcaption></figure><ul><li>Add a page rule to always use HTTPS:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*3PEAoCMjHQfudJExMc8xUw.png" /><figcaption>Add a Page Rule to always use HTTPS</figcaption></figure><h3>Troubleshooting</h3><h4>404 error</h4><p>Check that you put the CNAME file in the right directory. For example, if you have multiple directories under your <em>docs</em> directory.</p><p>Also make sure that you set the custom domain from GitHub settings, especially if you move the CNAME file.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e86c786194a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Website prelaunch checklist]]></title>
            <link>https://medium.com/@samdutton/website-prelaunch-checklist-14d0002b4c7c?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/14d0002b4c7c</guid>
            <category><![CDATA[pwa]]></category>
            <category><![CDATA[website-performance]]></category>
            <category><![CDATA[progressive-web-app]]></category>
            <category><![CDATA[accessibility]]></category>
            <category><![CDATA[seo]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Fri, 16 Nov 2018 00:56:07 GMT</pubDate>
            <atom:updated>2018-11-16T00:56:07.828Z</atom:updated>
            <content:encoded><![CDATA[<p>Before you launch a website, it’s crucial to check for simple mistakes.</p><h3>Code</h3><p>If you don’t have linting in place for all your code, now is the time!</p><p>• <a href="http://htmlhint.com/">HTMLHint</a><br>• <a href="https://stylelint.io/">stylelint</a><br>• <a href="http://eslint.org/">ESLint</a></p><p>These tools can all be used as editor plugins or run from your build process and with continuous integration tools such as <a href="https://travis-ci.org/">Travis</a>.</p><h3>Accessibility</h3><p>It’s quick and easy to do an <a href="https://developers.google.com/web/fundamentals/accessibility/how-to-review">accessibility review</a> that will benefit all your users and reduce the risk of problems post-launch.</p><p>If nothing else, run a <a href="https://developers.google.com/web/tools/lighthouse/">Lighthouse</a> accessibility audit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbE2-kfOsknCFCqnFxybew.png" /></figure><p>Other tools include <a href="https://wave.webaim.org/sitewide">Wave</a>, <a href="https://tenon.io/">Tenon</a> and <a href="https://www.deque.com/#">axe</a>, all of which can be run programatically or from a browser extension. Make sure to test examples of all relevant content, and not just your home page.</p><h3>Broken links and images</h3><p>There are lots of tools to test for broken links or missing images at build and runtime, including Chrome extensions (<a href="https://chrome.google.com/webstore/detail/check-my-links/ojkcdipcgfaekbeaelaapakgnjflfglf">this one</a> is good) and Node tools.</p><p><a href="https://github.com/stevenvachon/broken-link-checker">Broken Link Checker</a> can be used via an API or from the command line:</p><pre>$ npm install broken-link-checker -g<br>$ blc <a href="http://yoursite.com/">http://yoursite.com</a> -ro</pre><h3>Spelling</h3><p>There are several free tools to check spelling. <a href="http://aspell.net/">Aspell</a> is a good one.</p><p>Here’s how to check all the Markdown files in your repo (other installation tools are available!):</p><pre>$ brew install aspell<br>$ for f in **/*.md; do aspell check $f; done</pre><p><a href="https://twitter.com/alexwlchan">Alex Chan</a> wrote a <a href="https://alexwlchan.net/2016/09/please-use-aspell/">great guide</a> that goes into more detail.</p><h3>Images</h3><p>It’s easy to end up with images that are poorly compressed or larger in pixel dimensions than necessary. You may also have images that use the wrong format, such as PNG for photos.</p><p>One simple technique for checking is to search your repo for images, order by size, then optimise as necessary. Here’s an example on Mac:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h2LWCp71H5vgc_Cb" /></figure><p>This immediately highlights problems. Some images need individual attention — <em>hero.png</em> looks problematic. Open the original asset in an image editor, check the dimensions, compression settings and format (maybe it should be a JPEG?)</p><p>Better still, you can use a tool like <a href="https://imageoptim.com/">ImageOptim</a> to optimise all the files in place:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bNvaZRa5iOv96F8ATcILig.png" /></figure><p>Here’s a screencast of the whole process:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F6caW1ZtPJF4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6caW1ZtPJF4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F6caW1ZtPJF4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb4e6242a93d4991e1f0be19f8eb408a/href">https://medium.com/media/eb4e6242a93d4991e1f0be19f8eb408a/href</a></iframe><h3>Random glitches</h3><p>Minor content problems are often repeated. For example, on a site for web developers:</p><ul><li>Inconsistent capitalisation and spelling: JavaScript not Javascript</li><li>Inconsistency with spacing before units: 10 MB not 10MB</li><li>Curly quotes in code samples</li></ul><p>You can check manually (search across your repo from your code editor) or write a test script to check for common problems. Web Fundamentals <a href="https://github.com/google/WebFundamentals/blob/8ccac682487125126fb84587881b7e630adca11e/gulp-tasks/tests/commonTypos.js">does this really well</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iSRO4ycYlN-jtvZwZxMnXA.png" /></figure><p>Automated tests are also a great way to maintain an editorial stylesheet: a team standard for spelling, capitalisation and other elements of house style.</p><h3>Prepare for 404s</h3><p>There are lots of ways to wind up with 404 errors: pages inadvertently get moved or deleted, and external sites get links wrong. You should prepare for the worst by using analytics.</p><p>It’s easy to set up a <a href="https://support.google.com/analytics/answer/1151300?hl=en">custom report</a> from Google Analytics to keep tabs on what’s going wrong:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*nmmVetll9Iitwuj1yEeyOQ.png" /></figure><p>Here’s an example of a report:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*DJ4HORR03Ufzie9flz0ryA.png" /></figure><p>Armed with this data, you can fix the most urgent problems and potentially add redirects as a workaround.</p><h3>Social tags</h3><p>It’s critical that <a href="https://css-tricks.com/essential-meta-tags-social-media/">social sharing tags</a> work properly at launch time.</p><p>Google’s <a href="https://search.google.com/structured-data/testing-tool">structured data testing tool</a> is great for testing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dyc1r8khUPu6v1rUYQXZZQ.png" /></figure><p>While you’re at it, make sure everything in your &lt;head&gt; is shipshape: check out Josh Buchea’s <a href="https://github.com/joshbuchea/HEAD">HEAD guide</a>.</p><h3>Performance, SEO, security</h3><p>The following tools are all useful for quick and easy prelaunch sanity testing:</p><ul><li>Run <a href="https://developers.google.com/web/tools/lighthouse/">Lighthouse</a> and <a href="https://developers.google.com/web/tools/lighthouse#gists">save the report data</a>. You get a quantified baseline and a todo list for improvements.</li><li><a href="https://www.webpagetest.org/easy">WebPageTest</a>: check performance for different locations and connection types, caching, time to first byte, CDN usage.</li><li><a href="https://developers.google.com/web/tools/chrome-devtools/security">Chrome DevTools Security panel</a>: HTTPS usage.</li><li><a href="https://developers.google.com/web/tools/chrome-devtools/network-performance/">Chrome DevTools Network Panel</a>: load timings, resource sizes and number of requests. Check for outliers.</li><li>Chrome Task Manager: if your site constantly uses significant CPU or more memory than other apps then you may need to fix memory leaks, task running or resource loading problems. Make sure to test your site on devices representative of your users.</li><li><a href="https://developers.google.com/speed/pagespeed/insights/">Pagespeed Insights</a>: load performance, data cost and resource usage, including Chrome User Experience report data highlighting real-world performance statistics.</li><li><a href="https://www.thinkwithgoogle.com/feature/mobile/">Speed Scorecard and Impact Calculator</a>: compare site speed against peers and estimate the potential revenue opportunity of improving site speed.</li></ul><p><a href="https://github.com/GoogleChrome/lighthouse">Lighthouse</a>, <a href="https://sites.google.com/a/webpagetest.org/docs/advanced-features/webpagetest-restful-apis">WebPageTest</a> and <a href="https://developers.google.com/speed/docs/insights/v5/reference/pagespeedapi">Pagespeed Insights</a> can all be run programatically, so you can integrate testing with your build process.</p><p>Google Search Console has a range of tools to ensure your site is discoverable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjmWhgzc53AcPQuRp_GcKA.png" /></figure><p><a href="https://chrome.google.com/webstore/detail/tag-assistant-by-google/kejbdjndbnbjgmefkgdddjlbokphdefk">Tag Assistant</a> is an extension for troubleshooting Google tags, including Analytics. Google Analytics can also be used to <a href="https://support.google.com/analytics/answer/1205784?hl=en">track site speed</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aShkW1al-9d6ph4mDDG1oQ.png" /></figure><p>Make sure to test a representative set of content — not just the home page. For example, on an e-commerce site, you might want to test product pages, category pages, search results and the checkout page.</p><p>Whatever you do, test your site on a range of devices—make sure to try using a cheap phone and poor connectivity (emulated if necessary). Cajole friends and colleagues to attempt to break your site—fresh eyes find new bugs. Build <a href="https://www.nngroup.com/articles/discount-usability-20-years/">discount usability testing</a> into your workflow. Simple, iterative user testing, done early, with a few participants, will help avoid expensive mistakes.</p><blockquote>If you have other suggestions for prelaunch checks, let me know in the comments!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14d0002b4c7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build a great search box]]></title>
            <link>https://medium.com/@samdutton/how-to-build-a-great-search-box-2b9a6d1dce0d?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b9a6d1dce0d</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[usability]]></category>
            <category><![CDATA[ux]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[html]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Mon, 17 Sep 2018 16:26:01 GMT</pubDate>
            <atom:updated>2018-09-19T17:23:16.631Z</atom:updated>
            <content:encoded><![CDATA[<p>High quality search is fundamental to e-commerce — and the gateway to search is the humble input element.</p><p>A prominent, well designed search input element invites engagement and interaction. It’s a way of saying ‘Welcome! How can I help?’ However, there’s a double distance between success and failure. A good search experience means more conversions and better discovery, but poorly implemented search annoys customers and turns them away.</p><p>This article shows some simple ways to make the search input an invitation, not a barrier.</p><h3>tl;dr</h3><p><strong>The ‘search box’ is probably the single most important component of any e-commerce site.</strong></p><ul><li>Make search visible and prominent</li><li>Put search where people expect it, on every page</li><li>Make the search input large enough to enter text and work as a touch target</li><li>Give the search input a label and a magnifying glass icon</li><li>Don’t force users to click on the search icon to display the text input</li><li>Show recent searches</li><li>Provide suggestions and completions</li><li>Enable synonym search or other methods of dealing with ambiguous queries</li><li>Be ‘typo tolerant’</li><li>Retain query text</li><li>Consider voice-generated queries</li><li>Test!</li></ul><p>Here are three demos if you want to dive straight into code:</p><ul><li><a href="https://codepen.io/samdutton/pen/djrVVd">Responsive search input using JavaScript</a></li><li><a href="https://codepen.io/samdutton/pen/MBxxbm">Responsive search input using a form</a></li><li><a href="https://codepen.io/samdutton/pen/ZjZgNP">Responsive header using Grid and Flex, with branding, search and cart</a></li></ul><blockquote>The search field represents the initial step of the user’s search process … design of the search field is instrumental at an even more fundamental level — the field design can influence the user’s entire approach to how they find products on the site.</blockquote><blockquote>— <a href="https://baymard.com/blog/search-field-design">Baymard Institute</a></blockquote><h3>Make it visible and prominent</h3><p>People give up quickly, attention spans are short. If users can’t find something immediately, they’re likely to give up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ILvNVFYka8q04RrsHTvkKA.png" /><figcaption>Amazon and eBay on mobile: search is prominent</figcaption></figure><p><a href="https://www.nngroup.com/articles/recognition-and-recall/">Showing users things they recognize improves usability</a>. This might seem obvious, but it’s surprising how many sites get this wrong. Successful e-commerce sites tend to use simple, obvious search inputs without much styling.</p><p>If you really don’t want to do that, at least make sure the search input still looks like a search input.</p><p>The e-commerce homepage below looks lovely — but where’s the search?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YiefZMfVtlHnKnTiUjQ2pA.png" /><figcaption>Where’s the search box?</figcaption></figure><p>How-do-I-search here fails the <a href="https://en.wikipedia.org/wiki/Don%27t_Make_Me_Think">Don’t Make Me Think!</a> test — and shows that search input prominence is about contrast as well as size.</p><p>On the mobile version of this page, ‘where to search’ is clearer, but it can take a moment to work out how to enter query text, since the UI uses non-standard input styling. The input doesn’t look like a ‘search box’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIZPBzAYZ-9JxcD0faKb1A.png" /><figcaption>Is that a search box, a link, or a heading?</figcaption></figure><p>Non-standard input designs of this kind can also be problematic because they <a href="https://joshuawinn.com/ux-input-placeholders-are-not-labels/">look like headings, not text inputs</a>, as in the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xAZyJcyUJXBRGFMdaqqgAQ.png" /><figcaption>Search input that looks a bit like a heading</figcaption></figure><blockquote>For every rule, there’s an exception.</blockquote><blockquote>A well designed, prominent search input will encourage people to navigate your site via search.</blockquote><blockquote>However, some retailers deliberately downplay the prominence of search UI to encourage browsing rather than searching.</blockquote><blockquote>This is often the case for sites selling clothes. For example, it seems pretty likely that <a href="http://www.topman.com/">Topman</a> and <a href="https://www.urbanoutfitters.com/">Urban Outfitters</a> (see below) know what they’re doing: encouraging people to browse.</blockquote><blockquote>The point is that you should make search prominent unless there’s a good reason not to. De-emphasising search should only be done as a deliberate choice.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*t-R2k8al9MGrLTnI" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQc5jJQY5YYeJmoakmR85g.png" /></figure><h3>Put search where users expect it</h3><p>The best-known e-commerce sites all put the search input at the top of the page, generally in the middle or at the right (or on the left for right-to-left languages). There are several examples <a href="https://baymard.com/ecommerce-search/benchmark/page-types/search-field">here</a> — though mostly from US sites.</p><p>Putting search somewhere else will force users to look where they’re not expecting. Don’t do it!</p><h3>Put search on every page</h3><p>Search should be accessible from every page including (especially!) on About, Contact, 404, and other types of non-product pages.</p><p>Be careful to clearly disambiguate product search from other type of search such as a store locator or account search. The simplest way to do that is to keep product search at the top of the page, and other types of search in the middle of the page.</p><h3>Make it accessible</h3><p>There are several simple ways to make input elements more accessible.</p><h4>Provide a label</h4><p>Give the input a label and give the label a for attribute:</p><pre>&lt;label for=&quot;search&quot;&gt;Search&lt;/label&gt;<br>&lt;input id=&quot;search&quot;&gt;</pre><p>Adding the for attribute means the input gets focus when a user taps, clicks or otherwise selects the label. This means the label then becomes an extension of the input as a target, which is especially useful for people with motor disabilities, and anyone using a site in contexts where it’s difficult to click or tap accurately (for example, on a moving train).</p><p>You can wrap the input element with the label if you don’t want to use the for attribute, or you don’t want to use an ID for the input:</p><pre>&lt;label&gt;Search: <br> &lt;input&gt;<br>&lt;/label&gt;</pre><p>This looks odd and feels semantically wrong (the input is part of the label?) but the markup is valid.</p><p>If you <em>must</em> hide the label visually, <a href="https://www.w3.org/WAI/tutorials/forms/labels/#note-on-hiding-elements">use the method recommended by the W3C</a>.</p><p>If you <em>really</em> don’t want to use a label, make sure to give the input a search icon.</p><h4>Avoid using a placeholder without a label</h4><p>There are <a href="https://www.smashingmagazine.com/2018/06/placeholder-attribute/">good reasons</a> not to use placeholder text without a label:</p><ul><li>Assistive technologies expect labels to describe the role of inputs. Assistive technologies <a href="https://www.w3.org/WAI/tutorials/forms/instructions/#placeholder-text">do not treat placeholder text as labels</a>.</li><li>The browser default for placeholder text color is light gray, which gives poor contrast against a white background. Styling the placeholder text for higher contrast makes it look like the placeholder text is actually search text!</li><li>A search input with text in it is less noticeable: <a href="https://www.nngroup.com/articles/form-design-placeholders/">users’ eyes are drawn to empty fields</a>.</li></ul><p>If you <em>must</em> use placeholder text, make it helpful. In the following example, what constitutes a ‘keyword’ — and why not just say ‘item number’?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*aF_uFHDCtnhGWURTsbymdA.png" /><figcaption>Maybe just Search?</figcaption></figure><h4>Consider using the search type attribute</h4><p>You can set the type of the input as search:</p><pre>&lt;label for=&quot;search&quot;&gt;Search&lt;/label&gt;<br>&lt;input id=&quot;search&quot; type=&quot;search&quot;&gt;</pre><p>This clarifies the purpose of the input for screenreaders and has the added advantage of adding an X icon for clearing the text (in Chrome, and Safari on desktop at least, with a <a href="http://nicolasgallagher.com/about-normalize-css/#normalize-css-corrects-common-bugs">selector for styling</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OZ8ZUCTaVmJRQL6sjh4h7A.png" /><figcaption>If you set input type to ‘<em>search’ you get an X icon to clear text on Chrome and Safari on desktop</em></figcaption></figure><h4>Add ARIA attributes</h4><p><a href="https://www.w3.org/WAI/tutorials/forms/labels/#using-aria-label">Adding an ARIA label</a> to your search input helps screenreaders and other assistive technologies:</p><pre>&lt;input type=&quot;search&quot; aria-label=&quot;Search&quot;&gt;</pre><p>Add labels to both the input and search button (if you use one):</p><pre>&lt;input type=&quot;search&quot; aria-labelledby=&quot;search-button&quot;&gt;<br>&lt;button id=&quot;search-button&quot; type=&quot;submit&quot;&gt;Search&lt;/button&gt;</pre><h4>Style for usability</h4><p>Make sure the input element is large enough to work as a touch target, and that the text styling provides <a href="https://www.w3.org/TR/WCAG20-TECHS/G18">adequate contrast</a>. Make text large enough to read and type: as a general rule, font-size should be at least 16px. Remember that a high proportion of users have less than perfect vision and/or may be attempting to search in sub-optimal conditions: bright sunlight, on a busy street or while in transit.</p><blockquote>Often, something that’s a complete blocker for a few users is also a pain point for many others, so by fixing the accessibility issue you improve the experience for everyone.</blockquote><blockquote>— <a href="https://developers.google.com/web/fundamentals/accessibility/">developers.google.com/web</a></blockquote><h3>Make it big enough</h3><p>Make sure there’s adequate space to tap or click into a search input.</p><p>Avoid this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2dAbJDtqUZtrfd3z6M3uRA.png" /><figcaption>Unstyled text input: small text, small target</figcaption></figure><p>Likewise, allow enough space to enter and display text without overflowing.</p><p>Avoid this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dO_R5mdlcV_KXkbN0lCzZg.png" /><figcaption>Poorly styled input: not enough width for search query</figcaption></figure><p>Major e-commerce sites virtually always have large search inputs: full width on mobile, and expanding to fit available width on desktop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ILvNVFYka8q04RrsHTvkKA.png" /><figcaption>Amazon and eBay on mobile: full-width search inputs at the top of the page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8FO_hFpN--uCNpyg" /><figcaption>eBay on desktop: full-width search input at the top of the page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m6WJSGDvtGdBF7Vy" /><figcaption>Amazon on desktop: full-width search input at the top of the page</figcaption></figure><p><a href="https://baymard.com/ecommerce-search/benchmark/page-types/search-field">This page</a> has over 100 other examples.</p><blockquote>On mobile a search input is a touch target just like any other UI component.</blockquote><blockquote>For many years, <a href="https://www.lukew.com/ff/entry.asp?1085">UI guidelines from Apple, Microsoft and others</a> have recommended touch target sizes corresponding to human finger size: a minimum physical display width and height of around 7–9 mm, with a minimum 2 mm gap between items.</blockquote><blockquote>Users may also want to select a search input using their thumb — and thumbs need larger targets than fingers.</blockquote><blockquote>Major e-commerce sites such as Amazon and eBay all follow this rule. Browsers use an even bigger input size for the URL bar.</blockquote><h3>Style it!</h3><p>The default browser styling for input elements is small and tight, using the default sans-serif font. This means that without adding CSS, input elements are much too small on mobile as a touch target, and unpleasantly constricted for viewing or entering text, even on desktop.</p><p>Unless you style it, an input element looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8_wnCelpfXmI5yO_WlOvQ.png" /><figcaption>Unstyled input element</figcaption></figure><p>You need to add adequate padding (at least 5px) and style text using a font that matches style for your site. You’ll probably also want to set the border width, colour (and possibly radius).</p><p><a href="https://www.johnlewis.com/">John Lewis</a> do this really well. Colors, type design and spacing follow house style and makes search feel like an integral part of the experience:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*igXy865pGcRtBx8Q" /></figure><p>Use CSS Flex and CSS Grid to ensure your your search input sizes responsively to fit mobile and desktop viewport widths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zWsQzRdcIz9o_lTY" /></figure><p>There is a complete code sample for this <a href="https://codepen.io/samdutton/full/djrVVd">here</a>.</p><h3>Add an SVG magnifying glass icon</h3><p>Most e-commerce sites display a magnifying glass icon next to or inside the search input element. Unless you have a very good reason not to, you should do the same. Consider your users’ expectations and follow the <a href="https://en.wikipedia.org/wiki/Principle_of_least_astonishment">Rule of Least Surprise</a>. The magnifying glass icon has been used for many years to represent search and is now one of the few icons that is <a href="https://www.nngroup.com/articles/magnifying-glass-icon/">well understood by most people</a>.</p><p>It’s straightforward to add an icon to an input with an inline SVG: see the <a href="https://codepen.io/samdutton/pen/MBxxbm">code sample</a>. SVG is <a href="https://caniuse.com/#search=SVG">supported by all modern browsers</a>, and many open source SVG magnifying glass icons are available, such as those from the <a href="https://thenounproject.com/term/open-source/">Noun Project</a> (and the one from <a href="https://material.io/icons/">material.io/icons</a> which is used in the <a href="https://codepen.io/samdutton/pen/MBxxbm">code sample</a>). Inline SVGs are also generally <a href="https://css-tricks.com/icon-fonts-vs-svg/">a better alternative to icon fonts</a>, especially if you’re only using one or two images: no additional requests, less page weight, less code to maintain and update.</p><h3>Don’t make users click to display the search input</h3><p>Many sites force you to click an icon to display the search input. This might not sound like a big deal, but every tap and click is one more hurdle between the user and a conversion.</p><p>Not displaying the search input makes search much less obvious. Usability research shows that when users want to search, they look for <a href="https://www.nngroup.com/articles/search-visible-and-simple/">‘the little box where I can type’</a>.</p><p>This page excerpted from an otherwise excellent site makes you click the magnifying glass, then enter text. Why not just show the search input?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L6UyZRg8SjYwbFtU1tiSPA.png" /><figcaption>Sample desktop e-commerce site: you need to click the magnifying glass to display the search input</figcaption></figure><p>On mobile, this is even more common.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tZyAGRME62KwptzhYFwHwQ.png" /><figcaption>Sample mobile e-commerce site: you need to tap the magnifying glass to display the search input</figcaption></figure><p>Why not just include the search input itself?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dXg-sKXI0Z_GJlEImk5XIg.png" /></figure><p>It’s worth noting that Amazon, eBay and many other major retailers always display the search input. The global fashion retailer Lyst found that replacing the search icon with a search box <a href="https://www.thinkwithgoogle.com/intl/en-cee/success-stories/local-case-studies/lyst-increases-overall-conversion-rate-25-making-usability-improvements/">increased usage by 43% on desktop, 13% on mobile</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kPj3KfGcNWY4paQS" /></figure><blockquote>Users may not understand that a magnifying glass represents search if the icon is heavily stylised.</blockquote><blockquote>For example, on some sites you need to understand that a circle with a tiny line to one side means ‘click on me to enter text so you can search’.</blockquote><blockquote>Consider visual and textual literacy: ‘lower-literacy users don’t scan text. As a result, for example, they can’t quickly glance at a list of navigation options to select the one they want.’ (<a href="https://www.nngroup.com/articles/writing-for-lower-literacy-users/">Norman Nielsen</a>).</blockquote><h3>Enable users to search by pressing return — or by clicking the search icon</h3><p>Sounds obvious, but many sites don’t do this.</p><p>To initiate search after entering query text, some people prefer to tap or click the search icon, whereas others like to press return. If a user tries either of these methods and it doesn’t work, they are forced to try again, or possibly just give up.</p><p>You can enable the tap/click-or-press-return combination by using a form, as shown in the code sample <a href="https://codepen.io/samdutton/pen/MBxxbm">here</a>.</p><p>Alternatively, you can use JavaScript as in the code sample <a href="https://codepen.io/samdutton/pen/djrVVd/?editors=1100">here</a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fsamdutton%2Fembed%2Fpreview%2FdjrVVd%3Fheight%3D600%26slug-hash%3DdjrVVd%26default-tabs%3Dcss%2Cresult%26host%3Dhttps%3A%2F%2Fcodepen.io%26embed-version%3D2&amp;url=https%3A%2F%2Fcodepen.io%2Fsamdutton%2Fpen%2FdjrVVd%2F%3Feditors%3D0010&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fi.cdpn.io%2F407537.djrVVd.small.90403b4d-af5d-4118-87cd-64101ca40306.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/69c7464b6e6096d5c88ec9de28d35dc6/href">https://medium.com/media/69c7464b6e6096d5c88ec9de28d35dc6/href</a></iframe><h3>Don’t make users sign in to search</h3><p>Enough said.</p><h3>Think twice about ‘instant search’</h3><p>In some contexts, it can make sense to display results as soon as a match is found while text is being entered in the search input. Algolia, for example, provides a JavaScript library for this, and they have a demo <a href="https://community.algolia.com/instantsearch.js/v2/examples/e-commerce/">here</a> that uses IKEA product data.</p><p>Pretty impressive — notice how the categories on the left update dynamically, as do the text highlights in the search results.</p><p>Instant search has the advantage of enabling users to try out alternative queries dynamically, without needing to tap, click or press return. There’s no need to navigate to a search results page. When done right, instant search feels highly responsive.</p><p>However, very few e-commerce sites use it: user action is required to retrieve results.</p><p>The problem is that ‘instant search’ feels hyperactive and unstable. It’s too ‘jerky’.</p><p>‘Instant’ search can also be janky an unpredictable on a poor connection — It’s hard for ‘instant’ to fail gracefully. Dynamic updates require processing and a request to the server every time the query changes, which can consume a lot of CPU, battery and available bandwidth.</p><p>A good tradeoff, which many sites now use, is not to update the content of the page, but provide a rich autocomplete experience, sometimes even with thumbnail previews of products. This enables a highly responsive input experience, with an acceptable fallback if connectivity is inadequate.</p><h3>Show recent searches</h3><p>A high proportion of searches are repeat searches: people often search for the same thing repeatedly, even during the same session.</p><p>Because of this it often makes sense to provide recent-search suggestions — and potentially cache assets for recent searches, such as images or product data.</p><p>Again, <a href="https://johnlewis.com/">John Lewis</a> do this well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*qvkRcyaGt71dXVd4" /></figure><p>A simple way to implement this is by using the form or input autocomplete attribute — see the <a href="https://codepen.io/samdutton/pen/MBxxbm">code demos</a>.</p><h3>Retain query text</h3><p>People often want to repeat a search, or refine a search by adding to a previous query. Being forced to re-enter search text is annoying — especially on mobile.</p><p>Retain text in the search input by saving the query value on the server, or by saving state on the client (for example, by using sessionStorage).</p><p>You can add functionality to enable clearing the input text — the standard UI component for this is an X in the input. Safari and Chrome on desktop provide this by default if you set the input type to <em>search</em>. You can see an example of this in the <a href="https://codepen.io/samdutton/pen/MBxxbm">code demos</a>.</p><h3>Provide suggestions and completions</h3><p>Automated suggestions and completions can make search quicker, easier and more successful:</p><ul><li>Reducing the need to type — especially important on mobile.</li><li>Suggesting products — potentially a product the customer was not aware of.</li><li>Reducing spelling errors.</li></ul><p>Automated completion can be as simple as adding an autocomplete attribute to the input element: see the <a href="https://codepen.io/samdutton/pen/MBxxbm">code sample</a>. This enables the browser to suggest completions based on recent searches. (Remember that <a href="http://www.cond.org/sigir07.pdf">a high proportion of searches are repeat searches</a>!)</p><p>Many search platforms offer enhanced autocomplete functionality. Algolia and other search engines provide <a href="https://github.com/algolia/autocomplete.js/blob/master/README.md">autocomplete integration</a> with a number of frameworks and platforms (there is a demo <a href="https://codepen.io/Algolia/pen/OXgkxv">here</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/0*i9suGusNPmyiZNDR" /></figure><p>Automated suggestions can help customers avoid irrelevant results. For example, a surf shop may sell surfboard wax, but surfboard descriptions may also mention <em>board </em>and <em>wax</em>, leading to irrelevant results for <em>board wax</em>, as in the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CiE5w3GiPGbAZwIv" /></figure><p>Providing category or product suggestions in the search input would fix that.</p><p>Automated suggestions also enable customers to clarify ambiguous queries: if a user searches for <em>light</em> the search input can suggest alternatives to narrow the search.</p><p>There are many more examples of auto-completion options <a href="https://baymard.com/ecommerce-search/benchmark/page-types/autocomplete-suggestions">here</a>.</p><h3>Enable synonym search</h3><p>This is crucial for product search, where people tend to use different words for the same thing: shoes or footwear, wellies or wellingtons or gumboots.</p><p>Synonym search is particularly important if you have customers from different regions who make different word choices.</p><h3>Be ‘typo tolerant’</h3><p>It’s easy to make mistakes while entering text, especially on mobile. ‘Typo tolerance’ and auto-correction are now expected by users as the norm for text entry.</p><p>Spelling can be particularly tricky for product names (‘Johnny Walker’ or ‘Johnnie Walker’?) and for product identifiers (‘GF1–9’ versus ‘GFI-9).</p><blockquote>A high proportion of people struggle with spelling.</blockquote><blockquote>Nielsen Norman estimate that <a href="https://www.nngroup.com/articles/writing-for-lower-literacy-users/">around 40% of people using the web have low literacy levels</a>. They recommend designing homepage content at a 6th grade reading level.</blockquote><blockquote>But even people with a high level of literacy can have difficulty getting spelling right — particularly those whose first language is not the language of your website.</blockquote><p>It’s true that <a href="https://lucene.apache.org/solr/guide/7_4/spell-checking.html">Solr</a>, <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/fuzzy-matching.html">Elasticsearch</a>, <a href="https://www.algolia.com/doc/guides/textual-relevance/typo-tolerance/">Algolia</a> and other search engines provide functionality to handle misspellings server-side. However, it’s generally better to try to correct input errors on the input side, instead of (or as well as) relying on the search engine itself. Auto-suggest and auto-complete functionality to correct spellings or alternative queries during input will usually work better than if a search engine needs to resort to techniques such as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness">fuzzy matching</a>, which can lead to irrelevant results (when you entered ‘shurts’, did you mean ‘shirts’ or ‘shorts’?).</p><p>Many search engines can cope with spelling errors by letting the user clarify on input:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/0*U2ftE4GNZ2I7gJLh" /></figure><h3>Find out what people search for</h3><p>People search and navigate in different ways for different types of products. You need to accommodate this from the input side — particularly when incorporating suggestion and completion.</p><p>When designing search input functionality, make sure to understand the different types of search queries used by your customers. Online stores need to be flexible about the way people want to find what they’re looking for — even for the same product:</p><ul><li>General text search: <em>google mobile, android phone</em></li><li>Product name: <em>pixel, pixel phone, pixel 2</em></li><li>Product specification: <em>pixel 64GB kinda blue</em></li><li>Product number: <em>G-2PW4100</em></li><li>Category: <em>phone, mobile, smartphone</em></li><li>Geospatial: <em>secondhand pixel kings cross</em></li><li>Search expressions: <em>mobile AND android</em></li></ul><p>You might want to consider broadening search queries that fail — for example substituting OR for AND, or expanding an exact-match search to include partial matches, or suggesting matches for similar queries.</p><p>Something is usually better than nothing!</p><p>What people search for varies between products and stores. For example, <a href="https://baymard.com/blog/search-field-design">UX testing from the Baymard Institute</a> shows that, ‘visually-driven industries such as apparel and home decoration in particular saw a clear bias toward category navigation — both in terms of the subjects’ initial preference and their subsequent success rate’. For your site, other types of search and navigation may be more dominant.</p><blockquote>For a global audience, all this functionality must work across different languages, character sets, text directionality, and geographical locations — and potentially handle linguistics and cultural differences.</blockquote><blockquote>For example, <a href="http://www.cjk.org/cjk/joa/joapaper.htm">Japanese stemming</a> is very different from the way it’s done for English, for example. Search engines and search services all provide different approaches to internationalisation and localisation.</blockquote><p>The point here is to increase conversions by understanding user behaviour. For that, you need to monitor input queries and observe how queries change during the day, week or year — potentially in real time, to learn how products are found (or not!) All search engine platforms offer logging tools, and <a href="https://support.google.com/analytics/answer/1012264?hl=en">Site Search</a> can be used with Google Analytics to help you log and analyse search behaviour.</p><h3>Consider voice input</h3><p>Some sites already use search query voice input on desktop as well as mobile — not least <a href="https://google.com/">google.com</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*ClP-dFu0hG61Tioi" /></figure><p>When entering text on Android or iOS, you can tap the microphone button in the keyboard UI, and speak instead of typing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaPnSz-pwW8-xqIuc0cJOw.png" /><figcaption>Speech input in keyboard UI for iOS and Android</figcaption></figure><p>On desktop browsers, this isn’t possible. To enable your site to handle speech input on desktop you will need to build your own UI and JavaScript components to initiate microphone access and convert speech to text, using the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API">Web Speech API</a> (demo <a href="https://simpl.info/stt">here</a>) or a service such as <a href="https://cloud.google.com/speech-to-text/">Google Cloud Speech to Text</a>.</p><p>Search without typing text can be invaluable in contexts where text entry is difficult — for example, while driving or cooking — and for people who have difficulty typing, or in regions with multiple dialects or spelling variations.</p><p>However, speech recognition requires a round trip to a server to provide a text result before then doing a search query. This uses battery and bandwidth and can result in frustrating inaccuracies and irrelevant results.</p><p>The text of a voice search can be very different from what’s entered via a keyboard and visual interface. For example, ‘Show me red party dresses under $200’ compared to a ‘party dress’ query refined by price and colour. The ability to handle natural language queries is becoming a critical requirement for search engines, as artificial intelligence, machine learning and digital assistant platforms become increasingly ubiquitous and capable.</p><p>All in all — try out voice search, and check if your customers are already using it on mobile. The blog post <a href="https://www.algolia.com/doc/tutorials/getting-started/build-voice-search/">Build a voice-to-text search</a> provides a brief summary of what’s required.</p><h3>Whatever you do, test</h3><p>Whenever you make changes, test the results and keep an eye on analytics and search logging.</p><p>Build <a href="https://www.nngroup.com/articles/discount-usability-20-years/">discount usability testing</a> into your workflow. You don’t need a fancy usability lab to improve search input design: simple, iterative user testing and prototyping, done early, with a few participants, will help avoid expensive design mistakes.</p><p>Make it easy for non-techies to monitor and understand search input data:</p><ul><li>What are people searching for most? What are people not searching for?</li><li>How does search input vary across platforms? When is search used for discovery rather than making a purchase?</li><li>When and why do some queries provide no results? Can these be fixed by adding synonyms, spellchecking, or other changes to search input? Are there data problems? Maybe you need to expand your product range.</li><li>When does a search lead to a sale or other type of ‘conversion’?</li><li>Do some searches result in users exiting the site rather than clicking on a result?</li><li>Does search input prominence detract from browsing — or vice versa?</li></ul><h3>Find out more</h3><ul><li>Baymard Institute: <a href="https://baymard.com/blog/search-field-design">E-Commerce Search Field Design and Its Implications</a> provides detailed consumer behaviour research. <a href="https://baymard.com/ecommerce-search/benchmark/page-types/search-field">This page</a> shows 110 different (though mainly US) examples of search field design.</li><li><a href="https://medium.com/dev-channel/how-to-add-full-text-search-to-your-website-4e9c80ce2bf4">How to add full text search to your website</a>: server and client-side alternatives for implementing search.</li><li><a href="https://www.nngroup.com/articles/form-design-placeholders/">Placeholders in Form Fields Are Harmful</a> and <a href="https://www.smashingmagazine.com/2018/06/placeholder-attribute/">Don’t Use The Placeholder Attribute</a>: good reasons not to use placeholders.</li><li>Google Analytics <a href="https://support.google.com/analytics/answer/1012264?hl=en">Site Search</a>: search analytics.</li></ul><h3>Demos</h3><ul><li><a href="https://codepen.io/samdutton/pen/djrVVd">Responsive search input using JavaScript</a></li><li><a href="https://codepen.io/samdutton/pen/MBxxbm">Responsive search input using a form</a></li><li><a href="https://codepen.io/samdutton/pen/ZjZgNP">Complete header element with branding, search and cart</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b9a6d1dce0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How (not) to teach web development]]></title>
            <link>https://medium.com/@samdutton/a-manifesto-for-teaching-web-development-ba886d4aeb27?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba886d4aeb27</guid>
            <category><![CDATA[css]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[html]]></category>
            <category><![CDATA[teaching-computer-science]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Thu, 14 Jun 2018 11:28:06 GMT</pubDate>
            <atom:updated>2022-05-25T18:21:35.908Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last two decades I’ve taught several Web Dev 101 courses.</p><p>To be honest, I don’t think any of those courses went very well.</p><p>In my experience, students start out excited and ready to learn, but soon glaze over because the content and teaching are boring or baffling or both.</p><p>By way of apology to those I have baffled and bored over the last 20 years, I’ve put together 10 principles for explaining HTML, CSS and JavaScript to absolute beginners.</p><p>I think we can do a better job of teaching basic web skills.</p><ol><li>Many people new to coding don’t use or understand hierarchical file systems. When you start, make sure everyone is comfortable with files and folders. Many problems for beginning web developers are caused by something ‘in the wrong place’.</li><li>Beginners make different types of mistakes than experienced developers. Be aware of this when you’re debugging. Show beginners what to watch out for: missing angle brackets (&lt;div class=&quot;foo&quot;), missing equals signs (&lt;div class&quot;foo&quot;&gt;), missing closing tags, copy-pasted curly quotes instead of straight quotes.</li><li>For live classes, consider <a href="https://en.wikipedia.org/wiki/Flipped_classroom">flipped classroom</a> techniques. Get students to learn at home (with specific goals and requirements) then complete activities with supervision in the classroom. Be upfront about the level of commitment expected. Provide a clear path: how to get help, where to find out more, what to do next.</li><li>Wherever possible, show don’t tell. Emphasise learning by doing. Online or live, avoid presenting or talking for more than a few minutes.</li><li>Describe an objective, then show how to achieve it: avoid putting the technology before the requirement. For example, demonstrate the need to set the maximum size of an image, then show how to do that with CSS (not the other way around). Focus on ‘getting things done’ rather than language features.</li><li>Avoid unnecessary information. For example, do not go into the gory details of CSS specificity algorithms or JavaScript function hoisting. (I’ve seen both of these in 101 courses.) Stick to the basics — that’s plenty.</li><li>Make examples as simple as possible. Get rid of all redundant content and code.</li><li>Avoid jargon, fussy nomenclature and unhelpful analogies. Concepts such as ‘the cascade’ and ‘the box model’ are quite easy to grasp in practice, without needing to know what they’re called. Formal naming just makes them sound difficult. Most students will intuitively understand that div.foo overrides div (ask them to guess!) Likewise for padding, borders and margins: good examples and simple diagrams work better than formal descriptions. Once students are comfortable with a conceptual model, you can mention the official name for it.</li><li>Present a single best-practice approach and, if necessary, go back later to show alternatives. <em>There’s more than one way to do it</em> is a healthy attitude to impart, but options can be confusing. In particular, beginner students do not need to consider browser compatibility handling.</li><li>Teaching and examples should follow best practice and industry norms: use semantic elements, ===, single quotes in JavaScript, double quotes in HTML — and so on. You don’t need to go into the whys and wherefores, just be consistent and get students to follow best practice by default from the start.</li></ol><p>Two extras:</p><ol><li>Don’t bother with the backstory, the history of the web, the things you used to do and why things are the way they are now. Don’t explain HTML5 versus HTML4 or why JavaScript was invented. Rather than recount the story of AJAX, just show the need to fetch assets and explain how to do that with fetch(). Web history doesn’t help students get started with coding.</li><li>Work out how to reach a variety of people and capture their interest via problems they need to solve. Tailor your content and teaching to your audience and what they want to achieve.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba886d4aeb27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to add full text search to your website]]></title>
            <link>https://medium.com/@samdutton/how-to-add-full-text-search-to-your-website-3f02a39545dd?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f02a39545dd</guid>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[apache-solr]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Tue, 20 Feb 2018 14:33:23 GMT</pubDate>
            <atom:updated>2018-02-20T14:33:23.045Z</atom:updated>
            <content:encoded><![CDATA[<p>Many types of website need efficient, accurate search.</p><p>This article explains server and client-side alternatives, and shows how to implement search that works offline.</p><h3>tl;dr</h3><p>There are lots of ways to do search:</p><ul><li>Via a back-end search engine such as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Elasticsearch</a> or <a href="http://lucene.apache.org/solr/">Solr</a>.</li><li>Commercial search services such as <a href="http://algolia.com/">Algolia</a> and <a href="https://aws.amazon.com/cloudsearch/">AWS Cloudsearch</a>.</li><li>Using a database with built-in search such as <a href="https://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html">MySQL</a> or <a href="https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835">MongoDB</a>.</li><li>Application platforms and database services with add-on search functionality, such as <a href="https://firebase.google.com/docs/firestore/solutions/search">Firebase</a> and <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a>.</li><li>Client-side JavaScript search libraries such as <a href="http://elasticlunr.com/">Elasticlunr</a>.</li><li>Client-side JavaScript libraries that synchronises data with a backend database: <a href="https://pouchdb.com/">PouchDB</a> for example.</li><li><a href="https://www.google.com/cse/">Google Custom Search Engine</a>.</li></ul><p>All the search engines, databases and managed services discussed in this article have integrations across multiple platforms, frameworks and languages — not just for the web. Whatever your target platforms, there are several key considerations when choosing a solution:</p><ul><li><strong>How often do you update your data?</strong> If updates are continual and search results absolutely must be kept up to date, you’ll probably need to use a server-side option.</li><li><strong>Do you need to search content, data, metadata — or all three?</strong> Search engines can index content and metadata from binary files such as PDF and Word documents, or access data and content from databases, or extract metadata such as EXIF from images. JavaScript search libraries need JSON.</li><li><strong>What are you searching, in what context? </strong>Log analysis, for example, is very different from consumer product search or a professional search service for a large collection of unstructured data such as an historical archive.</li><li><strong>How much data do you need to search?</strong> If you can cache all of your data locally on the client (say less than 20MB) you could consider client-side search.</li><li><strong>Would you like search to work offline?</strong> Maybe just for part of your site’s functionality, such as a store locator or customer account history? Offline search can be enabled by caching index data on the client (demo <a href="https://simpl.info/search/shakespeare">here</a>).</li></ul><p>This is just an overview of some of the issues. Pros and cons for each option are explored in more detail below.</p><h3>What is ‘full text search’?</h3><p>For a small amount of simple textual data it’s possible to provide basic search functionality via simple string matching. For example, using JavaScript you could store product data for a small online shop as an array of objects in a JSON file, then fetch the file and iterate over each object to find matches.</p><p>This simplistic approach can be better than nothing, but successful search needs more flexible functionality to find relevant results:</p><ul><li><strong>Stemming</strong>: <em>developing for mobile </em>matches results for <em>develop for mobile</em> and vice versa.</li><li><strong>Stopword handing</strong>: Search engines need to avoid irrelevant results caused by matching common words such as <em>a</em> and <em>the.</em> (Conversely, ignoring stopwords can also cause problems for queries such as <em>The Who</em> or <em>To be or not to be</em>.)</li><li><strong>Basic fuzzy matching</strong>: <em>service workers</em> matches results for <em>Service Worker.</em></li></ul><blockquote>In practice some of these features may not work as well as expected, especially in multilingual implementations or for small snippets of structured data. <a href="https://blog.algolia.com/algolia-v-elasticsearch-relevance/#multi-field">Some search engine developers</a> have made the case for preferring alternatives.</blockquote><p>High quality search implementations provide additional features on the input side:</p><ul><li><strong>Autocomplete</strong>: Typing <em>progre…</em> invokes suggestions including <em>Progressive Web App.</em></li><li><strong>Spellcheck and autosuggest: </strong>Typing <em>Porgessive Web Apps </em>suggests <em>Progressive Web Apps</em>, or does the search automatically. Spellcheck can work better than fuzzy matching, which can lead to irrelevant results. On e-commerce sites autosuggest can also be used for merchandising: to suggest products the customer may not be aware of.</li><li><strong>Synonym search:</strong> <em>PWA</em> should match results for <em>Progressive Web Apps</em>, and vice versa. This is crucial for product search, where people tend to use different words for the same thing: <em>shoes </em>or <em>footwear</em>, <em>wellies</em> or <em>wellingtons </em>or <em>gumboots</em>.</li><li><strong>Recent searches: </strong>A high proportion of searches are repeat searches: people often search for the same thing, even during the same session. You may want to provide recent-search suggestions — and potentially cache assets for recent searches, such as images or product data.</li><li><strong>Phrase matching</strong>: How much distance can be allowed between words in a search phrase? For example, should a search for <em>super moon</em> match descriptions that include the words <em>super</em> and <em>moon</em>, but not the exact phrase <em>super moon</em>?</li><li><strong>Scoped and faceted search</strong>: Provide the user with UI controls such as checkboxes and sliders to narrow the range of potential search results. For example, search only within a product type or brand, or find <em>quilted coats, for a large dog, on sale, priced less than $50.00</em>. Functionality like this combines full text search (products that match <em>quilted coats</em>) with metadata constraints (<em>less than $50</em>).</li><li><strong>Search expressions</strong>: Enable search to use AND, OR, NOT and other operators such as quote marks around exact phrases. For example: <em>grizzly -bears </em>returns matches for <em>grizzly </em>but not <em>grizzly bears.</em> <em>grizzly AND bears</em> returns only results that mention both <em>grizzly</em> and <em>bears</em>, though not necessarily as a single phrase. Search expressions are often a core requirement for search tools such as library catalogues.</li><li><strong>Advanced relevance features: </strong>Ensure results can be ranked properly, by measuring the frequency or position of the search term within documents, phrase word proximity, or with other techniques. For example, a surf shop may sell surfboard wax, but surfboard descriptions may also mention <em>board </em>and <em>wax</em>, leading to irrelevant results for <em>board wax</em>.</li><li><strong>Spatial and geospatial searches</strong>: A query for <em>London web developers</em> matches results for web developers from the King’s Cross area or postcode NC1.</li><li><strong>Aggregation, faceting and statistics</strong>: For example, count the number of matches for <em>London Web Developers</em>, or (for a customer database) calculate the median spend for customers from the UK who spent over £500 in the last year, grouped by region.</li></ul><blockquote>For a global audience, all this functionality must work across different languages, character sets, text directionality, and geographical locations — and potentially handle linguistics and cultural differences. <a href="http://www.cjk.org/cjk/joa/joapaper.htm">Japanese stemming</a> is very different from the way it’s done for English. Search engines and search services all provide different approaches to internationalisation and localisation.</blockquote><h3>What is a search index?</h3><p>It’s possible to search a small amount of data simply by scanning all of the data for every query. As the quantity of data increases, this becomes slow and inefficient.</p><p>In its simplest form a <a href="https://en.wikipedia.org/wiki/Search_engine_indexing">search indexer</a> gets around this problem by analysing a data set and building an index of search terms (words or phrases) and their location within the data — a bit like an index at the back of a book. The search implementation can then look up the query in the index rather than scanning all of the data. Indexers can also implement features such as stop-word handling and stemming.</p><p>You can view an example <a href="https://simpl.info/search/lunr/data/index.json">here</a> of a simple index built for <a href="https://simpl.info/search/lunr">this demo</a> using the <a href="https://lunrjs.com/">Lunr</a> JavaScript library:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94fc16c2ff9996a890a1d0fd939f6e22/href">https://medium.com/media/94fc16c2ff9996a890a1d0fd939f6e22/href</a></iframe><h3>What is a document?</h3><p>Confusingly, the word ‘document’ is used with two different meanings in relation to search engines:</p><ul><li>Data representing a single item, a ‘record’ or ‘object’, for example:<br>{name: ‘Fred Nerk’, job: ‘Web Developer’}</li><li>An binary file in a format such as PDF or Word.</li></ul><p>Providing high quality search results for a set of binary files can be much more complex than searching structured textual data. Imagine a video archive catalogue consisting of millions of legacy files with multiple different binary formats and a variety of content structures — how can you provide consistent and accurate search across the entire document set?</p><h3>What else do you need to think about?</h3><ul><li><strong>Data update latency</strong>: The delay from when you add, update or delete data until your changes appear in search results. This latency can be very low for search engines, potentially sub-second, even with large data sets. Low update latency is crucial for large-scale e-commerce search, or for any context where search results must reflect data updates within strict time limits specified by SLAs — for example, a news site that may be issued with takedown notices. Client-side search may not be able to provide low enough update latency or guarantee that updates have been propagated. Maintaining client-side search may also be highly inefficient, since every data or index update must be distributed to and installed by all users.</li><li><strong>Search accuracy and ranking</strong>: Successful search implementations need to trade off <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Precision">precision</a> and <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Recall">recall</a>, avoiding missing results as well as false positives. Understanding how to rank results — and what the indexer should actually ignore — is often crucial.</li><li><strong>Metadata search: </strong>You may need to search metadata as well as content, for example image EXIF data or PDF author and title fields.</li></ul><blockquote><a href="https://www.ft.com/">FT.com</a>’s recent search engine update needed expert tweaking to work well. Their <a href="https://www.maketea.co.uk/2017/12/20/building-a-new-search-for-ft-com.html#23-quality-vs-quantity">blog post</a> about the project describes how the implementation initially tended to return results that made sense but were not really relevant, such as a plethora of articles that mention <em>Trump</em> but are not ‘about’ him. They also had to ensure that page ranking preferred recent news stories rather than always returning what might otherwise seem to be ‘most relevant’.</blockquote><h3>So… What are the options?</h3><h4>Search engine</h4><p>Run your own search engine on a server. The two most popular are <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Elasticsearch</a> and <a href="http://lucene.apache.org/solr/">Solr</a>, both open source.</p><h4><strong>Pro</strong></h4><ul><li>A search engine can also serve as a data store — though there are <a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">caveats</a>.</li><li>Search engines can handle petabytes of data on hundreds of servers.</li><li>Queries are extremely fast, and updates can be searchable with very low latency.</li><li>Potentially relatively cheap and straightforward to set up and run if you already have a server team (or at least a dedicated system administrator). However, see the comments under managed search services below.</li></ul><h4><strong>Con</strong></h4><ul><li>Setup, maintenance and update cost. Large-scale, high volume installations can be complex!</li><li>Initial and ongoing infrastructure cost.</li><li>Search queries done on a server require reliable connectivity from the client, and obviously don’t work offline.</li><li>Modern search engines are extremely fast, but client connectivity bottlenecks and latency can make searches unresponsive.</li><li>Every search query requires a request to the server, which can incur significant data cost and battery usage on mobile clients. <a href="https://www.algolia.com/doc/tutorials/search-ui/instant-search/build-an-instant-search-results-page/instantsearchjs/">Instant search</a> can make this worse.</li></ul><h4>Managed search service</h4><p>Use a commercial service such as <a href="http://algolia.com/">Algolia</a>, <a href="https://aws.amazon.com/cloudsearch/pricing/">Amazon CloudSearch</a> or a platform such as <a href="https://firebase.google.com/docs/firestore/solutions/search">Firebase</a> or <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a> that integrates with third party search services (Firebase uses Algolia).</p><h4><strong>Pro</strong></h4><ul><li>It’s not simple to set up and maintain a large-scale search service able to withstand high concurrent demand and traffic spikes, along with complex, petabyte-scale data sets and a high volume of updates. It can be hard to find the right people to do the job.</li><li>DIY alternatives can incur greater infrastructure and human resource costs.</li><li>A managed service can be simpler and more reliable to scale.</li><li>Outsourcing search can simplify the management of setup, maintenance and updates — and reduce startup time.</li><li>In most cases, the performance of managed services should be at least as good as a DIY equivalent.</li><li>Managed services are potentially more reliable than self-hosted alternatives and should be able to guarantee minimum service levels.</li><li>It may be possible to include search relatively cheaply within existing cloud contracts.</li></ul><h4>Con</h4><ul><li>Potentially more expensive than DIY search engine options.</li><li>As with other server-side options: data cost, battery usage, and reliance on connectivity. <a href="https://www.algolia.com/doc/tutorials/search-ui/instant-search/build-an-instant-search-results-page/instantsearchjs/">Instant search</a> can make this worse.</li></ul><h4>Database with built-in search</h4><p>NoSQL databases including <a href="https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835">MongoDB</a> support full text search. CouchDB can implement search using <a href="https://github.com/rnewson/couchdb-lucene">couchdb-lucene</a> or in pre-built alternatives such as <a href="https://www.couchbase.com/products/full-text-search">Couchbase</a>.</p><p>Full text search is also supported by open source relational databases such as <a href="https://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html">MySQL</a> and <a href="https://www.postgresql.org/docs/9.5/static/textsearch.html">PostgreSQL</a> as well as many commercial alternatives.</p><h4>Pro</h4><ul><li>A database that supports full text search may be adequate for your needs, without the cost of setting up and maintaining a separate search engine.</li><li>There are <a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">good reasons</a> not to use a search engine as a data store. If you need to store and update a significant volume of data, it’s likely you’ll need a database. If your database enables search, that may be all you need.</li></ul><h4>Con</h4><ul><li>Built-in or add-on database search functionality may not be powerful or flexible enough to meet your requirements.</li><li>A full-scale relational or NoSQL database may be overkill for smaller data sets with simpler use cases.</li><li>Same connectivity issues other server-side options.</li></ul><h4>Google Site Search and Google Custom Search Engine</h4><p>Google Site Search <a href="https://enterprise.google.com/search/products/gss.html">is deprecated</a>, but <a href="https://cse.google.co.uk/cse/">Google Custom Search Engine</a> (CSE) is still available. The differences between the two are explained <a href="https://support.google.com/customsearch/answer/4541888?hl=en">here</a>.</p><p>You can try CSE with the example <a href="https://cse.google.com/cse/publicurl?cx=012061871733780540184:yuzohtakpbo">here</a>, which searches products from the <a href="https://github.com/Polymer/shop">Polymer Shop</a> project.</p><p>If you don’t want ads and if you’re happy to pay (or the free quota is enough) the <a href="https://developers.google.com/custom-search/json-api/v1/reference/cse/list">CSE API</a> might work for you.</p><h4>Pro</h4><ul><li>Quick and easy to set up.</li><li>Fast, reliable search.</li><li>Ad-free for non profits and educational organisations.</li></ul><h4>Con</h4><ul><li>API version costs $5.00 per 1000 queries.</li><li>Unless you use the API, and build your own, you get ads at the top of search results.</li><li>No customisation possible for autosuggest or other input features.</li><li>No control over result ranking or layout now that Google Site Search is deprecated.</li></ul><h4>Client-side search</h4><p>The Cache and Service Worker APIs enable websites to work offline and build resilience to variable connectivity. Local caching combined with client-side search can enable a number of use cases. For example:</p><ul><li>Searching an online shop while offline.</li><li>Enabling customers to use a store locator or search their purchase history.</li><li>Providing resilience and fallbacks in case of connectivity dropouts.</li></ul><p>Client-side search can be particularly compelling for a relatively small set of data that doesn’t change much. For example, the demo <a href="http://simpl.info/sh">here</a> searches Shakespeare’s plays and poems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*zgMeiUvzGPpshd1XqHKfgg.png" /></figure><p>Client-side JavaScript full text search libraries include <a href="https://lunrjs.com/">Lunr</a> or <a href="http://elasticlunr.com/">ElasticLunr</a>.</p><p>You provide a set of ‘documents’ in JSON format, such as a product list, then create an index. Here’s how to do that with the <a href="https://www.npmjs.com/package/elasticlunr">Elasticlunr Node module</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62ad5ddee507dcfe8fdb6c902a8ce187/href">https://medium.com/media/62ad5ddee507dcfe8fdb6c902a8ce187/href</a></iframe><p>To initiate search on the client, you first need to fetch the index data and load it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0428227064db2fdce349448ff61491d/href">https://medium.com/media/c0428227064db2fdce349448ff61491d/href</a></iframe><p>To enable offline search, the index file can be stored by the client using the Cache API. Alternatively, you could fetch document data and build the index on the client, then serialise and store that locally.</p><p>And finally:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29071e22c86977a077142ff053d3821f/href">https://medium.com/media/29071e22c86977a077142ff053d3821f/href</a></iframe><blockquote><a href="http://html5doctor.com/introducing-web-sql-databases/">WebSQL</a> enabled fast text matching (<a href="https://samdutton.wordpress.com/2011/04/08/instant-search-for-shakespeares-sonnets/">demo</a>) and full text search (<a href="https://gist.github.com/nolanlawson/0264938033aca2201012">demo</a>).</blockquote><blockquote>However, the WebSQL standard <a href="https://softwareengineering.stackexchange.com/questions/220254/why-is-web-sql-database-deprecated">has been discontinued</a> and only ever had partial browser support.</blockquote><blockquote>Full text search in WebSQL is now being <a href="https://www.chromestatus.com/feature/5704730961510400">removed</a>.</blockquote><h4>Pro</h4><ul><li>Relatively simple to set up and maintain.</li><li>No search engine, database or third party search service required.</li><li>Search is done on the client, so queries do not require a server round trip, so reduce server load and use less radio on mobile devices.</li><li>Client-side search is resilient to connectivity vagaries.</li><li>Client-side search can be extremely fast for smaller data sets.</li><li>Offline search capability can be enabled by caching index files using client-side storage such as the Cache API or localStorage.</li><li>Potentially useful for content that isn’t frequently updated, or where updates (though important) are not time critical. For example, an e-commerce site could use client-side search to enable offline access to store locations.</li></ul><h4>Con</h4><ul><li>Only viable for a limited amount of data — though this could be up to tens of thousands of documents, potentially more, as long as the data is relatively static and doesn’t change frequently.</li><li>High latency for data updates compared to server-side alternatives.</li><li>Potential for stale data.</li><li>Client-side search incurs memory, storage and processing cost. For example, the demo <a href="https://simpl.info/search/shakespeare/">here</a> (which searches Shakespeare’s plays and poems) uses over 200MB of memory when running in Chrome on desktop.</li><li>Currently there’s no way to share a JavaScript object, such as a search index, across multiple pages, except by serialising and storing it. (<a href="https://developer.mozilla.org/en/docs/Web/API/SharedWorker">Shared workers</a> get around this, but <a href="https://bugs.chromium.org/p/chromium/issues/detail?id=154571">aren’t implemented for Chrome for Android</a>. <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer">SharedArrayBuffer</a> might help, but has been <a href="https://www.chromium.org/Home/chromium-security/ssca">disabled by default</a> by all major browsers.) Unless you use a <a href="https://en.wikipedia.org/wiki/Single-page_application">SPA</a> architecture, you’ll need to recreate the index every time the user navigates to a new page, which is unworkable.</li><li>A server-side search service with an API can be used across multiple platforms and device types whereas client-side search may need to be implemented for both web and native, potentially with server-side fallbacks.</li><li>Offline use cases are not (yet) a priority for e-commerce or other types of sites, so why bother with client-side search?</li></ul><h4>Client-side search with automated replication</h4><p>JavaScript libraries such as <a href="https://pouchdb.com/">PouchDB</a> and <a href="https://github.com/paldepind/synceddb">SyncedDB</a> do much the same job as the client-side libraries described above, but they also offer the ability to automatically synchronise data on the client with a back-end database, optionally in both directions.</p><p>You can try an offline-enabled PouchDB demo <a href="https://simpl.info/search/pouchdb">here</a>.</p><h4>Pro</h4><ul><li>In theory ‘the best of both worlds’: search is done on the client, data is continually updated.</li><li>Enables offline search.</li><li>As with other types of client-side search, reduces the number of requests, and thereby potentially reduces server load and device radio usage.</li><li>Enables bidirectional synchronisation, potentially for multiple endpoints.</li></ul><h4>Con</h4><ul><li>Architecture is more complex than simply maintaining data as (for example) a JSON file: a database server is required.</li><li>As with other types of client-side search, processing incurs battery cost, and the potential size of the data set is limited by <a href="https://www.html5rocks.com/en/tutorials/offline/quota-research/">client-side storage quota</a>.</li><li>For a client with good connectivity, potentially slower than a fast backend database or search engine.</li></ul><h3>What about UX and UI?</h3><h4>Query input</h4><p>People have come to expect a high standard of design for search query input, particularly on shopping sites. Functionality such as synonym matching and autosuggest is now the norm.</p><p>For example, <a href="http://www.asos.com/">Asos</a> does a great job of highlighting matches and suggesting other categories and brands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3eMbUJiP44tFIkvr." /></figure><p>This site makes suggestions using product images (good!) but also adds confusing and irrelevant content:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TRdSlA7jSvCu5jM4." /></figure><p>For this shop, men’s slippers don’t show up at all in search suggestions — even though they actually sell 18 different types!:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R07tPma5_ajgAGRe." /></figure><p><a href="https://econsultancy.com/blog/66658-24-best-practice-tips-for-ecommerce-site-search">This article</a> has other great examples of high quality search, and provides sensible guidelines for search input design.</p><p>Make sure to understand the different types of search required by your users. For example, an online store needs to be flexible about the way people want to find what they’re looking for:</p><ul><li>General text search: <em>google phone</em>, <em>android mobile</em></li><li>Product name: <em>pixel</em>, <em>pixel phone</em>, <em>pixel 2</em></li><li>Product specification: <em>pixel 64GB kinda blue</em></li><li>Product number: <em>G-2PW4100</em></li><li>Category: <em>phone</em>, <em>mobile</em>, <em>smartphone</em></li></ul><h4>Search results</h4><p>Search result content and presentation is critical:</p><ul><li>Work out what people want and give it to them! If only one result is returned, it may be best to go directly to the relevant item — for example, when a customer enters a part number.</li><li>Give people as much information and functionality as possible in search results. For an online shop, that might include review ratings and style options, and the ability to make a one-click purchase. Don’t force people to go to the product page — every click loses users.</li><li>Check analytics for searches where users didn’t click on any results. Always ensure that configuration changes that work for some searches don’t mess up others. Accurate result ranking is fundamental to successful search.</li><li>Enable query term highlighting (using the mark element) and provide match context where that makes sense. This is crucial for longer documents.</li></ul><p><a href="https://www.screwfix.com/">Screwfix</a> provides checkout options right on the search results page, and automatically transforms a query (in this case <em>bosch drill</em>) into a filtered set of results, which each include review ratings and a sensible level of product detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HI8tr8uSO1XE3p_r." /></figure><p>By contrast, <a href="https://www.made.com/">Made</a> keeps results clean and uncluttered, which suits the brand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bo-esQWbuHkOIxRU." /></figure><p><a href="http://www.ft.com/">ft.com</a> orders news stories by date, and suggests sensible options for refining results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1vnsgVgX2wMbJO8." /></figure><p><a href="https://www.gettyimages.co.uk/photos/balham?autocorrect=none&amp;numberofpeople=none,group&amp;phrase=balham&amp;sort=best#license">Getty Images</a> focuses on… images! Filtering options are provided based on available metadata, along with layout and display options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8cGgfvfBxKln0NOR." /></figure><h4>Cross-functional considerations</h4><p>Search doesn’t live in a vacuum. Successful implementations need good communication between different stakeholders:</p><ul><li>Maintaining high quality search functionality requires continual cross-functional collaboration between subject experts, content providers, application developers, back-end developers, designers, marketing and sales teams. Different teams need to reconcile different priorities — and understand who has the final say!</li><li>Changes often cause problems, or unexpected knock-on effects. For example, increasing <a href="https://en.wikipedia.org/wiki/Approximate_string_matching">fuzziness</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/slop.html">slop</a> may improve some searches causing other searches to return irrelevant results.</li><li>Make it easy for non-techies to adjust features such as autocomplete and synonym search, or to add merchandising and other featured content to searches.</li></ul><h4>Searching audio and video</h4><p>If you have video or audio content, you can enable users to find it by searching metadata such as titles and descriptions.</p><p>There are two problems with this approach:</p><ul><li>Search is only for metadata, not content.</li><li>If a match is found, it’s only to the ‘top level’ of an audio or video file.</li></ul><p>If your audio or video has captions, subtitles or other kinds of ‘timed metadata’, search and navigation can be much more granular.</p><p>For example, the demo <a href="https://simpl.info/s">here</a> searches Google developer video captions, and enables navigation to specific points within videos. The content in the demo is hosted on <a href="https://support.google.com/youtube/answer/2734698?hl=en">YouTube</a>, which uses the <a href="https://en.wikipedia.org/wiki/WebVTT#Main_differences_from_.srt">SRT</a> format for captions. You can see this in action if you view the transcript for a manually captioned YouTube video such as the one <a href="https://www.youtube.com/watch?v=cmGr0RszHc8">here</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JafIBOWQHAvf6YFc." /></figure><p>Websites can use subtitles and captions with a <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/track">track element</a> (demo <a href="https://simpl.info/track/">here</a>) using the <a href="https://en.wikipedia.org/wiki/WebVTT#Main_differences_from_.srt">WebVTT</a> format, which is very similar to SRT:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/592e51fbdb80374da3edd8864516450f/href">https://medium.com/media/592e51fbdb80374da3edd8864516450f/href</a></iframe><h4>Testing</h4><p>Whatever you do, test changes and keep an eye on analytics and search logging. Build <a href="https://www.nngroup.com/articles/discount-usability-20-years/">discount usability testing</a> into your workflow.</p><p>Make it easy for non-techies to monitor and understand search statistics:</p><ul><li>What are people searching for most? What are people <em>not</em> searching for?</li><li>When and why do some queries provide no results? Can these be fixed by adding synonyms or spellchecking suggestions?</li><li>When does a search lead to a sale or other type of ‘conversion’?</li><li>Check analytics for searches that result in users exiting the site rather than clicking on a result.</li></ul><h3>Find out more</h3><h4>Installing and using search engines and managed services</h4><p>The two articles below explain in more detail how to set up and use several of the most popular search engines and managed services:</p><ul><li><a href="https://medium.com/p/b897f072de90">More about search engines</a></li><li><a href="https://medium.com/p/1e2a16b4ee9c">More about search as a service</a></li></ul><h4>Articles, books and podcasts</h4><ul><li><a href="https://econsultancy.com/blog/66658-24-best-practice-tips-for-ecommerce-site-search">24 best practice tips for ecommerce site search</a>: Sensible compilation of UI and UX tips.</li><li><a href="https://www.maketea.co.uk/2017/12/20/building-a-new-search-for-ft-com.html">Building a new search for FT.com</a>: How a successful news site rebuilt search functionality for 720,000 items of content.</li><li><a href="https://www.manning.com/books/relevant-search">Relevant Search</a>: An actual printed book! Free chapters online, and a <a href="http://opensourceconnections.com/blog/2016/07/01/relevant-search/">podcast</a>.</li><li><a href="https://nlp.stanford.edu/IR-book/information-retrieval-book.html">Introduction to Information Retrieval</a>: Nearly 10 years old, but still highly relevant and free to read online.</li><li><a href="https://www.scientificamerican.com/article/new-ai-based-search-engines-are-a-ldquo-game-changer-rdquo-for-science-research/">AI-Based Search Engines</a>: Scientific American article about free AI-based scholarly search engines.</li><li><a href="https://searchengineland.com/voice-search-changes-everything-286665">Voice Search</a><br><a href="https://www.campaignlive.co.uk/article/just-say-it-future-search-voice-personal-digital-assistants/1392459#DX23YEYTF52jRj8A.99">The future of search is voice and personal digital assistants</a><br><a href="https://www.forbes.com/sites/jaysondemers/2018/01/09/why-you-need-to-prepare-for-a-voice-search-revolution/2/#4891718ae65e">Why You Need To Prepare For A Voice Search Revolution</a><br>How audio-only interfaces are changing search and leading to different types of human–computer interactions. More stats about voice search <a href="https://www.branded3.com/blog/google-voice-search-stats-growth-trends/">here</a>. According to <a href="https://www.campaignlive.co.uk/article/just-say-it-future-search-voice-personal-digital-assistants/1392459">comScore</a>, ‘50% of all searches will be voice searches by 2020’.</li><li><a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">Why shouldn’t I use ElasticSearch as my primary datastore?</a>: Discussion of why you probably still need a database.</li><li><a href="https://alligator.io/vuejs/vue-client-side-search/">Implementing Client-Side Search with Vue.js</a>: Simple code to add text search of a JSON dataset using <a href="https://www.npmjs.com/package/fuzzaldrin-plus">fuzzaldrin-plus</a>.</li><li><a href="https://blog.algolia.com/how-algolia-tackled-the-relevance-problem-of-search-engines/#comment-2686171655">How Algolia tackled the relevance problem of search engines</a>: How the <a href="http://algolia.com/">Algolia</a> search engine ranks results, with an interesting comment thread disputing some of their claims.</li><li><a href="https://www.youtube.com/watch?v=BNHR6IQJGZs">How Search Works</a>: A video overview of Google search.</li></ul><p><em>Thank you to all the people who helped with this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f02a39545dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to add full text search to your website]]></title>
            <link>https://medium.com/dev-channel/how-to-add-full-text-search-to-your-website-4e9c80ce2bf4?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e9c80ce2bf4</guid>
            <category><![CDATA[algolia]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[solr]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 18:10:06 GMT</pubDate>
            <atom:updated>2019-02-21T12:27:50.067Z</atom:updated>
            <content:encoded><![CDATA[<p>Many types of website need efficient, accurate search.</p><p>This article explains server and client-side alternatives, and shows how to implement search that works offline.</p><h3>tl;dr</h3><p>There are lots of ways to do search:</p><ul><li>Via a back-end search engine such as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Elasticsearch</a> or <a href="http://lucene.apache.org/solr/">Solr</a>.</li><li>Commercial search services such as <a href="http://algolia.com/">Algolia</a> and <a href="https://aws.amazon.com/cloudsearch/">AWS Cloudsearch</a>.</li><li>Using a database with built-in search such as <a href="https://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html">MySQL</a> or <a href="https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835">MongoDB</a>.</li><li>Application platforms and database services with add-on search functionality, such as <a href="https://firebase.google.com/docs/firestore/solutions/search">Firebase</a> and <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a>.</li><li>Client-side JavaScript search libraries such as <a href="https://www.npmjs.com/package/flexsearch">FlexSearch</a> and <a href="http://elasticlunr.com/">Elasticlunr</a>.</li><li>Client-side JavaScript libraries that synchronises data with a backend database: <a href="https://pouchdb.com/">PouchDB</a> for example.</li><li><a href="https://www.google.com/cse/">Google Custom Search Engine</a>.</li></ul><p>All the search engines, databases and managed services discussed in this article have integrations across multiple platforms, frameworks and languages — not just for the web. Whatever your target platforms, there are several key considerations when choosing a solution:</p><ul><li><strong>How often do you update your data?</strong> If updates are continual and search results absolutely must be kept up to date, you’ll probably need to use a server-side option.</li><li><strong>Do you need to search content, data, metadata — or all three?</strong> Search engines can index content and metadata from binary files such as PDF and Word documents, or access data and content from databases, or extract metadata such as EXIF from images. JavaScript search libraries need JSON.</li><li><strong>What are you searching, in what context? </strong>Log analysis, for example, is very different from consumer product search or a professional search service for a large collection of unstructured data such as an historical archive.</li><li><strong>How much data do you need to search?</strong> If you can cache all of your data locally on the client (say less than 20MB) you could consider client-side search.</li><li><strong>Would you like search to work offline?</strong> Maybe just for part of your site’s functionality, such as a store locator or customer account history? Offline search can be enabled by caching index data on the client (demo <a href="https://simpl.info/search/shakespeare">here</a>).</li></ul><p>This is just an overview of some of the issues. Pros and cons for each option are explored in more detail below.</p><h3>What is ‘full text search’?</h3><p>For a small amount of simple textual data it’s possible to provide basic search functionality via simple string matching. For example, using JavaScript you could store product data for a small online shop as an array of objects in a JSON file, then fetch the file and iterate over each object to find matches.</p><p>This simplistic approach can be better than nothing, but successful search needs more flexible functionality to find relevant results:</p><ul><li><strong>Stemming</strong>: <em>developing for mobile </em>matches results for <em>develop for mobile</em> and vice versa.</li><li><strong>Stopword handing</strong>: Search engines need to avoid irrelevant results caused by matching common words such as <em>a</em> and <em>the.</em> (Conversely, ignoring stopwords can also cause problems for queries such as <em>The Who</em> or <em>To be or not to be</em>.)</li><li><strong>Basic fuzzy matching</strong>: <em>service workers</em> matches results for <em>Service Worker.</em></li></ul><blockquote>In practice some of these features may not work as well as expected, especially in multilingual implementations or for structured, shorter-length data. <a href="https://blog.algolia.com/algolia-v-elasticsearch-relevance/#multi-field">Some search engine developers</a> have made the case for preferring alternatives.</blockquote><p>High quality search implementations provide additional features on the input side:</p><ul><li><strong>Autocomplete</strong>: Typing <em>progre…</em> invokes suggestions including <em>Progressive Web App.</em></li><li><strong>Spellcheck and autosuggest: </strong>Typing <em>Porgessive Web Apps </em>suggests <em>Progressive Web Apps</em>, or does the search automatically. Spellcheck can work better than fuzzy matching, which can lead to irrelevant results. On e-commerce sites autosuggest can also be used for merchandising: to suggest products the customer may not be aware of.</li><li><strong>Synonym search:</strong> <em>PWA</em> should match results for <em>Progressive Web Apps</em>, and vice versa. This is crucial for product search, where people tend to use different words for the same thing: <em>shoes </em>or <em>footwear</em>, <em>wellies</em> or <em>wellingtons </em>or <em>gumboots</em>.</li><li><strong>Recent searches: </strong>A high proportion of searches are repeat searches: people often search for the same thing, even during the same session. You may want to provide recent-search suggestions — and potentially cache assets for recent searches, such as images or product data.</li><li><strong>Phrase matching</strong>: How much distance can be allowed between words in a search phrase? For example, should a search for <em>super moon</em> match descriptions that include the words <em>super</em> and <em>moon</em>, but not the exact phrase <em>super moon</em>?</li><li><strong>Scoped and faceted search</strong>: Provide the user with UI controls such as checkboxes and sliders to narrow the range of potential search results. For example, search only within a product type or brand, or find <em>quilted coats, for a large dog, on sale, priced less than $50.00</em>. Functionality like this combines full text search (products that match <em>quilted coats</em>) with metadata constraints (<em>less than $50</em>).</li><li><strong>Search expressions</strong>: Enable search to use AND, OR, NOT and other operators such as quote marks around exact phrases. For example: <em>grizzly -bears </em>returns matches for <em>grizzly </em>but not <em>grizzly bears.</em> <em>grizzly AND bears</em> returns only results that mention both <em>grizzly</em> and <em>bears</em>, though not necessarily as a single phrase. Search expressions are often a core requirement for search tools such as library catalogues.</li><li><strong>Advanced relevance features: </strong>Ensure results can be ranked properly, by measuring the frequency or position of the search term within documents, phrase word proximity, or with other techniques. For example, a surf shop may sell surfboard wax, but surfboard descriptions may also mention <em>board </em>and <em>wax</em>, leading to irrelevant results for <em>board wax</em>.</li><li><strong>Spatial and geospatial searches</strong>: A query for <em>London web developers</em> matches results for web developers from the King’s Cross area or postcode NC1.</li><li><strong>Aggregation, faceting and statistics</strong>: For example, count the number of matches for <em>London Web Developers</em>, or (for a customer database) calculate the median spend for customers from the UK who spent over £500 in the last year, grouped by region.</li></ul><blockquote>For a global audience, all this functionality must work across different languages, character sets, text directionality, and geographical locations — and potentially handle linguistics and cultural differences. <a href="http://www.cjk.org/cjk/joa/joapaper.htm">Japanese stemming</a> is very different from the way it’s done for English. Search engines and search services all provide different approaches to internationalisation and localisation.</blockquote><h3>What is a search index?</h3><p>It’s possible to search a small amount of data simply by scanning all of the data for every query. As the quantity of data increases, this becomes slow and inefficient.</p><p>In its simplest form a <a href="https://en.wikipedia.org/wiki/Search_engine_indexing">search indexer</a> gets around this problem by analysing a data set and building an index of search terms (words or phrases) and their location within the data — a bit like an index at the back of a book. The search implementation can then look up the query in the index rather than scanning all of the data. Indexers can also implement features such as stop-word handling and stemming.</p><p>You can view an example <a href="https://simpl.info/search/lunr/data/index.json">here</a> of a simple index built for <a href="https://simpl.info/search/lunr">this demo</a> using the <a href="https://lunrjs.com/">Lunr</a> JavaScript library:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94fc16c2ff9996a890a1d0fd939f6e22/href">https://medium.com/media/94fc16c2ff9996a890a1d0fd939f6e22/href</a></iframe><h3>What is a document?</h3><p>Confusingly, the word ‘document’ is used with two different meanings in relation to search engines:</p><ul><li>Data representing a single item, a ‘record’ or ‘object’, for example:<br>{name: ‘Fred Nerk’, job: ‘Web Developer’}</li><li>An binary file in a format such as PDF or Word.</li></ul><p>Providing high quality search results for a set of binary files can be much more complex than searching structured textual data. Imagine a video archive catalogue consisting of millions of legacy files with multiple different binary formats and a variety of content structures — how can you provide consistent and accurate search across the entire document set?</p><h3>What else do you need to think about?</h3><ul><li><strong>Data update latency</strong>: The delay from when you add, update or delete data until your changes appear in search results. This latency can be very low for search engines, potentially sub-second, even with large data sets. Low update latency is crucial for large-scale e-commerce search, or for any context where search results must reflect data updates within strict time limits specified by SLAs — for example, a news site that may be issued with takedown notices. Client-side search may not be able to provide low enough update latency or guarantee that updates have been propagated. Maintaining client-side search may also be highly inefficient, since every data or index update must be distributed to and installed by all users.</li><li><strong>Search accuracy and ranking</strong>: Successful search implementations need to trade off <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Precision">precision</a> and <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Recall">recall</a>, avoiding missing results as well as false positives. Understanding how to rank results — and what the indexer should actually ignore — is often crucial.</li><li><strong>Metadata search: </strong>You may need to search metadata as well as content, for example image EXIF data or PDF author and title fields.</li></ul><blockquote><a href="https://www.ft.com/">FT.com</a>’s recent search engine update needed expert tweaking to work well. Their <a href="https://www.maketea.co.uk/2017/12/20/building-a-new-search-for-ft-com.html#23-quality-vs-quantity">blog post</a> about the project describes how the implementation initially tended to return results that made sense but were not really relevant, such as a plethora of articles that mention <em>Trump</em> but are not ‘about’ him. They also had to ensure that page ranking preferred recent news stories rather than always returning what might otherwise seem to be ‘most relevant’.</blockquote><h3>So… What are the options?</h3><h4>Search engine</h4><p>Run your own search engine on a server. The two most popular are <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Elasticsearch</a> and <a href="http://lucene.apache.org/solr/">Solr</a>, both open source.</p><h4>Pro</h4><ul><li>A search engine can also serve as a data store — though there are <a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">caveats</a>.</li><li>Search engines can handle petabytes of data on hundreds of servers.</li><li>Queries are extremely fast, and updates can be searchable with very low latency.</li><li>Potentially relatively cheap and straightforward to set up and run if you already have a server team (or at least a dedicated system administrator). However, see the comments under managed search services below.</li></ul><h4>Con</h4><ul><li>Setup, maintenance and update cost. Large-scale, high volume installations can be complex!</li><li>Initial and ongoing infrastructure cost.</li><li>Search queries done on a server require reliable connectivity from the client, and obviously don’t work offline.</li><li>Modern search engines are extremely fast, but client connectivity bottlenecks and latency can make searches unresponsive.</li><li>Every search query requires a request to the server, which can incur significant data cost and battery usage on mobile clients. <a href="https://www.algolia.com/doc/tutorials/search-ui/instant-search/build-an-instant-search-results-page/instantsearchjs/">Instant search</a> can make this worse.</li></ul><h4>Managed search service</h4><p>Use a commercial service such as <a href="http://algolia.com/">Algolia</a>, <a href="https://aws.amazon.com/cloudsearch/pricing/">Amazon CloudSearch</a> or a platform such as <a href="https://firebase.google.com/docs/firestore/solutions/search">Firebase</a> or <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a> that integrates with third party search services (Firebase uses Algolia).</p><h4>Pro</h4><ul><li>It’s not simple to set up and maintain a large-scale search service able to withstand high concurrent demand and traffic spikes, along with complex, petabyte-scale data sets and a high volume of updates. It can be hard to find the right people to do the job.</li><li>DIY alternatives can incur greater infrastructure and human resource costs.</li><li>A managed service can be simpler and more reliable to scale.</li><li>Outsourcing search can simplify the management of setup, maintenance and updates — and reduce startup time.</li><li>In most cases, the performance of managed services should be at least as good as a DIY equivalent.</li><li>Managed services are potentially more reliable than self-hosted alternatives and should be able to guarantee minimum service levels.</li><li>It may be possible to include search relatively cheaply within existing cloud contracts.</li></ul><h4>Con</h4><ul><li>Potentially more expensive than DIY search engine options.</li><li>As with other server-side options: data cost, battery usage, and reliance on connectivity. <a href="https://www.algolia.com/doc/tutorials/search-ui/instant-search/build-an-instant-search-results-page/instantsearchjs/">Instant search</a> can make this worse.</li></ul><h4>Database with built-in search</h4><p>NoSQL databases including <a href="https://code.tutsplus.com/tutorials/full-text-search-in-mongodb--cms-24835">MongoDB</a> support full text search. CouchDB can implement search using <a href="https://github.com/rnewson/couchdb-lucene">couchdb-lucene</a> or in pre-built alternatives such as <a href="https://www.couchbase.com/products/full-text-search">Couchbase</a>.</p><p>Full text search is also supported by open source relational databases such as <a href="https://dev.mysql.com/doc/refman/5.7/en/fulltext-search.html">MySQL</a> and <a href="https://www.postgresql.org/docs/9.5/static/textsearch.html">PostgreSQL</a> as well as many commercial alternatives.</p><h4>Pro</h4><ul><li>A database that supports full text search may be adequate for your needs, without the cost of setting up and maintaining a separate search engine.</li><li>There are <a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">good reasons</a> not to use a search engine as a data store. If you need to store and update a significant volume of data, it’s likely you’ll need a database. If your database enables search, that may be all you need.</li></ul><h4>Con</h4><ul><li>Built-in or add-on database search functionality may not be powerful or flexible enough to meet your requirements.</li><li>A full-scale relational or NoSQL database may be overkill for smaller data sets with simpler use cases.</li><li>Same connectivity issues other server-side options.</li></ul><h4>Google Site Search and Google Custom Search Engine</h4><p>Google Site Search <a href="https://enterprise.google.com/search/products/gss.html">is deprecated</a>, but <a href="https://cse.google.co.uk/cse/">Google Custom Search Engine</a> (CSE) is still available. The differences between the two are explained <a href="https://support.google.com/customsearch/answer/4541888?hl=en">here</a>.</p><p>You can try CSE with the example <a href="https://cse.google.com/cse/publicurl?cx=012061871733780540184:yuzohtakpbo">here</a>, which searches products from the <a href="https://github.com/Polymer/shop">Polymer Shop</a> project.</p><p>If you don’t want ads and if you’re happy to pay (or the free quota is enough) the <a href="https://developers.google.com/custom-search/json-api/v1/reference/cse/list">CSE API</a> might work for you.</p><h4>Pro</h4><ul><li>Quick and easy to set up.</li><li>Fast, reliable search.</li><li>Ad-free for non profits and educational organisations.</li></ul><h4>Con</h4><ul><li>API version costs $5.00 per 1000 queries.</li><li>Unless you use the API, and build your own, you get ads at the top of search results.</li><li>No customisation possible for autosuggest or other input features.</li><li>No control over result ranking or layout now that Google Site Search is deprecated.</li></ul><h4>Client-side search</h4><p>The Cache and Service Worker APIs enable websites to work offline and build resilience to variable connectivity. Local caching combined with client-side search can enable a number of use cases. For example:</p><ul><li>Searching an online shop while offline.</li><li>Enabling customers to use a store locator or search their purchase history.</li><li>Providing resilience and fallbacks in case of connectivity dropouts.</li></ul><p>Client-side search can be particularly compelling for a relatively small set of data that doesn’t change much. For example, the demo <a href="http://simpl.info/sh">here</a> searches Shakespeare’s plays and poems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*zgMeiUvzGPpshd1XqHKfgg.png" /></figure><p>Client-side JavaScript full text search libraries include <a href="https://lunrjs.com/">Lunr</a> or <a href="http://elasticlunr.com/">ElasticLunr</a>.</p><p>You provide a set of ‘documents’ in JSON format, such as a product list, then create an index. Here’s how to do that with the <a href="https://www.npmjs.com/package/elasticlunr">Elasticlunr Node module</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/62ad5ddee507dcfe8fdb6c902a8ce187/href">https://medium.com/media/62ad5ddee507dcfe8fdb6c902a8ce187/href</a></iframe><p>To initiate search on the client, you first need to fetch the index data and load it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0428227064db2fdce349448ff61491d/href">https://medium.com/media/c0428227064db2fdce349448ff61491d/href</a></iframe><p>To enable offline search, the index file can be stored by the client using the Cache API. Alternatively, you could fetch document data and build the index on the client, then serialise and store that locally.</p><p>And finally:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29071e22c86977a077142ff053d3821f/href">https://medium.com/media/29071e22c86977a077142ff053d3821f/href</a></iframe><blockquote><a href="http://html5doctor.com/introducing-web-sql-databases/">WebSQL</a> enabled fast text matching (<a href="https://samdutton.wordpress.com/2011/04/08/instant-search-for-shakespeares-sonnets/">demo</a>) and full text search (<a href="https://gist.github.com/nolanlawson/0264938033aca2201012">demo</a>).</blockquote><blockquote>However, the WebSQL standard <a href="https://softwareengineering.stackexchange.com/questions/220254/why-is-web-sql-database-deprecated">has been discontinued</a> and only ever had partial browser support.</blockquote><blockquote>Full text search in WebSQL is now being <a href="https://www.chromestatus.com/feature/5704730961510400">removed</a>.</blockquote><h4>Pro</h4><ul><li>Relatively simple to set up and maintain.</li><li>No search engine, database or third party search service required.</li><li>Search is done on the client, so queries do not require a server round trip, so reduce server load and use less radio on mobile devices.</li><li>Client-side search is resilient to connectivity vagaries.</li><li>Client-side search can be extremely fast for smaller data sets.</li><li>Offline search capability can be enabled by caching index files using client-side storage such as the Cache API or localStorage.</li><li>Potentially useful for content that isn’t frequently updated, or where updates (though important) are not time critical. For example, an e-commerce site could use client-side search to enable offline access to store locations.</li></ul><h4>Con</h4><ul><li>Only viable for a limited amount of data — though this could be up to tens of thousands of documents, potentially more, as long as the data is relatively static and doesn’t change frequently.</li><li>High latency for data updates compared to server-side alternatives.</li><li>Potential for stale data.</li><li>Client-side search incurs memory, storage and processing cost. For example, the demo <a href="https://simpl.info/search/shakespeare/">here</a> (which searches Shakespeare’s plays and poems) uses over 200MB of memory when running in Chrome on desktop.</li><li>Currently there’s no way to share a JavaScript object, such as a search index, across multiple pages, except by serialising and storing it. (<a href="https://developer.mozilla.org/en/docs/Web/API/SharedWorker">Shared workers</a> get around this, but <a href="https://bugs.chromium.org/p/chromium/issues/detail?id=154571">aren’t implemented for Chrome for Android</a>. <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer">SharedArrayBuffer</a> might help, but has been <a href="https://www.chromium.org/Home/chromium-security/ssca">disabled by default</a> by all major browsers.) Unless you use a <a href="https://en.wikipedia.org/wiki/Single-page_application">SPA</a> architecture, you’ll need to recreate the index every time the user navigates to a new page, which is unworkable.</li><li>A server-side search service with an API can be used across multiple platforms and device types whereas client-side search may need to be implemented for both web and native, potentially with server-side fallbacks.</li><li>Offline use cases are not (yet) a priority for e-commerce or other types of sites, so why bother with client-side search?</li></ul><h4>Client-side search with automated replication</h4><p>JavaScript libraries such as <a href="https://pouchdb.com/">PouchDB</a> and <a href="https://github.com/paldepind/synceddb">SyncedDB</a> do much the same job as the client-side libraries described above, but they also offer the ability to automatically synchronise data on the client with a back-end database, optionally in both directions.</p><p>You can try an offline-enabled PouchDB demo <a href="https://simpl.info/search/pouchdb">here</a>.</p><h4>Pro</h4><ul><li>In theory ‘the best of both worlds’: search is done on the client, data is continually updated.</li><li>Enables offline search.</li><li>As with other types of client-side search, reduces the number of requests, and thereby potentially reduces server load and device radio usage.</li><li>Enables bidirectional synchronisation, potentially for multiple endpoints.</li></ul><h4>Con</h4><ul><li>Architecture is more complex than simply maintaining data as (for example) a JSON file: a database server is required.</li><li>As with other types of client-side search, processing incurs battery cost, and the potential size of the data set is limited by <a href="https://www.html5rocks.com/en/tutorials/offline/quota-research/">client-side storage quota</a>.</li><li>For a client with good connectivity, potentially slower than a fast backend database or search engine.</li></ul><h3>What about UX and UI?</h3><h4>Query input</h4><p>People have come to expect a high standard of design for search query input, particularly on shopping sites. Functionality such as synonym matching and autosuggest is now the norm.</p><p>For example, <a href="http://www.asos.com/">Asos</a> does a great job of highlighting matches and suggesting other categories and brands:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3eMbUJiP44tFIkvr." /></figure><p><a href="https://econsultancy.com/blog/66658-24-best-practice-tips-for-ecommerce-site-search">This article</a> has other great examples of high quality search, and provides sensible guidelines for search input design.</p><p>Make sure to understand the different types of search required by your users. For example, an online store needs to be flexible about the way people want to find what they’re looking for:</p><ul><li>General text search: <em>google phone</em>, <em>android mobile</em></li><li>Product name: <em>pixel</em>, <em>pixel phone</em>, <em>pixel 2</em></li><li>Product specification: <em>pixel 64GB kinda blue</em></li><li>Product number: <em>G-2PW4100</em></li><li>Category: <em>phone</em>, <em>mobile</em>, <em>smartphone</em></li></ul><h4>Search results</h4><p>Search result content and presentation is critical:</p><ul><li>Work out what people want and give it to them! If only one result is returned, it may be best to go directly to the relevant item — for example, when a customer enters a part number.</li><li>Give people as much information and functionality as possible in search results. For an online shop, that might include review ratings and style options, and the ability to make a one-click purchase. Don’t force people to go to the product page — every click loses users.</li><li>Check analytics for searches where users didn’t click on any results. Always ensure that configuration changes that work for some searches don’t mess up others. Accurate result ranking is fundamental to successful search.</li><li>Enable query term highlighting (using the mark element) and provide match context where that makes sense. This is crucial for longer documents.</li></ul><p><a href="https://www.screwfix.com/">Screwfix</a> provides checkout options right on the search results page, and automatically transforms a query (in this case <em>bosch drill</em>) into a filtered set of results, which each include review ratings and a sensible level of product detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HI8tr8uSO1XE3p_r." /></figure><p>By contrast, <a href="https://www.made.com/">Made</a> keeps results clean and uncluttered, which suits the brand:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bo-esQWbuHkOIxRU." /></figure><p><a href="http://www.ft.com/">ft.com</a> orders news stories by date, and suggests sensible options for refining results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1vnsgVgX2wMbJO8." /></figure><p><a href="https://www.gettyimages.co.uk/photos/balham?autocorrect=none&amp;numberofpeople=none,group&amp;phrase=balham&amp;sort=best#license">Getty Images</a> focuses on… images! Filtering options are provided based on available metadata, along with layout and display options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8cGgfvfBxKln0NOR." /></figure><h4>Cross-functional considerations</h4><p>Search doesn’t live in a vacuum. Successful implementations need good communication between different stakeholders:</p><ul><li>Maintaining high quality search functionality requires continual cross-functional collaboration between subject experts, content providers, application developers, back-end developers, designers, marketing and sales teams. Different teams need to reconcile different priorities — and understand who has the final say!</li><li>Changes often cause problems, or unexpected knock-on effects. For example, increasing <a href="https://en.wikipedia.org/wiki/Approximate_string_matching">fuzziness</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/slop.html">slop</a> may improve some searches causing other searches to return irrelevant results.</li><li>Make it easy for non-techies to adjust features such as autocomplete and synonym search, or to add merchandising and other featured content to searches.</li></ul><h4>Searching audio and video</h4><p>If you have video or audio content, you can enable users to find it by searching metadata such as titles and descriptions.</p><p>There are two problems with this approach:</p><ul><li>Search is only for metadata, not content.</li><li>If a match is found, it’s only to the ‘top level’ of an audio or video file.</li></ul><p>If your audio or video has captions, subtitles or other kinds of ‘timed metadata’, search and navigation can be much more granular.</p><p>For example, the demo <a href="https://simpl.info/s">here</a> searches Google developer video captions, and enables navigation to specific points within videos. The content in the demo is hosted on <a href="https://support.google.com/youtube/answer/2734698?hl=en">YouTube</a>, which uses the <a href="https://en.wikipedia.org/wiki/WebVTT#Main_differences_from_.srt">SRT</a> format for captions. You can see this in action if you view the transcript for a manually captioned YouTube video such as the one <a href="https://www.youtube.com/watch?v=cmGr0RszHc8">here</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JafIBOWQHAvf6YFc." /></figure><p>Websites can use subtitles and captions with a <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/track">track element</a> (demo <a href="https://simpl.info/track/">here</a>) using the <a href="https://en.wikipedia.org/wiki/WebVTT#Main_differences_from_.srt">WebVTT</a> format, which is very similar to SRT:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/592e51fbdb80374da3edd8864516450f/href">https://medium.com/media/592e51fbdb80374da3edd8864516450f/href</a></iframe><h4>Testing</h4><p>Whatever you do, test changes and keep an eye on analytics and search logging. Build <a href="https://www.nngroup.com/articles/discount-usability-20-years/">discount usability testing</a> into your workflow.</p><p>Make it easy for non-techies to monitor and understand search statistics:</p><ul><li>What are people searching for most? What are people <em>not</em> searching for?</li><li>When and why do some queries provide no results? Can these be fixed by adding synonyms or spellchecking suggestions?</li><li>When does a search lead to a sale or other type of ‘conversion’?</li><li>Check analytics for searches that result in users exiting the site rather than clicking on a result.</li></ul><h3>Find out more</h3><h4>Installing and using search engines and managed services</h4><p>The two articles below explain in more detail how to set up and use several of the most popular search engines and managed services:</p><ul><li><a href="https://medium.com/@samdutton/more-about-search-engines-b897f072de90">More about search engines</a></li><li><a href="https://medium.com/@samdutton/more-about-search-as-a-service-1e2a16b4ee9c">More about search as a service</a></li></ul><h4>Articles, books and podcasts</h4><ul><li><a href="https://econsultancy.com/blog/66658-24-best-practice-tips-for-ecommerce-site-search">24 best practice tips for ecommerce site search</a>: Sensible compilation of UI and UX tips.</li><li><a href="https://www.maketea.co.uk/2017/12/20/building-a-new-search-for-ft-com.html">Building a new search for FT.com</a>: How a successful news site rebuilt search functionality for 720,000 items of content.</li><li><a href="https://www.manning.com/books/relevant-search">Relevant Search</a>: An actual printed book! Free chapters online, and a <a href="http://opensourceconnections.com/blog/2016/07/01/relevant-search/">podcast</a>.</li><li><a href="https://nlp.stanford.edu/IR-book/information-retrieval-book.html">Introduction to Information Retrieval</a>: Nearly 10 years old, but still highly relevant and free to read online.</li><li><a href="https://www.scientificamerican.com/article/new-ai-based-search-engines-are-a-ldquo-game-changer-rdquo-for-science-research/">AI-Based Search Engines</a>: Scientific American article about free AI-based scholarly search engines.</li><li><a href="https://searchengineland.com/voice-search-changes-everything-286665">Voice Search</a><br><a href="https://www.campaignlive.co.uk/article/just-say-it-future-search-voice-personal-digital-assistants/1392459#DX23YEYTF52jRj8A.99">The future of search is voice and personal digital assistants</a><br><a href="https://www.forbes.com/sites/jaysondemers/2018/01/09/why-you-need-to-prepare-for-a-voice-search-revolution/2/#4891718ae65e">Why You Need To Prepare For A Voice Search Revolution</a><br>How audio-only interfaces are changing search and leading to different types of human–computer interactions. More stats about voice search <a href="https://www.branded3.com/blog/google-voice-search-stats-growth-trends/">here</a>. According to <a href="https://www.campaignlive.co.uk/article/just-say-it-future-search-voice-personal-digital-assistants/1392459">comScore</a>, ‘50% of all searches will be voice searches by 2020’.</li><li><a href="https://www.quora.com/Why-shouldnt-I-use-ElasticSearch-as-my-primary-datastore">Why shouldn’t I use ElasticSearch as my primary datastore?</a>: Discussion of why you probably still need a database.</li><li><a href="https://alligator.io/vuejs/vue-client-side-search/">Implementing Client-Side Search with Vue.js</a>: Simple code to add text search of a JSON dataset using <a href="https://www.npmjs.com/package/fuzzaldrin-plus">fuzzaldrin-plus</a>.</li><li><a href="https://blog.algolia.com/how-algolia-tackled-the-relevance-problem-of-search-engines/#comment-2686171655">How Algolia tackled the relevance problem of search engines</a>: How the <a href="http://algolia.com/">Algolia</a> search engine ranks results, with an interesting comment thread disputing some of their claims.</li><li><a href="https://www.youtube.com/watch?v=BNHR6IQJGZs">How Search Works</a>: A video overview of Google search.</li></ul><p><em>Thank you to all the people who helped with this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e9c80ce2bf4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dev-channel/how-to-add-full-text-search-to-your-website-4e9c80ce2bf4">How to add full text search to your website</a> was originally published in <a href="https://medium.com/dev-channel">Dev Channel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[More about search as a service]]></title>
            <link>https://medium.com/@samdutton/more-about-search-as-a-service-1e2a16b4ee9c?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e2a16b4ee9c</guid>
            <category><![CDATA[algolia]]></category>
            <category><![CDATA[search]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[search-engines]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 18:08:03 GMT</pubDate>
            <atom:updated>2018-02-09T18:08:03.722Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article is a companion to <a href="https://medium.com/p/3f02a39545dd/edit"><strong>How to add full text search to your website</strong></a>.</blockquote><blockquote>It’s an overview from the perspective of a front end developer — and is in no way comprehensive!</blockquote><p>If you’re running a busy site with limited server resources (or not enough sys admins) you might want to use a managed <a href="https://en.wikipedia.org/wiki/Elasticsearch#Managed_services">search engine service</a> rather than install and maintain your own.</p><p>Many application platforms and database services, such as <a href="https://firebase.google.com/docs/firestore/solutions/search">Firebase</a> and <a href="https://www.ibm.com/cloud/cloudant">Cloudant</a>, also provide add-on search functionality.</p><p><a href="http://algolia.com/">Algolia</a> and <a href="https://aws.amazon.com/cloudsearch/pricing/">Amazon CloudSearch</a> are described below.</p><h3>Algolia</h3><p>Algolia is a closed-source search engine available as a managed service, implemented in C++, running as an nginx module. Algolia can also be used to <a href="https://www.algolia.com/doc/tutorials/indexing/3rd-party-service/firebase-algolia/">add search capability to Firebase</a> and other databases, CMS and e-commerce platforms.</p><p>Algolia is currently available as a free trial (without requiring a credit card!) The getting-started guides, API documentation and tutorials are excellent.</p><p><a href="https://www.algolia.com/doc/faq/why/what-makes-algolia-different-than-elasticsearch-or-solr/">The company claims</a> their engine makes successful ranking easier to configure, and returns results much faster than Lucene-based search engines. (<a href="http://opensourceconnections.com/blog/2016/06/01/thoughts-on-algolia/">One commentator</a> suggests an overall 10–20x improvement.) In terms of general functionality, Algolia provides features similar to other engines, such as spellcheck and result highlighting. Algolia <a href="https://blog.algolia.com/algolia-v-elasticsearch-relevance/#multi-field">point out</a> that their engine ‘is designed to index structured data, not large blocks of text’, and that other search engines are better suited to unstructured data.</p><p>API clients are provided for JavaScript and other languages and platforms, including Android and iOS, along with UI/UX components and <a href="https://community.algolia.com/instantsearch.js/v2/widgets.html">ready-made widgets</a>. Other types of integration are provided for a number of frameworks and libraries, as well as CMS, e-commerce and database platforms.</p><p>As with other search engines, either a REST API or a GUI dashboard can be used to upload data and search.</p><p>Data records (described as ‘objects’) are uploaded in JSON format, without requiring any extra boilerplate. Algolia give the following example:</p><pre>[<br>  {<br>    &quot;name&quot;: &quot;Monica Bellucci&quot;,<br>    &quot;alternative_name&quot;: &quot;Monica Anna Maria Bellucci&quot;,<br>    &quot;rating&quot;: 3956,<br>    &quot;image_path&quot;: &quot;/z3sLuRKP7hQVr.jpg&quot;<br>  },<br>  {<br>    &quot;name&quot;: &quot;Sean Connery&quot;,<br>    &quot;alternative_name&quot;: &quot;Sir Sean Connery&quot;,<br>    &quot;rating&quot;: 746,<br>    &quot;image_path&quot;: &quot;/ce84udJZ9QRSR44jxwK2apM3DM8.jpg&quot;<br>  },<br>  {<br>    &quot;name&quot;: &quot;Will Smith&quot;,<br>    &quot;alternative_name&quot;: null,<br>    &quot;rating&quot;: 492,<br>    &quot;image_path&quot;: &quot;/2iYXDlCvLyVO49louRyDDXagZ0G.jpg&quot;<br>  },<br>  {…}<br>]</pre><p>Search results are returned in the same format, but with an objectID value:</p><pre>{<br>  &quot;name&quot;: &quot;Monica Bellucci&quot;,<br>  &quot;alternative_name&quot;: &quot;Monica Anna Maria Bellucci&quot;,<br>  &quot;rating&quot;: 3956,<br>  &quot;image_path&quot;: &quot;/z3sLuRKP7hQVrvSTsqdLjGSldwG.jpg&quot;,<br>  &quot;objectID&quot;: &quot;5&quot;<br>}</pre><p>‘Searchable attributes’ can be defined and prioritised via the dashboard or REST API. For example, it probably makes sense in this example to prioritise <em>name</em> over <em>alternative_name</em> and not to search <em>image_path</em>. It’s also possible to specify fields to influence ranking — in this example, the <em>rating</em> field can be used to prioritise results (rather than, for example, alphabetical name order).</p><p>If required, Algolia can return results in HTML format rather than raw JSON, using a variety of templates. For example, a ‘detailed’ template response could look like this:</p><pre>&lt;li class=”results”&gt;<br>  &lt;img src=”<a href="https://image.tmdb.org/t/p/w154/%7B%7B">https://image.tmdb.org/t/p/w154/{{</a> hit.image_path }}” /&gt;<br>  &lt;h3&gt;{{{ hit._highlightResult.name.value }}}&lt;/h3&gt;<br>  {{ hit.alternative_name }}<br>&lt;/li&gt;</pre><p>There is a <a href="https://www.algolia.com/doc/tutorials/getting-started/quick-start-with-the-api-client/javascript/#before-starting">well-written tutorial</a> explaining how to use the Algolia APIs with JavaScript and other languages and platforms.</p><p>You can run the JavaScript port from Node:</p><pre>npm install algoliasearch — save</pre><p>Or from a client app:</p><pre>&lt;script src=”<a href="https://cdn.jsdelivr.net/algoliasearch/3/">https://cdn.jsdelivr.net/algoliasearch/3/</a><br> algoliasearch.min.js”&gt;&lt;/script&gt;</pre><p>A search-only client is also available, which weighs in at around 50KB:</p><pre>&lt;script src=”<a href="https://cdn.jsdelivr.net/algoliasearch/3/">https://cdn.jsdelivr.net/algoliasearch/3/</a><br> algoliasearchLite.min.js”&gt;&lt;/script&gt;</pre><p>Initialize the client:</p><pre>var client = algoliasearch(‘YourApplicationID’, ‘YourAPIKey’);<br>var index = client.initIndex(‘your_index_name’);</pre><p>Push data (sadly no Promises yet):</p><pre>var index = client.initIndex(‘contacts’);<br>var contactsJSON = require(‘./contacts.json’);<br>index.addObjects(contactsJSON, function(err, content) {<br>  if (err) {<br>    console.error(err);<br>  }<br>});</pre><p>Make configuration changes:</p><pre>index.setSettings({<br>  &#39;customRanking&#39;: [&#39;desc(followers)&#39;]<br>}, function(err, content) {<br>  console.log(content);<br>});</pre><pre>index.setSettings({<br>  &#39;searchableAttributes&#39;: [<br>    &#39;lastname&#39;,<br>    &#39;firstname&#39;,<br>    &#39;company&#39;,<br>    &#39;email&#39;,<br>    &#39;city&#39;,<br>    &#39;address&#39;<br>  ]<br>}, function(err, content) {<br>  console.log(content);<br>});</pre><p>And search!</p><pre>index.search(‘jimmie’, function(err, content) {<br>  console.log(content.hits);<br>});</pre><p><a href="https://www.algolia.com/doc/guides/search-ui/search-libraries/">InstantSearch.js</a> provides a library of UI widgets including components to provide a search bar, display results, sort, select filters and do pagination. React, Vue and other frameworks and libraries are also supported.</p><h3>Find out more</h3><ul><li><a href="https://blog.algolia.com/algolia-v-elasticsearch-latency/">Comparing Algolia and Elasticsearch for Consumer-Grade Search</a>: comprehensive comparison and details about Algolia support for different client platforms.</li><li><a href="http://opensourceconnections.com/blog/2016/06/01/thoughts-on-algolia/">Thoughts on Algolia (vs Solr &amp; Elasticsearch)</a>: from a self-confessed ‘pretty rabid Lucenist’.</li></ul><h3>Amazon CloudSearch</h3><p>CloudSearch is based on Solr; <a href="https://aws.amazon.com/elasticsearch-service/">Amazon Elasticsearch</a> uses Elasticsearch. It has a free tier, but you will need to sign up to Amazon Web Services with a credit card.</p><h4>Create and configure a search domain</h4><p>An Amazon <em>search domain</em> is a collection of related documents: for example, customer data or a product catalog. This corresponds to an <em>index</em> in Elasticsearch or a <em>collection</em> in Solr. Each domain has a unique URL endpoint.</p><p>The CloudSearch <a href="http://docs.aws.amazon.com/cloudsearch/latest/developerguide/getting-started-create-domain.html">Developer Guide</a> provides default configuration settings, along with sample data (5000 listings from <a href="http://www.imdb.com/">IMDB</a>). Setting up a search domain is a four step process:</p><ol><li>Configure indexing options and access-policies using an online wizard GUI.</li><li>Initialize resources for a domain — this takes around ten minutes.</li><li>Upload documents.</li><li>Test search from the Amazon CloudSearch console.</li></ol><p>You can then add <a href="https://docs.aws.amazon.com/cloudsearch/latest/developerguide/getting-suggestions.html">Suggesters</a> for tasks such as auto-completion and fuzzy matching.</p><p>Making changes to the index (such as adding a Suggester) can take several minutes, but users can continue to make searches using the old configuration while changes are being processed.</p><h4>Send search requests to your domain</h4><p>Each search domain corresponds to an endpoint:</p><p>search-foo-uvlaoh4rkf7gxj6maogmqmce2i.eu-west-1.cloudsearch.amazonaws.com</p><p>Searches are done via HTTP request to the endpoint, adding an API version and query string. By default results are returned as JSON:</p><pre>search-foo-uvlaoh4rkf7gxj6maogmqmce2i.eu-west-1.cloudsearch.amazonaws.com<br>  /2013-01-01/search?q=foo</pre><p>Amazon provides a structured query syntax for more complex requests: for example, to use sorting and numeric ranges:</p><pre>/search?q=(and genres:&#39;Sci-Fi&#39; year:{,2000])&amp;q.parser=structured&amp;<br>  return=title,year&amp;sort=title asc</pre><h4>Upload data</h4><p>You create <em>document batch files</em>, with a maximum size of 5MB, then send them to your endpoint. As described in the CloudSearch <a href="http://docs.aws.amazon.com/cloudsearch/latest/developerguide/preparing-data.html#creating-document-batches">documentation</a>, each batch file looks like the following example, which adds one document and deletes another:</p><pre>[<br> {&quot;type&quot;: &quot;add&quot;,<br>  &quot;id&quot;:   &quot;tt0484562&quot;,<br>  &quot;fields&quot;: {<br>    &quot;title&quot;: &quot;The Seeker: The Dark Is Rising&quot;,<br>    &quot;directors&quot;: &quot;Cunningham, David L.&quot;,<br>    &quot;genres&quot;: [&quot;Adventure&quot;,&quot;Drama&quot;,&quot;Fantasy&quot;,&quot;Thriller&quot;],<br>    &quot;actors&quot;: [&quot;McShane, Ian&quot;,&quot;Eccleston, Christopher&quot; ...]<br>  }<br> },<br> {&quot;type&quot;: &quot;delete&quot;,<br>  &quot;id&quot;:   &quot;tt0484575&quot;<br> }<br>]</pre><p>Note the constraints here:</p><ul><li>Batch files can be no more than 5MB in size.</li><li>You can’t just upload ‘vanilla’ JSON data: you need to follow the batch file format.</li></ul><p>You can submit a document batch via the Amazon CloudSearch console, by making a POST request to the endpoint, or by using the AWS command line interface (CLI).</p><h4>Use AWS from the command line</h4><p>First you need to install the AWS CLI — the easiest way to do that is to follow the Bundled Install <a href="http://docs.aws.amazon.com/cli/latest/userguide/awscli-install-bundle.html">instructions</a>.</p><p>You can then upload document batches using the aws tool:</p><pre>aws cloudsearchdomain --endpoint-url <br>  http://doc-movies-y6gelr4lv3jeu4rvoelunxsl2e.<br>  us-east-1.cloudsearch.amazonaws.com<br>  upload-documents --content-type application/json <br>  --documents movie-data-2013.json</pre><p>Alternatively, you can simply post batches to documents/batch:</p><pre>curl -X POST --upload-file movie-data-2013.json doc-movies-123456789012.us-east-1.cloudsearch.amazonaws.com/2013-01-01/documents/batch --header &quot;Content-Type:application/json&quot;</pre><h4>Search</h4><p>You can try out a test search <a href="http://search-samdutton-uvlaoh4rkf7gxj6maogmqmce2i.eu-west-1.cloudsearch.amazonaws.com/2013-01-01/search?q=dark&amp;return=_all_fields%2C_score&amp;highlight.actors=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&amp;highlight.directors=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&amp;highlight.plot=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&amp;highlight.title=%7B%22max_phrases%22%3A3%2C%22format%22%3A%22text%22%2C%22pre_tag%22%3A%22*%23*%22%2C%22post_tag%22%3A%22*%25*%22%7D&amp;sort=_score+desc">here</a>.</p><p>Search results look like this — note that query term highlighting is available by default:</p><pre>{<br>  &quot;status&quot;: {<br>    &quot;rid&quot;: &quot;+9HBsIws+sYBClMmzQ==&quot;,<br>    &quot;time-ms&quot;: 36<br>  },<br>  &quot;hits&quot;: {<br>    &quot;found&quot;: 90,<br>    &quot;start&quot;: 0,<br>    &quot;hit&quot;: [{<br>      &quot;id&quot;: &quot;tt0369226&quot;,<br>      &quot;fields&quot;: {<br>        &quot;rating&quot;: &quot;2.3&quot;,<br>        &quot;genres&quot;: [&quot;Action&quot;, &quot;Horror&quot;],<br>        &quot;plot&quot;: &quot;Based on the video game, Alone in the Dark focuses on Edward Carnby, a detective of the paranormal, who slowly unravels a mysterious events with deadly results.&quot;,<br>        &quot;release_date&quot;: &quot;2005-01-28T00:00:00Z&quot;,<br>        &quot;title&quot;: &quot;Alone in the Dark&quot;,<br>        &quot;rank&quot;: &quot;4404&quot;,<br>        &quot;running_time_secs&quot;: &quot;5760&quot;,<br>        &quot;directors&quot;: [&quot;Uwe Boll&quot;],<br>        &quot;image_url&quot;: &quot;http://ia.media-imdb.com/images/M/MV5BMTIxNDI5ODY2MF5BMl5BanBnXkFtZTcwNzQ1NzcyMQ@@._V1_SX400_.jpg&quot;,<br>        &quot;year&quot;: &quot;2005&quot;,<br>        &quot;actors&quot;: [&quot;Christian Slater&quot;, &quot;Tara Reid&quot;, &quot;Stephen Dorff&quot;],<br>        &quot;_score&quot;: &quot;8.61123&quot;<br>      },<br>      &quot;highlights&quot;: {<br>        &quot;actors&quot;: &quot;Christian Slater Tara Reid Stephen Dorff&quot;,<br>        &quot;directors&quot;: &quot;Uwe Boll&quot;,<br>        &quot;plot&quot;: &quot;Based on the video game, Alone in the *#*Dark*%* focuses on Edward Carnby, a detective of the paranormal, who slowly unravels a mysterious events with deadly results.&quot;,<br>        &quot;title&quot;: &quot;Alone in the *#*Dark*%*&quot;<br>      }<br>    }<br>    ...<br>    ]<br>  }<br>}</pre><h3>Find out more</h3><ul><li><a href="https://docs.aws.amazon.com/cloudsearch/latest/developerguide/getting-started-create-domain.html">Developer Guide</a></li><li><a href="https://optimalbi.com/blog/2016/02/16/aws-elasticsearch-vs-aws-cloudsearch/">AWS Elasticsearch vs AWS CloudSearch</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e2a16b4ee9c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[More about search engines]]></title>
            <link>https://medium.com/@samdutton/more-about-search-engines-b897f072de90?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/b897f072de90</guid>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[apache-solr]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Fri, 09 Feb 2018 18:06:47 GMT</pubDate>
            <atom:updated>2018-02-09T18:11:00.549Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>This article is a companion to <a href="https://medium.com/dev-channel/how-to-add-full-text-search-to-your-website-4e9c80ce2bf4"><strong>How to add full text search to your website</strong></a>.</blockquote><blockquote>It’s an overview from the perspective of a front end developer — and is in no way comprehensive!</blockquote><p>The two most popular open source search engines are <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Elasticsearch</a> and <a href="http://lucene.apache.org/solr/">Solr</a>. Both are based on the <a href="https://en.wikipedia.org/wiki/Apache_Lucene">Apache Lucene</a> Java library.</p><p>In reality, large-scale, high-volume search can be difficult to implement and maintain, but potentially search engines can cope with petabytes of data on hundreds of servers, providing extremely fast search, and low latency when data is updated. You can find out more about the differences between the two <a href="https://stackoverflow.com/questions/10213009/solr-vs-elasticsearch">here</a>.</p><blockquote>If you’re a front-end developer you may feel somewhat daunted at the prospect of installing a search engine. However, Elasticsearch and Solr are well documented and relatively easy to get up and running.</blockquote><h3>Installation and usage</h3><p>The following instructions describe the installation for the Java versions of Elasticsearch and Solr.</p><p>Elasticsearch is also available as a <a href="https://github.com/elastic/elasticsearch-js">Node module</a> with similar functionality and a promise-based API. Likewise, Solr can be used via Node modules such as <a href="https://www.npmjs.com/package/solr-client">solr-client</a>.</p><p>Elasticsearch and Solr use REST interfaces to perform search queries and also for administration and index management tasks:</p><ul><li>CRUD operations on index data: create, read, update and delete.</li><li>Health checks and statistics.</li><li>Configuration of search engine settings and indexes.</li><li>Advanced features such as pagination, sorting and filtering.</li></ul><p>Both engines also provide GUI apps for administration and query testing.</p><h3>Elasticsearch</h3><h4><strong>Install the search engine</strong></h4><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html#_installation_with_homebrew">Instructions</a></li><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html">Basic concepts</a></li></ul><p>On Mac OS X you can install Elasticsearch using Homebrew:</p><p>brew install elasticsearch</p><h4><strong>Test the installation</strong></h4><p>curl -XGET ‘localhost:9200/_cat/health?v&amp;pretty’</p><h4><strong>Create an index</strong></h4><p>curl -XPUT &#39;localhost:9200/customer?pretty&#39;</p><p>An <em>index</em> in Elasticsearch is a collection of related documents: for example, customer data or a product catalog. This corresponds to a <em>collection</em> in Solr.</p><h4><strong>Delete an index</strong></h4><p>curl -XDELETE &#39;localhost:9200/customer?pretty&#39;</p><h4><strong>Add a document</strong>:</h4><pre>curl -XPUT &#39;localhost:9200/customer/doc/1?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;name&quot;: &quot;John Doe&quot;<br>}<br>&#39;</pre><h4><strong>Retrieve a document</strong></h4><pre>curl -XGET &#39;localhost:9200/customer/doc/1?pretty&#39;</pre><p>The response looks like this:</p><pre>{<br>  &quot;_index&quot; : &quot;customer&quot;,<br>  &quot;_type&quot; : &quot;doc&quot;,<br>  &quot;_id&quot; : &quot;1&quot;,<br>  &quot;_version&quot; : 1,<br>  &quot;found&quot; : true,<br>  &quot;_source&quot; : { &quot;name&quot;: &quot;John Doe&quot; }<br>}</pre><h4><strong>Replace a document</strong></h4><pre>curl -XPUT &#39;localhost:9200/customer/doc/1?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;name&quot;: &quot;Jane Doe&quot;<br>}<br>&#39;</pre><h4><strong>Update a document</strong></h4><pre>curl -XPOST &#39;localhost:9200/customer/doc/1/_update?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;doc&quot;: { &quot;name&quot;: &quot;Jane Doe&quot;, &quot;age&quot;: 20 }<br>}<br>&#39;</pre><p>You can also use scripts to programmatically increment or otherwise change specific fields.</p><h4><strong>Delete a document</strong></h4><pre>curl -XDELETE &#39;localhost:9200/customer/doc/2?pretty&#39;</pre><p>There’s also an API for deleting a set of documents matching specific queries.</p><p>Bulk operations are done like this:</p><pre>curl -XPOST &#39;localhost:9200/customer/doc/_bulk?pretty&#39; -H<br>  &#39;Content-Type: application/json&#39; -d&#39;<br>  {&quot;index&quot;:{&quot;_id&quot;:&quot;1&quot;}}<br>  {&quot;name&quot;: &quot;John Doe&quot; }<br>  {&quot;index&quot;:{&quot;_id&quot;:&quot;2&quot;}}<br>  {&quot;name&quot;: &quot;Jane Doe&quot; }<br> &#39;</pre><h4><strong>Search for documents</strong></h4><p>This example matches any document, and sorts results by name in ascending order:</p><pre>curl <br>  -XGET &#39;localhost:9200/customer/_search?q=*&amp;sort=name:asc&amp;pretty&#39;</pre><p>The response includes information about the number of hits and time for the search, as well as the actual results:</p><pre>{<br>  &quot;took&quot; : 1,<br>  &quot;timed_out&quot; : false,<br>  &quot;_shards&quot; : {<br>    &quot;total&quot; : 5,<br>    &quot;successful&quot; : 5,<br>    &quot;skipped&quot; : 0,<br>    &quot;failed&quot; : 0<br>  },<br>  &quot;hits&quot; : {<br>    &quot;total&quot; : 2,<br>    &quot;max_score&quot; : 1.0,<br>    &quot;hits&quot; : [<br>      {<br>        &quot;_index&quot; : &quot;customer&quot;,<br>        &quot;_type&quot; : &quot;doc&quot;,<br>        &quot;_id&quot; : &quot;2&quot;,<br>        &quot;_score&quot; : 1.0,<br>        &quot;_source&quot; : {<br>          &quot;name&quot; : &quot;Jane Doe&quot;<br>        }<br>      },<br>      {<br>        &quot;_index&quot; : &quot;customer&quot;,<br>        &quot;_type&quot; : &quot;doc&quot;,<br>        &quot;_id&quot; : &quot;1&quot;,<br>        &quot;_score&quot; : 1.0,<br>        &quot;_source&quot; : {<br>          &quot;name&quot; : &quot;John Doe&quot;<br>        }<br>      }<br>    ]<br>  }<br>}</pre><p>You can make the same request using a JSON body:</p><pre>curl -XGET &#39;localhost:9200/customer/_search?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;query&quot;: { &quot;match_all&quot;: {} },<br>  &quot;sort&quot;: [<br>    { &quot;name&quot;: &quot;asc&quot; }<br>  ]<br>}<br>&#39;</pre><p>The number of results returned defaults to 10.</p><p>This request queries a list of accounts and only returns lastname values:</p><pre>curl -XGET &#39;localhost:9200/accounts/_search?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;query&quot;: { &quot;match_all&quot;: {} },<br>  &quot;_source&quot;: [&quot;lastname&quot;]<br>}<br>&#39;</pre><p>Search queries are straightforward:</p><pre>curl -XGET &#39;localhost:9200/customer/_search?pretty&#39; <br>  -H &#39;Content-Type: application/json&#39; -d&#39;<br>{<br>  &quot;query&quot;: { &quot;match&quot;: { &quot;name&quot;: &quot;Doe&quot; } }<br>}<br>&#39;</pre><p>It’s also possible to filter results, and to aggregate results to group and extract statistics.</p><h3>Solr</h3><h4><strong>Install the search engine</strong></h4><ul><li><a href="http://lucene.apache.org/solr/guide/7_1/installing-solr.html">Instructions</a></li><li><a href="http://lucene.apache.org/solr/guide/7_1/a-quick-overview.html">Basic concepts</a></li></ul><h4><strong>Start an interactive session</strong></h4><p>./bin/solr start -e cloud</p><h4><strong>Test it</strong></h4><p><a href="http://localhost:8983/solr/#/">http://localhost:8983/solr/#/</a></p><h4><strong>Create a new document collection</strong></h4><p>./bin/solr create -c mydocs -s 2 -rf 2</p><p>In Solr a <em>collection</em> is a set of related documents: for example, customer data or a product catalog. This corresponds to an <em>index</em> in Elasticsearch.</p><h4><strong>Index some data</strong></h4><p>bin/post -c techproducts example/exampledocs/*</p><p>This indexes files in the <em>example/exampledocs</em> directory, which is the set of sample data provided with the Solr installation. Data can be in any format that Solar recognises: XML, JSON, CSV. Solr can also index plain text files and formats such as PDFs and Word documents.</p><h4><strong>Do a search</strong></h4><pre>curl &#39;localhost:8983/solr/techproducts/select?q=foundation&#39;</pre><p>The response looks like this:</p><pre>{<br>  &quot;responseHeader&quot;: {<br>    &quot;zkConnected&quot;: true,<br>    &quot;status&quot;: 0,<br>    &quot;QTime&quot;: 8,<br>    &quot;params&quot;: {<br>      &quot;q&quot;: &quot;foundation&quot;<br>  }<br>},<br>  &quot;response&quot;: {<br>    &quot;numFound&quot;: 4,<br>    &quot;start&quot;: 0,<br>    &quot;maxScore&quot;: 2.7879646,<br>    &quot;docs&quot;:[{<br>        &quot;id&quot;: &quot;0553293354&quot;,<br>        &quot;cat&quot;: [&quot;book&quot;],<br>        &quot;name&quot;: &quot;Foundation&quot;,<br>        &quot;price&quot;: 7.99,<br>        &quot;price_c&quot;: &quot;7.99,USD&quot;,<br>        &quot;inStock&quot;: true,<br>        &quot;author&quot;: &quot;Isaac Asimov&quot;,<br>        &quot;author_s&quot;: &quot;Isaac Asimov&quot;,<br>        &quot;series_t&quot;: &quot;Foundation Novels&quot;,<br>        &quot;sequence_i&quot;: 1,<br>        &quot;genre_s&quot;: &quot;scifi&quot;,<br>        &quot;_version_&quot;: 1574100232473411586,<br>        &quot;price_c____l_ns&quot;: 799<br>    }]<br>}}</pre><h4><strong>Do a field search</strong></h4><pre>curl &#39;localhost:8983/solr/techproducts/select?q=cat:electronics&#39;</pre><h4><strong>Add a single document, using JSON</strong></h4><pre>curl -X POST -H &#39;Content-Type: application/json&#39;<br>  &#39;<a href="http://localhost:8983/solr/techproducts/update/json/docs&#39;">http://localhost:8983/solr/techproducts/update/json/docs&#39;</a> --data-binary &#39;<br>{<br>  &quot;id&quot;: &quot;1&quot;,<br>  &quot;title&quot;: &quot;Doc 1&quot;<br>}&#39;</pre><h4><strong>Add multiple documents</strong></h4><pre>curl -X POST -H &#39;Content-Type: application/json&#39; &#39;<a href="http://localhost:8983/solr/techproducts/update&#39;">http://localhost:8983/solr/techproducts/update&#39;</a> --data-binary &#39;<br>[<br>  {<br>    &quot;id&quot;: &quot;1&quot;,<br>    &quot;title&quot;: &quot;Doc 1&quot;<br>  },<br>  {<br>    &quot;id&quot;: &quot;2&quot;,<br>    &quot;title&quot;: &quot;Doc 2&quot;<br>  }<br>]&#39;</pre><h4><strong>Update existing documents</strong></h4><pre>curl -X POST -H &#39;Content-Type: application/json&#39;<br>  &#39;<a href="http://localhost:8983/solr/techproducts/update&#39;">http://localhost:8983/solr/techproducts/update&#39;</a> --data-binary &#39;<br>{<br>  &quot;add&quot;: {<br>    &quot;doc&quot;: {<br>      &quot;id&quot;: &quot;DOC1&quot;,<br>      &quot;my_field&quot;: 2.3  <br>    }<br>  },<br>  &quot;add&quot;: {<br>    &quot;commitWithin&quot;: 5000, <br>    &quot;overwrite&quot;: false<br>  },</pre><pre>&quot;commit&quot;: {},<br>  &quot;optimize&quot;: { &quot;waitSearcher&quot;:false },</pre><pre>&quot;delete&quot;: { &quot;id&quot;:&quot;ID&quot; },  <br>  &quot;delete&quot;: { &quot;query&quot;:&quot;QUERY&quot; } <br>}&#39;</pre><p>This request means that this document should be committed within 5 seconds and that there shouldn’t be a check for existing documents with the same <em>uniqueKey</em>.</p><p>Find out more <a href="https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-JSONFormattedIndexUpdates">here</a> about working with JSON in Solr.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b897f072de90" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Developer Portraits: Southeast Asia]]></title>
            <link>https://medium.com/dev-channel/developer-portraits-southeast-asia-4ff54c07f491?source=rss-84c8e3357532------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ff54c07f491</guid>
            <category><![CDATA[malaysia]]></category>
            <category><![CDATA[vietnam]]></category>
            <category><![CDATA[thailand]]></category>
            <category><![CDATA[southeast-asia]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Sam Dutton]]></dc:creator>
            <pubDate>Wed, 26 Jul 2017 19:54:37 GMT</pubDate>
            <atom:updated>2017-07-27T17:16:58.283Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>In this video series we present portraits of Web developers from around the world, starting our journey in Southeast Asia.</strong></h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9Bz5TikmJ-A%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9Bz5TikmJ-A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9Bz5TikmJ-A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fa542494d351ded8f2d6541676d04a5/href">https://medium.com/media/8fa542494d351ded8f2d6541676d04a5/href</a></iframe><p>Our first three videos focus on Jecelyn Yeen from Malaysia, Trung Đinh Quang in Vietnam, and Warat Wongmaneekit in Thailand — all countries with thriving tech communities, vibrant startup scenes, and large numbers of users coming online for the first time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*5oMX0Af1qKh-8yuQYBnuiw.png" /><figcaption>Jecelyn Yeen</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*1EDGCRieHge9TJMOIlrlSA.png" /><figcaption>Trung Đinh Quang</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*TeyYuoCAGvwLL1nVrbEE5g.png" /><figcaption>Warat Wongmaneekit</figcaption></figure><p>A few facts:</p><ul><li>Vietnam (population 91 million), Thailand (68 million) and Malaysia (31 million) have over <a href="http://databank.worldbank.org/data/reports.aspx?source=world-development-indicators">100 million internet users online</a>, and that number is <a href="https://www.webworldwide.io/country/viet-nam">rising fast</a>.</li><li>Metropolitan areas in all three countries have widespread reliable 4G. Cellular rather than wifi connections predominate. Outside urban areas, flakey connectivity and 2G are far more common.</li><li>Internet usage in the region is predominantly from mobile phones, but all three countries have a significant proportion of users on desktop devices.</li><li>Vietnam and Thailand both have large rural populations (around <a href="http://databank.worldbank.org/data/reports.aspx?source=world-development-indicators">66% and 50%</a> respectively) many of whom have poor access to connectivity. In Vietnam, for example <a href="https://www.webworldwide.io/country/viet-nam">only 54.9%</a> of people have access to 4Mbps+ broadband, as the OpenSignal <a href="https://opensignal.com/networks?z=6&amp;minLat=7.8&amp;maxLat=20.6&amp;minLng=93.9&amp;maxLng=116.8&amp;s=all&amp;t=2-3-4">cell coverage map</a> shows below.</li></ul><p>Southeast Asia is unique in having <a href="https://techcrunch.com/2015/09/08/why-southeast-asia-is-leading-the-worlds-most-disruptive-mobile-business-models/">leapfrogged other countries</a> in recent years to become the world’s first truly ‘mobile first’ region. Thailand, for example, has over <a href="http://data.worldbank.org/indicator/IT.CEL.SETS.P2?locations=TH">150 mobile subscriptions for every 100 people</a>, with <a href="https://en.wikipedia.org/wiki/List_of_countries_by_number_of_mobile_phones_in_use">connectivity more widespread than China, the US or India</a>. Vietnam and Thailand also <a href="https://www.forbes.com/sites/dongroves/2015/12/13/the-top-youtube-ads-of-2015-in-asia-pacific-vietnam-and-thailand-dominate/#79221be7466b">rank among the ten biggest</a> (and fastest growing) markets worldwide for total viewing time on YouTube. Fast growth also means there are still huge opportunities for user-generated content on the Web in Southeast Asia.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bcO7G_bXGdQw2YNo." /><figcaption>Map from opensignal.com, showing 2G and 3G coverage</figcaption></figure><p>Filmmaker Nilesh Bell-Gorsia describes how the videos were made:</p><blockquote>Over 12 days we visited three countries on a whistle stop tour. I was accompanied by a talented crew in Co-Producer Jamie Baughman, Director of Photography Andy Hoffman and Tasha Van Zandt with Assistant Camera Sebastian Zeck and Sound Recordist Bryan Gordon. Together we forged through fish markets at 3am, filmed out the back of moving tuk tuks and ran the length of football pitches to capture that perfect moment!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*S9tBvgiej2EH7ikYOgYh4A.png" /><figcaption>Jecelyn Yeen in Kuala Lumpur</figcaption></figure><blockquote>We traveled as light as physically possible so we were able to be pretty nimble on location. Armed with a pair of Canon C300 Mark II cameras, at 4k UHD 3840x2160, shooting on Canon L-Series Lenses, 35mm, 85mm, 16–35mm, 24–70, 70–200mm, with a couple of Sachtler FSB 6 tripods with carbon fiber sticks. To light the interviews we made use of two Wescott 1x1&#39; daylight balanced flex lights. Recording audio on a Sound Devices 702T Recorder with a Sennheiser MK-416 Shotgun microphone, Sennheiser ew 100 G3 Wireless and a Countryman B6 Omnidirectional Lavalier.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M0isfv5YKUHZZx_DotqPhg.png" /><figcaption>Trung Đinh Quang after football</figcaption></figure><blockquote>We sampled life in three different cities and were graciously invited into the lives of three brilliant Web developers. We had the privilege of being welcomed into the homes of each with the theme of ‘family’ imbuing our trip. Each short film explores their unique journeys as well as touching upon themes that conjoin them all.</blockquote><blockquote>In Malaysia, we learned about the importance of Jecelyn’s relationship with her father during her career, as well as her role in helping to build developer communities for women, alongside creating tutorials and code samples.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4nviMiA5h3GQmKzjklGgSw.png" /><figcaption>Fish market in Kuala Lumpur</figcaption></figure><blockquote>In Vietnam, Trung left his hometown for Ho Chi Minh City to become a Web developer. We documented the life he left behind and shared a beautiful home-cooked meal with Trung and his family. We were also taught how to fish by local fisherman in the rural <a href="https://en.wikipedia.org/wiki/B%C3%ACnh_Ph%C6%B0%E1%BB%9Bc_Province">Bin Phuoc</a>!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wM9vC_jPKvpNIOXe5r7ryA.png" /><figcaption>Filming a football match in Ho Chi Minh City</figcaption></figure><blockquote>In the bustle of metropolitan Thailand, we explored the turning point in Warat’s working life and the changes in his own technological perspective, along with his relationship with his mother as she grapples with new technology.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MOErYnevJVPZ8E70giS1PA.png" /><figcaption>Filming a tuk tuk from a tuk tuk: Warat Wongmaneekit in Bangkok</figcaption></figure><blockquote>As we witnessed these strong familial relationships, it was evident each developer was also growing a respective developer family of their own.</blockquote><p>In the future we hope to speak to developers in other regions where the greatest number of people are coming online for the first time. The Web’s technological and cultural centre of gravity is moving steadily away from the US and Western Europe, and we want to find out what this means for developers and users.</p><h3>Find out more</h3><ul><li><a href="https://developers.google.com/billions/">developers.google.com/billions</a></li><li><a href="https://docs.google.com/document/d/1LV55gzhvvfIvCTrn9yx6kvRrtXgjxIrzbQT-hSLcHZ0/edit#">The Global Web: data, information and tools</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ff54c07f491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dev-channel/developer-portraits-southeast-asia-4ff54c07f491">Developer Portraits: Southeast Asia</a> was originally published in <a href="https://medium.com/dev-channel">Dev Channel</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>