<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ying Ray Lu on Medium]]></title>
        <description><![CDATA[Stories by Ying Ray Lu on Medium]]></description>
        <link>https://medium.com/@yingray?source=rss-ea675b0d7c2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bRtFBPRKEcEu-HOwgpgIug@2x.jpeg</url>
            <title>Stories by Ying Ray Lu on Medium</title>
            <link>https://medium.com/@yingray?source=rss-ea675b0d7c2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:35:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yingray/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A simple example for Helm chart]]></title>
            <link>https://medium.com/htc-research-engineering-blog/a-simple-example-for-helm-chart-fbb5c7208e94?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbb5c7208e94</guid>
            <category><![CDATA[helm]]></category>
            <category><![CDATA[charts]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[tiller]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 09:51:11 GMT</pubDate>
            <atom:updated>2019-08-21T16:03:23.986Z</atom:updated>
            <content:encoded><![CDATA[<p>In Kubernetes’ world, while you want to deploy a microservices, you need to have some of the requirements like docker images, manifest YAML of deployments, services, etc. For docker images, it easily controls the version with the version tag of the images and usual stores in the docker registry.</p><p>Helm chart packages all of Kubernetes manifests and also provides the version tag. Besides, Helm provides Go-Template syntax for you. You can configure all the values, inject to manifests when deploying to the cluster.</p><p>Sounds good right? Let’s start to play a simple example with Helm!</p><pre>brew install kubernetes-helm</pre><p>Before playing with Helm, you need a Kubernetes cluster. You can use your existing cluster or refer to this <a href="https://medium.com/htc-research-engineering-blog/quick-start-local-kubernetes-cluster-with-minikube-a8e914b22887">instruction</a> and set up a local cluster.</p><p>Okay, let’s try Helm install command to install the WordPress chart which is provided from Helm. And you’ll get the error message.</p><pre>helm install --name wordpress stable/wordpress<br>&gt;&gt; Error: could not find tiller</pre><p>Before learning Helm, you need to know what Tiller is.</p><p>Tiller is the server portion of Helm, and you need to install it into your Kubernetes cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WRlTslNkaEPgFI1EPC67MQ.png" /><figcaption>Helm Tiller vs Kubectl</figcaption></figure><p>You can simply use Helm init command to deploy Tiller into the cluster.</p><pre>helm init</pre><p>Now, Tiller runs inside of your Kubernetes cluster. Check Tiller running into the Kube-system namespace:</p><pre>kubectl get pods --namespace kube-system<br>&gt;&gt;<br>...<br>tiller-deploy-845fb7cfc6-qkwgh     1/1     Running   0          83m</pre><p>And then do Helm install command again, you may get another warning like:</p><pre>&gt;&gt; Error: release wordpress failed: namespaces &quot;default&quot; is forbidden: User &quot;system:serviceaccount:kube-system:default&quot; cannot get resource &quot;namespaces&quot; in API group &quot;&quot; in the namespace &quot;default&quot;</pre><p>It means Tiller has no permission to release the chart. Let’s create a cluster role binding because it needs cluster-admin permission to deploy the services.</p><pre>kubectl create serviceaccount tiller -n kube-system<br>kubectl create clusterrolebinding <strong>tiller</strong> \<br>    --clusterrole cluster-admin \<br>    --serviceaccount=kube-system:<strong>tiller</strong></pre><p>Then initialized Helm Tiller within the Tiller service account</p><pre>helm init --service-account <strong>tiller</strong></pre><p>Ensure the version of Helm server and client:</p><pre>helm version<br>&gt;&gt; <br>Client: &amp;version.Version{SemVer:&quot;v2.14.2&quot;, GitCommit:&quot;a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7&quot;, GitTreeState:&quot;clean&quot;}<br>Server: &amp;version.Version{SemVer:&quot;v2.14.2&quot;, GitCommit:&quot;a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7&quot;, GitTreeState:&quot;clean&quot;}</pre><p>Install again and get the release notes. You can follow the notes and get your own WordPress website with the browser. Today, we will learn how to create and customize a Helm chart, so please remove it from the cluster.</p><pre>helm del --purge wordpress</pre><h3>Create a Helm chart</h3><p>Firstly, let’s create a Helm chart with CLI:</p><pre>helm create what-the-helm</pre><p>Inside of the directory:</p><pre>what-the-helm<br>├── Chart.yaml <br>├── charts<br>├── templates<br>│   ├── NOTES.txt<br>│   ├── _helpers.tpl<br>│   ├── deployment.yaml<br>│   ├── ingress.yaml<br>│   ├── service.yaml<br>│   └── tests<br>│       └── test-connection.yaml<br>└── values.yaml</pre><p>In this Helm chart, there’s deployment, service, ingress manifests in the templates folder…</p><p>Let’s start to deploy the Helm chart:</p><pre>helm install --name what-the-helm .<br>&gt;&gt; <br>NAME:   what-the-helm<br>LAST DEPLOYED: <br>NAMESPACE: default<br>STATUS: DEPLOYED</pre><pre>RESOURCES:<br>==&gt; v1/Deployment<br>NAME           READY  UP-TO-DATE  AVAILABLE  AGE<br>what-the-helm  0/1    1           0          0s</pre><pre>==&gt; v1/Pod(related)<br>NAME                            READY  STATUS             RESTARTS  AGE<br>what-the-helm-7d78fbf657-xgftv  0/1    ContainerCreating  0         0s</pre><pre>==&gt; v1/Service<br>NAME           TYPE          CLUSTER-IP  EXTERNAL-IP  PORT(S)       AGE<br>what-the-helm  ClusterIP  10.0.27.42  &lt;pending&gt;    80:31081/TCP  0s</pre><pre>NOTES:<br>1. Get the application URL by running these commands:<br>     NOTE: It may take a few minutes for the LoadBalancer IP to be available.<br>           You can watch the status of by running &#39;kubectl get --namespace default svc -w what-the-helm&#39;<br>  export SERVICE_IP=$(kubectl get svc --namespace default what-the-helm -o jsonpath=&#39;{.status.loadBalancer.ingress[0].ip}&#39;)<br>  echo <a href="http://$service_ip/">http://$SERVICE_IP:80</a></pre><p>Wait for pod running:</p><pre>kubectl get pods -w<br>&gt;&gt;<br>NAME                             READY   STATUS    RESTARTS   AGE<br>what-the-helm-7d78fbf657-xgftv   1/1     Running   0          76s</pre><p>Use port-forward to forward the local port to service.</p><pre>kubectl port-forward svc/what-the-helm 8888:80</pre><p>Check browser: <a href="http://localhost:8888/">http://localhost:8888</a>, then you’ll see Nginx welcome page.</p><h3>Helm Revision</h3><p>Now we make a little change for what-the-helm. Firstly, let’s create a ConfigMap Yaml in templates folder with the given configuration below:</p><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nginx-configmap<br>data:<br>  index.html: |<br>    &lt;h1&gt;What the Helm!!!&lt;/h1&gt;</pre><p>And mount this ConfigMap volume to Pod:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FpU2Z6iJ1tekmyYbAce9UA.png" /></figure><pre>    volumeMounts:<br>      - name: nginx<br>        mountPath: /usr/share/nginx/html<br>volumes:<br>  - name: nginx<br>    configMap:<br>      name: nginx-configmap</pre><p>Then upgrade it!</p><pre>helm upgrade what-the-helm .</pre><p>And check <a href="http://localhost:8080/">http://localhost:8080</a> again, you can see the welcome page was changed to our custom page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*4T_uRlxtskZPrDt_5ba9tw.png" /></figure><p>Also, try to use go template syntax like this:</p><pre># configmap.yaml<br>index.html: {{ .Values.pageContent }}</pre><p>For this example, Helm will render it from the values.YAML, so that we need to add the parameter to values.yaml.</p><pre># values.yaml<br>pageContent: | <br>  &lt;h1&gt; What the Helm Template 0.0 &lt;/h1&gt;</pre><p>Before upgrading the chart, if you want to test the template rendering, but not actually install anything, you can use the command below. This will send the chart to the Tiller server, which will render the templates. But instead of installing the chart, it will return the rendered template to you.</p><pre>helm install . --dry-run --debug</pre><p>Also, you can use the template command. This will directly send the rendered template for you. Let’s check it out!</p><pre>helm template .<br>&gt;&gt;&gt;<br>---<br># Source: what-the-helm/templates/configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nginx-configmap<br>data:<br>  index.html: &lt;h1&gt; What the Helm Template 0.0 &lt;/h1&gt;<br>---<br>...</pre><p>Helm provides a simple linter to verify that your chart is well-formed.</p><pre>helm lint .</pre><p>Okay, it seems like everything all right. Upgrade it and check it out with browser!</p><p>Helm also provides the feature for accessing files inside templates, let’s try to create a new HTML file:</p><pre># what-the-helm/static/index.html<br>&lt;h1&gt; What the Helm accessing files :)) &lt;/h1&gt;</pre><p>Modify configmap.yaml.</p><pre># what-the-helm/templates/configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nginx-configmap<br>data:<br>  {{- (.Files.Glob &quot;static/*&quot;).AsConfig | nindent 2 }}</pre><p>Upgrade it and check it out with browser again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*kuVl9gxZpsar2L8Sxft2Eg.png" /></figure><p>Nice job!</p><pre>helm history what-the-helm<br>&gt;&gt;&gt;<br>REVISION  UPDATED   STATUS     CHART               DESCRIPTION<br>1         ...       SUPERSEDED what-the-helm-0.1.0 Install complete<br>2         ...       SUPERSEDED what-the-helm-0.1.0 Upgrade complete<br>3         ...       SUPERSEDED what-the-helm-0.1.0 Upgrade complete<br>4         ...       DEPLOYED   what-the-helm-0.1.0 Upgrade complete</pre><p>Let’s try to rollback the chart to the first revision.</p><pre>helm rollback what-the-helm 1</pre><p>Then you’ll see Nginx welcome page come back and add another revision into the history as well.</p><pre>helm history what-the-helm<br>&gt;&gt;&gt;<br>REVISION  UPDATED   STATUS     CHART               DESCRIPTION<br>...<br>5         ...       DEPLOYED   what-the-helm-0.1.0 Rollback to 1</pre><p>By the way, if you want to keep the Helm history shortly, set the history max when initializing Tiller.</p><pre>helm init --service-account tiller --history-max 3</pre><p>Or edit the tiller-deploy deployment and change TILLER_HISTORY_MAX.</p><p>The final step, let’s list all of the releases by Helm and delete the chart we made.</p><pre>helm list <br>&gt;&gt;<br>NAME          REVISION UPDATED                  STATUS   CHART               APP VERSION NAMESPACE<br>what-the-helm 8        Sun Aug  4 17:43:28 2019 DEPLOYED what-the-helm-0.1.0 1.0         default</pre><pre>helm del --purge what-the-helm<br>&gt;&gt; release &quot;what-the-helm&quot; deleted</pre><h3>Distributing Charts</h3><p>To distribute your own chart, it definitely to package your chart first.</p><pre>helm package .<br>&gt;&gt; Successfully packaged chart and saved it to: /what-the-helm/what-the-helm-0.1.0.tgz</pre><p>In this example, I will create a Github project for storing helm charts, so that let’s make a new empty folder and connect with your own Github project.</p><pre>mkdir charts<br>cd charts</pre><pre>git init ...</pre><p>Copy the package of your chart into this folder. Then generate index.yaml which is the repository manifest with the command below:</p><pre>mv /what-the-helm/what-the-helm-0.1.0.tgz .<br>helm repo index .</pre><pre>cat index.yaml<br>&gt;&gt;<br>apiVersion: v1<br>entries:<br>  what-the-helm:<br>  - apiVersion: v1<br>    appVersion: &quot;1.0&quot;<br>    created: &quot;&quot;<br>    description: A Helm chart for Kubernetes<br>    digest: xxx<br>    name: what-the-helm<br>    urls:<br>    - what-the-helm-0.1.0.tgz<br>    version: 0.1.0</pre><p>After pushing, uploading the charts to your storage, add this repo URL to Helm.</p><pre>helm repo add my-repo <a href="https://raw.githubusercontent.com/yingray/charts/master">https://raw.githubusercontent.com/ACCOUNT/REPO_NAME/master</a></pre><pre>helm repo list<br>&gt;&gt;<br>NAME    URL<br>stable  <a href="https://kubernetes-charts.storage.googleapis.com/">https://kubernetes-charts.storage.googleapis.com</a><br>my-repo <a href="https://raw.githubusercontent.com/ACCOUNT/REPO_NAME/master">https://raw.githubusercontent.com/ACCOUNT/REPO_NAME/master</a><br>...</pre><p>With Helm searching command, you can find your own repo and charts.</p><pre>helm search</pre><p>For now on, you can directly install your own chart with only one command:</p><pre>helm install --name what-the-helm my-repo/what-the-helm</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbb5c7208e94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/a-simple-example-for-helm-chart-fbb5c7208e94">A simple example for Helm chart</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quick start local Kubernetes cluster with Minikube]]></title>
            <link>https://medium.com/htc-research-engineering-blog/quick-start-local-kubernetes-cluster-with-minikube-a8e914b22887?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8e914b22887</guid>
            <category><![CDATA[minikube]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubectl]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Sun, 04 Aug 2019 08:03:33 GMT</pubDate>
            <atom:updated>2022-05-03T04:31:19.671Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quick setup a local Kubernetes cluster with Minikube</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2C37AygKUVz7JDPA" /><figcaption>Photo by <a href="https://unsplash.com/@danielleone?utm_source=medium&amp;utm_medium=referral">Daniel Leone</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To develop microservices with Kubernetes, it usually needed a local environment for development. Here’s how to quickly set up a local cluster with Minikube on MacOS.</p><h3>Prerequisites</h3><p>Run the command to install kubectl (Kubernetes cluster manager)</p><pre>brew install kubernetes-cli</pre><p>Test to ensure the version you installed is up-to-date</p><pre>kubectl version</pre><p>Install Virtualbox using Homebrew, or download the dmg from <a href="https://www.virtualbox.org/wiki/Downloads)">https://www.virtualbox.org/wiki/Downloads</a></p><pre>brew cask install virtualbox</pre><p>The easiest way to install Minikube on macOS is by using Homebrew</p><pre>brew cask install minikube</pre><p>Then test to ensure the version you installed is up-to-date</p><pre>minikube version</pre><h3><strong>Setup local cluster</strong></h3><p>Start Minikube and create a cluster</p><pre>minikube start</pre><p>After installing the cluster, set docker-env to let Minikube cluster access your local images</p><pre>eval $(minikube docker-env)</pre><h3>Setup local cluster by Docker(new)</h3><p>Install Docker desktop and go to Preferences &gt; Kubernetes &gt; Enable Kubernetes, Done!</p><h3>Useful tools for Kubernetes</h3><ul><li><strong>Kubectl Plugin(oh-my-zsh)</strong></li></ul><p>This plugin adds completion for the <a href="https://kubernetes.io/docs/reference/kubectl/kubectl/">Kubernetes cluster manager</a>, as well as some aliases for common kubectl commands. (<a href="https://github.com/robbyrussell/oh-my-zsh/tree/master/plugins/kubectl">https://github.com/robbyrussell/oh-my-zsh/blob/master/plugins/kubectl/kubectl.plugin.zsh</a>)</p><ul><li>K9s</li></ul><p>This CLI provides a terminal UI to interact with your Kubernetes clusters.</p><p>(<a href="https://github.com/derailed/k9s">https://github.com/derailed/k9s</a>)</p><ul><li><strong>kubectx + kubens</strong></li></ul><p>This project helps you switching faster between clusters and namespaces. (<a href="https://github.com/ahmetb/kubectx">https://github.com/ahmetb/kubectx</a>)</p><ul><li><strong>Helm</strong></li></ul><p>Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. (<a href="https://github.com/helm/helm">https://github.com/helm/helm</a>)</p><ul><li><strong>Skaffold</strong></li></ul><p>Skaffold is an open-source project built by Google which is a command-line tool that facilitates continuous development for Kubernetes applications. (<a href="https://skaffold.dev/">https://skaffold.dev/</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8e914b22887" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/quick-start-local-kubernetes-cluster-with-minikube-a8e914b22887">Quick start local Kubernetes cluster with Minikube</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[秒玩LINE BOT聊天機器人Front-end Framework: LIFF]]></title>
            <link>https://medium.com/@yingray/%E7%A7%92%E7%8E%A9line%E7%9A%84front-end-framework-liff-2dad471c2d98?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dad471c2d98</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[liff]]></category>
            <category><![CDATA[line-front-end-framework]]></category>
            <category><![CDATA[messaging]]></category>
            <category><![CDATA[line]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Thu, 07 Jun 2018 06:26:15 GMT</pubDate>
            <atom:updated>2018-11-14T07:06:01.622Z</atom:updated>
            <content:encoded><![CDATA[<h3>LINE Front-end Framework = LIFF</h3><p>LINE於昨日(2018/6/6)公佈了一項全新功能，這項功能目的似乎是為了提升使用者聊天與網站互動、切換的體驗，LINE在昨天推出了內置WebView的功能，產生下圖的效果。這項功能也緊接著造福我們開發者使用，讓我們深入淺出，瞭解LIFF要怎麼使用吧🧐。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*WKYW3KA98ML5Gco9.png" /><figcaption>LINE Front-end Framework</figcaption></figure><p>首先，你必須要有你聊天機器人的頻道，與Channel Access Token，然後就可以開始試玩LIFF。</p><p>我們先從最好理解的方式下手，塞一個Google網站在我們的LIFF中，接著使用CURL或任何提出http request的方式提出下列的請求：</p><pre>curl -X POST https://api.line.me/liff/v1/apps \<br>-H &quot;Authorization: Bearer {channel access token}&quot; \<br>-H &quot;Content-Type: application/json&quot; \<br>-d &#39;{<br>  &quot;view&quot;:{<br>    &quot;type&quot;:&quot;<strong>compact</strong>&quot;,<br>    &quot;url&quot;:&quot;<strong>https://www.google.com</strong>&quot;<br>  }<br>}&#39;</pre><p>發送並通過所有認證後，你會收到一組LIFF ID：</p><pre>{&quot;liffId&quot;:&quot;1565953323-RLmdD77Z&quot;}</pre><p>試著在您的聊天機器人頻道內打上以下連結並發送：</p><pre>line://app/{liffId}</pre><p>點選此連結後(初次需要通過認證)：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_rt5MaI-v_7TUsaNUKToWA.jpeg" /></figure><p>我們可以發現，Google首頁以一個優雅的姿態覆蓋在聊天室之上，而非跳轉至LINE內建瀏覽器中。</p><p>除了他非常優雅美觀的依附在我們的聊天室上外，也提供了幾組實用的API給瀏覽器端使用。LINE非常好心的幫我們製作好Web example如下：</p><p><a href="https://github.com/line/line-liff-starter">line/line-liff-starter</a></p><p>除了官方的建議，使用Heroku部署方式外，我們也可以直接部署到Github pages，步驟如下：</p><ul><li>點選右上角的Fork，在你帳號下建立副本。</li><li>成功建立副本後，在你的Repo下點選Settings。</li><li>在Settings options中，至Github pages，選擇master branch並存擋。</li><li>稍待片刻，範例網站便部署到https://{account}.github.io/line-liff-starter/</li><li>不想部署沒關係，用我的<a href="https://yingray.github.io/line-liff-starter/">https://yingray.github.io/line-liff-starter/</a></li></ul><p>部署完成後，試著把剛才CURL中的Google網址換成範例網站的網址，並調整你想要的LIFF框架大小(compact, tall, full)，如下：</p><pre>curl -X POST https://api.line.me/liff/v1/apps \<br>-H &quot;Authorization: Bearer {channel access token}&quot; \<br>-H &quot;Content-Type: application/json&quot; \<br>-d &#39;{<br>  &quot;view&quot;:{<br>    &quot;type&quot;:&quot;<strong>tall</strong>&quot;,<br>    &quot;url&quot;:&quot;<a href="https://yingray.github.io/line-liff-starter/"><strong>https://yingray.github.io/line-liff-starter/</strong></a>&quot;<br>  }<br>}&#39;</pre><p>依照之前的方式，將LIFF呼叫出來，可以得到官方的範例頁面：</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPoEW2sEm4BxovL4XNB85w.png" /></figure><p>目前LINE提供LIFF網頁的SDK共有5組API，除了初始化的API之外，上面的4個按鈕代表了另外4組的API。其中有，開啟瀏覽器、關閉LIFF、取得User的資訊、發送訊息。使用方法都可以參考fork過來的源碼，非常淺顯易懂。</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dad471c2d98" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn how to use Observable with ReactiveX — Javascript ⚾️]]></title>
            <link>https://medium.com/htc-research-engineering-blog/learn-how-to-use-observable-with-reactivex-javascript-859eb2c4508b?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/859eb2c4508b</guid>
            <category><![CDATA[xr]]></category>
            <category><![CDATA[observables]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[rxjs]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Mon, 04 Jun 2018 02:53:03 GMT</pubDate>
            <atom:updated>2018-06-04T09:37:12.252Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dKQ4wauj61rZ4wOv" /><figcaption>“Colorful software or web code on a computer monitor” by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In Javascript’s world, how to handle the async event is a big issue. You may come up with <strong>Promise</strong> function, <strong>Async</strong> function, etc. But sometimes you need to control a series of tasks with the async event and it’s always hard to deal with. For example, there is no method for stopping the promise when it’s executing. It still has some solutions to work around, but it makes code hard reading and maintaining.</p><p>ReactiveX provides a good pattern to control the async event. It combines the Observer pattern with the Iterator pattern and functional programming with collections. So that it makes sequences of events like an array and uses the operators like the Lodash. There have several topics for you to understand the good parts of ReactiveX below:</p><ol><li>What is the difference between the “rx” and “rxjs” from npm packages?</li><li>Start to create a sample observable and subscribe it!</li><li>Introduce some of the core operators.</li><li>How to use it with React project?</li></ol><h3>What is the difference between the “rx” and “rxjs” from npm packages?</h3><p>First, let’s figure out the difference between the “<a href="https://github.com/Reactive-Extensions/RxJS">rx</a>” and “<a href="https://github.com/ReactiveX/rxjs">rxjs</a>”. In short, “rxjs” is rewrite of “rx” and is the latest production-ready version of RxJS which have better performance, better modularity, better debuggable call stacks while staying mostly backward compatible.</p><p><a href="https://github.com/Reactive-Extensions/RxJS">Reactive-Extensions/RxJS</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*lyqbiRpHw07ABEZ9ftJy_g.png" /><figcaption>Using operator with the previous version of rxjs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*iLbGfJTbhWNh-8wP_LpdeA.png" /><figcaption>Using operator with the previous version of rxjs</figcaption></figure><p><a href="https://github.com/ReactiveX/rxjs">GitHub - ReactiveX/rxjs: A reactive programming library for JavaScript</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*3wnak2bPMsZ3jiDx-3yMWQ.png" /><figcaption>Using operator with the latest version of rxjs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*2Hk89_AAdVTOAe3PgvMtcw.png" /><figcaption>Using operator with the latest version of rxjs</figcaption></figure><h3>Start to create a sample observable and subscribe it!</h3><p>Before creating an observable, let’s recognize the essential concepts in RxJS:</p><ul><li><strong>Observable:</strong> represents the idea of an invokable <strong><em>collection</em></strong> of future values or events.</li><li><strong>Observer:</strong> is a collection of callbacks that knows how to listen to values delivered by the Observable.</li><li><strong>Subscription:</strong> represents the execution of an Observable, is primarily useful for canceling the execution.</li><li><strong>Operators:</strong> are pure functions that enable a functional programming style of dealing with collections with operations like map, filter, flatMap, etc.</li></ul><p>Reference: <a href="http://reactivex.io/rxjs/manual/tutorial.html">http://reactivex.io/rxjs/manual/tutorial.html</a></p><p>Let’s create a source observable by the operator of create, and internally produce new events. Then, subscribe the source Observable. (I used the previous version of Rx.js which imports all of the operators to let us get the autocompletion)</p><pre>// create.js</pre><pre>import Rx from &quot;rx&quot;;<br><br>const source = Rx.Observable.create(observer =&gt; {<br>  observer.onNext(&quot;yingray say hi!&quot;);<br>  observer.onNext(&quot;frank say hi!&quot;);<br>  observer.onCompleted();<br>});<br><br>const subscription = source.subscribe(<br>  value =&gt; console.log(`Next: ${value}`),<br>  error =&gt; console.log(`Error: ${error}`),<br>  () =&gt; console.log(&quot;Completed!&quot;)<br>);</pre><p><a href="http://reactivex.io/documentation/operators/create.html">ReactiveX - Create operator</a></p><h3>Introduce some of the core operators.</h3><p>Let’s recognize the core operators and find the examples below:</p><p><a href="https://github.com/yingray/rx-demo">yingray/rx-demo</a></p><p>range.js</p><p><a href="http://reactivex.io/documentation/operators/range.html">ReactiveX - Range operator</a></p><p>interval.js</p><p><a href="http://reactivex.io/documentation/operators/interval.html">ReactiveX - Interval operator</a></p><p>map.js</p><p><a href="http://reactivex.io/documentation/operators/map.html">ReactiveX - Map operator</a></p><p>scan.js</p><p><a href="http://reactivex.io/documentation/operators/scan.html">ReactiveX - Scan operator</a></p><h3>FlatMap vs ConcatMap</h3><p><a href="http://reactivex.io/documentation/operators/flatmap.html">ReactiveX - FlatMap operator</a></p><p>baseball.js: <a href="https://github.com/yingray/rx-demo/blob/master/baseball.js">https://github.com/yingray/rx-demo/blob/master/baseball.js</a></p><h4>flatMap result</h4><pre>.<br>.<br>.<br>1006 ms: 120: H<br>.<br>.<br>.<br>2008 ms: 103: H<br>.<br>.<br>.<br>3008 ms: Ball landing<br>3009 ms: 151: swing and miss<br>.<br>.<br>.<br>.<br>4010 ms: Ball landing<br>4010 ms: 98: H<br>.<br>.<br>.<br>.<br>.<br>.<br>6011 ms: Ball landing</pre><h4>concatMap result</h4><pre>.<br>.<br>.<br>1004 ms: 120: H<br>.<br>.<br>.<br>.<br>.<br>.<br>3006 ms: Ball landing<br>3007 ms: 103: H<br>.<br>.<br>.<br>.<br>.<br>.<br>.<br>5009 ms: Ball landing<br>5009 ms: 151: swing and miss<br>5009 ms: 98: H<br>.<br>.<br>.<br>.<br>.<br>.<br>.<br>7011 ms: Ball landing</pre><h3><strong>How to use it with React project?</strong></h3><p>We back to the web application and figure out how to make good use of ReactiveX on React project. There is an example which needs to fetch the RESTful API to get the data from another service or 3rd party.</p><pre>const getData = () =&gt; fetch(‘https://yingray-services/data’);</pre><p>In this case, while the component has already mounted, it fetches the request to the service which is the async event and returns a promise object.</p><pre>MyPage extends React.Component {</pre><pre>  componentDidMount() {<br>    getDate()<br>      .then(response =&gt; response.json())<br>      .then(data =&gt; this.setState({ data }));<br>  }</pre><pre>}</pre><p>Let’s convert to the observable from promise object. In the latest version of Rx.js, we use the operator of ‘from’. Next, let’s make a subscription to subscribe this observable. Don’t forget to let it know how to deal with the success event and fail event.</p><pre>import { from } from &#39;rxjs&#39;;</pre><pre>MyPage extends React.Component {</pre><pre>  componentDidMount() {</pre><pre>    const observable = from(<br>                         getData()<br>                           .then(response =&gt; response.json())<br>                       );</pre><pre>    this.subscription = observable.subscribe(<br>      (data) =&gt; this.setState({ data }),<br>      (err) =&gt; console.error(err),<br>    );</pre><pre>  }</pre><pre>}</pre><p>Finally, to avoid the forceUpdate issues, let’s do unsubscribe before the component is unmounted. You must pay attention to the scope of including the promise because it only stops watching the observable but not the execution of the promise. (<strong>It means it may execute the response.json() after it’s unsubscribed</strong>)</p><pre>componentWillUnmount() {<br>  if (this.subscription &amp;&amp; !this.subscription.isStopped) {<br>    this.subscription.unsubscribe();<br>  }<br>}</pre><h3>References</h3><ol><li>Tutorial: <a href="http://reactivex.io/rxjs/manual/overview.html">http://reactivex.io/rxjs/manual/overview.html</a></li><li>Documentation: <a href="http://reactivex.io/documentation/operators.html">http://reactivex.io/documentation/operators.html</a></li><li>Node module: <a href="https://github.com/ReactiveX/rxjs">https://github.com/ReactiveX/rxjs</a></li><li>React integration: <a href="http://reactivex.io/rxjs/manual/tutorial.html#react">http://reactivex.io/rxjs/manual/tutorial.html#react</a></li><li>Pipeable Operators: <a href="https://github.com/ReactiveX/rxjs/blob/master/doc/pipeable-operators.md">https://github.com/ReactiveX/rxjs/blob/master/doc/pipeable-operators.md</a></li><li>The introduction to Reactive Programming you’ve been missing: <a href="https://gist.github.com/staltz/868e7e9bc2a7b8c1f754">https://gist.github.com/staltz/868e7e9bc2a7b8c1f754</a></li><li>Techbridge: <a href="https://blog.techbridge.cc/2017/12/08/rxjs/">https://blog.techbridge.cc/2017/12/08/rxjs/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=859eb2c4508b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/learn-how-to-use-observable-with-reactivex-javascript-859eb2c4508b">Learn how to use Observable with ReactiveX — Javascript ⚾️</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview the browser prompts of A2HS and Notification (PWA)]]></title>
            <link>https://medium.com/htc-research-engineering-blog/overview-the-browser-prompts-of-a2hs-and-notification-pwa-ce9b3029ba02?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce9b3029ba02</guid>
            <category><![CDATA[push-notification]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[progressive-web-app]]></category>
            <category><![CDATA[add-to-home-screen]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Tue, 08 May 2018 07:03:26 GMT</pubDate>
            <atom:updated>2018-05-08T08:02:28.161Z</atom:updated>
            <content:encoded><![CDATA[<h4>Agenda</h4><ul><li>The prompts always show up once!</li><li>How to handle the problems after we block or ignore the prompts?</li><li>Add to Home Screen(A2HS): Let’s refer to the good patterns, anti-patterns with the real world PWA. And we will talk about how to defer the event of before-install-prompt.</li><li>Push Notification: Let’s think about the UX requirements of push notification. And also, see the pattern which can <strong>NOT</strong> fulfill the requirements. How to do the good practice?</li></ul><p><a href="https://github.com/yingray/pwa-demo">yingray/pwa-demo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce9b3029ba02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/overview-the-browser-prompts-of-a2hs-and-notification-pwa-ce9b3029ba02">Overview the browser prompts of A2HS and Notification (PWA)</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch: Optimizing your search(2)]]></title>
            <link>https://medium.com/htc-research-engineering-blog/elasticsearch-optimizing-your-search-2-e91f46be8ad6?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e91f46be8ad6</guid>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[filters]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[analyzer]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Thu, 01 Mar 2018 07:46:15 GMT</pubDate>
            <atom:updated>2018-03-02T08:40:53.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gByCd-DsgfSddLIv." /><figcaption>“Neatly organized cutting board with hamburger, cheese, and vegetable fixings” by <a href="https://unsplash.com/@oelli?utm_source=medium&amp;utm_medium=referral">Elli O.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the previous article, we have acquired an initial understanding how to handle the documents with Elasticsearch. However, if you want to improve the ability of search, especially <strong>mandarin</strong>, we still have lots of efforts needed to address. In this topic, I’d like to talk about the “term” of Elasticsearch and share some solutions for searching improvement.</p><h3>Inverted Index</h3><p>Inverted index, is the good structure of Elasticsearch. It is usually designed to allow very fast full-text searches. To understand the inverted index, let me give you the example from the official website, we first create 2 documents, each with a content field containing the following:</p><ol><li><em>The quick brown fox jumped over the lazy dog.</em></li><li><em>Quick brown foxes leap over lazy dogs in summer.</em></li></ol><pre>PUT /ii_example/_doc/1<br>{<br>  &quot;content&quot;: &quot;The quick brown fox jumped over the lazy dog.&quot;<br>}</pre><pre>PUT /ii_example/_doc/2<br>{<br>  &quot;content&quot;: &quot;Quick brown foxes leap over lazy dogs in summer.&quot;<br>}</pre><pre>GET /ii_example/_doc/_search</pre><h4>Terms</h4><p>At index time, the built-in <strong><em>english analyzer</em></strong> would convert this sentence of each document into separate words (which we call <strong><em>terms</em></strong>, or <strong><em>tokens</em></strong>):</p><pre>Term      Doc_1  Doc_2<br>-------------------------<br>quick   |   X   |  X<br>brown   |   X   |  X<br>dog     |   X   |<br>dogs    |       |  X<br>fox     |   X   |<br>foxes   |       |  X<br>in      |       |  X<br>jumped  |   X   |<br>lazy    |   X   |  X<br>leap    |       |  X<br>over    |   X   |  X<br>summer  |       |  X<br>------------------------</pre><p>Now, if we want to search for a lazy dog, Elasticsearch queries the terms of lazy and dog, and then finds out the mapping documents.</p><pre>Query Terms   ===&gt;   Get Document&#39;s ID</pre><p>The request body:</p><pre>GET /ii_example/_doc/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;content&quot;: &quot;a lazy dog&quot;<br>    }<br>  }<br>}</pre><p>This same analysis process is applied to the query string at search time. The query text will be converted into terms of the same form as those that are stored in the inverted index. The result we find will be like:</p><pre>Term      Doc_1  Doc_2<br>-------------------------<br>lazy    |   X   |  X<br>dog     |   X   |<br>------------------------<br>Total   |   2   |  1</pre><pre>{<br>  &quot;_index&quot;: &quot;ii_example&quot;,<br>  &quot;_type&quot;: &quot;_doc&quot;,<br>  &quot;_id&quot;: &quot;1&quot;,<br>  &quot;_score&quot;: 0.5753642,<br>  &quot;_source&quot;: {<br>    &quot;content&quot;: &quot;The quick brown fox jumped over the lazy dog.&quot;<br>  }<br>},<br>{<br>  &quot;_index&quot;: &quot;ii_example&quot;,<br>  &quot;_type&quot;: &quot;_doc&quot;,<br>  &quot;_id&quot;: &quot;2&quot;,<br>  &quot;_score&quot;: 0.2876821,<br>  &quot;_source&quot;: {<br>    &quot;content&quot;: &quot;Quick brown foxes leap over lazy dogs in summer.&quot;<br>  }<br>}</pre><blockquote>Queries for term level: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.x/term-level-queries.html">Term level queries</a></blockquote><h3>Analysis</h3><p>Analysis is the process of converting <strong><em>text</em></strong> into <strong><em>terms</em></strong> which are added to the <strong>inverted index</strong> for searching.</p><p>Analysis is performed by an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html"><em>analyzer</em></a> which can be either a built-in analyzer or a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html">custom</a> analyzer defined per index. In addition, <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html"><em>analyzer</em></a> is just a package which contains <strong>3 lower-level building blocks</strong>: character filters, tokenizers, token filter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*jmRx433TPVJtLW3jSx8sgA.png" /><figcaption><em>Analyzer Pipeline</em></figcaption></figure><h4>Character filters (Optional)</h4><p>A <em>character filter</em> receives the original text as a stream of characters and can transform the stream by adding, removing, or changing characters. For instance, a character filter could be used to convert mandarin numerals (零一二三‎) into their Arabic-Latin equivalents (0123), or to strip HTML elements like &lt;b&gt; from the stream.</p><p>Let’s use the html_strip char filter which is the build-in analyzer:</p><pre># Try it first!</pre><pre>GET _analyze<br>{<br>  &quot;tokenizer&quot;: &quot;whitespace&quot;,<br>  &quot;text&quot;: &quot;&lt;h1&gt;Quick&lt;/h1&gt; brown fox!&quot;<br>}</pre><pre># Then add the char_filter!</pre><pre>GET _analyze<br>{<br>  &quot;tokenizer&quot;: &quot;whitespace&quot;,<br>  &quot;char_filter&quot;:  [ &quot;html_strip&quot; ],<br>  &quot;text&quot;: &quot;&lt;h1&gt;Quick&lt;/h1&gt; brown fox!&quot;<br>}</pre><h4>Tokenizers (Must have exactly one)</h4><p>A <em>tokenizer</em> receives a stream of characters, breaks it up into individual <em>tokens</em> (usually individual words), and outputs a stream of <em>tokens(terms)</em>.</p><pre># whitespace tokenizer<br>&quot;Quick brown fox!&quot;   ===&gt;   [Quick, brown, fox!]</pre><p>Elasticsearch provides the analyze API for testing and viewing the terms which produced by analyzer. Therefore, let’s use the API to test the whitespace tokenizer:</p><pre>GET _analyze<br>{<br>  &quot;tokenizer&quot;: &quot;whitespace&quot;,<br>  &quot;text&quot;: &quot;Quick brown fox!&quot;<br>}</pre><p>And the response body will be like this:</p><pre>{<br>  &quot;tokens&quot;: [<br>    {<br>      &quot;token&quot;: &quot;Quick&quot;,<br>      &quot;start_offset&quot;: 0,<br>      &quot;end_offset&quot;: 5,<br>      &quot;type&quot;: &quot;word&quot;,<br>      &quot;position&quot;: 0<br>    },<br>    {<br>      &quot;token&quot;: &quot;brown&quot;,<br>      &quot;start_offset&quot;: 6,<br>      &quot;end_offset&quot;: 11,<br>      &quot;type&quot;: &quot;word&quot;,<br>      &quot;position&quot;: 1<br>    },<br>    {<br>      &quot;token&quot;: &quot;fox!&quot;,<br>      &quot;start_offset&quot;: 12,<br>      &quot;end_offset&quot;: 16,<br>      &quot;type&quot;: &quot;word&quot;,<br>      &quot;position&quot;: 2<br>    }<br>  ]<br>}</pre><p>As the result, the tokenizer is also responsible for recording the order or <em>position</em> of each term and the start and end <em>character offsets</em> of the original word which the term represents.</p><h4>Token filters (Optional)</h4><p>A <em>token filter</em> receives the token stream and may add, remove, or change tokens. For example, a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-lowercase-tokenfilter.html">lowercase</a> token filter converts all tokens to lowercase, a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stop-tokenfilter.html">stop</a> token filter removes common words (<em>stop words</em>) like the from the token stream, and a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-tokenfilter.html">synonym</a> token filter introduces synonyms into the token stream.</p><pre>GET _analyze<br>{<br>  &quot;tokenizer&quot;: &quot;whitespace&quot;,<br>  &quot;filter&quot;: [&quot;lowercase&quot;], <br>  &quot;text&quot;: &quot;The Quick brown fox!&quot;<br>}</pre><pre>GET _analyze<br>{<br>  &quot;tokenizer&quot;: &quot;whitespace&quot;,<br>  &quot;filter&quot;: [&quot;lowercase&quot;, &quot;stop&quot;], <br>  &quot;text&quot;: &quot;The Quick brown fox!&quot;<br>}</pre><p><a href="https://www.elastic.co/blog/found-text-analysis-part-1">All About Analyzers, Part One</a></p><h3>Let’s try some cool stuff!</h3><h4>Customize the analyzer</h4><p>Usually, when we create the new index, we also set up the analysis functions from settings, customize the analyzer and the important things, <strong>don’t forget setting up the properties’ analyzer</strong>.</p><pre>PUT my_cool_index<br>{<br>  &quot;settings&quot;: {<br>    &quot;analysis&quot;: {<br>      &quot;analyzer&quot;: {<br>        &quot;cool_analyzer&quot;: { <br>          &quot;type&quot;: &quot;custom&quot;,<br>          &quot;tokenizer&quot;: &quot;standard&quot;,<br>          &quot;filter&quot;: [<br>            &quot;lowercase&quot;,<br>            &quot;asciifolding&quot;<br>          ]<br>        }<br>      }<br>    }<br>  }<br>}</pre><pre>PUT my_cool_index<br>{<br>  &quot;mappings&quot;: {<br>    &quot;_doc&quot;: {<br>      &quot;properties&quot;: {<br>        &quot;my_field&quot;: {<br>          &quot;type&quot;: &quot;text&quot;,<br>          &quot;analyzer&quot;: &quot;cool_analyzer&quot; <br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Also, it can be referred to when running the analyze API on a specific index:</p><pre>GET my_cool_index/_analyze <br>{<br>  &quot;analyzer&quot;: &quot;std_folded&quot;, <br>  &quot;text&quot;:     &quot;Is this déjà vu?&quot;<br>}</pre><pre>GET my_cool_index/_analyze <br>{<br>  &quot;field&quot;: &quot;my_text&quot;, <br>  &quot;text&quot;:  &quot;Is this déjà vu?&quot;<br>}</pre><h4>Customize the synonym token filter</h4><p>Synonyms can replace existing tokens or be added to the token stream. Before using the synonym token filter, we also need to set the synonyms at the analyzer.</p><pre>PUT my_synonym_index<br>{<br>  &quot;settings&quot;: {<br>    &quot;analysis&quot;: {<br>      &quot;analyzer&quot;: {<br>        &quot;synonym_analyzer&quot;: {<br>          &quot;tokenizer&quot;: &quot;standard&quot;,<br>          &quot;filter&quot;: [<br>            &quot;lowercase&quot;,<br>            &quot;my_synonym_filter&quot;<br>          ]<br>        }<br>      },<br>      &quot;filter&quot;: {<br>        &quot;my_synonym_filter&quot;: {<br>          &quot;type&quot;: &quot;synonym&quot;,<br>          &quot;synonyms&quot;: [<br>            &quot;british, english&quot;,<br>            &quot;queen, monarch&quot;<br>          ]<br>        }<br>      }<br>    }<br>  }<br>}</pre><pre>PUT my_synonym_index/_doc/_mappings<br>{<br>  &quot;properties&quot;: {<br>    &quot;my_field&quot;: {<br>      &quot;type&quot;: &quot;text&quot;,<br>      &quot;analyzer&quot;: &quot;synonym_analyzer&quot;<br>    }<br>  }<br>}</pre><p>Testing our analyzer with the analyze API shows the following:</p><pre>GET my_synonym_index/_analyze<br>{<br>  &quot;analyzer&quot; : &quot;synonym_analyzer&quot;,<br>  &quot;text&quot; : &quot;Elizabeth is the English queen&quot;<br>}</pre><p>Also, let’s test our synonym_analyzer by query:</p><pre>PUT my_synonym_index/_doc/1<br>{<br>  &quot;my_field&quot;: &quot;Elizabeth is the English queen&quot;<br>}</pre><pre>GET my_synonym_index/_doc/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;my_field&quot;: &quot;british&quot;<br>    }<br>  }<br>}</pre><h3>Mandarin search</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yajd7hcBQQFkrooY." /><figcaption>Photo by <a href="https://unsplash.com/@andrew_haimerl?utm_source=medium&amp;utm_medium=referral">Andrew Haimerl</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>If your service provides the searching with mandarin, you might understand that is not easy to solve the problem which is the process of analysis. In this part, I select <a href="https://code.google.com/archive/p/ik-analyzer/">ik-analyzer</a> which is well-known <a href="https://github.com/medcl/elasticsearch-analysis-ik">plugin</a> and is well-integrated with Elasticsearch.</p><h4>Installation</h4><p>Following the previous topics, let’s reuse the same environment and execute docker exec into the container, install the plugin of ik-analyzer. After installing successfully, we restart the container.</p><pre><strong>docker</strong> exec -it handsonelasticsearch_elasticsearch_1 bash \<br><strong>./bin/elasticsearch-plugin</strong> install <a href="https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.1/elasticsearch-analysis-ik-6.2.1.zip">https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.2.1/elasticsearch-analysis-ik-6.2.1.zip</a></pre><h4>Play with IK-analyzer</h4><p>Before trying the analyzer, let’s test the standard analyzer with analyze API for mandarin first.</p><pre>GET _analyze <br>{<br>  &quot;analyzer&quot;: &quot;standard&quot;, <br>  &quot;text&quot;:  &quot;本日的Elasticsearch教程為(如何最佳化您的搜索。)&quot;<br>}</pre><p>The elasticsearch-analysis-ik provides 2 analyzers: ik_smart and ik_max_word. Let’s try:</p><pre>GET _analyze <br>{<br>  &quot;analyzer&quot;: &quot;ik_smart&quot;, <br>  &quot;text&quot;:  &quot;本日的Elasticsearch教程為(如何最佳化您的搜索。)&quot;<br>}</pre><p>and</p><pre>GET _analyze <br>{<br>  &quot;analyzer&quot;: &quot;ik_max_word&quot;, <br>  &quot;text&quot;:  &quot;本日的Elasticsearch教程為(如何最佳化您的搜索。)&quot;<br>}</pre><h4>Add the synonym token filter</h4><p>Let’s try some different configuration of synonym token filter. This time, we create a file name synonyms.txt and put it into the path of config/analysis. Follow with the previous installation with docker, I have made the config folder map to the container’s folder. For the reason, we can edit the synonyms.txt outside.</p><pre># synonyms.txt<br>ipod, i-pod, i pod =&gt; ipod</pre><pre>番茄, 西紅柿<br>台大 =&gt; 台灣大學</pre><p>Then, we create an index with setting up the analyzer:</p><pre>PUT mandarin<br>{<br>  &quot;settings&quot;: {<br>    &quot;index&quot;: {<br>      &quot;analysis&quot;: {<br>        &quot;analyzer&quot;: {<br>          &quot;mandarin_analyzer&quot;: {<br>            &quot;tokenizer&quot;: &quot;ik_smart&quot;,<br>            &quot;filter&quot;: [<br>              &quot;synonym&quot;<br>            ]<br>          }<br>        },<br>        &quot;filter&quot;: {<br>          &quot;synonym&quot;: {<br>            &quot;type&quot;: &quot;synonym&quot;,<br>            &quot;synonyms_path&quot;: &quot;analysis/synonyms.txt&quot;<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><pre>PUT mandarin/_doc/_mappings<br>{<br>  &quot;properties&quot;: {<br>    &quot;my_field&quot;: {<br>      &quot;type&quot;: &quot;text&quot;,<br>      &quot;analyzer&quot;: &quot;mandarin_analyzer&quot;<br>    }<br>  }<br>}</pre><p>After setting up the analyzer, we can try:</p><pre>GET mandarin/_analyze<br>{<br>  &quot;analyzer&quot;: &quot;mandarin_analyzer&quot;,<br>  &quot;text&quot;: &quot;番茄&quot;<br>}</pre><h3>Conclusion</h3><p>In this topic, we understood the inverted index which is the important part of Elasticsearch. Then we knew what the term is, and how Elasticsearch recognizes the terms. Finally, we played the built-in analyzers, created several custom_analyzers and install the plugins to fulfill our requirements. To make a deeper understanding of analyzers and plugins, I provide some useful tools and documents for your reference below.</p><ul><li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/analysis.html">Analysis | Elasticsearch Reference [6.2] | Elastic</a></li><li><a href="https://github.com/medcl/elasticsearch-analysis-ik">medcl/elasticsearch-analysis-ik</a></li></ul><p>Also, You can find the full source code of my Github repo. Thanks for reading, and hopefully this tutorial is helpful for you! :D</p><p><a href="https://github.com/yingray/hands-on-elasticsearch">yingray/hands-on-elasticsearch</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e91f46be8ad6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/elasticsearch-optimizing-your-search-2-e91f46be8ad6">Elasticsearch: Optimizing your search(2)</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Elasticsearch: Hands-on with searching(1)]]></title>
            <link>https://medium.com/htc-research-engineering-blog/elasticsearch-hands-on-with-searching-1-57a3b8a8b1f6?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/57a3b8a8b1f6</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[search-engines]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Mon, 26 Feb 2018 09:16:40 GMT</pubDate>
            <atom:updated>2018-03-02T08:44:34.700Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iBqrExD-V6XWsXXR." /></figure><p>Learning how to search by Elasticsearch always is a big lesson. Therefore, I will share and distinguish between several interesting topics. In this topic, I’d like to share some experiences for searching using Elasticsearch.</p><h3>Installation</h3><p>Before we understand Elasticsearch, it is very troublesome when we set up the environment. Thence, I propose a simple solution using docker-compose to help us set up the environment conveniently and quickly. There is the docker-compose config for setting up the Elasticsearch-Kibana below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/699840bd5e496c22c7dbaf69ab7c1fd9/href">https://medium.com/media/699840bd5e496c22c7dbaf69ab7c1fd9/href</a></iframe><p>And then start it up:</p><pre><strong>docker-compose</strong> up -d &amp;&amp; <strong>docker-compose</strong> logs -f</pre><p>Test it out by opening another terminal window and running the following:</p><pre><strong>curl</strong> &#39;http://localhost:9200&#39;</pre><h3>Communication</h3><p>How you talk to Elasticsearch depends on whether you are using Java. Elasticsearch provides 2 ways: <strong>Java API</strong>, <strong>RESTful API</strong>. Java API comes with two built-in clients that talk to the cluster over port 9300, using the native Elasticsearch transport protocol, but it allows only Java to access. Therefore, on this topic, we only talk about RESTful API which can be communicated to all others languages over port 9200.</p><h4>RESTful API</h4><p>A request to Elasticsearch consists of the same parts as any HTTP request:</p><pre><strong>curl</strong> -X&lt;VERB&gt; &#39;&lt;PROTOCOL&gt;://&lt;HOST&gt;:&lt;PORT&gt;/&lt;PATH&gt;?&lt;QUERY_STRING&gt;&#39;\<br>     -d       &#39;&lt;BODY&gt;&#39;</pre><p>We will use HTTP expression to make it concise and readable.</p><pre>&lt;VERB&gt; /&lt;PATH&gt;?&lt;QUERY_STRING&gt;<br>&lt;BODY&gt;</pre><p><strong>Kibana</strong> comes with development tools named <strong>“Console(Sense)”</strong> to make us use HTTP expression easily.</p><ul><li>Enter to <a href="http://localhost:5601/app/kibana#/dev_tools/console">http://localhost:5601/app/kibana#/dev_tools/console</a></li><li>Copy and paste the target expression.</li><li>Run the expression(command+return) to dispatch the HTTP request directly.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*QZHBW9KXI-R8bU2eHL9mHw.png" /></figure><h3>Build a Catalog</h3><p>Before we start up the tutorial, we need to build a catalog for playing the Elasticsearch. First, we create a document with a single command:</p><pre>PUT /megacorp/employee/1<br>{<br>    &quot;first_name&quot; : &quot;John&quot;,<br>    &quot;last_name&quot; :  &quot;Smith&quot;,<br>    &quot;age&quot; :        25,<br>    &quot;about&quot; :      &quot;I love to go rock climbing&quot;,<br>    &quot;interests&quot;: [ &quot;sports&quot;, &quot;music&quot; ]<br>}</pre><p>Notice that the request path /megacorp/employee/1 contains 3 pieces of information:</p><ul><li>megacorp is the <strong>Index</strong> name</li><li>employee is the <strong>Type</strong> name</li><li>1 is the <strong>ID</strong> of this particular employee</li></ul><p>The request body contains all the information about this employee.</p><p>After we dispatch this request, we will get the response like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*KhCCuyClu6WLDIyUnXvHKw.png" /></figure><p>Let’s add a few more employees to the directory before moving on:</p><pre>PUT /megacorp/employee/2<br>{<br>    &quot;first_name&quot; :  &quot;Jane&quot;,<br>    &quot;last_name&quot; :   &quot;Smith&quot;,<br>    &quot;age&quot; :         32,<br>    &quot;about&quot; :       &quot;I like to collect rock albums&quot;,<br>    &quot;interests&quot;:  [ &quot;music&quot; ]<br>}<br><br>PUT /megacorp/employee/3<br>{<br>    &quot;first_name&quot; :  &quot;Douglas&quot;,<br>    &quot;last_name&quot; :   &quot;Fir&quot;,<br>    &quot;age&quot; :         35,<br>    &quot;about&quot;:        &quot;I like to build cabinets&quot;,<br>    &quot;interests&quot;:  [ &quot;forestry&quot; ]<br>}</pre><h3>Search the document</h3><p>Let’s talk about how to search and read the document. If you familiar with RESTful API, you’ll subconsciously use the GET method request like this:</p><pre>GET /megacorp/employee/1</pre><p>That’s right, after you request, you will get the John Smith’s information from the _source field of the response.</p><p>And then, let’s try using _search to put into the id part of the request path. Then we get all the employees of the megacorp index.</p><pre>GET /megacorp/employee/_search</pre><p>It also support query-string searching to let you put the parameters behind the request path:</p><pre>GET /megacorp/employee/_search?q=last_name:Smith</pre><p>Query-string searching is the fairly easy method to search, but it has its limitations. Elasticsearch provides a rich, flexible, query language called the <em>Query-DSL</em>, which allows us to build much more complicated, robust queries. <em>It </em>will make the search like this:</p><pre>GET /megacorp/employee/_search<br>{<br>    &quot;query&quot; : {<br>        &quot;match&quot; : {<br>            &quot;last_name&quot; : &quot;Smith&quot;<br>        }<br>    }<br>}</pre><p>Let’s see the request body, we will find the field named match which is one of several types of queries.</p><p>Finally, we try more complicated searching to query Smith who is older than 30:</p><pre>GET /megacorp/employee/_search<br>{<br>    &quot;query&quot; : {<br>        &quot;bool&quot; : {<br>            &quot;must&quot; : {<br>                &quot;match&quot; : {<br>                    &quot;last_name&quot; : &quot;smith&quot;<br>                }<br>            },<br>            &quot;filter&quot; : {<br>                &quot;range&quot; : {<br>                    &quot;age&quot; : { &quot;gt&quot; : 30 }<br>                }<br>            }<br>        }<br>    }<br>}</pre><blockquote>Furthermore, it’s not the common thing that put the request body in GET method request. It also works with POST method request. In order to keep with the tutorials intuition and reference to official documents, I will keep using GET method when I do the Elasticsearch searching.</blockquote><h4>Full-Text Queries</h4><p>Elasticsearch is built-on Lucene, so that makes it easy for us to achieve full-text search. Let’s use the same match query as before to search the about field for “rock climbing”.</p><pre>GET /megacorp/employee/_search<br>{<br>    &quot;query&quot; : {<br>        &quot;match&quot; : {<br>            &quot;about&quot; : &quot;rock climbing&quot;<br>        }<br>    }<br>}</pre><p>Then, we will get back 2 matching documents which have the string of “rock” or “climbing” in the about field.</p><p>If you want to query that contain both “rock” and “climbing”, we should add the operator and to the query body:</p><pre>GET /megacorp/employee/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;about&quot;: {<br>        &quot;query&quot;: &quot;climbing rock&quot;,<br>        &quot;operator&quot;: &quot;and&quot;<br>      }<br>    }<br>  }<br>}</pre><p>If you want to query the text exactly matching “rock climbing”, we should use a slight variation of the match query called the match_phrase query:</p><pre>GET /megacorp/employee/_search <br>{     <br>    &quot;query&quot; : {<br>         &quot;match_phrase&quot; : {<br>             &quot;about&quot; : &quot;rock climbing&quot;<br>         }<br>     }<br>}</pre><pre>GET /megacorp/employee/_search <br>{     <br>    &quot;query&quot; : {<br>         &quot;match_phrase&quot; : {<br>             &quot;about&quot; : &quot;climbing rock&quot;<br>         }<br>     }<br>}</pre><p>Then, we will get back 1 employee who really loves to go rock climbing not collect rock albums.</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html">Full text queries | Elasticsearch Guide [8.14] | Elastic</a></p><h4>Geo Queries</h4><p>Elasticsearch has a convenience ability which supports searching data in geography. In order to make this example more specific, let’s go to the Google Map to get the actual data. Refer to the picture below, we can find the longitude and latitude parameters of 7–11 in the real world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*BaQ5k4sDK9RpUNUHamTOAQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*-mvCzsQPQWfri3f3j1UE-w.png" /></figure><p>Before we start to make Geo-Queries, we should define the type of geo_point or geo_shape. In the example, I choice geo_point for demonstration.</p><pre>PUT /my_locations<br>{<br>  &quot;mappings&quot;: {<br>    &quot;_doc&quot;: {<br>      &quot;properties&quot;: {<br>        &quot;pin&quot;: {<br>          &quot;properties&quot;: {<br>            &quot;location&quot;: {<br>              &quot;type&quot;: &quot;geo_point&quot;<br>            }<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p>Then, let’s create the documents. And check the documents by _search.</p><pre>PUT /my_locations/_doc/1<br>{<br>  &quot;pin&quot;: {<br>    &quot;name&quot;: &quot;7-Eleven&quot;,<br>    &quot;location&quot;: {<br>      &quot;lat&quot;: 24.9840705,<br>      &quot;lon&quot;: 121.5399874<br>    }<br>  }<br>}</pre><pre>PUT /my_locations/_doc/2<br>{<br>  &quot;pin&quot;: {<br>    &quot;name&quot;: &quot;Mos burger&quot;,<br>    &quot;location&quot;: {<br>      &quot;lat&quot;: 24.9827922,<br>      &quot;lon&quot;: 121.5399175<br>    }<br>  }<br>}</pre><pre>GET /my_locations/_search</pre><p>I provide a pin location which is the location of our company. And then, let’s use the Geo-Queries to find the stores which are near to our company with only 20 meters.</p><pre>GET /my_locations/_search<br>{<br>  &quot;query&quot;: {<br>    &quot;bool&quot;: {<br>      &quot;must&quot;: {<br>        &quot;match_all&quot;: {}<br>      },<br>      &quot;filter&quot;: {<br>        &quot;geo_distance&quot;: {<br>          &quot;distance&quot;: &quot;20m&quot;,<br>          &quot;pin.location&quot;: {<br>            &quot;lat&quot;: 24.9841894,<br>            &quot;lon&quot;: 121.539891<br>          }<br>        }<br>      }<br>    }<br>  }<br>}</pre><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-queries.html">Geo queries | Elasticsearch Reference [7.2] | Elastic</a></p><h4>Aggregations</h4><p>Elasticsearch has a functionality to let you query and analyze the data. For example, let’s find the user who named “Smith”, and then calculate the average of age. In this case, we should use “Aggregations”.</p><pre>GET /megacorp/employee/_search<br>{<br>  &quot;size&quot;: 0,<br>  &quot;query&quot;: {<br>    &quot;match&quot;: {<br>      &quot;last_name&quot;: &quot;Smith&quot;<br>    }<br>  },<br>  &quot;aggs&quot;: {<br>    &quot;average_of_age&quot;: {<br>      &quot;avg&quot;: {<br>        &quot;field&quot;: &quot;age&quot;<br>      }<br>    }<br>  }<br>}</pre><p>Aggregations provide the ability which is similar to GROUP BY in SQL, group and extract statistics from your data. For example, let’s find the most popular interests enjoyed by our employees:</p><pre>GET /megacorp/employee/_search<br>{<br>  &quot;size&quot;: 0, <br>  &quot;aggs&quot;: {<br>    &quot;all_interests&quot;: {<br>      &quot;terms&quot;: {<br>        &quot;field&quot;: &quot;interests.keyword&quot;<br>      }<br>    }<br>  }<br>}</pre><p>In SQL, the above aggregation is similar in concept to:</p><pre># database = megacorp<br><strong>SELECT</strong> interests, COUNT(*) <br><strong>FROM</strong> employee <br><strong>GROUP BY</strong> interests <br><strong>ORDER BY</strong> COUNT(*) DESC</pre><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations.html">Aggregations</a></p><h3>Conclusion</h3><p>In this topic, we understood the Elasticsearch from going hands-on search part. However, these are very basic functions. When we face some complicated queries, we still need to figure out the solutions from the documentation of official website. There is the reference from official website below which is very easy to read.</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">Quick start | Elasticsearch Guide [8.1] | Elastic</a></p><p>Also, You can find the full source code of my Github repo. Thanks for reading, and hopefully this tutorial is helpful for you! :D</p><p><a href="https://github.com/yingray/hands-on-elasticsearch">yingray/hands-on-elasticsearch</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57a3b8a8b1f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/elasticsearch-hands-on-with-searching-1-57a3b8a8b1f6">Elasticsearch: Hands-on with searching(1)</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Elasticsearch]]></title>
            <link>https://medium.com/htc-research-engineering-blog/overview-of-elasticsearch-c388220f3e98?source=rss-ea675b0d7c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c388220f3e98</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[analysis]]></category>
            <category><![CDATA[search]]></category>
            <dc:creator><![CDATA[Ying Ray Lu]]></dc:creator>
            <pubDate>Wed, 21 Feb 2018 09:41:10 GMT</pubDate>
            <atom:updated>2018-03-02T05:37:58.894Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3PDCW7g-rE1vIDDEyapnHQ.jpeg" /></figure><p>When it comes to databases, we usually think of <a href="https://www.mysql.com/">MySQL</a>, <a href="https://www.postgresql.org/">PostgreSQL</a>, and the distributed DBMS such as <a href="https://www.mongodb.com/">MongoDB</a>. There is the figure below which refers to the <a href="https://db-engines.com/en/ranking">DB-Engines Ranking</a> and reflects the popularity of major databases in recent years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*Wx7cA9OWPFazaCCuF72fTA.png" /><figcaption>DB-Engines Ranking Table (Feb, 2018)</figcaption></figure><p>Today, we will talk about the search engine which reached the top ten databases in this year’s breakthrough - <a href="https://www.elastic.co/"><strong>Elasticsearch</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*RrKMZXY9pRvdv4phlOo4Mw.png" /><figcaption>DB-Engines Ranking Chart (Feb, 2018)</figcaption></figure><h3>Advantage</h3><h4>Elasticsearch is a <strong>distributed</strong> store, search and analytics engine capable of solving a growing number of use cases.</h4><ol><li>ES provides the cluster which is a collection of one or more nodes (servers) to hold your entire data and provide federated indexing and search capabilities across all nodes. Therefore, it can easily achieve scaling out your search volume.</li><li>ES provides the ability to subdivide your index into multiple pieces called shards. When you create an index, you can simply define the number of shards that you want. Each shard is in itself a fully-functional and independent “index” that can be hosted on any node in the cluster.</li><li>ES is a near real time search platform. What this means is there is a slight latency (normally one second) from the time you index a document until the time it becomes searchable.</li></ol><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.x/_basic_concepts.html">Basic Concepts | Elasticsearch Reference [6.x] | Elastic</a></p><h4>Using RESTful for communication, familiar to mainstream developers, intuitive interface, most languages are supported.</h4><h4>The core of Elasticsearch is built on top of Apache Lucene, is a well-known, open source, full-text search-engine library. However, Elasticsearch is much more than just Lucene and much more than “just” full-text search. It can also be described as follows:</h4><ol><li>A distributed real-time document store where <em>every field</em> is indexed and searchable.</li><li>A distributed search engine with real-time analytics.</li><li>Capable of scaling to hundreds of servers and petabytes of structured and unstructured data</li></ol><h3>Weak points</h3><ul><li>Not support scheduling.</li><li>Have a slight latency in indexing.</li><li>Not support permission management.</li></ul><h3>Use cases</h3><ul><li>Wikipedia uses Elasticsearch to provide<strong> full-text search</strong> with highlighted search snippets, and<strong> <em>search-as-you-type</em></strong> and <strong><em>did-you-mean</em></strong> suggestions.</li><li><em>The Guardian</em> uses Elasticsearch to combine <strong>visitor logs</strong> with <strong>social -network data</strong> to provide real-time feedback to its editors about the public’s response to new articles.</li><li>Stack Overflow combines <strong>full-text search</strong> with <strong>geolocation queries</strong> and uses <strong><em>more-like-this</em></strong> to find related questions and answers.</li><li>GitHub uses Elasticsearch to query 130 billion lines of code.</li><li>SoundCloud: Use Elasticsearch to intelligently search and deliver content to tens of millions.</li><li>Baidu: Baidu currently uses ES for analysis. It collects all kinds of index data and user-defined data on all servers in Baidu. Through multi-dimensional analysis and display of various data, it assists in positioning an exception or business exception. At present, it covers more than 20 business lines within Baidu, up to 100 machines in a single cluster and 200 ES nodes, importing 30TB+ daily data .</li></ul><h3>Comparison</h3><pre>+--------+----------+------------+<br>|   ES   |   SQL    |  MongoDB   |<br>+--------+----------+------------+<br>| Index  | Database | Database   |<br>| Shard  | Shard    | Shard      |<br>| Type   | Table    | Collection |<br>| Field  | Column   | Field      |<br>| Object | Record   | Record     |<br>+--------+----------+------------+</pre><h3>Conclusion</h3><p>In the era of big data, it often encounters billions of data and real-time search, so that we have NoSQL. However, it’s still not enough in real world, you may be find the “needle in the haystack“ for billions of data. When full-text search, structured search, and analysis become the necessary requirements, this will be the best condition to use Elasticsearch. It saves on the learning costs of understanding Lucene’s complex operating principles and communicates with the RESTful API behind the open-source community and experience. I believe Elasticsearch will be keeping in the top 10 popular database for a while.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c388220f3e98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/htc-research-engineering-blog/overview-of-elasticsearch-c388220f3e98">Overview of Elasticsearch</a> was originally published in <a href="https://medium.com/htc-research-engineering-blog">DeepQ Research Engineering Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>