<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abdul Rahman Babil on Medium]]></title>
        <description><![CDATA[Stories by Abdul Rahman Babil on Medium]]></description>
        <link>https://medium.com/@abdulrahmanbabil?source=rss-fe90c759a21e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*owm5YBAvcWT-VtF2dUhqoQ.png</url>
            <title>Stories by Abdul Rahman Babil on Medium</title>
            <link>https://medium.com/@abdulrahmanbabil?source=rss-fe90c759a21e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:40:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abdulrahmanbabil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to apply GitOps CD with GKE and Cloud-build in 5 minutes!]]></title>
            <link>https://medium.com/google-cloud/how-to-apply-gitops-cd-with-gke-and-cloud-build-in-5-minutes-58455a2bf063?source=rss-fe90c759a21e------2</link>
            <guid isPermaLink="false">https://medium.com/p/58455a2bf063</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[google-cloud-build]]></category>
            <category><![CDATA[google-kubernetes-engine]]></category>
            <dc:creator><![CDATA[Abdul Rahman Babil]]></dc:creator>
            <pubDate>Sun, 04 Apr 2021 07:15:04 GMT</pubDate>
            <atom:updated>2021-04-06T06:45:55.499Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*RLcGyanmtkde8jU2KA0gnA.jpeg" /></figure><p>Effective engineers will always prefer high leverage work to get done first, high leverage work is a kind of time investment, it’s to spend some time and efforts to build something that will save a lot of time in the future or prevent many human errors, one of these good investments is CI/CD pipelines.</p><p>CI/CD pipelines have never been a need only for big teams, even a single developer will need it, whatever the project it’s, however, it could have a few or many steps based on team size and needs.</p><h3>Kubernetes and Cloud-native applications</h3><p>I’m gonna assume that your application is a Cloud-native application built on top of microservices, each microservice is built by a different programming language and it’s communicating with other microservices in an organized way.</p><p>When we talk about Cloud-native applications and microservices probably means we use containers and containers orchestration (Kubernetes), I won’t talk about all benefits of Kubernetes in this article, but I would mention the greatness of IAC (Infrastructure as a code) when there is a code describes the current state of your infrastructure, so you can move it easily or rebuild again in seconds/minutes.</p><h3>What is GitOps?</h3><blockquote>GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools. — <a href="https://www.gitops.tech/">https://www.gitops.tech/</a></blockquote><p>The main idea is to have a Git Repo (Environment Repo) that hosts the current state of infrastructure (Kubernetes’ manifests), then changing application on the cloud would be coming from changes done on the Environment repo after it got sync with Kubernetes cluster.</p><h3>Pros of GitOps</h3><ul><li>Deploy faster on the cloud, no need to configure your local machine, useful if you manage multiple environments</li><li>Since it’s Git-based, only revert the last commit to recovery!</li><li>Safe, no need to store credentials for infrastructure on local machines</li><li>Use Git permissions to know who can read and write</li><li>Use Git history to know what happened and when it happened</li></ul><h3>Methods of GitOps</h3><p>After preparing the Environment repo and putting all required manifests to run the applications on Kubernetes we have to sync these manifests with the Kubernetes cluster, which means any changes that happen on the Environment repo should get reflect on the current running infrastructure.</p><ul><li>Pushed-based updates: changes on the Environment repo will trigger a job to update the current Kubernetes state to follow the new one in the Environment repo.</li><li>Pull-based updates: Kubernetes cluster will run a job periodically to check/observe the current state described in the Environment repo and update itself if needed.</li></ul><h3>GitOps example</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Iw-m54OybVsmLF3gStVNw.png" /><figcaption>Source: <a href="https://www.gitops.tech/">https://www.gitops.tech/</a></figcaption></figure><p>My simple example would be about having a web application repo and the environment repo, I use <a href="https://cloud.google.com/kubernetes-engine">GKE</a> to manage my Kubernetes cluster, we need two <a href="https://cloud.google.com/build">Cloud build</a> triggers here:</p><ul><li>Application repo trigger: which is gonna build a new docker image, push it to <a href="https://cloud.google.com/container-registry">container registry</a> / <a href="https://hub.docker.com/">Docker hub</a>, last but not least, it will update the environment repo to use the last web application image instead of the old one.</li><li>The environment repo trigger: which is gonna run when there is a new commit to apply the latest changes on the Kubernetes cluster.</li></ul><p>The environment repo will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ysmrs08B_jwf3z1bcu7lfQ.png" /><figcaption>The environment repo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zsC4jS57nKtSxYWdDRUfkg.png" /><figcaption>Demo app deployment, the image here considered as a variable</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*krOZgVncDJNymevQiJIo1g.png" /><figcaption>the kustomization file to create Kubernetes objects will replace the image variable with a specific build tag</figcaption></figure><p>I’m using <a href="https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/">kustomize</a>, it will make things easier in changing parts of Kubernetes objects, for example, if you have many cronjobs and all of them using the same image and you need to use a newer one, you’ve to update many YAML files and apply all of them, but if you’re using kustomize, you update and apply one file and that will update all cronjobs! very nice!.</p><p>So here are the required changes to be implemented in Application Repo (Demo API)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/472d6358dc9d7ad2b80c393d9023a2d5/href">https://medium.com/media/472d6358dc9d7ad2b80c393d9023a2d5/href</a></iframe><p>This cloudbuild.yaml should be added to the top level of the Application repo.</p><p>ENV_REPO_FILES: These are the paths of kustomization files you want to update in the environment repo.</p><p>I used a custom step omegaes/kubci as POC how to build a custom step for Cloud-build, this step will do three tasks:</p><ul><li>Clone the environment repo, checkout target branch to do required changes</li><li>Update Kustomiztion file, change the image tag of the application’s deployment</li><li>Commit these changes and merge them with the main branch in the Environment repo.</li></ul><blockquote>* It’s better to break the build tasks into multiple steps.</blockquote><blockquote>* It’s better to let app pipeline to create a target branch from main branch in the Environment repo, another trigger on target branch will check if it’s possible to apply these changes, if it’s okay it will apply and merge to main branch on the Environment repo.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64ce07bc965fa03d015cd3921f33e433/href">https://medium.com/media/64ce07bc965fa03d015cd3921f33e433/href</a></iframe><p>This cloudbuild.yaml should be added to the top level of your environment repo.</p><p>Finally, create two Cloud-build triggers and the secret “Environment-repo-auth-URL”:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22b7e0f11a79c93afc19fb87f745da58/href">https://medium.com/media/22b7e0f11a79c93afc19fb87f745da58/href</a></iframe><p>“Environment-repo-auth-URL” will be used in the custom step previously to clone the Environment Repo and push a new image tag there.</p><p>Let’s make a small change on Application Repo:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*F132tPBKeYmMDDE7CXRtzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7pzWb-fxnS_0kEaMblMS-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*6hRuT-uhv1dP--_v8nGE5A.png" /><figcaption>application repo trigger, update infrastructure code’s commit and the environment repo trigger,</figcaption></figure><h3>Conclusion</h3><p>What we’ve gone through was only a simple example for GitOps, in real-world projects you will find many struggles, this approach was good for me in controlling multiple Kubernetes clusters from the Environment Git repository, please don’t hesitate to ask for any clarification or to share your thoughts!!</p><p>You can check out these Github repositories to see how I build the demo here:</p><ul><li><a href="https://github.com/omegaes/GKE-Demo-App">GKE Demo App</a> (Hello World in node.js)</li><li><a href="https://github.com/omegaes/GKE-Demo-Environment">GKE-Demo-Environment</a> (host Kubernetes objects manifests)</li><li><a href="https://github.com/omegaes/Kubci">Kubci</a> (A Docker image meant to used as a custom step in Cloud build, this will clone the environment repo, update kustomization yaml and push these changes back to the environment repo)</li></ul><p>Resources I was inspired by:</p><ul><li><a href="https://www.gitops.tech/">GitOps</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/gitops-cloud-build">GitOps-style continuous delivery with Cloud Build | Kubernetes Engine | Google Cloud</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58455a2bf063" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/how-to-apply-gitops-cd-with-gke-and-cloud-build-in-5-minutes-58455a2bf063">How to apply GitOps CD with GKE and Cloud-build in 5 minutes!</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stream Millions of events from your client to BigQuery in a Serverless way, Part #2]]></title>
            <link>https://medium.com/google-cloud/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-2-d2913ecee763?source=rss-fe90c759a21e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2913ecee763</guid>
            <category><![CDATA[google-pubsub]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cloud-monitoring]]></category>
            <category><![CDATA[cloud-logging]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Abdul Rahman Babil]]></dc:creator>
            <pubDate>Sun, 21 Mar 2021 07:44:17 GMT</pubDate>
            <atom:updated>2021-03-22T13:12:55.566Z</atom:updated>
            <content:encoded><![CDATA[<p>This article is the completion of another article:</p><p>Part #1: <a href="https://medium.com/@abdulrahmanbabil/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-1-a38c4f9cd6e4">Building an API to receive raw data from clients and push it to Pub/Sub</a></p><p><strong>Part #2: Running ETL Jobs to transfer and load events into BigQuery</strong></p><p>You can check the<a href="https://github.com/omegaes/serverless-streaming"> Github Repository</a> to run/view-code this example, full details to run this example is there, just one command!</p><h3>ETL Job deploy</h3><p>Events subscription could have 10 messages or 10 million messages, so the number of jobs will vary based on the number of outstanding messages!, there are many ways to load background jobs in GCP, you can use Cloud Function, App engine, GKE, Cloud Build, Cloud Dataflow, Compute Engine, and Cloud Run.</p><p><a href="https://cloud.google.com/dataflow">Cloud Dataflow</a> is a managed solution for ETL, but it will restrict you to use <a href="https://beam.apache.org/">Apache beam</a> which is available only within few programming languages, but my approach in this article will work with any programming language you prefer.</p><p>For this example I will go with <strong>Cloud Run, </strong>a job would be an API call that will run for &lt; 10 mins, during this period, it will pull messages as much as possible, do transfer for each message(validate, map, filter, … etc) put these transferred messages in a (CSV, JSON-LD) and call BigQuery API to append these events into a table.</p><p>We create a new<strong> Job Cloud Run</strong> service, you have to make sure this one is independent of Client API one, also to make sure only authenticated scripts can call it to do the job, <strong>the</strong> <strong>timeout for API call to be around 15 minutes</strong>, the job must finish its work within 10 mins because we gonna set Acknowledge-Deadline to 600 seconds, we will keep pulling messages from <strong>events subscription </strong>for 7~8 minutes, then we load these events to BigQuery, once it’s a success from BigQuery side, we acknowledge all processed messages in <strong>events subscription</strong> so it doesn’t get delivered again, later the job will return a 200 response to acknowledge the message coming from <strong>invoking subscription.</strong></p><h3>ETL Job flow</h3><p>Extract: clients’ events are now in events subscription, waiting to be transformed and to load into the final destination, ETL job will pull a batch of events while the job is running and subscription dose have more events.</p><p>Transform: the job will do a required transform on each event, like mapping, filtering, .. etc</p><p>Load: will send a big bunch of transformed events into BigQuery</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*El_UzUeaJLe9hm77e8SbwA.png" /></figure><h3>ETL Load to BigQuery</h3><blockquote>Question: How we can load transformed events data into BigQuery?</blockquote><p>BigQuery has two ways to load data, Streaming insert and Batch load, for this example I would recommend using a Batch load, is cheaper than Streaming insert and more scalable, you can read about loading data from <a href="https://cloud.google.com/bigquery/docs/loading-data">here</a>.</p><p>The main idea behind batch loading is to create a structured text file has data rows and create a job in BigQuery to load the data from the file into a BigQuery table.</p><p>Preparing events to load into BigQuery in batch loading could take few seconds, obviously, that will make API slower!, the better solution is to send these events from API to a queue service (Pub/Sub), then later we run jobs to process a big batch of events and load it into BigQuery.</p><h3><strong>ETL Job trigger</strong></h3><blockquote>Now, ETL job needs to get invoked from an HTTP request to start processing, how we can do that??</blockquote><p>1- Using Cloud schedule, send Pub/Sub invoke msg in a corn way, every 5 mins for example!</p><blockquote>Cons:</blockquote><blockquote>a- If subscription didn’t receive messages for hour, Job will run multiple times and do nothing hence subscription is empty.</blockquote><blockquote>b- If subscription got to many messages, job instance has a limited capacity to process in one run, that means messages will stuck in the subscription for long time because processing power won’t scale up when load scale up.</blockquote><p>2- Using Cloud monitoring, monitor the number of undelivered events in events subscriptions, and based on the number it will trigger X ETL jobs.</p><p>You can use both Solution #1 and Solution #2, #1 will have regular ETL every X minutes, #2 will add more ETL jobs if events in subscription grow in fast-paced, that will help to do parallel ETL processing at the same time.</p><p>Unfortunately, Incident can send only 1 message</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_R4K56rnHOD5Y6luvv82w.png" /><figcaption>Alert policy over events subscription</figcaption></figure><p>We add a policy on events subscription, the policy has many conditions with OR operator to trigger an incident, the incident will send a Pub/Sub message to <strong>invoke topic, invoke subscription </strong>will post an HTTP call to trigger ETL Job and wait until ETL job until it finishes processing, the incident will keep sending invoke messages while subscription is empty!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qDpPxEY1N6F0r-L_gL3tXg.png" /><figcaption>Number of messages sent to invoke topic</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMOVfSdmVILkxHCuxlVLrw.png" /><figcaption>Transformed events in BigQuery after ETL got called</figcaption></figure><h3>Conclusion</h3><p>This is an approach to handle BigData coming from clients, these data will be delayed for minutes before reaching BigQuery, you might face some challenges when your application receives a high number of events, but be sure there is always a solution around!.</p><p>I would like to hear your thoughts and experience in handling such scenarios, what kind of technologies you would use and how hard is to build?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2913ecee763" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-2-d2913ecee763">Stream Millions of events from your client to BigQuery in a Serverless way, Part #2</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stream Millions of events from your client to BigQuery in a Serverless way, Part #1]]></title>
            <link>https://medium.com/google-cloud/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-1-a38c4f9cd6e4?source=rss-fe90c759a21e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a38c4f9cd6e4</guid>
            <category><![CDATA[google-cloud-run]]></category>
            <category><![CDATA[cloud-monitoring]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[stackdriver]]></category>
            <dc:creator><![CDATA[Abdul Rahman Babil]]></dc:creator>
            <pubDate>Sun, 21 Mar 2021 07:43:17 GMT</pubDate>
            <atom:updated>2021-03-22T13:12:40.162Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the success keys for startups is <strong>DATA, </strong>collecting data never been a bad idea, even it might be costly in time and resources, but it’s that treasure that you will use to understand your product and move your platform to a new level of ML.</p><p>I’m gonna show you a way to handle millions of data events in a scalable way, using Serverless services provided from GCP and do have a free tier!!</p><p>This article would be split into 2 parts:</p><p><strong>Part #1: Building an API to receive raw data from clients and push it to Pub/Sub</strong></p><p>Part #2: <a href="https://medium.com/@abdulrahmanbabil/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-2-d2913ecee763">Running ETL Jobs to transfer and load events into BigQuery</a></p><p>You can find out the code for <strong>API</strong> and <strong>gcloud</strong> to create all required resources with more in-depth details in the <a href="https://github.com/omegaes/serverless-streaming">Github repository</a>.</p><h3>Events flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57Wowb6Ot7zi-LWPiPejCA.png" /></figure><h3>Producer API!</h3><p>in most cases, you have one endpoint to receive data from the client-side, maybe the client will send a batch of data to the endpoint, this endpoint should be reliable and available all the time with low latency and high capacity of scaling, for this example I will choose a simple way to build this API, I’m gonna use CloudRun from GCP.</p><p><a href="https://cloud.google.com/run"><strong>Cloud Run</strong></a> does support hosting your contained app in a Serverless way with scaling from 0 to N based on your traffic, it’s very easy to deploy and lets you focus on building your application.</p><p>I did use node.js with express to write a simple endpoint that can accept a JSON array of JSON events.</p><p>We expect millions of events data in a short period of time, these data could be a low priority for your app, so no need to store it in your primary DB, but you can store it in a warehouse/data leak, in this example I will use BigQuery as the final destination for these data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*Ga38B55jLSNWVYJVFz8lfA.png" /></figure><h3>Stackdriver and Pub/Sub</h3><blockquote>Question: How we can send events into Pub/Sub?</blockquote><p>Usually, by using Pub/Sub SDK, it provides a set of functions to push messages into a Pub/Sub topic, then the topic will deliver to its subscriptions, this way consider good but might have some cons, like network latency to make an HTTP request to Pub/Sub service for every API call, or higher cost when you send a small number of messages for every API call.</p><p>The other way is to use <a href="https://cloud.google.com/logging">Stackdriver</a> with <a href="https://cloud.google.com/logging/docs/export/configure_export_v2"><strong>Stackdriver sink</strong></a><strong>s</strong>, by default Stackdriver will collect logs from all GCP services, so any logs in stdout will be collected by Stackdriver, and you have the option to store these logs as text files in GCP or to send these logs to BigQuery or Pub/Sub.</p><p>So first we have to let the API writing events into stdout as a JSON format, for node.js I will recommend you to use “Bunyan”.</p><pre>const bunyan = require(&#39;bunyan&#39;);</pre><pre>const logger = bunyan.createLogger({</pre><pre>name: &#39;events-service&#39;,</pre><pre>streams: [{stream: process.stdout, level: &#39;info&#39;}]</pre><pre>});</pre><pre>app.post(&#39;/receive&#39;, (req, res) =&gt; {</pre><pre> req.body.forEach(event =&gt; {</pre><pre>    event.receive_timestamp = Date.now()</pre><pre>    logger.info(event);</pre><pre> });</pre><pre> res.status(200).send({done: true});</pre><pre>});</pre><p>After preparing the code + Dockerfile, we need to deploy it on<strong> Cloud Run</strong>, next step is to create a <strong>Stackdriver sink</strong>, the first sink will direct all logs to a Pub/Sub topic, you can create the sink from the GCP console or gcloud CLI.</p><blockquote>Better to stop/exclude these logs from Stackdriver _default sink , just to save cost!</blockquote><p>You can call your Producer API with some events, later you go to Cloud logging (Stackdriver), search for logs with this filter:</p><pre>jsonPayload.name=&quot;events-service&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JDEukwRlSFZ4VFXhj-01WQ.png" /></figure><p>These are events sent when Producer API got called, just landed in logs, within few seconds, messages would be pushed into the target topic waiting for consumer jobs to do the ETL work!.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OGgyaBGaGr5kk2GpjDEKGA.png" /></figure><p>Congratulations! your events are waiting in Pub/Sub just within few lines of code, no need to care about the amount of events clients sending, Pub/Sub and Cloud Run will handle it, bare with me to read part #2 so you can transform and load these events into BigQuery.</p><p>Feel free to share your thoughts in the comments below, If you have any questions or feedback don’t hesitate to ask so!</p><p>Continue reading Part #2: <a href="https://medium.com/@abdulrahmanbabil/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-2-d2913ecee763">Running ETL Jobs to transfer and load events into BigQuery</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a38c4f9cd6e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/stream-millions-of-events-from-your-client-to-bigquery-in-a-serverless-way-part-1-a38c4f9cd6e4">Stream Millions of events from your client to BigQuery in a Serverless way, Part #1</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recommend personalized content for your app users using BigQuery]]></title>
            <link>https://medium.com/google-cloud/recommend-personalised-content-for-your-app-users-using-bigquery-239b1398a46a?source=rss-fe90c759a21e------2</link>
            <guid isPermaLink="false">https://medium.com/p/239b1398a46a</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Abdul Rahman Babil]]></dc:creator>
            <pubDate>Sat, 09 Jan 2021 07:52:06 GMT</pubDate>
            <atom:updated>2021-03-25T18:15:06.663Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*Nx2xLFwMCnFniv75JQYCiw.jpeg" /></figure><p><strong>How many times you were surprised when your favorite app show you relevant content?</strong></p><p>Whether you’re using Youtube, Netflix, or any other app you will notice that recommendation system is the main player in their success, but having a recommendation system could be a bit heavy for your startup, adding Machine learning algorithms like collaborative filtering / content-based filtering, after that, you need to train a model based on your big data, at the end you will end up with a static model to predict content relevance to the user.</p><p>The previous approach is hard and expensive to apply, any model training will take hours, and that might make your recommendation engine useless by time!!, Model drift is an issue that will chase you all the time, users’ data keep changing in a very fast-paced way.</p><h3>Alternative solution</h3><p><strong>The solution I would propose here is related to News aggregators, It’s using explicit rules, unlike ML! you can change the rules to match your domain.</strong></p><p>News aggregators build a curated list of articles from different publishers/topics and let users able to follow publishers and read articles, smart news aggregators try to serve the user with relevant articles, based on the study case we noticed few rules to improve the user experience when it comes to serving articles:</p><p>The user is interested in articles from specific topics.</p><p>User prefers some publishers more than other publishers.</p><p>User prefers articles talking having some main keywords.</p><p>The recommender should not recommends articles that users have already been read.</p><p>This Recommender is simple as it’s a command that runs as a cronjob to aggregate recommended articles for users on an hourly basis, work will be done in offline mode to prepare recommended articles to be pushed/highlighted for users, this job can take few mins to execute even when you have thousands of articles and hundreds of thousands of users!</p><h3>BigQuery</h3><p><strong>Why I preferred to use BigQuery in the first place?</strong></p><p>BigQuery is a serverless service from GCP (Google Cloud Platform), very efficient and powerful for BigData work, it can be integrated with other GCP services in a very easy way, no need to do any sysops or hosting, and it has CLI and SDKs in many modern programming languages like PHP, Go, node.js … etc, easy to load data, have 2 different approaches for billing and many other reasons, just take a look over it from <a href="https://cloud.google.com/bigquery">here</a>.</p><p>Make sure you use BigQuery as a warehouse, you keep a copy there for long term storage, you probably need to build ETL pipelines to keep moving data from your DB/ES to BigQuery, GCP providing a tool to handle that for you <a href="https://cloud.google.com/data-fusion/">Google Data Fusion</a></p><h3>Main dish!</h3><p><strong>First, all the queries here and example data are available on </strong><a href="https://github.com/omegaes/Recommendation-Engine-BQ"><strong>Github</strong></a><strong>.</strong></p><p>I won’t discuss how you gonna use this query, for me I would use <a href="https://laravel.com/">Laravel</a>, it has a nice way to build commands using <a href="https://laravel.com/docs/8.x/artisan">Artisan Console</a>, you just need to add <a href="https://github.com/googleapis/google-cloud-php-bigquery">PHP BigQuery SDK</a> to your composer!</p><p>Here is the data we need to process, article table for candidate articles, user the table that has users, each article belongs to topic and publisher</p><p>user_article_feed keeps articles that the user has seen in the feed.</p><p>user_article_read keeps articles that the user has read before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*Wmt7ExRH1e94TfVkNAQScQ.png" /></figure><p>To explore the article table, let us run:</p><pre>select * from test_data.article</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CbessPH7xFQXw3UKn9dWKg.png" /></figure><p>In user table, we have 1000 users (id, name).</p><p>Based on historical user data, we can notice that user X could be interested in some topics/publishers based on views/shares/reactions/CTR .. etc, from these data we can generate a user-METRIC relevance score (0 -&gt; 1), and here were user_publisher and user_topic come, both are pre-calculated tables and have to be well maintained and updated frequently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*7E7bnzLMteFCF4yzL07SaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*NfcHipPmCCJP73sEHxvWmA.png" /></figure><p>In this example, we have 100 different articles and 1000 users, which means we have 100,000 different combinations, each user has 100 different articles he can read, but at the end, a single user needs top X relevant articles, BigQuery can offer the great and scalable experience to do that, just using Standard SQL!</p><p>I will show you the simple query that can achieve that:</p><pre>with article as <br>( <br>  select * from test_data.article<br>),<br>user as (<br>  select * from test_data.user<br>),<br>topic as (<br>  select * from test_data.topic<br>),<br>user_read as (<br>  select * from `test_data.user_article_read` <br>),<br>user_feed as (<br>  select * from `test_data.user_article_feed` <br>),<br>user_publisher as (<br>  select * from test_data.user_publisher<br>),<br>user_topic as (<br>  select * from test_data.user_topic<br>),<br>enhanced_articles as (<br>  select *, <br>  ROW_NUMBER() over (Partition BY article.topic_id ORDER BY score DESC ) as article_rank,<br>  max(score) over (Partition BY article.topic_id) as article_max_score,<br>  min(score) over (Partition BY article.topic_id) as article_min_score,<br>   1000 * (score / max(score) over (Partition BY article.topic_id)) as enhanced_scroll</pre><pre>from article<br>),<br>user_article as (<br>  select user.id as user_id, article.id as article_id, article.score  as original_score, article.publisher_id , article.topic_id, article.enhanced_scroll  <br>  from user cross join enhanced_articles as article<br>),<br>user_article_with_dimension_relevance as (<br>  select user_article.*,  <br>  user_publisher.relevance as user_publisher_relevance,<br>  user_topic.relevance as user_topic_relevance<br>  from user_article <br>  left join user_publisher on user_publisher.user_id = user_article.user_id  and user_article.publisher_id = user_publisher.publisher_id <br>  left join user_topic on user_topic.user_id = user_article.user_id and user_topic.topic_id  = user_article.topic_id <br>),<br>user_article_check_read_and_feed as (<br>  select distinct user_article_with_dimension_relevance.*, <br>  IF(user_read.article_id is null, 0, 1) as read,<br>  IF(user_feed.article_id is null, 0, 1) as feed<br>  from user_article_with_dimension_relevance<br>  left join user_feed on user_feed.user_id = user_article_with_dimension_relevance.user_id and user_feed.article_id  = user_article_with_dimension_relevance.article_id<br>  left join user_read on user_read.user_id = user_article_with_dimension_relevance.user_id and user_read.article_id  = user_article_with_dimension_relevance.article_id<br>),<br>user_article_filtered as (</pre><pre>select * from user_article_check_read_and_feed where read = 1 or feed = 1 or user_publisher_relevance = 0 or user_topic_relevance = 0<br>),<br>user_article_with_relevance_score as (<br>  select u.user_id, u.article_id, u.original_score, u.publisher_id, u.topic_id, ((user_publisher_relevance+user_topic_relevance)/2) as relevance_score, u.enhanced_scroll <br>  from user_article_filtered as u</pre><pre>),<br>user_article_with_calculated_score as (<br>   select *, (enhanced_scroll*relevance_score) user_article_relevance_score,<br>   ROW_NUMBER() over (Partition BY user_id ORDER BY (enhanced_scroll*relevance_score) DESC ) as user_article_rank,</pre><pre>from user_article_with_relevance_score<br>),<br>user_with_top_articles as (<br>   select user_id, <br>   to_json_string(ARRAY_AGG(<br>    STRUCT&lt;article_id INT64, original_score INT64, publisher_id INT64, topic_id INT64, relevance_score FLOAT64, enhanced_scroll FLOAT64, user_article_relevance_score FLOAT64, user_article_rank INT64&gt;(article_id,original_score,publisher_id, topic_id, relevance_score, enhanced_scroll, user_article_relevance_score, user_article_rank)<br>   ) ) data<br>   from user_article_with_calculated_score<br>   where user_article_rank &lt;= 5<br>   group by user_id</pre><pre>)<br> select * from user_with_top_articles</pre><p>So this query is considered as a pipeline of steps, or a recursive query, execution is bottom to up, but we will discuss it up to down.</p><p>I will skip the first few sub-queries because it’s too simple, enhance_articles meant to do common ground for articles score, because article’s score value might vary from different topics, so we make article’s score as a value from 0 -&gt; 1000.</p><p>user_article_with_dimension_relevance Will have all user-article combinations with article’s score, user article-topic score, and user article-publisher score.</p><p>user_article_check_read_and_feed in this step, we will filter some rows, if the user reads the article or he got inside his feed, no need to recommend it again, so that will lower the number of candidate articles for the user.</p><p>user_article_with_calculated_score here we calculate user articles score based on the article’s score and user-article-topic/user-article-publisher score</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-MowHdiRBtvvD0elJULOkg.png" /></figure><p>The first Recommended article for user #2 has a low score comparing with another article, but its’ topic and publisher is the preferred for the user, and that article has the best score (1000) in its’ topic, and the user hasn’t read that article yet, that means there is a high chance it will grab user attention!</p><p>Also, we notice the 2nd recommended article for user #2 is an article that has lower article’s score comparing to 3rd one, both of them has the same publisher, but from different topic, and that user is more interested in topic #34 comparing to topic#20, so the new score will be (relevance score + article enhanced scroll) /2 as the final score to be used in articles ranking.</p><p>user_with_top_articles this step is just to change the output way, to make it easier for your command to get results, so rather than getting X rows for each user, you get 1 row per user, and articles are grouped in JSON array!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bctcwCyBVJDOsNViYhaq3Q.png" /></figure><h3>Conclusion</h3><p>BigQuery is a powerful tool that can process terabytes of data in seconds/minutes, you can use that power in background jobs in many different ways, here I showed a simple approach I use to process ~100GB of data in 2 minutes, only by using SQL.</p><p>The recommendation engine is a very crucial part in your app, this one is a very simple one for teams they don’t have ML scientist, also it’s an example to show you how BigQuery can be used</p><p>Feel free to share your thoughts in the comments below, If you have any questions or feedback don’t hesitate to ask so!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=239b1398a46a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/recommend-personalised-content-for-your-app-users-using-bigquery-239b1398a46a">Recommend personalized content for your app users using BigQuery</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>