<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ahmad Hamouda on Medium]]></title>
        <description><![CDATA[Stories by Ahmad Hamouda on Medium]]></description>
        <link>https://medium.com/@ahmad.ramadan-hamouda?source=rss-2ddacb2f6e6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*gSGxPJFGWxUp7ZVxrl1l8A.jpeg</url>
            <title>Stories by Ahmad Hamouda on Medium</title>
            <link>https://medium.com/@ahmad.ramadan-hamouda?source=rss-2ddacb2f6e6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 03:41:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ahmad.ramadan-hamouda/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Customer Segmentation at Glovo]]></title>
            <link>https://medium.com/glovo-engineering/customer-segmentation-at-glovo-8b46a787ac5e?source=rss-2ddacb2f6e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b46a787ac5e</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[system-architecture]]></category>
            <dc:creator><![CDATA[Ahmad Hamouda]]></dc:creator>
            <pubDate>Mon, 04 Jul 2022 07:11:08 GMT</pubDate>
            <atom:updated>2022-07-13T12:09:45.910Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2: Segmentation system design</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*79-nF8F0uh-aVjhG" /></figure><p>In part 1, we talked about the customer segmentation needs at Glovo and why we need to customize the customer’s UX. For example, we needed to target specific audiences for certain discounts. We also discussed the need to separate the segment’s calculations task from the segment’s restful API. In part 2, we will dive into the two components: explaining their respective responsibilities and the overall architecture.</p><p>We named the two components <strong><em>Core</em></strong> and <strong><em>Store</em></strong>. <strong><em>The core</em></strong> is responsible for performing Segment calculations in a scheduled fashion (every X minutes/hours), based on the segment definitions provided. <strong><em>The store</em></strong> is in charge of segment configuration management, auditing, and serving the calculated segments to the rest of the system. Additionally, between these components, we needed integrations to exchange segment configurations and segment results to expose those results to the external parties.</p><p>See the full architecture below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/0*AfP7JPrfoIj79z5N" /></figure><h4>Core</h4><p>For data calculations, we decided to leverage Glovo’s existing Data Lake and Big Data tools. Here were our reasons:</p><ul><li>We knew we needed to query data from multiple domains, initially orders, users, and products but possibly many others in the future as we defined new segment types. The Data Lake, built on S3, solved this requirement out of the box. It is a centralized store already gathering data from most of the domains in the system. Although data in the Data Lake was not <em>live </em>data (it was refreshed every half an hour), we didn’t need live data for most of the segmentation cases.</li><li>Performing the segment calculations would involve hundreds of millions of rows, and terabytes of data. It was clear to us that the best way to perform these calculations was using Big Data processing tools. At Glovo we use<a href="https://aws.amazon.com/emr/"> AWS EMR</a> and<a href="https://spark.apache.org/docs/latest/api/python/"> PySpark</a> to run calculations on the Data Lake.</li></ul><p>The full stack for the core also included <a href="https://www.jenkins.io/">Jenkins</a> and<a href="https://luigi.readthedocs.io/en/stable/"> Luigi</a>. New segment calculations would be put in an<a href="https://aws.amazon.com/s3/"> S3</a> prefix for the given DateTime and historical data could be eventually deleted by using<a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html"> object expiration policies</a>,</p><h3>Store</h3><p>This component is responsible for segment configurations management and also serving segment calculations. It consists of several parts: the backend exposing a restful API, an RDBMS holding the segment configurations, a separate data store (Elasticsearch) to serve the segments calculations to the backend, and Redis for caching customer-segments queries; to achieve the low latency needed.</p><p>As explained earlier, in order to move the calculated segments from the Store to the Core, we designed an ingestion process.</p><h4>Elasticsearch</h4><p>We took multiple factors into consideration to select the proper data store for segment calculations search. First and foremost, was the volume of data initial estimations. We expected 1K segments to be created by the business and partners for the initial phase. Beyond that, we anticipated over 100K segments once more segment types were added. Each one of these segments could have millions of customers and on average be around 50 MB. The second critical factor was that a given customer could belong to a multitiude of segments but different stakeholders would only be interested in a subset of segments for a given customer. For example, one might only care about segments for a particular store, or specific categories (pharmacies, not restaurants), and so forth. This meant we needed a search engine that could scale horizontally and was capable of querying by multiple, unbounded fields. And potentially, in the future, use-cases using text search. We agreed the best choice was<a href="https://www.elastic.co/?ultron=B-Stack-Trials-EMEA-S-Exact&amp;gambit=Stack-Core&amp;blade=adwords-s&amp;hulk=paid&amp;Device=c&amp;thor=elasticsearch&amp;gclid=Cj0KCQjwm6KUBhC3ARIsACIwxBioYniNLnDpcTPIcOzFfCmwoZ2FVsBmK8als7BUqBmvVXQM8G7BOK0aAv0oEALw_wcB"> ElasticSearch</a>.</p><p>We faced several interesting and challenging points during the ElasticSearch setup. We discovered performance improvements by using<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html"> term</a> queries from our Spring elastic client. We learned to select the proper data types for each indexed field (such as using<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword-field-type"> keyword</a> type for id fields). We also tweaked the elastic index<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#:~:text=By%20default%2C%20Elasticsearch%20periodically%20refreshes,in%20the%20last%2030%20seconds."> refresh rate</a> to reduce the latencies of the queries and partitioned large docs to remain below the<a href="https://www.elastic.co/guide/en/app-search/current/limits.html"> maximum document</a> size.</p><h4>Store clusters: Api and Ingestion</h4><p>Given the very different workloads of ingesting data from the Core and serving that data to outside domains, we opted for separate clusters for each task; splitting reads and writes. This would allow us to ingest in a decoupled and efficient way, scaling each cluster independently.</p><p>The ingestion flow would work as follows: When the task to process a new calculation is triggered, the<a href="https://spark.apache.org/docs/latest/api/python/"> PySpark</a> cluster starts generating segments that are later ingested by the Store Ingestion cluster and stored in a new Elastic Search index. To achieve efficient ingestion we leveraged several techniques:</p><ul><li>We used “<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html">Elastic Search alias switch</a>” to quickly swap between the old and the new index.</li><li>We used <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html">bulk API</a>, parallelization, and retries in case of errors to efficiently insert into Elastic Search.</li><li>We used a Redis lock and tweaked the message visibility timeout to expire old messages so as to have an at-most-once semantic; distinct from the at-least-once of<a href="https://aws.amazon.com/sqs/"> SQS</a>.</li></ul><h4>Redis Cache in the API Cluster</h4><p>We had committed to an SLO of p99 &lt; 50ms and to achieve it we leveraged the eventually-consistent nature of the segment calculations we were serving. We could cache heavily without worrying so much about the<a href="https://redis.io/commands/expire/#:~:text=Normally%20Redis%20keys%20are%20created,instance%20using%20the%20DEL%20command."> time to live</a> (TTL) of the entries or the cache invalidation. With Redis as a simple<a href="https://docs.aws.amazon.com/whitepapers/latest/database-caching-strategies-using-redis/caching-patterns.html"> cache-aside</a> strategy, we cached the most frequently accessed customer segments. Not only did we drop the latency to less than 10ms but we also reduced the load on the Elastic cluster. Before adding Redis due to a load of requests [10K / sec at the peak] elastic was going up to the 250ms, at least for some of the requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*p-XAWrNWCfTLCspL" /></figure><h3>Rollout Checklist</h3><p>Before we rolled out the project, we had a large checklist to validate the readiness of the multiple components and clients. We prepared an exhaustive rollout plan to minimize the chances of failure and to be ready for possible rollbacks. We also ensured documentation is in a centralized place, including the design, observability links, and<a href="https://swagger.io/specification/"> OpenAPI Specification</a> of our Restful endpoints. The rollout would start in a single city, then expand to the whole country, and finally to other countries. The gradual increase in the traffic could unveil any errors or performance issues early. AB tests were used to validate the results.</p><p>Performance testing</p><p>To accommodate the load from the client domains, we had to define initial sizes for our API, Ingestion, ElasticSearch, and ElastiCache clusters. Since we didn’t have a proper load testing tool by that time, we used JMeter to perform a first load test on the service. Later we enabled<a href="https://microsoft.github.io/code-with-engineering-playbook/automated-testing/shadow-testing/"> shadow traffic</a> from the real clients with all the proper observability around the key metrics. We adjusted the clusters node minimum and maximum node count, the node instance sizes, storage types, etc. until we reached the performance we required. Beyond that, autoscaling would adapt according to the load.</p><p>Resiliency</p><p>We couldn’t risk harming our existing clients’ SLOs and we had to define our Segmentation SLO respecting the<a href="https://www.ibm.com/docs/ru/imdm/11.6?topic=solution-integrating-upstream-downstream-systems"> upstream</a> services. On the ingestion process side, resilience was needed when reading segment parts from S3 and persisting to ElasticSearch. Moreover, we didn’t want to halt the entire ingestion of terabytes of data when a single part, out of thousands, would fail so we set up error tolerance rates. On the Elastic side, we wanted to keep the previously ingested data from old indexes for a while. In this case, the new index wouldn’t be created for failures during segments calculations or ingestion. Therefore, we used<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#ilm-gs-create-policy"> indexes lifecycle policies</a>. Finally, the client’s synchronous calls to our API cluster were protected with<a href="https://martinfowler.com/bliki/CircuitBreaker.html"> circuit breakers</a>, timeouts, and retries.</p><p>Observability</p><p>To deliver the project proper observability was critical. This included logs, metrics, dashboards, monitors, and alerts. Initially, the alerts would point to a dedicated<a href="https://slack.com/"> Slack</a> channel and later promoted to<a href="https://www.pagerduty.com/"> Pagerduty</a> to reach the on-call engineer. On-call<a href="https://www.pagerduty.com/resources/learn/what-is-a-runbook/"> run-books</a> were updated as well.</p><p>Main observability resources:</p><ul><li>Datadog’s<a href="https://www.datadoghq.com/dg/apm/ts-benefits-os/?utm_source=advertisement&amp;utm_medium=search&amp;utm_campaign=dg-google-apm-emea-general&amp;utm_keyword=application%20performance%20monitoring&amp;utm_matchtype=p&amp;utm_campaignid=15419168583&amp;utm_adgroupid=131788862153&amp;gclid=CjwKCAjw4ayUBhA4EiwATWyBrj374lFcZmYeTkTWO_ptkxUE3Xat4CXbiSWcNPFcBpQBTdiv-6fDfRoCOEgQAvD_BwE"> Application Performance Monitoring</a> (APM) for our different clusters, including API, Ingestion, and ElasticSearch.</li><li>General segmentation metrics dashboard, including endpoints, ingestions, segments calculations, SQS queues, and HTTP Client’s health</li><li><a href="https://www.elastic.co/kibana?ultron=B-Stack-Trials-EMEA-S-Exact&amp;gambit=Stack-Kibana&amp;blade=adwords-s&amp;hulk=paid&amp;Device=c&amp;thor=kibana&amp;gclid=CjwKCAjw4ayUBhA4EiwATWyBrj5MadTXgXR4k8pHPMhsQqNn3q24SngGmtw2Ur0VA91FNZ2phlj0bBoCUx4QAvD_BwE">Kibana</a> to keep track of the index’s status, including their size, document count, and policies, and to have a handy way to execute queries against Elastic.</li></ul><p>Rollout results</p><p>The production rollout was a success, no downtime nor rollbacks were needed. The application was already serving up to 10K requests per second, with a p99 of 12.05ms on average, and p75 avg 1.23ms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SsifR1bag9jyLy44" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*WGztT4wpIOMo0p0J" /></figure><p>The segmentation project proved to be effective by improving the conversion rate for discounts for new customers. It was a clear example of our <a href="https://about.glovoapp.com/culture-and-values/">Gas culture</a> but also keeping the bar high, as thanks to the good design this solution is now being rolled out for other use cases. Analysts are still gathering insights about the impact and exploring ways to further leverage the segmentation capabilities to personalize the user experience.<br>As the solution grows and new teams start using it, we may have to revisit our current solution, but until now, it’s meeting the expectations and objectives of this initial design.</p><p><strong>Authors: Ahmad Hamouda, Marcos Paulucci</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b46a787ac5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/glovo-engineering/customer-segmentation-at-glovo-8b46a787ac5e">Customer Segmentation at Glovo</a> was originally published in <a href="https://medium.com/glovo-engineering">The Glovo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer Segmentation at Glovo]]></title>
            <link>https://medium.com/glovo-engineering/customer-segmentation-at-glovo-8d2ecef2549a?source=rss-2ddacb2f6e6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d2ecef2549a</guid>
            <category><![CDATA[segmentation]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[audience-targeting]]></category>
            <dc:creator><![CDATA[Ahmad Hamouda]]></dc:creator>
            <pubDate>Mon, 27 Jun 2022 09:59:23 GMT</pubDate>
            <atom:updated>2022-06-27T09:59:23.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1: What is segmentation and why do we need it?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*Wx-WB-1GJuAcfzUH" /></figure><h3>Business context</h3><p>Glovo orchestrates a variety of discount campaigns. Before the Segmentation Service, these campaigns targeted our entire customer base, regardless of individual customer behavior or preference. For Glovo and its partners, this was highly inefficient in terms of budget and<a href="https://www.investopedia.com/terms/r/returnoninvestment.asp#:~:text=Return%20on%20investment%20%28ROI%29%20is,relative%20to%20the%20investment%27s%20cost."> ROI</a>, limiting customer acquisition, retention, and overall customer experience. We didn’t have a system capable of classifying the most appropriate discount campaign for each kind of customer. For example:</p><ul><li>Users who never ordered from a specific store</li><li>Users who hadn’t ordered in the last month</li><li>Users who order recurrently</li></ul><p>We began by providing a definition of this new concept called “Segmentation” and a new<a href="https://martinfowler.com/bliki/BoundedContext.html"> bounded context</a> called “Segments”; based on <a href="https://martinfowler.com/tags/domain%20driven%20design.html">DDD</a> concepts. Its responsibility was to “categorize customers based on different traits such as their profile, their order history, analytic events, and machine learning models”.</p><h3>Scope for an MVP</h3><p>We established the boundaries of our system and prioritized among the multiple stakeholders interested in different segment types. We needed a system capable of supporting most of their use cases. Therefore, we had to design the system properly with a focus on time to market, service response, scalability, and extensibility. Our first segment types were:</p><ul><li>New customers with 0 orders in certain stores</li><li>Customers ordering recurrently from some stores</li><li>Inactive customers who used to order in the past but had churned over a period of time</li></ul><p>Time-to-market was a hard constraint. With “Gas” as one of the <a href="https://about.glovoapp.com/culture-and-values/">Glovo’s core values</a>, we committed to delivering a solution in 3 months. We aimed to cover the first segmentation cases but also build an extensible system, capable of supporting different types of segments for the remaining and future use cases.</p><h3>Challenges</h3><p><em>Time-to-market</em></p><p>ASAP was our business constraint. Our partners were in deep need and we didn’t want to lose their trust so we had to act fast and with vision.</p><p><em>Scaling calculations</em></p><p>We would start with only a few hundred segments but, shortly thereafter, would scale to hundreds of thousands of segments. A single segment calculation could involve millions of customers and hundreds of millions of order history items.</p><p><em>Fast-growing data volume</em></p><p>Not only would the number of segments grow, but so would our customer base, order history, and other new data source involved. We had to prepare to process big data.</p><p><em>Serving segments with high availability and low latency</em></p><p>The segments would be used in most customer flows to personalize their journey therefore maintaining low latency was a crucial constraint. We committed to a tight<a href="https://en.wikipedia.org/wiki/Service-level_agreement"> SLA</a>, with an affordable latency of p99 &lt;= 50 milliseconds and availability &gt;= 99.99%.</p><p><em>Offline data vs Real-time data</em></p><p>Processing and serving terabytes of data to millions of users was challenging but having this system working with real-time data was even more difficult. Luckily, not all segments required real-time data. They would work just fine with eventually consistent segments wherein updates were delayed by a few minutes. We used this to our advantage and started by tackling the use cases that could be near real-time. This simplified our first version of the system; postponing the implementation of real-time segments for later.</p><p><em>Integration</em></p><p>How would these new customer segments be served to the whole Glovo ecosystem in a standardized way? We aimed for a solution that would be agnostic of the calling domain but flexible enough to serve each domain’s needs. Also, we wanted to integrate other components that could calculate other types of segments in their own way. For example, some Machine Learning models were already generating new kinds of customer segments. We needed to be able to feed these results into our service just as a new type of segment.</p><p><em>Cost</em></p><p>Cost doesn’t come only from marketing campaigns, but also from infrastructure and maintenance costs to new feature development. The cost of the solution could not be more expensive than the benefit the project would bring. Therefore, we aimed for a solution that was cost-effective yet maintainable. One that was reasonably extensible to cover our main use cases and could easily evolve to support new use cases. This allowed us to shorten the delivery of an MVP; meeting our strict time-to-market constraint.</p><p><em>Customer UX validation</em></p><p>We used<a href="https://martinfowler.com/articles/feature-toggles.html"> Feature toggles</a> and<a href="https://en.wikipedia.org/wiki/A/B_testing"> AB testing</a>:</p><ul><li>To customize each customer’s UX and applicable discounts</li><li>To compare the<a href="https://en.wikipedia.org/wiki/Return_on_investment"> Return on Investment</a> (ROI) against non-segmented discount campaigns</li><li>To assess the cost of customer acquisition, flexibility of defining new configurations, and scalability</li></ul><h3>High-level architecture</h3><p>This is how our Segmentations component would serve data to the rest of the Glovo ecosystem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/0*J8w80hZCg8IWnVvt" /></figure><p>We assessed the nonfunctional requirements and quickly realized that the segment’s calculation flows were quite different from the serving data; especially in terms of data volume and required latency. This led us to split calculations and reads into two different components on the architectural level. The <strong>Segments Core</strong> would be responsible for calculating the segments by processing vast and diverse types of data. The <strong>Segment Store</strong> would be responsible for serving the calculated segments to the rest of the system, with high availability and low latency. Having these two separate components would also allow us to choose the appropriate technologies for each task.</p><p>If you are interested to know what the architecture looks like check part 2 “Segmentation service zoom in”!</p><p><strong>Authors: </strong><a href="https://medium.com/@ahmad.ramadan-hamouda"><strong>Ahmad Hamouda</strong></a><strong>, </strong><a href="https://medium.com/@marcos.paulucci"><strong>Marcos Paulucci</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d2ecef2549a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/glovo-engineering/customer-segmentation-at-glovo-8d2ecef2549a">Customer Segmentation at Glovo</a> was originally published in <a href="https://medium.com/glovo-engineering">The Glovo Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>