<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Constructor Engineering - Medium]]></title>
        <description><![CDATA[​​Constructor is the only AI product discovery platform designed explicitly for ecommerce retailers. Machine learning and data science are in Constructor’s DNA, bringing dynamic personalization to the customer experience and lifting KPIs unlike any other discovery solution. - Medium]]></description>
        <link>https://medium.com/constructor-engineering?source=rss----25df6fa77ec4---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Constructor Engineering - Medium</title>
            <link>https://medium.com/constructor-engineering?source=rss----25df6fa77ec4---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:36:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/constructor-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Observe and record performance of Databricks jobs]]></title>
            <link>https://medium.com/constructor-engineering/observe-and-record-performance-of-databricks-jobs-11ffe236555e?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/11ffe236555e</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[victoriametrics]]></category>
            <category><![CDATA[performance-monitoring]]></category>
            <dc:creator><![CDATA[Sergey Ivanychev]]></dc:creator>
            <pubDate>Tue, 28 Nov 2023 17:32:49 GMT</pubDate>
            <atom:updated>2023-11-28T17:32:49.808Z</atom:updated>
            <content:encoded><![CDATA[<h3>Observe and record performance of Spark jobs with Victoria Metrics</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODiTP-3dAQjQznYwfmwqwA.jpeg" /></figure><p>In this article, we will explore our approach to monitoring and observability of performance of Spark pipelines. We’ll describe our approach to building the system with Databricks clusters, but the technique can be applied to any Spark setup. Our requirements for the systems were:</p><ul><li>Be able to inspect metrics in real-time</li><li>Be able to inspect metrics after the cluster shutdown</li><li>Automate the performance insights generation using collected metrics</li><li>Be able to <em>push</em> metrics from driver and workers (because we can’t <em>pull</em> metrics from them with Prometheus)</li></ul><p>The article is divided into three sections:</p><ul><li><strong>Background</strong> section describes the existing approaches for monitoring.</li><li><strong>Making Spark pipelines observable </strong>section<strong> </strong>describes the chosen approach on high-level.</li><li><strong>Using performance metrics in action</strong> section describes the concrete details used in our monitoring systems.</li></ul><h3><strong>Background</strong></h3><p>At Constructor, multiple teams use data platforms to build data pipelines and process terabytes of data every day. The data platform team is responsible for providing tools for other teams to effectively do it. Each team owns dozens of different pipelines.</p><p>We care about pipeline performance because it inevitably influences the overall cloud bill and time of execution. Bad performance can lead to pipeline failures or to a disproportional increase of its execution time (e.g., when too much RAM is used and swap is used).</p><p>So, we need to observe performance, but we can’t afford to manually check in on every pipeline. There are too many pipeline runs happening daily (in our case, ~10k in total). Thus, we need to build an automated way to collect performance metrics from Databricks, store them for later inspection, and build a tool on top of this data to gather performance insights. Some of the requirements for the monitoring system are:</p><ul><li><strong>Metrics should be stored and readable by both human and machine.</strong> We would like to look at the metrics in real-time during the pipeline run. Also, we want to be able to examine them after it finishes. We also want to process them and get insights.</li><li><strong>Metrics should be <em>pushed</em> from driver/executors.</strong> Our pipelines use <a href="https://docs.databricks.com/en/clusters/custom-containers.html">Databricks Container Services</a>. So, we’re very limited with what we can expose from the container running Spark. The vanilla Prometheus way of pulling metrics won’t work for us.</li></ul><h4><strong>The state of Spark performance observability</strong></h4><p>The most important metrics exposed by Spark are represented in <a href="https://spark.apache.org/docs/3.5.0/web-ui.html"><strong>Spark Web UI</strong></a>. It provides a lot of information about Spark job execution and metrics, such as GC time, time to serialize/fetch results, peak execution memory, and shuffle read/write sizes. <strong>Most of these metrics are Spark specific and aggregated over a period of time.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Jat-wH4OY7Mbz3Fn0SyMw.png" /><figcaption>Example of exported Spark metrics per executor for a stage</figcaption></figure><p>These metrics can be gathered automatically using a built-in Spark prometheus integration. This can be enabled with settingspark.ui.prometheus.enabled=true. The metrics will be exposed in Prometheus format at /metrics/executors/prometheus path.</p><ul><li>If you’re using Databricks, you can use the following URL format to access them:</li></ul><pre>https://dbc-&lt;instance&gt;.cloud.databricks.com/?o=...#setting/sparkui/&lt;cluster_id&gt;/driver-.../metrics/executors/prometheus</pre><ul><li>I’m not aware of how to fetch these metrics in plain text. Feel free to comment on this article if you know if these are exposed.</li></ul><p>The problem with these metrics is that<strong> it does not expose any time series information and detailed performance metrics </strong>like CPU time breakdown, IOPS, etc. Moreover, the data that is exposed by UI is inconvenient to extract automatically to make it queryable.</p><h4><strong>Before the Databricks Runtime 13.0</strong></h4><p><a href="https://en.wikipedia.org/wiki/Ganglia_(software)">Ganglia</a> was the main performance observability solution for Apache Spark prior to the <a href="https://docs.databricks.com/en/archive/runtime-release-notes/13.0.html">DBR 13.0</a> version. It exposed the performance metrics in a separate webpage, letting a user query what’s curious to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FhuJV7jh6lp0rp_WLyI7ww.png" /><figcaption>Ganglia interface</figcaption></figure><p>The biggest problem with Ganglia is as soon as the cluster dies, Databricks only has a collection of screenshots of metrics taken with a 15-minute interval. These screenshots are sometimes helpful, but can’t provide a detailed picture of what’s going on.</p><p>In addition, Ganglia is not developed and not supported on modern operation systems (e.g., you can’t install it via apt since Ubuntu 20.04).</p><h4><strong>Databricks Runtime 13.0 and later</strong></h4><p>Starting from DBR 13.0, Databricks replaced Ganglia with <a href="https://docs.databricks.com/en/compute/cluster-metrics.html">cluster metrics</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CVlG5nGmqKOPIyKADQH_EQ.png" /><figcaption>The screenshot from the new cluster metrics tool</figcaption></figure><p>Compared to Ganglia, it’s capable of storing live and historical cluster metrics and exposing them through Databricks UI.</p><p>This system greatly improves observability, but still is not good enough to build monitoring upon. This performance data is not exposed via any API. This means no alerts or Grafana dashboards.</p><h3>Making Spark pipelines observable</h3><p>Here’s the solution we came up with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeGTo64hixcbGaHd8pUmwA.png" /></figure><ul><li>Deploy a container with Victoria Metrics accessible within our infrastructure.</li><li>During initialization time (init script), we install <a href="https://github.com/prometheus/node_exporter">node_exporter</a>, <a href="https://github.com/utkuozdemir/nvidia_gpu_exporter">nvidia_gpu_exporter</a> and run them in the background. They collect performance metrics and expose them locally.</li><li>During initialization time, we run a Python script in the background that periodically pulls metrics from node exporters and pushes them to Victoria Metrics.</li><li>Connect Grafana to it as a Prometheus source. Create <a href="https://prometheus.io/docs/alerting/latest/alertmanager/">AlertManager</a> based on the Prometheus source. Periodically gather all metrics, store them as a <a href="https://delta.io/">Delta</a> table, and automatically find performance problems.</li></ul><h4><strong>Victoria metrics</strong></h4><p><a href="https://docs.victoriametrics.com/">Victoria Metrics</a> is a fast, cost-effective, and scalable monitoring solution and time series database. It is developed in Go, Prometheus-compatible, and supports both <em>pulling</em> metrics from targets (as Prometheus) and <em>pushing</em> metrics to it via an API. It also supports PromQL and can serve as a Prometheus-source for Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lJZQg_CRmcpR5PR1SxX86g.png" /><figcaption>The screenshot of VM UI exposing the collected metrics</figcaption></figure><p>Victoria Metrics can be deployed easily in any environment, which can run Docker containers (ECS in our case).</p><pre>docker run -it - rm \<br>  -v /where/to/store/data/locally:/victoria-metrics-data \<br>  -p 8428:8428 \<br>  victoriametrics/victoria-metrics \<br>  -retentionPeriod=1w</pre><p>Two things to highlight:</p><ul><li>Mount the local folder to /victoria-metrics-data to specify where the VM data will be stored</li><li>--retentionPeriod=1w controls how many days the metrics will be stored by VM. After the period, these will be cleaned up automatically.</li></ul><p>Victoria metrics exposes data in multiple ways:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W7a4xiFayOnjKjiMZBOfLA.png" /><figcaption>The example of Grafana dashboard that builds per-cluster performance page</figcaption></figure><ul><li>You can use VM as Prometheus sources in Grafana and use PromQL to describe data requests.</li><li>You can use <a href="https://docs.victoriametrics.com/url-examples.html#apiv1export">Victoria Metrics API</a> to fetch data in CSV, JSON, and native formats for further processing. Here’s the example of how to download these metrics in JSON format from Python:</li></ul><pre>response = requests.get(<br>    url=&quot;https://localhost:8428/api/v1/export&quot;,<br>    params={<br>        &quot;match[]&quot;: &quot;{job=\&quot;job-id\&quot;}&quot;,<br>    },<br>)</pre><h4><strong>Node exporter and Nvidia GPU exporter</strong></h4><p><a href="https://github.com/prometheus/node_exporter">node_exporter</a> collects various system metrics, such as CPU, disk I/O, memory, network, etc., and exposes them via API in <a href="https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md#text-format-details">Prometheus format</a>. <a href="https://github.com/utkuozdemir/nvidia_gpu_exporter">nvidia_gpu_exporter</a> does the same and exposes metrics in the same format, but only cares about GPU metrics. Example of metrics in this format:</p><pre>node_disk_read_bytes_total{device=&quot;disk0&quot;} 2.89638715392e+11<br>node_disk_read_errors_total{device=&quot;disk0&quot;} 0<br>node_disk_read_retries_total{device=&quot;disk0&quot;} 0<br>node_disk_read_sectors_total{device=&quot;disk0&quot;} 4616.966796875<br>node_disk_read_time_seconds_total{device=&quot;disk0&quot;} 4674.995333666</pre><p>Remember that Victoria Metrics supports pulling and pushing metrics in Prometheus format. Thus, we need something in between them. Running them during init time of the container is as easy as:</p><pre>nohup node_exporter &amp;<br>nohup nvidia_gpu_exporter &amp;</pre><h4><strong>Metric collection daemon</strong></h4><p>You need to implement a program that pulls metrics from node exporters and pushes them to Victoria Metrics in 30 seconds. In the simplest form, it can be something like this (implemented in Python):</p><pre>import requests as r<br>import time<br><br>while True:<br>    # Export metrics every 30 seconds.<br>    time.sleep(30) <br>    # Fetch node exporter metrics from local node_exporter.<br>    node_resp = r.get(&#39;http://127.0.0.1:9835&#39;).text<br>    # Fetch GPU metrics from local nvidia_gpu_exporter.<br>    node_gpu_resp = r.get(&#39;http://127.0.0.1:9100&#39;).text<br>    # Combine metrics and send them to Victoria Metrics.<br>    all_metrics = &#39;\n&#39;.join((node_resp, node_gpu_resp))<br>    r.post(&#39;http://victoria-metrics-url/metrics/job/my-job-id/instance/my-instance-id&#39;,<br>           body=all_metrics)</pre><p><strong>Note</strong> that the URL to Victoria metrics contains all labels assigned during upload. In the example above, two labels will be assigned to the metrics: job=my-job-id and instance=my-instance-id . This is the place where you can add needed labels, such as Databricks cluster id, instance IP, instance type, pipeline name, owner team, etc. To run this in the background of the driver/executor container, use nohup</p><pre># Forward stdout and stderr to files.<br>nohup python dbx_daemon.py &gt; /tmp/dbx_daemon.log 2&gt; /tmp/dbx_daemon.err &amp;</pre><h3>Using performance metrics in action</h3><p>During the cluster initialization time, we save and later expose the following labels from driver and executors. Note that we use initialization time environment variables and metadata from <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html">EC2 instance metadata IP</a> (169.254.169.254)</p><ul><li>is_driver — is the current instance Spark driver or not. This information is available via DB_IS_DRIVER environment variable, available during cluster initialization (in init script).</li><li>job — main Victoria Metrics label. We use the Databricks Cluster Id for that. Available at DB_CLUSTER_ID environment variable during initialization.</li><li>instance_type — AWS EC2 instance type. Available on board of EC2 instance at 169.254.169.254/latest/meta-data/instance-type</li><li>instance_lifecycle — is the instance <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html">Spot</a> or On-Demand. 169.254.169.254/latest/meta-data/instance-life-cycle</li><li>instance_region, instance_az, instance_hostname — these are available at 169.254.169.254 as well.</li></ul><p><strong>VM system requirements</strong></p><p>Regarding the needed resources for the Victoria Metrics setup, we currently store:</p><ul><li><strong>32,218,411 </strong>Total time series</li><li>With <strong>6 labels</strong> with <strong>&gt;10K of unique values each </strong>daily</li><li>Running in container with 1 CPU core, 4G of RAM, and 100G of disk storage</li></ul><p><strong>Real-time performance dashboard</strong></p><p>In Grafana, add a <a href="https://grafana.com/docs/grafana/latest/datasources/prometheus/">Prometheus data source</a> and use Victoria Metrics URL for that. It will work. Using this data source, it’s straightforward to create a dashboard parameterized with job (label of metrics pushed to VM).</p><p>Using a concrete job value, we can fetch and aggregate all of the metrics we need from VM in Grafana. (This is the <a href="https://gist.github.com/ivanychev/3f02e95e5d8438c18bdd27a6f136bc2c">full example of our dashboard</a>.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eENDMZIiwP4CDZ2W4HaT5g.png" /><figcaption>Parameterize dashboard with job. Note that our labels start with dbx-</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VK_vSN5rcvCaUOlypS0--g.png" /><figcaption>You can pick the cluster to watch</figcaption></figure><p><strong>Note</strong> that in this setup, you can compose the Grafana link only with a Databricks cluster ID. In the example above, the URL parameter added to the Grafana dashboard URL var-ClusterId=&lt;cluster_id&gt; will open the dashboard for the concrete cluster. We can provide the user with this link on cluster start.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0u2aQcEBtClRgQM9M-LI9Q.png" /><figcaption>Printing the URL to the performance dashboard during the start of pipeline</figcaption></figure><p>That way, we provide our users with the ability to monitor performance metrics during or after pipeline execution.</p><p><strong>Prometheus alerts</strong></p><p>Victoria Metrics is compatible with Alertmanager (look at <a href="https://docs.victoriametrics.com/vmalert.html">these docs</a> to set it up). Alertmanager rules can be used to monitor performance and issue alerts. The simplest example is to alert if the execution time for the past day for a given MyPipelineName data pipeline is too high.</p><pre>- alert: &quot;Too much time executing MyPipelineName&quot;<br>  expr: max_over_time(max(node_time_seconds{task_family=&quot;MyPipelineName&quot;} - node_boot_time_seconds{task_family=&quot;MyPipelineName&quot;})[1d]) &gt; 60 * 60<br>  for: 5m<br>  labels:<br>    severity: warning<br>    oncall_team: data-science</pre><p><strong>Performance insights</strong></p><p>Using Victoria Metrics API, we can export the metrics and run computations on top of it. Here’s an example of how to do it:</p><pre># Fetch all job names that have perf data in VM.<br>resp = r.get(<br>  &#39;https://vm-url/api/v1/label/job/values&#39;,<br>  params={<br>    &#39;start&#39;: &#39;-1d&#39;,<br>  }).json()<br>job_names = resp[&#39;data&#39;]<br><br># Download JSONL files with metrics to local machine.<br>for job in job_names:<br>    with r.get(&#39;https://vm-url/api/v1/export&#39;, params={<br>        &#39;match[]&#39;: f&#39;&#39;&#39;{{job=&quot;{job}&quot;}}&#39;&#39;&#39;<br>    }, stream=True) as resp, pathlib.Path(f&quot;/dir/{job}.jsonl&quot;).open(&quot;wb&quot;) as f:<br>        f.write(resp.content)</pre><p>The imported metrics can be uploaded to any analytical database or data lake. Use SQL to find jobs with undesired performance metric values. Some metrics worth monitoring:</p><ul><li>Average CPU I/O Wait time</li><li>Max RAM usage / Total RAM available</li><li>Swap usage</li><li>Spikes in total disk capacity (in Databricks, it means storage autoscaling kicked in)</li></ul><h3>What didn’t work for us</h3><h4><strong>CloudWatch</strong></h4><p>CloudWatch is a monitoring service provided by Amazon Web Services (AWS). It allows users to view and collect key metrics, logs, set alarms, and automatically react to changes in AWS resources.</p><p>We considered sending metrics to CloudWatch but eventually disabled the integration. The main reason was <strong>high operational costs</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9mvYLZI3lNEWJWznfs437g.png" /></figure><p>Logging metrics from each Spark driver/executor means extremely high cardinality of metric labels (in the best case, it’s # metrics * # of VM instances run daily). CloudWatch <a href="https://aws.amazon.com/cloudwatch/pricing/">charges based on number of unique metrics</a>, which was simply too high in our case.</p><h4><strong>Prometheus + Push Gateway</strong></h4><p>Prometheus alone can’t be used to push metrics to it. <a href="https://github.com/prometheus/pushgateway">Pushgateway</a> is a small go service that can be used to push metrics to it and wait for Prometheus to scrap metrics from it.</p><p>We couldn’t use Pushgateway because it doesn’t support TTL for metrics. Due to very high metric cardinality, we needed to clean up Pushgateway manually, which was quite complex to do.</p><h3>Conclusion</h3><p>In this article, we discovered the performance monitoring and observability approach of Spark Databricks pipelines. We demonstrated how to send, store, and retrieve metrics from Victoria Metrics and use them for performance alerts and insights generation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11ffe236555e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/observe-and-record-performance-of-databricks-jobs-11ffe236555e">Observe and record performance of Databricks jobs</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attribute Enrichment: Under the Hood]]></title>
            <link>https://medium.com/constructor-engineering/attribute-enrichment-under-the-hood-acd10b8cf7a7?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/acd10b8cf7a7</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Anton Baltachev]]></dc:creator>
            <pubDate>Wed, 15 Nov 2023 18:48:54 GMT</pubDate>
            <atom:updated>2023-11-15T18:48:54.106Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rnjZU0K4dHQ_XRTpSnJblQ.png" /></figure><p>In the previous <a href="https://medium.com/constructor-engineering/introducing-attribute-enrichment-at-constructor-143ed55ed765">article</a>, we gave an overview of how Attribute Enrichment at Constructor works. In this part, we’ll explore the machine learning techniques used for this process in more detail.</p><p>Choosing which attributes to display alongside products depends largely on what customers want to see and know. For instance, when shopping for a highly technical product like a computer chip or a specific set of screws, many customers delve deeper into technical details, reading about features, where the products can fit, and technical capabilities. They also prioritize these over the images associated with the products. In these domains, words and descriptions play a crucial role.</p><p>When someone’s looking for a new Lego set, it’s all about how cool the model looks once it’s put together. Sure, the size and recommended age might be good to know, but what really grabs customers’ attention is the design and mini figures that come with it. With Lego, it’s those eye-catching details that usually seal the deal.</p><p>Now, let’s look at furniture. Think of someone arranging their living room and looking for a new sofa. They’d want to see pictures of the sofa from different sides to get a good idea of its design. At the same time, they’d want to read about its size to make sure it fits in their room, in addition to what it’s made of to avoid poor quality materials. In cases like this, both pictures and words are really important to help customers decide what to buy.</p><p>You might see what I’m getting at. There’s no silver bullet, and it’s essential to have a full arsenal at hand, including image-based, text-based, and multimodal approaches that merge visual and textual information.</p><h3>Image-Based Approach</h3><p>An image-based approach is an obvious choice for attribute enrichment due to the availability of corresponding images for all items. In our exploration of various methods, we have uncovered some valuable insights that we would like to share.</p><h4>Fine-Tuning an Image Encoder</h4><p>When we talk about fine-tuning an image encoder, we mean adjusting a model that already knows a lot about pictures to get better at spotting product details. Think of it like this: models like ConvNets or Visual Transformers have seen tons of images before. So, they know general things about pictures. With fine-tuning, we’re helping them get better at specific tasks, like spotting the details of a shirt’s pattern or the color of a shoe. So, if you have products with unique details that need to be noticed, fine-tuning is a good way to help the model see and understand those details better. However, it comes with its own set of limitations:</p><ul><li><strong>Overfitting risks.</strong> If the training data lacks diversity or contains noise, the model may overfit and struggle to generalize well on unseen data.</li><li><strong>Ambiguity in prediction.</strong> Let’s say you’re using this model for an e-commerce platform that sells home decor. If you have an image of a living room that has both a sofa without any pattern and a diagonal checkered pattern rug, the model could get confused because there’s no clear pattern for which item should be identified.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xsoLLoBV7TqXDBNf4B4aLQ.png" /></figure><ul><li><strong>Limited consideration of textual data. </strong>Imagine browsing for a raincoat online. The image showcases its design, color, and fit on the model. However, textual data is crucial here. It can detail whether the material is breathable, if the coat is 100% waterproof or just water-resistant, and the types of temperatures or conditions it’s best suited for.</li></ul><p>Now, consider looking for a pair of hiking boots. The picture gives you a sense of the boots’ style and construction, but it’s the textual information that could specify features like ankle support, type of insulation, or the kind of terrain it’s designed for. Customer reviews could reveal how comfortable they are during long treks or if they’re truly slip-resistant in wet conditions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vONWc9yVFvEiF6otzDVgFg.png" /></figure><p>In both apparel scenarios, the picture grabs attention, but the textual data provides the depth and context necessary for making a purchase. Ignoring such textual insights can lead the model to make less informed predictions, potentially missing key attributes that customers care about.</p><h3>Multimodal Approach</h3><h4>Fully-Connected Layer on Top of a Multimodal Encoder</h4><p>To overcome the problem of ambiguous predictions, we chose to explore a multimodal approach. This method involves using a fully-connected layer on top of a multimodal encoder, combining visual and textual information.</p><p><strong>How It Works:</strong></p><ul><li><strong>Visual Encoding.</strong> Firstly, images of products go through an image encoder, much like our previous approach, turning the image into a list of numbers that computers can understand (often called a feature vector).</li><li><strong>Textual Encoding.</strong> At the same time, any text associated with that product — product description, title, specifications, etc. — is processed by a text encoder, turning that text into its own list of numbers or a feature vector.</li><li><strong>Combining The Two.</strong> Once we have the image and text both turned into these lists of numbers, they are combined. This combination can be as simple as just stacking them on top of each other or as complex as merging them with specialized mathematical operations.</li><li><strong>Fully-Connected Layer.</strong> The combined data then goes through the fully-connected layer, which works as the final decision maker, figuring out which attributes best fit the product.</li></ul><p>However, it’s not the best solution either. There are several factors to consider:</p><ul><li><strong>Overfitting risks.</strong> As mentioned, if our training data isn’t varied enough or if it has mistakes, our model might get too fixated on wrong details. The fully-connected layer, especially when not pre-trained, can amplify these mistakes. It starts from a “noisy” or random state and can latch onto this noise.</li><li><strong>Underfitting risks.</strong> On the flip side, if the fully-connected layer is too simple, it might not capture all the intricate details from the combined data. This means it might miss out on some important patterns, leading to more generic or inaccurate predictions.</li><li><strong>Model maintenance challenges.</strong> As with any model, changes or updates can be a headache. If a new type of product attribute comes into the mix, it might mean retraining a large part of the model.</li></ul><h4>Fine-Tuning Multimodal Models for Zero-Shot Classification</h4><p>Unfortunately, we ran into issues with overfitting when trying to create fully-connected layers from scratch. So, we looked into adjusting the layers that were already trained by fine-tuning multimodal models in a zero-shot setting.</p><p>We employ a sophisticated model known as CLIP for this purpose. CLIP is quite impressive in its abilities. For instance, when a user uploads an image of a t-shirt with an unconventional neckline like a “mock neck,” CLIP can intelligently select the most suitable option from the list of possible choices: v-neck, round neck, square neck, or mock neck. Remarkably, it accomplishes this even though it wasn’t explicitly trained to identify necklines. These remarkable capabilities are a result of its extensive training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EaC1PFsQQeAgHYi3L5kRjg.png" /><figcaption>CLIP Pretraining Method. Image retrieved from <a href="https://arxiv.org/abs/2103.00020">https://arxiv.org/abs/2103.00020</a>.</figcaption></figure><p>However, like any powerful tool, CLIP does have its limitations:</p><ul><li><strong>Risk of forgetting.</strong> Multimodal models are susceptible to forgetting previously learned information when fine-tuning, making it challenging to maintain overall model quality. The improvements in specific tasks or domains may come at the cost of losing other valuable qualities.<br>One way to prevent this is by freezing most of the layers, meaning we don’t change them during fine-tuning. This way, we don’t erase what the model already knows, and it’s less likely to forget important information.</li><li><strong>More complex training setup.</strong> Fine-tuning multimodal models for zero-shot classification requires careful consideration of negative sampling strategies, adding complexity to the training process compared to regular classification.</li></ul><p>In summary, we find value in utilizing CLIP as it greatly assists us in addressing challenging issues, such as identifying t-shirt necklines. Nonetheless, it’s worth noting that not all aspects can be discerned from images alone. In some cases, we rely on textual information like titles and descriptions to categorize items, even though CLIP primarily specializes in visual understanding.</p><h3>Text-Based Approaches</h3><p>Keeping these considerations in mind, we’ve started exploring text-based approaches, where our primary objective is to extract attributes from textual data. In the following sections, we will delve deeper into these approaches, exploring their variations, practical applications, and advantages and disadvantages.</p><h4>Regular Expressions</h4><p>One prominent technique within text-based approaches is the use of regular expressions, which are incredibly powerful for pinpointing specific patterns within text data. By defining predefined patterns or rules, we can achieve remarkable results through this simple yet effective concept.</p><p>To illustrate how we can extract valuable product information from textual descriptions, consider the following real-world examples:</p><p>Product descriptions often include technical specifications such as screen size, weight, or camera resolution. You can use regular expressions to extract these specifications by defining patterns that match relevant numeric values and units:</p><ul><li>Regular Expression Pattern for Screen Size: <strong>(\d+(.\d+)?)-inch (screen|display)</strong></li><li>Regular Expression Pattern for Weight: <strong>(\d+(.\d+)?) (pounds|kg)</strong></li></ul><p>Suppose the product descriptions contain sentences like:</p><ul><li>“This laptop features a <strong>15.6-inch</strong> display.”</li><li>“Our camera weighs only <strong>0.5 pounds</strong>.”</li></ul><p>Applying the respective regular expression patterns would extract screen sizes (e.g., “15.6-inch”) and weights (e.g., “0.5 pounds”) from the descriptions. So regular expressions offer several advantages:</p><ul><li><strong>Suitable for Multi-Word Text Patterns.</strong> Regular expressions excel in extracting multi-word patterns from text, making them ideal for scenarios where attribute values span multiple words.</li><li><strong>Fast and Effective. </strong>Regular expressions are efficient for extracting attributes from large volumes of text data, making them a speedy choice for processing.</li><li><strong>Deterministic. </strong>You have full control over attribute extraction results, ensuring consistency.</li><li><strong>One-Time Extraction.</strong> Results can be extracted once and used offline, reducing computational overhead.</li><li><strong>Customizable. </strong>Regular expressions can be easily modified to accommodate changes or extensions in the extraction rules.</li></ul><p>However, there are some challenges to consider:</p><ul><li><strong>Complexity.</strong> Regular expressions can become intricate, especially when dealing with complex text patterns, potentially leading to difficult-to-maintain code.</li><li><strong>Limited Contextual Understanding.</strong> They lack contextual understanding, making it challenging to cover all possible variations of attribute values and limiting their suitability for tasks like sentiment analysis.</li></ul><h4>Named Entity Recognition (NER)</h4><p>Named Entity Recognition (NER) is a powerful technique for extracting specific entities — such as product brands, names, sizes, colors, and materials — from unstructured text. NER models can significantly enhance attribute enrichment by automatically identifying and categorizing these entities. Let’s explore NER in more detail, including real-world examples of its application:</p><p>Suppose you have a dataset of fashion product descriptions:</p><ul><li>“This T-shirt is available in Small, Medium, and Large sizes.”</li><li>“Our backpack collection includes sizes from 15 to 30 liters.”</li><li>“Choose from shoe sizes ranging from 5 to 12.”</li></ul><p>A NER model trained on your product data can recognize size information like small, medium, and large as well as numeric sizes like 5, 12, 15, and 30.</p><p>Open-sourced NER models work well if you know your data is similar to the data that these models expect for the input. This way, these models have:</p><ul><li><strong>Semantic Understanding. </strong>If your data aligns with the model’s training data, it can accurately detect entities. For instance, if both the data and the NER model focus on food, it can correctly identify food items.</li><li><strong>Automatic Categorization. </strong>These models can automatically categorize extracted values into predefined attributes, such as brands, names, and sizes, eliminating the need for additional clustering methods.</li></ul><p>However, these models have cons, such as:</p><ul><li><strong>Limited Entities.</strong> The models come with predefined attributes, which might not cover all the entities you need.</li><li><strong>Missed Values.</strong> Since these models are trained on external datasets, they might not extract or recognize specific values unique to your data.</li></ul><p>NER models fine-tuned on your data are very good for your specific data if you have enough qualified labeled data for training. If we lived in an ideal world, these models would be a solution for the majority of AE tasks because the models:</p><ul><li><strong>Understand Data.</strong> Because these models are trained on your data, they have a deep understanding of it.</li><li><strong>Recognize Semantics.</strong> They grasp the semantic properties of your data.</li><li><strong>Identify Unlimited Attribute Values.</strong> They can identify a vast range of attribute values, from known ones to entirely new entries in your dataset.</li></ul><p>However, these models also have their own drawbacks, such as:</p><ul><li><strong>Data Requirement.</strong> A significant amount of labeled data is required for training, which can be challenging to obtain.</li><li><strong>Resource Intensive. </strong>Training these models demands time and robust computational resources, like GPUs.</li><li><strong>Validation Needs. </strong>Post-training, robust validation techniques are essential to ensure the model’s accuracy.</li></ul><h4>Large Language Models</h4><p>Question Answering (QA) is a technique commonly used to enrich the values of specific attributes. While you can ask any question about your data, including requests to extract all mentioned attributes, being too broad might not guarantee accurate results. It’s straightforward to understand how to use QA models for Attribute Enrichment (AE): the idea is to provide the model with a prompt containing product details, followed by a question about the specific attribute you’re interested in. Then, the model provides an answer containing the value of that attribute.</p><p>For instance, imagine you have a product description, and you want to extract flavor attributes using a QA model:</p><p><strong>Context:</strong> “Our gourmet chocolate box is a delightful indulgence for chocolate aficionados. It boasts an array of handcrafted chocolates with flavors like dark truffle, sea salt caramel, and raspberry ganache, all crafted from top-notch, sustainably harvested cocoa beans.” <br><strong>Question:</strong> “What are the flavors of the chocolates?” <br><strong>Answer:</strong> “Dark Truffle, Sea Salt Caramel, Raspberry.”</p><p>We selected a GPT-like model because it’s the best for QA tasks. When provided with a clear prompt about specific details, these models can offer precise answers. Their strengths lie in their extensive training on a vast amount of data and their inherent ability to identify entities, eliminating the need for separate NER models. However, there are points you should consider:</p><ul><li><strong>Commercial Restrictions. </strong>Some models may not be available for commercial use.</li><li><strong>Performance Limitations. </strong>Cloud-based models may have token and request limitations.</li><li><strong>Processing Time.</strong> It can be lengthy, whether using an API or running the model locally.</li></ul><h3><strong>Conclusion</strong></h3><p>In the ever-evolving landscape of attribute enrichment, there’s no one-size-fits-all solution. The key lies in having a versatile toolkit at our disposal, with the flexibility to choose the right approach based on the specific needs of each product category. As technology continues to advance, we remain committed to refining our methods and ensuring that customers have all the information they need to make confident purchasing decisions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd10b8cf7a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/attribute-enrichment-under-the-hood-acd10b8cf7a7">Attribute Enrichment: Under the Hood</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to optimize AWS S3 costs via granular visualisation]]></title>
            <link>https://medium.com/constructor-engineering/how-to-optimize-aws-s3-costs-via-granular-visualisation-d921328306e9?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/d921328306e9</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[cost-optimization]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Sergey Ivanychev]]></dc:creator>
            <pubDate>Thu, 31 Aug 2023 02:53:16 GMT</pubDate>
            <atom:updated>2023-08-31T10:46:37.220Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Optimize AWS S3 Costs via Granular Visualization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aGpiS3JzOuiQO1HVBvsucQ.jpeg" /><figcaption>What happens when a data lake turns into a data swamp. Credit: Photo by <a href="https://unsplash.com/fr/@krystian_piatek?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Krystian Piątek</a> on <a href="https://unsplash.com/s/photos/swamp?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>TL;DR: </strong>Expressive visualisations help to save money. For example, if you’re using S3 as your cloud storage of choice, then <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Server access logs</a> show <em>who</em> accessed <em>which</em> S3 object. <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Inventory</a> regularly exports the metadata of each S3 object in your bucket. These two data sources can be used to answer questions like <em>“How much does this prefix cost per month?”</em>, <em>“How frequently was this prefix accessed for past 30 days?”, </em>or <em>“How much will it cost to transfer this prefix to Glacier?” </em>Go collect and join this data. Then, build treemap visualizations for insights on your spending.</p><p>Wow, that’s a huge cloud bill!</p><p>Companies build remarkable AI products that store and process a lot of data. <a href="https://www.wired.com/insights/2014/07/data-new-oil-digital-economy/">Data is the new oil</a>, they say. But how do we make sure we don’t spend too much on oil tanks?</p><p>Many data-driven companies have been there. To provide intelligent products, they need to be trained on significant volumes of data. So, they build a <a href="https://aws.amazon.com/ru/big-data/datalakes-and-analytics/what-is-a-data-lake/">data lake</a>. They save every single useful event, message, log, etc., hoping to extract value from it later on. Data engineers build sophisticated pipelines. There are tables that depend on other tables (some get deprecated, and others are unused). Basically, it all eats up a lot of space.</p><p>This is one of many ways a data lake can become a data swamp, which is hard to manage and usually costs a lot of money to maintain.</p><p>In this article, we’ll explore techniques that can help cloud infrastructure owners figure out how to clean up data. We’ll explore ways of solving problems in AWS and similar techniques that can be replicated in other cloud providers with similar APIs. I’ll add some links to the APIs that can be used if you use Azure or GCP.</p><h4>Source of inspiration</h4><p>As a macOS user and a software engineer, I sometimes struggle with my hard drive running out of space. There’s another similar problem: here’s a folder that consumes a lot of space. Why is it so heavy? What consumes that precious disk space?</p><p>To solve this problem locally, I use a tool called <a href="https://daisydiskapp.com/">DaisyDisk</a>. (There are probably many similar tools, even for other operating systems — e.g., <a href="https://windirstat.net/">WinDirStat</a> of Windows.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*jSoTRHQ3cEGQBhVYoODnQQ.png" /></figure><p>What is so cool about this tool is the visualization technique it uses to represent disk space. It’s called a <strong>sunburst graph</strong>. Here are some examples of <a href="https://plotly.com/python/sunburst-charts/">how it can be built using Plotly</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*VsoAMadZmRybBh2hXDWC4w.png" /></figure><h4>Definition of Done</h4><p>Building the sunburst visualization for S3 buckets would help us find the most expensive S3 folders to examine their cost and data size.</p><p>But we can do even better than that. What if we could see <em>who </em>accessed files from any given S3 prefix, including <em>how frequently</em>? We can build that visualization as well.</p><p>It’s also important to note that costs come <a href="https://aws.amazon.com/s3/pricing/">not only from storage</a>. We also pay for API requests to the storage infrastructure. If a big chunk of the AWS S3 bill comes from API requests, then how can we figure out what causes it? We can build the prefix to request visualization.</p><p>In this article, we’ll build:</p><ul><li>A sunburst graph of the S3 bucket that visualizes the money we spend on each prefix</li><li>A visualization of the usage of each S3 prefix</li><li>A visualization of the frequency of requests to each S3 prefix</li></ul><h4>Data sources</h4><p>We will use Python + <a href="https://spark.apache.org/docs/latest/api/python/">PySpark</a> for data processing and <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/index.html">boto3</a> Python library for accessing the AWS API, but the logic can be replicated in any programming language. Make sure that the data processing framework you use can fit significant amount of logs. (For example, <a href="https://pandas.pydata.org/">Pandas</a> will choke when analyzing buckets with too many objects.)</p><p>To accomplish this task, we need to answer two types of questions for each S3 object:</p><ul><li>How much does it cost to store this object per month?</li><li>How many requests have been done to this object for the past X# days?</li></ul><p><a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Server access logs</a> and the <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Inventory</a> will help us answer these questions.</p><p>Other cloud providers have similar capabilities, but the APIs and data sources might be different. For example, Azure has <a href="https://learn.microsoft.com/en-us/azure/storage/blobs/blob-inventory">blob inventory</a> and <a href="https://learn.microsoft.com/en-us/azure/storage/common/storage-analytics-logging?tabs=dotnet">storage analytics logging</a>. GCP has <a href="https://cloud.google.com/storage/docs/insights/inventory-reports">storage inventory reports</a> and <a href="https://cloud.google.com/storage/docs/access-logs">access logs</a>.</p><h4>Step 0: Learn about the costs in your AWS region</h4><p>First, we need to learn how much S3 charges us for requests/storage in our region. These costs differ from region to region. Learn yours <a href="https://aws.amazon.com/s3/pricing/">on the AWS website</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Tm2X7J-fmcUebObgYcZ7gA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*-_bmP5jIYdLM7uEtHnSjiQ.png" /><figcaption>Note that the costs are per storage class. Each storage class provides different sets of tradeoffs.</figcaption></figure><p>Another option is to use the API to fetch up-to-date prices.</p><pre>from mypy_boto3_pricing import PricingClient<br><br>pricing: PricingClient = boto3.client(&#39;pricing&#39;)<br>pricing.get_products(<br>    ServiceCode=&#39;AmazonS3&#39;,<br>    Filters=[ <br>      { <br>         &quot;Field&quot;: &quot;regionCode&quot;,<br>         &quot;Type&quot;: &quot;TERM_MATCH&quot;,<br>         &quot;Value&quot;: &quot;us-east-1&quot; # Change to your favourite region.<br>      }<br>    ]<br>)</pre><p>The S3 pricing is sophisticated, but in this article, we’re interested in two cost components. Depending on the <a href="https://aws.amazon.com/ru/s3/storage-classes/">S3 Storage class</a> of the object, we pay different rates for:</p><ul><li>Storage (pay for GB per month)</li><li>Requests [pay for 1000 requests, GETs, COPYs, Lifecycle transitions and data retrieval requests (change of storage classes)]</li></ul><p>For a given object, we know:</p><ul><li>its size</li><li>its <a href="https://aws.amazon.com/ru/s3/storage-classes/">storage class</a></li><li>how many requests of each type were done to it for a past month</li></ul><p>From that, we know how much this object costs us per month (the prices from step 0) and if it makes sense to delete or move it to another storage class.</p><h4>Step 1: Learn about the storage (and storage costs)</h4><p>The AWS API already supports retrieving all the metadata about specific S3 objects:</p><pre>import boto3<br>from mypy_boto3_s3 import S3Client<br><br>s3: S3Client = boto3.client(&#39;s3&#39;)<br>resp = s3.head_object(<br>    Bucket=&#39;my-bucket&#39;,<br>    Key=&#39;some/file.txt&#39;<br>)<br>print(resp[&#39;StorageClass&#39;], resp[&#39;ContentLength&#39;])</pre><p>This works if you need to fetch this metadata for thousands of objects. But what if your bucket has millions of objects, and you want this metadata to be updated daily? On a daily basis, this would cost us:</p><pre>object_count / 1000 * cost_of_1000_head_requests</pre><p>And it will also work painfully slow because of the amount of daily requests we might have to make to fetch all metadata.</p><p>Luckily, S3 already provides us with a great tool to dump all of this metadata daily. It’s called <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Inventory</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qkF8IN8WFEWZyn2FyD4Fag.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Mjhlx7-7EJhisgdDPXn17g.png" /><figcaption>Parquet is the preferred data format.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FmfgMdHo-FTPyrwB-WC6xw.png" /></figure><p>You can create per-bucket Inventory configuration in the <em>Management</em> section of S3 bucket. When enabled, S3 generates a report with the following file structure. Each timestamp prefix contains a manifest with a list of files from data/ folders that represent the state of the bucket for a given timestamp. data/contains files. (Note: This is in the format that you picked during configuration. I picked <a href="https://parquet.apache.org/">Apache Parquet</a>.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*wUvJdFcP8Bo1_achYNlgCg.png" /><figcaption>This is the prefix that is updated by the S3 Inventory.</figcaption></figure><p>To read this report, you can use <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html">AWS Athena</a> or manually parse the manifest for a given data and read corresponding files. I use Apache Spark to read and process the files (and assume that SparkSession spark exists) because their size can be significant.</p><pre>from mypy_boto3_s3 import S3ServiceResource<br>import json<br><br>s3 = boto3.resource(&#39;s3&#39;)<br><br>object = s3.Object(<br>    &#39;some-bucket&#39;,<br>    &#39;.../2023-07-22T01-00Z/manifest.json&#39;).get()[&#39;Body&#39;].read()<br>manifest = json.loads(object)<br><br>bucket = manifest[&#39;destinationBucket&#39;][len(&#39;arn:aws:s3:::&#39;) :]<br>file_keys = [f[&#39;key&#39;] for f in manifest[&#39;files&#39;]]<br>files = [f&#39;s3://{bucket}/{key}&#39; for key in file_keys]<br><br># We use Apache Spark to read manifest files. You can use<br># any other approach that is suitable for you.<br>s3_inventory = spark.read.parquet(*files)<br># Here we preprocess this data and store the result as parquet dataframe.<br>(<br>    s3_inventory.withColumn(&#39;parts&#39;, F.split(F.col(&#39;key&#39;), &#39;/&#39;))<br>    .withColumn(&#39;full_path_length&#39;, F.size(F.col(&#39;parts&#39;)))<br>    .select(<br>        F.explode(F.sequence(F.lit(0), F.size(F.col(&#39;parts&#39;)))).alias(&#39;path_length&#39;),<br>        F.array_join(F.slice(F.col(&#39;parts&#39;), F.lit(1), F.col(&#39;path_length&#39;)), &#39;/&#39;).alias(<br>            &#39;prefix&#39;<br>        ),<br>        (F.col(&#39;path_length&#39;) == F.col(&#39;full_path_length&#39;)).alias(&#39;is_file&#39;),<br>        F.col(&#39;bucket&#39;),<br>        F.col(&#39;storage_class&#39;),<br>        F.col(&#39;intelligent_tiering_access_tier&#39;),<br>        F.col(&#39;size&#39;),<br>    )<br>    .write<br>    .parquet(...)<br>)</pre><p>We have a daily updated dataset that contains metadata for all S3 objects in the given bucket. Note that for an object with key a/b/c.txt, the resulting DataFrame will have 4 rows with path_length equal to 0, 1, 2, and 3 along with corresponding prefixes a, a/b, and a/b/c.txt. This is required to group this data frame by prefix and aggregate total sizes and storage costs.</p><p>Moreover, the logs contain the following important fields:</p><ul><li>bucket and prefix — the object that is being manifested</li><li>storage_class — the S3 Object storage class</li><li>intelligent_tiering_access_tier — if storage_class is Intelligent-Tiering, this field defines the tier</li><li>size — the file size</li></ul><h4>Step 2: Collect object access data</h4><p>The S3 bucket <em>Properties</em> section provides <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiFksP88KSAAxVfEFkFHV6lAJEQFnoECBoQAQ&amp;url=https%3A%2F%2Fdocs.aws.amazon.com%2FAmazonS3%2Flatest%2Fuserguide%2FServerLogs.html&amp;usg=AOvVaw3MTCcb_TAojrCtxm-fu3Cb&amp;opi=89978449">S3 Server access logs</a> settings, where you can configure exporting S3 object access events. They are exported as CSV files with about 1 minute latency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ygdg91ns01QOFz3-6-2fiw.png" /><figcaption>S3 Server access logging settings.</figcaption></figure><p>The format of the logs is described <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html">here</a>. Here’s how to read the logs using Apache Spark:</p><pre>RAW_CSV_SCHEMA = schema = StructType([StructField(f&#39;_c{idx}&#39;, StringType()) for idx in range(25)])<br><br>def read_bucket_logs(spark: SparkSession, bucket: str, path: str, day: date):<br>    return (<br>        spark.read.option(&#39;header&#39;, False)<br>        .option(&#39;delimiter&#39;, &#39; &#39;)<br>        .schema(RAW_CSV_SCHEMA)<br>        .csv(os.path.join(path, f&#39;{day.isoformat()}-*&#39;))<br>        .select(<br>            f.col(&#39;_c0&#39;).alias(&#39;owner&#39;).cast(StringType()),<br>            f.col(&#39;_c1&#39;).alias(&#39;bucket&#39;).cast(StringType()),<br>            f.to_timestamp(<br>                f.substring(f.concat(f.col(&#39;_c2&#39;), f.lit(&#39; &#39;), f.col(&#39;_c3&#39;)), 2, 26),<br>                &#39;dd/MMM/yyyy:HH:mm:ss Z&#39;,<br>            ).alias(&#39;time&#39;),<br>            f.col(&#39;_c4&#39;).alias(&#39;ip&#39;).cast(StringType()),<br>            f.col(&#39;_c5&#39;).alias(&#39;requester&#39;).cast(StringType()),<br>            f.col(&#39;_c6&#39;).alias(&#39;request_id&#39;).cast(StringType()),<br>            f.col(&#39;_c7&#39;).alias(&#39;operation&#39;).cast(StringType()),<br>            f.col(&#39;_c8&#39;).alias(&#39;key&#39;).cast(StringType()),<br>            f.col(&#39;_c9&#39;).alias(&#39;request_uri&#39;).cast(StringType()),<br>            f.col(&#39;_c10&#39;).alias(&#39;http_status&#39;).cast(IntegerType()),<br>            f.col(&#39;_c11&#39;).alias(&#39;error_code&#39;).cast(StringType()),<br>            f.col(&#39;_c12&#39;).alias(&#39;bytes_sent&#39;).cast(LongType()),<br>            f.col(&#39;_c13&#39;).alias(&#39;object_size&#39;).cast(LongType()),<br>            f.col(&#39;_c14&#39;).alias(&#39;total_time_millis&#39;).cast(LongType()),<br>            f.col(&#39;_c15&#39;).alias(&#39;turnaround_time_millis&#39;).cast(LongType()),<br>            f.col(&#39;_c16&#39;).alias(&#39;referer&#39;).cast(StringType()),<br>            f.col(&#39;_c17&#39;).alias(&#39;user_agent&#39;).cast(StringType()),<br>            f.col(&#39;_c18&#39;).alias(&#39;version_id&#39;).cast(StringType()),<br>            f.col(&#39;_c19&#39;).alias(&#39;host_id&#39;).cast(StringType()),<br>            f.col(&#39;_c20&#39;).alias(&#39;signature_version&#39;).cast(StringType()),<br>            f.col(&#39;_c21&#39;).alias(&#39;cipher_suite&#39;).cast(StringType()),<br>            f.col(&#39;_c22&#39;).alias(&#39;authentication_type&#39;).cast(StringType()),<br>            f.col(&#39;_c23&#39;).alias(&#39;host_header&#39;).cast(StringType()),<br>            f.col(&#39;_c24&#39;).alias(&#39;tls_version&#39;).cast(StringType()),<br>        )<br>        .withColumn(&#39;day&#39;, f.lit(day))<br>        .withColumn(&#39;bucket&#39;, f.lit(bucket))<br>    )</pre><p>With this data frame, we can do the same trick as with inventory, where we aggregated the dataset by path and path_length. Note that the logs contain the following important fields:</p><ul><li>bucket and prefix — the object that is being manifested</li><li>time — the timestamp when the request occurred</li><li>requester — the IAM role of the subject, who made the request</li><li>operation — the S3 operation</li></ul><h4>Step 3: Join and visualise</h4><p>We ended up with two datasets:</p><ul><li><strong>Inventory dataset</strong>: keyed by bucket and prefix. Contains aggregate storage sizes per each storage class. Costs for storage per month storage_cost_per_month. (Remember Step 0? Multiply the storage sizes by the cost per storage class, and you’ll get the storage cost.)</li><li><strong>Server access logs dataset</strong>: keyed by bucket and prefix. Contains an aggregate number of <a href="https://dev.to/aws-builders/aws-s3-understanding-s3-billing-usage-for-beginners-3ple">Tier 1 and Tier 2 requests</a> per path. requests_cost_per_month costs for requests to prefix per month. (Multiply the number of requests by the cost per 1000 requests from Step 0.) log_access_count_30d is the logarithm of the number of requests for the given prefix.</li></ul><p>When we join the datasets by bucket and prefix, we’ll have all the data to visualize the bucket costs. Furthermore, we add a column parent_prefix , which is equal to the folder, that contains the folder/object in prefix (e.g., for prefix=a/b/c parent_prefix=a/b).</p><p>Let’s use the <a href="https://plotly.com/python/plotly-express/">plotly</a> library to visualize the storage costs. The following call creates the visualization. Note that you can filter the df with a particular prefix and limit the depth (the number of inner folders) of visualization, as the rendering can be taxing for the web browser.</p><pre>import plotly.express as px<br><br>px.treemap(df, <br>    names=&#39;prefix&#39;,<br>    parents=&#39;parent_prefix&#39;,<br>    values=&#39;storage_cost_per_month&#39;,<br>    color_continuous_scale=&#39;RdBu&#39;,<br>    color=&#39;log_access_count_30d&#39;,<br>    branchvalues=&#39;total&#39;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FnOJk5SWSODs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnOJk5SWSODs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnOJk5SWSODs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ca92f30f8a172a72568c292cacb75e5/href">https://medium.com/media/9ca92f30f8a172a72568c292cacb75e5/href</a></iframe><p>The same trick can be done with requests (just replace storage_cost_per_month with requests_cost_per_month):</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F7-nyNyO-J0c%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D7-nyNyO-J0c&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F7-nyNyO-J0c%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/99404545226285d37f0dccf077b58440/href">https://medium.com/media/99404545226285d37f0dccf077b58440/href</a></iframe><p><strong>Did you notice the grey rectangles in the storage costs visualization?</strong> Grey comes from the color argument of the px.treemap call and means that the prefix hasn’t been accessed for the past month. These prefixes are great candidates for removal or another storage class transition.</p><h4>Conclusion</h4><p>The proposed way of visualization of S3 bucket costs can save a significant amount of funds by making S3 observability more granular. I managed to save dozens of thousands of dollars on cloud bills using these data sources, and at Constructor, we managed to save up to 40% of S3 costs using this technique. I suggest you try it as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d921328306e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/how-to-optimize-aws-s3-costs-via-granular-visualisation-d921328306e9">How to optimize AWS S3 costs via granular visualisation</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Attribute Enrichment at Constructor]]></title>
            <link>https://medium.com/constructor-engineering/introducing-attribute-enrichment-at-constructor-143ed55ed765?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/143ed55ed765</guid>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[ecommerce-web-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Serhii Yevtukhov]]></dc:creator>
            <pubDate>Wed, 26 Jul 2023 14:17:03 GMT</pubDate>
            <atom:updated>2023-07-26T14:17:03.693Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*61pSFtF0a5EJtrKstfpLFw.png" /></figure><p>Bad product data makes it hard for customers to find what they need on websites, even with well formulated search queries or specific search filters. This happens when products don’t have searchable attributes or complete and correct tags.</p><p>Working with <a href="https://constructor.io/customers/">some of the biggest retailers in the world</a>, Constructor is continually developing new solutions to help ecommerce teams run successful businesses and improve their customer experience. We got tired of poor product attribute data leading to poor results, and so we set out to improve it in a way that wouldn’t only make search, filtering, and product discovery better, but could optimize a B2C or B2B e-commerce company’s entire value chain.</p><blockquote>Enter <a href="https://constructor.io/products/product-data-enrichment">Attribute Enrichment</a>: Constructor’s new solution to automatically generate and fix product attribute data using a mix of Constructor’s existing best-in-class AI and new innovations in the fields of machine vision and text classification.</blockquote><p>Attribute Enrichment automatically tags a customer’s products with new relevant attributes and product categories. Enriched product attributes enable better search and browse experiences, leading to increased conversions and revenue and minimizing returns from people mistakenly buying products based on incomplete or incorrect data. On top of those direct lifts, Attribute Enrichment also provides valuable input to other systems like Product Information Management (PIM) platforms and improves operational efficiency, saving time and effort for employees who no longer have to manually enter this information.</p><p>We know a lot of companies are making big claims around AI these days without a lot of substance, so we wanted to dive into how Attribute Enrichment works from the technical side. In this article, we give an overview of Constructor’s Attribute Enrichment engine, including overall system design, data we work with, and challenges we tackled while building the system.</p><h3>Methodology</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ijXmNq4RXm8Lqubs02o2Ng.png" /></figure><p>Our Attribute Enrichment engine leverages machine learning and deep learning techniques under the hood to predict both product categories and product attributes based on image data, text data, and clickstream data.</p><h4>Available data</h4><p>Data is an essential part of Attribute Enrichment as we train our models on it. As inputs, we take raw data from an ecommerce company’s product catalogs, along with the ecommerce website’s behavioral clickstream data.</p><p>Clickstream data contains important user/website and user/app interactions including search queries, browse filters, shopping sessions, clicks on search results, add-to-carts, purchases, etc. Currently, Attribute Enrichment relies heavily on search query history to detect high-demand attributes and prioritize them during enrichment.</p><p>Because clickstream has always been critical to Constructor’s algorithms, we’ve collected an incredibly large amount of this data across many industries and countries over many years, with years of experience ensuring the clickstream data we receive is correct and trustworthy. Fusing this clickstream data with data from product catalogs gives us the ability to predict the most accurate and relevant attributes for every product.</p><p>Customers send us product catalog data on a daily basis. Typically, this data set contains product categories, SKUs, product titles, descriptions, images, and product variations and categories for each product, along with various product attribute data like color, size, or material. However, it’s not uncommon for this attribute data to be missing or broken.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdu-qNJHQZTGV18_ok87ag.png" /></figure><h4>Attributes we predict</h4><p>Constructor works with some of the largest ecommerce companies in the world, both in B2C and B2B and spanning many different domains, including but not limited to apparel fashion, grocery, home decor, pet supplies, B2B manufacturing/distribution, and more. Customers across the same domain have similar product taxonomies, attributes, and attribute values.</p><p>We began this new product by focusing on fashion and home, as those domains<strong> </strong>both displayed the most interest and the most robust models already trained. But as we are constantly developing our models for other domains and adapting them to customer needs, we have since applied Attribute Enrichment to everything from eyeglasses and frames to grocery to esoteric construction supplies.</p><p>We define attribute taxonomy as a tree of the following structure: domain &gt; product category &gt; attribute &gt; attribute values. Below are several examples of attributes we extract in different domains (though this list is certainly not exhaustive).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*y0bz92BQUKONMUAVqmtWmg.png" /></figure><h4>Enrichment methods</h4><p>Every e-commerce website contains textual and visual information on available products. To extract that information, we leverage three approaches:.</p><ul><li><strong>Image-based approaches</strong> are an obvious choice for attribute enrichment due to the availability of corresponding images for all items. We use a cadre of different visual enrichment methods based on pretrained and fine-tuned neural networks, zero-shot vision models, and others.</li><li><strong>Text-based approaches</strong> are also super helpful here as lots of product information is represented in textual form, e.g. product titles, descriptions, reviews, categories, and tags. We developed a variety of methods from very basic (like those based on regular expressions or token matching) to more advanced NLP techniques like transformer-backed or GPT-based question answering.</li><li><strong>Multimodal approaches</strong> can combine both visual and textual information for learning and inference.</li></ul><p>The choice of approach depends on the specific needs of each customer. For instance, textual information plays a significant role in domains like grocery and electronics where ingredients or tech specs matter most, while visual information is more crucial for apparel and home decor.</p><p>At the end of the day, it’s most important for us to find and enrich attributes <em>that are important for e-commerce businesses and their customers</em>. We use clickstream data to achieve this. While businesses may have their own theories of what attributes are important for shoppers, the customers themselves know the exact answer to this question. Through historical analysis of search queries, we can discover the most necessary attributes and attribute values for users (as well as product categories) and enrich them.</p><p>Finally, rather than limiting ourselves to existing methods, we are constantly improving our models and researching new approaches that work better for our customers. We’ll cover all of them in detail in the next article soon — stay tuned.</p><h3>Challenges</h3><p>There are a number of challenges in product data enrichment that can hinder accurate identification of product or category attributes.</p><h4>Lack of domain understanding</h4><p>In fashion, for example, understanding important attributes like fabric type, sleeve length, or neckline requires domain expertise (what is a cap sleeve? A boat neck?). Without this knowledge, it becomes difficult to enrich product descriptions with the appropriate attributes.</p><p>Below are some examples of fabric patterns (first two products) and neckline (the last one) which are hard to figure out without being a fashion expert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n5D-1VyvIh_YdTIZcv_rGQ.png" /></figure><p>To address this challenge, we can take several approaches:</p><ul><li><strong>Investigate the Domain.</strong> Explore customers’ websites and their competitors’ sites to gain insights into how attributes are described and presented. This can provide valuable information about common attributes used in the domain.</li><li><strong>Research Customer Search Queries.</strong> Conduct research on customers’ search queries to understand what attributes or product features they search for the most. This helps in identifying important attributes from the customers’ perspective.</li><li><strong>Collaborate with Experts.</strong> Engage with domain experts who have a deep understanding of the specific industry or product domain. Collaborating with these experts can provide valuable insights and ensure that attribute enrichment efforts align with industry standards and customer expectations.</li><li><strong>Leverage External Sources of Information</strong>. Use tools like GPT as a dense source of information from the internet. This can help in acquiring a broader understanding of the domain and gathering relevant information for Attribute Enrichment.</li></ul><p>By combining these approaches, it is possible to bridge the gap in domain understanding and improve the accuracy of attribute identification and annotation for better attribute enrichment outcomes.</p><h4>Lack of annotated data</h4><p>Another challenge in attribute enrichment is the limited availability of annotated data. Not all of our customers provide complete or accurate annotations, which makes it difficult to enrich attributes effectively (which explains why our customers wanted Attribute Enrichment in the first place!).</p><p>Sure, we could hire people to label images or texts, but it can be costly because the ecommerce field is vast and has many unique cases. Instead, it’s important to focus on showing the model the most difficult cases.</p><p>Active learning methods can help with this by selecting samples that the model is less sure about. To determine these probabilities, we can use pretrained models that can make predictions or collect a small amount of data to train a model without external help. Even if it’s not perfect, this can be a starting point to identify the most challenging samples.</p><h4>Huge attribute taxonomy</h4><p>As Constructor works with many customers in many different domains, attribute taxonomy grows very quickly. Every customer has from tens to thousands of unique product categories. Each unique category has its own set of attributes, and each attribute has a list of values associated with it.</p><p>The sheer magnitude of this taxonomy doesn’t give us a chance to develop specific models to specific categories to address specific customer needs. Instead, we are forced to look for generic yet efficient techniques which can be transferred between domains and categories. Here is where deep learning comes in.</p><p>Pretrained neural networks are able to interpret our data and extract low-level features from it. All we need is to fine-tune them for specific attributes. This approach we use to enrich attributes for apparel and fashion. We also have a team of data annotators who label products with relevant attributes, and then later use these annotations to fine-tune backbone models.</p><h3>System Design</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XXLUINLqu52YUAqu" /></figure><p>Our Attribute Enrichment system begins with the product catalog that a customer sends us daily, which is a major piece of data for model training and validation. To obtain even more quality data we involve a data annotation team whose goal is to manually annotate products with relevant attributes and tags for further model training.</p><p>To train ML &amp; DL models on a regular basis, we use data pipelines built on <a href="https://github.com/spotify/luigi">Spotify’s Luigi</a>. Then enrichment pipelines use these models for inference. For example, we have enrichment pipelines to predict product category, colors, apparel attributes, text entity recognition, and others. Enrichment pipelines run on a daily basis, taking the most recent catalog and predicting new attributes for products. Thus we have fresh attributes even if a catalog has recently been updated.</p><p>The enrichment pipelines save the attributes into a MySQL database. Another microservice builds search indices for production use and delivers them right to the search backend.</p><p>Our search backend is largely microservices implemented in Rust, C++ and Python and designed for high-performance index data access. It takes each processed search query as an input and returns relevant products from a search index that matches the query.</p><p>The search backend reloads search indices when they’re updated. So when a new search index with enriched attributes is built, the search backend uploads it immediately and serves search requests with this new index under the hood.</p><h3>Results</h3><h4>Customer dashboard</h4><p>A customer dashboard serves as a valuable tool for merchandisers, allowing them to review and manage enriched attributes that impact search and browse functionalities. This is necessary because no model can achieve 100% accuracy, and our domain knowledge can be limited.</p><p>The purpose of the customer dashboard is twofold:</p><ul><li><strong>Reviewing Enriched Attributes.</strong> Merchandisers should have the ability to review the attributes that have been enriched. This allows them to assess the accuracy and relevance of the attributes that influence the search and browse experience for customers. By reviewing these attributes, merchandisers can leverage their domain expertise to identify any inaccuracies or irrelevant values. We also make sure this work creates more global value as it feeds back into our Attribute Enrichment system, helping ensure we both produce better results in the future and telling us about the attribute preferences of each individual company we work with.</li><li><strong>Attribute Management.</strong> The dashboard should provide functionality for merchandisers to remove attribute values that they consider irrelevant. This action should have a direct and timely impact on search and browse functionalities, ensuring that customers are presented with the most relevant and accurate results. Modifications made here also help train the system.</li></ul><p>To fulfill these requirements, we have developed a dedicated web application as a separate tool for merchandisers. The application presents a user-friendly table where merchandisers can access a comprehensive overview of items and their variations. Each item is associated with its unique attribute values, and merchandisers have the ability to remove or correct any irrelevant values as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hsoL19KRUdcpuv8Z" /></figure><p>Once a review session is completed and customers have made edits in the dashboard, it is essential that the changes take effect on search and browse functionalities in a short period of time. To achieve this, we trigger an index building process. This process ensures that the updated attribute information is reflected accurately in the search index, allowing our customers to experience the impact of the attribute modifications promptly.</p><h4>Enrichment examples</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*trbU5NDPSdPRfw4NnhUVQA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDfTX66NUio9dmjlP39_Zw.png" /></figure><h3>Conclusion</h3><p>Constructor’s Attribute Enrichment is a relatively new product offering for us, but with the early results we’ve seen, we believe that there is a real opportunity to help customers further augment their product data by leveraging powerful deep learning techniques. For too long, bad product data was an excuse search engines gave for bad results. We want to remove that excuse and ensure attractive, revenue-optimizing results, even for product catalogs with poor data.</p><p>In the next installment in this series, we’ll cover in detail some machine learning and deep learning techniques we use to enrich products with new attributes.</p><p>We’re constantly improving our enrichment models, covering new domains and attributes to address customer needs. Our goal is to make sure end users find what they are looking for, they are satisfied with the whole experience and end their sessions by easily finding and purchasing products they’ll love.</p><p>Ready to learn more about Attribute Enrichment? See our product page <a href="https://constructor.io/products/product-data-enrichment">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=143ed55ed765" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/introducing-attribute-enrichment-at-constructor-143ed55ed765">Introducing Attribute Enrichment at Constructor</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Store Design at Constructor]]></title>
            <link>https://medium.com/constructor-engineering/feature-store-design-at-constructor-330b65f64b18?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/330b65f64b18</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[feature-store]]></category>
            <dc:creator><![CDATA[Dmitry Persiyanov]]></dc:creator>
            <pubDate>Wed, 18 Jan 2023 15:46:24 GMT</pubDate>
            <atom:updated>2023-01-19T14:56:21.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BErWNg8jmH1kaKRNiphZA.jpeg" /></figure><h4>How we compute, store, and deliver ranking signals for our ML</h4><p>One of the central components in product ranking at Constructor is a Feature Store. It is a centralized storage of ML features. The Feature Store has two APIs: offline and online. The offline API is used during the model train phase and experiments, while the online API is requested by our production Machine Learning models to retrieve features before making a prediction.</p><p>In this article, we’ll discuss feature challenges in real-time Machine Learning, walk through the current design of the Feature Store in Constructor with code examples, go over key motivations and decision drivers behind it, show how Data Science experimentation workflow benefits from the Feature Store API, and discuss our future plans.</p><p><strong>Table of Contents<br></strong>· <a href="#5504">A Recap of Product Ranking at Constructor</a><br>· <a href="#455a">Feature Challenges in Real-Time ML</a><br> ∘ <a href="#a285">Motivations</a><br> ∘ <a href="#8f41">E-Commerce ML Ranking Overview</a><br>· <a href="#c429">Existing Approaches for Feature Delivery</a><br> ∘ <a href="#f4d7">Features Logging</a><br> ∘ <a href="#f131">Feature Store</a><br>· <a href="#5079">Constructor’s Feature Store</a><br> ∘ <a href="#77e2">Requirements</a><br> ∘ <a href="#f398">Offline Batch Layer</a><br> ∘ <a href="#9695">Online Serving Layer</a><br> ∘ <a href="#9b68">Ranking Service</a><br>· <a href="#167e">Results</a><br>· <a href="#2a9b">Future Plans</a></p><h3>A Recap of Product Ranking at Constructor</h3><figure><img alt="Overview of Constructor’s Search Engine" src="https://cdn-images-1.medium.com/max/1024/0*-_XZ5in274EX1pE1" /><figcaption>Overview of Constructor’s Search Engine</figcaption></figure><p>If you haven’t read <a href="https://medium.com/constructor-engineering/product-ranking-at-constructor-3c3ae44791f5">Product Ranking at Constructor</a>, please go ahead and read it!</p><p><strong><em>Terminology Disclaimer</em></strong></p><ul><li><strong>Customer</strong> refers to a business that uses Constructor, i.e. our client</li><li><strong>Shopper or user </strong>refers to a particular user on a website performing some product discovery activity</li></ul><p>Serving HTTP requests at Constructor has a lot of requirements, such as real-time personalization (no caching), high volume, high performance, and unique data sets per customer.</p><p>Here is the high-level flow when we receive a search HTTP request:</p><ul><li>Retrieve a set of matching products with an initial ranking</li><li>Send top-K candidates to the Ranking Service for enhanced ML ranking and reranking</li><li>Apply Business/Merchandising rules to results</li></ul><p>First, the Retrieval System takes a query and returns a matching set of candidate products with initial lightweight ranking. Second, Top-K candidates are sent to the Ranking Service. Using dozens of computationally complex features, the Ranking Service calculates all the features needed for an ML model and runs model inference to score candidates. Predictions are returned back to the Backend Service for subsequent reranking. Finally, Business/Merchandising rules (slotting, boosting, and burying) are applied, affecting the results order for the last time. Merchants create rules to infuse business specifics into product rankings, for example to handle sales or promotions or increase visibility of new products.</p><p>In the rest of this article, we’ll concentrate on what happens inside the Ranking Service. In particular, we’ll discuss common feature challenges in online ML, serving hundreds of models with varying configurations, and how we solved feature calculation, delivery, and storage via our Feature Store.</p><h3>Feature Challenges in Real-Time ML</h3><h4>Motivations</h4><p>As mentioned in <a href="https://medium.com/constructor-engineering/product-ranking-at-constructor-3c3ae44791f5">Product Ranking at Constructor</a>, when deciding to build the Ranking Service, we had several motivations to address:</p><ul><li><strong>Improve time-to-market for ranking improvements</strong> (from months to days)</li><li><strong>Support more ranking signals</strong> (&lt;10 to 10–100)</li><li><strong>Support almost any ML ranking model</strong> (e.g., Gradient Boosted Trees, Linear Models, Neural Networks)</li><li>Have a flexible, <strong>customer-specific model configuration system</strong></li><li>Have a unified way to <strong>handle customer-provided ranking factors</strong></li></ul><p>These motivations arose for various reasons. The implementation of the “ranking part” in the retrieval system was built with specific ranking factors in mind, such as token-to-product relevance and lack of extensibility. Flexible configuration was needed due to Constructor serving <strong>many customers</strong>, and we wanted to serve <strong>many models per customer</strong> (e.g., for A/B testing or serving in different contexts). Moreover, every customer has a unique combination of domain specifics and patterns in shopper behavior, which leads to <strong>different optimal model configurations and feature sets</strong>.</p><h4>E-Commerce ML Ranking Overview</h4><p>Let’s take a concrete case as an example.</p><p>We want to train an ML model that predicts the probability of a conversion event like add-to-cart or purchase. A training dataset is constructed using request logs (search results shown in the past) along with behavioral logs (user activity such as an event of searching, clicking, adding a product to cart). The dataset row is a logged product shown in a specific request (request has associated timestamp, search query, user ID, and other info). Label is a binary indicator whether an interaction has happened within T seconds after request was made.</p><figure><img alt="E-Commerce Ranking Dataset Example" src="https://cdn-images-1.medium.com/max/1024/0*8BuFfNvFns6r6OBq" /><figcaption><em>E-Commerce ML Ranking Dataset Example</em></figcaption></figure><p>When training the model against the dataset, we want to utilize various kinds of features that our data can express. For example:</p><ul><li>Product metadata (categories, facets, sizes, stock levels, colors etc)</li><li>Behavioral statistics over any period of time (global product CTRs, query-product CTRs, query-category CTRs, etc.)</li><li>Real-time user history</li><li>User metadata (geographical region, device)</li></ul><p>Once a model is trained by an offline batch process, we save the artifact to cloud storage like S3 or GCS. The Ranking Service regularly picks up new artifacts for serving.</p><p>When it comes to online serving of ranking models, a few questions arise:</p><ul><li><strong>How do we get features for every request? Where do we store &amp; retrieve them from?</strong></li><li><strong>How do we know what features the model needs?</strong> We have many customers and hundreds of models served in parallel and theset of features varies for every model so it can’t be hardcoded</li><li><strong>How do we ensure consistency between the feature values model observed during the train phase and what we see now in serving?</strong></li></ul><p>In order to address all the motivations and answer questions above, we adopted some ideas from the <a href="https://www.featurestore.org/">Feature Store</a> concept and implemented a solution that covered all our needs at that point.</p><h3>Existing Approaches for Feature Delivery</h3><p>Let’s discuss two approaches for feature delivery that are popular in the industry. We don’t use them <em>as is,</em><strong><em> </em></strong>but talking through them will help us to understand design choices made for Constructor’s Feature Store.</p><h4>Features Logging</h4><figure><img alt="Features Logging for Feature Delivery" src="https://cdn-images-1.medium.com/max/1024/1*kwyH_ShmHa3u5Qf_rn5KLQ@2x.png" /><figcaption>Features Logging for Feature Delivery</figcaption></figure><p>One approach is what we call <strong>Features Logging</strong>. In this approach, the Feature Calculation code lives on the backend side, close to model inference. Usually, that code uses additional features data (e.g., product CTRs or user history) saved in an external storage such as MySQL or Redis. When a request comes in, features are calculated and then sent to request logs. Since the training dataset is constructed from request logs, it has features available straight away. Once the model is picked up for serving, the backend runs the same feature calculation code and calls model inference.</p><p>In experiments, data scientists have two options for adding features to the model:</p><ul><li>Use features that are already available in request logs</li><li>Construct new features using custom offline code</li></ul><p>With the first approach, you are limited to only the features that are already logged and their derivations. With the second approach, when moving to production you’ll need to either rewrite offline code for serving or have duplicate feature logic for training and serving. Both options come with disadvantages. When re-implementing a feature for serving, you have double the work and backfilling problems (like needing to run code live for weeks to collect logged feature values). In the case of duplicate logic, there’s always a chance for train-inference mismatch that might ruin online model quality.</p><p>Feature logging is a good approach when you have a bunch of heuristics already in your codebase and want to apply ML using them as features — or when you want to use request-time information for features. Log them and use them for training. As you scale to more features, challenges with train-inference mismatch, backfilling, and double work arise, making the feature logging approach less attractive.</p><p>We opted for a feature logging approach for real-time features, i.e. the ones that have sub-hour update frequency.</p><h4>Feature Store</h4><figure><img alt="Feature Store for Feature Delivery" src="https://cdn-images-1.medium.com/max/1024/0*4zmMvEdmuF32IMP7" /><figcaption>Feature Store for Feature Delivery</figcaption></figure><p>Another approach that’s become popular in the last few years is the <a href="https://www.featurestore.org/"><strong>Feature Store</strong></a>. There are several differences from the feature logging approach:</p><ul><li><strong>Responsibility for the feature calculation is shifted from the online to offline environment. </strong>This allows feature code to be written once in an offline environment, and its result will be available for both training and serving.</li><li><strong>Features are decoupled from particular log streams or datasets. </strong>Features are defined for particular entities and have specific time granularity. This allows joining features to any dataset during training with point-in-time correctness. This also allows backfilling features.</li><li><strong>Feature Storage is centralized. </strong>This allows features to be reused by different consumers for different tasks.</li></ul><p>One of the core pieces in the Feature Store that allows having a source of truth in an offline environment is a streaming engine. In order to support real-time features such as up-to-date user history for any point in time, events need to be processed by a streaming engine such as Apache Spark or Apache Flink.</p><p>In the rest of this article, we’ll go through Constructor’s Feature Store design and discuss our hybrid approach that has allowed us to utilize most of the FS principles while supporting real-time features without having a streaming engine in our architecture.</p><h3>Constructor’s Feature Store</h3><h4>Requirements</h4><p>Here’s a list of our key decision-making drivers for the Feature Store solution:</p><p><strong>Usability</strong></p><ul><li>Minimized code duplication (reducing chances of train-inference mismatch)</li><li>Convenient DS experiment workflow (maximize iteration speed from idea to prod) for both ad-hoc experiments and scheduled runs</li><li>Convenient configuration engine<strong> </strong>that supports many customers and many models per customer</li></ul><p>We focused on usability since our ML teams are small but we have many customers. We need to iterate very quickly to deliver massive value across the whole customer base.</p><p><strong>Efficiency</strong></p><ul><li>Features are stored once, then reused by multiple models</li><li>Store 100s of millions of feature values with minimal increase in infra costs</li><li>Read latency for 1K feature values as P99 &lt; 10ms</li></ul><p>We focused on efficiency because our Backend Service sets a high standard for response time percentiles. Our customers have already gotten used to Constructor’s great performance, and we didn’t want to change expectations as we rolled out the Ranking Service. We set SLO for Ranking Service response time as P99 &lt; 100ms. Since Feature Store lookup is just one step in ranking service processing, it should be blazingly fast so we could put most of the complexity into the model.</p><p><strong>Considerations</strong></p><ul><li>Few real-time features, many daily-updated features</li><li>Reuse existing infrastructure if possible</li></ul><p>We were ready to step aside from state-of-the-art and take advantage of the specifics of our existing state in order to bring the Ranking Service to life as soon as possible.</p><h4>Offline Batch Layer</h4><figure><img alt="Constructor’s Feature Store: Offline Batch Layer" src="https://cdn-images-1.medium.com/max/1024/0*sWw0lr3yinTfK6AR" /><figcaption>Constructor’s Feature Store: Offline Batch Layer</figcaption></figure><p><strong>Entity, Feature Spec</strong></p><p>Every feature definition consists of three components:</p><ul><li>A List of Entities</li><li>Type (e.g. float, int, str)</li><li>Name</li></ul><p>Entities are dictated by business domain, and in our case include but are not limited to:</p><ul><li>user</li><li>query</li><li>browse page</li><li>product</li><li>product variation</li></ul><p>Every particular instance of an entity has a <em>join_key</em> — column which is used as an identifier (e.g. <em>user_id</em> for user, <em>product_id</em> for product, <em>query</em> itself for query).</p><p>Such design allows for declarative definitions of features. This ultimately helps us to talk about features in the code, communicating what features a model needs, what features to retrieve from FS, and what features are available in FS.</p><p><a href="https://gist.github.com/persiyanov/b7209ac3e729d15546514c5bb4b26a68#file-fs_article-py-L1-L56">As an example</a>, a feature like “product CTR over the last 7 days” is defined for a single “product” entity, has type “float,” and has a name like “product_ctr_7_days”.</p><p><strong>Feature Registry</strong></p><p>A set of batch jobs (<a href="https://github.com/spotify/luigi">Luigi</a> PySpark tasks) calculate features in the offline environment on a daily basis. Each job’s output is a Parquet table with multiple feature columns. For example, a product CTR job saves CTRs for different actions (clicks, add-to-carts, purchases) into separate columns. Such groups of features are called <strong>Feature Groups.</strong></p><p><strong>Feature Registry</strong> is a Python class that keeps track of Feature Groups. Every day, a batch job goes through all feature groups in the registry and runs corresponding feature tasks for the current day’s partition. Thus offline, we have feature values available for the whole timeline and can backfill features for older dates.</p><p>All feature groups are registered in a declarative Python file (<a href="https://gist.github.com/persiyanov/b7209ac3e729d15546514c5bb4b26a68#file-fs_article-py-L59-L106">example here</a>).</p><p><strong>Feature Joiner</strong></p><p>Now, as we have all features registered in a single place, we need the Offline API to retrieve and join features to the dataset. For ranking, we’d have a dataset of logged results pages with an attached label indicating behavioral action presence. We want to enrich the dataset with features, e.g. product metadata, query-product CTRs, etc.</p><p>Since all the features conform to the same interface, it’s possible to write a generic logic that adds any set of features to a dataset. It takes a dataset and feature specs as input, and adds actual feature values with point-in-time correctness. Remember how we used <em>join_key </em>in entity definition? It is now used to match feature values with rows in a dataset. A component that does such a join is called <a href="https://gist.github.com/persiyanov/b7209ac3e729d15546514c5bb4b26a68#file-fs_article-py-L109-L115"><strong>Feature Joiner</strong></a><strong> </strong>and is also implemented as a Pyspark Luigi job.</p><p>As an example, suppose we have a ranking training dataset of form <em>(timestamp, request_id, user_id, query, product_id, position, label). </em>If we have a product CTR feature in the registry defined for PRODUCT entity, the way joiner would add it is:</p><ol><li>For dataset, calculate daycolumn based on timestamp</li><li>Calculate min(day)and max(day) in dataset</li><li>Read feature group data for a period from min(day) to max(day)</li><li>Left join dataset with features data on (product_id, day)</li></ol><p>Enriched with features, the dataset is final and can be used for training.</p><figure><img alt="E-Commerce ML Ranking Dataset enriched with Features" src="https://cdn-images-1.medium.com/max/1024/1*weTONHOcRDiYWwIY3icMjQ@2x.png" /><figcaption><em>E-Commerce ML Ranking Dataset enriched with Features</em></figcaption></figure><p><strong>Model Interface</strong></p><p>We are on the bridge with online part of the system, and questions we need to answer at this point are:</p><ul><li>After the model is picked up in the Ranking Service, how does it know what features the model needs?</li><li>How do we make the Ranking Service model agnostic to allow as much flexibility for DS as possible?</li></ul><p>To solve these questions, a <a href="https://gist.github.com/persiyanov/b7209ac3e729d15546514c5bb4b26a68#file-fs_article-py-L118-L134">simple model interface</a> is defined for all actual model implementation to inherit from.</p><p>Knowledge about features is carried in the model artifact. The Ranking Service doesn’t need to know any implementation details of the model, but it just needs to have the ability to a) load the artifact from cloud storage, b) get the required features list, and c) call the predict method.</p><p>This interface lives in a shared library and is used both during offline model training and Ranking Service.</p><p><strong>Basis and Processed Features</strong></p><p>In experiments, DS might want to apply some transformation on a feature and see if it improves metrics. For example, we might want to apply min-max normalization to a numerical feature before training the model, or calculate textual distance features based on query and product title.</p><p>How does this play with the Feature Store? Do we store transformation outputs in the FS as a separate feature or do we make this processing a part of the model itself?</p><p>We decided to keep both options open. On one hand, if reuse or better performance is needed, it makes sense to put the transformed value into the FS as a separate feature. On the other hand, some transformations are only possible at request-time, such as calculating textual distance mentioned above.</p><p>Features stored in the Feature Store are also referred to as <em>basis features (returned by ModelInterface.get_required_features(), serving as input in ModelInterface.predict()</em>), while features obtained by post-processing basis features are called <em>processed features. </em>For the examples above, the numerical feature is a basis feature, while min-max normalized feature is a processed one; product title is a basis feature, while textual distance is a processed one.</p><p><strong>Configuration</strong></p><p>As a consequence of bringing features to a uniform interface, we get quite a nice experience for DS. They can define a config file for any experiment with all the features to use in training. Once satisfied with the result, the same config is scheduled for production runs. All the configs are stored and versioned with the rest of the codebase.</p><pre>model:<br>    type: boosting<br>    n_estimators: 500<br>    max_depth: 10<br>transforms:<br>    - name: text_distances<br>      input_columns: [&#39;product_title&#39;, &#39;query&#39;]<br>dataset:<br>    name: main_ranking_dataset<br>    period: 30d<br>    sampling: uniform<br>basis_features:<br>    - &#39;product_title&#39;<br>    - &#39;qp_view_to_click_rate_30d&#39;<br>    - &#39;qp_view_to_add_to_cart_rate_30d&#39;<br>    - ...&lt;DS can add / remove features here&gt;...</pre><h4>Online Serving Layer</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gUcZclr6Me672Wu9" /><figcaption>Constructor’s Feature Store: Online Serving Layer</figcaption></figure><p>When it comes to the online part, a central question is the choice of what low-latency storage to use for request-time features retrieval.</p><p><strong>Index Service</strong></p><p>The<strong> Index Service</strong> is an in-house built service that manages multiple memory-mapped data structures. Data lives on disk and, thanks to memory mapping, doesn’t eat up a lot of RAM. The Index Service performs well when there’s a need for fast low-latency reads, but not for quick writes or updates. Data structures are updated regularly in a background by <em>Index Builders</em> and put on the disk for serving. One type of data structure the Index Service supports is a simple key-value, which is enough for online Feature Store use cases.</p><p>We compared Index Service with Redis, and both options had similar read performance, so for us the deciding factor was pricing. Based on our back-of-the-envelope calculations, going with Redis would mean an increase in total production costs of more than 10%. With the Index Service all data lives on disk, so cost increase was negligible.</p><p>After going to production with the Feature Store based on the Index Service, we saw even better read times than in our benchmark. Read latencies for production requests have P99 &lt; 25ms while storing 10x more feature values, and 10x more feature values are requested from FS for every call.</p><p><strong>Daily Batch Ingestion</strong></p><p>Every day, a Luigi task goes through all model configs scheduled for production, collects all the latest feature values needed by models, and pushes them to the cloud storage. Index Builders pick up new data and put prepared indexes in the <strong>Online Feature Store </strong>for serving.</p><p><strong>Real-Time Features Support</strong></p><p>Everything we discussed so far works for features with batch updates. Such features are usually based on a high period of aggregation or cover cases where we don’t need to react to changes immediately. For example, product CTRs based on the last 30 days of data can be updated daily.</p><p>But what about features where we do need to propagate updates within seconds?</p><p>For such features, we decided to leave the “features logging” path open. Our source of truth lives not in an Offline Feature Store, but rather in external storage. Once a shopper performs some activity, our behavioral API receives an event and updates Redis. This way, feature updates are propagated to the model almost immediately (milliseconds-seconds lag). The Ranking Service, in addition to calling the Online Feature Store (Index Service), also checks Redis for real-time features.</p><p>Another type of real-time feature is quickly changing dimensions in the product catalog, such as stock levels. Customers regularly (sometimes every 5 minutes) send catalog updates which are picked up by Index Builders and then served by the Index Service.</p><p>To summarize different types of features and how we handle them, a table below can be helpful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*f_LYDmI429_T2Xy_yucVuQ@2x.png" /><figcaption>Historical vs Real-Time Features Comparison</figcaption></figure><p>A cleaner architecture for handling both historical and online would be a <a href="https://en.wikipedia.org/wiki/Lambda_architecture">Lambda or Kappa architecture</a>, but given the split between historical and real-time features and having Redis / Index Service components already in our architecture, we decided to leave streaming for future improvements.</p><p>Below, you can see the overall system architecture diagram which covers all the components we have discussed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sdQztjwUDEECk4eyhF2cxw@2x.png" /><figcaption>Constructor’s Feature Store: Overall Architecture</figcaption></figure><h4>Ranking Service</h4><p>All the design described so far actually allowed us to make the Ranking Service completely model-agnostic. There is no need to write custom code in the Ranking Service for a particular model or customer. All implementation details are encapsulated in the model interface and in uniform feature design.</p><p>Here’s what happens when a request comes into the Ranking Service:</p><ol><li>RS looks up a requested model by (customer_id, model_name)</li><li>RS calls model.get_required_features() to receive required feature specs</li><li>RS looks at additional features passed in the request</li><li>RS calls FS for the rest of features (present in required specs but not passed in the request)</li><li>RS calls model.predict() with retrieved basis feature values</li><li>RS returns predicted scores back to Backend Service</li><li>RS logs request data along with all the feature values</li></ol><h3>Results</h3><p>By incorporating best industry practices for ML feature operations and adjusting for the specifics of our existing architecture, we have been able to get a lot of benefits:</p><ol><li>Drastically improve time-to-market for new features</li><li>Achieve convenient DS workflow</li><li>No code duplication</li></ol><p>And all that comes with a minimal increase in infrastructure costs. Online Feature Store performance satisfies our needs at the moment, allowing us to put all the complexity into the model. Our design supports both batch features and real-time features.</p><h3>Future Plans</h3><p>While the described solution covers most of our needs, there are always things to improve. A few directions we’re exploring at the moment include:</p><ul><li>Hourly updates for batch features. For features that don’t really need online updates, we still can update more often than daily via Offline to Online ingestion dataflow.</li><li>With more online features to support, explore Spark streaming (Lambda or Kappa architecture)</li></ul><p>Dmitry Persiyanov is a data science team lead at <a href="https://www.constructor.io/">Constructor</a> where our data science and engineering teams collaborate on e-commerce search and discovery and serve billions of requests each year for retail’s top companies.</p><p>Interested in joining the future of search? <a href="https://constructor.io/about/careers/">We’re hiring.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=330b65f64b18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/feature-store-design-at-constructor-330b65f64b18">Feature Store Design at Constructor</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product Ranking at Constructor]]></title>
            <link>https://medium.com/constructor-engineering/product-ranking-at-constructor-3c3ae44791f5?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/3c3ae44791f5</guid>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[ecommerce-web-development]]></category>
            <category><![CDATA[ecommerce]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Arshak Mkhoyan]]></dc:creator>
            <pubDate>Thu, 25 Aug 2022 19:25:58 GMT</pubDate>
            <atom:updated>2022-11-14T18:34:00.697Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Improving ecommerce business metrics with product ranking" src="https://cdn-images-1.medium.com/max/1024/1*HeFz5z3SSpO-Pa7QVZVrSg.jpeg" /></figure><h4>Increasing Business Metrics by Reordering Items</h4><p>Any e-commerce shopper knows the experience of scrolling through irrelevant and unattractive products looking for the right item. This experience leads to two possibilities: finally finding what you’re looking for, or leaving the site. Either way, it’s not a good experience for the shopper — and it can impact an e-commerce retailer’s ability to drive profit and build a long-term relationship.</p><p>Working with <a href="https://constructor.io/customers/">some of the biggest retailers in the world</a>, Constructor is continually developing new solutions to help e-commerce teams run successful businesses and improve the customer experience.</p><p>One such solution has been effective product ranking to satisfy the end user by helping them find ideal products more quickly. The culmination of seven years of R&amp;D, our ML-powered product ranking system satisfies users’ needs and drives retail company goals by surfacing relevant, <a href="https://constructor.io/blog/search-relevance-is-it-really-the-most-valuable-metric/">attractive</a> products higher up in search results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*h_BKQ4-j-ci1QIwMY0kdKA.gif" /></figure><p>In this article, we introduce a high-level view of the ranking system, including why we developed it, how the system works, and the results we’ve seen in A/B tests against other solutions. This is the first in a series of upcoming articles which will build on this one and provide more technical details.</p><h4><strong>Table of Contents</strong></h4><ol><li>A Quick Intro to Constructor’s Search Engine</li><li>The Retrieval System and Its Limitations</li><li>How Constructor’s Product Ranking System Works</li><li>Why a Separate Service for Ranking?</li><li>A/B Test Results</li><li>On the Roadmap</li></ol><h3>A Quick Intro to Constructor’s Search Engine</h3><p>In 2015, Constructor was founded explicitly to help e-commerce companies improve their search and discovery experiences through AI and machine learning. We launched with an <a href="https://constructor.io/products/autosuggest/">autosuggest</a> service, and then in 2019 released <a href="https://constructor.io/products/search/">search</a>, <a href="https://constructor.io/products/browse">browse</a>, and <a href="https://constructor.io/products/recommendations/">recommendations</a>, creating a holistic e-commerce ecosystem built from scratch on a foundation of data science.</p><p>Before we dive into the ranking system specifically, it’s important to review the overall architecture of Constructor’s search engine. It consists of 3 parts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KkJE_6Q16FzW5t7mVwH07Q.png" /><figcaption>Overview of Constructor’s Search Engine</figcaption></figure><ol><li><strong>Retrieval System<br></strong>Given a search query, retrieve all relevant/attractive products (candidates). The <em>recall</em> metric indicates the percentage of relevant items retrieved from all relevant ones. The higher the recall, the more relevant items are returned.</li><li><strong>Ranking System<br></strong>Given a set of products, rank products in the most effective way to satisfy business needs. The <em>precision</em> metric describes the percentage of attractive items in the retrieved set. We particularly optimize the<em> precision@K </em>metric, where <em>K</em> stands for the number of items shown at the top of search results to consider for computing <em>precision</em>. In other words, we want to rank attractive items at the top positions. Higher precision@K indicates more interactions (i.e., clicks, add to carts, purchases) between users and products shown.</li><li><strong>Business/Merchant Rules<br></strong>Applying e-commerce business-specific rules defined by the customer (optimize for profit margin, revenue per visit, conversion rate, inventory balancing, etc.) before returning the final ranked set of items.</li></ol><p>In this article, we’ll focus mainly on the second part of the search engine: the <strong>ranking system</strong>. But first, let’s briefly discuss retrieval.</p><h3>The Retrieval System and Its Limitations</h3><p>Constructor has two main product retrieval systems: an <a href="https://en.wikipedia.org/wiki/Inverted_index"><strong>Inverted Index</strong></a><strong> (II)</strong>, which was built in-house, and vector space search, which we’ll call by the name of our proprietary implementation, <a href="https://constructor.io/blog/say-goodbye-to-zero-search-results/"><strong>Cognitive Embeddings Search</strong></a><strong> (CES)</strong>.</p><p>Constructor’s<strong> II</strong> is e-commerce first and outperforms other popular variants in the field like SOLR, Lucene, and ElasticSearch. The index data structure is built based on searchable fields, including the product name, size, color, brand, AI-generated keywords, etc. Customers can also send us any item’s metadata in a separate field to include it in the II. It is also used in production to quickly get items relevant to a given query by matching the words (tokens) in the query and words (tokens) in searchable fields.</p><p>Constructor’s <strong>CES</strong> uses unsupervised and supervised (DSSM) models to learn vector representation of items and queries. By computing query and item distances (cosine distances between two vectors), we return Top-K items as decided through rounds of A/B tests for each customer. It greatly improves the recall metric and provides awesome lifts for conversion and purchase metrics based on internal A/B tests!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6bPDIZON-TjNAo6h-lxeNg.png" /><figcaption>Constructor’s Cognitive Embeddings Search improves search performance by greatly reducing zero-results rates with contextual awareness through vector/spatial representation of a catalog.</figcaption></figure><p>Both of these systems have some notion of ranking already: II uses token-item scores (as opposed to more traditional algorithms built into Solr and Elasticsearch like TF-IDF or BM25) and CES uses distances produced as scores.</p><p>These scores are then used in a weighted sum in addition to several other ranking factors based on clickstream data, such as query/user and item interactions, the personalization score based on each user’s unique history, and item attractiveness, to calculate the final score used to rank the items. After getting the base ranking, Business/Merchant Rules are applied to change the ranking accordingly.</p><p>Ranking using CES combined with II<em> </em>has proven to be a great technology, winning all external customer A/B tests comparing our system both to other search vendors as well as search engines built in-house on technology like Solr and Elasticsearch. However, there was still room for improvement based on the following limitations:</p><ol><li><strong>Time-consuming delivery of new features/signals.</strong> The logic that applies ranking factors was scattered throughout the backend, leaving no isolated place where data scientists could easily make changes. The lack of a unified pipeline created other limitations: a slow experimentation cycle, difficulty in adding and scaling new signals, and difficulty implementing changes for all customers at once.</li><li><strong>Difficulty in delivering “complex” features/signals.</strong><em> </em>Ranking applied within the <em>retrieval system </em>was done for all items returned by the system. Bounded by inference time constraints, we could only add simple signals that were “cheap” enough to calculate for potentially tens of thousands of products.</li><li><strong>Lack of flexibility in customer-specific configuration</strong>. Ranking logic was hard to configure on a customer basis, especially when we wanted to use different sets of signals for different customers. We could only ingest “all features” into production, many of which were unused. This could potentially overload the system.</li><li><strong>Difficulty in adding customers’ external ranking factors.</strong> We have customers who want to send us features to be considered by our ranking algorithm. These features needed to be custom added for each business.</li></ol><p>In order to surpass these limitations and also add more business value (which will be discussed later in the article), the Constructor team decided to create a separate service aimed at even better product ranking.</p><h3>How Constructor’s Product Ranking System Works</h3><p>Constructor’s ranking system is a real-time personalized ranking<strong> </strong>service, powered by machine-learning models with efficient feature delivery and experiment workflow<strong>.</strong></p><p>Below you can find a diagram with a simplified process for product ranking. Generally, it works on top of the retrieval system by taking Top-K candidates returned by the retrieval system and passing them through the ML model to get predicted scores that are used to rank the items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zxOIFVuowMTvGjT_" /><figcaption>Constructor’s Products Retrieval and Ranking Infrastructure Overview</figcaption></figure><p>Overall, the system consists of 2 parts: Offline and Online.</p><h3>Offline</h3><p>To be able to process requests to the Ranking Service online, we run<strong> </strong>dataset creation and<em> </em>model training/ingestion on a daily basis for each customer. To do so, we have a job that when triggered creates a DAG (Directed Acyclic Graph) to compute, save, and collect all necessary data (mostly using Spark via Databricks), and then train and save the ML model.</p><p><strong>Dataset Creation</strong></p><p>To construct data for training the ML model, we use behavioral and search results logs to collect data around each user’s search requests, items shown, and interactions (clicks, add-to-carts, purchases, etc.) We also run a Feature Computation Job and use the Offline Feature Store to store four types of collected features: item metadata, user metadata, user history, and interactions. The collected features are saved to the Online Feature Store on a daily basis to be used in real-time inference.</p><p><strong>Model Training/Ingestion</strong></p><p>After the dataset and features are collected, we run a job to train and evaluate our ML model offline. All produced logs generated from training and validating the model (paths to model and data, metric values, statistics, etc.) are saved into Neptune.ai for experiment management, while the model and Model Artifacts are saved into object storage, and Model Metadata is ingested into a database.</p><h3>Online</h3><p>When the user types a search query, the request is sent to the Backend Service<em>, </em>which makes all needed calls to other services to return a ranked set of items. The process happens in the following order:</p><ol><li><strong>Receive<em> </em>Personalization &amp; Real-Time Features</strong><em> </em>based on user’s metadata and history to later use in ranking results.</li><li><strong>Retrieve candidates</strong> for ranking:<br>- The II receives the query and finds and returns a set of matching items, applying spelling correction and query expansion automatically.<br> - In parallel, CES is triggered to enrich the result set. It uses vector representations of items, and thus, has much higher (100%) coverage compared to the II. If results from the II are not available, only results from CES are used. Otherwise, results from the two systems are combined to get all relevant/attractive products.</li><li>A retrieved set of items (from II and/or CES) is passed to the <strong>Ranking Service</strong>. It retrieves features from the Online Feature Store<em>, </em>computes some “complex” real-time features, and retrieves the appropriate model with its artifacts using the Model Registry. By passing features to the model, we produce final items’ scores to rank the results.</li><li>The last step in ranking is applying <strong>Business/Merchant Rules</strong>, which is done in the Backend Service.</li></ol><h3>Why a Separate Service for Ranking?</h3><p>A separate ranking service has many benefits over ranking done within retrieval systems. It specifically overcomes the limitations mentioned previously:</p><ol><li><strong>Fast delivery of new features/signals.</strong> Where it used to take two months to add a new feature to be considered in the rankings, it now takes less than two weeks. This was achieved by having an ML model to learn ranking and separate storage for features. Other benefits include a faster experimentation cycle, scalable addition of new signals, and ease of providing changes to all customers at once.</li><li><strong>Easy to deliver “complex” signal features/signals. </strong>As ranking is applied to the Top-K candidates, we can now easily add “complex” features that will be computed only for a small set of items and not reach inference time constraints.</li><li><strong>Flexibility in customer-specific configuration. </strong>Ranking logic is easy to configure on a customer basis, as we can set different (model, feature, dataset) parameters for each customer in the configuration file alone.</li><li><strong>Easy to add Customer’s external ranking factors. </strong>Customers can send us their ranking factors in a predefined way to be included in our ML model.</li><li><strong>Optimal results for merchant tasks. </strong>As a bonus, merchandiser rules can now be integrated into the ranking system by providing the respective set of features (promotions/sales data and current items’ stock data for promotions and inventory optimization, for example) to include in the ML model to reflect business logic.</li></ol><h3>A/B Test Results</h3><p>Constructor embraces an experimentation culture, so A/B tests are our best friends to validate the changes made in the ranking service: new features, new dataset creation logic, code optimization, etc. We constantly conduct A/B tests to prove the value we provide to our customers and decide on the general path of future work.</p><p>We have already had many rounds of A/B tests for the ranking system, each of which helped us to improve the algorithm and the service.</p><p>All tests were mostly concentrated on improving product ranking and making sure speed was fast enough so the overall effect was a lift in conversions and revenue. Altogether, we were able to reach positive, statistically significant <em>incremental</em> lifts for most customers on top of the results they’d already attained by switching to Constructor:</p><ul><li><strong>+2% average incremental lift in purchase rate</strong></li><li><strong>+2% average incremental lift in conversion rate</strong></li><li><strong>+3% average incremental lift in click rate</strong></li></ul><h3>On the Product Ranking Roadmap</h3><p>The ranking system is still in the process of rolling out for all customers. We’re doing it slowly to optimize respective ML models for every customer to ensure maximum lifts. While it’s already proving its value by increasing our customers’ revenue, conversions, and profit, we’re not stopping there! We have several plans for how we can further develop and optimize the ranking service:</p><ul><li><strong>Applying the separate ranking service to the browse experience.</strong> Constructor ranks the items presented when users search for products by going to a category page from the website’s navigation panel instead of typing a search query. Driving incremental improvement for browse with the separate ranking service is next on our list. We have seen great offline lifts and are hoping for the same results in upcoming A/B tests.</li><li><strong>Better usage of customer-provided data to improve ranking. </strong>We can make ML ranking more personalized if we can apply data from third-party vendors our customers collaborate with or from the Customer’s own DS team.</li></ul><h3>Conclusion</h3><p>Constructor’s ranking system is a promising service to make e-commerce businesses more successful by finding an additional way to leverage user data to power the ML model. We’re constantly working on improving the system and specifically the ranking algorithm to make end users happy. When shoppers see more attractive results that they find more appealing, they buy more and retailers see more conversions. It’s a win-win.</p><p><em>Arshak Mkhoyan is a machine learning engineer at </em><a href="https://www.constructor.io/"><em>Constructor</em></a><em>, where our data science and engineering teams collaborate on e-commerce search and discovery and serve billions of requests each year for retail’s top companies.</em></p><p><em>Interested in joining the future of search? </em><a href="https://constructor.io/about/careers/"><em>We’re hiring.</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c3ae44791f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/product-ranking-at-constructor-3c3ae44791f5">Product Ranking at Constructor</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Faster PySpark Unit Tests]]></title>
            <link>https://medium.com/constructor-engineering/faster-pyspark-unit-tests-1cb7dfa6bdf6?source=rss----25df6fa77ec4---4</link>
            <guid isPermaLink="false">https://medium.com/p/1cb7dfa6bdf6</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[unit-testing]]></category>
            <category><![CDATA[pytest]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Sergey Ivanychev]]></dc:creator>
            <pubDate>Fri, 24 Jun 2022 14:39:47 GMT</pubDate>
            <atom:updated>2022-06-24T14:39:47.835Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYgJS7hNS8-QZ0EiCDnNuA.jpeg" /><figcaption>Credit for photo: <a href="http://twitter.com/tetrebbien">@tetrebbien</a> from Unsplash</figcaption></figure><blockquote><strong>TL;DR:</strong> A PySpark unit test setup for pytest that uses efficient default settings and utilizes all CPU cores via pytest-xdist is available on <a href="https://github.com/ivanychev/medium-faster-pyspark-tests">GitHub</a>.</blockquote><p>Spark is an excellent framework for massively parallel computation. As with every software system, the data pipelines written in Spark require thorough testing. Every engineer that has ever written a unit test that involved Spark computation has noticed that those typically perform much slower than ones testing Pandas logic.</p><p>In this article, we provide you with battle-tested best practices and good defaults on how to write a faster Spark unit test that utilizes 100% of your CPU and RAM.</p><p><strong>Disclaimer:</strong> This article uses the <a href="https://docs.pytest.org/en/7.1.x/">pytest</a> unit testing framework. When I mention specific Java options, you can put them in the configuration step of the SparkBuilderseparated by a space:</p><pre>.config(<br>    &#39;spark.driver.extraJavaOptions&#39;,<br>    &#39;-Dsome.option=true -Dother.option=false&#39;<br>)</pre><h3>Configure Your Spark Session Wisely</h3><h4>Use one executor</h4><p>Let’s see how PySpark works when run locally:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*_cGCAqn6ljPifwcfB3iTyA.png" /><figcaption>Diagram of moving components of local Spark session</figcaption></figure><p>Unit tests don’t involve crunching gigabytes of data stored in multiple partitions. The number of Spark calls, on the other hand (manipulation with the tables, writes, and DataFrame manipulations), is quite high. This means that the most CPU overhead you encounter in unit tests is spent on the Python-to-JVM (Java Virtual Machine) interop.</p><p>In this case, having more than one executor (=1 CPU core) per session is not necessary as there’s not much data to crunch, but it still occupies some CPU. We can tell Spark to use one core by specifying local[1] when creating a session.</p><pre>SparkSession.builder<br>  .master(&#39;local[1]&#39;)</pre><h4>Override default partitioning</h4><p>The default value for spark.sql.shuffle.partitions <a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options">is 200 as of version 3.2.1</a>. This is a good default for big data, but since there’s not much local data anyway, you spend more overhead iterating over the partitions after the shuffle than actually doing computation. Limiting this number to 1 drastically improves performance.</p><pre>.config(&quot;spark.sql.shuffle.partitions&quot;, &quot;1&quot;)<br># In case you&#39;re using Delta<br>.config(&#39;spark.databricks.delta.snapshotPartitions&#39;, &#39;2&#39;)</pre><pre># Java options<br># -Ddelta.log.cacheSize=3</pre><p>If you’re using <a href="http://delta.io/">Delta</a> as a storage format you might need to override some of its defaults as well. During the reading of the transaction log, Delta repartitions it, and the spark.databricks.delta.snapshotPartitions setting defines the number of partitions (<a href="https://github.com/delta-io/delta/blob/v0.5.0/build.sbt#L70">here is the config</a> used in the Delta project itself).</p><p>Java option -Ddelta.log.cacheSize=3 reduces the memory footprint by <a href="https://books.japila.pl/delta-lake-internals/DeltaLog/#deltalog-cache">limiting the Delta log cache</a>:</p><pre>.config(&#39;spark.ui.showConsoleProgress&#39;, &#39;false&#39;)<br>.config(&#39;spark.ui.enabled&#39;, &#39;false&#39;)<br>.config(&#39;spark.ui.dagGraph.retainedRootRDDs&#39;, &#39;1&#39;)<br>.config(&#39;spark.ui.retainedJobs&#39;, &#39;1&#39;)<br>.config(&#39;spark.ui.retainedStages&#39;, &#39;1&#39;)<br>.config(&#39;spark.ui.retainedTasks&#39;, &#39;1&#39;)<br>.config(&#39;spark.sql.ui.retainedExecutions&#39;, &#39;1&#39;)<br>.config(&#39;spark.worker.ui.retainedExecutors&#39;, &#39;1&#39;)<br>.config(&#39;spark.worker.ui.retainedDrivers&#39;, &#39;1&#39;)</pre><p>Here we tell Spark that there’s no need for the Spark progress bar or Spark UI during unit test execution (though it might be helpful for debugging unit tests) and that it shouldn’t track the history of the jobs. This reduces RAM usage and doesn’t spin up the local HTTP server with the UI.</p><pre>.config(&#39;spark.driver.memory&#39;, &#39;2g&#39;)</pre><pre># Java options:<br># -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops</pre><p>In local mode, the Spark driver shares its memory with its executors (just the one, in our case). You should fix the RAM expectations upfront. This is essential if you want to tell if this threshold is reached when your pipeline evolves. Setting this parameter also helps assess how much memory is consumed in total when running unit tests in parallel. (We’ll come back to this later).</p><p>The -XX:+CMSClassUnloadingEnabled option allows the JVM garbage collector to remove unused classes that Spark generates a lot of dynamically. <a href="https://stackoverflow.com/questions/3334911/what-does-jvm-flag-cmsclassunloadingenabled-actually-do">This wins back some RAM</a>.</p><p>If you’re not planning to use more than 32G of RAM in Spark, -XX:+UseCompressedOops will do no harm and will return to you a few megabytes of memory. <a href="https://medium.com/r?url=http%3A%2F%2Fblog.leneghan.com%2F2012%2F03%2Freducing-java-memory-usage-and-garbage.html">This setting</a> tells JVM to use 32-bit addresses instead of 64.</p><h3>Don’t Create a Spark Session Unless You Need To</h3><p>A common pattern for writing unit tests is to have a single god-like fixture that sets up all of the resources (including the Spark session) and is then used in all tests implicitly via the setting autouse=True :</p><pre>@pytest.fixture(scope=&#39;session&#39;, autouse=True)<br>def all_resources():<br>  spark = get_spark_session()<br>  mock_some_objects()<br>  data_dir = prepare_test_data()<br>  <br>  yield (spark, data_dir) # yielding tuple resources.<br>  <br>  # Cleanup.<br>  shutil.rmtree(data_dir)<br>  spark.stop()</pre><pre># ...</pre><pre>def test_compute_something_big(all_resources):<br>  spark = all_resources[0]<br>  assert spark.table(...).count() == 42</pre><pre># Here we don&#39;t need Spark session, this test might<br># work much faster if executed on its own.</pre><pre>def test_pandas(all_resources):<br>  data_dir = all_resources[1]  <br>  df = pd.read_csv(&quot;...&quot;)<br>  expected_types = ...<br>  assert df.dtypes == expected_types</pre><pre># This test doesn&#39;t need Spark session either,<br># but it still implicitly created due to autouse=True</pre><pre>def test_spark_version():<br>  import pyspark<br>  assert pyspark.__version__ == &#39;3.2.1&#39;</pre><p>This approach works, but in your project there may be tests that don’t need Spark (or other expensive resources) to run. When you’re implementing some Pandas logic and running test_pandas iteratively, it starts up pretty slowly because it depends on all_resources, which spins up a Spark session every time you launch test_pandas .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*YfNmx2SbnAisOU2O-qaGlA.png" /><figcaption>Dependency in the naive approach</figcaption></figure><p>We can do better.</p><p>pytest is excellent at dealing with your unit test dependencies through <a href="https://docs.pytest.org/en/7.1.x/explanation/fixtures.html?highlight=fixtures">fixtures</a>. To allocate the resources your test needs, you can decouple the fixtures and use only those that are required by your test:</p><pre>TestingContext = collections.namedtuple(&quot;data_dir&quot;)<br>SparkTestingContext = collections.namedtuple(&quot;test_ctx spark&quot;)</pre><pre># This is cheap to execute, so autouse=True is fine<br>@pytest.fixture(scope=&#39;session&#39;, autouse=True)<br>def testing_context():<br>  mock_some_objects()<br>  data_dir = prepare_test_data()<br>  <br>  yield TestingContext(data_dir)<br>  shutil.rmtree(data_dir)</pre><pre># The spark_testing_context &quot;wraps&quot; the testing_context.<br># Test can require only testing_context, in this case<br># Spark session is not created at all!<br>@pytest.fixture(scope=&#39;session&#39;)<br>def spark_testing_context(testing_context):<br>  spark = get_spark_session()<br>  <br>  yield SparkTestingContext(testing_context, spark)<br>  <br>  spark.stop()</pre><p>Here we introduced testing_context and spark_testing_context fixtures, where the second depends on the first. The expensive initialization of the Spark session is moved to a different fixture that does not auto-initialize. This makes the 2nd and the 3rd tests evaluate much faster:</p><pre>def test_pandas(testing_context):<br>  data_dir = testing_context.data_dir<br>  df = pd.read_csv(&quot;...&quot;)<br>  expected_types = ...<br>  assert df.dtypes == expected_types</pre><pre>def test_spark_version():<br>  import pyspark<br>  assert pyspark.__version__ == &#39;3.2.1&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*XLdSuUhlaNtaaFn4d1GhJQ.png" /><figcaption>Tests with fixtured dependency</figcaption></figure><h3>Use all of your cores</h3><p>When running vanilla pytest tests implemented this way, only 1 to 2 CPU cores will be utilized (the first because of <a href="https://realpython.com/python-gil/">GIL</a> and the second because Spark lives in a separate JVM process that communicates with the Python interpreter via <a href="https://www.py4j.org/">py4j</a>).</p><p>This means a lot of unrealized potential if you have a CPU with ≥ 4 cores. How can we benefit from many cores when running tests? There’s already an excellent library: <a href="https://github.com/pytest-dev/pytest-xdist">pytest-xdist</a>.</p><pre>pytest tests_folder/</pre><p>With this module and two simple command-line options, you can run your pytest test cases on multiple cores:</p><pre>pytest tests_folder/ \<br>  # Spawn 2 processes for testing<br>  -n 2\<br>  # The tests are distributed by file across processes.<br>  # Given a file, all tests in it are executed in a single process.<br>  --dist loadfile</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*4gYmlWk2g9iHqKbCbHv07A.png" /></figure><p>This setup can lead to problems, however, when your tests involve Spark. Session-scoped fixtures are created once per process; in the example above, at most two Spark sessions are created. If configured poorly, the sessions will interfere with each other in a data race, so we need to carefully configure them. To solve this problem, pytest provides the <a href="https://docs.pytest.org/en/6.2.x/tmpdir.html#the-tmp-path-fixture">tmp_path_factory</a> session-scoped fixture, which generates a temporary directory to use in tests — a different directory in each pytest-xdist process.</p><p>At this point, let&#39;s list possible places where issues might happen.</p><h4><strong>Metastore path</strong></h4><p>When running locally, Spark uses the <a href="https://db.apache.org/derby/">Apache Derby</a> database. This database uses RAM and the local disc to store files. Using the same metastore in all processes can lead to concurrent modifications of the same table metadata. This can lead to errors (e.g. unknown partitions) and undesired interference between tests.</p><p>To avoid it, use a separate metastore in each process. This can be achieved via setting the -Dderby.system.home directory:</p><pre>-Dderby.system.home={tmp_path_factory.mktemp(&#39;derby&#39;)}</pre><h4><strong>Test output data path</strong></h4><p>Do not write Spark outputs from different processes to the same output folder, as this can lead to data races and test interference. Use tmp_path_factory to set up the output folder in the session-scoped fixture. In Spark, we can omit to specify the output path while calling the write operation like saveAsTable</p><pre>df.write.saveAsTable(&quot;test_db.some_table&quot;)</pre><p>In this case, the data is saved to the folder set up by spark.sql.warehouse.dir configuration. You should use separate paths in different processes as well.</p><pre>.config(&#39;spark.sql.warehouse.dir&#39;,<br>        tmp_path_factory.mktemp(&#39;warehouse&#39;))</pre><h4>Checkpoint location</h4><p>If you use a checkpoint operation in your code, you should use the same technique to set up output locations.</p><pre>spark.sparkContext.setCheckpointDir(<br>  tmp_path_factory.mktemp(&#39;checkpoints&#39;)<br>)</pre><h4>Delta setup</h4><p>Delta Lake is <a href="https://docs.delta.io/latest/quick-start.html#python">straightforward to set up with Spark</a>. The configure_spark_with_delta_pip function does a great job at setting up the SparkBuilder to work with Delta. However, this approach doesn’t work when executing tests in parallel as configure_spark_with_delta_pip doesn’t find the JAR with Delta in separate processes and tries to download them to the same temporary location.</p><p>To my knowledge, this location can’t be configured, so that means we need to manually configure Delta. This isn’t hard, though:</p><pre>.config(&#39;spark.sql.extensions&#39;,<br>        &#39;io.delta.sql.DeltaSparkSessionExtension&#39;)<br>.config(&#39;spark.sql.catalog.spark_catalog&#39;,<br>        &#39;org.apache.spark.sql.delta.catalog.DeltaCatalog&#39;)<br>.config(&#39;spark.jars&#39;, f&#39;file://{delta_core_jar}&#39;)</pre><p>delta_core_jar stores the path to the delta-core JAR file (which you can download <a href="https://mvnrepository.com/artifact/io.delta">here</a>). In my experience, this file isn’t too big so it’s fine to put it in your Git near the conftest.pyand use a relative path to point Spark to it.</p><pre># conftest.py<br>delta_core_jar = str(<br>  pathlib.Path(__file__).parent / &#39;delta-core_2.12-1.1.0.jar&#39;<br>)</pre><p><strong>Note: </strong>Use the same Delta JAR version as your delta-spark version. You should even write a unit test for that!</p><p>The final diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*O8JKFH-Xh5c6SVXNyhfKjA.png" /></figure><h3>Other notes</h3><ul><li><strong>Use a profiler to find bottlenecks.</strong> <a href="https://github.com/manahl/pytest-plugins">pytest-profiling</a> is a fantastic library that profiles your tests and generates an SVG with a call graph for the percentage of time spent in every function. It’s very helpful to find slow tests or slow code reused across tests.</li><li><strong>Profile your code at run time. </strong>If you want to attach to the already running code, <a href="https://github.com/benfred/py-spy">py-spy</a> is an excellent tool for the job.</li><li><strong>Speed up tests collection step.</strong> Before executing the tests, pytest collects the tests to execute. This process might become slow due to heavy imports or expensive logic put in the module scope that gets executed during import. To find out what’s causing the issue, use the pytest-profiling library with pytest’s --collect-only flag.</li></ul><h3>Conclusion</h3><p>Check out the final PySpark unit tests setup <a href="https://github.com/ivanychev/medium-faster-pyspark-tests">here in GitHub</a>. The techniques described in this article helped us to speed up Constructor’s test suite roughly 2x when running on anr5a.xlarge AWS instance equipped with 4 cores and 32G of RAM.</p><h3>References</h3><ul><li>Some resources from Matthew Powers: an article on <a href="https://mrpowers.medium.com/testing-spark-applications-8c590d3215fa">testing Spark applications</a> and some handy <a href="https://github.com/MrPowers/chispa">Spark unit testing helpers</a></li><li>The latest <a href="https://spark.apache.org/docs/latest/configuration.html">Spark configuration options</a></li></ul><p><em>Sergey Ivanychev is a data engineer at </em><a href="https://www.constructor.io/"><em>Constructor</em></a><em>, where our data science and engineering teams collaborate on e-commerce search and discovery and serve billions of requests each year for retail’s top companies.</em></p><p><em>Interested in joining the future of search? </em><a href="https://constructor.io/about/careers/"><em>We’re hiring.</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cb7dfa6bdf6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/constructor-engineering/faster-pyspark-unit-tests-1cb7dfa6bdf6">Faster PySpark Unit Tests</a> was originally published in <a href="https://medium.com/constructor-engineering">Constructor Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>