<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Artem Kozhevnikov on Medium]]></title>
        <description><![CDATA[Stories by Artem Kozhevnikov on Medium]]></description>
        <link>https://medium.com/@artem_30369?source=rss-7eec23789b61------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VA2pcAYacBUUOnZZJAbK9w.png</url>
            <title>Stories by Artem Kozhevnikov on Medium</title>
            <link>https://medium.com/@artem_30369?source=rss-7eec23789b61------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 04:02:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@artem_30369/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Group-by data augmentation for e-commerce datasets]]></title>
            <link>https://medium.com/tinyclues-vision/group-by-data-augmentation-for-e-commerce-datasets-508d5080c0c6?source=rss-7eec23789b61------2</link>
            <guid isPermaLink="false">https://medium.com/p/508d5080c0c6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ecommerce]]></category>
            <dc:creator><![CDATA[Artem Kozhevnikov]]></dc:creator>
            <pubDate>Fri, 24 Mar 2023 09:36:26 GMT</pubDate>
            <atom:updated>2023-03-24T09:36:26.806Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing a new training strategy for learning user preferences to multi-attribute offers based on a multi-task approach</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1bTzKvXTdXWf7Frw" /><figcaption>Making sense of various groups of offer attributes</figcaption></figure><blockquote><strong>Code</strong> is available on <a href="https://github.com/tinyclues/group-by-augmentations-model">https://github.com/tinyclues/group-by-augmentations-model</a></blockquote><h3>Motivation</h3><p>In <em>tinyclues SaaS</em> business context, we want to model a user propensity to buy a given offer in the presence of its multiple offer attributes. In our previous <a href="https://medium.com/tinyclues-vision/universal-machine-learning-approach-for-targeted-marketing-campaigns-b6df27f38532">medium blog</a>, we explained that a core tinyclues platform functionality is to provide marketers with an extremely flexible ML tool for creating targeted marketing campaigns.</p><p>In particular, the definition of an “offer” is very broad and may include different product categories, attributes, and transaction contexts. It can even work in a case where no product catalog exists at all.</p><p>To solve this challenge, we considered several machine-learning approaches. In this blog, we’ll dive deep into a complete implementation of one of the previously discussed ML designs. Mainly we’ll talk about the offer aggregation strategy internal to the model. We’ll also share some insights about model architecture and discuss various results. So, let’s start!</p><h4>Main idea</h4><p>Our guiding intuition is to represent an offer as a bag of features composing it. What does this mean concretely? Take an offer defined as a SQL filter over a transactions (user-offer interaction) table. So, when we apply that filter, we get all corresponding offer rows (instances). Next, for a given set of offer features, we capture the distribution of each offer feature (independently of others).</p><p>In e-commerce datasets, offer features are typically categorical and encoded as 1-hot labels. Thus, the resulting bags will be weighted multi-hot features encoded as sparse vectors. Note that we may optionally modify the relative row weights by giving more importance to recent events. <a href="https://medium.com/tinyclues-vision/universal-machine-learning-approach-for-targeted-marketing-campaigns-b6df27f38532#2122">At inference time</a>, a trained model receives those multi-hot bags-of-attributes features as input (along with other features) to generate a prediction for a selected offer.</p><p>However, it turns out that is not enough yet! We need to modify the training schema as well to get accurate predictions when using this offer aggregation (because otherwise, the model performance may drop a lot.) Now we’ll explain how precisely to do that!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4PwExLUq3hwQMjaeAT_VGg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i2v79J8CHHYFSuNNYuEsZA.jpeg" /><figcaption><strong>left: </strong>A dummy example of offer attributes on transactional table || <strong>right: </strong>Bags representation of some offers generated from that table. (Offer definition attributes are shown in red. Here, “pid” is a short for “product_id”.)</figcaption></figure><h3>Data augmentation ingredients</h3><p>Keeping the main ideas in mind, our main logic becomes quite straightforward: we want to <em>mimic the inference pattern</em> of feature transformation during the model training as well. Mainly, we should replace original non-aggregated offer features with their bags-aggregated analogs for all training batches. In machine learning literature, this process is called <strong>data augmentation</strong>, and we’ll now take a closer look at it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DcXyMRX78nV13kB0V7-QTA.png" /><figcaption>Results of Group-By (with “key” column) on categorical (”brand” and “size” that are labeled encoded) and dense features (”emb”) followed by <strong>mean</strong> aggregation</figcaption></figure><h4>Mini-batch approximation of offer aggregation</h4><p>Note that for the inference, we can access offer bag statistics from the full (or sufficiently well-sampled) transaction data. Yet, for neural network training, having a mini-batch-based approximation can be convenient for building augmentations on the fly.</p><p>Fortunately, for the training of our models, we already used a batch size that is large enough (~10k), and we did not notice any major drawbacks of using the mini-batch stats in our experiments (even if, in theory, such an approximation adds more randomness and may not be accurate for rare offers).</p><p>Ok, how to generate a mini-batch augmentation? Let’s pick some random offer attributes (it can be one or many), let’s say <em>brand</em> and <em>size</em>. The obtained tuple (<em>brand, size</em>) is called a <strong><em>key</em></strong>, and a component performing this choice is called KeyGenerator. Next, for any row i ∈ MiniBatch, we modify the value of an offer feature <strong><em>F</em></strong> by a bag of values taken from all mini-batch rows sharing the same key. The resulting feature <strong>Grp(F)</strong> will be of variable length if <strong><em>F</em></strong> is categorical. Since this is similar to a common SQL operation, we’ll also baptize it <strong><em>group-by :</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tuX99wS7CPFI8QpCS0gqnA.png" /></figure><p>For a given mini-batch we generate <strong>nb_augmentations</strong> (typically, 5 …10) group-by augmented batches.</p><pre># input: batch (dict: feature_name -&gt; tensor)<br># params: offer_features, average_key_length, nb_augmentations<br># output: [batch_a1, batch_a2, ... , batch_an] (augmented batches)<br><br>def KeyGenerator(batch, offer_features, average_key_length):<br>    &quot;&quot;&quot;<br>    Generate group-by key from offer attributes<br>    &quot;&quot;&quot;<br>  ...<br><br>def GroupBy(feature_tensor, group_by_key):  # in TF 2.*<br>    unique_values, unique_idx = tf.unique(group_by_key)  # unique keys positions<br>    grouped_ragged = tf.ragged.stack_dynamic_partitions(feature_tensor, unique_idx, len(unique_values))<br>    return tf.gather(grouped_ragged, unique_idx)  # broadcast to batch size -&gt; a ragged tensor of shape bs x None<br><br>output = []<br>for i in range(nb_augmentations):<br>    augmented_batch = batch.copy()<br>    group_key = KeyGenerator(batch, offer_features, average_key_length)  # shape bs x 1<br>    for feature in offer_features:  # we don&#39;t modify other features<br>        augmented_batch[f&#39;{feature}_grp&#39;] = GroupBy(batch[feature], group_key)<br>    output.append(augmented_batch)</pre><h3>KeyGenerator</h3><p>So, what about the behavior of KeyGenerator? Many possible key exploration strategies could be used. For instance, it is reasonable to select the offer keys which are more frequently used by our clients on the platform with a higher probability. Yet, we went with a very simple and usage-agnostic approach: each offer feature can be selected randomly and independently of others with some fixed probability proba = average_key_length / number_of_offer_features, which produces a binomial distribution. We typically set average_key_length ~ 2. Once key features are selected, we define a “key” column as a tuple of key features values (or by hashing them), and next, we perform an offer features group-by based on it. If an offer feature is a multi-hot feature, we apply <a href="https://en.wikipedia.org/wiki/MinHash">MinHash</a> on it (with random seeding for each batch) to get back to the 1-hot case.</p><h4>Offer mixture</h4><p>A basic KeyGenerator above corresponds to the AND conjunction between several attributes in the offer SQL filter. To support the OR disjunction case (as pid=”942” OR pid=”661” in the example above), we need to perform an extra offer mixture. There are many ways to do so, but we consider here a simple method that preserves a mini-batch logic. In particular, we split the biggest groups obtained above into smaller sub-groups, which we then collision randomly to create a new key column. We do it in such a way that collided groups contain from 2 to 6 “pure” sub-groups. That simulates the sparse mixtures of various offers with random weights. In the process, we apply those collisions only to a random fraction (like ~50%) of generated keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCGbBQB4Z0vrhe5SaxmyPw.png" /><figcaption>Offer mixture illustration : original groups are randomly split into smaller sub-groups, and next randomly merged to form new collided groups (colors match different groups)</figcaption></figure><p>Note that it is very similar to generating a random linear combination of sample representations in classical semi-supervised methods such as <a href="https://arxiv.org/pdf/1905.02249.pdf">MixUp</a>.</p><h4>Multi-task</h4><p>Now that we have found how to modify batches to represent any offer, we’ll feed those batches into the original model. Of course, one should also ensure that a model handles ragged inputs correctly. A loss per step can be defined merely as a sum of the respective losses. And in fact, we may think about it as if it were just a multi-task learning process where each choice of grouping keys corresponds to a different task. And that’s an important point since it opens the room to various multi-task optimization methods.</p><pre>loss = 0<br>for i in range(nb_augmentations)<br>  output_i = Model(augm_batch_i)<br>  loss += LossFn(output_i, response)</pre><p>Note that it is worth creating several augmentations from one batch of data. It would allow to average gradients of different multi-tasks at each learning step, making it more stable. It also speeds up the training itself. Indeed, grouped-by batches can be processed in parallel by a model while user features embeddings (or other model nodes) can be shared across all augmentations.</p><h4>Group-by with fused aggregation</h4><p>In practice, we do a group-by aggregation in a more optimized way. We first apply an embedding layer on offer feature inputs (only once per batch) and then perform a data augmentation <strong>fused</strong> with a vectorial aggregation (like <em>mean </em>on “emb” column in the example above) in one single operation on the offer embeddings. By doing so, we can share offer embeddings computation across different batch augmentations and thus be more memory efficient.</p><p>Of course, this trick may not work for more complex aggregations. Technically speaking, thanks to on-the-fly group-by with fused aggregation, we could to implement everything inside the model itself without modifying our training data-loaders at all. So, let’s now look at what happens to the model!</p><h3>Elements of model design</h3><p>Group-by data augmentation can be combined with many <a href="https://arxiv.org/pdf/1807.00311.pdf">possible ML/DL models</a>. And the most noticeable performance improvement will come from the group-by itself. However, choosing the right model can also be very important. We thus want to share some principles we considered for our model architecture choice. From our experiences, the latter is typically responsible for around 2%-6% of a relative metric improvement compared to a simple two-towers model (in notebooks, we’ll provide some <a href="https://github.com/tinyclues/group-by-augmentations-model/blob/master/training/ablation%20rees.ipynb">ablation tests</a>).</p><h4>Mean and variance extraction for internal feature importance</h4><p>To capture the main offer semantics, it is natural to extract a <em>mean </em><strong><em>μ</em></strong> of embeddings within groups. Yet besides <strong><em>μ</em></strong>, we also extract the embeddings’ intra-group variance <strong><em>σ</em></strong>. On many datasets, we observed clear benefits from using <strong><em>σ</em></strong> in our models, and the intuition behind it is the following: higher variance <strong><em>σ</em></strong> indicates the potential presence of noisy components among offer embeddings.</p><p>Therefore, thanks to <strong><em>σ</em></strong>, a model can reduce their relative importance (one may think about <strong><em>μ / √(1 + σ)</em></strong> signal over noise kind of formula), whereas with only <strong><em>μ</em></strong>, a model should learn to sum up useless offer components to zero (neutral vector) which becomes much harder, thus making the model less robust to the generalization on unseen offers.</p><p>For instance, to represent a given <em>brand</em>, a model would rather rely on the <em>brand</em> feature embedding (whose <strong><em>σ</em></strong> is zero) and may completely ignore the component of <em>size</em> embeddings (whose <strong><em>σ</em></strong> is typically high because there is a variety of <em>sizes</em> within any <em>brand</em>).</p><p>Instead of taking a precise formula of σ action on <strong><em>μ</em></strong><em>,</em> we decided to learn it. Hence, we introduced an additional sub-network MaskNet, acting on <strong><em>μ</em></strong> as pointwise multiplication :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VkJWHAgUxCrmZ5L35q7XDQ.png" /></figure><p>We parametrize MaskNet as a sequence of non-linear dense layers (DenseNetwork or DN) with some sparse (<a href="https://arxiv.org/pdf/1905.10510.pdf"><strong>k-WTA</strong></a><strong>)</strong> connections suitable for multi-task learning.</p><p>Note that one can also extract other statistics from group embeddings as soon as they are consistent, i.e., converge to the ones over the entire dataset when a batch size grows. And, to follow a more modern idea, one could use a <a href="https://en.wikipedia.org/wiki/Attention_%28machine_learning%29"><em>self-attention</em></a> type of model as an aggregator instead of deterministically extracted statistics.</p><h4>Compressed feature-wise interactions</h4><p>Another important principle we identified in our work: each offer (and user) feature should possess its own single embedding space. This is important for sharing the maximum amount of information while reducing the number of model parameters.</p><p>However, this comes at the price of a more complex interaction module. For instance, it can be harmful to go with a simple sum of different feature embeddings (like <em>size</em> and <em>brand</em>). Indeed, suppose the<em> size</em> feature provides a very strong interaction. In that case, the mixed embedding space will be completely driven by <em>size</em> semantics and can be too constraining for a <em>brand</em> representation.</p><p>This reasoning naturally brings us to a feature-wise <a href="https://www.atailab.cn/seminar2019fall/pdf/jiangxinyu1.pdf">bi-linear interaction model</a> which captures all individual user-offer feature interactions and allows the coexistence of different embedding spaces. Mainly, for all pairs of user <strong><em>u_i</em></strong> and offer <strong><em>o_j</em></strong> features, we extract a scalar interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fg9hxCGFAa8s7NlSMnEB3Q.png" /></figure><p>where the kernels</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yBHpr4ifDqXPWGLYA9vXhw.png" /></figure><p>It is an adequate choice when there are few interactions. But it becomes impractical (due to a higher number of parameters and expensive computational time) when the number of offer and user features grows. To solve this issue, we suggest the following tradeoff: before computing the feature-wise interactions, we combine some offer features into a “meta”-feature.</p><p>Thus, this piece of the model looks (in <a href="https://www.tensorflow.org/api_docs/python/tf/einsum">einsum</a> tensor notations) as follows,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZwETkPc6icPLjroM7TZHQ.png" /></figure><p>where <strong>b</strong> stands for batch, <strong>o</strong> for offer features, <strong>m</strong> for meta offer features, and <strong>d</strong> for embedding dimensions, respectively. The feature compression matrix <strong><em>ℂ := ℂ(μ, σ) </em></strong>is (instance-wise) parametrized by a DN, and MaskNet is now acting directly on the meta-features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxlBHCrz6f7TLoFe3Cgdfw.png" /></figure><p>We perform a similar meta-compression on user features as well.</p><p>We argue that this feature compression is a valid choice for the groups of hierarchical features (like for ‘<strong>cat1</strong>’, ‘<strong>cat2</strong>’, ‘<strong>pid</strong>’ sharing a compatible semantic), and as a usual number of hierarchical groups within offer features is rather low, we keep the number meta features rather lower (≤ 5). In some situations, it could be useful to require a more sparse structure from ℂ (for a stronger feature selection effect).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*rsKJRu7qcY1aZdzwbWYDMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*VoK2X12h_8maDLhxpuaKiA.png" /></figure><p>We finally apply a DN again (with gelu or tanh nonlinear activation) over extracted meta features interactions to get scalar output. For more details (technical implementations, model hyper-parameters choices, evaluations, and so on), we invite the reader to look at these <a href="https://github.com/tinyclues/group-by-augmentations-model">notebooks</a>, where we reproduce our models on publicly available <em>Movielens</em> and <em>Rees</em> datasets.</p><h4>Training time</h4><p>Thanks to fused aggregation and suitable training params choice (we typically double the number of epochs for group-by training), we found that the overall training time increase due to group-by augmentation is rather acceptable (at worst-case scenario ≤ x3) compared to a large number of combinations of offer attributes the model learns.</p><h3>Results</h3><p>We’ll present several results of our model with group-by data augmentation on the internally available datasets. Group-by models are now used in production on the majority of our datasets (on ≥ 100 of them) but due to a lack of space, we’ll only report the most typical and illustrative cases here. We also provided the notebooks that allowed us to reproduce these results on two public datasets.</p><h4>Evaluation</h4><p>Our cross-validation protocol consists of the models training on historical events (around one year of data) and the performance evaluation over two weeks post-training time. Let’s know the metric we look at. For a given grouping key (let’s say “<em>brand</em>” above), we focus first on the corresponding most popular offers (i.e., queries like <em>(”brand” = X)</em> ). For any such offer we evaluate the model <a href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">AUC</a> on the offer binary classification task (<em>1</em> if the event belongs to that offer and <em>0</em> otherwise).</p><p>Finally, we average the computed AUCs (with weight = offer frequency) to get <strong>wAUC</strong> <strong>— </strong>the metric we’ll follow. We report <strong>wAUC</strong> for a subset of available offer keys. While looking at the top offers is more appropriate for group-by effect (since all groups will contain a lot of elements), for cold start benchmarks, we are taking less frequent offers whose number of occurrences is ~ 20–200 in the evaluation window.</p><h4>Multi-task model vs. <strong>specialized</strong> mono-task models</h4><p>To challenge our multi-task approach, for any chosen offer attribute, we train a mono-task model which is restricted to the corresponding offer feature. These specialized models are trained with no augmentation but share the same architecture (except using <strong><em>σ</em></strong>, which is zeros in that case).</p><p>Here we report models wAUC performance for various offer attributes (taking offers with ≥ 200 occurrences):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dXOxC9D-NPJ1So1Fsie6aw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*76GNdDEfDZb94L9M3Kn11A.png" /><figcaption><strong>left</strong>: <strong>Retail A</strong> (Classical retail with atypical size feature) || <strong>right</strong>: <strong>Retail B</strong> (Classical retail with strong geographical shop_id feature)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ijWxbyI1hZYeRmHvzT4fpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4Xz95-_6GdxXaOw8olHoQ.png" /><figcaption><strong>left</strong>: <strong>Travel</strong> (Travel industry standard case) || <strong>right: </strong>public <a href="https://www.kaggle.com/datasets/mkechinov/ecommerce-behavior-data-from-multi-category-store"><strong>Rees</strong></a> retail dataset</figcaption></figure><p>Let’s take a closer look at the results. We note several important observations :</p><ul><li>First of all, as expected, mono task models show strong results when evaluating on their corresponding task but typically underperform on the others.</li></ul><blockquote>GROUP_BY model provides decent results across ALL tasks with few cases where metrics are slightly lower than the specialized counterpart model. And even if it is not a winner everywhere, a group-by model shows clear multi-task capabilities !</blockquote><ul><li>Model doing offer group-by only in inference with only classical training (without augmentations) has a significant performance drop for most tasks.</li></ul><h3>Further analysis and future work</h3><p>We see that group-by trained models perform very well compared with specialized models, and that’s precisely the main effect we wanted to create! But beyond that, we’ll show you below some other interesting results that came along the way.</p><h4>Cold start scenario</h4><p>When looking at wAUC of rare offers (with a number of occurrences between 20 and 200), we see that our multi-task group-by model starts to <a href="https://medium.com/tinyclues-vision/beyond-cold-start-13da56b946b0">show even better results than specialized mono-task models</a>, which proves that group-by models can effectively leverage the information of all available offer features in cold-start scenario. Note also that group-by model leverages it much better than without augmentations model which has access to all offer features as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*uvhFHB-PRbQ1sGbsDOSfWQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*e2PDEX61QKy5WiWw2t2SHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*oWP6PS0wlakFmklWB7JfCw.png" /><figcaption>Cold start results. <strong>left</strong> : Retail A || <strong>center</strong> : Retail B || <strong>right</strong> : public Rees retail dataset</figcaption></figure><h4>Offer mixture experiments</h4><p>To validate the importance of offers mixture in group-by data augmentation, we generate an artificial “mixed” topics by sampling randomly a <strong><em>λ</em></strong> fraction of rows from one offer and <strong><em>(1 — λ)</em></strong> fraction from the second offer (<strong><em>λ = 0.5</em></strong> corresponds to ORquery between of the two offers).</p><p>We vary <strong><em>λ ∈ [0, 1] </em></strong>and follow the AUC on the <strong><em>λ</em></strong>-mixed offer of two models: one trained with only pure groups and another with an extra group-mixture. What we note is that for the extreme <strong><em>λ</em></strong> values, both models perform similarly. However, the collision stimulated model shows much better result in the middle; and if two offers are more “dissimilar,” one should expect a bigger gap.</p><p>On the plot below we show some typical behavior :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*78svq108gjySXLC2lMBr_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vRb5eu8X31LZKECNo2S48A.png" /><figcaption><strong>left:</strong> AUC on mixture (with param lambda) of two offers for two models : one trained with keys collisions (AND_OR) and the other not (only_AND) || <strong>right: </strong>Same for another offers choice</figcaption></figure><h4>Negative transfer</h4><p>We see that sometimes, the learning of some offer attributes may result in a performance drop for others (hence, we are talking about <a href="https://arxiv.org/pdf/2012.09575.pdf"><em>conflicting attributes</em></a>, like shop_id and product_id on <a href="https://www.notion.so/Group-by-data-augmentation-for-e-commerce-datasets-bbe25bbaff6f4445b4fb4f86e337f1f4">Dataset B</a>). This phenomenon, called <em>negative transfer,</em> is well-known and actively studied in the <a href="https://ojs.aaai.org/index.php/AAAI/article/view/5125/4998">multi-task world</a>.</p><p>To mitigate it, a variety of techniques can be applied. Here is a list of several ideas for future work :</p><ol><li>A guided (aka <a href="https://ronan.collobert.com/pub/2009_curriculum_icml.pdf">curriculum learning</a>) keys exploration;</li><li>Exploration that avoids taking the same or similar keys but rather tries to create complementary offer keys (based on their mutual information) for each batch;</li><li><a href="https://arxiv.org/abs/1708.02002">Focal loss</a> (we applied it with success on datasets with an important disparity of offer features interaction strength);</li><li>A <a href="https://proceedings.neurips.cc//paper/2020/file/3fe78a8acf5fda99de95303940a2420c-Paper.pdf">gradient surgery</a> for conflicting tasks and more multi-task-friendly <a href="https://arxiv.org/pdf/1911.05034.pdf">model architecture</a>.</li></ol><p>To illustrate the idea of curriculum learning, we observe in the example below that during the training “<em>shop_id</em>” attribute (which is easy to learn thanks to the “<em>zipcode</em>” user feature) reaches its top wAUC quickly whilst it takes more epochs for the offer attributes (item dependent). This is rather a common pattern, and therefore, it is worth making <strong>KeyGenerator</strong> more focused on the “hard” keys after the first epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*zDVt6gH3op8frmzsFzWmPQ.png" /><figcaption>wAUC evolution during training epochs on <strong>dataset B</strong></figcaption></figure><h4>Feature representation quality</h4><p>Another noticeable effect of group-by training is the improvement of the quality of both user and offer embeddings. We observed a better performance of user embeddings transferred to other downstream tasks. Thus, group-by data augmentation can be used as an efficient pre-training strategy or as an auxiliary task for semi-supervised training.</p><p>Group-by trained offer embeddings (more precisely, offer meta features) used for the (cos-)similarity search also showed more relevant results. Moreover, the overall model diversity (meaning the capacity to select different audiences for different offers while keeping the same prediction accuracy) has been increased as well (especially for lower entropy categories such as <em>cat1</em>, <em>cat2</em>, or<em> brand</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8jc-Q3IO8jNMEJDlWX-59g.png" /><figcaption>Offer meta features CosSimilarity for top-40 directors on Movielens dataset. Comparison of two models. <strong>left : group-by </strong>model || <strong>right : </strong>model <strong>without group-by</strong></figcaption></figure><p>The main intuition is as follows: the group-by training forces a model to make sense of all offer embeddings individually and their combinations, whereas with no data augmentation, the lower entropy features serve mainly as compensation for poorly sampled ones of higher entropy. Note that to some extent, a similar effect is reached with a classical masking/<a href="https://www.cs.toronto.edu/~mvolkovs/nips2017_deepcf.pdf">dropping</a> feature type of <a href="https://arxiv.org/pdf/2007.12865.pdf">data augmentation</a> (and the group-by can be successfully combined with the latter methods!).</p><h3>Conclusion</h3><p>To wrap it up, we’ve suggested a new type of data augmentation strategy called <strong>group-by</strong>. This strategy allows us to learn in a multi-task fashion user preferences from all combinations of offer attributes producing high-capacity models. And that’s with an acceptable negative transfer and a limited increase in the training time! Beyond that <strong>group-by</strong> trained models prove to be powerful for improving the embeddings quality and tackling the cold-start problem.</p><p>Last but not least, we believe that <strong>group-by</strong> data augmentation can find other applications for tabular datasets and is not limited to learning only offer-user interactions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=508d5080c0c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tinyclues-vision/group-by-data-augmentation-for-e-commerce-datasets-508d5080c0c6">Group-by data augmentation for e-commerce datasets</a> was originally published in <a href="https://medium.com/tinyclues-vision">Tinyclues Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 design principles for antifragile predictive systems]]></title>
            <link>https://medium.com/tinyclues-vision/5-design-principles-for-antifragile-predictive-systems-47d6b3216c2b?source=rss-7eec23789b61------2</link>
            <guid isPermaLink="false">https://medium.com/p/47d6b3216c2b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-systems]]></category>
            <dc:creator><![CDATA[Artem Kozhevnikov]]></dc:creator>
            <pubDate>Thu, 25 Jun 2020 13:34:31 GMT</pubDate>
            <atom:updated>2020-06-26T15:22:22.864Z</atom:updated>
            <content:encoded><![CDATA[<h4>To build successful real-world predictive systems, control theory is as important as machine learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*1b7BiOinrGTZBBwi.png" /></figure><p>On Kaggle, you can win by blending the machine-learning algorithms with the best raw performance metrics. In real life, this approach rarely works.</p><p>A common mistake of inexperienced data scientists is to believe that machine learning is the only science they need to build successful predictive systems.</p><p>Machine-learning textbooks usually focus on individual algorithms that work in isolation. When you assemble multiple algorithms into a complex system, when this complex system is fed real-word data and interacts with real-world users, a lot of unexpected stuff will happen.</p><p><strong>In real-life, systemic stability is what separates success from failure.</strong> It isn’t determined by the individual algorithms but by the overall architecture. Achieving systemic stability requires a control-theory mindset at all stages of the design process.</p><p>Here are 5 practical guidelines that summarize the design philosophy that we’ve successfully used to achieve systemic stability at Tinyclues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fOAWYO3ehV9DFUez.png" /><figcaption>Seeking performance without caring for stability results in fragile designs. To get operability at scale, you need deep stability and simple controls.</figcaption></figure><h3>1. Build an architecture that survives noise</h3><p>Machine-learning algorithms optimize loss functions under assumptions on noise (resulting into an extra regularization term). Gaussian noise is a popular noise model in machine-learning theorems, not because it happens in real life, but because it makes the math easier.</p><p>The actual noise that you can encounter is unlikely to fit any simple model:</p><ul><li>The original <strong><em>noise from the actual problem</em></strong> that you are studying may be intrinsically hard to model. Eg, the B2C consumptions patterns that we are working on exhibit a customer attributes noise component (more or less Gaussian), a time-series noise component (extremely sparse and hard to model beyond the obvious cycles), plus some ticket-size noise (totally different patterns). Thus, the noise for the resulting <em>revenue </em>turns out to be impossible to reliably model.</li><li><strong><em>Noise from the data collection process</em></strong>. Data connections will never be 100% reliable and your system will have to continue to function reasonably well under your typical <em>data collection noise</em> patterns. Examples of <em>data collection noise</em> patterns we encounter so frequently that they influenced our predictive strategy: duplicate events, missing events for certain dates (recovered later, or never recovered), aberrations on all numerical values (such as prices), encoding inconsistencies.</li><li><strong><em>Noise artefacts from earlier algorithms in the processing pipeline.</em></strong> When you combine multiple algorithms, you multiply noise. If you’re not careful, you can end up with big problems.</li></ul><blockquote><em>Do not assume that you can model noise. Do not assume that you can get rid of noise. Noise happens. Your architecture must survive noise.</em></blockquote><p>This is why you need to have noise reduction in mind at all steps of the design process. In practice: whenever you combine algorithm X with algorithm Y (eg, a feature engineering algorithm with a scoring algorithm), make sure that all possible artefacts of algorithm X are damped and not amplified by algorithm Y.</p><h3>2. Design indestructible controllers</h3><p>A controller is a piece of software that handles the final decision-making downstream of the core predictive engine.</p><p>Compared to fancy machine-learning algorithms, controllers look simple and it is easy to overlook them. <strong>However</strong>, <strong>they are the most critical components for systemic stability. So, never overlook controllers! </strong>Consider them as external regularizers to your learning problem.</p><p>The <em>campaign dispatch</em> problem provides a great example. A key feature of Tinyclues is the possibility for marketing teams to optimize customer marketing plans where each customer is eligible to multiple campaigns but there is a cap on how many campaigns he can receive per week.</p><p>Let’s look at the simplified problem with two campaigns A and B and max_campaigns_per_user=1.</p><p>Upstream of the controller, your core predictive engine generates well-calibrated scores: for each user in the customer database, you have good predictions of the expected revenues if the user receives campaign A or campaign B.</p><p>Here the controller is the algorithm that consumes these scores and splits the database into customers receiving A and customers receiving B.</p><p>In the early days of Tinyclues, the first controller we tried was <strong>Argmax</strong>: each customer receives the campaign where his score is maximal. <strong>Argmax is the logical mathematical solution to the problem but, in the real world, it is an absolute disaster. </strong>Here is why:</p><ul><li>First, Argmax amplifies all the flaws from the scoring algorithm. It greedily allocates customers to wherever their score overfits the most.</li><li>Even if there was zero overfit, Argmax would still exhibit unacceptable macro behaviors. Scores between different campaigns tend to be highly correlated: some customers are just better customers, whatever the campaign. If the ROC curve for campaign A is bit sharper than the ROC curve for campaign B, then all the best customers will be allocated to A, which results in really bad outcomes (for customers and for the marketing team).</li></ul><p>A typical workaround consists in the score randomization via <em>Thompson sampling</em>. That may work under the hypothesis of simple noise model and would result into explore/exploit extra parameter for this controller (yet it did not provide a suitable macro behaviour in our case).</p><p>Looking more carefully at the situation, we realized that scores really have two components: an <em>intensity</em> component (how likely a customer will buy any product) and a <em>specificity</em> component (assuming a customer will buy a product, how likely will it be a product associated with campaign A?).</p><p>While it is tricky to reliably separate the two components, knowing that they exist helped us design an incredibly robust dispatch controller.</p><p>Rather than naive numerical optimization, our source of inspiration was combinatorial thinking taking into account business constraints and marketing savviness. This approach resulted in a dispatch algorithm that displays healthy behaviors even in the presence of overfit or scoring calibration issues.</p><h3>3. Monitor macro behaviors, not just loss functions</h3><p>The dispatch problem also illustrates a typical control issue that arises if you blindly trust machine-learning algorithms: loss functions that are defined as sums of local penalties fail to penalize unhealthy macro behaviors.</p><p>You may end up in situations where the optimization is OK at a local level (eg, each customer receives content that is reasonably relevant) but catastrophic at a macro level (eg, there are massive imbalances in the allocation of campaigns, or customers are locked in a content bubble.)</p><blockquote><em>The real-world problem that you are solving cannot be modelled by a technical loss function.</em></blockquote><blockquote><em>Monitor the macro behavior of your system with high level metrics that are pragmatic and business-centric.</em></blockquote><p>The high-level metrics of self-driving cars (miles between disengagements, accident rates, dispatch reliability) are different from the loss functions used by their machine-learning algorithms.</p><p>At Tinyclues, we closely monitor the macro behavior of our clients’ marketing programs across multiple months, with high-level metrics that capture audience overlaps between campaigns and customer-exposure rates for different engagement levels.</p><p>Building the right set of metrics is business-specific and solution-specific. It requires a deep understanding of the real-world problem that you are solving and the failure modes of your system. And if you’re lucky enough, you may find a way to integrate those metrics into your learning losses.</p><blockquote>Building these metrics takes years and constitutes a form of data network effect.</blockquote><h3>4. Beware hyperparameter creep</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tYAg18ZjX84mVU3U" /></figure><p>Exposing too many hyperparameters can be a symptom that a predictive system isn’t ready for production. If you can’t figure out decent default values for your core hyperparameters, or if you expect to have to constantly tweak them in production, then your system is immature and probably unstable.</p><p>Machine learning is about machines that learn. If you have too many hyperparameters, if they change too often, then you’re adding an unpenalized and uncontrollable human-learning metalayer.</p><p>This is better understood in organizational terms. At Tinyclues, we have two Data Science teams:</p><ul><li>the R&amp;D Data Science team, which designs the predictive stack;</li><li>the Data Ops team, responsible for client setups and predictive quality in production.</li></ul><p>The Data Ops team has serious machine-learning skills but their job isn’t to second-guess our predictive engine. They have access to a complete admin back-office but the core predictive hyperparameters have very clear default values.</p><p>The very few parameters that are modified on a client-by-client basis are all related to client-specific business goals. A typical example is the parameter that controls the “<em>intensity-vs-specificity”</em> trade-off (favoring short-term revenue by overexposing the most active customers vs re-engaging less active customers.)</p><p>But the job of the Data Ops team isn’t to adjust gradient descent rates, penalization coefficients or dimensions of latent feature layers. <strong>The job of the R&amp;D Data Science team is to design a system that is 100% controllable with business-centric inputs only.</strong></p><h3>5. Always fail on the right side</h3><p>Let’s face it: horrible things will happen. Someone will push a predictive bug in production. Weird anomalies in incoming data will mess up with key components of your stack. Users will do crazy things that you never expected.</p><p>Of course you need to build the QA and the instrumentation layer that will allow you to catch these incidents early on. But the best way to prepare is to build each component with the worst-case-scenario in mind.</p><p><strong>Failure modes are hugely asymmetric in terms of business outcomes.</strong> A confused self-driving car should rather pull over rather than randomly accelerate. But on the left-lane of a highway it shouldn’t brutally decide to park.</p><p>Based on your domain-specific knowledge, identify the preferred failure mode for each component of your stack and design your controllers with these failure modes in mind.</p><blockquote><em>All systems fail.</em></blockquote><blockquote><em>Antifragile systems fail on the right side.</em></blockquote><h3>Suggested further reading</h3><ul><li>On regularizing outputs of recommendation systems based on business goals: <a href="https://dl.acm.org/doi/10.1145/3240323.3240372">https://dl.acm.org/doi/10.1145/3240323.3240372</a></li><li>On Combinatorial Multi-Armed Bandit: <a href="http://proceedings.mlr.press/v28/chen13a.pdf">http://proceedings.mlr.press/v28/chen13a.pdf</a></li><li>On uncertainty estimation in Deep Learning: <a href="http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles.pdf">http://papers.nips.cc/paper/7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles.pdf</a></li><li>Thompson sampling methods comparison: <a href="https://arxiv.org/pdf/1802.09127.pdf">https://arxiv.org/pdf/1802.09127.pdf</a></li><li>Bayesian Models for Click-Through Rate Prediction: <a href="https://quinonero.net/Publications/AdPredictorICML2010-final.pdf">https://quinonero.net/Publications/AdPredictorICML2010-final.pdf</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47d6b3216c2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tinyclues-vision/5-design-principles-for-antifragile-predictive-systems-47d6b3216c2b">5 design principles for antifragile predictive systems</a> was originally published in <a href="https://medium.com/tinyclues-vision">Tinyclues Vision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>