<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Katerina Henclova on Medium]]></title>
        <description><![CDATA[Stories by Katerina Henclova on Medium]]></description>
        <link>https://medium.com/@kate.henclova?source=rss-f74e89338760------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Mz7KD2MYkEuQ8KsYUH0bNQ.png</url>
            <title>Stories by Katerina Henclova on Medium</title>
            <link>https://medium.com/@kate.henclova?source=rss-f74e89338760------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:07:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kate.henclova/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Benchmarking Alternative Feature Selection Algorithms (Part 3: Results on Synthetic Problems)]]></title>
            <link>https://medium.com/@kate.henclova/benchmarking-alternative-feature-selection-algorithms-part-3-results-on-synthetic-problems-0ccb1c12a4c3?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ccb1c12a4c3</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[feature-selection]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Wed, 14 Oct 2026 11:01:02 GMT</pubDate>
            <atom:updated>2026-10-14T11:01:02.249Z</atom:updated>
            <content:encoded><![CDATA[<p>Six algorithms search for three planted answers, needle-in-a-haystack style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AO1DVp0uG2Z2POrcVXzIxA.jpeg" /></figure><h3>Where we are in the benchmarking series</h3><p>On the synthetic benchmark described in detail in <a href="https://medium.com/datamole/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee">Part 1</a>, three equally valid answers are planted in the data, so we know exactly what an algorithm should find. Six algorithms (introduced in <a href="https://medium.com/@kate.henclova/benchmarking-alternative-feature-selection-algorithms-part-2-the-algorithms-20b42ae1d6d5">Part 2</a>) went looking.</p><p>This article constitutes Part 3, where we show that GEMSS outperforms its competitors.</p><p>Part 4 will take us to real data, validating the claim that GEMSS indeed finds practically usable solutions and it does so better than the other methods.</p><p>For further details, we refer the reader to <a href="https://arxiv.org/abs/2602.08913">this paper</a>.</p><h3>The setup</h3><h4>The default synthetic problem</h4><p>The default benchmark problem contains <strong><em>n</em> = 100 samples</strong>. It hides three solutions consisting of 10 features among <strong><em>p</em> = 200 features</strong>. Every solution predicts the target equally well, by construction. One knob sets how many features the three solutions share: 0, 2, 4 or 8. That is, <strong>depending on the overlap, we want the algorithms to correctly identify 30, 26, 22, and 14 features out of 200</strong>, respectively. The greater the overlap, the harder the solutions are to tell apart.</p><h4>The methods</h4><p>There are 6 tested methods:</p><ol><li><strong>Iterative masking </strong>(= reruns of LASSO with masked solutions),</li><li><strong>RLE</strong> (Randomized-LASSO Ensemble),</li><li><strong>EnumLASSO</strong> (Enumeration LASSO),</li><li><strong>ALFESE </strong>(Alternative Feature Selection, using Mutual Information score),</li><li><strong>BB-SSL </strong>(Bayesian Bootstrap Spike-and-Slab LASSO),</li><li><strong>GEMSS</strong> (Gaussian Ensemble for Multiple Sparse Solutions).</li></ol><p>Each method is tuned on its own terms and must return three solutions of ten features.</p><h4>Scoring metrics</h4><p>Two scores judge the result:</p><ul><li><strong>Union-F1</strong> pools the returned features and compares them with the pooled planted features. It asks whether the right features came back at all, wherever they landed among the three solutions, and it penalizes wrong extra ones.</li><li><strong>Dissimilarity</strong> asks how different the returned solutions are from each other, from 0 (identical) to 1 (nothing in common). It is read against the true value, not maximized.</li></ul><p><a href="https://medium.com/datamole/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee">Part 1</a> explains why both are scored this way.</p><blockquote>Note that Union-F1 is rather strict: it counts every planted feature missed AND every wrong feature picked.</blockquote><h4>Averaging independent runs</h4><p>Every result below is averaged over 15 to 25 runs with different random seeds.</p><h3>1. Results on the default problem</h3><h4>1.1 Which methods find the hidden answers?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_b1Rq04bFEPijRUgWlbYxg.png" /></figure><p><strong>GEMSS recovers the planted features best at every level of overlap. </strong>ALFESE comes second.</p><p><strong>The gap widens as the answers overlap more.</strong> When the true answers share most of their features they sit close together, and a method has to keep nearby answers apart instead of merging them into one.</p><p><strong>A guarantee did not translate into recovery.</strong> BB-SSL is the only one of the six whose draws are proven to contract around the truth, if only asymptotically. Yet it lands far below GEMSS and ALFESE, and below even iterative masking at 3 of 4 overlap levels.</p><h4>1.2 How different are the discovered solutions?</h4><p>Union-F1 does not say whether the features came back as different answers — that is why we observe dissimilarity as well. As the planted answers share more features, they become more alike. <strong>A method that finds them should follow the true dissimilarity.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NEJrWWTARDx190hkgxgZ9w.png" /></figure><p>We see that iterative masking and BB-SSL stay at the maximum whatever the truth. Iterative masking removes what it has found, so its answers cannot share anything. BB-SSL behaves the same way here.</p><p>RLE and EnumLASSO barely react to the overlap.</p><p>ALFESE, though promising, collapses at high overlap, where its three answers are identical (even though the corresponding parameter was being optimized).</p><p><strong>GEMSS moves with the truth and does not collapse.</strong> Its solutions grow more alike as the planted ones do, closest to the truth at middle overlap.</p><p><a href="https://medium.com/@kate.henclova/benchmarking-alternative-feature-selection-algorithms-part-2-the-algorithms-20b42ae1d6d5">Part 2</a> described what sets how different each method’s solutions are. Here it shows: where the procedure sets it, as in iterative masking, the solutions stay maximally different whatever the data holds.</p><h4>1.3 The fairness checks</h4><p>Is it possible we gave GEMSS an unfair advantage?</p><p><strong>1.3.1 Could RLE simply run longer?</strong></p><p>RLE finds several answers by restarting many times. Each restart is cheap, so it might just be under-resourced.</p><p><strong>Conclusion: more restarts do not help.</strong> RLE levels off within a few hundred restarts, a hundred times more changes nothing, and a single GEMSS fit stays far ahead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1QrC_CbbstE_TzDuwkWqQ.png" /></figure><p><strong>1.3.2 Did GEMSS need to know the answer to be tuned?</strong></p><p>Its main setting is the number of mixture components, <em>K</em>, explained in <a href="https://medium.com/@kate.henclova/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">the theory post</a>. GEMSS fits <em>K</em> components and clusters them down to the three solutions it returns. The results above use the best <em>K</em> for each problem, which in practice you would not know.</p><p><strong>Conclusion: no oracle is needed to set up K.</strong> Choosing <em>K</em> by the model’s own fit score, the ELBO, needs no knowledge of the true answers and costs very little recovery. The result still beats every competitor at its best-tuned setting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GIFq3n3YP_bBy64EUEKvHg.png" /></figure><h3>2. Stress-testing</h3><p>This is where things get most interesting: how do the methods behave under pressure?</p><p>We run only three methods that are worth comparing. GEMSS faces ALFESE, its closest competitor on this benchmark, and RLE, the method a data scientist can reach for in practice.</p><h4><strong>2.1 High noise</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejujiuikS1lJBcleeGoZJg.png" /></figure><p><strong>GEMSS keeps the highest recovery and degrades gracefully under noise.</strong> ALFESE degrades quickly and, at the highest noise level, falls to RLE’s level.</p><h4><strong>2.2</strong> <strong>Missing values</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*36K4LzJAyM2BQElA-MsWew.png" /></figure><p><strong>GEMSS natively supports missing values</strong> because it evaluates its likelihood only on the observed entries, i.e. it needs no imputation. RLE needs the gaps filled in first — conveniently for RLE, we use the optimal imputation here (mean value) so its performance is rather overestimated. ALFESE cannot run at all.</p><h4><strong>2.3 Extremely high dimension and n &lt;&lt; p</strong></h4><p>The figure below shows the Union-F1 scores for GEMSS (green), ALFESE (dark gray) and RLE (light gray) for two overlap regimes. The winning average scores are bolded and the min-max intervals are shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BtVh4BLEnA7mLxt2VplPQg.png" /></figure><p><strong>GEMSS ranks first in 15 of the 18 settings (green cells) and is fully competitive in the rest</strong>, where ALFESE leads only within the error bars. The lead holds up to <em>p</em> = 5,000. Only at <em>p</em> = 10,000 (not shown), do GEMSS and ALFESE meet, as every method degrades.</p><blockquote><strong>Scores of .5 to .7 here are practically far better than they look at first sight.</strong> Take <em>n</em> = 30 and <em>p</em> = 5,000 at overlap 2: the three planted solutions use 26 features hidden among 5,000, and the method sees only 30 samples. A random pick of (optimistically) 26 features would hit about 0.14 planted features on average (26 × 26 / 5,000), for a union-F1 of about .0052. GEMSS’s result .52 is 100× that.</blockquote><h3>Limitations of this evaluation</h3><ol><li>The data are synthetic and linear.</li><li>Every planted answer has the same size.</li><li>Each method was tuned on its own grid and is reported at its best setting we were able to find (programmatically).</li><li>GEMSS is my own method.</li></ol><h3>Conclusion</h3><p>We have shown that GEMSS clearly outperforms its competitors on the synthetic benchmark problems where mainly the recovery of all possible supporting features is being scored.</p><h3>Resources on GEMSS</h3><ul><li><strong>Paper:</strong> <a href="https://arxiv.org/abs/2602.08913">GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems</a> (Henclová &amp; Šmídl, 2026)</li><li><strong>Open benchmarking framework:</strong> <a href="https://github.com/kat-er-ina/gemss_testing">github.com/kat-er-ina/gemss_testing</a></li><li><strong>GEMSS code/package:</strong> <a href="https://github.com/kat-er-ina/gemss">github.com/kat-er-ina/gemss</a> (pip install gemss)</li><li><strong>No-code app for GEMSS:</strong> <a href="https://huggingface.co/spaces/kat-er-ina/gemss">huggingface.co/spaces/kat-er-ina/gemss</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ccb1c12a4c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking Alternative Feature Selection Algorithms (Part 2: The Algorithms)]]></title>
            <link>https://blog.datamole.ai/benchmarking-alternative-feature-selection-algorithms-part-2-the-algorithms-20b42ae1d6d5?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/20b42ae1d6d5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 13:46:12 GMT</pubDate>
            <atom:updated>2026-10-12T18:58:58.671Z</atom:updated>
            <content:encoded><![CDATA[<p>The phenomenon of a dataset supporting several valid solutions keeps being rediscovered and renamed, e.g. the Rashomon effect, predictive multiplicity or statistically equivalent signatures. This post introduces six algorithms we later compare in our benchmark.</p><blockquote><strong>This is part 2 of 4</strong> on benchmarking algorithms that return several valid feature subsets. <a href="https://medium.com/datamole/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee">Part 1</a> built the benchmark.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vOBngHRXDwBXGsBuY6X9CA.jpeg" /><figcaption>The six competing algorithms.</figcaption></figure><h3>One problem, many names</h3><p>When features far outnumber samples, or are strongly correlated, several small feature sets can explain the data equally well. A standard selector such as the <a href="https://en.wikipedia.org/wiki/Lasso_(statistics)">LASSO</a> typically returns just one of them.</p><blockquote>Why someone would want to see all the valid sets is the subject of an earlier post on <a href="https://medium.com/datamole/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb">assisted discovery</a>.</blockquote><p>Different fields have their own name for this:</p><ul><li><strong>Rashomon effect.</strong> Named after <a href="https://en.wikipedia.org/wiki/Rashomon">Kurosawa’s film</a>, in which witnesses give contradicting but individually coherent accounts of the same event.</li><li><strong>Predictive multiplicity.</strong> The fairness view: competing models of near-equal accuracy can give different predictions (see the intro to <a href="https://medium.com/datamole/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">this article of mine</a>).</li><li><strong>Statistically equivalent signatures.</strong> The bioinformatics term, with dedicated software in the R package MXM.</li><li><strong>Multiple Markov boundaries.</strong> The causal-discovery term. A <a href="https://en.wikipedia.org/wiki/Markov_blanket">Markov boundary</a> is a minimal set of variables that makes all other variables irrelevant for predicting the target.</li><li><strong>Multimodal optimization.</strong> Finding all or most of a problem’s optima, not just the single best one. Evolutionary algorithms have long tackled it with <a href="https://en.wikipedia.org/wiki/Evolutionary_multimodal_optimization">niching</a>: techniques that keep the population spread over several promising regions, so the search does not converge to a single solution.</li></ul><h3>Algorithm families by their output</h3><p>Most important grouping of the below-described algorithms is by what they actually return.</p><h4>One solution per run</h4><p>Solve, change something, solve again. The change is either a random perturbation (RLE, BB-SSL) or the removal of what was just found (iterative masking).</p><h4><strong>Several solutions per run</strong></h4><p>The method looks for more than one answer internally and returns them together: GEMSS, ALFESE in its simultaneous mode, and EnumLASSO.</p><h3>The six competing methods</h3><p>Since this field is only emerging, we struggled to find true competitors for GEMSS. Below is the list of both dedicated and somewhat improvised methods used in the comparison.</p><blockquote><strong>A support is the set of features with nonzero coefficients. </strong>One support corresponds to one solution, so I use the two words interchangeably. However, strictly speaking, a solution is only found when coefficients are computed by a downstream classification/regression model.</blockquote><h4>1. Iterative masking</h4><p>This is the most naive approach: fit a selector, remove the features it picked, fit again on what is left, and repeat. In our benchmark, the selector is LASSO.</p><p><strong>The resulting solutions never share any features, </strong>because their total dissimilarity is designed by the procedure, not learned from the data. Obviously, when the true alternative solutions overlap, this approach yields poor results.</p><h4>2. Randomized-LASSO Ensemble (RLE)</h4><p>In practice, this is the method a data scientist reaches for. Resample the rows, reweight the penalties at random, solve an ordinary LASSO, and repeat many times. Then cluster the supports that come back. <strong>Each solution comes from a differently randomized problem, so the differences between them are down to chance.</strong></p><p>The randomized LASSO is closely related to stability selection, which keeps the features chosen most often across such runs. RLE keeps the different supports instead.</p><h4>3. Enumeration LASSO (EnumLASSO)</h4><p>EnumLASSO lists the <em>k</em> supports with the lowest LASSO objective, explicitly so that a person can inspect several candidates. Its authors prove that enumerating this way recovers a true feature set exactly, under weaker conditions than the ordinary LASSO needs.</p><p><strong>Nothing in its objective asks the sets to differ.</strong> How far apart they end up is left to the LASSO objective.</p><h4>4. Alternative Feature Selection (ALFESE) with Mutual Information feature selector</h4><p>ALFESE makes diversity an explicit requirement in a constrained optimization problem.</p><p>It <strong>maximizes a feature-quality score under the constraint that every pair of returned sets differs by at least a threshold <em>τ</em></strong>. The difference is measured with the <a href="https://en.wikipedia.org/wiki/Dice-S%C3%B8rensen_coefficient">Dice coefficient</a>. At <em>τ</em> = 0 the sets may be identical, at <em>τ</em> = 1 they may not share a single feature.</p><p>There are two run options that determine whether the feature sets are found one after another (the sequential mode) or all at once (the simultaneous mode). True to our problem, we use the latter.</p><p>The quality score comes from a conventional feature selector that ALFESE wraps. There are multiple feature selectors available in the package and we chose the best-performing one based on mutual information.</p><p>Mutual information rates each feature against the target on its own, so five correlated copies of the same signal all rate highly. The constraint is what keeps the returned sets apart. <strong>The amount of diversity is a setting the user chooses</strong>, and raising <em>τ</em> pushes the sets apart whether or not the data supports it.</p><h4>5. Bayesian Bootstrap Spike-and-Slab LASSO (BB-SSL)</h4><p>BB-SSL starts from the spike-and-slab LASSO, a prior under which each coefficient is either shrunk almost to zero or left free to be large. It draws approximate posterior samples by <strong>solving many randomly perturbed versions of the problem</strong>: the likelihood gets random weights, and the prior gets random shrinkage targets. The idea builds on the weighted likelihood bootstrap.</p><p>Its authors prove that these draws contract around the truth at a near-optimal rate, in sparse normal-means problems and in high-dimensional regression. No other method here has a result of that kind, though it is an asymptotic one. (However, as we shall see in Part 3, this theoretical result actually does not translate to good performance.)</p><h4>6. Gaussian Ensemble for Multiple Sparse Solutions (GEMSS)</h4><p>GEMSS also puts a structured spike-and-slab prior on the coefficients. That way, sparsity and multiplicity are encoded.</p><p>When several sparse feature sets explain the data equally well, the resulting posterior has several peaks.</p><p>However, the such a posterior is intractable, so GEMSS approximates it with a Gaussian mixture. After a single optimization run (with Stochadtic Gradient Descent), each mixture component yields one candidate solution.</p><p><strong>The solutions may share features, and how much they share comes out of the fit.</strong></p><p>One term in the objective, the entropy of the mixture, penalizes two components that sit on the same peak, so they spread out over different peaks. An optional penalty on the Jaccard similarity between solutions allows for direct user control (our experiments leave it off). <a href="https://medium.com/@kate.henclova/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">This theory post</a> explains the method in more detail.</p><blockquote>GEMSS and BB-SSL go after the same kind of target, a sparse posterior with several peaks. BB-SSL samples it, while GEMSS approximates it with a mixture.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fGsKYGTkAyVM0yQJHY0FCQ.jpeg" /></figure><h3>Nearby methods that solve a different problem</h3><p><strong>Other related methods study the set of all good models, but none returns a shortlist of sparse linear solutions. For example:</strong></p><ul><li><strong>Model Class Reliance.</strong> The range of a feature’s importance across all well-performing models in a model class the user specifies.</li><li><strong>Variable Importance Clouds.</strong> Each feature’s importance across all good models.</li><li><strong>TreeFARMS.</strong> Every near-optimal sparse decision tree: concrete models, but a complete set of trees.</li></ul><h3>Where we are in the benchmarking series</h3><p>This was Part 2 of 4: introducing the algorithms.</p><ul><li><a href="https://medium.com/datamole/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee">Part 1</a> built the benchmark: synthetic data with several equally valid solutions planted in it, scored on which features come back, not on how well a model predicts.</li><li>Part 3 (coming soon) tests all six methods on the synthetic benchmark.</li><li>Part 4 (coming soon) takes the methods to real data.</li></ul><h3>Further reading</h3><p>Bach, J. (2023). <a href="https://arxiv.org/abs/2307.11607">Finding Optimal Diverse Feature Sets with Alternative Feature Selection</a>. arXiv:2307.11607.</p><p>Bach, J., &amp; Böhm, K. (2024). <a href="https://doi.org/10.1007/s41060-024-00527-8">Alternative Feature Selection with User Control</a>. International Journal of Data Science and Analytics.</p><p>Breiman, L. (2001). <a href="https://doi.org/10.1214/ss/1009213726">Statistical Modeling: The Two Cultures</a>. Statistical Science, 16(3), 199–231.</p><p>Dong, J., &amp; Rudin, C. (2020). <a href="https://arxiv.org/abs/1901.03209">Exploring the Cloud of Variable Importance for the Set of All Good Models</a>. Nature Machine Intelligence, 2, 810–824.</p><p>Fisher, A., Rudin, C., &amp; Dominici, F. (2019). <a href="https://arxiv.org/abs/1801.01489">All Models Are Wrong, but Many Are Useful</a>. JMLR, 20, 1–81.</p><p>Hara, S., &amp; Maehara, T. (2017). <a href="https://doi.org/10.1609/aaai.v31i1.10793">Enumerate Lasso Solutions for Feature Selection</a>. AAAI 31, 1985–1991.</p><p>Henclová, K., &amp; Šmídl, V. (2026). <a href="https://arxiv.org/abs/2602.08913">GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems</a>.</p><p>Lagani, V., Athineou, G., Farcomeni, A., Tsagris, M., &amp; Tsamardinos, I. (2017). <a href="https://www.jstatsoft.org/article/view/v080i07">Feature Selection with the R Package MXM</a>. Journal of Statistical Software, 80.</p><p>Li, X., Epitropakis, M. G., Deb, K., &amp; Engelbrecht, A. (2016). <a href="https://doi.org/10.1109/TEVC.2016.2638437">Seeking Multiple Solutions: An Updated Survey on Niching Methods and Their Applications</a>. IEEE TEVC, 21(4), 518–538.</p><p>Marx, C., Calmon, F. du Pin, &amp; Ustun, B. (2020). <a href="https://arxiv.org/abs/1909.06677">Predictive Multiplicity in Classification</a>. ICML.</p><p>Meinshausen, N., &amp; Bühlmann, P. (2010). <a href="https://arxiv.org/abs/0809.2932">Stability Selection</a>. JRSS-B, 72(4), 417–473.</p><p>Newton, M. A., &amp; Raftery, A. E. (1994). <a href="https://doi.org/10.1111/j.2517-6161.1994.tb01956.x">Approximate Bayesian Inference with the Weighted Likelihood Bootstrap</a>. JRSS-B, 56(1), 3–26.</p><p>Nie, L., &amp; Ročková, V. (2023). <a href="https://doi.org/10.1080/01621459.2022.2025815">Bayesian Bootstrap Spike-and-Slab LASSO</a>. Journal of the American Statistical Association.</p><p>Ročková, V., &amp; George, E. I. (2018). <a href="https://doi.org/10.1080/01621459.2016.1260469">The Spike-and-Slab LASSO</a>. JASA, 113(521), 431–444.</p><p>Statnikov, A., Aliferis, C. F., et al. (2013). <a href="https://jmlr.org/papers/v14/statnikov13a.html">Algorithms for Discovery of Multiple Markov Boundaries</a>. JMLR, 14, 499–566.</p><p>Tsamardinos, I., Lagani, V., &amp; Pappas, D. (2012). <a href="https://www.scribd.com/document/166480556/MultipleSignatureHSCBB12v0-2">Discovering Multiple, Equivalent Biomarker Signatures</a>. PRIB, 152–163.</p><p>Xin, R., Zhong, C., Chen, Z., Takagi, T., Seltzer, M., &amp; Rudin, C. (2022). <a href="https://arxiv.org/abs/2209.08040">Exploring the Whole Rashomon Set of Sparse Decision Trees</a>. NeurIPS 35, 14071–14084.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20b42ae1d6d5" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/benchmarking-alternative-feature-selection-algorithms-part-2-the-algorithms-20b42ae1d6d5">Benchmarking Alternative Feature Selection Algorithms (Part 2: The Algorithms)</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Benchmarking Alternative Feature Selection Algorithms (Part 1: Design)]]></title>
            <link>https://blog.datamole.ai/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/972722b8e6ee</guid>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Wed, 02 Sep 2026 07:25:38 GMT</pubDate>
            <atom:updated>2026-09-02T07:25:38.068Z</atom:updated>
            <content:encoded><![CDATA[<p>How do you benchmark an algorithm whose job is to return several valid answers? The obvious way to score it turns out to be wrong.</p><p><strong>TL;DR:</strong> To compare algorithms that return several equally valid answers, we must design a synthetic benchmark where we plant those answers. The hard part is the metric. Pairing each returned solution against “its” planted counterpart is the obvious choice and it is wrong, because which solution a feature belongs to is not well defined. Pool the features and score the union instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gz8_w20ZA_RkMhI4ZAr13w.jpeg" /></figure><h3>The predictive multiplicity problem: some models are right, not all are useful</h3><p>You are looking for biomarkers of a specific disease. You have a metabolomics dataset: a few dozen patients, several hundred measured analytes. A feature selector returns a small subset that separates the sick from the healthy almost perfectly. Then you look at what it picked: general inflammation markers.</p><p>Mathematically, nothing went wrong. The correlation is real, it survives cross-validation, and sick people are indeed inflamed. It is just the wrong answer to the question you asked — a hundred other conditions would have produced the same signal.</p><p><strong>“Statistically as good as” and “useful for what I actually want” are different questions, and a feature selector is not equipped to tell them apart.</strong></p><p>And those inflammation markers were almost certainly not the only viable subset. A disease-specific one may well have been sitting in the same data, scoring slightly lower, never surfaced — because the algorithm was built to return one answer, and it did.</p><blockquote>This example is a typical problem of predictive multiplicity or the Rashomon effect or alternative feature selection (and a few more names), depending on whom you ask. They are underdetermined problems that can have more than one solution.</blockquote><p><strong>The fix is to change what the algorithm delivers</strong>: surface the alternative valid models and let a domain expert pick the one that makes sense in the real world — scientifically, economically or ethically. I’ve written about that framing, <a href="https://medium.com/datamole/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb">assisted discovery</a>, and about <a href="https://medium.com/@kate.henclova/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">the method I built to do it</a>.</p><p>Which leaves an obvious question. <strong>If you are going to trust an algorithm to hand you a shortlist, you need to know it is a good shortlist.</strong></p><blockquote>Data from physical measurement carry measurement errors and few samples, producing underdetermined systems that naturally admit multiple sparse solutions. As such, in this work, we assume each set of explanatory features is <strong>sparse</strong> relative to the full feature space.</blockquote><h3>Why this makes evaluation genuinely hard</h3><p>That turns out to be remarkably hard to establish. An algorithm’s job here is to hand an expert several defensible explanations instead of one arbitrary pick — and neither of the two things you would naturally reach for will tell you whether it managed that.</p><p><strong>Prediction metrics alone cannot tell you.</strong> A method can return three highly predictive feature sets having missed the actual mechanism in the data, e.g. in the form of cosmetic re-dressings of the same useless solution. On a leaderboard those look identical to a genuine discovery. Worse yet, the <em>more useful</em> solution may get disqualified for scoring somewhat lower even though it is still perfectly acceptable.</p><p><strong>Real data cannot tell you either.</strong> No dataset arrives labelled “here is the true and complete set of alternative explanations”. What you <em>can</em> do on real data is show that several solutions with comparable, usable performance exist. This is good evidence that a method is practically useful, but nothing you can score recovery against.</p><h3>Manufacturing a ground truth you can actually check</h3><p>In this situation, we stop looking for the benchmark in the wild and build it deliberately:</p><ol><li><strong>Generate the data from one shared low-rank signal.</strong> Signal and noise features are matched in marginal variance, so no method can cheat by chasing variance instead of structure.</li><li><strong>Plant several distinct feature subsets that each reconstruct that signal.</strong> They are then equally valid predictors by definition rather than by assumption. Each planted subset recovers the noiseless signal at R² ≈ 0.99, because each one is a <em>support</em>: the set of features a solution selects. (By design, and unlike real problems, every planted support has the same number of features.)</li><li><strong>Parametrize solution overlap.</strong> Set up deliberately how many features are shared across all planted supports. No overlap implies disjoint solutions.</li><li><strong>Add realistic mess. </strong>Gaussian noise, missing values, class imbalance, high dimension are added to make the solution recovery harder.</li></ol><p>Binarizing the latent target turns the whole thing into a classification benchmark, so <strong>one generator serves both regression and classification</strong>.</p><p><strong>Step 3 is where the real difficulty lives.</strong> Heavily overlapping alternatives sit close together in the space of possible answers, so a method has to tell nearby-but-distinct apart rather than lumping them into one. Judging by the datasets we have worked with, that is also the regime real problems live in.</p><blockquote><strong>Our main benchmark (as published) is deliberately small and hard:</strong> <em>n</em> = 100 samples, <em>p</em> = 200 features, three planted supports of ten features each, with the shared core swept across 0, 2, 4 and 8 of those ten, every condition averaged over 25 random seeds. Separate sweeps push noise to twenty times baseline, the missing rate to 50%, and the dimension to <em>p</em> = 5000 and beyond.</blockquote><h3>Measuring recovery, not accuracy …and the trap in the first-choice metric</h3><p>With ground truth in hand you need a metric that scores <em>structure recovered</em> rather than predictions made. That sounds easy until you write it down.</p><p><strong>The first-choice metric is one-to-one set matching. </strong>Pair each returned solution with “its” planted support, score each pair for feature overlap, average across pairs. We used the Hungarian algorithm to assign the pairs optimally rather than by index, which felt like the careful version of the idea. However, it was a blind alley.</p><p>The reason it misleads is actually just first-semester linear algebra. Every planted support in this <em>linear</em> benchmark reconstructs the same signal, so the features in each span the same low-dimensional subspace — and they are not the only sets that do. If two features are strongly correlated, or more generally if one is a linear combination of others, swapping one for the other inside a support does not affect the span, and therefore the solution either.</p><p><em>Stated precisely: any equally-sized set of features whose span contains the signal is a viable solution.</em></p><p><strong>Which means “which solution does this feature belong to” does not have a well-defined answer.</strong> A method can recover every relevant feature and still distribute them across its solutions differently from the way you planted them, without having got anything wrong.</p><h4>A counterexample</h4><p>Here is a small case that breaks the set-matching metric. Two planted supports of three features sharing a common core, where features 2 and 4 are interchangeable and so are 3 and 5:</p><ul><li><strong>planted:</strong> {1, 2, 3} and {1, 4, 5}</li><li><strong>returned:</strong> {1, 4, 3} and {1, 2, 5}</li></ul><p>Every planted feature came back, nothing spurious was added, and both returned sets span the signal. This is a perfect recovery. Score it by pairing, though, and each pair shares two features out of three: F1 = 0.67. Swap which candidate goes with which support and you get 0.67 again, so the Hungarian assignment cannot rescue it. <strong>Both pairings are equally penalized, because the pairing was never the problem.</strong></p><h4>The fix is to stop pairing</h4><ul><li><strong>Compare unions, not pairs.</strong> Take the union of all features returned across all candidate solutions, and compare it against the union of all features across all planted supports. In the example above that is {1,2,3,4,5} against {1,2,3,4,5}: a perfect score, correctly.</li><li><strong>Score that with F1.</strong> Among the available set-matching metrics we found F1 the most descriptive here. It balances recall of the ground truth against precision of the returned set, which suits both the use case and the way the ground truth was built.</li></ul><p>We call this the <strong>Union-F1 score </strong>in order to distinguish this metric from the predictive quality of an individual solution.</p><h4>Dissimilarity as a complementary descriptor</h4><p>We also measure <strong>dissimilarity</strong>: the average pairwise distance (one minus Jaccard similarity) between returned solutions. It carries its own trap. Maximum dissimilarity is trivially achievable by forcing zero overlap, and achieving it that way means nothing.</p><p>Therefore, dissimilarity is not a quality score but a descriptor, read against the <em>true</em> dissimilarity planted in the data — and that target is available in closed form.</p><p><strong>The goal is to match the true dissimilarity, not maximization.</strong> A method reporting 1.00 at every overlap level is telling you about its own construction rather than about your data.</p><blockquote>For supports of <em>D</em> features sharing a core of <em>k</em>, the planted mean pairwise 1−Jaccard is (2<em>D</em>−2<em>k</em>) / (2<em>D</em>−<em>k</em>), which for the main benchmark’s ten-feature supports gives <strong>1.00, 0.89, 0.75 and 0.33</strong> as the shared core goes 0, 2, 4, 8.</blockquote><h3>Comparing methods that do not share much</h3><p>Two more practical problems.</p><p><strong>The easy one is timing.</strong> The candidate methods span several libraries and languages, so wall-clock would mostly measure whose implementation happened to be faster. Counting computational budget (iterations/restarts/fits) is the standard fix when benchmarking optimization algorithms.</p><p><strong>The harder problem is that a “run” does not mean the same thing to every method.</strong> There are two families:</p><ul><li><strong>Several solutions per fit.</strong> The method is internally searching for more than one answer and returns them together. GEMSS (the Gaussian Ensemble for Multiple Sparse Solutions, the subject of <a href="https://medium.com/@kate.henclova/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">the previous post</a>) works this way, as does ALFESE (in simultaneous mode), which wraps a conventional selector in a constraint demanding further, sufficiently different solutions.</li><li><strong>One solution per run.</strong> Perturb the data, re-solve the single-answer problem, repeat: the randomized-LASSO ensemble. Getting a second, genuinely different solution is not a matter of running it longer. It takes enough differently-perturbed reruns that one lands elsewhere in solution space, plus some clustering to notice that it did. And “elsewhere” can be a very small target to hit by chance.</li></ul><p><strong>The two cannot be <em>fairly</em> budgeted the same way.</strong> Capping the reruns at three because you asked for three solutions does not subject the second family to the same test; it guarantees it never gets a chance.</p><p>So, internal effort must no longer be a shared currency. Instead, it becomes a private, per-method hyperparameter, tuned to whatever is optimal and feasible for each. <strong>Fairness is enforced only at the finish line</strong>: every method returns the same number of solutions, at the same sparsity (the same number of selected features in each solution).</p><h3>What the benchmark cannot tell you</h3><p>Three limitations worth stating plainly.</p><ul><li><strong>It is synthetic, by necessity.</strong> Knowing the complete set of true solutions is the entire point, and no real dataset offers that. Evidence that real data do contain multiple usable solutions has to come from somewhere else.</li><li><strong>One generator family, and a linear one.</strong> Whether the same conclusions hold for a nonlinear mechanism is untested.</li><li><strong>Every solution is the same size.</strong> Fixing the support size is what makes automated comparison possible across thousands of runs. As a result, the benchmark measures methods in a regime tidier than the one you would deploy them in practice.</li></ul><p>These are the reasons why this series will not stop at synthetic data.</p><h3>Where this leaves us</h3><p>We now have a benchmark that:</p><ul><li><strong>Builds ambiguous ground truth on purpose</strong>, and tunes how hard that ambiguity is to resolve.</li><li><strong>Scores recovery by comparing unions</strong> of features, rather than pairing solutions one-to-one.</li><li><strong>Lets every method tune itself freely</strong>, holding only the output format fixed.</li></ul><p>It is introduced formally in <a href="https://arxiv.org/abs/2602.08913">the GEMSS paper</a>, and the code is <a href="https://github.com/kat-er-ina/gemss_testing">openly available</a>.</p><h3>Conclusion</h3><p>This approach does not advise you which alternative explanation to use, deliberately. Go back to the inflammation markers example. A benchmark can establish that a method reliably surfaces the alternatives, i.e., that the disease-specific subset of markers is indeed offered alongside the practically useless candidate solutions. However, <strong>recognising which subset answers your actual question is not a statistical judgement,</strong> and no metric should pretend otherwise; <strong>that call stays with the person who understands the domain.</strong></p><p>The benchmark’s job is narrower and still worth doing: separating the algorithms that genuinely recover alternative solutions from those that merely appear to, because diversity was imposed from outside rather than found in the data.</p><h4>Follow the full series</h4><p>This post is the first in the series on benchmarking algorithms for alternative feature selection. It provides reasoning on how the synthetic benchmark is built.</p><p>See further:</p><ul><li>Part 2 (TBA) introduces the current state of the art and, in particular, the six methods that we shall compare.</li><li>Part 3 (TBA) tests the six algorithms on the benchmark.</li><li>Part 4 (TBA) showcases predictive performance on real data.</li></ul><p>For introductory articles, see the big-picture in <a href="https://medium.com/datamole/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb">The assisted discovery of GEMS(S)</a> and the math in <a href="https://medium.com/@kate.henclova/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">The problem of predictive multiplicity and how GEMSS deals with it</a>.</p><h3>Resources</h3><ul><li><strong>GEMSS + benchmarking paper:</strong> <a href="https://arxiv.org/abs/2602.08913">GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems</a> (Henclová &amp; Šmídl, 2026)</li><li><strong>Benchmarking framework:</strong> <a href="https://github.com/kat-er-ina/gemss_testing">github.com/kat-er-ina/gemss_testing</a></li><li><strong>Python package:</strong> <a href="https://github.com/kat-er-ina/gemss">github.com/kat-er-ina/gemss</a></li><li><strong>No-code app:</strong> <a href="https://huggingface.co/spaces/kat-er-ina/gemss">huggingface.co/spaces/kat-er-ina/gemss</a></li><li><strong>ALFESE paper:</strong> Bach, J., &amp; Böhm, K. (2024). <a href="https://arxiv.org/abs/2307.11607">Alternative Feature Selection with User Control</a>. International Journal of Data Science and Analytics.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=972722b8e6ee" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/benchmarking-alternative-feature-selection-algorithms-part-1-design-972722b8e6ee">Benchmarking Alternative Feature Selection Algorithms (Part 1: Design)</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is Predictive Multiplicity in Your Data And How GEMSS Deals With It]]></title>
            <link>https://blog.datamole.ai/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/28b7eca61a09</guid>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 12:18:26 GMT</pubDate>
            <atom:updated>2026-10-11T18:00:04.859Z</atom:updated>
            <content:encoded><![CDATA[<p>When your data has more than one story to tell, an ordinary feature selector will pick one and never mention the rest. GEMSS hands you the menu of several valid sparse solutions in a single fit.</p><p><strong><em>TL;DR:</em></strong><em> When features are many or correlated and samples are scarce, several small feature subsets explain the data equally well, producing multiple solutions. GEMSS is a variational method that finds the multiple sparse solutions in a single fit. It uses Bayesian inference and approximates the multimodal posterior with a Gaussian mixture.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*C0y28w3YL1aLNnr-tVactw.jpeg" /><figcaption>The raw data can be cut in multiple ways to produce the desired outcome.</figcaption></figure><h3>When your data has more than one story to tell</h3><p>You are diagnosing failures on a machine with a few hundred sensor channels. Vibration and temperature sensors both track the same underlying degradation, so either group predicts the failure well. A conventional sparse feature selector picks one, drops the other, and reports a clean answer.</p><p><strong>That answer is not wrong. It is just one of several, chosen arbitrarily by whatever tie-breaking happens inside the optimizer.</strong> And if the vibration channels are the expensive, drift-prone ones and the temperature channels are cheap and reliable, you would very much like to know that the second option exists. A single-solution method never tells you.</p><p>This is a very common scenario whenever data comes from physical measurements:</p><ul><li><strong><em>n ≪ p</em>.</strong> There can be many more features than (effective) samples in the dataset, sometimes by orders of magnitude. When there are more unknowns than equations, uniqueness is off the table. Most notably, biological datasets with few specimen and modern measurement methods like mass spectrometry, are of this type.</li><li><strong>Collinearity.</strong> Features can be interchangeable for domain reasons (they report the same biological or physical mechanism) or for purely statistical ones (with <em>p</em> that large, some features are near-linear combinations of others by accident). Either way, the ambiguity survives even when <em>n</em> &gt; <em>p</em>.</li></ul><p>Formally, what fails is the <a href="https://andrewcharlesjones.github.io/journal/irrepresentable.html">irrepresentable condition</a>, and with it the support-recovery guarantee of classical sparse selectors. They still return a support; it just isn’t <em>the</em> support, because there isn’t just one.</p><p><strong>The practical response is to stop asking for the answer and start asking for the shortlist.</strong> Hand a domain expert several statistically competitive feature subsets and let them apply what the algorithm cannot know — their domain knowledge and human intuition. I’ve written about that framing, <strong>assisted discovery</strong>, in <a href="https://medium.com/datamole/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb">an earlier post</a>.</p><p>This post is about the machinery of the method I built for such problems: <strong>GEMSS, the Gaussian Ensemble for Multiple Sparse Solutions</strong>. Testing it is a separate story, told across further posts: designing a proper benchmark, how GEMSS and its competitors fare against it, and what comes out on real data.</p><blockquote>If you want to try it before reading the math, there’s a <a href="http://huggingface.co/spaces/kat-er-ina/gemss">free online app</a>. Upload data (you can borrow <a href="https://github.com/kat-er-ina/gemss/tree/main/data">our preprocessed datasets</a>), request candidate solutions, and validate them with cross-validated end models.</blockquote><h3>The fundamentals of Bayesian inference</h3><p>Skip ahead if you already think in posteriors. Otherwise: Bayesian inference is a rule for updating a belief once data arrives. Three pieces:</p><ul><li><strong>The prior</strong>, π(θ) — what you believe about the unknown θ <em>before</em> looking at anything. The prejudicial starting point.</li><li><strong>The likelihood</strong>, p(y | θ) — how well each candidate θ accounts for the evidence, meaning the data you actually observed.</li><li><strong>The posterior</strong>, p(θ | y) — that starting point adjusted once the evidence has been taken into account.</li></ul><p>Bayes’ theorem is how the first two combine into the third:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*VD8EaUKdaqrm07IzE650rQ.png" /><figcaption>The Bayes theorem</figcaption></figure><p>The denominator p(y) is a normalizing constant: the probability of the observed data under the model, averaged over every θ. Confusingly, <em>this quantity</em> is also called the <strong>evidence</strong> — and the <em>evidence lower bound</em> that turns up later in this article is a lower bound on its logarithm.</p><p>Two features matter for what follows. The result is a <strong>whole distribution rather than a single estimate</strong>, so it can have several peaks — and each peak is a candidate answer. And <strong>the prior is a modelling choice you make deliberately</strong>, so with the right one you can build that multiplicity in on purpose. Which is exactly what GEMSS does.</p><h3>The mental model: a mountain range, not a peak</h3><p>Put a sparsity-inducing prior π(β) on the coefficients of a generalized linear model and look at the posterior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*u3Yj_KLSZu8ljBMHxyBXPw.png" /><figcaption>The posterior distribution</figcaption></figure><p><strong>Read this as a landscape over coefficient vectors, where height is plausibility given the data.</strong> A single-solution method assumes one summit and climbs it. But when several sparse supports explain <em>y</em> equally well, the landscape has <strong>several summits of near-equal height</strong> — and the task becomes mapping them, not reporting the coordinates of whichever one you happened to climb.</p><p>Everything below is that one picture. Two things follow from it, and they structure the article:</p><ol><li>Something has to <em>make</em> the posterior multimodal → the prior’s job.</li><li>Something has to <em>find several modes at once</em> → the variational mixture’s job, which only works because the objective actively dislikes duplicates.</li></ol><h3>Step 1: the prior that manufactures the modes</h3><p>If you have 1000 features you already believe most of them are not important for predicting your target. The standard way to encode that is a <a href="https://en.wikipedia.org/wiki/Spike-and-slab_regression">spike-and-slab prior</a>: each coefficient is drawn either from a <strong>spike</strong>, a narrow Gaussian at zero that crushes irrelevant features, or from a <strong>slab</strong>, a wide Gaussian that lets a selected feature take whatever value it needs.</p><p>GEMSS defaults to the <em>structured</em> variant, which fixes the number of nonzeros at exactly <em>D</em> by mixing uniformly over all supports A of size <em>D</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*MynWbRsunnoh8yi7RtV2-g.png" /><figcaption>The structured Spike-and-Slab prior</figcaption></figure><p>The formula looks heavier than the idea. It says: pick a size-<em>D</em> subset of features, let those <em>D</em> be free, pin everything else at zero, and average over every way of picking.</p><blockquote>When the number of combinations is astronomically large the supports are subsampled — this is a prior, not an enumeration.</blockquote><p>Two consequences worth pausing on.</p><p><strong>Sparsity is a knob, not a cap.</strong> <em>D</em> is set directly rather than tuned for via shrinkage — but it sets the regime the prior searches in, not a hard limit on the output. If the evidence supports a denser solution, the fit accommodates it. So <strong>do not overestimate <em>D</em></strong>.</p><p><strong>The prior is itself a mixture, so the posterior is too.</strong> Each size-<em>D</em> support contributes its own term, and after multiplying by the likelihood every support that fits well keeps a term of comparable mass. The mountain range is <em>built</em>, not stumbled upon.</p><h3>Step 2: <em>K</em> search parties instead of one hiker</h3><p>Now the search. <strong>The usual approach drops one blindfolded hiker on the map, lets them climb, perturbs the map and drops them again</strong>. Repeat a few thousand times, cluster the landing sites. That is precisely what constitutes the naive approach: the randomized-LASSO ensemble.</p><p><strong>GEMSS instead models the whole range at once</strong>, with a parametric mixture of <em>K</em> diagonal Gaussians:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*wH8xasixrCwBNXXplDC8_Q.png" /><figcaption>Approximation by a mixture of Gaussian distributions</figcaption></figure><p>Each component is a search party with its own location <em>μ_k</em>, per-feature uncertainty <em>σ²_k</em>, and a weight <em>α_k</em> for how much posterior mass it claims. All <em>K</em> are optimized together by maximizing the <a href="https://en.wikipedia.org/wiki/Evidence_lower_bound">Evidence Lower BOund</a> (ELBO) with stochastic gradient descent, using implicit reparameterization to automatically differentiate through the mixture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*hkJ73v6zR392BO6GDEj-DQ.png" /><figcaption>The Evidence Lower Bound (ELBO)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KlktE4lXzMSHpWqCDjFgQ.png" /><figcaption><em>The posterior (top) and the fitted variational mixture (bottom). Each component settles under one peak, so a single fit maps several solutions at once.</em></figcaption></figure><p>So far this is textbook variational inference, and the natural worry is that all <em>K</em> components simply pile onto the same summit. They don’t — and the reason is already sitting in the objective.</p><h3>Why the components push each other apart?</h3><p>Repulsion between modes is already present in the objective: the mixture entropy <em>-E_q[log q]</em> term inside the KL divergence contains a logarithm of a sum over all components. So, <strong>two components placed on the same mode increase <em>log q</em> at the samples, and are penalized for it.</strong> Maximizing the ELBO maximizes the entropy term, and that pushes the component means apart like magnets of the same polarity. (This is an explicit <strong>parameter-space repulsion</strong>, the same principle behind repulsive mixtures and <a href="https://arxiv.org/abs/1608.04471">SVGD</a>.) <strong>But it is a mechanism, not a theorem.</strong></p><p>Hence, the components do not need an additional regularizer to behave: it is what maximizing the ELBO of a mixture already implies. <strong>The diversity is not manufactured.</strong> Also, we have <strong>a single objective function</strong>: Prior, likelihood and diversity are all covered by one objective, and are optimized in the same run, once.</p><h3>The λ knob: forcing solutions apart on purpose</h3><p>On top of that built-in repulsion, GEMSS gives you an explicit diversity control: a penalty on the pairwise <a href="https://en.wikipedia.org/wiki/Jaccard_index">Jaccard similarity</a> between solutions, with scale <em>λ</em>. It is <strong>off by default (<em>λ</em> = 0)</strong>, and the built-in repulsion is often enough.</p><p><strong>Turn it on when you want more separation than the model alone would give you.</strong> <strong>How different your candidates should be is partly a design and domain-specific choice</strong>, not purely a property of the data — more of a choice than one would like, in practice. Just note which mechanism produced the diversity you end up looking at: the entropy repulsion separates components where the posterior has separate mass, <em>λ</em> separates them whether the data agrees or not.</p><blockquote>Tuning <em>λ </em>is not fiddly. Sensible values run from 0 to roughly 4000, and the method is not especially sensitive within that band, so stepping by ~500 is plenty. Our most common choices were ~2000.</blockquote><h3>Choosing K: ask for more peaks than you need</h3><p><strong><em>K</em> is the number of peaks you are asking the model to find </strong>but it should not equal the number of candidate solutions you actually want, because the landscape is messier than the answer: a component can settle on a shoulder rather than a summit, or two can split one broad mode between them.</p><p><strong>Rule of thumb: choose K to be ~3× the number of solutions you want.</strong> Too few components and well-separated modes go uncovered; far too many and one broad structure ends up fragmented across several. Nothing clever happens afterwards — you just look at what came back and discard the candidates you don’t want, for any reason.</p><h3>Reading the features off a component</h3><p>Each component gives you one candidate solution, but you still have to set the cutoff threshold to select the top few features. Two options:</p><ul><li><strong>Top-D by |μ_k|</strong> — the <em>D</em> coefficients with the largest posterior means, so every solution has exactly <em>D</em> features. This is what the benchmark uses because automated comparison needs a fixed output size (number of features in a solution).</li><li><strong>Z-score outlier detection</strong> — the coefficients that stand out from the component’s own bulk (at 2.5 standard deviations, say). This approach allows for varying support size, i.e. your alternative explanations will not have to contain the same number of features. <strong>Use this approach in practice.</strong></li></ul><h3>Missing data: marginalize, don’t guess</h3><p>Being a probabilistic model buys one more thing for free. Rather than imputing or dropping rows, GEMSS evaluates the per-sample likelihood <strong>over the observed coordinates only</strong> — missing entries are marginalized out, so a sample with three features missing contributes exactly the information it has and nothing invented. Nothing to configure, nothing to impute.</p><h3>Dealing with nonlinearities</h3><p>GEMSS is a generalized linear model, so nonlinear mechanisms sit outside the model class. The naive workaround — engineer transformations and interaction terms into the feature space — normally backfires by multiplying the dimension and manufacturing collinearity, exactly what breaks conventional selectors. However, <strong>GEMSS can handle high dimensions and collinearities, so the naive feature engineering approach is practically usable.</strong></p><h3>What to take away</h3><p><strong>Underdetermined problems have multimodal posteriors, and that is structure rather than noise.</strong> The prior manufactures those modes and the mixture entropy separates them. Not through a regularizer bolted on to keep the components apart, though, but through single-objective ELBO maximization.</p><p>Four choices are yours to make:</p><ul><li><strong><em>D</em>, the sparsity</strong> — the estimated/desired number of features per solution. It is not a hard limit, data can override it. Better to set it lower than too high.</li><li><strong><em>K</em>, the number of peaks</strong> — ask for ~3× the number of solutions you want, then discard the candidates you don’t like.</li><li><strong><em>λ</em>, the diversity penalty</strong> — turn it on when you want your solutions to overlap less. We observed most suitable values to be ~2000.</li><li><strong>The extraction rule</strong> —set up a suitable z-score threshold.</li></ul><h3>Benchmarking GEMSS</h3><p>Whether GEMSS actually beats its competitors is a harder question than it looks. That’s where my next three posts go: creating a fair benchmark, the benchmarking itself, and testing on real-world data.</p><h3>Resources</h3><ul><li><strong>Paper:</strong> <a href="https://arxiv.org/abs/2602.08913">GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems</a> (Henclová &amp; Šmídl, 2026) — accepted for the Discovery Science 2026 conference.)</li><li><strong>Python package repo:</strong> <a href="https://github.com/kat-er-ina/gemss">github.com/kat-er-ina/gemss</a> (pip install gemss)</li><li><strong>Benchmarking framework:</strong> <a href="https://github.com/kat-er-ina/gemss_testing">github.com/kat-er-ina/gemss_testing</a></li><li><strong>No-code online app:</strong> <a href="https://huggingface.co/spaces/kat-er-ina/gemss">huggingface.co/spaces/kat-er-ina/gemss</a></li></ul><h3>Acknowledgements</h3><p>This research has been done with prof. Václav Šmídl at Faculty of Electrical Engineering, Czech Technical University. Many thanks also go to the data science team at Datamole for the research time and support behind this open-source release, especially to Marek Nevole for the software engineering input and Markéta Jůzlová for our discussions.</p><h3>Further reading</h3><p>Andersen, M. R., Winther, O., &amp; Hansen, L. K. (2014). Bayesian Inference for Structured Spike and Slab Priors. NeurIPS 27.</p><p>Figurnov, M., Mohamed, S., &amp; Mnih, A. (2018). Implicit Reparameterization Gradients. NeurIPS 31.</p><p>Hastie, T., Tibshirani, R., &amp; Wainwright, M. (2015). Statistical Learning with Sparsity: The Lasso and Generalizations. CRC Press.</p><p>Henclová, K., &amp; Šmídl, V. (2026). GEMSS: A Variational Method for Discovering Multiple Sparse Solutions in Classification and Regression Problems. arXiv:2602.08913.</p><p>Liu, Q., &amp; Wang, D. (2016). Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm. NeurIPS 29.</p><p>Petralia, F., Rao, V., &amp; Dunson, D. B. (2012). Repulsive Mixtures. NeurIPS 25.</p><p>Xie, F., &amp; Xu, Y. (2020). Bayesian Repulsive Gaussian Mixture Model. Journal of the American Statistical Association, 115(529).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28b7eca61a09" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/the-problem-of-predictive-multiplicity-and-how-gemss-deals-with-it-28b7eca61a09">What Is Predictive Multiplicity in Your Data And How GEMSS Deals With It</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multivariate SSA: Meet the PCA for Time Series]]></title>
            <link>https://blog.datamole.ai/multivariate-ssa-meet-the-pca-for-time-series-f921960950d7?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/f921960950d7</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 19:04:56 GMT</pubDate>
            <atom:updated>2026-07-14T19:04:56.599Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>TL;DR </em></strong><em>Multivariate Singular Spectrum Analysis (mSSA) is a rigorous and highly effective method for time series preprocessing. You can think of it as Principal Component Analysis (PCA) adapted for time series data. In a single algorithm, mSSA handles missing values, removes noise, and engineers robust features without introducing any lag. It successfully captures both temporal and spatial relationships across multiple signals, which makes it especially suitable for multivariate IoT data streams. Moreover, this principled, mathematical approach consistently outperforms complex deep learning models in imputation and forecasting.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBNT2_hft6bj7q6qc0LLVQ.png" /></figure><h3>The Importance of Preprocessing</h3><p>Every data scientist knows the first and most important rule: “garbage in — garbage out”. But even messy data can sometimes be salvaged with the right kind of preprocessing.</p><p>In my previous post <a href="https://medium.com/@kate.henclova/no-labels-no-problem-unlocking-iot-time-series-with-weakly-supervised-learning-9543c47c7be0">No Labels? No Problem: Unlocking IoT Time Series with Weakly Supervised Learning</a>, I mention that changing the usual preprocessing and feature engineering was the crucial prerequisite for making the weakly supervised models lift above random guessing and become competitive with their fully supervised counterpart. This post deep dives into the useful method I had employed.</p><p>When you prepare IoT time series data for predictive modeling (e.g., classification with algorithms like XGBoost or MINIROCKET), you typically face a sequence of tasks. You need to handle missing values, smooth the signals, and engineer features.</p><p>Traditionally, practitioners handle the tasks with imputation methods, rolling mean smoothing, classical time series decomposition and/or Fourier analysis.</p><p>As an alternative, one can incorporate the Multivariate Singular Spectrum Analysis (mSSA) into the standard pipeline. This effectively replaces all of those disparate steps.</p><h3>The Basics: Singular Spectrum Analysis</h3><p>To understand mSSA, we first need to look at the basic Singular Spectrum Analysis (SSA) algorithm. SSA acts as PCA for a single time series, relying on the singular decomposition of a matrix constructed from the time series.</p><p>At its core, SSA comprises four steps:</p><h4>Step 1: Embedding</h4><p>The algorithm constructs a <a href="https://en.wikipedia.org/wiki/Hankel_matrix">Hankel matrix</a> (below denoted as <em>X</em>). This matrix comprises lagged vectors of the time series (number of time steps is denoted as <em>N</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mi_57tW_PsRgNB8vNTJfvg.png" /></figure><h4>Step 2: Decomposition</h4><p>Perform the <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">Singular Value Decomposition</a> (SVD) of the Hankel matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*diJYTVEXB3qIvj1cqFx7fA.png" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*lHxxbNlkCLiVIxd7V9vycw.png" /></figure><p>by standard notation.</p><h4>Step 3: Grouping</h4><p>Group the eigentriples (eigenvalues and the corresponding left and right eigenvectors) based on the eigenvalue size. Then “hankelize” the groups, i.e. average the corresponding skew-diagonals</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*X0Al1dV76PZLoo37MdZAZA.png" /></figure><p>creating a new Hankel matrix.</p><p>This process effectively breaks down the time series into parts analogous to the traditional trend, seasonality, and noise components.</p><h4>Step 4: Reconstruction</h4><p>Finally, reconstruct the time series by mapping back from the Hankel matrices.</p><h4>The Summary of SSA</h4><p>In the above steps, we transformed the time series into a matrix in order to use SVD. This method extracts the essential components so that we are able to analytically separate the signal from noise. Once noise is discarded, we reverse the process to obtain the cleaned time series.</p><h3>The Upgrade: Multivariate SSA</h3><p><strong><em>The essential reading on this topic and the source of the following text: </em></strong><a href="https://arxiv.org/abs/2006.13448"><em>On Multivariate Singular Spectrum Analysis and its Variants</em></a><em> by Agarwal, Alomar, and Shah (MIT, 2022).</em></p><p>When you work with data from multiple sensors, you need an algorithm that synthesizes information across all of them. mSSA extends the basic SSA algorithm to capture relationships across multiple dimensions simultaneously. Below are the changes <a href="https://github.com/AbdullahO/mSSA">implemented</a> in the 2022 version of mSSA.</p><p><strong>Embedding and Stacking. </strong>Create Page matrices for each individual sensor. If there is missing data, the algorithm initializes those gaps with zeros. It then concatenates these matrices column-wise to form a massive stacked matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p1d9JRgeFqPj2fe4CMsw5Q.jpeg" /><figcaption>The stacked Page matrix</figcaption></figure><p><strong>Decomposition.</strong> Similar to the basic algorithm, it calculates the SVD of the stacked matrix.</p><p><strong>Denoising. </strong>The algorithm applies “hard singular value thresholding”. By removing small singular values, it mathematically strips out the noise.</p><p><strong>Imputation. </strong>It reconstructs the matrix, which results in imputation by reconstruction.</p><p><strong>Forecasting. </strong>Finally, you can perform out-of-sample forecasting via ordinary least squares regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Q-EhqOwELSUDvMFUh9wo2Q.png" /><figcaption><em>The schema of the mSSA algorithm. Image borrowed directly from the paper </em><a href="https://arxiv.org/abs/2006.13448"><em>On Multivariate Singular Spectrum Analysis and its Variants</em></a><em> by Agarwal, Alomar, and Shah (MIT, 2022).</em></figcaption></figure><h3>The Benchmark: Beating the Deep Learning Giants</h3><p>If you are skeptical about using a matrix transformation over a modern neural network, the performance metrics are highly compelling. Besides the rigorous theoretical foundations, the researchers benchmarked mSSA across diverse, real-world datasets, including electricity, traffic, synthetic, and financial data.</p><p>When evaluated on imputation and mean forecasting tasks, mSSA statistically outperforms or is comparable with other state-of-the-art time series algorithms. This includes heavy-hitting deep learning architectures like Long Short-Term Memory (LSTM) networks and DeepAR, as well Temporal Regularized Matrix Factorization (TRMF) and Vector Autoregression (VAR).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U4nvnMqmgkyGBg9hp2UKzg.png" /><figcaption><em>Credit for this comparison goes entirely to Agarwal, Alomar, and Shah <br>(paper </em><a href="https://arxiv.org/abs/2006.13448"><em>On Multivariate Singular Spectrum Analysis and its Variants</em></a><em>).</em></figcaption></figure><h3>Conclusion</h3><p>mSSA is a robust mathematical technique that handles the heavy lifting during the preprocessing stage of time series. It is well suited for multiple tasks including missing value imputation, smoothing, trend extraction, feature engineering, and forecasting. Due to its ability to capture not only temporal but also spatial relationships, it is perfect when processing multivariate IoT/sensor data streams.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f921960950d7" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/multivariate-ssa-meet-the-pca-for-time-series-f921960950d7">Multivariate SSA: Meet the PCA for Time Series</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[No Labels? No Problem: Unlocking IoT Time Series with Weakly Supervised Learning]]></title>
            <link>https://blog.datamole.ai/no-labels-no-problem-unlocking-iot-time-series-with-weakly-supervised-learning-9543c47c7be0?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/9543c47c7be0</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[weakly-supervised]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[iot]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 12:35:38 GMT</pubDate>
            <atom:updated>2026-07-08T12:37:11.342Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mq95Bb-bsLtOdpWkUd5Pww.png" /></figure><p>If you work in machine learning, you know the data bottleneck all too well: we can get mountains of raw data, but getting the high-quality, ground-truth labels needed to train supervised models is tedious and expensive.</p><p>In natural language processing and computer vision, practitioners have widely adopted programmatic Weakly Supervised Learning (WSL) to bypass this issue. Instead of hand-labeling thousands of data points, domain experts write heuristic rules that automatically assign noisy, probabilistic labels.</p><p>However, when it comes to multivariate time series and (physiological) IoT sensor data, WSL remains surprisingly under-explored. Recently, we in Datamole decided to explore it ourselves, applying a programmatic weak labeling pipeline to a binary classification problem using multivariate time series data in the domain of animal health.</p><p>Here is an admittedly subjective, but highly practical, look at our internal experiment, alongside the current research and key lessons we learned. (Disclaimer: the full results are not shared due to confidentiality.)</p><h3>The Programmatic Weak Supervision Pipeline</h3><p>For those new to the concept, the standard programmatic WSL pipeline operates in three rigorous steps:</p><ul><li><strong>Labeling Functions (LFs):</strong> Domain experts encode heuristics, pattern matchers, or external knowledge base rules to output either a noisy label or abstain from voting.</li><li><strong>Label modeling:</strong> Because LFs have unknown accuracies, varying coverage, and underlying conflicts, a generative label model aggregates their outputs. By analyzing agreements and disagreements without access to ground truth, it estimates unobserved accuracies and computes a single probabilistic training label.</li><li><strong>End model training:</strong> These probabilistic labels are used to train a robust downstream model, such as a neural network or Gradient Boosting Machine. This allows the final high-capacity model to learn robust feature representations and generalize beyond the initial heuristic rules.</li></ul><p><strong>The difference between labeling functions and features </strong>is easy to overlook at first. While they have much in common, labeling functions can utilize information that is not available at the time of inference, e.g. the future.</p><h3>Our Sanity-Check Experiment: A Subjective Deep Dive</h3><p>We evaluated WSL on a binary classification task using physiological IoT multivariate time series data. For practical reasons, we introduced several deliberate constraints and shortcuts. As such, the experiment was not fully rigorous, yet it served as a neat real-world proving ground.</p><h3>Experiment Methodology</h3><p>Our setup:</p><ul><li><strong>Data: </strong>~500,000 weakly labeled datapoints compared to 11,000 gold labels. For the golden labels, there were 140 anomalous events, each comprising multiple gold labels. The data was limited to one biological cycle per subject.</li><li><strong>Labeling functions: </strong>To build our heuristics, we brought an LLM (Claude Opus 4.6) into the loop as a “domain expert proxy”, which successfully helped us brainstorm and refine 34 weak labeling functions and their default categorical weights. A minor, deliberate data leak occurred at this point.</li><li><strong>Tested WSL models</strong>: Snorkel, Flying Squid, Naive Bayes, Enhanced Bayesian Classifier Combination, Dawid-Skene, Bayesian Balsubramani-Freund, and manual weighting.</li><li><strong>End models: </strong>XGBoost and a basic 3-layer Multi-Layer Perceptron (MLP), both with a deliberately robust setup and no fine tuning.</li></ul><h3>Key Learnings &amp; Best Practices</h3><p>Below we summarize our most interesting observations.</p><h4>1. Quality Preprocessing is Non-Negotiable</h4><p>You cannot weak-label your way out of bad data. We found that weak labeling is virtually useless when only uninformative features and/or labeling functions are available, even though the same features were used to train the fully supervised counterpart.</p><p>Initially, features and labeling functions based on our usual preprocessing failed to produce any useful results. It was only after incorporating advanced preprocessing that our results lifted above random chance.</p><p>In other words, if your LFs are operating directly on time series data, finding the right preprocessing and parameterization is essential.</p><h4>2. The Power of Manual Weights and the BBF Label Model</h4><p>We tested several label models to combine our weak LFs. Interestingly, our manually weighted model outperformed all but one automated model <em>by a great margin</em>. This was likely caused by the automated methods’ conservative approach to LF’s weights, whereas the manual setup allowed us to assign an extremely high weight to one select labeling function. Industry-standard models like Snorkel struggled to assign large weights to single functions and seemingly could not capture the underlying correlations among our rules.</p><p>Among the automated models, the standout performer was the <strong>Bayesian Balsubramani-Freund model </strong>(BBF). Introduced by Steven An (2026, NeurIPS), BBF is an adversarial model that implicitly constructs a geometric polytope of plausible labelings and outputs the center of that polytope. Because it is a Bayesian approach, BBF gave us a massive structural advantage: it allowed us to inject our manual LF weights directly into the model as priors. By doing this, the model started with our domain knowledge but could still add value.</p><p>Most notably, using the weak labeling approach (and different data preprocessing), we were able to <strong>outperform the current model trained only with the gold labels</strong>.</p><p>On the other side of the spectrum was, by a large margin, the Dawid-Skene method, which performed worse than random.</p><h4>3. Weak-to-Strong Generalization is Real</h4><p>A striking observation was the impact of the final step: training the end model.</p><p>When evaluating the raw, composed BBF labels alone, their performance was subpar, sometimes even dropping below random chance. However, when we trained an XGBoost classifier on those exact same probabilistic labels, they showed supreme generalization properties: the overall performance skyrocketed and became highly competitive.</p><p>In our tests, the default XGBoost consistently matched or outperformed the simple MLP at this denoising step. For simplicity sake, we did not train a more complex end model.</p><p>Our observations perfectly mirror recent research by Lang et al. (2024, NeurIPS) on weak-to-strong generalization. They note two phenomena that make WSL successful:</p><ul><li><strong>Pseudolabel correction:</strong> The strong student model actually learns to correct the weak teacher’s errors.</li><li><strong>Coverage expansion:</strong> The model performs well even on the portion of the example space that was not covered by the initial heuristic rules.</li></ul><h3>Validating with the Broader Literature</h3><p>Our subjective dive into programmatic labeling confirmed what current literature suggests: WSL is a highly viable, affordable alternative to fully supervised learning for IoT time series data.</p><p>For example, Goswami et al. (2022) explored WSL for ECG time series data. They found that with as few as six intuitive time series heuristics, they could infer high-quality probabilistic labels for over 100,000 heartbeats. Crucially, their weakly supervised ResNet model performed on par with a fully supervised model.</p><h3>The Verdict</h3><p>If you are dealing with extreme label scarcity, a massive pool of unlabeled data, and enough domain knowledge to write a few sensible heuristics, programmatic labeling is a legitimate option even for sensor time series data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9543c47c7be0" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/no-labels-no-problem-unlocking-iot-time-series-with-weakly-supervised-learning-9543c47c7be0">No Labels? No Problem: Unlocking IoT Time Series with Weakly Supervised Learning</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The assisted discovery of GEMS(S)]]></title>
            <link>https://blog.datamole.ai/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb?source=rss-f74e89338760------2</link>
            <guid isPermaLink="false">https://medium.com/p/2353907474cb</guid>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <dc:creator><![CDATA[Katerina Henclova]]></dc:creator>
            <pubDate>Tue, 12 May 2026 10:29:15 GMT</pubDate>
            <atom:updated>2026-08-04T12:24:20.454Z</atom:updated>
            <content:encoded><![CDATA[<p>Your feature selector picks one explanation and never mentions the alternatives. When the goal is understanding a mechanism, that choice belongs to a domain expert.</p><p><strong><em>TL;DR </em></strong><em>In underdetermined predictive problems, multiple statistically comparable and sparse solutions may exist. Standard feature selection returns exactly one of them, chosen arbitrarily, and gives no hint of others — fine if you only want predictions, a real loss if you are trying to identify a mechanism, because the solution you were handed may be the least useful of the valid ones. Assisted discovery reverses the handoff: the algorithm surfaces a menu of statistically plausible candidate solutions, and a domain expert picks among them using what no algorithm knows . GEMSS is the method I built to produce that menu.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_-J6wP2p6obUyUiyWUNVcg.png" /><figcaption>Multiple solutions to the same problem. Only you know which one is best suitable in practice.</figcaption></figure><p>For a long time, we have been taught a very specific way to do machine learning: feed the data into an algorithm, optimize for a metric, and hunt down the one best model. But what happens when the data contains multiple possible solutions?</p><p>In industrial, agricultural, and biomedical data science, we constantly run into the “<em>n &lt;&lt; p</em>” problems with precious few samples and possibly many more features. The reality of this isn’t just a mathematical quirk; it comes down to the practicalities of life and economics. In these fields, acquiring samples is often very expensive, both money and time-wise, meaning we as data scientists just have to make do with what we have, sometimes with only a few dozen observations. Meanwhile, modern measurement methods easily generate hundreds or thousands of features for every sample. When searching for explanatory mechanisms and building interpretable models, as is common in our R&amp;D applications, we have a real challenge: to obtain a handful of quality modeling features, effectively looking for a needle in a combinatorial haystack.</p><p>Most conventional feature selection algorithms are built to pick a single subset of features and ignore the rest. They collapse all possible explanations of the data into one. Subsequently, it is used for model training and testing — you know the drill. (Alternatively, they may provide estimates of feature importances — but that does not help when we need specific combinations.)</p><p>Now, the single solution these methods find usually isn’t <em>wrong</em>. Mathematically, it is perfectly legitimate. But it might not be the one we actually want.</p><p>Consider a medical diagnostic test: standard ML might easily identify general inflammation markers to perfectly predict that a patient is sick. That is a highly accurate, statistically valid solution. However, it is completely useless if your actual goal is to find biomarkers specific to a rare, underlying condition. By arbitrarily isolating just one legitimate solution, standard methods risk obscuring alternative explanatory mechanisms that might actually be the specific drivers we are looking for. Or those that are more practical or cost-effective.</p><p>(The very same problem arises in scenarios with highly correlated features that yield multiple solutions even if “<em>n &gt; p</em>”.)</p><h3>Predictive multiplicity: when the data has more than one story to tell</h3><p>This brings us to a harsh data-centric reality. Real-world physical and biological measurements are messy. In complex environments, the “truth” is rarely hidden in just one place. Instead, we encounter what is known as predictive multiplicity, or the <a href="https://arxiv.org/abs/2407.04846"><em>Rashomon effect</em></a>. Simply put, multiple distinct subsets of features can explain the data equally well.</p><p>When the primary objective of our modeling shifts from pure prediction to generating actionable scientific or operational insights, traditional metrics are often insufficient to adjudicate between these equally valid solutions. We don’t just need a model that predicts accurately, we need to understand the actual physical or biological mechanisms behind the data.</p><h3>Assisted discovery: putting the expert back in the loop</h3><p>This is where we want to change how we think about the role of machine learning in R&amp;D: let us shift the paradigm to <em>assisted discovery</em>.</p><p>Algorithms lack real-world context and human intuition. They don’t know the cost of a certain chemical test or the biological irrelevance of combinations composed solely of general inflammation markers. But domain experts do. So, how can we best mine this information?</p><p>Instead of an algorithm dictating a single answer, what if it provided a diverse “menu” of statistically plausible hypotheses? Presenting multiple candidate hypotheses allows a domain expert to factor in their domain knowledge, practical constraints, and intangible human experience to pick the mechanism that is actually useful in practice.</p><h3>GEMSS offers a menu of valid explanations to data</h3><p>This specific frustration is exactly why I developed GEMSS: <strong>G</strong>aussian <strong>E</strong>nsemble for <strong>M</strong>ultiple <strong>S</strong>parse <strong>S</strong>olutions.</p><p>At its heart, GEMSS is a variational <a href="https://en.wikipedia.org/wiki/Bayesian_inference">Bayesian algorithm</a>. If we break down the name, the “ensemble” refers primarily to the group of statistically plausible candidate explanations for your data. Each one is represented by a component in a <a href="https://en.wikipedia.org/wiki/Mixture_model">Gaussian mixture</a> that is used to approximate a complex, multimodal posterior distribution (which is, incidentally, another Gaussian mixture, since it has been produced by a structured <a href="https://en.wikipedia.org/wiki/Spike-and-slab_regression">Spike&amp;Slab prior distribution</a>, i.e. yet another Gaussian mixture).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KlktE4lXzMSHpWqCDjFgQ.png" /><figcaption>Finding 4 likely solutions: approximating the multimodal posterior by a Gaussian mixture with 4 components.</figcaption></figure><p>GEMSS is designed to simultaneously discover multiple, diverse and sparse (i.e. small) feature combinations. Crucially, it is built for the messy realities of our problem. Extensive testing has shown its <strong>robustness to class imbalance and gaussian noise</strong>. Moreover, it <strong>natively handles missing data</strong>: it simply uses only the observed values. And because it utilizes stochastic gradient descent to optimize all of these modes simultaneously, it scales practically and <strong>efficiently even when your feature space is massive</strong>.</p><h3>Proving it in the wild</h3><p>To ensure this wasn’t just a theoretical exercise, we put GEMSS to the test on real-world datasets that suffer from the exact problems we face in the trenches.</p><p>Broadly, we evaluated two types of data:</p><ol><li><strong>Public metabolomics data:</strong> We tested three datasets from the rigorous MetaboLights archive. This included <a href="https://www.ebi.ac.uk/metabolights/editor/MTBLS1">data of diabetic patients</a> as a classic <em>n &lt; p </em>scenario (n=132, p = 222) , an <a href="https://www.ebi.ac.uk/metabolights/editor/MTBLS2"><em>Arabidopsis</em> plant genomics dataset</a> representing extreme sample scarcity (16 samples), as well as a <a href="https://www.ebi.ac.uk/metabolights/editor/MTBLS12968">dataset of mothers</a> with(out) PCOS and (pre)term birth (generally lesser predictive performance — not included in the table below).</li><li><strong>Proprietary physical chemistry data:</strong> A commercial dataset characterized by collinear variables of various sources, measurement errors, and highly imbalanced labels.</li></ol><p>Each problem threw a different challenge at the algorithm, requiring us to extract anywhere from 1 to 33 features per candidate solution depending on the specific context and dataset.</p><p>Here is the crucial takeaway: <strong>in all of these cases, GEMSS successfully identified multiple distinct candidate solutions</strong>.</p><p>Naturally, the absolute predictive metrics differed across the datasets depending on their inherent data quality. But the key finding was that within any given problem, the discovered alternative solutions achieved comparable predictive performance.</p><p>It proves that predictive multiplicity is a real problem — and we now have a tool to navigate it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*XEl1u-Bqq4AdDA-LvECfZQ.png" /><figcaption>Summary of the best 8 candidate solutions for 3 real-world datasets (P = number of features selected in a candidate solution).</figcaption></figure><h3>Democratizing assisted discovery with GEMSS Explorer</h3><p>Because our goal is to put the “expert in the loop”, I built the <a href="https://huggingface.co/spaces/kat-er-ina/gemss"><strong>GEMSS Explorer</strong></a>. It is a free online application, no ads, fully open-sourced, hosted at Hugging Face. The app guides data scientists as well as tech-savvy domain experts through the entire workflow: from data upload, to obtaining multiple sparse candidate solutions, to rigorously validating them with cross-validated models.</p><p>The app is ready for you to use today, no strings attached. You don’t need to write a single line of code to use it. You just need your data (we do not see them) and your domain intuition.</p><p>I invite you to check out also the public <a href="https://github.com/kat-er-ina/gemss">GitHub repository</a> (where a downloadable version of the app is available too, should you be worried about safety) or read the <a href="https://arxiv.org/abs/2602.08913">preprint</a> paper (peer-reviewed, accepted for publication). I would love to hear your feedback, and I am happy to provide assistance if you are a researcher and want to try it out on your own tough datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*39GGxJ-pDffCEfcN1D9J9A.png" /><figcaption>Try out the free online <a href="https://huggingface.co/spaces/kat-er-ina/gemss">GEMSS Explorer app</a>, hosted at Hugging Face.</figcaption></figure><p><em>This open-source release would never have happened without the dedicated research time and expert support from the data science team at Datamole. Reach out to us for R&amp;D tailored to your commercial needs.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2353907474cb" width="1" height="1" alt=""><hr><p><a href="https://blog.datamole.ai/the-assisted-discovery-of-gemss-multiple-solutions-2353907474cb">The assisted discovery of GEMS(S)</a> was originally published in <a href="https://blog.datamole.ai/">DATAMOLE</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>