<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hiroki Takizawa on Medium]]></title>
        <description><![CDATA[Stories by Hiroki Takizawa on Medium]]></description>
        <link>https://medium.com/@hiroki-takizawa?source=rss-2e0ccdf447f1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DNdXClHU6Zmh4Sw9HDdCbw.png</url>
            <title>Stories by Hiroki Takizawa on Medium</title>
            <link>https://medium.com/@hiroki-takizawa?source=rss-2e0ccdf447f1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 03:02:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hiroki-takizawa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing A New Terminator: Early Termination of Black-box Optimization Based on Expected…]]></title>
            <link>https://medium.com/optuna/introducing-a-new-terminator-early-termination-of-black-box-optimization-based-on-expected-9a660774fcdb?source=rss-2e0ccdf447f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a660774fcdb</guid>
            <category><![CDATA[optuna]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[gaussian-process]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hiroki Takizawa]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 04:08:26 GMT</pubDate>
            <atom:updated>2024-10-15T04:08:26.936Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introducing A New Terminator: Early Termination of Black-box Optimization Based on Expected Minimum Model Regret</h3><h3><strong>Introduction</strong></h3><p>Optuna includes a Terminator module, which terminates the optimization. In the v4.0 update, new versatile algorithms have been added to the Terminator module. This new Terminator is a powerful option that can be used in cases where cross-validation (CV) is not performed. In this article, we’ll provide an overview of the Terminator and explore examples of how to use the new Terminator.</p><h3>What is Optuna Terminator ?</h3><p>The Terminator module implemented in Optuna predicts the potential for improvement based on the history of optimization trials. Based on these predictions, it offers functionalities such as automatic early termination of the optimization process and visualization of the predicted “room for improvement” at each trial stage. It is named “Terminator” because it enables the early termination of the optimization itself. According to a <a href="https://proceedings.mlr.press/v188/makarova22a.html">paper</a>, the advantages of early termination include not only saving computational costs but also the potential to prevent overfitting.</p><p>The Terminator was first introduced to Optuna in version 3.2, based on a <a href="https://proceedings.mlr.press/v188/makarova22a.html">paper published in 2022</a>. In Optuna 4.0, a new implementation was added based on another <a href="https://proceedings.mlr.press/v206/ishibashi23a.html">paper published in 2023</a>.</p><h3>Trying The New Terminator</h3><p><a href="https://proceedings.mlr.press/v188/makarova22a.html">The earlier paper</a> and the Terminator based on it (e.g., <a href="https://optuna.readthedocs.io/en/stable/reference/generated/optuna.terminator.CrossValidationErrorEvaluator.html#optuna.terminator.CrossValidationErrorEvaluator">CrossValidationErrorEvaluator</a>) focused on the statistical properties of score variance across each fold of cross-validation (CV) and used this analysis to determine when to terminate the optimization. This was a significant breakthrough; however, a downside was that its primary application was limited to problem settings that involve CV.</p><p>On the other hand, the new Terminator introduced in Optuna 4.0 (e.g., <a href="https://optuna.readthedocs.io/en/stable/reference/generated/optuna.terminator.EMMREvaluator.html#optuna.terminator.EMMREvaluator">EMMREvaluator</a>) employs a method that estimates the potential for improvement based on the series of final evaluation values obtained from each trial. This approach allows the Terminator to be applied to a wide range of problem settings, independent of CV.</p><p>In the derivation of the new Terminator, the Expected Minimum Simple Regret (EMSR) is defined, representing the potential for improvement. To work around the computational difficulty of EMSR, the paper derives a computable upper bound. This upper bound is implemented in Optuna and is named the Expected Minimum Model Regret (EMMR). During this implementation, we communicated with <a href="https://orcid.org/0000-0001-7751-7102">Prof. Hideaki Ishibashi</a> from Kyushu Institute of Technology, one of the paper’s authors, and received valuable advice and guidance, including the suggestion to name it “Expected Minimum Model Regret.”</p><p>As an example of a program incorporating early termination with the new Terminator, you can write it as follows.</p><pre>import optuna<br>from optuna.terminator import EMMREvaluator<br>from optuna.terminator import MedianErrorEvaluator<br>from optuna.terminator import Terminator<br>from optuna.terminator import TerminatorCallback<br><br>emmr_improvement_evaluator = EMMREvaluator()<br><br># By default, 1/100 of the median value of the 10th to 30th step of<br># emmr_improvement_evaluator<br>median_error_evaluator = MedianErrorEvaluator(emmr_improvement_evaluator)<br><br># If the value of emmr_improvement_evaluator falls below the value of<br># median_error_evaluator, the early termination occurs.<br>terminator = Terminator(<br>    improvement_evaluator=emmr_improvement_evaluator,<br>    error_evaluator=median_error_evaluator,<br>)<br><br>def objective(trial):<br>    xs = [trial.suggest_float(f&quot;x{i}&quot;, -10.0, 10.0) for i in range(5)]<br>    return sum(x**2 for x in xs)<br><br>sampler = optuna.samplers.TPESampler(seed=0)<br>study = optuna.create_study(sampler=sampler, direction=&quot;minimize&quot;)<br>study.optimize(objective, n_trials=1000, callbacks=[TerminatorCallback(terminator)])<br><br># visualize<br>fig = optuna.visualization.plot_terminator_improvement(<br>    study,<br>    plot_error=True,<br>    improvement_evaluator=emmr_improvement_evaluator,<br>    error_evaluator=median_error_evaluator,<br>)<br>fig.write_image(&quot;example-figure.png&quot;)</pre><p>The visualization of the above code is as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*ADROojXEZRRE96Sv19aIeQ.png" /><figcaption>Figure: The blue line is the output of Terminator’s improvement_evaluator, showing the amount of room for improvement. The red line is the output of `error_evaluator`, which is 1/100 of the median blue value of the first 10 to 30th trials. The optimization stops immediately after blue drops below red. (The actual values during optimization are slightly different because Improvement is recalculated for the visualization and the pseudo-random numbers take different values during the recalculation.)</figcaption></figure><h3>Conclusions</h3><p>Try the new Optuna Terminator, a powerful tool that helps you in saving computational costs during optimization and visualizing the room for improvement within a certain theoretical framework! Since these are also emerging research topics in academia, your feedback will provide valuable insights not only for us but for the research community. Optuna is committed to rapidly integrating new research developments and empowering our users.</p><h3>Acknowledgement</h3><p>In adding the new Terminator, we received valuable advice and the <a href="https://github.com/hideaki-ishibashi/stopping_BO">authors’ implementation</a> from <a href="https://orcid.org/0000-0001-7751-7102">Prof. Hideaki Ishibashi</a> of Kyushu Institute of Technology. We express our gratitude.</p><h3>References</h3><p>Makarova, A., Shen, H., Perrone, V., Klein, A., Faddoul, J.B., Krause, A., Seeger, M. &amp; Archambeau, C.. (2022). Automatic Termination for Hyperparameter Optimization. <em>Proceedings of the First International Conference on Automated Machine Learning</em>, in <em>Proceedings of Machine Learning Research</em> 188:7/1–21 Available from <a href="https://proceedings.mlr.press/v188/makarova22a.html">https://proceedings.mlr.press/v188/makarova22a.html</a>.</p><p>Ishibashi, H., Karasuyama, M., Takeuchi, I. &amp; Hino, H.. (2023). A stopping criterion for Bayesian optimization by the gap of expected minimum simple regrets. <em>Proceedings of The 26th International Conference on Artificial Intelligence and Statistics</em>, in <em>Proceedings of Machine Learning Research</em> 206:6463–6497 Available from <a href="https://proceedings.mlr.press/v206/ishibashi23a.html">https://proceedings.mlr.press/v206/ishibashi23a.html</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a660774fcdb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/optuna/introducing-a-new-terminator-early-termination-of-black-box-optimization-based-on-expected-9a660774fcdb">Introducing A New Terminator: Early Termination of Black-box Optimization Based on Expected…</a> was originally published in <a href="https://medium.com/optuna">Optuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the Stabilized JournalStorage in Optuna 4.0: From Mechanism to Use Case]]></title>
            <link>https://medium.com/optuna/introducing-the-stabilized-journalstorage-in-optuna-4-0-from-mechanism-to-use-case-e320795ffb61?source=rss-2e0ccdf447f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/e320795ffb61</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Hiroki Takizawa]]></dc:creator>
            <pubDate>Wed, 18 Sep 2024 07:03:02 GMT</pubDate>
            <atom:updated>2024-09-18T07:03:02.292Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>The default storage class in Optuna is InMemoryStorage. However, InMemoryStorage does not persist trial histories and cannot be used for distributed optimization. Therefore, Optuna provides multiple storage classes.</p><p>In Optuna 4.0, <a href="https://optuna.readthedocs.io/en/v4.0.0/reference/generated/optuna.storages.JournalStorage.html#optuna.storages.JournalStorage">JournalStorage</a> and <a href="https://optuna.readthedocs.io/en/v4.0.0/reference/generated/optuna.storages.journal.JournalFileBackend.html#optuna.storages.journal.JournalFileBackend">JournalFileBackend</a>, which are among the storage classes, have been officially supported. In this blog, we will introduce these technical points, as well as their use cases and how to utilize them.</p><h3><strong>About JournalStorage</strong></h3><p>JournalStorage is one of the storage classes in Optuna. The name originates from the fact that it records the operational logs of Optuna in a stacked, journal-like manner. The primary motivation for its introduction was to make it easier to implement various backends (such as databases) as storage for Optuna. To achieve this, the design separates the responsibility, where the JournalStorage class acts as Optuna’s storage, while another class is responsible for reading and writing to the backend. The JournalStorage class is designed to accept objects of classes prepared for each backend during initialization (see Fig. 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*ZF1QJ9MGSLj4c196hxVTOg.png" /><figcaption>Figure 1: A diagram showing the relationships between JournalStorage and related classes with arrows. The JournalStorage class, shown in gray in the top, takes the backend class objects, shown in blue, as arguments. There is a backend class that can also take another class related to a locking mechanism, shown in green, as an optional argument.</figcaption></figure><p>JournalStorage was experimentally introduced in Optuna v3.1, and the official support began in v4.0. With the official support, backward compatibility of log files will be guaranteed. Additionally, class names and module paths have been reorganized, and stability has been improved.</p><p>A simple code example using JournalStorage can be written as follows:</p><pre>import optuna<br>from optuna.storages import JournalStorage<br>from optuna.storages.journal import JournalFileBackend<br><br><br>def objective(trial):<br>    xs = [trial.suggest_float(f&quot;x{n}&quot;, -1.0, 1.0) for n in range(3)]<br>    return sum(x ** 2 for x in xs)<br><br><br>storage = JournalStorage(JournalFileBackend(&quot;./optuna_journal_storage.log&quot;))<br>study = optuna.create_study(storage=storage)<br>study.optimize(objective, n_trials=20)</pre><p>As mentioned earlier, the JournalStorage class takes a class object that implements the functionality to access the backend (in this code, the JournalFileBackend class) as an argument during initialization. This is one of the key features of JournalStorage, making it an easy method to add various backends to Optuna. As of Optuna 4.0, in addition to JournalFileBackend, the JournalRedisBackend class has been implemented to use <a href="https://redis.io/">Redis</a>, a famous NoSQL database, as a backend. (However, the focus of stabilization in this release is limited to JournalFileBackend.)</p><p>For further details on JournalStorage, please also refer to the <a href="https://medium.com/optuna/distributed-optimization-via-nfs-using-optunas-new-operation-based-logging-storage-9815f9c3f932">past blog post</a>.</p><h3><strong>About JournalFileBackend</strong></h3><p>The JournalFileBackend class provides storage functionality compatible with distributed optimization. It can be used by passing an object to the initialization of the JournalStorage class, as shown in the sample code above. The greatest advantage of JournalFileBackend is that it enables distributed optimization via Network File System (NFS). This is achieved by implementing mutual exclusion using system calls defined as atomic by the NFS specification. Two methods for acquiring locks are each implemented as separate classes, and you can switch between them by passing an optional argument during the initialization of the JournalFileBackend class. For usage instructions, please refer to the <a href="https://optuna.readthedocs.io/en/v4.0.0/reference/storages.html">documentation</a>, and for further details on the mechanism, please refer to the <a href="https://medium.com/optuna/distributed-optimization-via-nfs-using-optunas-new-operation-based-logging-storage-9815f9c3f932">past blog post</a>, which explains this alongside details of JournalStorage.</p><p>In addition to JournalStorage, another method to store Optuna’s studies is RDBStorage. RDBStorage supports SQLite3 as well as MySQL. Like JournalFileBackend, SQLite3 can also use a single file as storage, making it a convenient option. However, it is known that when the SQLite3 file is located on an NFS, simultaneous access from multiple nodes or processes does not function well. This issue is mentioned in the <a href="https://www.sqlite.org/faq.html#q5">official SQLite FAQ</a>. Given these considerations, JournalFileBackend is the only method available for distributed optimization in Optuna 4.0 when using a single file on NFS as storage.</p><h3><strong>A Use-Case</strong></h3><p>While executing large-scale distributed optimization using RDBStorage and MySQL, the load on the MySQL server became a bottleneck during the analysis of the optimization results. To streamline the analysis process and reduce the load, we considered migrating the studies stored in MySQL to another storage using the optuna.copy_study function. While we also looked into converting to storage options like Redis and SQLite3, we found that converting to JournalStorage with JournalFileBackend was the fastest and most suitable for our workload. This allowed us to avoid placing additional load on the MySQL server and perform the analysis more quickly.</p><h3><strong>Changes for Stabilization in Optuna 4.0</strong></h3><p>In Optuna 4.0, class names and module paths were reorganized to make the API more intuitive. The code in the example above conforms to the specifications of v4.0. (Excerpted and re-posted below:)</p><pre>from optuna.storages import JournalStorage<br>from optuna.storages.journal import JournalFileBackend<br>storage = JournalStorage(JournalFileBackend(&quot;./optuna_journal_storage.log&quot;))</pre><p>Code written using the v3.x syntax will continue to function with backward compatibility for the time being, but it is deprecated. For details about the new module paths, please refer to the <a href="https://optuna.readthedocs.io/en/v4.0.0/reference/storages.html">documentation</a> and <a href="https://github.com/optuna/optuna/discussions/5573">migration guide</a>.</p><p>All log files created since the release of v3.1 will remain usable in v4.0 and beyond! Please use the file from the JournalFileBackend class in the same way as shown in the code example above.</p><h3>Conclusions</h3><p>In this article, we introduced the usage and a practical use-case of the storage features stabilized in Optuna 4.0. JournalStorage with JournalFileBackend is a powerful method that supports distributed optimization while only relying on NFS. We encourage you to give it a try!</p><p>Optuna 4.0 has made significant progress in many areas, in addition to stabilizing JournalStorage. For more details, check out the <a href="https://medium.com/optuna/optuna-4-0-whats-new-in-the-major-release-3325a8420d10">Optuna 4.0 release blog</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e320795ffb61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/optuna/introducing-the-stabilized-journalstorage-in-optuna-4-0-from-mechanism-to-use-case-e320795ffb61">Introducing the Stabilized JournalStorage in Optuna 4.0: From Mechanism to Use Case</a> was originally published in <a href="https://medium.com/optuna">Optuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Natural Gradient-Based Optimization Algorithm Registered on OptunaHub]]></title>
            <link>https://medium.com/optuna/a-natural-gradient-based-optimization-algorithm-registered-on-optunahub-0dbe17cb0f7d?source=rss-2e0ccdf447f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/0dbe17cb0f7d</guid>
            <category><![CDATA[optimization-algorithms]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hiroki Takizawa]]></dc:creator>
            <pubDate>Wed, 28 Aug 2024 08:46:51 GMT</pubDate>
            <atom:updated>2024-08-28T08:46:51.322Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>We have been offering a beta version of <a href="https://hub.optuna.org/">OptunaHub</a>, a new feature-sharing platform designed for Optuna, since July. In this article, we introduce a newly implemented optimization algorithm called Implicit Natural Gradient Optimization (INGO) [1], which performs optimization based on natural gradient. INGO is an algorithm closely related to CMA-ES (Covariance Matrix Adaptation Evolution Strategy), a powerful evolutionary algorithm. We demonstrate that the registered INGO outperforms CMA-ES in our experiment.</p><h3><strong>INGO Algorithm Registered on OptunaHub</strong></h3><p>In this section, we will run the INGO-based sampler registered on OptunaHub. <a href="https://github.com/nel215">Yuhei Otomo</a> helped us with this implementation. Please check <a href="https://hub.optuna.org/samplers/implicit_natural_gradient/">here</a> for more details.</p><p>The implementation of this sampler was verified by basic test cases and by checking whether the benchmarking results are consistent with the claims made in the original paper [1]. Figure 1 shows that the registered INGO outperforms CMA-ES on 100-dimensional L₁-Norm Ellipsoid as claimed in the paper . The horizontal axis shows the number of trials, and the vertical axis shows the objective function value, which is better when it is lower. The solid lines (red: INGO, blue: CMA-ES) show the minimum objective function value found up to each trial, and the weak-color bands are collections of dots that represent the objective function values found at each trial. The appearance of bands is due to the high density of dots. Similar to Figure 1 (b) in the paper, you can see that INGO steadily improves, whereas CMA-ES does not improve as effectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*YMCLgUtNvtt8Rs62ZCHAjw.png" /><figcaption>Figure 1. Comparison of INGO (red) and CMA-ES (blue) using L₁-Norm Ellipsoid.</figcaption></figure><p>The code used for benchmarking is as follows:</p><pre>from argparse import ArgumentParser<br><br>import numpy as np<br>import optuna<br>import optunahub<br><br>parser = ArgumentParser()<br>parser.add_argument(&quot;--use-ingo&quot;, action=&quot;store_true&quot;)<br>args = parser.parse_args()<br><br><br>def objective(trial):<br>    dimension = 100<br>    X = np.abs([trial.suggest_float(f&quot;x{d}&quot;, -10.0, 10.0) for d in range(dimension)])<br>    coefficients = [10 ** (6 * d / (dimension - 1)) for d in range(dimension)]<br>    return float(coefficients @ X)<br><br><br>if args.use_ingo:<br>    mod = optunahub.load_module(&quot;samplers/implicit_natural_gradient&quot;)<br>    sampler = mod.ImplicitNaturalGradientSampler(sigma0=0.5, seed=0)<br>else:<br>    sampler = optuna.samplers.CmaEsSampler(sigma0=0.5, seed=0)<br><br><br>study = optuna.create_study(sampler=sampler)<br>study.optimize(objective, n_trials=20000)</pre><p>The code above, let’s say we name it as demo.py, can be run with the command below.</p><p>In addition to the INGO sampler, many other packages are already registered on OptunaHub, and a number of them can be used simply by installing optunahub via pip! (Of course, some packages require additional dependencies, but their requirements are usually described in each description)</p><pre># Installation of Dependencies.<br>$ pip install optuna optunahub cmaes<br><br># Run INGO.<br>$ python demo.py --use-ingo<br><br># Run CMA-ES.<br>$ python demo.py</pre><h3><strong>Brief Description of the INGO Algorithm</strong></h3><p>Implicit Natural Gradient Optimization (INGO) is a black-box optimization method that does not require the explicit gradient of the objective function. Lyun and Tsang [1] made it possible to apply the natural gradient method well-known in information geometry to black-box optimization. Note that INGO approximates natural gradient based on the Fisher information matrix estimated from observations, so it also does not require an explicit gradient with respect to the search space.</p><p>Covariance Matrix Adaptation Evolution Strategy (CMA-ES), widely used in practice and available in Optuna, is a related method to INGO. Both methods determine the search region by updating the parameters (mean vector and covariance matrix) of a multivariate normal distribution defined over the search space. While INGO updates the parameters via the natural gradient, the update of the parameters in CMA-ES under certain conditions is performed using the Monte-Carlo approximation of the natural gradient [2]. Since INGO estimates the natural gradient in a mathematically less biased manner, INGO is expected to be more efficient. In fact, the original paper reports that some benchmark results in continuous spaces are competitive to those of CMA-ES [1].</p><h3><strong>Conclusion</strong></h3><p>In this article, we introduced INGO, a powerful natural gradient-based optimization algorithm available on OptunaHub. Please give it a try!</p><p><a href="https://hub.optuna.org/samplers/implicit_natural_gradient/">Implicit Natural Gradient Sampler (INGO) | OptunaHub</a></p><p>On OptunaHub, we welcome many contributions from the community, not only pull requests for new packages, but also code refactoring, bug reports, and<a href="https://github.com/optuna/optuna/discussions/5606"> suggestions for desired packages</a>. If you’re interested, please check out the <a href="https://github.com/optuna/optunahub-registry">OptunaHub GitHub repository</a> and help us make OptunaHub even better together with your contributions!</p><h3><strong>References</strong></h3><p>[1] Yueming Lyu and Ivor Tsang (2020). Black-box Optimizer with Implicit Natural Gradient. arXiv Preprint arXiv:1910.04301.</p><p>[2] Youhei Akimoto, Yuichi Nagata, Isao Ono, and Shigenobu Kobayashi (2010). Bidirectional Relation between CMA Evolution Strategies and Natural Evolution Strategies. PPSN, pp. 154–163 (2010).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0dbe17cb0f7d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/optuna/a-natural-gradient-based-optimization-algorithm-registered-on-optunahub-0dbe17cb0f7d">A Natural Gradient-Based Optimization Algorithm Registered on OptunaHub</a> was originally published in <a href="https://medium.com/optuna">Optuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Timeline Plot: a New Feature of Optuna v3.2]]></title>
            <link>https://medium.com/optuna/introducing-timeline-plot-a-new-feature-of-optuna-v3-2-2628a0ee9502?source=rss-2e0ccdf447f1------2</link>
            <guid isPermaLink="false">https://medium.com/p/2628a0ee9502</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[optuna]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hiroki Takizawa]]></dc:creator>
            <pubDate>Fri, 23 Jun 2023 08:30:28 GMT</pubDate>
            <atom:updated>2024-08-28T14:21:33.687Z</atom:updated>
            <content:encoded><![CDATA[<p>The newest version of Optuna, Optuna v3.2, an open source software tool for black box optimization, was released at the end of May. This article introduces one of its new features, Timeline Plot.</p><p>Timeline Plot is a type of plot that displays each trial as a horizontal bar, with the time on the x-axis and the trial number on the y-axis, as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*LmbSKjgLnx4GjDL981CJ3g.png" /></figure><h3>hands-on</h3><p>Here is an example for a hands-on exercise: let us use the sample classification dataset provided by scikit-learn to perform machine learning with LightGBM. The Python source code is as follows:</p><pre>import lightgbm as lgb<br>import numpy as np<br>import optuna<br>import sklearn.datasets<br>import sklearn.metrics<br>from sklearn.model_selection import train_test_split<br><br>data, target = sklearn.datasets.load_breast_cancer(return_X_y=True)<br>train_x, val_x, train_y, val_y = train_test_split(<br>    data, target, test_size=0.25<br>)<br><br><br>def objective(trial):<br>    dtrain = lgb.Dataset(train_x, label=train_y)<br>    param = {<br>        &quot;objective&quot;: &quot;binary&quot;,<br>        &quot;metric&quot;: &quot;binary_logloss&quot;,<br>        &quot;boosting_type&quot;: &quot;gbdt&quot;,<br>        &quot;lambda_l1&quot;: trial.suggest_float(&quot;lambda_l1&quot;, 1e-8, 10.0, log=True),<br>        &quot;lambda_l2&quot;: trial.suggest_float(&quot;lambda_l2&quot;, 1e-8, 10.0, log=True),<br>        &quot;num_leaves&quot;: trial.suggest_int(&quot;num_leaves&quot;, 2, 256),<br>        &quot;feature_fraction&quot;: trial.suggest_float(&quot;feature_fraction&quot;, 0.4, 1.0),<br>        &quot;bagging_fraction&quot;: trial.suggest_float(&quot;bagging_fraction&quot;, 0.4, 1.0),<br>        &quot;bagging_freq&quot;: trial.suggest_int(&quot;bagging_freq&quot;, 1, 7),<br>        &quot;min_child_samples&quot;: trial.suggest_int(&quot;min_child_samples&quot;, 5, 100),<br>    }<br>    gbm = lgb.train(param, dtrain)<br>    preds = gbm.predict(val_x)<br>    pred_labels = np.rint(preds)<br>    return sklearn.metrics.accuracy_score(val_y, pred_labels)<br><br><br>study = optuna.create_study(direction=&quot;maximize&quot;)<br>study.optimize(objective, n_jobs=4, n_trials=50)<br>fig = optuna.visualization.plot_timeline(study)<br>fig.show()</pre><p>Executing the above source code will output the following figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*zwyCaVkClQ5D_1wFW750oQ.png" /></figure><p>From the Timeline Plot, it can be observed that the computations for trials after the 15th trial took relatively longer time. Optuna recognizes that values of certain parameters of LightGBM that take longer to calculate tend to result in higher accuracy, so such trials are likely to increase in number from the middle of the search process. Specifically, increasing the value of min_child_samples parameter can lengthen the computation time.</p><ul><li>As an aside, in the aforementioned code, the dataset was partitioned into two subsets (for training and validation) and then optimized across 50 trials. Were this to be increased to, for instance, 1000 trials, the validation accuracy would likely improve; however, the generalization performance for test data may deteriorate. This potential outcome indicates that the hyperparameters could overfit the validation dataset. To mitigate this issue, a new feature dubbed the ‘<a href="https://github.com/optuna/optuna-examples/blob/main/terminator/terminator_simple.py">Optuna Terminator</a>’, which was introduced in Optuna v3.2. Optuna Terminator can be utilized to halt the optimization early, hence curtailing overfitting and simultaneously reducing computation time.</li></ul><h3>Situations where the Timeline Plot is particularly effective</h3><h4>Monitoring trials in distributed environments</h4><p>Timeline Plot is particularly effective in visualizing issues that often arise in distributed environments. When running distributed optimization in a shared cluster, Optuna workers are often preempted, causing the computation to be forcibly terminated midway. If Optuna is not notified about the worker being preempted, the status of the trial that was being worked on at the time of preemption will remain ‘Running’ indefinitely. Of course, there could also be cases where it genuinely remains ‘Running’ indefinitely, such as if there is a mistake in the implementation of the evaluation function causing an infinite loop. Such mistakes are easy to overlook in a distributed environment, but they can be clearly visualized using a Timeline Plot.</p><p>The Timeline Plot function can accept an ongoing study as an argument. This feature enables quick detection of any anomalous trials, facilitating the initiation of bug fixes or introducing solutions to these problems, such as implementing a ‘heartbeat’ to detect preemption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*4FbeFdTS62MK8zXSJKHwQg.png" /><figcaption>An example of visualizing anomalous Trials. The trial indicated by the yellow arrow persists in the ‘RUNNING’ state. In this scenario, even though initial parallel computations launched two independent trials, the throughput was reduced by half for all subsequent trials, following the one that became stalled.</figcaption></figure><h4>Checking trial details interactively</h4><p>Timeline Plot can visualize delays or failures on a per-trial basis. Furthermore, in the case of the plotly version, you can hover the mouse cursor over the horizontal bars representing each trial to view the parameters and evaluation function values of that trial as hover text. This feature would be of great help in quickly eliminating difficult bugs, such as ones causing infinite loops only when a parameter has a certain value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*7bP_iX4wZSqgDjkvHKUaFA.png" /><figcaption>This is a screenshot of the Timeline Plot. The states of trials are displayed according to the sequence of trial numbers. The red bubble in the center represents hover text, which reveals the parameter and evaluation function value (shown as null due to an error) for a single trial when you hover the mouse cursor over it.</figcaption></figure><h4>Timeline Plot in Optuna Dashboard</h4><p>Timeline Plot is available in <a href="https://github.com/optuna/optuna-dashboard">Optuna Dashboard</a> v0.10.0, as shown in the screenshot below. As an example of the functionality that can be leveraged through this combination, if a trial that remains in the ‘Running’ status exists, it can be easily and promptly identified and removed using the dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I93haESJGIbQjSMz7qZQuA.png" /></figure><h3>Conclusions</h3><p>In this article, I introduced the Timeline Plot added in <a href="https://medium.com/optuna/announcing-optuna-3-2-cfbfbe104d5f">Optuna v3.2</a>. The Timeline Plot can play a crucial role in troubleshooting in distributed environments and tuning machine learning hyperparameters that require trial and error. It is particularly useful for promptly identifying the presence of anomalous trials and parameters of noteworthy trials.</p><p>In addition to this, other useful features were added in <a href="https://medium.com/optuna/announcing-optuna-3-2-cfbfbe104d5f">Optuna v3.2</a>. The Optuna Terminator is a feature for early stopping optimization, helping to reduce computation time and prevent overfitting.</p><p>By using new features for Optuna, you can expect improvements in processing speed for optimization tasks in machine learning, as well as the discovery of better solutions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2628a0ee9502" width="1" height="1" alt=""><hr><p><a href="https://medium.com/optuna/introducing-timeline-plot-a-new-feature-of-optuna-v3-2-2628a0ee9502">Introducing Timeline Plot: a New Feature of Optuna v3.2</a> was originally published in <a href="https://medium.com/optuna">Optuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>