<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kento Nozawa on Medium]]></title>
        <description><![CDATA[Stories by Kento Nozawa on Medium]]></description>
        <link>https://medium.com/@nzw0301?source=rss-19af64176674------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Kento Nozawa on Medium</title>
            <link>https://medium.com/@nzw0301?source=rss-19af64176674------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 02:59:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nzw0301/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Optuna Meets Weights and Biases]]></title>
            <link>https://medium.com/optuna/optuna-meets-weights-and-biases-58fc6bab893?source=rss-19af64176674------2</link>
            <guid isPermaLink="false">https://medium.com/p/58fc6bab893</guid>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[optuna]]></category>
            <category><![CDATA[wandb]]></category>
            <dc:creator><![CDATA[Kento Nozawa]]></dc:creator>
            <pubDate>Mon, 07 Mar 2022 05:10:34 GMT</pubDate>
            <atom:updated>2024-08-29T01:59:53.759Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GNW1B5Pb8hV-E8miQUar4g.png" /></figure><p><a href="https://wandb.ai/">Weights and Biases</a> (WandB) is one of the most powerful machine learning platforms that offer several useful features to track machine learning experiments. I demonstrate how to combine Optuna-based experiments and WandB’s <a href="https://docs.wandb.ai/guides/track">Experiment Tracking</a> to enhance machine learning projects. This post mainly aims to explain how to apply WandB to Optuna’s code with minimal modification. The advantages of combining optuna and WandB are as follows:</p><ol><li>We enjoy the rich visualization functionality provided by WandB for Optuna’s optimization results.</li><li>We can easily share the optimization results with your colleagues by sharing WandB’s project pages or <a href="https://wandb.ai/site/reports">Reports</a>.</li><li>By using Optuna’s integration callback for WandB, we can seamlessly introduce WandB to Optuna-based experiments.</li></ol><p>The codes used in this post are available at <a href="https://github.com/nzw0301/optuna-wandb">https://github.com/nzw0301/optuna-wandb</a>.</p><h3>Preliminaries</h3><p>To use WandB’s web interface, please sign up WandB from <a href="https://wandb.ai/login?signup=true">https://wandb.ai/login?signup=true</a> and install wandbvia pip on your terminal as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FHanvHkca5y79GCgNM9d3%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FHanvHkca5y79GCgNM9d3&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/824a48c3bff8293dad7c0379b4690abe/href">https://medium.com/media/824a48c3bff8293dad7c0379b4690abe/href</a></iframe><p>Of Course, optuna as well:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FSKXGPeXRjzGFJTdbNbNA%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FSKXGPeXRjzGFJTdbNbNA&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e89f39f024f105ca47eb950ba49806c4/href">https://medium.com/media/e89f39f024f105ca47eb950ba49806c4/href</a></iframe><p>In this post, we use optuna==2.10.0and wandb==0.12.10. To synchronize experimental results to a server hosted by WandB, please login via command-line interface by running the following command.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FRy5eV0sAWUYiBueRaDtr%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FRy5eV0sAWUYiBueRaDtr&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d1679080c490711e07029cf5712c5de/href">https://medium.com/media/1d1679080c490711e07029cf5712c5de/href</a></iframe><p>Fortunately, WandB can run on your local machine as well! <a href="https://docs.wandb.ai/guides/self-hosted/local">https://docs.wandb.ai/guides/self-hosted/local</a> describes details to do so.</p><p>In this example, we use a few external libraries, if you would like to run the same scripts in this post, please install the following libraries as well:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FZTgyQSgllJ8k2VFOez2s%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FZTgyQSgllJ8k2VFOez2s&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c015f44ce05bf26f873f656c4ad741ad/href">https://medium.com/media/c015f44ce05bf26f873f656c4ad741ad/href</a></iframe><h3>Apply Weights and Biases to Optuna’s code</h3><p>Let’s start with an example code of Optuna. The following code maximizes validation accuracy on the FashionMNIST dataset by using shallow neural networks. If you are not familiar with Optuna, we recommend checking <a href="https://optuna.readthedocs.io/en/stable/tutorial/index.html#key-features">key feature tutorials</a>​.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2F5ZJMezq6DNCNxCkcZxnH%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F5ZJMezq6DNCNxCkcZxnH&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b7c5d5c05e7c65a66cd44007c4515e0/href">https://medium.com/media/8b7c5d5c05e7c65a66cd44007c4515e0/href</a></iframe><p>The full code is available at <a href="https://github.com/nzw0301/optuna-wandb/blob/main/part-1/naive_optuna.py">https://github.com/nzw0301/optuna-wandb/blob/main/part-1/naive_optuna.py</a>.</p><p>As in the code above, Optuna attempts to tune the following hyper-parameters of neural networks in objective:</p><ul><li>Optimizer</li><li>Optimizer’s learning rate</li><li>The number of layers of neural networks</li><li>The number of hidden units of neural networks</li><li>Dropout ratio</li></ul><p>WandB enables us to track the optimization by adding about 10 lines! Let’s apply WandB to the Optuna’s script step by step. The code is available at <a href="https://github.com/nzw0301/optuna-wandb/blob/main/part-1/wandb_optuna.py">https://github.com/nzw0301/optuna-wandb/blob/main/part-1/wandb_optuna.py</a>.</p><p><a href="https://docs.wandb.ai/ref/python/run">run</a> is defined as a unit of tracked computation by WandB. In neural network training, we’ll treat each training of a realized neural network as a single run. In the example code above, each training corresponds to an Optuna trialWe create run by calling <a href="https://docs.wandb.ai/ref/python/init">wandb.init</a> method in objective as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2Fd0VRuxiVPImrNjWq73mo%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2Fd0VRuxiVPImrNjWq73mo&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd244c1501fb577a88ecc854f71050f6/href">https://medium.com/media/dd244c1501fb577a88ecc854f71050f6/href</a></iframe><p>Let’s look at a few parameters in the wandb.init method. project takes a name to send run. If you send multiple runs for comparison, please specify the same name among runs. entity takes your username of WandB. In this example, nzw0301 is my WandB’s username. So please do not forget to replace it with your username! config stores tracked parameters as dict. In this case, I store the suggested hyperparameters by Optuna. In addition, I store trial.number in config to distinguish trials on the WandB web interface. reinit argument is necessary to call wandb.init method multiple times in the same process because objective is called multiple times in Optuna’s optimization.</p><p>Then we move to the part to record the optimization history in neural network training. In the PyTorch example code, we compute validation accuracy at the end of every epoch. To record the computed value, we call <a href="https://docs.wandb.ai/ref/python/log">wandb.log</a> method as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2F38iEMl09Fpf0kpepI1Z1%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F38iEMl09Fpf0kpepI1Z1&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2395557b8fc90a28a1925510feb62b54/href">https://medium.com/media/2395557b8fc90a28a1925510feb62b54/href</a></iframe><p>where data takes a dict to store tracked metrics. To specify the current step of reported validation accuracy, we use epoch as step of wandb.log. wandb.logenables us to store multiple metrics such as flops or average training accuracy to show detailed information about optimization on WandB’s Web UI. Additional recorded metrics are helpful to notice the trade-off between validation accuracy and computing time or not to widen hyperparameter space for underfitting.</p><p>The final part is for pruning and completing trials.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FWsDpKSuyqQhdar4ArC6W%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FWsDpKSuyqQhdar4ArC6W&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a213257bfd65bc8129bab5cf901ea3d9/href">https://medium.com/media/a213257bfd65bc8129bab5cf901ea3d9/href</a></iframe><p>run.summary stores an aggregated metric such as final validation accuracy or test accuracy. To distinguish the pruned trial and complete trial, we set pruned as state of run.summary . By calling <a href="https://docs.wandb.ai/ref/python/finish">wandb.finish</a>, we make sure run is completed in the current trial. Otherwise, the new trial’s tracked metrics are stored as records of the previous run.</p><p>Similarly, when the trial completes without pruning, we set completed as state of run.summary. In addition, we also add the final validation accuracy as final accuracy to run.summary.</p><p>By running the code, <a href="https://wandb.ai/[ENTITY]/optuna">https://wandb.ai/[ENTITY]/optuna</a> shows the reported results automatically! Recall that [ENTITY] is your WandB’s username specified in wandb.init. So in my case, <a href="https://wandb.ai/nzw0301/optuna">https://wandb.ai/nzw0301/optuna</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaFuc82VbCsOn9BfiBvp0A.png" /></figure><p>As you can see in the screenshot, WandB provides rich visualizations to understand the experimental results such as parameter importance and average validation accuracy by the optimizer. The visualizations are available at <a href="https://wandb.ai/nzw0301/optuna?workspace=user-nzw0301">https://wandb.ai/nzw0301/optuna?workspace=user-nzw0301</a>, where workplace is specified as user-nzw0301, because these visualizations depend on the login user. Note that we need to manually specify the type of figure and quantities such as hyperparameters or objective value from Add Panel button on WandB’s Web UI to add a new figure.</p><h3>Apply WandB with a callback</h3><p>If your optimization can be solved in a non-iterative manner unlike the example above, Optuna’s WandB callback, namely, <a href="https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.WeightsAndBiasesCallback.html">optuna.integration.WeightsAndBiasesCallback</a>, offers a much simpler interface to track Optuna’s optimization! Suppose we would like to compare sampler algorithms: RandomSampler and TPESampler. To do so, we solve the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_olivetti_faces.html#sklearn.datasets.fetch_olivetti_faces">Olivetti faces classification dataset</a> with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">RandomForest</a>. Concretely, we tune the following hyper-parameters of RandomForest:</p><ul><li>n_estimators</li><li>min_samples_leaf</li><li>max_depth</li><li>min_samples_split</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FWsDpKSuyqQhdar4ArC6W%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FWsDpKSuyqQhdar4ArC6W&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a213257bfd65bc8129bab5cf901ea3d9/href">https://medium.com/media/a213257bfd65bc8129bab5cf901ea3d9/href</a></iframe><p>The full script is available at <a href="https://github.com/nzw0301/optuna-wandb/blob/main/part-2/main.py.">https://github.com/nzw0301/optuna-wandb/blob/main/part-2/main.py.</a></p><p>The main code is like this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FILHlofipk6C7aryYoVXE%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FILHlofipk6C7aryYoVXE&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ceb55f7b1700e76dd266925feb4a896/href">https://medium.com/media/7ceb55f7b1700e76dd266925feb4a896/href</a></iframe><p>As we can see, we perform optimization num_runs times by sampler: RandomSampler or TPESampler. Each optimization evaluates objective n_trials=30 times. We track each optimization as a run of WandB. wandb_kwargs is a dict to be passed to wandb.init method internally. So we are already familiar with each key and value of wandb_kwargs.</p><p>We instantiate <a href="https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.WeightsAndBiasesCallback.html">WeightsAndBiasesCallback</a> with wandb_kargs. To clarify the name of the tracked metric by WandB, we set val_accuracy as metric_name. The callback is passed to <a href="https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html#optuna.study.Study.optimize">Study.optimize</a>’s callbacks.</p><p>Technically, when we instantiate WeightsAndBIasesCallback, it calls wandb.initto create a run. Then at the end of a trial, the callback calls wandb.log with the finished trial’s information.</p><p>To see the best suggested parameters and the best validation accuracy, we need to set a few parameters wandb.run.summary manually as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2F7Qzoquyr2NdTly9WTzfW%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2F7Qzoquyr2NdTly9WTzfW&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1a9dff00092a6aac94e1eea8bb39ada/href">https://medium.com/media/a1a9dff00092a6aac94e1eea8bb39ada/href</a></iframe><p>As a result, we can compare the following performance results by sampler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZcqZyCoyNbphlpbk4I7scA.png" /></figure><p>If you would like to save <a href="https://optuna.readthedocs.io/en/stable/reference/visualization/index.html">visualizations by Optuna</a> on WandB, wandb.log can save Plotly and Matplotlib’s figure as described at <a href="https://docs.wandb.ai/guides/track/log/plots#matplotlib-and-plotly-plots">https://docs.wandb.ai/guides/track/log/plots#matplotlib-and-plotly-plots</a>. In the following example, we save <a href="https://optuna.readthedocs.io/en/latest/reference/visualization/generated/optuna.visualization.plot_optimization_history.html">plot_optimization_history</a> and <a href="https://optuna.readthedocs.io/en/stable/reference/visualization/generated/optuna.visualization.plot_param_importances.html">plot_param_importances</a> after optimization. These plots are stored in each run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GdewaxGplB46xmXpDwVTgw.png" /></figure><p>Finally, if you instantiate the callback multiple times in the same process like the example code above, please call wandb.finish() at the end of each optimization. Otherwise, the new trial’s tracked metrics are stored as records of the previous run.</p><p>I hope you enjoy reading this post! For further information, please check <a href="https://docs.wandb.ai/">WandB</a> and <a href="https://optuna.readthedocs.io/en/stable/reference/index.html">Optuna</a> documentation!</p><p>Further materials:</p><ul><li><a href="https://medium.com/optuna/visualizing-hyperparameters-in-optuna-86c224bd255f">Visualizing Hyperparameters in Optuna</a>: A post explaining Optuna’s visualization APIs with examples.</li><li><a href="https://medium.com/optuna/easy-hyperparameter-management-with-hydra-mlflow-and-optuna-783730700e7d">Easy Hyperparameter Management with Hydra, MLflow, and Optuna</a>: A post explaining how to combine Optuna and MLFlow, which is another library to track machine learning experiments.</li><li><a href="https://github.com/optuna/optuna-dashboard">Optuna dashboard</a>: Official web dashboard for Optuna’s experiments.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58fc6bab893" width="1" height="1" alt=""><hr><p><a href="https://medium.com/optuna/optuna-meets-weights-and-biases-58fc6bab893">Optuna Meets Weights and Biases</a> was originally published in <a href="https://medium.com/optuna">Optuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>