<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Siddharth Sah on Medium]]></title>
        <description><![CDATA[Stories by Siddharth Sah on Medium]]></description>
        <link>https://medium.com/@siddharthksah?source=rss-1aa18b2cd060------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*RTWoIcWgxb9qaY9qBDHikA.jpeg</url>
            <title>Stories by Siddharth Sah on Medium</title>
            <link>https://medium.com/@siddharthksah?source=rss-1aa18b2cd060------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:48:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@siddharthksah/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Navigating the Maze: Streamlined Project Structure for Data Science and everything around it]]></title>
            <link>https://siddharthksah.medium.com/navigating-the-maze-streamlined-project-structure-for-data-science-and-everything-around-it-dbd7ac19ca0e?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/dbd7ac19ca0e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[full-stack]]></category>
            <category><![CDATA[project-structure]]></category>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 10:31:06 GMT</pubDate>
            <atom:updated>2023-03-15T10:31:06.324Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*PDHm7cyAj5K3TuT80i4ZmQ.jpeg" /></figure><p>In the world of data science and machine learning, structuring your project is crucial for ensuring smooth collaboration, maintainability, and scalability. This article will guide you through the best practices for organising your data science or machine learning projects, complete with code snippets and bullet points for easy understanding.</p><h4>Table of Contents:</h4><ol><li>Overview of Project Structure</li><li>Directory Structure</li><li>Managing Dependencies</li><li>Version Control</li><li>Jupyter Notebooks and Code Organization</li><li>Testing and Validation</li><li>Deployment and Monitoring</li></ol><h3>Overview of Project Structure:</h3><p>A well-structured project helps in managing code, data, and resources efficiently, and it facilitates collaboration between team members.</p><h3>Directory Structure:</h3><p>The directory structure should be organized in a way that separates different components of the project. Here’s a suggested structure:</p><pre>project_name/<br>|-- data/<br>|   |-- raw/<br>|   |-- processed/<br>|-- docs/<br>|-- models/<br>|-- notebooks/<br>|-- src/<br>|   |-- __init__.py<br>|   |-- data/<br>|   |   |-- __init__.py<br>|   |   |-- data_preparation.py<br>|   |-- features/<br>|   |   |-- __init__.py<br>|   |   |-- feature_extraction.py<br>|   |-- models/<br>|   |   |-- __init__.py<br>|   |   |-- model_building.py<br>|   |   |-- model_evaluation.py<br>|-- tests/<br>|-- README.md<br>|-- .gitignore<br>|-- requirements.txt</pre><ul><li>data/: Contains raw and processed data files</li><li>docs/: Documentation for the project</li><li>models/: Trained machine learning models</li><li>notebooks/: Jupyter notebooks for exploratory analysis and experimentation</li><li>src/: Source code for the project, organized into subdirectories for data preparation, feature extraction, and modeling</li><li>tests/: Unit tests and integration tests</li><li>README.md: Project description, setup instructions, and usage information</li><li>.gitignore: List of files and directories to be ignored by Git</li><li>requirements.txt: List of Python packages required for the project</li></ul><h3>Managing Dependencies:</h3><p>To manage dependencies effectively, use virtual environments and list your project’s dependencies in a requirements.txt file. This makes it easier for others to set up the project and ensures consistency in the development environment.</p><pre>python -m venv venv<br>source venv/bin/activate<br>pip install -r requirements.txt</pre><h4>Managing Conda Environments and Reproducing Them</h4><p>Conda is a popular package and environment manager for Python and R programming languages. It allows you to manage multiple environments and isolate dependencies, making it easier to reproduce your project on different machines. In this guide, we will walk through creating and managing Conda environments and demonstrate how to reproduce them.</p><h4>Installing Conda</h4><p>To use Conda, you need to install either Anaconda or Miniconda. Anaconda is a full distribution that includes many scientific libraries, while Miniconda is a lightweight distribution containing only the essential packages. Choose the one that best suits your needs and follow the installation instructions from the official websites:</p><ul><li>Anaconda: <a href="https://www.anaconda.com/products/distribution">https://www.anaconda.com/products/distribution</a></li><li>Miniconda: <a href="https://docs.conda.io/en/latest/miniconda.html">https://docs.conda.io/en/latest/miniconda.html</a></li></ul><h4>Creating a new Conda environment</h4><p>To create a new Conda environment, open a terminal and run the following command:</p><pre>conda create --name my_env python=3.8</pre><p>Replace my_env with the desired name for your environment and 3.8 with the Python version you want to use.</p><h4>Activating and deactivating environments</h4><p>To activate the environment, use the following command:</p><pre>conda activate my_env</pre><h4>To deactivate the environment and return to the base environment, run:</h4><pre>conda deactivate</pre><h4>Installing packages</h4><p>With your environment activated, you can install packages using the conda install command. For example, to install NumPy and Pandas, run:</p><pre>conda install numpy pandas</pre><h4>Exporting environment dependencies</h4><p>To reproduce your Conda environment, you need to export its dependencies to a file. This file, often called an “environment file,” contains a list of all the packages and their versions installed in the environment. To create an environment file, run:</p><pre>conda env export &gt; environment.yml</pre><p>This command will create a file named environment.yml in your current directory. Make sure to include this file in your project repository to ensure reproducibility.</p><h4>Reproducing the environment</h4><p>To reproduce the environment on another machine or for another user, follow these steps:</p><ol><li>Install Conda (either Anaconda or Miniconda) on the target machine.</li><li>Clone your project repository, which should include the environment.yml file.</li><li>Open a terminal, navigate to your project directory, and create a new Conda environment using the environment.yml file:</li></ol><pre>conda env create -f environment.yml</pre><p>4. Activate the environment:</p><pre>conda activate my_env</pre><p>Now, the environment has been reproduced with all dependencies installed.</p><h4>Updating the environment</h4><p>When you update your project dependencies, remember to update the environment.yml file as well. You can do this by exporting the updated environment as follows:bashCopy cod</p><pre>conda env export &gt; environment.yml</pre><h4>Removing an environment</h4><p>To remove an environment, run:</p><pre>conda env remove --name my_env</pre><p>Replace my_env with the name of the environment you want to remove.</p><p>By following these steps, you can effectively manage and reproduce your Conda environments, ensuring that your data science and machine learning projects run smoothly across different machines and environments.</p><h3>Version Control:</h3><p>Version control is essential for tracking changes, collaborating with others, and maintaining the project’s history. Git is a popular choice for version control.</p><pre>git init<br>git add .<br>git commit -m &quot;Initial commit&quot;<br>git remote add origin &lt;your-repository-url&gt;<br>git push -u origin master</pre><p>In data science and machine learning projects, the dataset is as important as the code. Changes in data can lead to significant variations in model performance and accuracy. Therefore, it is crucial to maintain proper version control for your datasets, just as you would for your codebase.</p><p>Version control for datasets can be achieved using various tools and techniques. Here, we will discuss some popular tools and best practices for dataset version control:</p><h4>Data Version Control (DVC):</h4><p>DVC is an open-source tool specifically designed for dataset version control. It works alongside traditional version control systems like Git and enables efficient management of large datasets. DVC tracks changes in data files and directories, storing them in a separate remote storage such as Amazon S3, Google Cloud Storage, or local file storage. Here are the key benefits of DVC:</p><ul><li>Storage-efficient: DVC uses data deduplication and delta encoding techniques to store only the changes between different dataset versions.</li><li>Easy integration: DVC works well with existing Git repositories, making it easy to incorporate dataset versioning into your current workflow.</li><li>Reproducibility: DVC keeps track of the relationship between code and data, allowing you to recreate any previous version of your dataset and the corresponding model.</li></ul><p>In this example, we will walk through a simple DVC workflow for beginners. We will create a DVC project, add data to it, and demonstrate basic DVC commands.</p><ol><li>Install DVC</li></ol><p>First, you need to install DVC. You can do this using pip:bashCopy coe</p><pre>pip install dvc</pre><p>2. Initialize a DVC project</p><p>To create a new DVC project, navigate to your project directory and run the following command:</p><pre>dvc init</pre><p>This will initialize a DVC project and create a .dvc folder in your project directory.</p><p>3. Configure remote storage</p><p>Next, configure your remote storage. In this example, we will use a local directory as remote storage. Replace /path/to/remote/storage with the path to your desired remote storage location:</p><pre>dvc remote add -d myremote /path/to/remote/storage</pre><p>4. Add data to the DVC project</p><p>Now, let’s add a dataset to the DVC project. Assume you have a CSV file called data.csv in your project directory. To start tracking the dataset with DVC, run the following command:</p><pre>dvc add data.csv</pre><p>This command will create a data.csv.dvc file in your project directory, which contains metadata about the dataset. Make sure to add this file to your Git repository to track dataset changes:</p><pre>git add data.csv.dvc</pre><p>5. Push data to remote storage</p><p>To store the dataset in your remote storage, run:</p><pre>dvc push</pre><p>6. Retrieve a specific version of the dataset</p><p>If you want to retrieve a specific version of the dataset from remote storage, first checkout the desired Git commit, and then run</p><pre>dvc pull</pre><p>This will download the dataset associated with that Git commit from the remote storage.</p><h3>Jupyter Notebooks and Code Organization:</h3><p>Jupyter notebooks are great for exploratory analysis, visualization, and prototyping. However, they can become unwieldy for large projects. It’s a good idea to refactor reusable code from notebooks into Python scripts within the src/ directory.</p><p>Jupyter Notebooks are an essential tool for data science and machine learning projects. They provide an interactive environment to write, run, and document your code. However, as projects grow in complexity, it is crucial to organize your notebooks and code effectively. In this section, we will discuss some best practices for organizing Jupyter Notebooks and code.</p><h4>1. Separate code into logical sections</h4><p>Use Jupyter Notebooks to break your code into logical sections or “cells.” Each cell should ideally contain a single task or step, making it easier to understand the flow of your project. To create a new cell, click the “+” button in the toolbar or press Shift + Enter after running a cell.</p><h4>2. Use Markdown cells for documentation</h4><p>Jupyter Notebooks support Markdown, a lightweight markup language that allows you to format text easily. Use Markdown cells to provide explanations, instructions, and context for your code. To create a Markdown cell, select the cell type as “Markdown” from the dropdown menu in the toolbar, or press M when the cell is selected.</p><p>Here’s an example of a Markdown cell:</p><pre># This is a header<br><br>This is a paragraph with *italic text* and **bold text**.<br><br>- This is a bullet point list<br>- Another item</pre><h4>3. Modularise your code</h4><p>As your project grows, consider modularising your code by moving reusable functions and classes into separate Python files. This helps keep your notebook clean and focused on high-level tasks.</p><p>For example, let’s say you have a utility function process_data() in your notebook:</p><pre>def process_data(data):<br>    # Do something with the data<br>    return processed_data</pre><p>Move this function to a separate Python file, e.g., utils.py, and then import it into your notebook:</p><pre>from utils import process_data<br># Use the function in your notebook<br>processed_data = process_data(raw_data)</pre><p>Useful things you can do in Jupyter Notebook, along with code examples:</p><h4>1. Interactive plotting with Matplotlib</h4><p>Jupyter Notebook allows you to create and display interactive plots directly in the notebook. By using the %matplotlib inline magic command, you can render your plots in the notebook itself.</p><pre>%matplotlib inline<br>import matplotlib.pyplot as plt<br>import numpy as np<br>x = np.linspace(0, 10, 100)<br>y = np.sin(x)<br>plt.plot(x, y)<br>plt.xlabel(&#39;x&#39;)<br>plt.ylabel(&#39;sin(x)&#39;)<br>plt.title(&#39;Sine Wave&#39;)<br>plt.show()</pre><h4>2. Displaying images and videos</h4><p>Jupyter Notebook can display images and videos from local files or URLs. Use the IPython display module to load and display multimedia content.</p><pre>from IPython.display import Image, display<br># Display an image from a URL<br>url = &#39;https://www.example.com/path/to/image.jpg&#39;<br>display(Image(url=url))<br># Display a local image<br>local_image_path = &#39;path/to/local/image.jpg&#39;<br>display(Image(filename=local_image_path))</pre><h4>3. Running shell commands</h4><p>You can run shell commands directly within a Jupyter Notebook using the ! prefix. This is useful for installing packages, checking file contents, or running scripts.</p><pre># List the contents of the current directory<br>!ls</pre><pre># Install a package using pip<br>!pip install numpy</pre><h4>4. Magic commands</h4><p>Magic commands are special commands that provide additional functionality in Jupyter Notebook. They are prefixed with % for line magics and %% for cell magics.</p><pre># Measure the execution time of a single line of code<br>%timeit sum(range(1000))<br># Measure the execution time of an entire cell<br>%%timeit<br>total = 0<br>for i in range(1000):<br>    total += i</pre><h3>5. Interactive widgets</h3><p>Jupyter Notebook supports interactive widgets that allow you to create interactive user interfaces directly in the notebook. Use the ipywidgets library to create and display widgets.</p><pre>import ipywidgets as widgets<br># Create a slider widget<br>slider = widgets.IntSlider(min=0, max=10, step=1, value=5)<br>display(slider)<br># Get the current value of the slider<br>slider.value</pre><p>I have started using Jupyter Lab more than only the notebook. It comes with some great tools.</p><p>Here are some of the key features and usage of JupyterLab:</p><ol><li>Flexible workspace: JupyterLab allows you to arrange multiple notebooks, text editors, terminals, and other components using a tabbed and paneled interface. You can customize your layout by resizing, splitting, and arranging these components to suit your workflow.</li><li>File browser: JupyterLab comes with a built-in file browser that enables you to navigate, create, and manage files and directories in your project. You can also drag and drop files between the file browser and your workspace.</li><li>Code editor: JupyterLab includes a powerful code editor with syntax highlighting, code completion, and automatic indentation. The code editor supports various programming languages, including Python, R, Julia, and more.</li><li>Integrated terminal: You can open and use multiple terminals directly within JupyterLab, allowing you to run shell commands, start external processes, or interact with version control systems like Git.</li><li>Extensions: JupyterLab has a growing ecosystem of extensions that add new functionality and customize the user interface. Some popular extensions include JupyterLab Git (for Git integration), JupyterLab LSP (for Language Server Protocol support), and JupyterLab DrawIO (for creating diagrams and flowcharts).</li><li>Rich output and interactivity: Just like the classic Jupyter Notebook, JupyterLab supports rich output and interactive widgets for data visualization and exploration. You can display images, videos, plots, and interactive elements directly within your notebook cells.</li></ol><p>To get started with JupyterLab, you can install it using pip or conda:</p><pre># Install using pip<br>pip install jupyterlab</pre><pre># Install using conda<br>conda install -c conda-forge jupyterlab</pre><p>Once installed, you can launch JupyterLab by running the following command in your terminal:</p><pre>jupyter lab</pre><p>This will open JupyterLab in your default web browser, allowing you to create, open, and edit notebooks, as well as use other features of JupyterLab.</p><h3>Testing and Validation:</h3><p>To ensure the reliability and robustness of your project, write unit tests and integration tests. Store tests in the tests/ directory and use a testing framework like pytest to run them.</p><pre>pip install pytest<br>pytest tests/</pre><p>Unit testing: Unit tests check the correctness of individual functions and classes in isolation. Python has a built-in library called unittest for writing and running unit tests. There are also third-party libraries like pytest that offer more advanced features and a simpler syntax.</p><p>To demonstrate unit testing, let’s consider a simple function that adds two numbers:</p><pre>def add(a, b):<br>    return a + b</pre><p>We can write a unit test for this function using unittest:</p><pre>import unittest<br>class TestAddition(unittest.TestCase):<br>    def test_add(self):<br>        self.assertEqual(add(2, 3), 5)<br>        self.assertEqual(add(-1, 1), 0)<br>        self.assertEqual(add(0, 0), 0)<br>if __name__ == &#39;__main__&#39;:<br>    unittest.main()</pre><p>Or using pytest:</p><pre>def test_add():<br>    assert add(2, 3) == 5<br>    assert add(-1, 1) == 0<br>    assert add(0, 0) == 0</pre><p>Model validation: To assess the performance of machine learning models, you can use validation techniques like k-fold cross-validation, train-test split, or holdout validation. Scikit-learn provides useful tools for model validation:</p><pre>from sklearn.model_selection import train_test_split, cross_val_score<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.metrics import accuracy_score<br>from sklearn.datasets import load_iris<br># Load the Iris dataset<br>data = load_iris()<br>X, y = data.data, data.target<br># Split the data into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)<br># Train a logistic regression model<br>model = LogisticRegression()<br>model.fit(X_train, y_train)<br># Test the model on the test set<br>y_pred = model.predict(X_test)<br>accuracy = accuracy_score(y_test, y_pred)<br>print(f&quot;Accuracy: {accuracy:.2f}&quot;)<br># Perform k-fold cross-validation<br>cv_scores = cross_val_score(model, X, y, cv=5)<br>print(f&quot;Cross-validation scores: {cv_scores}&quot;)<br>print(f&quot;Mean CV accuracy: {cv_scores.mean():.2f}&quot;)</pre><p>Testing Jupyter Notebooks: If you’re using Jupyter Notebooks for your project, you can use nbval to run tests within the notebook. nbval is a plugin for pytest that allows you to validate the output of your notebook cells. To use nbval, install it using pip:</p><pre>pip install nbval</pre><p>Then, add assertions in your notebook cells and run the tests with the following command:</p><pre>pytest --nbval your_notebook.ipynb</pre><p>Hyperparameter optimization and model export are essential steps in a machine learning pipeline, enabling you to fine-tune your models and deploy them for predictions. This article will provide an in-depth guide to hyperparameter optimization techniques, model exporting, and using these optimized models for making predictions.</p><h3>Hyperparameter Optimization</h3><p>Hyperparameter optimization is the process of searching for the optimal combination of hyperparameters to achieve the best model performance. Here, we’ll discuss three popular hyperparameter optimization techniques:</p><ol><li>Grid Search</li><li>Random Search</li><li>Bayesian Optimization</li></ol><h4>Grid Search</h4><p>Grid search is a brute-force approach to hyperparameter optimization. It involves defining a set of hyperparameter values and then evaluating all possible combinations.</p><pre>from sklearn.model_selection import GridSearchCV<br>from sklearn.ensemble import RandomForestClassifier<br>param_grid = {<br>    &#39;n_estimators&#39;: [10, 50, 100, 200],<br>    &#39;max_depth&#39;: [None, 10, 20, 30],<br>    &#39;min_samples_split&#39;: [2, 5, 10]<br>}<br>model = RandomForestClassifier()<br>grid_search = GridSearchCV(estimator=model, param_grid=param_grid, cv=5)<br>grid_search.fit(X_train, y_train)<br>print(&quot;Best parameters found: &quot;, grid_search.best_params_)</pre><h4>Random Search</h4><p>Random search, unlike grid search, samples hyperparameter values randomly from a specified distribution. This method is faster and can sometimes achieve better results than grid search.</p><pre>from sklearn.model_selection import RandomizedSearchCV<br>from scipy.stats import randint<br>param_dist = {<br>    &#39;n_estimators&#39;: randint(10, 200),<br>    &#39;max_depth&#39;: [None] + list(randint(1, 30).rvs(size=3)),<br>    &#39;min_samples_split&#39;: randint(2, 11)<br>}<br>random_search = RandomizedSearchCV(estimator=model, param_distributions=param_dist, cv=5, n_iter=50)<br>random_search.fit(X_train, y_train)<br>print(&quot;Best parameters found: &quot;, random_search.best_params_)</pre><h4>Bayesian Optimization</h4><p>Bayesian optimization is a more advanced technique that leverages probability distributions to search for optimal hyperparameter values.</p><pre>from skopt import BayesSearchCV<br>from skopt.space import Integer, Categorical<br>param_space = {<br>    &#39;n_estimators&#39;: Integer(10, 200),<br>    &#39;max_depth&#39;: Categorical([None] + list(range(1, 31))),<br>    &#39;min_samples_split&#39;: Integer(2, 10)<br>}<br>bayes_search = BayesSearchCV(estimator=model, search_spaces=param_space, cv=5, n_iter=50)<br>bayes_search.fit(X_train, y_train)<br>print(&quot;Best parameters found: &quot;, bayes_search.best_params_)</pre><p>HyperOpt is another popular library for hyperparameter optimization using a Sequential Model-based Global Optimization (SMBO) algorithm. The following example demonstrates how to use HyperOpt for optimizing the YOLOv5 model’s hyperparameters.</p><p>First, you’ll need to install the hyperopt library:</p><pre>pip install hyperopt</pre><p>Now, let’s create a custom evaluation function that will train a YOLOv5 model and return its loss.</p><pre>from hyperopt import fmin, tpe, hp<br>import subprocess<br>import re<br>def train_yolov5(params):<br>    # Set your YOLOv5 training command<br>    command = f&quot;python train.py --img 640 --batch 16 --epochs 100 --data dataset.yaml --weights yolov5s.pt --cache --nosave --hyp {params[&#39;hyp&#39;]}&quot;<br>    # Execute the training command<br>    output = subprocess.check_output(command, shell=True).decode(&quot;utf-8&quot;)<br>    <br>    # Extract loss from the training output<br>    loss = float(re.search(r&quot;best_fitness=([\d.]+)&quot;, output).group(1))<br>    <br>    return loss</pre><pre>space = {<br>    &#39;hyp&#39;: {<br>        &#39;lr0&#39;: hp.loguniform(&#39;lr0&#39;, -5, -2),<br>        &#39;lrf&#39;: hp.loguniform(&#39;lrf&#39;, -5, -2),<br>        &#39;momentum&#39;: hp.uniform(&#39;momentum&#39;, 0.6, 0.98),<br>        &#39;weight_decay&#39;: hp.loguniform(&#39;weight_decay&#39;, -9, -4),<br>        &#39;warmup_epochs&#39;: hp.quniform(&#39;warmup_epochs&#39;, 0, 5, 1),<br>        &#39;warmup_momentum&#39;: hp.uniform(&#39;warmup_momentum&#39;, 0.6, 0.98),<br>        &#39;box&#39;: hp.uniform(&#39;box&#39;, 0.02, 0.2),<br>        &#39;cls&#39;: hp.uniform(&#39;cls&#39;, 0.2, 2),<br>        &#39;cls_pw&#39;: hp.uniform(&#39;cls_pw&#39;, 0.5, 2),<br>        &#39;obj&#39;: hp.uniform(&#39;obj&#39;, 0.2, 2),<br>        &#39;obj_pw&#39;: hp.uniform(&#39;obj_pw&#39;, 0.5, 2),<br>        &#39;iou_t&#39;: hp.uniform(&#39;iou_t&#39;, 0.1, 0.7),<br>        &#39;anchor_t&#39;: hp.uniform(&#39;anchor_t&#39;, 2, 5),<br>        &#39;fl_gamma&#39;: hp.uniform(&#39;fl_gamma&#39;, 0, 2),<br>        &#39;hsv_h&#39;: hp.uniform(&#39;hsv_h&#39;, 0, 0.1),<br>        &#39;hsv_s&#39;: hp.uniform(&#39;hsv_s&#39;, 0, 0.9),<br>        &#39;hsv_v&#39;: hp.uniform(&#39;hsv_v&#39;, 0, 0.9),<br>        &#39;degrees&#39;: hp.uniform(&#39;degrees&#39;, 0, 45),<br>        &#39;translate&#39;: hp.uniform(&#39;translate&#39;, 0, 0.9),<br>        &#39;scale&#39;: hp.uniform(&#39;scale&#39;, 0, 0.9),<br>        &#39;shear&#39;: hp.uniform(&#39;shear&#39;, 0, 10),<br>        &#39;perspective&#39;: hp.uniform(&#39;perspective&#39;, 0, 0.001),<br>        &#39;flipud&#39;: hp.uniform(&#39;flipud&#39;, 0, 1),<br>        &#39;fliplr&#39;: hp.uniform(&#39;fliplr&#39;, 0, 1),<br>        &#39;mosaic&#39;: hp.uniform(&#39;mosaic&#39;, 0, 1),<br>        &#39;mixup&#39;: hp.uniform(&#39;mixup&#39;, 0, 1)<br>    }<br>}</pre><p>Finally, let’s run the optimization using the fmin function from HyperOpt. This function will minimize the loss returned by the train_yolov5 function by searching for the best hyperparameters in the defined search space.</p><pre># Set the number of evaluations<br>max_evals = 50<br># Run the optimization<br>best = fmin(<br>    fn=train_yolov5,<br>    space=space,<br>    algo=tpe.suggest,<br>    max_evals=max_evals,<br>    verbose=2<br>)<br>print(&quot;Best hyperparameters found: &quot;, best)</pre><p>After the optimization is complete, you’ll get the best hyperparameters for your YOLOv5 model. You can then use these hyperparameters to train your final YOLOv5 model and achieve improved performance.</p><h3>Exporting Models for Prediction</h3><p>After optimizing your model, it’s time to export it for predictions. We’ll use the popular library joblib to save and load models.</p><pre>import joblib<br># Save the model<br>joblib.dump(grid_search.best_estimator_, &quot;optimized_model.pkl&quot;)<br># Load the model<br>loaded_model = joblib.load(&quot;optimized_model.pkl&quot;)</pre><h3>Making Predictions with the Exported Model</h3><p>Once you’ve loaded the optimized model, you can use it to make predictions on new, unseen data.</p><pre># Make predictions<br>predictions = loaded_model.predict(X_test)<br># Evaluate the performance<br>from sklearn.metrics import accuracy_score<br>accuracy = accuracy_score(y_test, predictions)<br>print(&quot;Accuracy: &quot;, accuracy)</pre><p>Exporting a trained model and using it for prediction is an essential step in any machine learning workflow. In this guide, we’ll walk through the process of exporting a PyTorch model and using it for making predictions.</p><ol><li>Train a simple PyTorch model</li></ol><p>First, let’s create a simple neural network and train it on some dummy data. We’ll use the Iris dataset for this example.</p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from sklearn.datasets import load_iris<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br># Load the Iris dataset<br>iris = load_iris()<br>X = iris.data<br>y = iris.target<br># Split the dataset into training and testing sets<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br># Scale the data<br>scaler = StandardScaler()<br>X_train = scaler.fit_transform(X_train)<br>X_test = scaler.transform(X_test)<br># Convert numpy arrays to PyTorch tensors<br>X_train = torch.tensor(X_train, dtype=torch.float32)<br>X_test = torch.tensor(X_test, dtype=torch.float32)<br>y_train = torch.tensor(y_train, dtype=torch.long)<br>y_test = torch.tensor(y_test, dtype=torch.long)<br># Define the neural network<br>class IrisNet(nn.Module):<br>    def __init__(self):<br>        super(IrisNet, self).__init__()<br>        self.fc1 = nn.Linear(4, 16)<br>        self.fc2 = nn.Linear(16, 8)<br>        self.fc3 = nn.Linear(8, 3)<br>    def forward(self, x):<br>        x = torch.relu(self.fc1(x))<br>        x = torch.relu(self.fc2(x))<br>        x = self.fc3(x)<br>        return x<br># Initialize the model, loss function, and optimizer<br>model = IrisNet()<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters(), lr=0.001)<br># Train the model<br>num_epochs = 100<br>for epoch in range(num_epochs):<br>    optimizer.zero_grad()<br>    outputs = model(X_train)<br>    loss = criterion(outputs, y_train)<br>    loss.backward()<br>    optimizer.step()<br>    if (epoch+1) % 10 == 0:<br>        print(f&#39;Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item()}&#39;)</pre><p>2. Save the trained model</p><p>After training the model, we can save it to a file for later use.</p><pre>torch.save(model.state_dict(), &#39;iris_net.pth&#39;)</pre><p>3. Load the saved model</p><p>To use the saved model for predictions, we’ll first need to load the model state into a new instance of the neural network.</p><pre>loaded_model = IrisNet()<br>loaded_model.load_state_dict(torch.load(&#39;iris_net.pth&#39;))<br>loaded_model.eval()  # Set the model to evaluation mode</pre><p>4. Make predictions</p><p>Now we can use the loaded model to make predictions on new, unseen data.</p><pre>with torch.no_grad():<br>    test_outputs = loaded_model(X_test)<br>    _, predicted = torch.max(test_outputs, 1)<br>    accuracy = (predicted == y_test).sum().item() / len(y_test)<br>    print(f&#39;Accuracy: {accuracy * 100}%&#39;)</pre><p>This example demonstrates the entire process of training a simple PyTorch model, saving it to a file, loading the saved model, and making predictions using the loaded model. You can adapt this process to your specific use case and model architecture.</p><h3>Deployment and Monitoring:</h3><p>Once your project is complete, you may need to deploy it in a production environment. This involves packaging your code, setting up the necessary infrastructure, and configuring the deployment process. Here are some best practices to consider during deployment:</p><ul><li>Use continuous integration and continuous deployment (CI/CD) tools such as Jenkins, CircleCI, or GitHub Actions to automate the deployment process.</li><li>Containerise your application using Docker to ensure consistent and reproducible environments across development and production stages.</li><li>Use cloud platforms such as AWS, Google Cloud Platform, or Microsoft Azure for scalable and flexible infrastructure.</li><li>Monitor your deployed models with tools like Prometheus, Grafana, or custom-built monitoring solutions to track performance, usage, and errors.</li></ul><p>Deployment and monitoring are important aspects of machine learning and data science projects, as they enable you to put your models into production and ensure their performance over time. In this section, we’ll discuss some popular tools for deploying and monitoring your models, including Streamlit for building web applications and cloud platforms for scalable deployments.</p><p>Streamlit: Streamlit is an open-source Python library that makes it easy to create custom web applications for machine learning and data science projects. With just a few lines of code, you can build interactive dashboards and visualize your data. To get started, install Streamlit using pip:</p><pre>pip install streamlit</pre><p>Next, create a Python script (e.g., app.py) with the following code to build a simple Streamlit application:</p><pre>import streamlit as st<br>st.title(&quot;My First Streamlit App&quot;)<br>user_input = st.text_input(&quot;Enter your name:&quot;)<br>st.write(f&quot;Hello, {user_input}!&quot;)</pre><p>Run the Streamlit app by executing the following command:</p><pre>streamlit run app.py</pre><p>This will launch your web application in the browser.</p><h4>Docker</h4><p>Docker is a containerization platform that allows developers to package applications along with their dependencies into a standardized unit called a container. This ensures that the application will run consistently across different environments. In this guide, we’ll demonstrate how to create a simple Python web app using Streamlit and package it using Docker.</p><p>Step 1: Create a Streamlit web app</p><p>First, let’s create a simple Streamlit web app. Create a new Python file named app.py and add the following code:</p><pre>import streamlit as st<br>st.title(&quot;Hello, Streamlit!&quot;)<br>st.write(&quot;Welcome to our simple Python web app built using Streamlit!&quot;)<br>st.write(&quot;Enter your name below and press &#39;Submit&#39; to see a personalized message:&quot;)<br>name = st.text_input(&quot;Your Name&quot;)<br>if st.button(&quot;Submit&quot;):<br>    st.write(f&quot;Hello, {name}! Nice to meet you!&quot;)</pre><p>Step 2: Install Streamlit</p><p>If you haven’t already, install Streamlit using pip:</p><pre>pip install streamlit</pre><p>Step 3: Run the Streamlit web app</p><p>To run the web app, use the following command:</p><pre>streamlit run app.py</pre><p>Your app should now be running on <a href="http://localhost:8501.">http://localhost:8501.</a></p><p>Step 4: Create a Dockerfile</p><p>To containerize the web app using Docker, create a new file named Dockerfile in the same directory as your app.py file and add the following content:</p><pre># Use an official Python runtime as a parent image<br>FROM python:3.9-slim<br># Set the working directory<br>WORKDIR /app<br># Copy requirements.txt into the container<br>COPY requirements.txt .<br># Install any needed packages specified in requirements.txt<br>RUN pip install --trusted-host pypi.python.org -r requirements.txt<br># Make port 8501 available to the world outside this container<br>EXPOSE 8501<br># Copy the rest of the application files into the container<br>COPY . .<br># Define environment variable<br>ENV NAME World<br># Run app.py when the container launches<br>CMD [&quot;streamlit&quot;, &quot;run&quot;, &quot;app.py&quot;]</pre><p>Step 5: Create a requirements.txt file</p><p>Create a new file named requirements.txt in the same directory as your app.py file and add the following content:</p><pre>streamlit</pre><p>Step 6: Build the Docker image</p><p>Navigate to the directory containing your Dockerfile and run the following command to build the Docker image:</p><pre>docker build -t streamlit-webapp .</pre><p>Step 7: Run the Docker container</p><p>After building the image, run the Docker container using the following command:</p><pre>docker run -p 8501:8501 streamlit-webapp</pre><p>Your Streamlit web app should now be running inside a Docker container and accessible at <a href="http://localhost:8501.">http://localhost:8501.</a></p><p>With these steps, you’ve successfully created a simple Python web app using Streamlit and containerized it using Docker. This setup allows you to easily deploy and scale your application in different environments.</p><h3>Deployment on the cloud:</h3><p>Cloud platforms like AWS, Google Cloud, and Microsoft Azure provide various services to deploy and manage your machine learning models. These platforms offer scalable, cost-effective solutions for production deployments.</p><p>For example, to deploy a machine learning model on Google Cloud using AI Platform, follow these steps:</p><ol><li>Train and save your model in a format supported by AI Platform, such as TensorFlow SavedModel or Scikit-learn model:</li></ol><pre>import joblib<br>from sklearn.datasets import load_iris<br>from sklearn.linear_model import LogisticRegression<br># Train a logistic regression model<br>data = load_iris()<br>X, y = data.data, data.target<br>model = LogisticRegression()<br>model.fit(X, y)<br># Save the model<br>joblib.dump(model, &#39;model.joblib&#39;)</pre><p>2. Create a Google Cloud Storage bucket and upload your model:</p><pre>gsutil mb gs://your-bucket-name<br>gsutil cp model.joblib gs://your-bucket-name/model.joblib</pre><p>3. Create an AI Platform model and version:</p><pre>gcloud ai-platform models create your_model_name<br>gcloud ai-platform versions create your_version_name \<br>  --model=your_model_name \<br>  --origin=gs://your-bucket-name/ \<br>  --runtime-version=2.6 \<br>  --python-version=3.7 \<br>  --framework=SCIKIT_LEARN</pre><p>4. Send prediction requests to your deployed model:</p><pre>from googleapiclient import discovery<br>import numpy as np<br># Create the AI Platform API client<br>api = discovery.build(&#39;ml&#39;, &#39;v1&#39;)<br># Prepare a sample for prediction<br>sample = np.array([5.1, 3.5, 1.4, 0.2]).reshape(1, -1).tolist()<br># Send the prediction request<br>response = api.projects().predict(<br>    name=&#39;projects/your_project_id/models/your_model_name/versions/your_version_name&#39;,<br>    body={&#39;instances&#39;: sample}<br>).execute()<br># Print the prediction results<br>print(response[&#39;predictions&#39;])</pre><p>Monitoring: Once your models are deployed, it’s important to monitor their performance and usage. Cloud platforms provide monitoring and logging services to help you track your model’s performance, resource usage, and errors. For example, Google Cloud’s AI Platform integrates with Cloud Monitoring and Cloud Logging for real-time monitoring and logging of your deployed models.</p><h3>Conclusion</h3><p>In this comprehensive guide, we’ve ventured through the winding path of structuring a data science or machine learning project. From organizing directories and managing dependencies, to deploying and monitoring your models, we’ve covered it all — with a sprinkle of code snippets and a dash of humor.</p><p>Remember, a well-structured project is the secret ingredient to success in the kitchen of data science and machine learning. It keeps your code fresh, your collaborators happy, and your models performing at their best. So, go forth, dear reader, and conquer the world of machine learning, one well-organized project at a time.</p><p>And if you ever find yourself lost in the maze of code, data, and models, remember that a pinch of humour and a healthy dose of best practices will guide you back to the light. Happy coding!</p><h3>References:</h3><ol><li>Jupyter Project. (n.d.). Jupyter Notebook. <a href="https://jupyter.org/">https://jupyter.org</a></li><li>Jupyter Project. (n.d.). JupyterLab. <a href="https://jupyterlab.readthedocs.io/">https://jupyterlab.readthedocs.io</a></li><li>DVC (Data Version Control). (n.d.). <a href="https://dvc.org/">https://dvc.org</a></li><li>Conda. (n.d.). Conda: Package, dependency, and environment management for any language. <a href="https://conda.io/">https://conda.io</a></li><li>Streamlit. (n.d.). Streamlit — The fastest way to build custom ML tools. <a href="https://www.streamlit.io/">https://www.streamlit.io</a></li><li>Hyperopt. (n.d.). Distributed Asynchronous Hyperparameter Optimization in Python. <a href="https://hyperopt.github.io/hyperopt">https://hyperopt.github.io/hyperopt</a></li><li>Ultralytics. (n.d.). YOLOv5. <a href="https://github.com/ultralytics/yolov5">https://github.com/ultralytics/yolov5</a></li><li>PyTorch. (n.d.). PyTorch: An open-source machine learning library. <a href="https://pytorch.org/">https://pytorch.org</a></li><li>pytest. (n.d.). pytest: Simple, rapid testing framework for Python. <a href="https://pytest.org/">https://pytest.org</a></li><li>Docker. (n.d.). Docker: Empowering App Development for Developers. <a href="https://www.docker.com/">https://www.docker.com</a></li></ol><p><strong><em>If you found this article helpful and insightful, don’t forget to show your support by giving it a clap, or maybe even a standing ovation! Feel free to follow me for more articles that’ll keep you up-to-date on the latest trends and best practices in data science, machine learning, and beyond. Let’s keep learning and growing together!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dbd7ac19ca0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating SSH Login and Jupyter Notebook Setup for Machine Learning Projects]]></title>
            <link>https://siddharthksah.medium.com/automating-ssh-login-and-jupyter-notebook-setup-for-machine-learning-projects-bd2967a832d?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd2967a832d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[servers]]></category>
            <category><![CDATA[remote-work]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[remote-access]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Tue, 07 Mar 2023 08:26:53 GMT</pubDate>
            <atom:updated>2023-03-07T08:34:02.125Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uXCVNW0noXAH1uS4nZIBXw.jpeg" /><figcaption>Used the title of this article as a prompt to generate this image</figcaption></figure><p>SSH Login: SSH (Secure Shell) is a cryptographic network protocol for operating network services securely over an unsecured network. It is commonly used for remote command-line login and remote command execution. To automate SSH login, we will use the Paramiko library.</p><p>Paramiko is a Python implementation of the SSH protocol. It allows you to create SSH connections and execute commands on the remote server. In our script, we get the SSH ID, password, Python version, Conda flag, and venv flag as command-line arguments. We also check if Paramiko is installed and install it if it is not installed.</p><p>Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. It is widely used in the machine learning community. To automate Jupyter Notebook setup, we will use the following steps:</p><ol><li>Check if Conda is installed on the remote server, and install it if it does not exist.</li><li>Create a new Conda environment with the specified Python version if the Conda flag is set to True.</li><li>Create a new Python virtual environment using venv if the venv flag is set to True.</li><li>Install Jupyter Notebook and create a new configuration.</li><li>Set the password for the Jupyter Notebook server.</li><li>Open a tunnel to the remote server using SSH port forwarding.</li><li>Start the Jupyter Notebook server and open it in a web browser.</li><li>Wait for the user to exit the Jupyter Notebook.</li><li>Kill the SSH tunnel process and close the SSH connection.</li></ol><p>Automating SSH login and Jupyter Notebook setup for machine learning projects is a great way to save time and increase productivity. It allows you to focus on developing your machine learning models instead of spending time on server setup and configuration. In this article, we have discussed how to automate SSH login and Jupyter Notebook setup using Python. By following these steps, you can easily set up your machine learning project on a remote server and start developing your models.</p><pre>import sys<br>import paramiko<br>import os<br>import subprocess<br>import webbrowser<br><br># Get the SSH ID, password, Python version, Conda flag, and venv flag as command line arguments<br>if len(sys.argv) != 6:<br>    print(&quot;Usage: python ssh_login.py &lt;ssh_id&gt; &lt;password&gt; &lt;python_version&gt; &lt;conda&gt; &lt;venv&gt;&quot;)<br>    sys.exit(1)<br>ssh_id = sys.argv[1]<br>password = sys.argv[2]<br>python_version = sys.argv[3]<br>conda_flag = sys.argv[4].lower() == &#39;true&#39;<br>venv_flag = sys.argv[5].lower() == &#39;true&#39;<br><br># Check if paramiko is installed and install it if it is not installed<br>try:<br>    import paramiko<br>except ImportError:<br>    print(&quot;Paramiko is not installed. Installing Paramiko...&quot;)<br>    os.system(&quot;pip install paramiko&quot;)<br><br># Create an SSH client object<br>ssh = paramiko.SSHClient()<br><br># Set the policy for the client object to auto add the hostname and key to known_hosts file<br>ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())<br><br># Connect to the SSH server using the provided hostname, port, username, and password<br>try:<br>    ssh.connect(hostname=&#39;&lt;hostname&gt;&#39;, port=&lt;port&gt;, username=ssh_id, password=password)<br>    print(&quot;Successfully logged in to the SSH server!&quot;)<br>except paramiko.AuthenticationException:<br>    print(&quot;Authentication failed. Please check your credentials.&quot;)<br>    sys.exit(1)<br><br># Check if Conda is installed on the remote server, and install it if it does not exist<br>if conda_flag:<br>    conda_check_cmd = &quot;conda --version&quot;<br>    stdin, stdout, stderr = ssh.exec_command(conda_check_cmd)<br>    output = stdout.read().decode()<br>    if &quot;conda: command not found&quot; in output:<br>        print(&quot;Conda is not installed on the remote server. Installing Conda...&quot;)<br>        conda_install_cmd = &quot;wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh &amp;&amp; bash ~/miniconda.sh -b -p $HOME/miniconda&quot;<br>        stdin, stdout, stderr = ssh.exec_command(conda_install_cmd)<br>        print(stdout.read().decode())<br>    else:<br>        print(&quot;Conda is already installed on the remote server.&quot;)<br><br>    # Install nb_conda<br>    nb_conda_install_cmd = &quot;conda install nb_conda -y&quot;<br>    stdin, stdout, stderr = ssh.exec_command(nb_conda_install_cmd)<br>    print(stdout.read().decode())<br><br># Create a new Conda environment with the specified Python version, if the Conda flag is set to True<br>if conda_flag:<br>    conda_cmd = f&quot;conda create -n myenv python={python_version}&quot;<br>    stdin, stdout, stderr = ssh.exec_command(conda_cmd)<br>    print(stdout.read().decode())<br><br>    # Activate the new Conda environment<br>    activate_cmd = &quot;conda activate myenv&quot;<br>    stdin, stdout, stderr = ssh.exec_command(activate_cmd)<br><br># Create a new Python virtual environment using venv, if the venv flag is set to True<br>if venv_flag:<br>    venv_cmd = f&quot;python{python_version} -m venv myenv&quot;<br>    stdin, stdout, stderr = ssh.exec_command(venv_cmd)<br>    print(stdout.read().decode())<br><br>    # Activate the new virtual environment<br>    activate_cmd = &quot;source myenv/bin/activate&quot;<br>    stdin, stdout, stderr = ssh.exec_command(activate_cmd)<br><br># Install Jupyter Notebook and create a new configuration<br>install_jupyter_cmd = &quot;pip install jupyter&quot;<br><br>stdin, stdout, stderr = ssh.exec_command(install_jupyter_cmd)<br>print(stdout.read().decode())<br><br># Generate a new configuration file for Jupyter Notebook<br>generate_config_cmd = &quot;jupyter notebook --generate-config&quot;<br>stdin, stdout, stderr = ssh.exec_command(generate_config_cmd)<br>print(stdout.read().decode())<br><br># Set the password for the Jupyter Notebook server<br>password_hash = subprocess.check_output([&#39;python&#39;, &#39;-c&#39;, &quot;from notebook.auth import passwd; print(passwd(&#39;&quot; + password + &quot;&#39;))&quot;]).decode().strip()<br><br>config_file = os.path.expanduser(&quot;~/.jupyter/jupyter_notebook_config.py&quot;)<br>with open(config_file, &#39;a&#39;) as f:<br>    f.write(&quot;\n&quot;)<br>    f.write(&quot;# Set the password for the Jupyter Notebook server\n&quot;)<br>    f.write(f&quot;c.NotebookApp.password = &#39;{password_hash}&#39;\n&quot;)<br><br># Open a tunnel to the remote server using SSH port forwarding<br>localhost = &#39;localhost&#39;<br>port = 8888<br>remote_port = 8888<br><br>tunnel_cmd = f&quot;ssh -N -L {port}:{localhost}:{remote_port} {ssh_id}@&lt;hostname&gt;&quot;<br><br># Start the SSH tunnel process<br>tunnel_proc = subprocess.Popen(tunnel_cmd, shell=True)<br><br># Start the Jupyter Notebook server and open it in a web browser<br>start_jupyter_cmd = f&quot;jupyter notebook --no-browser --port={remote_port}&quot;<br>stdin, stdout, stderr = ssh.exec_command(start_jupyter_cmd)<br>print(stdout.read().decode())<br><br>webbrowser.open_new_tab(f&#39;http://{localhost}:{port}/&#39;)<br><br># Wait for the user to exit the Jupyter Notebook<br>print(&quot;Press ENTER to close the SSH tunnel and exit the program.&quot;)<br>input()<br><br># Kill the SSH tunnel process<br>tunnel_proc.kill()<br><br># Close the SSH connection<br>ssh.close()</pre><p>It can be challenging to train and test machine learning models on a single computer due to the limited computational resources. Therefore, it is often necessary to use remote servers or cloud computing platforms to run machine learning experiments. In this article, we explored a Python script that facilitates logging into a remote server via SSH, setting up a machine learning environment, and running Jupyter Notebook.</p><p>The code above starts with importing several libraries, including paramiko, os, subprocess, and webbrowser. The paramiko library is a Python implementation of the SSH protocol and can be used to create an SSH client object. The os library provides a way to interact with the operating system, and the subprocess library can be used to execute shell commands from Python. Finally, the webbrowser library provides an interface to interact with web browsers.</p><p>The next section of the code gets the command-line arguments for the SSH ID, password, Python version, Conda flag, and venv flag. The Conda flag is used to specify whether Conda should be installed, and the venv flag is used to specify whether a Python virtual environment should be created. The argparse library can also be used to parse command-line arguments in Python.</p><p>After getting the command-line arguments, the code checks if the paramiko library is installed and installs it if it is not installed. This is done using the os.system() method to execute the pip install command. Once the paramiko library is installed, an SSH client object is created using the paramiko.SSHClient() method.</p><p>The next step is to set the policy for the client object to auto add the hostname and key to known_hosts file using the ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) method. This ensures that the client object trusts the remote server and does not prompt the user to manually verify the server’s fingerprint.</p><p>After setting the policy, the code connects to the remote server using the ssh.connect() method. The hostname, port, SSH ID, and password are provided as arguments. The code also catches the paramiko.AuthenticationException exception to handle authentication errors gracefully.</p><p>If the Conda flag is set to True, the code checks if Conda is installed on the remote server using the conda — version command. If Conda is not installed, the code downloads and installs the latest version of Miniconda using the wget command and the bash command to execute the installation script. The nb_conda package is also installed using the conda install nb_conda -y command.</p><p>If the Conda flag is set to True, the code creates a new Conda environment using the specified Python version using the conda create -n myenv python={python_version} command. The environment is activated using the conda activate myenv command. If the venv flag is set to True, the code creates a new Python virtual environment using the python -m venv myenv command and activates it using the source myenv/bin/activate command.</p><p>The code then installs Jupyter Notebook using the pip install jupyter command and generates a new configuration file using the jupyter notebook — generate-config command. The password for the Jupyter Notebook server is set using the notebook.auth.passwd() method, which generates a SHA-1 hash of the password. The hash is added to the jupyter_notebook_config.py file using the f.write() method.</p><p>The code then opens an SSH tunnel to the remote server using port forwarding using the ssh -N -L {port}:{localhost}:{remote_port} command. The tunnel is started using the subprocess.Popen() method, which executes the command in a new process.</p><p>Once the tunnel is open, the code starts the Jupyter Notebook server on the remote server with the start_jupyter_cmd command. The --no-browser option is used to start the server without opening a browser window. Instead, the code opens a new tab in the default web browser of the user with the webbrowser module. The URL of the Jupyter Notebook server is <a href="http://localhost:8888/">http://localhost:8888/.</a></p><p>After starting the Jupyter Notebook server, the code prompts the user to press ENTER to close the SSH tunnel and exit the program. Once the user presses ENTER, the code kills the SSH tunnel process and closes the SSH connection with the tunnel_proc.kill() and ssh.close() commands, respectively.</p><p>This code is a powerful tool for machine learning professionals who need to work with remote servers to train and deploy machine learning models. By automating the setup of a remote machine, installing the necessary software, and starting a Jupyter Notebook server, this code saves valuable time and resources that can be better spent on developing and improving machine learning models.</p><h4>Limitations</h4><p>The code performs various tasks to automate the setup of a Jupyter Notebook server on a remote server using SSH. However, there are several limitations and potential areas for improvement in the code:</p><ol><li>Lack of input validation: The code assumes that the command-line arguments and inputs provided by the user are valid, which can lead to unexpected errors or security issues. For example, the code does not check if the provided hostname or port are valid or if the provided password is strong enough.</li><li>Limited error handling: The code does not have robust error handling mechanisms to handle unexpected errors or exceptions that may occur during the execution of the script. For instance, the code assumes that the SSH connection will always be successful, but it does not handle cases where the connection fails due to network issues or incorrect credentials.</li><li>Security concerns: The code uses a plaintext password to authenticate with the remote server, which is not secure. Additionally, the code generates and stores the Jupyter Notebook password in plaintext in the configuration file, which is also a security concern.</li><li>Compatibility issues: The code assumes that the remote server is running a Linux-based operating system and that the user has administrative privileges to install packages and create environments.</li></ol><p>To improve the code, here are some suggestions which can be incorporated in the next version:</p><ol><li>Add input validation: The code should validate user inputs and command-line arguments to prevent unexpected errors or security issues. For example, the code can use regular expressions to validate the hostname, port, and password inputs.</li><li>Improve error handling: The code should have robust error handling mechanisms to handle unexpected errors or exceptions that may occur during the execution of the script. For instance, the code can use try-except blocks to catch and handle exceptions that may occur during SSH connection or package installation.</li><li>Enhance security: The code can use more secure methods for password authentication, such as SSH keys or two-factor authentication. Additionally, the code can use a secure method to store the Jupyter Notebook password, such as the keyring module.</li><li>Improve compatibility: The code can check the remote server’s operating system and the user’s privileges before attempting to install packages or create environments. Additionally, the code can use platform-independent package managers such as conda or pipenv instead of system-specific package managers.</li><li>Add logging: The code can add logging statements to track the progress and errors during the script’s execution. This can help in debugging and troubleshooting issues that may occur during the script’s execution.</li><li>Use command-line arguments for all inputs: Currently, the code uses command-line arguments for the SSH ID, password, Python version, Conda flag, and venv flag. However, the hostname and port are hard-coded in the code. It would be better to use command-line arguments for all inputs so that the user can provide all the necessary information when running the script.</li><li>Use context managers for SSH connection: The code currently uses the connect method of the SSHClient class to establish an SSH connection, but it does not use a context manager to ensure that the connection is properly closed. It would be better to use a with statement to create a context manager that automatically closes the connection when the code inside the context is finished.</li><li>Use subprocess instead of os.system: The os.system function is deprecated and should be replaced with the subprocess module, which provides a more flexible and secure way to run external commands.</li><li>Use f-strings for string interpolation: The code currently uses string concatenation and string formatting with the % operator. It would be better to use f-strings, which are more readable and less error-prone.</li><li>Use more descriptive variable names: Some of the variable names in the code are not very descriptive, which makes it harder to understand what the code is doing. For example, the variable stdin is used to store the standard input stream of an SSH command, which is not very clear. It would be better to use more descriptive names like stdin_stream or input_stream.</li></ol><p>I have tried below to incorporate some of them. This is still experiment please use it with caution.</p><pre>def check_and_install_package(package):<br>    &quot;&quot;&quot;<br>    Check if a package is installed and install it if it is not installed<br>    &quot;&quot;&quot;<br>    try:<br>        __import__(package)<br>    except ImportError:<br>        print(f&quot;{package} is not installed. Installing {package}...&quot;)<br>        subprocess.run([&quot;pip&quot;, &quot;install&quot;, package])</pre><pre>def ssh_login(hostname, port, ssh_key, python_version, use_conda=False, use_venv=False):<br>    &quot;&quot;&quot;<br>    Connects to a remote server using SSH and executes Python code.<br>    :param hostname: Hostname of the remote server.<br>    :param port: Port number to connect to the remote server.<br>    :param ssh_key: Path to the SSH private key file.<br>    :param python_version: Version of Python to use on the remote server.<br>    :param use_conda: Flag indicating whether to use conda as the Python environment manager.<br>    :param use_venv: Flag indicating whether to use virtual environments as the Python environment manager.<br>    &quot;&quot;&quot;<br>    install_package(&quot;paramiko&quot;)<br><br>    # Create an SSH client object<br>    ssh = paramiko.SSHClient()<br><br>    # Set the policy for the client object to auto add the hostname and key to known_hosts file<br>    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())<br><br>    # Load the private key file<br>    ssh_key = paramiko.RSAKey.from_private_key_file(ssh_key)<br><br>    try:<br>        # Connect to the SSH server using the provided hostname, port, and SSH key<br>        ssh.connect(hostname, port=port, username=os.getlogin(), pkey=ssh_key)<br><br>        # Execute the Python code on the remote server<br>        command = f&quot;python{python_version}&quot;<br>        if use_conda:<br>            command = f&quot;conda run -n my_env {command}&quot;<br>        elif use_venv:<br>            command = f&quot;source my_env/bin/activate &amp;&amp; {command}&quot;<br>        stdin, stdout, stderr = ssh.exec_command(command)<br><br>        # Print the output of the command<br>        for line in stdout:<br>            print(line.strip())<br><br>        # Print the errors, if any<br>        for line in stderr:<br>            print(line.strip())<br><br>    except Exception as e:<br>        print(f&quot;Error: {e}&quot;)<br>    finally:<br>        # Close the SSH connection<br>        ssh.close()</pre><pre>if __name__ == &quot;__main__&quot;:<br>    if len(sys.argv) != 7:<br>        print(&quot;Usage: python ssh_login.py &lt;hostname&gt; &lt;port&gt; &lt;ssh_key&gt; &lt;python_version&gt; &lt;use_conda&gt; &lt;use_venv&gt;&quot;)<br>        sys.exit(1)<br><br>    hostname = sys.argv[1]<br>    port = int(sys.argv[2])<br>    ssh_key = sys.argv[3]<br>    python_version = sys.argv[4]<br>    use_conda = sys.argv[5].lower() == &quot;true&quot;<br>    use_venv = sys.argv[6].lower() == &quot;true&quot;<br><br>    ssh_login(hostname, port, ssh_key, python_version, use_conda, use_venv)</pre><pre>import argparse<br>import subprocess<br><br># Check if paramiko is installed<br>try:<br>    import paramiko<br>except ImportError:<br>    # Install paramiko using pip<br>    subprocess.check_call([&#39;pip&#39;, &#39;install&#39;, &#39;paramiko&#39;])<br>    import paramiko<br><br># Prompt the user for input<br>hostname = input(&quot;Enter the hostname: &quot;)<br>username = input(&quot;Enter the username: &quot;)<br>password = input(&quot;Enter the password: &quot;)<br><br># Create an SSH client object<br>ssh = paramiko.SSHClient()<br><br># Automatically add the host key<br>ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())<br><br># Attempt to connect to the SSH server<br>try:<br>    ssh.connect(hostname=hostname, username=username, password=password)<br>except paramiko.ssh_exception.AuthenticationException:<br>    print(&quot;Authentication failed. Please check your username and password.&quot;)<br>except paramiko.ssh_exception.NoValidConnectionsError:<br>    print(&quot;Unable to connect to the server. Please check the hostname.&quot;)<br>except Exception as e:<br>    print(f&quot;An error occurred while connecting to the server: {e}&quot;)<br>else:<br>    print(&quot;Successfully connected to the server.&quot;)<br><br>import subprocess<br><br># Set up SSH tunnel<br>local_port = 8888<br>remote_port = 8888<br>transport = paramiko.Transport((hostname, 22))<br>transport.connect(username=username, password=password)<br>local_socket = (&#39;localhost&#39;, local_port)<br>remote_socket = (&#39;localhost&#39;, remote_port)<br>transport.request_port_forward(*remote_socket)<br><br># Start Jupyter Notebook server on remote server<br>command = f&quot;jupyter notebook --no-browser --port {remote_port}&quot;<br>ssh = transport.open_session()<br>ssh.exec_command(command)<br><br># run this command in the terminal &quot;ssh -N -L localhost:&lt;local-port&gt;:localhost:&lt;remote-port&gt; &lt;remote-user&gt;@&lt;remote-host&gt;&quot;<br>subprocess.Popen([&quot;ssh&quot;, &quot;-N&quot;, &quot;-L&quot;, f&quot;localhost:{8888}:localhost:{8888}&quot;, f&quot;{username}@{hostname}&quot;])<br><br># Open browser window to connect to Jupyter Notebook server<br>import webbrowser<br>url = f&quot;http://localhost:{local_port}&quot;<br>webbrowser.open_new(url)<br><br># Wait for user to close browser window<br>input(&quot;Press Enter to close the SSH connection and shut down the Jupyter Notebook server...&quot;)<br><br># Shut down Jupyter Notebook server<br>ssh.exec_command(&quot;pkill -f jupyter&quot;)<br>ssh.close()<br><br># Close SSH tunnel<br>transport.close()</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd2967a832d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real time Object tracking and Segmentation using YoloV8 with Strongsort, Ocsort and Bytetrack]]></title>
            <link>https://siddharthksah.medium.com/real-time-object-tracking-and-segmentation-using-yolov8-with-strongsort-ocsort-and-bytetrack-180eef43354a?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/180eef43354a</guid>
            <category><![CDATA[realtime-video-processing]]></category>
            <category><![CDATA[multiple-object-tracking]]></category>
            <category><![CDATA[yolov8]]></category>
            <category><![CDATA[object-segmentation]]></category>
            <category><![CDATA[object-detection]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Thu, 09 Feb 2023 07:33:40 GMT</pubDate>
            <atom:updated>2023-06-08T16:03:57.052Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*EEfntv-LLPWDubqpEmOrRw.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*2TdhzwMgw6XwPXdRdbJuTQ.gif" /></figure><p>The goal of object tracking is to keep track of an object as it moves through the frame and to locate it in subsequent frames.</p><p>There are various methods for object tracking, including:</p><ul><li>Feature-based tracking: This method involves tracking an object based on its features, such as colour, shape, or texture.</li><li>Template matching: This method involves matching a pre-defined template to each frame in the video sequence.</li><li>Correlation-based tracking: This method involves computing the similarity between the target object and candidate regions in subsequent frames.</li><li>Deep learning-based tracking: This method uses neural networks trained on large datasets to detect and track objects in real-time.</li></ul><p>Object tracking has many potential applications, including:</p><ul><li>Video surveillance: Object tracking can be used in security systems to track objects of interest, such as vehicles or people, in real-time.</li><li>Human-computer interaction: Object tracking can be used to detect and track body movements in order to enable gesture recognition.</li><li>Robotics: Object tracking can be used to enable robots to track and follow objects, such as people or objects of interest.</li><li>Sports analysis: Object tracking can be used to track athletes and analyse their performance in sports such as soccer or basketball.</li><li>Automated driving: Object tracking can be used in autonomous vehicles to track other vehicles, pedestrians, and road signs in real-time.</li></ul><p>The tracking is divided into two parts. The first one is detecting the object and the second one is using an algorithm to track that object in subsequent frames.</p><p>Object detection using YOLO</p><p>YOLO (You Only Look Once) is a popular object detection algorithm that uses a convolutional neural network (CNN) to detect objects in images and videos. YOLO was developed to be fast and efficient, and it can detect objects in real-time on a standard computer.</p><p>The YOLO algorithm works by dividing an image into a grid of cells, and each cell is responsible for predicting the presence of objects within it. If an object spans multiple cells, each cell responsible for the object predicts the presence of the object. The YOLO network then predicts the bounding box coordinates and class probabilities for each object in the image.</p><p>One of the main advantages of YOLO is its speed. YOLO uses a single CNN to make predictions for an entire image, which makes it faster than other object detection algorithms that use multiple CNNs or a sliding window approach. Additionally, YOLO uses anchor boxes to handle the problem of object aspect ratios, which helps it to detect objects with varying shapes.</p><p>YOLO has been trained on a large dataset and can detect a wide range of objects, including people, cars, animals, and more. YOLO is commonly used for a variety of applications, including object tracking, video surveillance, and autonomous vehicles.</p><p>YOLO is a powerful object detection algorithm that is fast, efficient, and capable of detecting a wide range of objects. If you’re interested in using YOLO for object detection, there are several pre-trained models available that you can use, or you can train your own model on your own dataset.</p><p>We will be using yolov8.</p><p>YOLOv8 is the latest version of the YOLO (You Only Look Once) object detection algorithm. It builds on the previous versions of YOLO and has several new features and improvements.</p><p>Here are some of the new features and improvements in YOLOv8:</p><ol><li>Improved architecture: YOLOv8 has a new architecture that is designed to be more efficient and accurate than previous versions. This includes a new backbone network and a more effective use of anchor boxes.</li><li>Better object detection: YOLOv8 includes improved object detection capabilities, with a higher accuracy and improved ability to detect small objects.</li><li>Increased speed: YOLOv8 is faster than previous versions, making it suitable for real-time applications such as video surveillance and autonomous vehicles.</li><li>Enhanced performance on various platforms: YOLOv8 has been optimized for various platforms, including GPUs, CPUs, and mobile devices, making it more versatile and accessible for a wider range of applications.</li><li>Improved data augmentation: YOLOv8 includes improved data augmentation techniques to increase the robustness of the model and prevent overfitting.</li></ol><p>YOLOv8 is a significant improvement over previous versions of YOLO and represents the state-of-the-art in real-time object detection. If you’re interested in using YOLOv8 for object detection, there are several pre-trained models available, or you can train your own model on your own dataset.</p><p>We have 3 different types of trackers namely StrongSORT OSNet, OCSORT and ByteTrack.</p><p>StrongSORT is an object tracking algorithm that is based on a combination of deep neural networks and traditional computer vision techniques. StrongSORT was developed to overcome some of the limitations of other object tracking algorithms and to provide improved accuracy and robustness in challenging tracking scenarios.</p><p>One of the key features of StrongSORT is its use of deep neural networks to extract features from an image that are robust to appearance changes, such as changes in illumination, viewpoint, and occlusion. StrongSORT also uses a Kalman filter to model the motion of the target object and to estimate its position in subsequent frames.</p><p>Another advantage of StrongSORT is its ability to handle occlusions and partial occlusions, which can be a challenging problem for other object tracking algorithms. StrongSORT uses a combination of feature matching and Kalman filtering to handle occlusions and estimate the position of the target object even when it is partially occluded.</p><p>StrongSORT is a powerful object tracking algorithm that provides improved accuracy and robustness in challenging tracking scenarios. If you’re interested in using StrongSORT for object tracking, there are several implementations available, or you can implement your own version based on the published research.</p><p>OCSORT (Online Continuous-Time Object Tracking) is an online object tracking algorithm that is designed to track objects in real-time. It is designed to handle challenging tracking scenarios such as occlusions and changes in object appearance.</p><p>One of the key advantages of OCSORT is its real-time performance. OCSORT is designed to track objects in real-time, even in complex and cluttered scenes, and it is capable of handling high-frame rate videos.</p><p>OCSORT is a powerful object tracking algorithm that provides real-time performance and high accuracy in challenging tracking scenarios.</p><p>ByteTrack was developed to provide improved accuracy and robustness in object tracking, and it has been applied to a wide range of tracking scenarios, including person tracking, vehicle tracking, and drone tracking.</p><p>One of the key advantages of ByteTrack is its use of reinforcement learning to train a deep neural network to perform object tracking. Another advantage of ByteTrack is its ability to track objects in real-time. ByteTrack is designed to run in real-time on a standard computer or even on mobile devices, and it has been shown to achieve high frame rates and low latency in a range of tracking scenarios.</p><p>ByteTrack is a powerful object tracking algorithm that provides improved accuracy and robustness in challenging tracking scenarios. If you’re interested in using ByteTrack for object tracking, there are several implementations available, or you can implement your own version based on the published research.</p><p>Let’s talk code.</p><pre>conda create -n tracking python=3.7 -y<br>conda activate tracking<br>git clone --recurse-submodules https://github.com/mikel-brostrom/yolov8_tracking.git  # clone recursively<br>cd yolov8_tracking<br>pip install --upgrade pip<br>pip install -r requirements.txt  # install dependencies<br><br>#object detection and tracking<br>python3 track.py --source output.mov --yolo-weights yolov8s.pt --save-vid<br><br># object detection with segmentation and tracking<br>python3 track.py --source output.mov --yolo-weights yolov8s-seg.pt --save-vid<br><br># selecting the tracking method<br>python3 track.py --tracking-method strongsort --source 0<br>python3 track.py --tracking-method ocsort --source 0<br>python3 track.py --tracking-method bytetrack --source 0</pre><p>YoloV8 Re-identification weights</p><pre>python3 track.py --yolo-weights yolov8s.pt --reid-weights osnet_x0_25_msmt17.pt --source 0 --save-vid</pre><p>ReID (Re-Identification) weight is a term used in the context of object tracking algorithms. It refers to the weight assigned to the ReID (Re-Identification) loss term in the objective function of the tracking algorithm.</p><p>ReID loss is used in object tracking algorithms to ensure that the target object is accurately re-identified in subsequent frames of a video. The ReID loss term compares the appearance of the target object in the current frame to the appearance of the target object in previous frames. The ReID weight determines the importance of this term in the overall objective function of the tracking algorithm.</p><p>In some object tracking algorithms, the ReID weight is learned during training, while in others it is set based on prior knowledge or heuristics. The ReID weight can be critical to the performance of the object tracking algorithm, as it determines the balance between the importance of accurately re-identifying the target object and the importance of accurately estimating its motion.</p><p>In general, setting the ReID weight too low may result in poor re-identification performance, while setting it too high may result in poor motion estimation performance. The optimal ReID weight will depend on the specific scenario and requirements of the tracking problem.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=180eef43354a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synthetic training data from Blender + Object Detection with Transfer Learning = Deep Learning on…]]></title>
            <link>https://siddharthksah.medium.com/synthetic-training-data-object-detection-with-transfer-learning-deep-learning-on-steroids-e20f76bd4269?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/e20f76bd4269</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[image-generation]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Sun, 05 Feb 2023 06:36:07 GMT</pubDate>
            <atom:updated>2023-02-05T12:48:19.849Z</atom:updated>
            <content:encoded><![CDATA[<h3>Synthetic training data from Blender + Object Detection with Transfer Learning = Deep Learning on Steroids!</h3><p>A deep learning model is only as good as its data. If the data has inherent biases, the model will reflect them. The fine-tuning of models still has the original biases.</p><p>Collecting non-biased data is next to impossible; it is often woven into the very fabric of the data acquisition and generation.</p><p>Bias in training data is when a machine learning model treats different groups of people in different ways based on things like race, gender, or socioeconomic status. This happens when the training data used to build the model shows that bias and discrimination are common in society as a whole. Because of this, the model can keep and amplify these biases in the decisions and predictions it makes. For example, a biased training set for facial recognition could mean that people with darker skin tones will make more mistakes. It&#39;s important to deal with bias in training data to make sure that machine learning models are fair and reliable.</p><p>There are several ways to address biases in deep learning models:</p><ol><li>Diverse training data: Use a diverse and representative training dataset to ensure that the model is exposed to a range of examples from different groups and does not over-represent or under-represent any particular group.</li><li>Fairness constraints: Apply mathematical constraints during the training process to ensure that the model makes fair predictions and decisions, such as equal opportunity or equal accuracy across different groups.</li><li>De-biasing algorithms: Use algorithms that specifically target and remove bias from the model, such as adversarial training or re-sampling methods.</li><li>Human review: Incorporate human review into the decision-making process to catch and correct any biases that may emerge.</li><li>Monitoring and evaluation: Regularly monitor and evaluate the model’s performance and decision-making to identify and address any potential biases.</li></ol><p>Apart from biases, deep learning models also have major data privacy concerns surrounding them.</p><p>Data privacy is a major concern with deep learning models because they often use large amounts of personal data to train and make predictions. This personal data can include sensitive information like financial records, medical records, or biometric data. If this information is not kept safe or is used in an unethical way, it can cause serious privacy violations and hurt people.</p><p>To address data privacy concerns in deep learning, several techniques can be used, such as:</p><ol><li>Data anonymisation: Remove or mask personal identifiable information from the data used for training.</li><li>Data encryption: Encrypt sensitive data to protect it from unauthorised access.</li><li>Data aggregation: Combine multiple data sources into a single, aggregate representation to reduce the risk of re-identification.</li><li>Access control: Implement strict access controls to limit who can access the data and under what circumstances.</li><li>Privacy preserving machine learning: Use privacy-preserving techniques such as differential privacy or federated learning to train models without compromising the privacy of individuals.</li></ol><p>Synthetic data helps solve these problems. If implemented well, it can dampen both the biases and the data security concerns.</p><p>Let’s talk about adversarial attacks on deep learning models for a second. They are part of the data privacy concern. We can reverse-engineer our way to the data from the model it was trained on from only the model weights. Google came up with decentralised training called Federated Learning to solve these issues. In that case, sensitive data never leaves the original device.</p><p>There are several types of adversarial attacks, including:</p><ul><li>Poisoning attacks: Injecting malicious data into the training set to cause the model to behave in a specific way when given similar inputs.</li><li>Evasion attacks: Modifying input data at test time in order to trick the model into making a wrong prediction.</li><li>Exploratory attacks: Attempting to find inputs that are misclassified by the model and understanding why this is the case.</li></ul><p>Trained models can have backdoors too.</p><p>Alright, let’s talk some code now. In this article, we are going to create our own synthetic training data and use transfer learning to build an object detection model.</p><p>Let’s take an example. We have a factory that manufactures beverages. The factory recently came up with a new types of beverages. We want to classify them on the production floor so that we can pack them in different boxes. Now because we do not have any training data, we can wait to take photos for a few days and then sort them, but that is not feasible financially. How about we create the data? We have access to the 3D model of the bottles (Usually not a bottleneck). If we render them in different scenarios and use that as training data, it should work, right? Let’s see.</p><p>There are ways to generate synthetic data from Blender using Python. For this example, I am going to stick to open source datasets.</p><p><a href="https://www.kaggle.com/datasets/vencerlanz09/bottle-synthetic-images-dataset">https://www.kaggle.com/datasets/vencerlanz09/bottle-synthetic-images-dataset</a></p><p>Let’s look at some synthetic data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*vBt7DUjaXZRB3DPwBaiN1g.jpeg" /><figcaption>Class I — Beer Bottle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*KznvDvsJ4xfIwTiBD4OGpg.jpeg" /><figcaption>Class II — Plastic Bottle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*JsLiVa8Xu4Y0n-N-Y1y7Bg.jpeg" /><figcaption>Type III — Soda Bottle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*l-iuR4RJ_r4VbPkOYJeBsQ.jpeg" /><figcaption>Type IV — Water Bottle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*cgiWNLABu077eY0bU-7PYg.jpeg" /><figcaption>Type V — Wine Bottle</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ptef-9U1U1p3u07AwN5Wvg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjkPfQM9RSPd6jRkJVE10A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8tS8b8F0iTqGH7RntM5P7g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7OJyiFTGw5ioushS8SLLA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFo8cD_WT4_y-yzIvYag3A.jpeg" /></figure><p>A collage of five different classes. We have 5000 images per class. Some classes have 512X512 images and some 300X300 — we will handle that in the training process.</p><p>I will be using yolov5 to train my custom dataset and Weights &amp; Biases to track the experiments.</p><p>The original dataset has the annotations in COCO JSON Format so we will need to convert it to YOLO TXT Format. I will be using <a href="https://github.com/pylabel-project/samples">https://github.com/pylabel-project/samples</a> for this.</p><pre>!pip install pylabel<br>from pylabel import importer<br><br># Copy images_raw to working directory<br># Note: This may take some time depending on the size of your images_raw folder<br>!cp -r ./input/bottle-synthetic-images-dataset/ImageClassesCombinedWithCOCOAnnotations/images_raw ./<br><br># Copy annotations to working directory<br>!cp -r ./input/bottle-synthetic-images-dataset/ImageClassesCombinedWithCOCOAnnotations/coco_instances.json ./<br><br># Copy test image to output directory<br>!cp -r ./input/bottle-synthetic-images-dataset/ImageClassesCombinedWithCOCOAnnotations/test_image.jpg ./<br><br>#Specify path to the coco.json file<br>path_to_annotations = r&quot;./coco_instances.json&quot;<br>#Specify the path to the images (if they are in a different folder than the annotations)<br>path_to_images = r&quot;./images_raw&quot;<br><br>#Import the dataset into the pylable schema <br>dataset = importer.ImportCoco(path_to_annotations, path_to_images=path_to_images, name=&quot;BCCD_coco&quot;)<br>dataset.df.head(5)<br><br>print(f&quot;Number of images: {dataset.analyze.num_images}&quot;)<br>print(f&quot;Number of classes: {dataset.analyze.num_classes}&quot;)<br>print(f&quot;Classes:{dataset.analyze.classes}&quot;)<br>print(f&quot;Class counts:\n{dataset.analyze.class_counts}&quot;)<br>print(f&quot;Path to annotations:\n{dataset.path_to_annotations}&quot;)<br><br>try:<br>    display(dataset.visualize.ShowBoundingBoxes(2))<br>    display(dataset.visualize.ShowBoundingBoxes(&quot;./images_raw/00000002.jpg&quot;))<br>except:<br>    pass<br><br># This cell may take some time depending on the size of the dataset.<br>dataset.path_to_annotations = &quot;labels&quot;<br>dataset.export.ExportToYoloV5(output_path=&#39;text_files&#39;);<br><br># Note!!! Only run this code once<br># this will change the start of class numbers from 1 to 0<br>path = &#39;./text_files&#39; #path of labels<br>labels = os.listdir(path)<br>for x in labels:<br>    lines = list()<br>    with open(path+&quot;/&quot;+x, &quot;r+&quot;) as f:<br>        for line in f.read().splitlines():<br>            split_line = line.split(&quot; &quot;)  # split on space character (and remove newline characters as well)<br>            split_line[0] = str(<br>              int(split_line[0]) - 1)  # update the value inside the loop. the loop used in later not needed.<br>            lines.append(split_line)  # add split list into list of lines<br><br>    with open(path+&quot;/&quot;+x, &#39;w&#39;) as file:  # rewrite to file<br>        for line in lines:<br>            write_me = &#39; &#39;.join(line)  # Use join method to add the element together<br>            file.write(write_me + &quot;\n&quot;)</pre><p>Now that we have the data ready in YOLO TXT format, We&#39;ll divide it into three parts: training, testing, and validation.</p><pre># Read images and annotations<br>image_dir = r&#39;./images_raw&#39;<br>images = [os.path.join(image_dir, x) for x in os.listdir(image_dir)]<br>annotations = [os.path.join(&#39;./text_files&#39;, x) for x in os.listdir(&#39;./text_files&#39;) if x[-3:] == &quot;txt&quot;]<br><br>images.sort()<br>annotations.sort()<br><br># Split the dataset into train-valid-test splits <br>train_images, val_images, train_annotations, val_annotations = train_test_split(images, annotations, test_size = 0.2, random_state = 1)<br>val_images, test_images, val_annotations, test_annotations = train_test_split(val_images, val_annotations, test_size = 0.5, random_state = 1)<br><br>len(train_images),len(train_annotations)<br><br># yolov5 expects annotations and images in directories in a specific format<br># let&#39;s first create those directories and then move the files into them<br>!mkdir images<br>!mkdir annotations<br>!mkdir images/train images/val images/test annotations/train annotations/val annotations/test<br><br>#Utility function to move images <br>def move_files_to_folder(list_of_files, destination_folder):<br>    for f in list_of_files:<br>        try:<br>            shutil.move(f, destination_folder)<br>        except:<br>            print(f)<br>            assert False<br><br># Move the splits into their folders<br>move_files_to_folder(train_images, &#39;images/train&#39;)<br>move_files_to_folder(val_images, &#39;images/val/&#39;)<br>move_files_to_folder(test_images, &#39;images/test/&#39;)<br>move_files_to_folder(train_annotations, &#39;annotations/train/&#39;)<br>move_files_to_folder(val_annotations, &#39;annotations/val/&#39;)<br>move_files_to_folder(test_annotations, &#39;annotations/test/&#39;)<br><br>!mv annotations labels<br>shutil.move(&quot;./images&quot;, &quot;./yolov5&quot;)<br>shutil.move(&quot;./labels&quot;, &quot;./yolov5&quot;)<br><br># Yolov5 has a dataset.yaml file which containes the directories of the training, test and validation data<br># Since we moved the files we will update the path in the yaml too<br><br># Viewing the original unprocessed yaml file<br># yaml_params = {}<br># with open(r&#39;dataset.yaml&#39;) as file:<br>#    # The FullLoader parameter handles the conversion from YAML<br>#    # scalar values to Python the dictionary format<br>#    yaml_file_list = yaml.load(file, Loader=yaml.FullLoader)<br>#    yaml_params = yaml_file_list<br>#    print(yaml_file_list)<br><br># Adjusting the parameters of the yaml file<br>yaml_params[&#39;path&#39;] = &#39;images&#39;<br>yaml_params[&#39;train&#39;] = &#39;train&#39;<br>yaml_params[&#39;val&#39;] = &#39;val&#39;<br>yaml_params[&#39;test&#39;] = &#39;test&#39;<br>yaml_params<br><br># Overwriting the new params from the previous ones.<br>with open(r&#39;dataset.yaml&#39;, &#39;w&#39;) as file:<br>    documents = yaml.dump(yaml_params, file)<br><br># Moving the dataset.yaml inside the yolov5/data folder.<br>shutil.move(&quot;dataset.yaml&quot;, &quot;yolov5/data&quot;)<br><br>shutil.move(&quot;./test_image.jpg&quot;, &quot;./yolov5&quot;)<br><br># Change the current directory inisde the yolov5<br>%cd ./yolov5</pre><p>Let’s talk about the training now. These are the hyperparameters we will tuning for this example.</p><ol><li>Size of the Image</li></ol><p>2. Batch Size</p><p>3. Epochs</p><p>4. Workers</p><p>5. Model Architecture — There are four choices available: yolo5s.yaml, yolov5m.yaml, yolov5l.yaml, yolov5x.yaml.</p><pre>!python train.py --img 300 --cfg yolov5s.yaml --hyp hyp.scratch-low.yaml --batch 128 --epochs 100 --data dataset.yaml --weights yolov5s.pt --workers 24 --name yolo_bottle_det</pre><p>Running inference and testing if everything is ok.</p><pre>!python detect.py --source images/test --weights runs/train/yolo_bottle_det/weights/best.pt --conf 0.25 --name yolo_bottle_det<br><br>detections_dir = &quot;runs/detect/yolo_bottle_det/&quot;<br>detection_images = [os.path.join(detections_dir, x) for x in os.listdir(detections_dir)]<br><br>fig, axes = plt.subplots(nrows=4, ncols=4, figsize=(15, 15),<br>                        subplot_kw={&#39;xticks&#39;: [], &#39;yticks&#39;: []})<br><br>for i, ax in enumerate(axes.flat):<br>    random_detection_image = PIL.Image.open(random.choice(detection_images))<br>    ax.imshow(random_detection_image)</pre><p>Let’s make some predictions on real life data now.</p><pre>!python detect.py --source ./test_image.jpg --weights runs/train/yolo_bottle_det/weights/best.pt --conf 0.25 --name yolo_bottle_det<br><br>detections_dir = &quot;runs/detect/yolo_bottle_det2/&quot;<br>detection_images = [os.path.join(detections_dir, x) for x in os.listdir(detections_dir)]<br>random_detection_image = PIL.Image.open(random.choice(detection_images))<br>plt.figure(figsize=(30,30));<br>plt.imshow(random_detection_image)<br>plt.xticks([])<br>plt.yticks([]);</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e20f76bd4269" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A hitchhiker’s guide to Synthetic data for Deep Learning]]></title>
            <link>https://siddharthksah.medium.com/a-sorta-kinda-hitchhikers-guide-to-synthetic-data-ad722798af63?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad722798af63</guid>
            <category><![CDATA[text-to-image-generation]]></category>
            <category><![CDATA[synthetic-tabular-data]]></category>
            <category><![CDATA[image-synthesis]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Sat, 04 Feb 2023 11:51:33 GMT</pubDate>
            <atom:updated>2023-02-09T16:57:49.195Z</atom:updated>
            <content:encoded><![CDATA[<p>What’s synthetic data?</p><p>Synthetic data in machine learning refers to the creation of artificially generated data sets that mimic real-world data and can be used for training machine learning models. The purpose of using synthetic data is to overcome limitations in collecting and using real-world data for training, such as data privacy concerns, the unavailability of large enough data sets, or the need to artificially balance the data. Synthetic data can be generated using various methods, such as generative models, simulations, or statistical sampling techniques.</p><p>Why should we care?</p><p>Several reasons actually.</p><ol><li>Overcoming data privacy concerns: In some cases, it may not be possible to use real-world data due to privacy or confidentiality issues. Synthetic data can provide a solution to this problem by generating data that resembles the real-world data but doesn’t contain any sensitive information.</li><li>Balancing the data: In some cases, the real-world data may be imbalanced, meaning that there is a disproportionate number of samples for certain classes. Synthetic data can be generated to balance the data and improve the performance of the machine learning model.</li><li>Lack of data: In some cases, there may not be enough real-world data to train a machine learning model. Synthetic data can be generated to supplement the real-world data and increase the size of the data set.</li><li>Testing and validation: Synthetic data can also be used to test and validate machine learning models by providing a controlled environment for experimentation.</li><li>Domain adaptation: When the real-world data for a particular domain is limited, synthetic data can be generated from a related domain to help train the machine learning model.</li></ol><p>What kind of synthetic data can we really generate, though?</p><ol><li>Tabular Synthetic Data: This type of synthetic data refers to artificially generated data in a tabular format, such as a spreadsheet or a database table. It is commonly used for testing machine learning algorithms and addressing class imbalance in datasets.</li><li>Image Synthetic Data: This type of synthetic data refers to artificially generated images, used for training machine learning models in computer vision and object detection tasks.</li><li>Text Synthetic Data: This type of synthetic data refers to artificially generated text, used for training machine learning models in natural language processing and text classification tasks.</li><li>Audio Synthetic Data: This type of synthetic data refers to artificially generated audio, used for training machine learning models in audio processing and speech recognition tasks.</li><li>Time Series Synthetic Data: This type of synthetic data refers to artificially generated time series data, used for training machine learning models in forecasting and time series analysis tasks.</li><li>Spatial Synthetic Data: This type of synthetic data refers to artificially generated spatial data, used for training machine learning models in geographic information systems and spatial analysis tasks.</li><li>Video Synthetic Data: This type of synthetic data refers to artificially generated video, used for training machine learning models in video processing and object detection tasks.</li></ol><p>Let’s talk about these in not-so-much detail</p><ol><li>Tabular Synthetic Data</li></ol><p>Tabular synthetic data refers to the generation of artificial data that resembles tabular data, such as a spreadsheet or a database table, with rows and columns of data. Each row in a table of synthetic data is a sample, and each column is a feature.</p><p>Tabular synthetic data can be generated using various methods, such as statistical sampling, simulation, or machine learning models such as generative adversarial networks (GANs) or variational autoencoders (VAEs). The methods used will depend on the desired properties of the synthetic data, such as the distribution of the data, the correlation between features, or the presence of outliers or anomalies.</p><p>Tabular synthetic data can be used in various applications, such as data privacy and confidentiality, data augmentation, model testing and validation, and model deployment. By generating synthetic data that resembles real-world data, it can help overcome challenges associated with using real-world data, such as data scarcity, data imbalance, or data privacy concerns.</p><p>However, it’s important to note that while synthetic data can be a valuable tool, it can also have limitations, such as not perfectly capturing the complex relationships and patterns in real-world data. Therefore, it’s important to evaluate the quality of the synthetic data and compare it to the real-world data to ensure that it meets the desired requirements.</p><p>Main libraries in Python —</p><ul><li><a href="https://github.com/gretelai/gretel-synthetics">gretel-synthetics</a> — Generative models for structured and unstructured text, tabular, and multi-variate time-series data featuring differentially private learning.</li><li><a href="https://github.com/sdv-dev/SDV">SDV</a> — Synthetic Data Generator for tabular, relational, and time series data.</li></ul><p>Let’s build a model and make ourselves some “new” tabular data. I will be using SDV for this example.</p><pre># create an environment and install libraries we will need<br>conda create -n synthetic_tabular python==3.8 -y<br>conda activate synthetic_tabular<br>pip install pandas_profiling<br>pip install sdv==0.13.1 -q</pre><p>I like to check all the dependencies if they have been properly installed before running my code</p><pre>pip show sdv<br>pip show pandas_profiling</pre><p>Let’s load a clean tabular data for this example. We will use pandas profiling library to get insights fast.</p><p>The pandas profiling library is used to generate exploratory reports for Pandas DataFrames. It provides a fast and simple way to perform exploratory data analysis (EDA) and identify patterns, trends, and potential problems in the data.</p><p>The library generates a comprehensive report that includes information such as:</p><ol><li>Overview: A summary of the DataFrame, including its shape, size, and data types.</li><li>Descriptive statistics: A summary of the central tendencies, dispersion, and distribution of the data.</li><li>Missing values: A summary of the missing values in the data, including the number and percentage of missing values for each column.</li><li>Correlations: A matrix showing the correlations between the columns in the data.</li><li>Outliers: Identification of outliers in the data using scatter plots, histograms, and box plots.</li><li>Duplicates: A summary of the duplicate records in the data.</li><li>Frequency tables: A summary of the frequency distribution of the categorical variables in the data.</li><li>High-cardinality variables: Identification of columns with high cardinality (a large number of unique values).</li></ol><pre>import warnings<br><br>import pandas as pd<br>from pandas_profiling import ProfileReport<br><br>from sdv.evaluation import evaluate<br>from sdv.metrics.tabular import CSTest, KSTest, LogisticDetection, SVCDetection<br>from sdv.tabular import CTGAN, TVAE, CopulaGAN, GaussianCopula<br><br>warnings.filterwarnings(&quot;ignore&quot;)<br><br>import numpy as np<br># you can download the adult.csv file from here<br># https://www.kaggle.com/datasets/uciml/adult-census-income?select=adult.csv<br>real_data = pd.read_csv(&quot;adult.csv&quot;).replace(&#39;?&#39;,np.nan).dropna().sample(1000)<br><br>print(real_data.shape)<br>real_data.head()<br><br>cat_columns = [<br>    &quot;workclass&quot;,<br>    &quot;education&quot;,<br>    &#39;education.num&#39;<br>    &quot;marital.status&quot;,<br>    &quot;occupation&quot;,<br>    &quot;relationship&quot;,<br>    &quot;race&quot;,<br>    &quot;sex&quot;,<br>    &quot;native.country&quot;,<br>    &quot;income&quot;,<br>]</pre><pre># lets do some visualization<br>profile = ProfileReport(<br>    real_data, title=&quot;Profiling Real Adult Data&quot;, html={&quot;style&quot;: {&quot;full_width&quot;: True}}<br>)<br># this saves a report in html in working dir and later rendered in the notebook<br>profile.to_file(&quot;real_data_report_ctgan.html&quot;)<br>profile.to_notebook_iframe()</pre><p>One of the most common ways to generate synthetic tabular data is via the CTGAN Model. The CTGAN model is based on the GAN-based deep learning data synthesizer, which was presented at the NeurIPS 2020 conference in the paper titled &quot;Modeling Tabular Data Using Conditional GAN.&quot;</p><p>We can add constraints to the synthetic data generation process too. For example, some values might be valid only in a particular range. Also, some columns, like DOB and age, have a strict correlation that the model might get wrong. We can add these logics into the constraints and pass them as an argument when we create an instance of the model.</p><pre># we have already imported CTGAN from sdv.tabular<br># lets create an instance and fit it on the real data<br># lets time it too<br>%%time<br>model_ctgan = CTGAN()<br>model_ctgan.fit(real_data)</pre><pre>new_data_ctgan = model_ctgan.sample(real_data.shape[0])<br><br>#lets generate a pandas profiling report on the new data too<br>profile_synthetic = ProfileReport(<br>    new_data_ctgan,<br>    title=&quot;Profiling Synthetic Adult Data&quot;,<br>    html={&quot;style&quot;: {&quot;full_width&quot;: True}},<br>)<br>profile_synthetic.to_file(&quot;fake_data_report_ctgan.html&quot;)<br>profile_synthetic.to_notebook_iframe()</pre><p>2. Image Synthetic Data</p><p>These images can be generated using computer graphics techniques or generative models such as Generative Adversarial Networks (GANs).</p><p>We can either talk about generating images from tags or from text. There are a lot of text-to-image data generators. In this article, we focus on the images generated, which can be used as synthetic data for training in specific cases.</p><h4>Synthetic Image Generation with Variational Autoencoders (VAE)</h4><p>Variational Autoencoders (VAEs) are a type of generative model that can be used to generate synthetic image data. VAEs are neural networks that learn to generate images by encoding an input image into a lower-dimensional representation, known as a latent representation or bottleneck, and then decoding the latent representation back into an image.</p><p>The encoding process is performed by an encoder network that maps an input image to its latent representation, while the decoding process is performed by a decoder network that maps the latent representation back to an image. The goal of VAEs is to learn a mapping from the input image space to the latent representation space that captures the underlying structure of the data.</p><p>One of the key features of VAEs is that they enforce a constraint on the distribution of the latent representations, which is typically modeled as a Gaussian distribution. This allows the VAE to generate new images by sampling from the Gaussian distribution and then decoding the samples back into images.</p><p>The generated images can be used as synthetic data for training and testing machine learning models, especially in domains where real-world data is scarce or privacy is a concern. VAEs have been shown to generate high-quality images that are similar to real-world data, making them a popular choice for image synthesis tasks.</p><p>In summary, VAEs are a type of generative model that can be used to generate synthetic image data by encoding and decoding images through a neural network architecture. They are a flexible and powerful tool for addressing data scarcity, data privacy, and data augmentation challenges in computer vision and machine learning.</p><h4>Synthetic Image Generation with Generative Adversarial Networks (GAN)</h4><p>Generative Adversarial Networks (GANs) are a type of generative model that can be used to generate synthetic image data. GANs consist of two networks: a generator network and a discriminator network. The generator network learns to generate new images, while the discriminator network learns to distinguish between real images and generated images.</p><p>The two networks are trained together in an adversarial manner, with the generator trying to generate images that are indistinguishable from real images, and the discriminator trying to correctly classify real images from generated images. Over time, the generator and discriminator both improve, and the generator is able to generate increasingly realistic images.</p><p>GANs have been shown to be capable of generating high-quality synthetic images that are similar to real-world images. They have been used for many different things, such as image synthesis, image translation, and style transfer.</p><p>One of the key strengths of GANs is their ability to generate images that are diverse and representative of the data distribution. This makes them well-suited for data augmentation and data generation tasks in computer vision and machine learning.</p><p>In short, GANs are a type of generative model that can be used to make fake image data by training a generator network and a discriminator network to work against each other. In computer vision and machine learning, they are a powerful way to deal with problems like not having enough data, keeping data private, and adding to data.</p><p>Head over to <a href="https://thispersondoesnotexist.com/">https://thispersondoesnotexist.com/</a> to look at some images generated by the legendary StyleGAN2/StyleGAN3.</p><h4>Image Diffusion Models</h4><p>Image diffusion models are a type of generative model that can be used to generate synthetic image data. They are based on the idea of diffusion processes, which describe the spread of a substance or signal through a medium over time. Image diffusion models treat an image as a signal that diffuses over time, and generate synthetic images by simulating the diffusion process.</p><p>The key idea behind image diffusion models is to model the relationship between an image and its transformed versions. This relationship is modeled as a diffusion process, in which a series of diffusion steps change the image into a new image. Each diffusion step can be thought of as a step in the image generation process, and the final image is generated by aggregating the results of the diffusion steps.</p><p>There are several types of image diffusion models, including non-linear diffusion models, anisotropic diffusion models, and random walk models. These models can be used to make fake images that look like real ones. They have been used in a wide range of ways, such as to remove noise from images, fix damaged images, and make new images.</p><p>Image Synthesis with Deep Convolutional Generative Adversarial Networks (DCGANs): DCGANs are a type of GAN that use deep convolutional neural networks as both the generator and the discriminator. DCGANs are capable of generating high-quality synthetic images that are similar to real-world images.</p><p>Generative Flow Models: Generative flow models are a type of generative model that can be used to generate synthetic image data by transforming a simple noise input into a high-dimensional image. Generative flow models are based on the idea of normalizing flow, which transforms a simple noise input into a complex distribution.</p><p>Let’s implement synthetic image generation in a few different ways.</p><ol><li>Generate fake faces with DCGANs</li><li>Synthetic Image generation with Flip. Generate new 2D images from a small batch of objects and backgrounds.</li><li>Text to Image Diffusion Model for photorealistic image generation</li></ol><h4>Generate fake faces with DCGANs</h4><p>I will be using four Nvidia A5000 RTX GPUs to generate these images since it is computationally very expensive.</p><h4>Synthetic Image generation with Flip. Generate new 2D images from a small batch of objects and backgrounds</h4><p>We focus on image overlaying and making it look realistic. We will be using Flip for this demo.</p><pre># create a conda environment with Python (&gt;= 3.7)Opencv (&gt;= 4.3.0) Numpy (&gt;= 1.19.1)<br># install flip<br>pip install flip-data</pre><p>Let’s make two directories, one for the background images and one for the objects to compose.</p><p>Object</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/0*I1VwzoL2SjNBP23O.png" /><figcaption>Object</figcaption></figure><p>Background</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ay-LvBXCok4nZOSd.jpg" /><figcaption>Background</figcaption></figure><p>Combination</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w5H1UTk6MwMLdt4e.jpg" /><figcaption>Overlayed Images</figcaption></figure><h4>Text to Image Diffusion Model for photorealistic image generation</h4><pre>from diffusers import StableDiffusionPipeline<br>import torch<br><br>model_id = &quot;runwayml/stable-diffusion-v1-5&quot;<br># do not use half precision on CPU<br># I will recommend using GPU for inference cause the CPU inference is slooow<br># for GPU feel free to change the float point precision to 16 from 32 in the next line of code<br>pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)<br># pipe = pipe.to(&quot;cuda&quot;)<br><br>prompt = &quot;a photo of a panda running through a field&quot;<br>image = pipe(prompt).images[0]  <br>    <br># image.save(&quot;panda.png&quot;)<br>image.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*bOTt3H-XpojJ_kIXPqTUnQ.jpeg" /><figcaption>Image generated by the diffusion model from the prompt “a photo of a panda running through a field”</figcaption></figure><p>3. Synthetic Text Data for OCR</p><p>I worked on OCR with Prudential, Singapore. OCR in real life is a prime example on OCR almost will never work in the wild without failsafes.</p><p>Also, ID Card data is sensitive so you need to generate your own.</p><p>We will be using the trdg library for this.</p><pre>pip install trdg</pre><p>Generate Images</p><pre>from trdg.generators import (<br>    GeneratorFromDict,<br>    GeneratorFromRandom,<br>    GeneratorFromStrings,<br>    GeneratorFromWikipedia,<br>)<br>impor time<br><br># The generators use the same arguments as the CLI, only as parameters<br>generator = GeneratorFromStrings(<br>    [&#39;Test1&#39;, &#39;Test2&#39;, &#39;Test3&#39;],<br>    blur=2,<br>    random_blur=True<br>)<br><br>for img, lbl in generator:<br>    # Do something with the pillow images here.<br>    img.show()<br>    time.sleep(2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/0*l4N8NImHg7HU-Php.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*oumTbTEU4fV0WuPy.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/0*Cv-dqWErJTnaVtOb.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*-rIJyp6Lf4M0EF0h.jpg" /></figure><h3>Final Remarks</h3><p>Synthetic data has become a crucial tool in the field of machine learning and data science. The idea of generating artificial data to solve real-world problems is not new, but recent advances in deep learning and generative models have dramatically increased its capabilities and applications.</p><p>One of the key benefits of synthetic data is its ability to address data scarcity and privacy issues. In many real-world situations, collecting large amounts of high-quality data can be difficult or even impossible. Synthetic data can be generated in vast quantities to fill in the gaps and provide additional training data for machine learning algorithms. Additionally, synthetic data can be used to protect sensitive information, such as personal data or confidential business information, by creating artificial datasets that contain similar patterns and structures, but not actual sensitive information.</p><p>Another advantage of synthetic data is its ability to improve the performance of machine learning algorithms. By generating artificial data that resembles real-world data, synthetic data can be used to train machine learning algorithms in ways that are not possible with real data alone. This can help to overcome the limitations of real data, such as class imbalance or overfitting, and lead to better model performance. Additionally, synthetic data can be used to augment real-world data, creating larger, more diverse datasets that can help to improve the accuracy and robustness of machine learning algorithms.</p><p>In recent years, the development of deep learning and generative models has also made it possible to generate high-quality synthetic image data. From Variational Autoencoders (VAEs) to Generative Adversarial Networks (GANs), and Image Diffusion Models to Deep Convolutional Generative Adversarial Networks (DCGANs), there are many techniques available for generating synthetic image data. These techniques can be used to create artificial datasets that closely resemble real-world images, and can be used for a variety of purposes, such as training computer vision algorithms, testing image processing algorithms, and even generating high-quality animations.</p><p>In addition to its technical benefits, synthetic data also has the potential to impact other areas of life and business. For example, synthetic data can be used to support innovation in fields such as autonomous vehicles, healthcare, and finance. By generating vast amounts of synthetic data that resembles real-world data, researchers and developers can create and test new algorithms and technologies in a safe and controlled environment, before deploying them in real-world scenarios.</p><p>Despite its many advantages, synthetic data is not a silver bullet, and there are still challenges and limitations that need to be addressed. For example, while synthetic data can be generated in vast quantities, it may not accurately capture the complexity and diversity of real-world data. Additionally, synthetic data may not fully capture the relationships and interdependencies between different variables, leading to models that are not fully representative of the real world.</p><p>In conclusion, synthetic data has become an increasingly important tool in the field of machine learning and data science. From addressing data scarcity and privacy issues to improving model performance and creating augmented datasets, the applications of synthetic data are many and varied. Whether we are using tabular data or image data, the generation of synthetic data is a powerful tool for solving real-world problems and advancing the field of machine learning. As machine learning and data science continue to evolve, it is likely that synthetic data will play an even more important role in shaping the future of artificial intelligence and data science.</p><p>References</p><ol><li><a href="https://www.kaggle.com/code/mcarujo/synthetic-data-generation-sdv-tutotial">https://www.kaggle.com/code/mcarujo/synthetic-data-generation-sdv-tutotial</a></li><li><a href="https://www.kaggle.com/code/sayakdasgupta/fake-faces-with-dcgans">https://www.kaggle.com/code/sayakdasgupta/fake-faces-with-dcgans</a></li><li><a href="https://github.com/LinkedAi/flip">https://github.com/LinkedAi/flip</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad722798af63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepSafe: Open source deepfake detection platform built for Researchers]]></title>
            <link>https://siddharthksah.medium.com/deepsafe-open-source-deepfake-detection-42f1c18f9500?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/42f1c18f9500</guid>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Thu, 02 Feb 2023 15:26:54 GMT</pubDate>
            <atom:updated>2025-05-23T09:41:05.298Z</atom:updated>
            <content:encoded><![CDATA[<pre>https://github.com/siddharthksah/DeepSafe<br>Hosted here — http://deepsafe.ml/ (If you want a demo and this is down please feel free to contact me)</pre><blockquote><strong><em>&gt; note</em></strong></blockquote><blockquote><strong><em>&gt;&gt; deepsafe has changed a lot since this article was published. we now support enterprise grade image manipulation detection. reach out to us for a demo deepsafe.hq [@] gmail.com</em></strong></blockquote><h3>Background</h3><p>DeepFake detection is important especially as the video content on the internet is getting more and more common. With the rise of influencer marketing on TikTok and Instagram Reels, creation of short video content is at an all-time high. In fact, companies like TikTok have beaten giants like Google in this sector. As these video contents are consumed on a variety of different platforms from smartphone to desktop, the deepfake detection solution needs to run on all these platforms seamlessly.</p><p>To understand the use case and practicality of deepfake detection on these devices, I explored multiple solutions in the form of different technologies and different use cases to understand which solution would be most practical. To quantify the results, a survey was carried out where users were given to try out all these solutions to deepfake detection and their experience was noted and studied. This type of study is unique and has not been carried with multiple solutions in the literature studied. This thesis aims to quantify the best proposed method to detect deepfake videos in the day to day social media content consumption. The solutions built and studied have been listed below.</p><ul><li>DeepSafeWebApp/DeepSafe Mobile App</li><li>DeepSafe Chrome Extension with client side processing</li><li>DeepSafe Chrome Extension with server side processing</li></ul><h3>Why DeepSafe?</h3><p>There is a need to keep in check on the fake information spread on social media more than ever. Fake news detection based on text data is a very well studied subject both in academia and industry. Most of the tools we use on the internet for texting come equipped for some sort of fake news detector.</p><p>There has been some work on creating platforms for DeepFake detection. DeepFakeo- meter created by Li et al was published in 2021. They incorporated a few of the existing detectors and ran inferences on images and videos on their webapp. Although, there is a lot of limitations within the webapp itself. It only supports uploading local files, no URLs. This renders the whole idea of using deepfake detection for social media unusable. There are also hard limits on the file size and no backend API is provided. DeepFake-o-meter has been only created for end users primarily. The features on API or adding custom detectors do not exist.</p><p>DeepSafe has been built with platform design in the core. It aims to provide additional custom features which can be incorporated on the platform right away. It comes with sanity checks and adding custom detection methods with minimal changes right out of the box. A platform like this is important because of the variety of generation and detection methods in academia being studied right now. An absence of a unified platform to use SOTA DeepFake Detectors is why DeepSafe was built.</p><p>Some of the deepfake creation techniques work well on edge devices. There are usually low res videos created using FaceSwap. They are usually harmless and created for memes. These apps run on mobile devices and need no technical knowledge to create these videos. As it has become handy the increase in the number of videos on being shared as memes has gone up exponentially. These creation methods are dynamic which means it gets more real every year. Currently, these videos are evident “enough” to look fake hence it has no intentional malicious use. But it is not very difficult to create high res ones, hence platforms like DeepSafe which can run right on your mobile phone are as relevant as ever</p><p>Companies like Deepware and Sensity aim to solve the deepfake detection issue as well. These companies are mostly closed source with some sample open source models. The platform design and code is closed source. Their target audience are SME and direct users. These are designed on the top of the front end and help these companies combat spread of fake media on their platform. The target audience is different. Deep- Safe is built by the researchers for the researchers solving the pain points of carrying out sustainable research in the deepfake detection domain. Building better detectors needs to instil knowledge from the previous generation ones. To create a better deepfake detector, researchers need to keep track and understand the previous built detectors. Every few months there is a new detector and generator solving the issues from the previous generations rendering the detectors useful in most cases. To keep a track of this highly dynamic research domain DeepSafe has been created.</p><p>DeepSafe not only is an aggregator of detection models but it also creates a dataset of all the potentially harmful circulated DF images and videos. An absence of a platform which can authenticate or flag a video is important to keep the trustworthiness in media alive. DeepSafe incorporates a 2 min quick guide to detect potential deepfakes in a glance based on the research until now. Blockchain and media file signatures can be integrated into DeepSafe to keep track of image and videos and its source. A standard way of detection is running detection models on a media file and outputting the probability. DeepSafe uses an extra step to save writing repetitive tests and provides a dashboard to keep track of all the experiments. Rather than running a detection model first, DeepSafe first looks for URL matching in the database. Image matching and Video fingerprint matching is also an important step to it. Video finger print is still in development. Currently, DeepSafe incorporates URL and image matching only. Although, by default this setup is toggled off. DeepSafe has a dynamic database, which means if we detect a deepfake we add it to our library of deepfakes after running it through other detectors to reduce the TN and FP.</p><p>There are a few competitors which do something similar. But they have their limitations, like 1. Closed source 2. Only supports uploaded image/video and only few URLs 3. All of them are static and use their proprietary detection algorithms 4. No open-source database</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUaS7VJ0ZvaVhHImNlV7jQ.png" /><figcaption>Key features of DeepSafe</figcaption></figure><h3>DeepSafe WebApp</h3><p>The DeepSafe WebApp provides a web UI which auto adapts to mobile and desktop views. The UI allows users to paste URLs of images and videos which they want to check the authenticity of. The URL supports the majority of the common image and video formats on the internet. DeepSafe downloads these media files and converts them into MP4 for video files and JPG for images. This ensures the URLs of video or images with non standard formats also work. Saving media in a single standard format ensures no failures in detection caused by file formats and support of different detectors on these formats.</p><p>The media downloader from the URL is based on the yt-dlp package. This supports the majority of the public videos and websites. A list of supported websites is provided in the appendix.</p><p>Along with the URL, users can also directly upload files from their local system. The image formats supported for uploading in the latest release as of July’22 are JPG, PNG, JPEG and video formats are MP4 and MOV. DeepSafe converts the uploaded files into MP4 for video files and JPG for images. This ensures the videos or images with non standard formats also work. Saving media in a single standard format ensures no failures in detection caused by file formats and support of different detectors on these formats.</p><p>Examples on DeepSafe WebApp</p><p>DeepSafe includes the examples of the most common SFW deepfake videos on the internet. These videos can be played right on the DeepSafe platform without leaving it. The idea behind this is to educate people what high quality deepfakes might look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*vFA79jR_zPFyXmN9YjvLFg.png" /><figcaption>An illustration of DeepSafe WebApp Main Page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*RD0dSO6TS2-PaaXe4Z9TtA.png" /><figcaption>DeepSafe WebApp About section — contains information about detectors incorporated onto the platform</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*6bTOqqLd1m3tn9wOT5luFA.png" /><figcaption>A quick guide to learn to spot DeepFakes</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*3AUbcOfNhrK09Rz4tCRN3A.png" /><figcaption>An illustration of how Deep Learning works in practice</figcaption></figure><p>DeepSafe Mobile App</p><p>DeepSafe mobile app has both the versions for ios and android. The processing in done on the server side and not on the mobile phones hence the user experience related to inference time remains the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*T9IZ7ThF85RYAios3_5ieg.png" /><figcaption>Mobile Apps for iOS and Android</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*I8t5j5GtoFcVxNAZMei_EA.png" /><figcaption>Usage of URLs and local uploads</figcaption></figure><h4>DeepSafe Chrome Extension with client side processing</h4><p>The first version of DeepSafe Chrome Extension was based on processing media files right in the browser. While this methodology ensures significantly lower server costs for the host platform but it comes with a great expense of user experience. DeepFake detection especially on video files can be computationally expensive which if run on a slower system can even freeze. Moreover, each video needs to be processed every time a user wants to check. If the detection is carried right onto the server after the video or image is uploaded onto the platform it ensures the check happens only once. Also, it can flag such videos without even reaching the public and spreading potential misinfomation. Processing data in user’s browser also solves the issue of data privacy and handles any data storage cost. In case of video files when the file size is bigger compared to image files, transferring the data between client and server takes time effectively increasing the total latency.</p><p>Building a detection system should not rely on the client side for processing unless the total inference is done under a second on a standard edge device. As DeepFake detection, especially on video files can be computationally expensive, the processing being done on the server side makes the user experience better. Although it comes at the cost of data transfer in case the user wants to upload their own custom video file. As that particular case wouldn’t be the user browsing, not having a real-time doesn’t affect the user experience.</p><p>The v1 of DeepSafe chrome extension focussed on doing the inference on images. The user experience was also limited to it. No inference was done on video files in this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*UAbzRSzzYl-L7wGficvFgQ.png" /><figcaption>An illustration of how the first version of DeepFake chrome extension work</figcaption></figure><h4>DeepSafe Chrome Extension with server side processing</h4><p>As having a chrome extension continuously running in the background can be computationally expensive, v2 of the extension focussed on understanding this current limitation.</p><p>If we look at the social media content, the majority of them are media files. With the rise of TikTok and other short videos platforms, consumption of media files has leaned towards short video clips. The DeepSafe detection system architecture is specifically designed to support these video files and their URL. V2 chrome extension opens the video in a new tab of the browser and processes it to get the deepfake probability.</p><h4>URL Matching</h4><p>There are multiple ways to reduce the processing time and computational expense during detection. The classic ones are reducing or downsizing the image and video files to lower resolution. This reduces the input layer size and hence reduces the total computation. Another method is model quantization and pruning. This method essentially reduces the model size removing weights by making them zero. Although it does impact the accuracy there is a trade off between inference time and accuracy but selecting the right parameters can significantly reduce the processing time hence increasing the user experience.</p><p>The entire discussion around processing time is the key because the user experience significantly improves if the inferences are as real time as possible. Inference shouldn’t be the bottleneck in the ideal situation.</p><p>One of the interesting ways to reduce computation is using URL matching. The idea is to store the URLs of the image or video which are flagged as deepfake by DeepSafe. If this flag is in infact true we can store it in our database. Hence before running an inference of a new media we can search for it in the database and return back the flag. Although this seems more like an ad hoc solution. The flag can be verified by crowd sourcing or using other deepfake detection models. All the detectors can test this media file and output a probability distribution. This probability distribution will increase the confidence score of the flag and in case it is higher than a preset threshold we can flag or unflag.</p><p>URL matching is especially important for video files. It reduces the computational cost by hundreds of times if not thousands. Additionally, saving URLs cheap and well practiced.</p><h4>Media similarity</h4><p>By definition, similarity is a hazy idea. It works well when two people talk, but it is not the simplest thing for a deterministic computer programme to cope with. If the aim is to analyse certain characteristics of a picture, creating and training a model for that purpose works, such as a standard image classification model. However, if the “similarity” requirements are visual but not well defined, or if there is a lack of finely labelled data for training, picture similarity using deep learning may help. Deep learning paves the way for reliable quantification of picture similarity, allowing for the automation of even ill-defined tasks.</p><p>The most apparent use is probably reverse searching. Reverse searching allows us to start with the content and get the keywords linked with it. Deep learning similarity approaches are also at the heart of some of the most effective methods for landmark identification, which may help us recall where to focus in photos. Picture similarity may be used for the data clustering if we want to keep things ordered. This enables us to use a mix of explicit information, such as clothing type, and visual cues learnt implicitly by a deep learning model.</p><p>However, unlike a traditional hash function, the compressed representations should be comparable such that the degree of similarity between their respective items may be plainly indicated.</p><p>“Cosine similarity” is the operation that is typically used to assess how close two things are when it comes to similarity. When we read that two photos are 90% similar, it most likely implies that their compressed representation’s cosine similarity is equal to 0.90, or that they are 25 degrees apart. The similarity of two vectors in an inner product space is measured by cosine similarity. It is calculated by taking the cosine of the angle between two vectors and determining if two vectors are heading in the same general direction. It is often used in text analysis to assess document similarity. Cosine similarity is a mathematical measure of the cosine of the angle between two vectors projected in a multidimensional space.</p><p>When plotted on a multi-dimensional space, where each dimension corresponds to a word in the text, the cosine similarity captures the direction (the angle) but not the amplitude. The cosine similarity is helpful because, even if two comparable papers are separated by the Euclidean distance due to size (for example, the word “cricket” occurred 50 times in one document and 10 times in another), they may still have a lesser angle between them. The greater the resemblance, the smaller the angle.</p><p>Using image and video similarity techniques can also help us look at the existing database and flag if an image or video is deepfake without running inferences on them, saving computation.</p><p><strong>Front-end and Back-end</strong></p><p>DeepSafe’s front end is based on Streamlit. DeepSafe automatically detects the file type from the URL or the uploaded file and presents only the relevant detection methods.</p><p><strong>DeepSafe Dashboard</strong></p><p>DeepSafe has a built-in dashboard which provides all the information and plots of the experiments run. This saves all the information about inference on different images and videos. Also, it saves a local copy of these files in case it is detected as DeepFake which can be used in retraining your detection models. The dashboard contains the following information. Number of Inferences, Timestamp, Average DF probability, File Type, File Size in MB, Total processing time in seconds.</p><p><strong>Streamlit</strong></p><p>Streamlit is an open-source Python application framework. It expedites the creation of web applications for data science and machine learning. It is interoperable with important Python libraries, including scikit-learn, PyTorch, Keras, NumPy, Matplotlib and pandas, among others.</p><p>Streamlit does not need callbacks since widgets are viewed as variables. Caching data simplifies and accelerates calculation processes. Streamlit monitors for modifications to the connected Git repository and deploys the application automatically in the shared link. Streamlit streamlines the interactive cycle of coding and visualisation in a web application.</p><p><strong>Project Structure</strong></p><p>The procedure to integrate new models onto the platform locally is simple. Unlike other methods, we do not need to change any parameters into the original model what- soever. The idea is to have an extra .py file which runs the inference using the command line right inside a python file. This is achieved by calling the inference code in a python pseudo terminal. The approach standardised both the inputs and outputs. Ideally the output should be a text file which the probability of deepfake in the root directory. The exception handling comes by default out of the box which includes readable error codes which point out exactly where the code broke during the process which makes it easier to debug. Also, the sanity check which comes out of the box when run checks for all the potential bugs and gives a report of the entire pipeline with just one click. The version control is handled such that all the models have their own environment which provides a very smooth integration of models over years with different versions of libraries.</p><p><strong>How does DeepSafe help DeepFake researchers?</strong></p><p>Since the last few years we have seen a rise in the amount of deepfake videos and photos on the internet. Some of these videos are created with malicious purposes which need to be detected/removed from the platform. As the new videos make the old detectors obsolete it is important to learn and understand how the old detectors work in order to create the new ones. DeepSafe comes handy here. Any researcher can create a local copy of DeepSafe and install the dependencies in just a few commands. These commands have been tested and customised anc verified for Windows, Mac and Ubuntu OS. Majority of these detectors are not in active development and do not come with the installation requirements which makes it tricky and at times next to impossible to create the right environment to test. Also, over time these detectors have different versions of python and associated libraries which makes it troublesome to test all at once. DeepSafe saves weeks of these trouble.</p><p><strong>How to add your own DeepFake Detector?</strong></p><p>DeepSafe allows a simple and intuitive way of adding any custom detector and comes with out of the box code to test it against other state of the art detectors.</p><ol><li><strong>Folder Structure</strong></li><li><strong>Meta Data</strong></li></ol><p>Metadata includes the licence, the URL to the project, type of detector. An example of Meta data is below</p><p>Boken, <a href="https://github.com/beibuwandeluori/DeeperForensicsChallengeSolution,">https://github.com/beibuwandeluori/DeeperForensicsChallengeSolution,</a> MIT Licence</p><ol><li><strong>Demo.py</strong></li></ol><p>This is the most important aspect which connects any deepfake detector to DeeSafe. The demo.py calls the model and saves the inference result in a text file in the model folder. Note that this is saved in the respective model folder and not the root.</p><ol><li><strong>requirements.txt</strong></li></ol><p>This should contain all the relevant libraries and their versions needed The folder name should be “detectorname-image” or “detectorname-video”, depending on whether it is an image or video detector.</p><ol><li><strong>Out of the box features</strong></li></ol><p>Run inference on any dataset with all the detectors. This will also provide an estimated time of completion. Drag and drop folders in the main repo without any extra configuration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*BZjX_86IQvaS0pXA7qmq2A.png" /><figcaption>info-graphics on the folder structure of the DeepSafe Models</figcaption></figure><p><strong>Sanity Check</strong></p><p>Checks all the required model settings and runs inference with all the models on test images and videos to check if everything is working fine.</p><p><strong>DockerFile</strong></p><p>Docker is essentially a very light-weight virtual machine that contains all of the applications and dependencies required to execute your programme. When we set up a large application with a UI, database, and many microservices, we must complete a setup, and each of these services need some type of software to function, such as JAVA. And setting up and installing everything properly on the new server would most likely take days.</p><p>However, most of the time, versions do not match and other issues arise, necessitating extensive debugging. So, what Dockers does is construct an image and describe what is required for our software. So we’re just saying these are the dependencies, this is where our programme is coming from, and then we need to construct a container with our software, which we’re simply fetching from the Docker registry. And a container containing a software will be running, ready to boot in a few seconds. So we merely download and start in a few seconds, and everybody who requests that image will have the exact same configuration. The biggest benefit of Dockers is the ability to have the exact same configuration everywhere and to swiftly bootstrap the whole infrastructure. Containers may also be built quite rapidly.</p><p>Docker is a great technology to use if we want to create a single application that can run and operate identically in any environment, laptop, or public cloud instance, owing to its convergent virtualized packages known as containers.</p><p>A Dockerfile is a basic text file that contains instructions for creating Docker images. The benefit of using a Dockerfile over just saving the binary image (or a snapshot/template in other virtualization systems) is that automated builds guarantee we always have the most recent version. This is a good thing from a security standpoint, as you want to guarantee that no susceptible software is installed.</p><p><strong>How are the examples on DeepSafe WebApp collected?</strong></p><p>DeepSafe web app contains Safe for Work(SFW) deepfake videos for educational purposes. These videos have been collected through web scraping on YouTube.</p><p><strong>Current versions of DeepSafe WebApp</strong></p><p>The DeepSafe webapp has 2 versions for the final prototype. The first one, hosted on Google Cloud Run, comes with limited features and can be used on the web without any installation whatsoever. This version has been built for end users who want to test if the video or image they are watching is DeepFake or not. The second version of the webapp is built for the Researchers who can use it to incorporate their custom detectors and test it against custom datasets. The features of the later version have been listed in the previous paragraphs. The first version of DeepSafe mentioned is Proof-of-concept work only. The thesis is based on the second version of it.</p><p><strong>DeepSafe API</strong></p><p>There are multiple options when using the DeepSafe API. Run inference on an entire folder across all the detectors out of the box. Run inference on an entire folder with specific detectors only. The API is available on GitHub and is also open source under MIT Licence.</p><p><strong>API Design</strong></p><p>Pass images in base64 encoding with a number which represents which SOTA algorithm to use for DeepFake Detection get json output with the probability float value.</p><p>API is a collection of procedures, protocols, and tools used to develop software applications. An API essentially defines how software components should communicate. They are also used to programme components of graphical user interfaces (GUIs). A good API facilitates programme development by offering all the necessary components. Modern APIs conform to developer-friendly, widely accessible, and commonly understood standards (usually HTTP and REST). They are handled as products rather than code. They are intended for consumption by certain audiences (e.g., mobile developers), are documented, and are versioned so that users may have certain maintenance and lifetime expectations. They have a much greater discipline for security and governance, as well as performance and scale monitoring and management, since they are much more standardised. Modern APIs have their own software development lifecycle (SDLC) consisting of designing, testing, constructing, maintaining, and versioning.</p><p>Flask is a popular micro web framework for developing Python APIs. It is a basic but powerful web framework built for rapid and easy application development, with the capacity to expand to large projects. “Micro” does not imply that your whole web application must be contained inside a single Python file, nor does it imply that Flask lacks capability. The “micro” in microframework indicates that Flask intends to maintain a minimal, expandable core.</p><p>APIs enable your product or service to communicate with other goods and services without requiring you to understand their implementations. This may facilitate app development simplification, therefore saving time and money. When building new tools and products or managing current ones, APIs allow flexibility, ease design, administration, and usage, and enable innovation.</p><p>Sometimes, APIs are seen like contracts, with documentation representing the parties’ agreement: If party 1 delivers a remote request formatted in a certain manner, party 2’s programme will answer. Because APIs make it easier for developers to incorporate new application components into an existing architecture, they facilitate collaboration between business and IT teams. In response to everevolving digital marketplaces, where new rivals may transform an entire sector with a single app, business requirements often undergo rapid transformation. To remain competitive, it is essential to encourage the quick development and implementation of new services. APIs are used to link a microservices application architecture to cloud-native application development, which is a determinable method for increasing development pace.</p><p>APIs are a streamlined method to link your own infrastructure through cloud- native application development, but they also enable you to share your data with clients and other external users. Public APIs provide distinctive economic value since they may simplify and broaden how you communicate with your partners, and possibly enable you to monetise your data (the Google Maps API is a popular example).</p><p>One of the primary benefits of APIs is that they enable the separation of functionality across systems. An API endpoint decouples the application using a service from its underlying infrastructure. As long as the specification for what the service provider is sending to the endpoint stays the same, the changes to the infrastructure underlying the endpoint should go unnoticed by the API-dependent apps. This provides the service provider with service providing flexibility. For instance, if the API’s underlying architecture consists of physical servers in a data centre, the service provider may simply migrate to cloud-based virtual servers.</p><p><strong>5.17 DeepFake Detection Methods incorporated on DeepSafe</strong></p><ol><li>MesoNet [15] is a CNN model that focuses on the mesoscopic characteristics of pictures. They provide two MesoNet variations, notably Meso and MesoInception. Meso employs typical convolutional layers, while Meso Inception is built on more complex Inception modules [3]; Meso Inception is included into the plat- form.</li><li>FWA [16] is built on ResNet-50, which identifies DeepFake movies by revealing face warping distortions resulting from resizing and interpolation procedures.</li><li>VA[18] addresses visual artefacts in face organs including the eyes, teeth, and facial features of synthetic faces. This approach is available in two variations: VAMLP and VA-LogReg. VA-MLP is based on a CNN whereas VA-LogReg em- ploys a logistic regression model. We include VA-MLP into the system.</li><li>Xception is equipped with the FaceForensics++ dataset. It is a DeepFake de- tection approach based on the XceptionNet model. This approach offers the Xception-raw, Xception-c23, and Xception-c40 variations. Xception-raw is learned on raw videos, while Xception-c23 and Xception-c40 are trained on movies com- pressed to varying degrees. Xception-c23 is included in the platform.</li><li>ClassNSeg [19] is a further CNN-based DeepFake detection approach that is de- signed to simultaneously identify forgeries pictures and segment modified re- gions.</li><li>Capsule uses the VGG19 capsule structure as the primary architecture for Deep- Fake classification.</li><li>DSP-FWA is a further enhanced approach based on FWA that includes a spatial pyramid pooling (SPP) module to better deal with the changes in face resolutions.</li><li>CNNDetection [20] employs a conventional image classifier trained on just Pro- GAN [15], discovering that it generalises very well to unobserved topologies, datasets, and training techniques.</li><li>According to Upconv [21], standard up-sampling approaches (upconvolution or transposed convolution) are incapable of accurately reproducing the spectral distributions of real training data. As a feature, they use the 2D amplitude spectrum and a simple SVM classifier.</li><li>To generate predictions, WM combines two WS-DAN models (using EfficientNet- b3 and Xception feature extractors, respectively) and an Xception classifier.</li><li>Selim employs a state-of-the-art encoder, EfficientNet B7, which has been pre trained with ImageNet and noisy student, and a heuristic method to pick 32 frames for each video to average predictions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JFyb_HTUkSNZyShIF-vBfQ.png" /><figcaption>An illustration of how RestAPIs work in DeepSafe</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*oGTBvcuGiLDyHjUgNTjxOA.png" /><figcaption>The list of DeepFake detector incorporated in DeepSafe</figcaption></figure><h3>Discussion</h3><p>One of the major aspects of developing DeepSafe was to address the ethical concerns and create a system to detect fake media. The research question talks about creating fake personas and putting systems and mechanisms in place to comply with the fair usage of this technology. These systems need to be fused with the fabric of social media and content sharing. Although, knowledge of the existence of these technologies solves the major problem of seeing and believing media content on the internet.</p><p>Media manipulation has been there since the start of media creation. Even before deepfake it was possible to create such high quality videos, it was just tedious and required professional experience. This technology just made it easier to do so.</p><p>When Adobe Photoshop came into existence, media manipulation became common. Adobe photoshop didn’t go extinct if anything it evolved to be more powerful and feature rich than ever. Photoshop focuses more on images, DeepFakes majorly on videos. The technological advancement will be analogous. Of course creating reliable detectors are important but what is more important is being aware that videos can be created and manipulated in high quality.</p><p>A few of our user journey interviews ended with feedback like including a photoshop manipulation detector and audio deepfake detector onto the platform as well. The future work could entail this.</p><p>The study focussed only on the platform and assumes that currently no universal detector exists and the results may not be generalizable to sub cases where a detector works for all the cases. URL and image matching can computation but the most computation theoretically will be saved by video matching.</p><p>DeepSafe works like a platform and can evolve over time without any major changes. As long as an API can be written to run a model and save the inference, DeepSafe will be relevant.</p><h3>Conclusion</h3><p>DeepFake detection research is an extremely active domain. DeepSafe not only helps keep track of the state of the art work until now but also provides a forward facing development platform to add new ones built by researchers.</p><p>For end users, DeepSafe was tested in different versions to find out which one is the most practical with the best user experience. We tested different versions of chrome extension which would let users use it right in the browser. The tests done with Androidand the ios app proved users want to use it only when they feel suspicious of a video or photo. Running it in the background the entire time ruins the user experience. The DeepSafe web app which supports all the major systems including edge devices had the best user experience. It had the lowest average inference time cause the processing was server side and it supported the most devices hence could be used by most number of users. Mobile app seems a great solution since it would mean using the same device to consume digital content and for detection, hence lower friction. The current mobile app has limited features hence we could not test this hypothesis but something to explore in the future work.</p><p>DeepFake detection is an intermediate solution to a much bigger problem. Al- though we need reliable detectors, what we need the most is educating the general population about the existence and usage of this technology.</p><h3>Future Work</h3><p>The platform incorporates eight cutting-edge detection algorithms, as well as interfaces allowing researchers to implement their algorithms into the platform. DeepSafe may continue to include more DeepFake detection algorithms onto the platform in the future. Furthermore, multi-GPU systems may be used to expedite the data analysis. APIs may be extended to provide more broad detection algorithms for different media types. Audio-based deepfake detectors may also be used since the detection principles are the same.</p><p>Blockchain systems store data in a decentralised, immutable ledger that is regu- larly reviewed and re-confirmed by every organisation that uses it, making it almost difficult to modify data once it has been established. One of the most prominent uses of blockchain is to track the movement of cryptocurrencies such as Bitcoin. However, since blockchain enables decentralised authentication and a clear chain of custody, it has the potential to be useful as a tool for monitoring and validating not just financial resources, but also a wide range of other types of content. Blockchain technology holds a lot of potential and can be explored in the later version of DeepSafe. Blockchain has immensely benefited the fight against deepfakes. However, it is not completely trust- worthy. To maximise its potential, other technologies such as artificial intelligence will need to be combined. Combining blockchain and artificial intelligence may dramatically help the battle against deepfakes. Even if we consider blockchain’s limitations in detecting deepfakes, it is still beneficial to have a system that can aid in recognising deepfakes and thereby reducing the amount of disinformation they distribute.</p><p>In the future versions of DeepSafe, sub video matching can be studied. We can check for a video in the database to return it if it is real or fake without running inferences.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42f1c18f9500" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use subprocess to efficiently use multiple conda environments in the same project]]></title>
            <link>https://siddharthksah.medium.com/how-to-use-subprocess-to-efficiently-use-multiple-conda-environments-in-the-same-project-2b1d45024611?source=rss-1aa18b2cd060------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b1d45024611</guid>
            <category><![CDATA[conda]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Siddharth Sah]]></dc:creator>
            <pubDate>Thu, 02 Feb 2023 09:50:47 GMT</pubDate>
            <atom:updated>2023-02-02T14:20:15.661Z</atom:updated>
            <content:encoded><![CDATA[<p>Let&#39;s start with the most basic method of managing conda environments.</p><ol><li>Create separate environments for each project requirement:</li></ol><pre>conda create -n env1 python=x.x <br>conda create -n env2 python=y.y</pre><p>2. Activate the desired environment before working on the project:</p><pre>conda activate env1</pre><p>3. Deactivate the current environment before switching to another:</p><pre>conda deactivate</pre><p>4. Export the environment’s specifications to a file:</p><pre>conda env export &gt; env1.yml</pre><p>By doing this, you can have different packages and dependencies for each environment, and switch between them easily as needed through the <em>terminal</em>.</p><p>This works fine unless you need to do this on the fly, i.e., between different conda environments for the same project.</p><p>Multiple conda environments in a Python project can be necessary for several reasons, including:</p><ol><li>Package management: Different projects may require different versions of packages or different packages altogether. Having separate conda environments ensures that each project has the specific packages it requires and reduces the risk of version conflicts.</li><li>Reproducibility: Separate environments ensure that the project can be easily run on different machines and that results can be reproduced.</li><li>Isolation: Different environments can be isolated from one another, which is useful for testing and development.</li><li>Compatibility: By using separate conda environments, you can create an environment that is compatible with the specific requirements of a project, such as a specific Python version.</li></ol><p>In short, using multiple conda environments provides greater control and organization for Python projects and reduces the risk of conflicts and compatibility issues.</p><p>The below code defines a function switch_conda_environment which takes the name of a conda environment as an argument and switches to that environment using the subprocess.run function. The shell=True argument is used to run the command in a shell, which is required to use the conda command. The function prints the name of the environment that was switched to, which can be useful for debugging purposes.</p><p>In the example usage section, the function is used to switch between two different environments, “env1” and “env2”. You can replace these names with the names of your own conda environments.</p><pre>import subprocess<br><br>def switch_conda_environment(env_name):<br>    subprocess.run(f&quot;conda activate {env_name}&quot;, shell=True)<br>    print(f&quot;Switched to conda environment: {env_name}&quot;)<br><br># Example usage:<br>switch_conda_environment(&quot;env1&quot;)<br>switch_conda_environment(&quot;env2&quot;)</pre><p>For deployment, I advise adding another function that checks if the environment exists else creates one for you with the given requirements. You can go crazy and handle errors too. Just purge and reinstall. For demo I am going to stick to the fundamentals.</p><pre>import subprocess<br><br>def create_conda_environment(env_name, requirements_file):<br>    env_exists = False<br>    try:<br>        subprocess.run(f&quot;conda activate {env_name}&quot;, shell=True, check=True)<br>        env_exists = True<br>    except subprocess.CalledProcessError as e:<br>        pass<br>    <br>    if not env_exists:<br>        subprocess.run(f&quot;conda create --name {env_name} --file {requirements_file}&quot;, shell=True)<br>        print(f&quot;Conda environment {env_name} created.&quot;)<br>    else:<br>        print(f&quot;Conda environment {env_name} already exists.&quot;)<br><br># Example usage:<br>create_conda_environment(&quot;env1&quot;, &quot;requirements.txt&quot;)</pre><p>This code defines a function `create_conda_environment which takes the name of a conda environment and a requirements file as arguments. The function first attempts to activate the environment using the subprocess.run function. If the environment does not exist, a CalledProcessError is raised, and the function proceeds to create the environment using the `conda create command and the requirements file. The shell=True argument is used to run the command in a shell, which is required to use the conda command. The function prints a message indicating whether the environment was created or already existed.</p><p>In the example usage section, the function is used to create an environment named “env1” using the requirements file &quot;requirements.txt&quot;. You can replace these values with your own environment name and requirements file.</p><p>Having multiple conda environments in the same project also means you have the flexibility to use different Python versions for different projects. This is a big relief if you want to work with papers that use different Python versions, but the Python versions are not compatible. It even works with Python 2. Here is the code, which also takes the Python version as an argument.</p><pre>import subprocess<br><br>def create_conda_environment(env_name, requirements_file, python_version):<br>    env_exists = False<br>    try:<br>        subprocess.run(f&quot;conda activate {env_name}&quot;, shell=True, check=True)<br>        env_exists = True<br>    except subprocess.CalledProcessError as e:<br>        pass<br>    <br>    if not env_exists:<br>        subprocess.run(f&quot;conda create --name {env_name} python={python_version} --file {requirements_file}&quot;, shell=True)<br>        print(f&quot;Conda environment {env_name} created.&quot;)<br>    else:<br>        print(f&quot;Conda environment {env_name} already exists.&quot;)<br><br># Example usage:<br>create_conda_environment(&quot;env1&quot;, &quot;requirements.txt&quot;, &quot;3.9&quot;)</pre><p>In this updated code, the function create_conda_environment takes a third argument python_version, which specifies the desired version of Python for the conda environment. This version is passed to the conda create command using the python option.</p><p>In the example usage section, the function is used to create an environment named “env1” with Python version 3.9 and the requirements file “requirements.txt”. You can replace these values with your own environment name, requirements file, and desired Python version.</p><p>Here is a master function which puts all of this together and works seamlessly when working with multiple environments.</p><pre>import subprocess<br><br>def handle_conda_environment(env_name, requirements_file, python_version):<br>    def create_conda_environment():<br>        subprocess.run(f&quot;conda create --name {env_name} python={python_version} --file {requirements_file}&quot;, shell=True)<br>        print(f&quot;Conda environment {env_name} created.&quot;)<br><br>    def switch_conda_environment():<br>        subprocess.run(f&quot;conda activate {env_name}&quot;, shell=True)<br>        print(f&quot;Switched to conda environment: {env_name}&quot;)<br><br>    env_exists = False<br>    try:<br>        switch_conda_environment()<br>        env_exists = True<br>    except subprocess.CalledProcessError as e:<br>        pass<br>    <br>    if not env_exists:<br>        create_conda_environment()<br>    else:<br>        print(f&quot;Conda environment {env_name} already exists.&quot;)<br><br># Example usage:<br>handle_conda_environment(&quot;env1&quot;, &quot;requirements.txt&quot;, &quot;3.9&quot;)</pre><p>This code defines a function handle_conda_environment which takes the name of a conda environment, a requirements file, and a Python version as arguments. The function contains two inner functions create_conda_environment and switch_conda_environment which perform the respective tasks as described in previous answers.</p><p>The handle_conda_environment function first attempts to switch to the specified environment using the switch_conda_environment function. If the environment does not exist, a CalledProcessError is raised, and the function proceeds to create the environment using the create_conda_environment function. The function prints messages indicating the state of the environment, whether it was created, switched to, or already existed.</p><p>In the example usage section, the function is used to handle an environment named “env1” with Python version 3.9 and the requirements file “requirements.txt”. You can replace these values with your own environment name, requirements file, and desired Python version.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b1d45024611" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>