<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashley Peterson on Medium]]></title>
        <description><![CDATA[Stories by Ashley Peterson on Medium]]></description>
        <link>https://medium.com/@ashley.peterson?source=rss-502ec8706481------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*58DD3daifH80fyNRju4rIA.jpeg</url>
            <title>Stories by Ashley Peterson on Medium</title>
            <link>https://medium.com/@ashley.peterson?source=rss-502ec8706481------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:25:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashley.peterson/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Comparison Between AWS and Azure to Enable Forecasting at Scale]]></title>
            <link>https://medium.com/@ashley.peterson/a-comparison-between-aws-and-azure-to-enable-forecasting-at-scale-f17c0b859e4e?source=rss-502ec8706481------2</link>
            <guid isPermaLink="false">https://medium.com/p/f17c0b859e4e</guid>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[aws-deepar]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[forecasting]]></category>
            <dc:creator><![CDATA[Ashley Peterson]]></dc:creator>
            <pubDate>Fri, 28 Aug 2020 19:48:32 GMT</pubDate>
            <atom:updated>2020-08-28T19:48:32.584Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dpLnC_HRQ93YGWyBkAA8xw.png" /></figure><p>By <a href="https://medium.com/@ashley.peterson">Ashley Peterson</a> and <a href="https://medium.com/@evan_schaeffer">Evan Schaeffer</a></p><p>Forecasting is a common tool with applications in all types of business. Whether this technique is used to predict sales, order volume, staffing requirements, or inventory, it is not a small task. Building an accurate forecasting model for one entity is challenging enough, and the difficulty level increases with the need to forecast at scale. Businesses need this ability not just at a global or regional level, but for individual stores, departments, or even products.</p><p>Forecasting at scale can be challenging to implement. Some of the biggest challenges include model customization and management, data processing and aggregation, and provisioning the compute power necessary for many forecasts.</p><p>Cloud platforms are developing tools to help address these challenges. We explored AWS’s DeepAR (Autoregressive Recurrent Neural Networks) capability and Azure’s AutoML in their Machine Learning Workspace to enable implementing forecasting at scale.</p><h3>Model Customization &amp; Model Management</h3><p>When working with limited resources, a common method is to develop a general model from one representative series and apply it to the rest of the data rather than creating custom models for individual series. If you have series that behave consistently this method could work, but if you are looking to forecast at different levels of granularity, the patterns may not be as consistent. Leveraging a cloud platform allows for additional model customization and managing those custom models.</p><p>AWS and Azure enable efficient customization of models for each series you wish to forecast. AWS DeepAR leverages a recurrent neural network to learn patterns from the entire set of time series data and applies this model to each individual series. Azure AutoML trains each series across multiple models and returns the best model for each series based on user criteria. With AutoML you can customize the list of models you would like to run for model selection. Both tools allow user input options to customize the parameters for the model training jobs.</p><p>Our preference of platform on this topic is dependent on the data and the level of effort required to support the models generated over time. If the individual time series have similar patterns, then AWS DeepAR uses a very suitable algorithm; time series that are full of different patterns would benefit from using Azure AutoML.</p><p>Retraining a model is important for maintaining accurate forecasts over time. Since AWS DeepAR only leverages one algorithm, changes that occur in retraining will be minor and wouldn’t require much oversight. Azure AutoML could see vastly different models as new trends unfold over time which should be reviewed by the developer. Azure naming convention and tagging become very important as the number of models increase to easily find the models that you’re looking for.</p><p>Since both solutions offer versioning, it is easy to compare differences or revert to an older model when necessary. Within Azure, model versioning is automatic, and you can add tags for additional information. In AWS, tags can be used for model versioning or for a more rigorous approach you can store models in an S3 bucket with versioning enabled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*RzUsr8UtO04MtOQSV941RQ.jpeg" /><figcaption>Table created with sheetsu</figcaption></figure><h3>Data Processing &amp; Forecasting Granularity</h3><p>Having the option to toggle granularity when creating forecasts may end up being a big benefit for organizations with multiple use cases for forecasting. The granularity of a forecasting model is decided at the data processing stage, where aggregation and formatting occur. Tools that provide flexibility in this space will save a lot of time and effort when changing the forecasting approach.</p><p>Azure AutoML and AWS DeepAR both require certain formatting and cleaning of the training data before it can be ingested by each algorithm. Azure offers some features for automatic pre-processing, including the imputation of missing data. Since DeepAR is a custom algorithm, AWS will require certain formatting that must be done prior to training, including transforming the data into JSON format. These steps will need to be repeated and potentially reworked if there are changes to the desired granularity when modeling.</p><p>Often, we want to see forecasting at multiple levels with the ability to roll up or drill down. With both AWS and Azure the model is trained at the granularity level of the input data. Leveraging the pipelines in Azure, you can train multiple models at various aggregation levels or you can create steps to aggregate your data post modeling. Since DeepAR is training one model for all series, you can set up multiple training jobs if you desire modeling at several granularities. Lambda functions can be leveraged to process your data after modeling.</p><p>As data scientists who are used to working with tabular data, we found Azure AutoML to provide a smoother pre-processing and formatting experience. There are fewer requirements for encoding, cleaning, and restructuring. However, with enough exposure to these tools the difference will become trivial as they both will follow a repeatable process.</p><p>In both platforms you can set up a flexible pipeline that will eliminate the need for code refactoring as you introduce new data or change the granularity of the model, but it must be a cognitive effort from the start if it will save any time in the long run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*JFnCviGzea1xxKbdYdGtCg.jpeg" /><figcaption>Table created with sheetsu</figcaption></figure><p><em>*Aggregating post forecast will require recalculation of error metrics using validation set</em></p><h3>Compute Power</h3><p>‘For’ loops are not a very efficient way to train hundreds of individual models. A cloud solution will open the door to training and retraining in a relatively short amount of time enabling data scientists to focus on tuning and improving models.</p><p>Cloud platforms provide flexible options to set up the compute power necessary to run computationally expensive data science processes such as forecasting at scale. With both AWS and Azure you can select different size compute instances for model training and model deployment, enable parallel processing, and set parameters that allow complete control into managing costs. The biggest challenge with setting up your compute is determining the appropriate cluster size. This can vary based on data volume, model training time, and cost. In both platforms, many compute options exist from general purpose to memory or compute optimized. Check out the following pages for more information: <a href="https://aws.amazon.com/ec2/instance-types/">AWS Compute Options</a>, <a href="https://docs.microsoft.com/en-us/azure/virtual-machines/sizes">Azure Compute Options</a>.</p><p>An added benefit of using Azure Machine Learning is having a built-in workspace section to create, manage and track the status of managed compute targets. We liked the built-in dashboard to see the status of our cluster during runs. We found this to be helpful for tracking the status of the model training jobs and being able to quickly identify if there is an unhealthy node. Below is a screenshot of the cluster status during a training pipeline run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jjFUqbsS6TGs5crX5VUsgw.png" /></figure><h3>Additional Tool Capabilities</h3><p>While both AWS and Azure provide solutions to the three challenges, each platform takes a different approach to enable forecasting at scale. AWS approaches forecasting at scale with DeepAR, a specialized model trained to handle forecasting many series. Azure’s AutoML allows for users to specify specific forecasting models with emphasis on creating pipelines to scale the process.</p><p>The table below summarizes some additional similarities and differences for implementing forecasting at scale in either platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*vOQsgvpTwtkhcqtTRA9fvQ.jpeg" /><figcaption>Table created with sheetsu</figcaption></figure><p>If you are interested in creating reusable pipelines and scheduled processes for running forecasts, Azure Machine Learning may be the tool for you. You can also set up scheduled processes in AWS, but you will need additional services outside of Sagemaker and DeepAR. On the other hand, if you require a model to continually learn and need to be able to make forecasts on data with limited history, AWS may be better suited to handle the task. Both solutions are an improvement over implementing forecasting at scale in an on-premise environment and both are continuing to evolve their products as this technology advances.</p><p><em>For more information on forecasting at scale in Azure check out the </em><a href="https://github.com/microsoft/solution-accelerator-many-models"><em>Many Models Solution Accelerator</em></a><em>. To learn more about AWS’s DeepAR check out </em><a href="https://medium.com/slalom-data-analytics/forecasting-weekly-department-sales-using-deepar-in-aws-5f7de09d0916"><em>this post</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*K6My4W9bUZCAQ_TS1DOrsg.jpeg" /></figure><p>Ashley Peterson (L) and Evan Schaeffer (R) are Consultants in Slalom’s Data and Analytics practice.</p><p><a href="https://www.slalom.com/"><em>Slalom</em></a><em> is a modern consulting firm focused on strategy, technology, and business transformation.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f17c0b859e4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scheduling & Deploying ML Pipeline Endpoints in Azure Machine Learning]]></title>
            <link>https://medium.com/@ashley.peterson/scheduling-deploying-ml-pipeline-endpoints-in-azure-machine-learning-497f57300e9d?source=rss-502ec8706481------2</link>
            <guid isPermaLink="false">https://medium.com/p/497f57300e9d</guid>
            <category><![CDATA[ml-model-deployment]]></category>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[azure-machine-learning]]></category>
            <category><![CDATA[predictions]]></category>
            <category><![CDATA[endpoints]]></category>
            <dc:creator><![CDATA[Ashley Peterson]]></dc:creator>
            <pubDate>Thu, 27 Aug 2020 16:19:41 GMT</pubDate>
            <atom:updated>2020-08-27T16:19:41.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYslOjFTIHANyeG33owWsg.jpeg" /></figure><p>By <a href="https://medium.com/@ashley.peterson">Ashley Peterson</a> and <a href="https://medium.com/@emily.huskins">Emily Huskins</a></p><p>In our <a href="https://medium.com/slalom-data-analytics/using-azure-ml-pipelines-automl-to-classify-airbnb-listings-b88f8d88916a">previous post</a>, we created a model training pipeline to classify AirBnb listings based on the dwelling attributes. After creating a Machine Learning (ML) Pipeline in Azure, the next step is to deploy the pipeline. Using the model training pipeline, we wanted to set a retraining schedule and create a second pipeline to generate predictions when a new file was added to Blob Storage.</p><p>First, we created the second pipeline, the model prediction pipeline, which would call the model we previously trained. Next, we published the two pipelines to REST endpoints. Then, we created schedules for each pipeline — one time-based and one change-based.</p><h3><strong>Creating the Model Prediction Pipeline</strong></h3><p>The model prediction pipeline has two steps. The first is the data preparation step, which mirrors the data preparation step from the model training pipeline to ensure the same features are created for predictions. The second step calls the model saved in the workspace and makes predictions on the new data. We are storing and reading our data directly from Azure Blob Storage. Within an Workspace, you can register your data and read datasets from the Workspace instead of connecting directly to a storage service.</p><h4>Step 1: Data Prep</h4><p>We used our data_prep script from the training pipeline as a starting point. The challenge we ran into with the AirBnb data was that some of the categorical features had numerous values. For example, property type had 28 possible categories. Not every category for property type was represented in the new data file, so we needed to update our script to preform <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html">One Hot Encoding</a> to include all the possible categorical values.</p><p>To allow for scheduling the pipeline later on, we created a <a href="https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.datapath.datapath?view=azure-ml-py">DataPath</a> parameter as a dynamic input to the pipeline. We will explain the DataPath parameter in the change-based scheduling section. For testing the prediction pipeline, the DataPath parameter is a static input that passes the path of our CSV file in Blob Storage.</p><pre>data_path_input = (datapath1_pipeline_param,<br>                   DataPathComputeBinding(mode = ‘mount’))<br>data_prep_step = PythonScriptStep(name = &#39;Data Preparation&#39;,</pre><pre>script_name = &#39;data_prep_predictions.py&#39;,</pre><pre>arguments = [&quot;--input_data_path&quot;, data_path_input,&quot;--string_parameter&quot;, string_pipeline_param,&quot;--output_data&quot;, cleansed_data],inputs = [data_path_input], outputs = [cleansed_data],compute_target = compute_target,runconfig = run_config,allow_reuse = True)</pre><h4>Step 2: Model Predictions</h4><p>Using the model registered to the Workspace from the AutoML Training Pipeline, we made predictions on a new set of data and output the results. The code below pulls the model registered in the Workspace and returns the predictions to a Blob container. The output_path is an input to this step and defines the path in Blob Storage where the results are stored.</p><pre>reg_model_path = Model.get_model_path(args.model_name)best_model = joblib.load(reg_model_path)<br>predictions = best_model.predict(x)</pre><pre>output_path = os.path.join(args.output_dir, &#39;predictions_&#39; +     <br>              str(datetime.datetime.now().date()))</pre><pre>airbnb.to_csv(output_path + &#39;.csv&#39;, index=False)</pre><h3><strong>Publishing the Pipelines to REST endpoints</strong></h3><p>After creating the model prediction pipeline, we leveraged the Python SDK to create a REST endpoint. Within the Workspace, you can create endpoints for either a model or a pipeline depending on how you would like to use the output. Since we were not creating a web app to make real-time predictions, we chose to create a pipeline endpoint. You also have the option to enable versioning for your pipeline, so that the endpoint remains the same even if you update the pipeline. Creating the endpoint allows us to run the pipeline outside of the Jupyter Notebook that was used to create it.</p><pre>published_prediction = pipeline.publish(<br>                        name =    <br>                         &#39;airbnb_automl_classification_predictions&#39;,<br>                        description = &#39;airbnb automl classification <br>                                       model prediction pipeline&#39;,<br>                        version = &#39;2&#39;)</pre><h3><strong>Calling the Endpoints: On-Demand and Scheduling</strong></h3><p>Once a pipeline is published, you can call that pipeline to execute a run. On-demand and Scheduled are the two options for calling the endpoints. With On-Demand the pipeline endpoint is active and can be called whenever you would like to run it. Scheduled pipelines have options to run your pipelines on a set schedule or in response to a change in the data source.</p><h4><strong>On-Demand</strong></h4><p>To test our endpoint, we used a new notebook to call the pipeline from the Workspace and submit a run. The code to run a pipeline could be used in other applications to trigger a run and return results. Using the .submit() function will run the pipeline on-demand when that line is executed.</p><pre>experiment_name2 = &#39;endpoint-run-airbnb-prediction&#39;<br>experiment2 = Experiment(ws, experiment_name2)</pre><pre>prediction_pipeline = PublishedPipeline.get(ws, id = pipeline.id)<br>run_prediction_pipeline = experiment2.submit(prediction_pipeline)</pre><h4><strong>Scheduling</strong></h4><p>Instead of having to do on-demand runs of the pipelines, we explored the options for using <a href="https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.schedule(class)?view=azure-ml-py">Scheduling</a>. Azure has two methods for scheduling: time-based and change-based. We chose to set our training pipeline on a time-based schedule and our model prediction pipeline on a change-based schedule.</p><p><em>Note: If there is a need to deactivate your pipeline, you must first cancel the schedule associated with that endpoint.</em></p><p><strong>Time-Based Schedule</strong></p><p>Creating a time schedule is a few lines of code. After you have your published pipeline, you can attach a schedule to the pipeline’s endpoint by using the pipeline’s ID. We set our training pipeline to a time-based schedule to retrain the model when a month’s worth of data had been collected.</p><pre>recurrence = ScheduleRecurrence(<br>                               frequency=&quot;Month&quot;,<br>                               interval=1,<br>                               start_time=&quot;2020-01-01T09:00:00&quot;) </pre><pre>recurring_schedule = Schedule.create(ws,<br>                           name=&quot;airbnb_training_pipeline_schedule&quot;,<br>                           description=&quot;Schedule Training Pipeline<br>                               to run on first day of every month&quot;,<br>                           pipeline_id=training_pipeline_id,<br>                           experiment_name=experiment.name,<br>                           recurrence=recurrence)</pre><p><strong>Change-based Schedule</strong></p><p>To enable scheduling based on a change in our Blob container, we created a <a href="https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.datapath.datapath?view=azure-ml-py">DataPath</a> object and PipelineParameter. The DataPath will update to the new uploaded file’s path which would be the input to the prediction pipeline. Along with the DataPath object, a <a href="https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.datapath.datapathcomputebinding?view=azure-ml-py">DataPathComputeBinding</a> needs to be defined to determine how the data will be consumed during a pipeline run. We set the mode to ‘mount’ to allow access to the data in Blob rather than downloading the dataset. We recommend using ‘mount’ when you have large amounts of data.</p><pre>from azureml.data.datapath import DataPath, DataPathComputeBinding<br>from azureml.pipeline.core import PipelineParameter</pre><pre>data_path = DataPath(datastore = dstore, path_on_datastore =  <br>                     &#39;listings_subset.csv&#39;)<br>datapath1_pipeline_param = PipelineParameter(name =  <br>                                      &#39;input_data_path&#39;,   <br>                                       default_value = data_path)<br>data_path_input = (datapath1_pipeline_param, <br>                   DataPathComputeBinding(mode = &#39;mount&#39;))</pre><pre>string_pipeline_param = PipelineParameter(name=&#39;input_string&#39;, <br>                        default_value = &#39;input_data.csv&#39;)</pre><p>To enable a change-based Schedule, we create a schedule as we did for the time-based schedule with additional parameters. The <em>datastore</em>, <em>path_on_datastore</em>, <em>polling _interval</em>, and <em>data_path_parameter_name</em> are set. These parameters define where to watch for changes to trigger a pipeline run. The <em>datastore</em> and <em>path_on_datastore</em> define where you would like to watch for changes. Our datastore is our Blob container and we want to watch for any changes in the container so the path is set to ‘/’. Setting a change-based schedule will only monitor the path given — it will not trigger based off changes in the subfolders. The <em>polling_interval</em> is the time in minutes that you want to check for changes in the data source. The <em>data_path_parameter_name</em> is the name of the object defined above in the datapath1_pipeline_param. Our prediction pipeline is set to check the airbnb-data container hourly for new data.</p><pre>datastore = Datastore(workspace=ws, name=&quot;airbnb&quot;)<br>pipeline_id = pipeline.id<br>experiment_name3 = &quot;blob_trigger_airbnb_predictions&quot;</pre><pre>reactive_schedule = Schedule.create(ws,<br>                           name=&quot;Blob_trigger_schedule&quot;,<br>                           description=&quot;Based on input file change <br>                                        in airbnb-data container.&quot;,<br>                           pipeline_id=pipeline_id,<br>                           experiment_name=experiment_name3,<br>                           datastore=datastore,<br>                           path_on_datastore = &#39;/&#39;,<br>                           polling_interval = 60,<br>                           data_path_parameter_name=<br>                                             &#39;input_data_path&#39;)</pre><p>After setting up the schedule, you can check the status using the code below.</p><pre>schedules = Schedule.list(ws)</pre><pre>for schedule in schedules:<br>    x = Schedule.get(ws, schedule.id)<br>    print(&quot;Schedule id {} status {}&quot;.format(x.id, x.status))</pre><p>Both pipelines are now set to run automatically to provide predictions on new data and to retrain the model monthly.</p><p>Azure Machine Learning provides an easy way to create REST endpoints to deploy ML pipelines. Once you have created pipelines for model training and model predictions, you can determine the best deployment option to enable your model or pipeline to be consumed. With the Scheduler, you can set your pipelines to run weekly or as you received data updates. If you are looking to build a web app, the on-demand call of the pipeline would be a good choice to get predictions as needed. Once the output is landed in a datastore, PowerBI dashboards can be created to monitor and explore the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*P0dBFa4yWNReQAYhr_z9fQ.jpeg" /></figure><p>Ashley Peterson (L) is a Consultant in Slalom’s Data and Analytics practice. Emily Huskins (R) is an Analyst in Slalom’s Data and Analytics practice.</p><p><a href="https://www.slalom.com/"><em>Slalom</em></a><em> is a modern consulting firm focused on strategy, technology, and business transformation.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=497f57300e9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Azure ML Pipelines & AutoML to Classify AirBnb Listings]]></title>
            <link>https://medium.com/@ashley.peterson/using-azure-ml-pipelines-automl-to-classify-airbnb-listings-b88f8d88916a?source=rss-502ec8706481------2</link>
            <guid isPermaLink="false">https://medium.com/p/b88f8d88916a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[automl]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ashley Peterson]]></dc:creator>
            <pubDate>Tue, 09 Jun 2020 17:03:35 GMT</pubDate>
            <atom:updated>2020-06-09T17:03:35.400Z</atom:updated>
            <content:encoded><![CDATA[<h3>Using Azure ML Pipelines &amp; AutoML to Classify Chicago AirBnb Listings</h3><p>Developing a reusable Azure ML Pipeline — by <a href="https://medium.com/@ashley.peterson">Ashley Peterson</a>, <a href="https://medium.com/@rachel.wiseley">Rachel Wiseley</a>, and <a href="https://medium.com/@alden.nail">Alden Nail</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*WdPcfeNRyiP9Kagj3vCtOA.png" /><figcaption>Model results for classifying ‘Top Stays’ and ‘Fixer Uppers’.</figcaption></figure><p>Data scientists need numerous “tools in their toolbox” to successfully develop, train, and deploy a model. They need to be able to prepare data for modeling, experiment with many models and parameters, set up a retraining process, and deploy their model to gain insights from the predictions. Each of the steps in the modeling process can be time consuming and require multiple tools and skills. With Azure’s Machine Learning Service, a data scientist can organize their experiments, track results, store and deploy models, and speed up their process. Two tools that enable a data scientist to experiment quicker and set up repeatable processes are Automated Machine Learning (AutoML) and Machine Learning Pipelines (ML Pipelines). AutoML democratizes the machine learning model development process.[<a href="#_ftn1">1</a>] ML Pipelines allow data scientist to develop repeatable workflows.</p><p>In this post, we will leverage Azure’s AutoML and ML Pipelines to train a classification model.</p><p>The key tools we used to develop the model are:</p><ul><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/overview-what-is-azure-ml"><strong>Azure Machine Learning Service</strong></a> — A cloud-based environment for training, tracking, and deploying machine learning models</li><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines"><strong>Azure Pipelines</strong></a> — A framework to create reusable machine learning workflows</li><li><a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-automated-ml"><strong>Azure AutoML</strong></a><strong> </strong>— A machine learning model that trains many models according to the input problem type of regression, classification, or forecasting</li><li><a href="https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-overview"><strong>Azure Blob Storage</strong></a> — Azure’s object storage service</li></ul><p>First, we will explore the data that we used to train a classification model. Then, we will dive into the pipeline and AutoML capabilities in Azure.*</p><p><em>*Note we have already set up a Machine Learning Workspace for this. To set up your workspace you can follow these </em><a href="https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb"><em>instructions</em></a><em>.</em></p><h3>Overview of Airbnb Dataset and Classification Use Case</h3><p>If you are looking to predict between two categories such as a win/loss or yes/no, then you would want to use a classification model. Some classification use cases include predicting whether a customer will default on a financial loan, classifying emails as spam, or determining if an image contains a specific item, like a car or snow-leopard. For this demo, we created a ML Pipeline to classify Airbnb properties from Chicago as “top stays” or “fixer-upper”. A “top stay” was defined as a listing where the average rating is greater than or equal to 90 (out of 100 ).</p><p>The data contained host-specific as well as listing-specific information that was used in developing the model. Feature selection was used to remove correlated variables; feature engineering was performed to clean up the date and categorical features. This data cleanup is done in the <strong>data_prep.py</strong> script that is called in the first step of the pipeline.**</p><p><em>**Note: We will not go into the details on the model itself. Instead we will focus on utilizing the AutoML and ML Pipeline features of Azure Machine Learning to quickly set up a machine learning workflow.</em></p><h3>Setting up the Machine Learning Pipeline and AutoML Step</h3><p>Machine Learning Pipelines are one piece of the larger MLOps framework in Azure. ML Pipelines allow you to create reusable workflows and track the results of each run. Within the pipeline, steps modularize the modeling process. The decoupled code enables easier debugging and development.</p><p>Our pipeline contains four steps: Data Preparation, Train-Test Split, AutoML, and Model Registration. A benefit of ML Pipelines is the ability to automate the pipeline to run again to retrain your model when new data is made available.</p><p>The steps of the ML Pipeline act like an outline for the process. Each step references a script. When the Pipeline is run, the steps call the referenced code and executes it. <a href="#_ftn1">[2]</a></p><h4>ML Pipeline Setup</h4><p>Before creating the steps of a pipeline, a few items need to be set up or referenced:</p><ul><li>Create the reference objects for the workspace and the datastore</li><li>Create a compute cluster (if not already done so with workspace set up) to run the pipeline on</li><li>Create the directory for the scripts which will be referenced by the steps of the pipeline</li><li>(Optional) You can reference/read data located in Blob and other Azure services. Additionally, you can register your data to the workspace to create a data reference. In our setup, we registered the raw data to our workspace.</li><li>Create the <a href="https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.runconfig.runconfiguration?view=azure-ml-py">RunConfiguration</a> for the scripts. This is where you set up the environment you need for the scripts to run. You can pip or conda install any dependencies. For pipelines and automl, we included <a href="https://docs.microsoft.com/en-us/python/api/overview/azure/ml/install?view=azure-ml-py">azureml-sdk</a>, <a href="https://docs.microsoft.com/en-us/python/api/azureml-dataprep/?view=azure-ml-py">azureml-dataprep</a>, <a href="https://docs.microsoft.com/en-us/python/api/azureml-automl-core/?view=azure-ml-py">automl,</a> and <a href="https://docs.microsoft.com/en-us/python/api/azureml-automl-runtime/?view=azure-ml-py">azureml-automl-runtime</a>.</li></ul><h4>Step 1: Data Preparation</h4><p>Now that our environment is set up, the first step in the Pipeline is the Data Preparation Step. This step pulls the dataset from the workspace and performs transformations and updates to the features. We created a <a href="https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.python_script_step.pythonscriptstep?view=azure-ml-py">PythonScriptStep</a> to perform the custom data preparation. The step calls a python script, <strong>data_prep.py</strong>, that contains the python code to do the transformations and clean up.</p><p>The key parameters to set in the step are the <em>outputs</em> and<em> arguments</em>. The output is a <a href="https://docs.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py">PipelineData</a> object which enables the next step to read in the cleaned data. The arguments parameter passes information into the script that the step calls. We are passing input_data and output_data as arguments to the<strong> data_prep.py</strong> script. Within the <strong>data_prep.py</strong> script, the passed arguments are strings, so we passed the dataset name,<em> airbnb</em>, into the script.</p><p><strong>Data Preparation Step</strong></p><pre># defined output after data prep step <br>cleansed_data = PipelineData(&quot;airbnb_cleaned&quot;, datastore=dstore)<br><br># create the Python Pipeline step <br>data_prep_step = PythonScriptStep(name = &#39;Data Preparation&#39;,<br>                                  script_name = &#39;data_prep.py&#39;,<br>                                  arguments = [&quot;--input_data&quot;,   <br>                                             &quot;airbnb&quot;,<br>                                             &quot;--output_data&quot;,      <br>                                             cleansed_data],<br>                                  inputs =   <br>                          [airbnb.as_named_input(&#39;airbnb_dataset&#39;)],<br>                                  outputs = [cleansed_data],<br>                                  compute_target = compute_target,<br>                                  runconfig = run_config, <br>                                  allow_reuse = True <br>                                )</pre><p><strong>Data_prep.py</strong></p><p>Our initial input data is registered to the Workspace and we gave it the name ‘<em>airbnb_dataset</em>’ in the input parameter for the data_prep_step. In the script, we can call the dataset by using the following code:</p><pre>from azureml.core import Run, Dataset</pre><pre>run_context = Run.get_context()<br>airbnb_dataset = run_context.input_datasets[‘airbnb_dataset’]<br>airbnb = airbnb_dataset.to_pandas_dataframe()</pre><p>After the step runs, the output is saved in the cleansed_data PipelineData object we created.</p><h4>Step 2: Train Test Split</h4><p>The second step is the Test Train Data Split Step. The set up for this step is like the Data Preparation step. It is also a PythonScriptStep with code to split the data into training and test sets. The input parameters for this script are the output from the Data Preparation step, the training and test datasets.</p><pre># test train split the data <br>output_train = PipelineData(&quot;output_train&quot;, datastore = dstore)     <br>output_test = PipelineData(&quot;output_test&quot;, datastore = dstore)<br><br>test_train_step = PythonScriptStep(name = &quot;Test Train Data Split&quot;,<br>                                 script_name =&quot;train_test_split.py&quot;,<br>                                 arguments = [&quot;--input_data&quot;,     <br>                                              cleansed_data,<br>                                              &quot;--output_train&quot;, <br>                                              output_train,<br>                                              &quot;--output_test&quot;, <br>                                              output_test],<br>                                  inputs = [cleansed_data],<br>                                  outputs = [output_train, <br>                                             output_test],<br>                                  compute_target = compute_target, <br>                                  runconfig = run_config, <br>                                  allow_reuse = True<br>                                  )</pre><h4>Step 3: AutoML</h4><p>We used the <a href="https://docs.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.automl_step.automlstep?view=azure-ml-py">AutoMLStep</a> to train our model. This step is setup with parameters like the PythonScriptStep, and it also needs the <a href="https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig?view=azure-ml-py">AutoMLConfig</a> settings. These settings are used to control what parameters you would like to use for your models. You can set the primary_metric to score on, the max time for the experiment to run, the number of cross-validations, etc. This is also where you set what type of modeling you would like to run. AutoML can perform classification, regression or forecasting.</p><p>The configuration uses the data_script parameter to read in the PipelineData from the previous step. With the latest update, AutoML uses Datasets that are in the Workspace. In order to read in the PipelineData you need to create a <strong>get_data.py</strong> file with a function called get_data() that pulls in the data from the previous step.***</p><p><em>***Note get_data() will be deprecated in the future, then the AutoML step will need an input of a Dataset. The commented out label_column and training_data will need to be used.</em></p><p><strong>AutoML Config</strong></p><pre>import logging<br>from azureml.train.automl import AutoMLConfig</pre><pre>#label = ‘review_scores_rating_binned’<br>project_folder = ‘./’</pre><pre>automl_settings = {<br> “iteration_timeout_minutes”: 2, <br> “n_cross_validations”: 3,<br> “primary_metric”: ‘accuracy’,<br> #“preprocess”: True,<br> #&quot;featurization&quot;: &#39;auto&#39;,<br> “enable_early_stopping”: True, <br> “max_concurrent_iterations”: 4,<br> “max_cores_per_iteration”: -1,<br> “verbosity”: logging.INFO,<br>}</pre><pre>automl_config = AutoMLConfig(task = ‘classification’,<br>                             experiment_timeout_minutes = 15,<br>                             #whitelist_models= [],<br>       #training_data = Dataset.get_by_name(ws, name = ‘air_train’),<br>                             #label_column_name = label,<br>                             data_script = ‘get_data.py’,<br>                             compute_target = compute_target, <br>                             path = project_folder,<br>                             **automl_settings<br>                             )</pre><pre>print(“AutoML config created.”)</pre><p><em>Note: Using logging with the AutoMLConfig will provide more details around the errors to help with debugging. You can set the appropriate logging level. For more details, visit this </em><a href="https://docs.python.org/3/library/logging.html#levels"><em>site</em></a><em>.</em></p><p>After setting up the AutoMLConfig, you can create the AutoMLStep. The input for the AutoMLStep is the PipelineData that the <strong>get_data.py</strong> script will use. Because AutoML runs through many combinations of models, this step can take some time to run. You set a limit on the time with the experiment_timeout_minutes and iteration_timeout_minutes parameters. The speed is also impacted by the size of the compute instance you have set up.</p><p><strong>AutoML Step</strong></p><pre>from azureml.train.automl import AutoMLStep<br>from azureml.pipeline.core import PipelineData, TrainingOutput</pre><pre>metrics_output = ‘metrics_output’<br>best_model_output = ‘best_model_output’</pre><pre>metrics_data = PipelineData(name = ‘metrics_data’, <br>                           datastore = dstore,<br>                           pipeline_output_name = metrics_output, <br>                   training_output = TrainingOutput(type=’Metrics’))<br>model_data = PipelineData(name = ‘model_data’,     <br>                          datastore=dstore,<br>                          pipeline_output_name=best_model_output, <br>                     training_output = TrainingOutput(type=’Model’))</pre><pre>automl_step = AutoMLStep(<br>                         name=’AutoML_Classification’,<br>                         automl_config=automl_config,<br>                         inputs=[output_train],<br>                         outputs = [metrics_data, model_data],<br>                         allow_reuse=True <br>                         )</pre><p>This step outputs two PipelineData objects with the metrics and model results to read into the final step.</p><h4>Step 4: Model Registration</h4><p>The last step in the pipeline is the Model Registration step. In this step, we register the best model to the workspace for deployment and future use. The model will show in the workspace’s Model tab. The model output from the AutoML step is read in as the input for this step.</p><p><strong>Model Registration Step</strong></p><pre>model_reg_step = PythonScriptStep(name = “Model Registration”,<br>                                  script_name = “register_model.py”,<br>                                  arguments = [“ — input_data”,    <br>                                               output_test, <br>                                               “ — best_model”,  <br>                                               best_model_output, <br>                                               “ — model_path”,  <br>                                               model_data, <br>                                               “ — train_data”,<br>                                               output_train],<br>                                  inputs = [output_test, model_data,<br>                                           output_train],<br>                                  compute_target = compute_target, <br>                                  runconfig = run_config<br>                                  )</pre><h4>Putting It All Together</h4><p>To connect the steps and run the workflow, you need to create a Pipeline. We used the StepSequence to confirm that our steps would execute in order. You can run the Pipeline in the Jupyter Notebook using the experiment.submit(pipeline) function or you can run the pipeline from the User Interface in the Pipelines tab.</p><pre>from azureml.pipeline.core import Pipeline, StepSequence<br>from azureml.widgets import RunDetails</pre><pre># build pipeline<br>four_steps = StepSequence(steps = [data_prep_step, test_train_step, <br>                                   automl_step, model_reg_step])<br>Pipeline = Pipeline(workspace = ws, steps= four_steps)</pre><pre># run the pipeline <br>pipeline_run = experiment.submit(pipeline)</pre><p>Now that the Pipeline is created, we can run it and view the results of the AutoML step and confirm the model registered to the workspace.</p><p>You can track the progress of a run in the experiments tab and see what steps have completed successfully. If a step fails, you can investigate the logs and output that is generated. You can add your own notes and logs within the scripts for debugging and they will print in the log files of the step.</p><h3>AutoML Output</h3><p>The pipeline has one output, the model that is registered to the workspace, and it also produces details on the model runs. Once the AutoML step has completed you can explore the models that were run and their scores. The best model will be selected based on the accuracy metric you defined. Under the models tab you can explore the other models that ran along with additional accuracy metrics.</p><p>Two additional resources are the Visualizations and Explanations. For classification, the Visualizations detail the model results in the Precision Recall, ROC, Calibration Curve, Lift Chart, Gain Curve and the Confusion Matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1XrpGXQ28mGfyf4o_JcwKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFd_Iv6X3o0NdagpuG3yfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4Vei_aLd6I_Fc5kq-TbLg.png" /></figure><p>The Explanations tab allows you to run an additional experiment to return the details of the model. This allows you to explore the feature importance. An option for AutoML is enabling <a href="https://docs.microsoft.com/en-us/python/api/azureml-train-automl-client/azureml.train.automl.automlconfig.automlconfig?view=azure-ml-py">auto featurization</a> which we leveraged for our model development. The top 8 created features can be seen below in the variable importance graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kRim6s0e-1G2gT3Ji0QuuQ.png" /></figure><p>These two tools allow a data scientist to quickly explore and interpret the model results instead of having to spend time creating these charts after a model runs.</p><h3>Next Steps</h3><p>With the reusable Pipeline, you can set up your model retraining schedule or re-run the Pipeline on demand. The next steps would be to publish the pipeline for future retraining and deploy the model to an API endpoint to use for predictions.</p><p>Machine Learning Pipelines and AutoML enable data scientists to quickly train and experiment with many models. Instead of having to write code for each algorithm they want to test, they can specify the modeling type or the specific list of models to try. Developing the Machine Learning Pipeline allows for integration into production and deployment of a model. Once you have a model framework set up, you can set a retraining schedule or use the model framework with a new dataset and update only pieces of the pipeline. Azure Machine Learning speeds up the modeling process and allows a data scientist to experiment quicker.</p><p><strong>References and Resources</strong></p><p>[<a href="#_ftnref1">1</a>] <a href="https://docs.microsoft.com/en-us/azure/machine-learning/concept-automated-ml">https://docs.microsoft.com/en-us/azure/machine-learning/concept-automated-ml</a></p><p><a href="#_ftnref1">[2]</a> <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-your-first-pipeline">https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-your-first-pipeline</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b88f8d88916a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>