<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jade Zhou on Medium]]></title>
        <description><![CDATA[Stories by Jade Zhou on Medium]]></description>
        <link>https://medium.com/@greenorange1994?source=rss-e58ec2c1f48c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*njWW24EnRkxWaEP_PbcMVg.jpeg</url>
            <title>Stories by Jade Zhou on Medium</title>
            <link>https://medium.com/@greenorange1994?source=rss-e58ec2c1f48c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:02:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@greenorange1994/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Build Entity Recognition Models in a Jiffy using Watson Discovery-III]]></title>
            <link>https://medium.com/inside-machine-learning/how-to-build-entity-recognition-models-in-a-jiffy-using-watson-discovery-iii-977da5e8f28c?source=rss-e58ec2c1f48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/977da5e8f28c</guid>
            <category><![CDATA[watson-discovery]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[watson]]></category>
            <dc:creator><![CDATA[Jade Zhou]]></dc:creator>
            <pubDate>Wed, 16 Oct 2019 18:48:37 GMT</pubDate>
            <atom:updated>2019-10-16T18:53:08.336Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Written by </strong><a href="https://medium.com/@KunalSavvy"><strong>Kunal Sawarkar</strong></a><strong> &amp; </strong><a href="https://medium.com/@greenorange1994"><strong>Jade Zhou</strong></a></p><p><em>A step by step guide to understand how data scientists can upend their game with sophisticated but efficient handling of NLP issues — while exploiting the integration of Watson Discovery with Jupyter Notebooks for business applications.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*owjWBDMHkeyQ3DA_EL1mHQ.png" /></figure><p>This is a final chapter in a three part series on this topic. You can see previous stories <a href="https://medium.com/inside-machine-learning/how-to-build-entity-recognition-models-in-a-jiffy-using-watson-discovery-i-b67d41b71281">here</a> about <em>how Watson Discovery can help business analysts jump-start entity recognition model building with zero coding. And how Watson Knowledge Studio empowers subject matter experts with a self-service domain-aware entity extraction capabilities.</em></p><p>Although Watson Discovery has good a graphical interface for people who are not familiar with coding to build their own entity recognition models, we also provide APIs in different languages for developers to integrate Watson Discovery service into their applications. Continuing on our entity problem, even after company entities have been identified in emails, it’s by no means the end of story. These entities are often used for other purposes like email routing.</p><p>Let’s assume you have multiple account managers to implement trading demands. “Peter” is one of these managers and is responsible for a bank beginning with “A”. “Mary” is another manager responsible for a different bank beginnig with “B”. To avoid Peter and Mary having to read all emails, we need an engine that can analyze various trading actions across different banks referred to in numerous emails — then allocate them to the corresponding account manager. This is where Watson Discovery APIs can be leveraged. Now let’s build that engine by consuming results from the Watson Discovery API to build the engine in python.</p><p>First, we need to connect to the API using the correct credentials. Username and password refer to your account information on Cloud Pak for Data. The url can be found in the Watson Discovery instance page below.</p><pre>from ibm_watson import DiscoveryV1 </pre><pre>discovery = DiscoveryV1( version=&#39;2019-08-21&#39;, username=&#39;***&#39;, password=&#39;***&#39;, icp4d_url=***&#39;, authentication_type=&#39;icp4d&#39;, url=&#39;***&#39; ) </pre><pre>discovery.disable_SSL_verification()</pre><p>Next, let’s send an empty query to our collection to take a look at what the response looks like.</p><pre>import json </pre><pre>env_id = &#39;default&#39; </pre><pre>collections = discovery.list_collections(env_id).get_result()<br> query_results = discovery.query(env_id, collections[&#39;collections&#39;][4][&#39;collection_id&#39;], return_fields=&#39;enriched_text&#39;, count = 93).get_result()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*MoziaN_KC6aadMb3.png" /><figcaption>Fig 1. Query Results From Watson Discovery Python API</figcaption></figure><p>Basically, the response covers document_id and its enrichment. It extracts company entities which we will use for building the engine. If you add other enrichment like sentiment analysis, it will show up here as well.</p><p>Finally, we can get the list of corresponding emails for different managers based on what entities we found in the email.</p><pre>def email_allocate(company_lst):<br>    eml_lst = []<br>        for doc in query_results[‘results’]:<br>            try:<br>                enrichment = doc[‘enriched_text’]<br>                for entity in enrichment[0][‘entities’]:<br>                    if entity[‘text’].replace(u’\ufeff’, ‘’) in company_lst:<br>                    eml_lst.append(doc[‘document_id’])<br>                    break<br>    except:<br>        continue<br>    return eml_lst</pre><pre>peter_company_lst = [‘bank of america’, &#39;boa&#39;, &#39;bofa&#39;, &#39;bac&#39;]<br>mary_company_lst = [‘barclay’, &#39;BAR&#39;]</pre><pre>peter_email_list = email_allocate(peter_company_lst)<br>mary_email_list = email_allocate(mary_company_lst<br></pre><p>Now, it’s time to embed this function into your enterprise application to finish the email routing task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VLesFIQ_RfZe-Vtr.png" /><figcaption>Fig 2. Routed Email List</figcaption></figure><p>Peter and Mary should now receive emails which are automatically extracted by their relevance and responsibility. The emails are routed to them which significantly improves process efficiency. This can be integrated to any front-end application for action and visualization. Note that extracted entities include all variations, abbreviations and formats for a given company name. So Peter will receive emails that have actionable items for Bank of A, BOA , bofa, BAC, etc,.</p><p>Lastly, data scientists can combine results from Watson Discovery with other NLP packages (like NLTK or LDA modeling results) using the same methods. This will continue to extend and integrate the above model to the NLP pipeline.</p><p>Just to recap, Watson Discovery on Cloud Pak for Data includes the items shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sa-uorTnph4FEM8q.png" /><figcaption>Fig 3. Overview of Watson Discovery on Cloud Pak For Data</figcaption></figure><ul><li>Smart Document Understanding - Visually annotate and enrich documents, including image formats using OCR</li><li>Integration with Watson Knowledge Studio for domain specific problems</li><li>Identify entities in customer conversations using Watson Assistant</li></ul><h4>Github Link:</h4><p><a href="https://github.com/greenorange1994/EmailRoutingByWatsonDiscovery">https://github.com/greenorange1994/EmailRoutingByWatsonDiscovery</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=977da5e8f28c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-machine-learning/how-to-build-entity-recognition-models-in-a-jiffy-using-watson-discovery-iii-977da5e8f28c">How to Build Entity Recognition Models in a Jiffy using Watson Discovery-III</a> was originally published in <a href="https://medium.com/inside-machine-learning">Inside Machine learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Survival Analysis]]></title>
            <link>https://medium.com/inside-machine-learning/survival-analysis-1b1793fb8195?source=rss-e58ec2c1f48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b1793fb8195</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[survival-analysis]]></category>
            <dc:creator><![CDATA[Jade Zhou]]></dc:creator>
            <pubDate>Wed, 02 Oct 2019 20:46:17 GMT</pubDate>
            <atom:updated>2019-10-02T22:44:27.612Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2 — An Example: How to Predict Future Repair Amount?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*WInAIPodSU2rj8eF.png" /></figure><h3><strong>Introduction</strong></h3><p>The idea of survival analysis comes from a businessman, <a href="http://userwww.sfsu.edu/efc/classes/biol710/survival/surv-anal.htm">John Gaunt</a>. He built the life table including 3 columns (Age, Died, Survived) to analyze mortality statistics in London. Thirty years after Mr.Gaunt publishing his book, Edmund Halley started to apply mathematics representation to life table and survival analysis gradually became a mature research field.</p><p><a href="https://en.wikipedia.org/wiki/Survival_analysis">Survival Analysis</a> is a branch of statistics for analyzing the expected duration of time until one or more events happen, such as death of patients and failure in mechanical systems. It can answer questions like after certain time, what percent of patients can still be alive? However, we don’t need to limit the usage of it to only survival analysis problem. Continuing on the <a href="https://medium.com/inside-machine-learning/survival-analysis-cb5832ffcd78">blog</a> from my colleague, <a href="https://medium.com/@KunalSavvy">Kunal</a>, we will show you how to reshape a repair amount prediction question, usually considered as time series forecasting problem, into survival analysis structure and how to fit survival models to get predictions. Data sets come from <a href="https://www.kaggle.com/c/pakdd-cup-2014">Kaggle — ASUS Malfunctional Components Prediction</a>. The goal is to predict future repair amount for each module-component pair at monthly level.</p><p>We use lifelines package for fitting survival models.</p><h3>0. Survival Analysis Basics</h3><p>Please refer to <a href="https://lifelines.readthedocs.io/en/latest/Survival%20Analysis%20intro.html">this page</a> to get familiar with key concepts in survival analysis. Make sure you understand following 3 concepts and 2 probabilities before reading this notebook: <br>- Event <br>Traditional: If a patient is dead. <br>Current: If a component needs to be repaired. <br>- Duration <br>How long before the event happens. <br>- Censoring <br>After observed period, we call those subjects which we didn’t observe event happening on as right-censored data. <br>- Survival Function <br>It defines the probability the death event has not occurred yet at time 𝑡. <br>S(t) = Pr(T &gt; t)<br>- Hazard Function <br>The probability of the death event occurring at time 𝑡, given that the death event has not occurred until time 𝑡.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*Aj7HEBcRu7nVyObT3M4tIA.png" /><figcaption>Fig1. Hazard Function</figcaption></figure><h3><strong>1. What data we have?</strong></h3><p>Basically, we have three tables:<br> 1) SaleTrain.csv<br> It contains information about date of sales and number of sales for each module-component pair. We need to sum over number of sales group by module, component and date of sale.<br> 2) RepairTrain.csv<br> It contains information about date of sales, date of repair and amount of repair for each pair. We need to sum over number of repairs group by module, component, date of sale and date of repair. Then we will merge “SaleTrain” table and “RepairTrain” table to get one input table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uVGGrsb_9sq6dng8.png" /><figcaption><strong>Fig 1. Input Table after Preprocessing</strong></figcaption></figure><p>3) Output_TargetID_Mapping.csv</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DLZRS8Qj62iMfHsa.png" /><figcaption><strong>Fig 2. Output Table after Preprocessing</strong></figcaption></figure><p>As shown in above pictures, we defined some new columns including “sale_date” and “repair_date”, converting actual dates into month index. I defined earliest month in data sets as my month 1 and then calculate the difference in how many months between sale/repair date and month 1. This step will simplify our later modeling and predicting process.</p><h3><strong>2. Analyze current problem under survival analysis setting?</strong></h3><p>It’s natural to consider our example as time series forecasting problem, so how to make up for the gap between time series and survival analysis? Actually, in every survival analysis problem, we have key concepts like right-censor waiting to define. Following this structure, we start to fill in blanks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*vPg2aqogRvaM02tNNUmiAA.png" /><figcaption><strong>Fig 3. Fill in Key Concepts</strong></figcaption></figure><p>Based on above table, we can rephrase the problem in matchmatical way. Actually, we need to predict future monthly cumulative hazard ratio for each component. Then apply following formula to get future repair amount of components regarding specific module and specific sale date:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*IjEKM-slnJbU3fTGV1NxIg.png" /><figcaption><strong>Fig 5. Math Representation of Our Problem</strong></figcaption></figure><p>Finally sum over the amount by component, module and repair date to map into output table Kaggle needs.</p><p>Back to our data, event is if the component was sent back to repair, so we need to add one column to called “event’ and values are all 1. Then duration, which is called “months_after_sold”, is how long the repair of specific component was requested after sale. As for right-censor, it means we didn’t observe the event during experiment period, so that we cannot rule out the possibility of event beyond time frame. A common misunderstanding about right-censor is the target for each survival model is event or not, but actually right-censored or not. Therefore, it’s really important to find our right-censored data which requires a trick. For each module-component pair, we already know the amount of sales in total and how many repairs were requested during observed period, so the deduction of them can get how many components are still “alive” after observed period. Therefore, we define right-censored data to be the number of components which didn’t require repairing during observed period here and the corresponding value in “event” column is 0. Finally, we define the number of events happening in one month as frequency.</p><pre>final_repair_date = round(((end_date - start_date)/np.timedelta64(1, &#39;M&#39;)))</pre><pre>def add_censor(x):</pre><pre>    new_row = pd.DataFrame(x.iloc[-1, :]).transpose()</pre><pre>    new_row[&#39;repair_date&#39;] = final_repair_date</pre><pre>    new_row[&#39;months_after_sold&#39;] = new_row[&#39;repair_date&#39;] - new_row[&#39;sale_date&#39;]</pre><pre>    new_row[&#39;event&#39;] = 0</pre><pre>    new_row[&#39;freq&#39;] = new_row[&#39;number_sale&#39;] - sum(x[&#39;freq&#39;])</pre><pre>    return x.append(new_row)</pre><pre>df_input = df_input.groupby([&#39;module_category&#39;, &#39;component_category&#39;, &#39;sale_date&#39;]).apply(add_censor).reset_index(drop = True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/0*jVtITr5NpFTSE5Ml.png" /><figcaption><strong>Fig 6. Input Table after Adding Right-Censored Data</strong></figcaption></figure><h3><strong>3. What models we will use?</strong></h3><p>In general, there are two kinds of models for survival analysis. The first type is univariate models, only taking information about time, right-censor and frequency. The second one is regression models which can incorporate influences from exogenous variables as well. In this blog, I’ll start with a very simple non-parametric model, Nelson-Aalen fitter, first and go through steps in detail and then mention cox regression not that deep because you can follow similar steps for Nelson-Aalen fitter.</p><p><strong>Modeling and Predicting — Nelson-Aalen Fitter</strong></p><ul><li>Modeling: Component Level Fitter</li></ul><p>Since missing data is all around, we will come into a problem that there are not enough observations for specific component-module pair to fit a model. Therefore, we can fit data at component-level first, which means fitting a model for every component, and then use predictions of repair amount for corresponding component to fill in missing pieces.</p><pre>from lifelines import NelsonAalenFitter<br>naf = lifelines.NelsonAalenFitter(nelson_aalen_smoothing = False)<br>comp_hazard = pd.DataFrame(columns = [&#39;component_category&#39;, &#39;months_after_sold&#39;, &#39;hazard&#39;])<br>for component in df_input[&#39;component_category&#39;].unique():<br>    print(&#39;Current component %s&#39; % component)<br>    cur_df = pd.DataFrame(columns = [&#39;component_category&#39;, &#39;months_after_sold&#39;, &#39;hazard&#39;])<br>    df_input_specific = df_input[df_input[&#39;component_category&#39;].str.match(component)]<br>    naf.fit(durations = df_input_specific[&#39;months_after_sold&#39;], event_observed = df_input_specific[&#39;event&#39;], weights = df_input_specific[&#39;freq&#39;])<br>    cur_df[&#39;months_after_sold&#39;] = naf.cumulative_hazard_.index<br>    cur_df[&#39;hazard&#39;] = naf.cumulative_hazard_.diff().fillna(0).values<br>    cur_df[&#39;component_category&#39;] = component<br>    comp_hazard = pd.concat([comp_hazard, cur_df])<br>    naf.plot_hazard(bandwidth=3)<br>plt.show()</pre><p>From above codes, you can feel it’s pretty simple to fit a survival model. You just need to specify duration, event and weight. That’s why we spent much time explaining how to find corresponding pieces in second part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*rbrOOoufaS0RoYyL.png" /><figcaption><strong>Fig 7. Estimated Monthly Cumulative Hazard Ratio at Component Level</strong></figcaption></figure><p>To give you a rough image of specific hazard ratio, I set bandwith = 3 for visualization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*vvxyjmNSnT1kSVS7.png" /><figcaption><strong>Fig 8. Estimated Hazard Ratio for Component P04</strong></figcaption></figure><p>In above figure, the unit of X axis is month and Y axis represents hazard ratio. This picture indicates a pattern that the risk of repair gets higher and higher at the beginning and then fluctuates within small range until 24 months after being sold. Then the risk decreases sharply and finally stabilizes around 0. I found similar patterns from most of components. Some people in Kaggle community said the reason might be warranty period is 2 years, so after 24 months, repair costs money so that people stop it.</p><ul><li>Modeling: Module-Component Level Fitter</li></ul><p>Run similar codes as above but at module-component level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/0*1JJ238d7Pb3_epHm.png" /><figcaption><strong>Fig 9. Estimated Monthly Cumulative Hazard Ratio at Module-Component Level</strong></figcaption></figure><ul><li>Predicting</li></ul><p>Once we have two above tables, we will first merge output table with the hazard ratio table for module-component pairs and then merge results from component level model with output table to fill in missing values from hazard column.</p><pre>df_final_output = df_output.merge(mod_comp_hazard[[&#39;module_category&#39;, &#39;component_category&#39;, &#39;months_after_sold&#39;, &#39;hazard&#39;]], left_on = [&#39;module_category&#39;, &#39;component_category&#39;, &#39;months_after_sold&#39;], right_on = [&#39;module_category&#39;, &#39;component_category&#39;, &#39;months_after_sold&#39;], how = &#39;left&#39;)</pre><pre>df_final_output_nona = df_final_output[~df_final_output[&#39;hazard&#39;].isnull()]<br>df_final_output_na = df_final_output[df_final_output[&#39;hazard&#39;].isnull()].drop(&#39;hazard&#39;, axis = 1)<br>df_final_output_na = df_final_output_na.merge(comp_hazard, left_on = [&#39;component_category&#39;, &#39;months_after_sold&#39;], right_on = [&#39;component_category&#39;, &#39;months_after_sold&#39;], how = &#39;left&#39;)<br>df_final_output_na.isnull().sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*Mb-4-L-G9BZO9UEd.png" /><figcaption><strong>Fig 10. Number of Missing Values for Each Column</strong></figcaption></figure><p>However, there are still lots of missing values, so we need to look into them and find ways of dealing that. After some exploration, we can conclude patterns for module-component pairs or components who have missing values:</p><ul><li>Duration is beyond the range of fitted models</li><li>There are not enough observations to fit a model whose time frame is long enough</li><li>Missing monthly cumulative hazard ratio for specific months</li></ul><p>To finish predictions simply, we propose preliminary ways to deal with above cases correspondingly:</p><ul><li>Replace missing values with 0 because the estimated hazard ratio plot shows ratio will stabilize around 0 in far future for these components.</li><li>Fit linear models on monthly cumulative hazard ratio based on historical data.</li><li>Directly interpolate hazard ratios.</li></ul><p>Lastly, we multiply number of sales by monthly cumulative hazard ratio and sum over that by module, component and repair date to get final predictions.</p><pre>df_final_output[&#39;pred_repair&#39;] = df_final_output[&#39;number_sale&#39;] * df_final_output[&#39;hazard&#39;]<br>df_final_output = df_final_output.groupby([&#39;module_category&#39;, &#39;component_category&#39;, &#39;repair_date&#39;])[&#39;pred_repair&#39;].sum().reset_index()<br>df_final_output.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/0*0yFonLNr4bpX4_tC.png" /><figcaption><strong>Fig 11. Final Predictions</strong></figcaption></figure><p><strong>Beginning of Cox Regression</strong></p><p>After playing with univariate models, we start to think about incorporating other information like how many sales were made at specific time into our model. Now it’s time to introduce cox proportional hazard model. To put it simply, this model thinks the log-hazard of an individual is a linear function of their static covariates and a population-level baseline hazard that changes over time. You can refer to <a href="https://lifelines.readthedocs.io/en/latest/Survival%20Regression.html">this page</a> here for better understanding.</p><p>We can use “LabelEncoder” from sklearn package to convert categorical variables like “module_category” to be integers and fit one model at first.</p><pre>from sklearn import preprocessing<br>le_comp = preprocessing.LabelEncoder()<br>le_mod = preprocessing.LabelEncoder()</pre><pre>df_cox_input = df_input[[&#39;module_category&#39;, &#39;component_category&#39;, &#39;number_sale&#39;, &#39;sale_date&#39;, &#39;months_after_sold&#39;, &#39;event&#39;, &#39;freq&#39;]]</pre><pre>le_comp.fit(df_cox_input[&#39;component_category&#39;])<br>df_cox_input[&#39;component_category&#39;] = le_comp.transform(df_cox_input[&#39;component_category&#39;])</pre><pre>le_mod.fit(df_cox_input[&#39;module_category&#39;])<br>df_cox_input[&#39;module_category&#39;] = le_mod.transform(df_cox_input[&#39;module_category&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WajO47RL_UpdIdMa.png" /><figcaption><strong>Fig 12. Input Table for Cox Regression</strong></figcaption></figure><p>One thing worth to mention is we need to check if proportional assumption holds for every cox regression model. There is a function called check_assumptions in lifelines to check that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zubWFTQh-nRDH7vJ.png" /><figcaption><strong>Fig 13. Results from Check_assumptions</strong></figcaption></figure><p>Following above results, we have to go back and fit component-level models again. Now it’s your turn to finish whole modeling and predicting flow!</p><h3><strong>Summary</strong></h3><p>Congrats! Now you know how to analyze time series problems into survival analysis structure and corresponding codes to fit models and make predictions. To conclude, there are many models to deal with time series forecasting, but best model type is always case by case. If you are stuck in time series, trying to change a way of thinking and starting to apply survival analysis in your case might be helpful. Hope this series of blogs about survival analysis can provide you with a new view when dealing with forecasting problems.</p><p>Click here to learn more about the <a href="https://www.ibm.com/community/datascience/elite/">IBM Data Science Elite</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b1793fb8195" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-machine-learning/survival-analysis-1b1793fb8195">Survival Analysis</a> was originally published in <a href="https://medium.com/inside-machine-learning">Inside Machine learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leverage Watson Machine Learning Function Deployments in Data Science Workflows]]></title>
            <link>https://medium.com/@greenorange1994/leverage-watson-machine-learning-function-deployments-in-data-science-workflows-476ce9ed726a?source=rss-e58ec2c1f48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/476ce9ed726a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[watson-machine-learning]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[lda]]></category>
            <dc:creator><![CDATA[Jade Zhou]]></dc:creator>
            <pubDate>Mon, 30 Sep 2019 20:25:52 GMT</pubDate>
            <atom:updated>2019-10-02T21:13:41.813Z</atom:updated>
            <content:encoded><![CDATA[<h3>Leverage Watson Machine Learning Function Deployment in Data Science Workflow</h3><h4>Part 1 –Deploy a Special Type of Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*HCutZZp-6mdj5oyz.png" /></figure><p>Yushu Zhou, IBM Data Science Elite Team,</p><p><a href="https://medium.com/@wantingwang">Wanting Wang</a>, IBM Data Science Elite Team</p><h3>Introduction</h3><p>IBM Watson Machine Learning (WML) is a full-service IBM Cloud offering that makes it easy for developers and data scientists to work together to integrate predictive capabilities with their applications. WML provides two deployment options: model deployment and function deployment, so literally, when you want to integrate your model into production environment, model deployment is the reasonable choice. However, <a href="https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/pm_service_supported_frameworks.html?audience=wdp&amp;context=wdp">WML model deployment</a> only supports limited number of types of models like scikit-learn and keras. In real world, apart from classification and regression models, we still have other types of models. For example, LDA is not supported by WML model deployment functionality currently. Thus, as a more flexible alternative, WML provides function deployment to be leveraged in above situations.</p><p>In this blog, we will take 2 different unsupported models as examples to introduce different ways of deploying models by WML function deployment.</p><h3>Set-up</h3><p>To follow this blog, we recommend you to create an <a href="https://console.bluemix.net/">IBM cloud account</a> and instantiate your personal Cloud Object Storage (COS) and Watson Machine Learning (WML) services under ‘Catalog’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*0yWKK2-1WOH3RSRCoxXg2w.png" /><figcaption>Fig 1. WML under Catalog</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*VmcN05A4nbRjm-obz9vEyw.png" /><figcaption>Fig 2. COS under Catalog</figcaption></figure><p>The good thing here is we have ‘Lite’ version for these services which you can play around without any cost. After you provision these two services, go to your <a href="https://console.bluemix.net/dashboard/apps">bluemix dashboard</a> and choose WML instance you just created. Click on ‘view credentials’ and copy them down. We will use these credentials later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*jFo459SMnXy-nh225ZKJ5A.png" /><figcaption>Fig 3. Get WML Credentials</figcaption></figure><p>Then go to <a href="https://dataplatform.cloud.ibm.com/">Watson Studio</a> and create a project based on COS you just created. Finally, associate WML instance with this project under ‘Settings’.</p><p>After you finish these steps, we can start playing with WML.</p><h3>How we leverage function deployment in different situations</h3><h4>1. You can save and deploy your model but cannot score by WML model deployment</h4><p>Sci-kit learn package in Python provides numerous machine learning algorithms and related methods for data processing. Many popular algorithms from sci-kit learn are well supported by WML. One exception is Latent Dirichlet Allocation (LDA), a useful topic modeling algorithm in text analytics.</p><p>To understand why sci-kit learn LDA models are not supported and accordingly how to use sci-kit learn LDA models with WML, let’s start with training a simple LDA model on 20 news groups dataset which can be downloaded through sci-kit learn.</p><p>(1) Load data and train LDA model</p><pre>import pandas as pd<br>from sklearn.datasets import fetch_20newsgroups<br>from sklearn.feature_extraction import stop_words<br>from sklearn.feature_extraction.text import CountVectorizer<br>from sklearn.decomposition import LatentDirichletAllocation</pre><pre>## download data<br>dict_news = fetch_20newsgroups(subset=’train’)<br>text = dict_news.data[:5000]<br>label = dict_news.target[:5000]</pre><pre>## vectorize data<br>stop_words = list(stop_words.ENGLISH_STOP_WORDS)<br>vectorizer = CountVectorizer(stop_words = stop_words, min_df = 0.01, max_df = 0.8))<br>text_bw = vectorizer.fit_transform(text)</pre><pre>## train LDA<br>lda = LatentDirichletAllocation(random_state=123)<br>lda.fit(text_bw)</pre><p>As what you may expect, I use the count vectorizer to vectorize the text data, and reduce the vocabulary by setting thresholds in min_df and max_df so that words unlikely to be informative are eliminated and the training doesn’t take too much time.</p><p>Now the model is trained, I want to deploy it in WML in order for my application to use.</p><p>(2) Save and deploy to WML as a model</p><p>To make the right WML instance be aware of what is going on, I need to set up the connection to the picked WML instance.</p><pre>credentials_wml = {<br>“apikey”: “****”,<br>“iam_apikey_description”: “****”,<br>“iam_apikey_name”: “****”,<br>“iam_role_crn”: “****”,<br>“iam_serviceid_crn”: “****”,<br>“instance_id”: “****”,<br>“password”: “****”,<br>“url”: “****”,<br>“username”: “****”<br>}</pre><pre>from watson_machine_learning_client import WatsonMachineLearningAPIClient</pre><pre>client = WatsonMachineLearningAPIClient(credentials_wml)</pre><p>Credentials can be obtained from the Service Credentials page of your WML instance which we showed previously. As a recap, you can go to the resource list in your IBM Cloud dashboard (<a href="https://cloud.ibm.com/resources">https://cloud.ibm.com/resources</a>), find the WML instance listed under Services, click on it and go to the Service Credentials page on the menu. You may use an existing key or create a new one.</p><pre>model_props = {client.repository.ModelMetaNames.NAME: “LDA model 1”}</pre><pre>model_artifact=client.repository.store_model(lda, meta_props=model_props)</pre><pre>published_model_uid = client.repository.get_model_uid(model_artifact)</pre><pre>created_deployment = client.deployments.create(published_model_uid, name=”LDA model 1 deployment”)</pre><p>Great, it says the deployment is successful! Before embedding the scoring endpoint into my application, I’d better test it first, just in case.</p><pre>scoring_endpoint = client.deployments.get_scoring_url(https://medium.com/created_deployment)</pre><pre>fields = df_bw.columns.tolist()</pre><pre>values = [df_bw.iloc[1].values.tolist()]</pre><pre>scoring_payload = {“fields”: fields,”values”: values}</pre><pre>predictions = client.deployments.score(scoring_endpoint, scoring_payload)</pre><pre>print(predictions)</pre><p>I pull one vectorized document (essentially one row) from the data frame, convert the values to a list, and wrap it as a payload with field names which I then send to the scoring endpoint.</p><p>Ideally I should see the result on this document given by the LDA model I just deployed, but I got the error message saying that my LDA model object does not have attribute predict.</p><p>It is clear that WML is trying to call the predict method on the LDA model object, however this operation fails simply because scikit-learn LDA models don’t have such method. Instead, when applying a scikit-learn LDA model on new data points, we need to call the transform method.</p><p>So far I have no idea if there is any way to hack into WML and change how it works — there should not, I suppose? Then we are left with the only choice: make adaptation on our side to make things work.</p><p>Thanks God, there is python deployable function we can turn to.</p><p>(3) Save and deploy to WML as a function</p><p>Python deployable function specifies what needs to be run when the function is deployed and what needs to be run when the deployment is called. In other words, you are able to customize what preparation WML does in the environment when you deploy the function, as well as what steps WML takes to generate the output when you call the API later on.</p><p>The only change we want to make for the deployment is to replace the predict method with the transform method. When it comes to the deployable function, it is straightforward but not that straightforward, as you don’t see the vanilla version of what is going on but you are asked to create your own.</p><p>What are the essential components to score a random data point? We need to a) fetch the model object back from WML, b) convert the payload to the same format as the training data, c) call the transform method on the model object over the payload, and d) construct the result to be returned.</p><p>Let’s see how these 4 components are put together.</p><pre>ai_parms = { “credentials_wml” : credentials_wml,<br>“model_uid” : published_model_uid }</pre><pre>def my_deployable_function( parms=ai_parms ):<br>    ## a) load model<br>    from watson_machine_learning_client import WatsonMachineLearningAPIClient<br>    client = WatsonMachineLearningAPIClient( parms[“credentials_wml”] )<br>    model = client.repository.load(parms[“model_uid”])</pre><pre>def score( payload ):<br>        ## b) assemble input<br>        df_payload = pd.DataFrame(payload[“values”])<br>        df_payload.columns = payload[“fields”]<br>        ## c) obtain result from the model<br>        res = model.transform(df_payload).tolist()<br>        ## d) assemble output<br>        output = {“result” : res}<br>        return output</pre><pre>return score</pre><p>The deployable function consists of the outer function (any place that is not within the score function) and the inner score function. The code sits in the outer function runs when the function is deployed, and the environment is then frozen and ready to be used whenever the API is called. The code sits in the inner score function runs when the API is called, in the environment customized by the outer function, basically takes the input json and outputs some result json.</p><p>If the score function is the only thing that WML runs when the deployment API is called, why you would even bother to care about the outer function? Why cannot just put everything in the score function?</p><p>The answer is, yes you can always put everything in the score function, but do you really want to afford the repeated cost in run time that can be isolated in environment preparation?</p><p>In this example, fetching the simple LDA model from WML may not take much time. What if you need to load a big neural network model that has hundreds of millions of parameters or some complex ensemble model that compromises dozens of weaker models? They could easily exceed 100MB or even 1GB. Loading models of such big size requires substantially longer time, and there is not benefit loading them every time when you call the API compared to loading them previously in the environment.</p><p>In addition to model loading that may take some time, other operations that are irrelevant to the scoring part (like package installation) can also be moved to the outer function.</p><p>After you configure the outer function, what needs to be done in the inner score function then is pretty easy to imagine, and actually is very much similar to how you will score a new data point with the loaded model in a plain cell in jupyter notebook.</p><p>Once the function is specified, deploying a function is as simple as deploying a model.</p><pre>meta_data = { client.repository.FunctionMetaNames.NAME : ‘LDA model 1 function’ }</pre><pre>function_details = client.repository.store_function( meta_props=meta_data, function=my_deployable_function )</pre><pre>function_id = function_details[“metadata”][“guid”]</pre><pre>function_deployment_details = client.deployments.create( artifact_uid=function_id, name=’LDA odel 1 function deployment’ )</pre><p>Again I see the familiar success message, although I know it does not necessarily indicate that the deployment API works as expected — from the surprise I had several minutes ago. So too early to celebrate at this moment. I have to keep it until the test is passed.</p><p>Similarly, from the deployment I obtain the scoring endpoint, and pass the same test payload to it.</p><pre>scoring_endpoint_function = client.deployments.get_scoring_url(https://medium.com/function_deployment_details)</pre><pre>predictions = client.deployments.score(scoring_endpoint_function, scoring_payload)</pre><pre>print(predictions)</pre><p>See what is the result?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*69PNDQzO5vyulG4_fSP7WQ.png" /><figcaption>Fig 4. Results from LDA function endpoint</figcaption></figure><h4>2. You don’t need a fixed model</h4><p>For most of time series forecast problems, simple stats models can beat machine learning without much effort, so there are many cases where we have to build not-ML models to have better predictions. Before explaining how to deploy this type of model, we have to rethink training and testing data that needs to be fed into the model.</p><p>Assume you are responsible for production and stock planning in a company, to make sure whenever clients request a specific product, your company has stock of that, you have to predict every product’s monthly demands ahead of time considering production needs time, so a gap period is needed and you have to predict demands month by month. Every time when you try to make monthly predictions, you need to refit a model because one more month data can be incorporated into your model. Thus, in this scenario, we don’t need a fixed model to be continuously called. Instead, we pass training data every time to get a new model and all predictions. To conclude, what we post to API should be training data. After that, the API should finish fitting a model and respond predictions.</p><p>Following this workflow, we should deploy a function with the ability to fit and forecast. Therefore, we already got a rough image of what should be included in the notebook.</p><p>(1) Set up WML Client</p><p>Same step as before.</p><p>(2) Define custom function — outer part</p><p>Next, create your own function. Inside function, to make sure we install all packages we need, I’ll add sub-processing here.</p><pre>def es():<br>    &quot;&quot;&quot;<br>        You will deploy this function.<br>        -------------------------------<br>        :return score function<br>    &quot;&quot;&quot; <br>    try:<br>         <br>        import subprocess<br>        subprocess.check_output( &quot;pip install -I statsmodels==0.9.0 --user&quot;, stderr=subprocess.STDOUT, shell=True)<br>        subprocess.check_output( &quot;pip install -I pandas==0.23.4 --user&quot;, stderr=subprocess.STDOUT, shell=True)<br>        subprocess.check_output( &quot;pip install -I numpy==1.15.4 --user&quot;, stderr=subprocess.STDOUT, shell=True)<br>        import pandas as pd<br>        import numpy as np<br>        from statsmodels.tsa.api import SimpleExpSmoothing<br>    <br>    except subprocess.CalledProcessError as e:<br>    <br>        return { &quot;error&quot; : &quot;subprocess.CalledProcessError:\n\n&quot; + &quot;cmd:\n&quot; + e.cmd + &quot;\n\noutput:\n&quot; + e.output.decode() }<br>    <br>    except Exception as e:<br>    <br>        return { &quot;error&quot; : repr( e ) }</pre><p>(3) Define custom function — inner scoring part</p><p>Since our input is training data, we will fit a model and finish multi-step or single-step forecast inside scoring function. Then we will return forecasts in json format and outer function has to return score function.</p><pre>def score(payload):<br>        &quot;&quot;&quot;<br>            To get returned values from es function, you have to rewrite your own score function here.<br>            ------------------------------------------------------------------------------------------<br>            :return any output in json format<br>        &quot;&quot;&quot;<br>        forecast_period = int(payload[&#39;values&#39;][0][&#39;forecast_period&#39;])<br>        ts = pd.read_json(payload[&#39;values&#39;][0][&#39;data&#39;])<br>        ts.index = pd.to_datetime(ts.index)<br>        <br>        # To put it simply, I built a simple exponential smoothing model here, but you can build more complicated models and return predictions as output field&#39;s values.<br>        fit = SimpleExpSmoothing(ts).fit(smoothing_level=0.2,optimized=False)<br>        fcast = fit.forecast(forecast_period)</pre><pre>        return { &quot;output&quot; : list(fcast)}<br>    <br>    return score</pre><p>(4) Deploy and test</p><p>Finally, we can deploy above function in our customized environment and post input data to the endpoint as a test. Remember to embed your data in json format which our function can recognize.</p><pre># store function<br># replace function_name with your own name<br>function_name = &#39;Exponential Smoothing&#39;<br>meta_data = {<br>    client.repository.FunctionMetaNames.NAME: function_name#,<br>    #Uncomment next line of code if you choose installing libraries by RUNTIME API<br>    #client.repository.FunctionMetaNames.RUNTIME_UID: runtime_guid <br>}</pre><pre>stored_function = client.repository.store_function( meta_props=meta_data, function=es)</pre><pre>function_uid = client.repository.get_function_uid( stored_function )</pre><pre># deploy function<br># replace deploy_name with your own name<br>deploy_name = &#39;ES Deployment&#39;<br>function_deployment_details = client.deployments.create( artifact_uid = function_uid, name = deploy_name )</pre><p>Here are results if we call the deployed function:</p><pre># NOTE: manually define and pass the array(s) of values to be scored in the next line<br># We&#39;re predicting next 3 months&#39; electricity demands based on 2003 monthly data only<br>function_payload = {&quot;fields&quot;: [[]], &quot;values&quot;: [{&#39;forecast_period&#39;: 3, &#39;data&#39;: pd.DataFrame(ts[&#39;2003&#39;]).to_json()}]}</pre><pre>scoring_endpoint = client.deployments.get_scoring_url(https://medium.com/function_deployment_details)</pre><pre>predictions = client.deployments.score(scoring_endpoint, function_payload)<br>print(predictions)</pre><h3>Conclusion</h3><p>Now it’s time to toast! Hope this blog can bring you new ideas about how to leverage WML into data science workflow from model deployment perspective. One missing piece here is how we deploy trained models when you cannot even save it into WML model repository. For example, spark ALS model is a common tool for agile development of recommender but it’s not directly supported by WML. Our potential solution is to upload it to COS by ibmos2spark and follow same steps in first part: load back the model in outer function and score in inner function. However, since the environment for function deployment is pure python environment, it’s hard to connect to spark engine there, but it will get solved in near future. Therefore, we’ll update this blog once it’s ready.</p><p>Finally, we know to train a machine learning model, the whole pipeline is much more than just final model deployment, so Wanting will introduce how to leverage WML function deployment at other stages in part 2.</p><p>Keep an eye on our updates!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=476ce9ed726a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating and Accelerating Hyperparameter Tuning for Deep Learning]]></title>
            <link>https://medium.com/@greenorange1994/automating-and-accelerating-hyperparameter-tuning-for-deep-learning-52184944eeb4?source=rss-e58ec2c1f48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/52184944eeb4</guid>
            <category><![CDATA[watson-studio]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jade Zhou]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 13:21:23 GMT</pubDate>
            <atom:updated>2019-03-14T17:36:20.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*k0KZNVb8rwOjD4dj.jpg" /></figure><p>Deep learning can be tedious work. Taking Long Short-Term Memory (LSTM) as an example, we have lots of hyperparameters, (learning rate, number of hidden units, batch size, and so on) waiting for us to find the best combination. Considering the size of a deep learning model, hyperparameter tuning usually takes long time.</p><p>One traditional way to automate the tuning process is grid search, trying every possible group of hyperparameters in the search space and remembering the best one, but this method is typically too computationally intensive. To overcome this issue, IBM developed a black-box optimization library called <a href="https://github.com/coin-or/rbfopt">RbfOpt</a> which is embedded in Experiment Builder.</p><p>In this blog, I’ll walk you through how to set up hyperparameter tuning with Experiment Builder on Watson Studio. Basically, we need to meet three requirements before we start our experiments:</p><ol><li>Provisioning a Cloud Object Storage (COS) Instance</li><li>Provisioning a Watson Machine Learning (WML) Instance</li><li>Setting up a Training Definition</li></ol><h3>Service Set-up</h3><p>To fulfill the first two requirements, you have to create an <a href="https://console.bluemix.net/">IBM cloud</a> account and instantiate your personal COS and WML services under ‘Catalog’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*iaup67xDChNj5R9PKoIkDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*3H3qlQ0bU20JyLb4I4GYzQ.png" /></figure><p>Look for the ‘Lite’ version of these services, available at no cost.</p><h3>Training Definition</h3><p>The training definition is a zip file that includes one main python script to train models and save results. You can also zip other scripts or files if needed. After Experiment Builder is activated, it actually runs the main script. That’s why we need this training definition.</p><p>For the main script, you can follow the <a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_code_guidelines.html?audience=wdp">coding guidelines</a> to write model training and evaluation functions and execute them in the main function. In addition, you’ll have to add and edit some parts of the code to do hyperparameter optimization (HPO). Let’s delve deeper into each part.</p><ol><li><strong>Specify directories and read data.</strong> You’ll see we need to specify two buckets (a source bucket and a result bucket) for input data and output results in the next section, so once you upload data into that source bucket, it’s pretty easy for your program to read them:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*4rRejIZw-df9DwbYICEJ4w.png" /></figure><p><strong>2. Read hyperparameters.</strong> Hyperparameters will be saved in a json file called ‘config.json’, within same directory as this script. In this example, I only explored learning rate and number of hidden units for LSTM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*fGxylr0E4Z9_qEP-ckDbnw.png" /></figure><p>Lastly, include three additional components. First, download ‘<a href="https://raw.githubusercontent.com/pmservice/wml-sample-models/master/deep-learning/metrics/emetrics.py">emetrics.py</a>’ and include it in your zip file. Remember to import it at the beginning of your main script.</p><p>What’s more, outside the main function, add a function called ‘getCurrentSubID’ and a class called ‘HPOMetrics’ that will pass metrics to your HPO program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*BICcNv2zEEzUsafB_p62Cw.png" /></figure><p><strong>3. Train and evaluate.</strong> Let’s return to the main function. After you’ve set up all the data and hyperparameters set-up, you can call functions for training and evaluating:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*1kPKYjl9HW5oFTDh9soWsQ.png" /></figure><p>Inside the ‘train_model’ function, to pass metrics to our HPO program after each epoch, we have to create an instance of HPOMetrics and include it in the model ‘callbacks’. After training finishes, don’t forget to close HPOMetrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*1FBjwjMUozNxqZGiFjKkTg.png" /></figure><p>If you want to save predictions or other results, you can write them directly into the result directory which is defined as ‘output_model_folder’ in the first part. Then you can go to your cloud account dashboard and enter the bucket you used for storing results to download those predictions or any metrics you want.</p><h3>Start Experimenting!</h3><p>We’ve got services and python scripts ready, so now we can start experimenting on hyperparameters.</p><p>To access the UI page of Experiment Builder on <a href="https://dataplatform.cloud.ibm.com/">Watson Studio</a>, create a project based on the COS you just created, and associate the WML instance with this project under ‘Settings’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*MT1xGmJWA7_jF5B2-etKgA.png" /></figure><p>Next, return to the project ‘Assets’ page and you’ll see ‘Experiments’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Nez1Ulim-ugMEo63Ejc7Lg.png" /></figure><p>Clicking ‘New experiment +’ and select the services you just associated with this project. Technically, COS is used for getting data, storing logs and results. WML can be a place to save and deploy your model (which this blog doesn’t cover). For Experiment Builder, we usually create a new bucket in COS. The source COS bucket stores all data your python scripts need. Results COS bucket stores all logs and results. I usually use the same bucket for these two purposes, but you can also just have two separate buckets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*NbvyAGqSTPMzyIp5iC8JSw.png" /></figure><p>Click the source bucket (marked by a red rectangle) and you’ll go to your Bluemix dashboard, then upload all your data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*UtMoffF5TZGclYEB5gTglA.png" /></figure><p>Next, we go back to our Experiment page to add the training definition. Apart from uploading the zip file including your python scripts, there are several things you need to configure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*K2baXMDLo293NcCkGiCHVg.png" /></figure><ol><li><strong>Framework.</strong> Choose the one (tensorflow/pytorch/caffe) you code with.</li><li><strong>Compute plan.</strong> Choose how many <strong>GPUs</strong> you want to use.</li><li><strong>Hyperparameter optimization method.</strong> If you don’t want to tune hyperparameter, just select ‘none’. Otherwise, choose RBFOpt or random. Here I chose RBFOpt.</li><li><strong>Number of optimizer steps.</strong> If you decide to tune hyperparameters, you have to clarify some parameters. This one equals the number of training runs.</li><li><strong>Objective.</strong> Since I’m using keras to build a LSTM model, the objective has to exist in the model history. Note that it even requires the same name, otherwise Experiment Builder will post error messages. During my training process, I only calculated ‘train_loss’ and ‘val_loss’. To avoid overfitting, my objective is ‘val_loss’.</li><li><strong>Maximize or minimize.</strong> Choose ‘minimize’ when the objective is loss and ‘maximize when it’s accuracy.</li><li><strong>Add hyperparameters.</strong> These must be consistent with the hyperparameters we set in our script. This step will automatically generate a json file called ‘config.json’, which is why in our script we have to read hyperparameters from this file.</li></ol><p>For each hyperparameter, you have to specify the range and data type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*KDAEQPkb_2qlqNYftSRpGA.png" /></figure><p>And you’re finished! Don’t forget to give your experiment a unique name. Hit ‘create and run’. Deep learning training takes a while, but you can supervise your model while it’s running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ByNIuk5As-xtls9x1P9Tng.png" /></figure><p>After Watson Studio completes the experiments, look for results under ‘Compare Runs’ and choose your best group of hyperparameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*wYdtGHmDFtil8FlnJebFtA.png" /></figure><p>Congratulations! You’ve done your first group of experiments using Experiment Builder on Watson Studio. Enjoy your results!</p><p><strong>Further Reading:</strong></p><ol><li><a href="https://dataplatform.cloud.ibm.com/analytics/notebooks/b9809f95-f8d5-42e7-b9f3-343d9dc673e9/view?access_token=f9491fe943c5376bc300fc1a571b4d1367f25660b14daeed69ddadad0104d148">From keras experiment to scoring with watson-machine-learning-client</a></li><li><a href="https://github.com/pmservice/wml-sample-models/tree/master/keras/mnist">wml-sample-model Github Repository</a></li><li><a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_hpo.html">HPO Guide — Watson Studio Document</a></li></ol><p><em>Originally published at </em><a href="https://medium.com/inside-machine-learning/placeholder-520a5274d07"><em>medium.com</em></a><em> on March 12, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52184944eeb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automating and Accelerating Hyperparameter Tuning for Deep Learning]]></title>
            <link>https://medium.com/inside-machine-learning/placeholder-520a5274d07?source=rss-e58ec2c1f48c------2</link>
            <guid isPermaLink="false">https://medium.com/p/520a5274d07</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[hyperparameter-tuning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jade Zhou]]></dc:creator>
            <pubDate>Tue, 12 Mar 2019 13:21:23 GMT</pubDate>
            <atom:updated>2019-03-12T13:22:53.411Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*8GCKL070jqMwFVrjJh-fPQ.png" /></figure><p>Deep learning can be tedious work. Taking Long Short-Term Memory (LSTM) as an example, we have lots of hyperparameters, (learning rate, number of hidden units, batch size, and so on) waiting for us to find the best combination. Considering the size of a deep learning model, hyperparameter tuning usually takes long time.</p><p>One traditional way to automate the tuning process is grid search, trying every possible group of hyperparameters in the search space and remembering the best one, but this method is typically too computationally intensive. To overcome this issue, IBM developed a black-box optimization library called RbfOpt (<a href="https://github.com/coin-or/rbfopt">https://github.com/coin-or/rbfopt</a>) which is embedded in Experiment Builder.</p><p>In this blog, I’ll walk you through how to set up hyperparameter tuning with Experiment Builder on Watson Studio. Basically, we need to meet three requirements before we start our experiments:</p><ol><li>Provisioning a Cloud Object Storage (COS) Instance</li><li>Provisioning a Watson Machine Learning (WML) Instance</li><li>Setting up a Training Definition</li></ol><h3><strong>Service Set-up</strong></h3><p>To fulfill the first two requirements, you have to create an IBM cloud account (https://console.bluemix.net/) and instantiate your personal COS and WML services under ‘Catalog’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iaup67xDChNj5R9PKoIkDA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3H3qlQ0bU20JyLb4I4GYzQ.png" /></figure><p>Look for the ‘Lite’ version of these services, available at no cost.</p><h3><strong>Training Definition</strong></h3><p>The training definition is a zip file that includes one main python script to train models and save results. You can also zip other scripts or files if needed. After Experiment Builder is activated, it actually runs the main script. That’s why we need this training definition.</p><p>For the main script, you can follow the coding guidelines (<a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_code_guidelines.html?audience=wdp">https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_code_guidelines.html?audience=wdp</a>) to write model training and evaluation functions and execute them in the main function. In addition, you’ll have to add and edit some parts of the code to do hyperparameter optimization (HPO). Let’s delve deeper into each part.</p><ol><li><strong>Specify directories and read data.</strong> You’ll see we need to specify two buckets (a source bucket and a result bucket) for input data and output results in the next section, so once you upload data into that source bucket, it’s pretty easy for your program to read them:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rRejIZw-df9DwbYICEJ4w.png" /></figure><p><strong>2. Read hyperparameters.</strong> Hyperparameters will be saved in a json file called ‘config.json’, within same directory as this script. In this example, I only explored learning rate and number of hidden units for LSTM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fGxylr0E4Z9_qEP-ckDbnw.png" /></figure><p>Lastly, include three additional components. First, download ‘emetrics.py’ (<a href="https://raw.githubusercontent.com/pmservice/wml-sample-models/master/deep-learning/metrics/emetrics.py">https://raw.githubusercontent.com/pmservice/wml-sample-models/master/deep-learning/metrics/emetrics.py</a>) and include it in your zip file. Remember to import it at the beginning of your main script.</p><p>What’s more, outside the main function, add a function called ‘getCurrentSubID’ and a class called ‘HPOMetrics’ that will pass metrics to your HPO program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BICcNv2zEEzUsafB_p62Cw.png" /></figure><p><strong>3. Train and evaluate.</strong> Let’s return to the main function. After you’ve set up all the data and hyperparameters set-up, you can call functions for training and evaluating:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1kPKYjl9HW5oFTDh9soWsQ.png" /></figure><p>Inside the ‘train_model’ function, to pass metrics to our HPO program after each epoch, we have to create an instance of HPOMetrics and include it in the model ‘callbacks’. After training finishes, don’t forget to close HPOMetrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1FBjwjMUozNxqZGiFjKkTg.png" /></figure><p>If you want to save predictions or other results, you can write them directly into the result directory which is defined as ‘output_model_folder’ in the first part. Then you can go to your cloud account dashboard and enter the bucket you used for storing results to download those predictions or any metrics you want.</p><h3><strong>Start Experimenting!</strong></h3><p>We’ve got services and python scripts ready, so now we can start experimenting on hyperparameters.</p><p>To access the UI page of Experiment Builder on Watson Studio (https://dataplatform.cloud.ibm.com/), create a project based on the COS you just created, and associate the WML instance with this project under ‘Settings’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MT1xGmJWA7_jF5B2-etKgA.png" /></figure><p>Next, return to the project ‘Assets’ page and you’ll see ‘Experiments’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nez1Ulim-ugMEo63Ejc7Lg.png" /></figure><p>Clicking ‘New experiment +’ and select the services you just associated with this project. Technically, COS is used for getting data, storing logs and results. WML can be a place to save and deploy your model (which this blog doesn’t cover). For Experiment Builder, we usually create a new bucket in COS. The source COS bucket stores all data your python scripts need. Results COS bucket stores all logs and results. I usually use the same bucket for these two purposes, but you can also just have two separate buckets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NbvyAGqSTPMzyIp5iC8JSw.png" /></figure><p>Click the source bucket (marked by a red rectangle) and you’ll go to your Bluemix dashboard, then upload all your data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UtMoffF5TZGclYEB5gTglA.png" /></figure><p>Next, we go back to our Experiment page to add the training definition. Apart from uploading the zip file including your python scripts, there are several things you need to configure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K2baXMDLo293NcCkGiCHVg.png" /></figure><ol><li><strong>Framework.</strong> Choose the one (tensorflow/pytorch/caffe) you code with.</li><li><strong>Compute plan.</strong> Choose how many <strong>GPUs</strong> you want to use.</li><li><strong>Hyperparameter optimization method.</strong> If you don’t want to tune hyperparameter, just select ‘none’. Otherwise, choose RBFOpt or random. Here I chose RBFOpt.</li><li><strong>Number of optimizer steps.</strong> If you decide to tune hyperparameters, you have to clarify some parameters. This one equals the number of training runs.</li><li><strong>Objective.</strong> Since I’m using keras to build a LSTM model, the objective has to exist in the model history. Note that it even requires the same name, otherwise Experiment Builder will post error messages. During my training process, I only calculated ‘train_loss’ and ‘val_loss’. To avoid overfitting, my objective is ‘val_loss’.</li><li><strong>Maximize or minimize.</strong> Choose ‘minimize’ when the objective is loss and ‘maximize when it’s accuracy.</li><li><strong>Add hyperparameters.</strong> These must be consistent with the hyperparameters we set in our script. This step will automatically generate a json file called ‘config.json’, which is why in our script we have to read hyperparameters from this file.</li></ol><p>For each hyperparameter, you have to specify the range and data type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDAEQPkb_2qlqNYftSRpGA.png" /></figure><p>And you’re finished! Don’t forget to give your experiment a unique name. Hit ‘create and run’. Deep learning training takes a while, but you can supervise your model while it’s running.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByNIuk5As-xtls9x1P9Tng.png" /></figure><p>After Watson Studio completes the experiments, look for results under ‘Compare Runs’ and choose your best group of hyperparameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wYdtGHmDFtil8FlnJebFtA.png" /></figure><p>Congratulations! You’ve done your first group of experiments using Experiment Builder on Watson Studio. Enjoy your results!</p><p><strong>Further Reading:</strong></p><p>“From keras experiment to scoring with watson-machine-learning-client”: <a href="https://dataplatform.cloud.ibm.com/analytics/notebooks/b9809f95-f8d5-42e7-b9f3-343d9dc673e9/view?access_token=f9491fe943c5376bc300fc1a571b4d1367f25660b14daeed69ddadad0104d148">https://dataplatform.cloud.ibm.com/analytics/notebooks/b9809f95-f8d5-42e7-b9f3-343d9dc673e9/view?access_token=f9491fe943c5376bc300fc1a571b4d1367f25660b14daeed69ddadad0104d148</a></p><p>“wml-sample-model”: <a href="https://github.com/pmservice/wml-sample-models/tree/master/keras/mnist">https://github.com/pmservice/wml-sample-models/tree/master/keras/mnist</a></p><p>“Hyperparameter Optimization”: <a href="https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_hpo.html">https://dataplatform.cloud.ibm.com/docs/content/analyze-data/ml_dlaas_hpo.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=520a5274d07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inside-machine-learning/placeholder-520a5274d07">Automating and Accelerating Hyperparameter Tuning for Deep Learning</a> was originally published in <a href="https://medium.com/inside-machine-learning">Inside Machine learning</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>