<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Caio Ishizaka Costa on Medium]]></title>
        <description><![CDATA[Stories by Caio Ishizaka Costa on Medium]]></description>
        <link>https://medium.com/@caio.ishizaka?source=rss-9b1abe8b6091------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XegJowgFwATLwjS9ZmfTQw.jpeg</url>
            <title>Stories by Caio Ishizaka Costa on Medium</title>
            <link>https://medium.com/@caio.ishizaka?source=rss-9b1abe8b6091------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:34:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@caio.ishizaka/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I used open source Dolly to write a LinkedIn post]]></title>
            <link>https://medium.com/@caio.ishizaka/imagehow-did-i-use-open-source-dolly-to-write-a-linkedin-post-fecb3436f93d?source=rss-9b1abe8b6091------2</link>
            <guid isPermaLink="false">https://medium.com/p/fecb3436f93d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[dolly]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Caio Ishizaka Costa]]></dc:creator>
            <pubDate>Fri, 31 Mar 2023 17:48:36 GMT</pubDate>
            <atom:updated>2023-03-31T19:30:47.819Z</atom:updated>
            <content:encoded><![CDATA[<p>Excitement around chatGPT and similar tools that are gaining popularity, a recent publication of an open model called Dolly by Databricks, and the willingness to write a LinkedIn post on my 2-year anniversary at Databricks, led me to try Dolly to help me on that task, this is how it went.</p><h3>What is Dolly?</h3><p>Dolly is an innovative tool developed by Databricks that aims to democratize access to the latest advances in Natural Language Processing (NLP) through a user-friendly interface. Dolly is a cheap-to-build LLM that exhibits a surprising degree of the instruction following capabilities exhibited by ChatGPT.</p><p>One of the key features of Dolly is its ability to adapt to specific use cases and contexts. By allowing users to fine-tune the underlying language model to their domain or industry, Dolly can provide even more accurate and relevant responses. This customization is made possible through Databricks’ scalable and secure cloud platform, which provides a streamlined pipeline for training and deploying custom NLP models. Overall, Dolly represents a major step forward in making NLP accessible to a broader audience, and has the potential to transform the way we interact with machines and each other through natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o4pGnB3QOLFgi8B6WyJC4A.png" /></figure><p>Learn more about Dolly at <a href="https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html"><em>Databricks blog post</em></a></p><h3>How to use Dolly?</h3><p>Contrary to what many may think, using Dolly does not require an incredible powerful computer with leading edge graphic cards. All my experiments were done in a modest personal computer, equipped with 32GB of RAM, an i7-6850k CPU, and an NVIDIA GeForce GTX1080 (that went unused in the end)</p><p>Also, in this exercise I did not tune the model further, I used what was available off-the-shelf by Databricks.</p><p>The easiest way to use the Dolly, I found, is to use the <a href="https://huggingface.co/databricks/dolly-v1-6b"><em>model made available in Hugging Face</em></a><em>. </em>Cloning it is pretty straight forward, with a simple git command</p><pre>git clone https://huggingface.co/databricks/dolly-v1-6b</pre><p>Main reason I cloned it (including the large .bin file) was to avoid having to download the model in every execution.</p><p>The basic requirements to use this is to install <em>transformers </em>(from Hugging Face) and <em>PyTorch</em></p><pre>%pip install transformers<br>%pip install torch</pre><p>If you have a CUDA compatible GPU, you may want to install torch with CUDA enabled. For that you need to <a href="https://developer.nvidia.com/cuda-downloads"><em>first install CUDA</em></a><em>,</em> and I advise checking if your GPU is <a href="https://developer.nvidia.com/cuda-gpus"><em>CUDA compatible first</em></a><em>.</em></p><p>Once CUDA is installed, I advise following the<a href="https://pytorch.org/get-started/locally/"> instructions on PyTorch webpage </a>to properly install a distribution with CUDA enabled</p><h3>Actually prompting it</h3><p>Now that all is setup, it is showtime. The example in the readme did not quite work, as it converts the input to cuda, but does not the same to the model. You have to either remove the <em>.to(“cuda”)</em> from the tokenizer, or do a model.cuda(). The former will use CPU only, while the latter will use the GPU instead.</p><p>Unfortunately the 8GB available in my GPU was not enough to make this work. It suggests optimizations, but I decided not to go down that path (for now).</p><p>Good news is that CPU did the work, with the following code adjusted from readme, with some additions to track time.</p><pre>from transformers import AutoTokenizer, AutoModelForCausalLM, PreTrainedModel, PreTrainedTokenizer<br>from datetime import datetime<br>import numpy as np<br><br>print(&quot;Loading Tokenizer&quot; + str(datetime.now()))<br>tokenizer = AutoTokenizer.from_pretrained(&quot;./&quot;)<br>print(&quot;Loading Model&quot; + str(datetime.now()))<br>model = AutoModelForCausalLM.from_pretrained(&quot;./&quot;)<br><br>PROMPT_FORMAT = &quot;&quot;&quot;Below is an instruction that describes a task. Write a response that appropriately completes the request.<br><br>### Instruction:<br>{instruction}<br><br>### Response:<br>&quot;&quot;&quot;<br><br>def generate_response(instruction: str, *, model: PreTrainedModel, tokenizer: PreTrainedTokenizer, <br>                      do_sample: bool = True, max_new_tokens: int = 256, top_p: float = 0.92, top_k: int = 0, **kwargs) -&gt; str:<br>    print(&quot;Tokenizing input&quot; + str(datetime.now()))<br>    input_ids = tokenizer(PROMPT_FORMAT.format(instruction=instruction), return_tensors=&quot;pt&quot;).input_ids<br><br>    # each of these is encoded to a single token<br>    response_key_token_id = tokenizer.encode(&quot;### Response:&quot;)[0]<br>    end_key_token_id = tokenizer.encode(&quot;### End&quot;)[0]<br>    <br>    print(&quot;Generating tokens&quot; + str(datetime.now()))<br>    gen_tokens = model.generate(input_ids, pad_token_id=tokenizer.pad_token_id, eos_token_id=end_key_token_id,<br>                                do_sample=do_sample, max_new_tokens=max_new_tokens, top_p=top_p, top_k=top_k, **kwargs)[0].cpu()<br><br>    # find where the response begins<br>    print(&quot;Extracting response&quot; + str(datetime.now()))<br>    response_positions = np.where(gen_tokens == response_key_token_id)[0]<br><br>    if len(response_positions) &gt;= 0:<br>        response_pos = response_positions[0]<br>        <br>        # find where the response ends<br>        end_pos = None<br>        end_positions = np.where(gen_tokens == end_key_token_id)[0]<br>        if len(end_positions) &gt; 0:<br>            end_pos = end_positions[0]<br><br>        return tokenizer.decode(gen_tokens[response_pos + 1 : end_pos]).strip()<br><br>    return None<br><br># Sample similar to: &quot;Excited to announce the release of Dolly, a powerful new language model from Databricks! #AI #Databricks&quot;<br>print(generate_response(&quot;&quot;&quot;Write a linkedin post about my 2 year anniversary on Databricks, highlighting the following accomplishments<br>- Promotion to Sr Solutions Architect<br>- Earned &quot;Big Hitter&quot; award for largest deal in Latam<br>- Earned &quot;Solution Excellence&quot; award for contributions to the team<br>- Participation in Data + AI Summit<br>And thanking my managers Marcelo Sales and Luiz Antunes, also my partners Thiago Silveira, Rafael Ricotta and Renan Valente&quot;&quot;&quot;, model=model, tokenizer=tokenizer))</pre><p>You can easily identify that the prompt was:</p><blockquote>Write a linkedin post about my 2 year anniversary on Databricks, highlighting the following accomplishments<br>- Promotion to Sr Solutions Architect<br>- Earned “Big Hitter” award for largest deal in Latam<br>- Earned “Solution Excellence” award for contributions to the team<br>- Participation in Data + AI Summit<br>And thanking my managers Marcelo Sales and Luiz Antunes, also my partners Thiago Silveira, Rafael Ricotta and Renan Valente</blockquote><h3>What was the result?</h3><p>The response was as follows</p><blockquote>Happy Anniversary to Me and Databricks! 2 years ago I started working at Databricks as a Solutions Architect, and I couldn’t be more grateful for this amazing journey. Since then, I’ve been part of the development team and have worked on projects such as expanding the Databricks cloud, solving complex problems in a highly scalable way and contributing to the continuous improvement of our product.</blockquote><blockquote>I’m also very proud of my “Big Hitter” award and “Solution Excellence” award for my biggest deal in Latin America, as well as the participation at Data + AI Summit. But the most important thing I’m grateful for is the support I’ve received from my managers Marcelo Sales, Luiz Antunes and the rest of the Databricks team.</blockquote><blockquote>2 years ago, Databricks wasn’t what it is today, but I know my 2 years as a Solutions Architect have been an amazing journey. I’m very thankful and thankful to all my partners for their support and encouragement. As Databricks continues to grow, I’m sure I’ll have many more 2 year anniversaries!<br>#Anniversary#Databricks#ForTwoYears</blockquote><p>Which is pretty good. I decided to use most of it, with small adjustments (especially the job description it just invented for me) and adding the parts it decided to omit (my promotion, and some people to thank</p><p>I also gave the same prompt to chatGPT (free tier) to compare the responses.</p><blockquote>I am thrilled to announce that I have reached a significant milestone in my career — my 2-year anniversary at Databricks! It has been an incredible journey filled with remarkable experiences and achievements, and I am truly grateful for every opportunity that has come my way.</blockquote><blockquote>In the past two years, I have had the privilege of working with an exceptional team and collaborating with some of the most brilliant minds in the industry. I am incredibly proud to have earned several accolades during my tenure, including the “Big Hitter” award for securing the largest deal in Latam and the “Solution Excellence” award for my contributions to the team. Additionally, I was thrilled to participate in the Data + AI Summit, which was an unforgettable experience.</blockquote><blockquote>I would like to express my sincere gratitude to my managers, Marcelo Sales and Luiz Antunes, for their unwavering support and guidance throughout my journey. I am immensely grateful for the opportunities they have provided me and the confidence they have shown in my abilities. I would also like to thank my partners, Thiago Silveira, Rafael Ricotta, and Renan Valente, for their valuable contributions and the amazing work we have accomplished together.</blockquote><blockquote>I am looking forward to continuing my journey at Databricks and contributing to the team’s success. Here’s to many more years of growth, learning, and achievements!</blockquote><p>I will leave the comparison to the reader. I will just note that I found very odd that both models decided that my promotion was not worth mentioning.</p><h3>How was the performance?</h3><p>Not great. Especially comparing to chatGPT almost instant response (when the website is not too crowded).</p><p>Unfair comparison, as I ran the model in my personal computer. For the record, it took around 3 minutes to load the model, and then 43 minutes to generate the response. Well, not bad for a personal computer running on CPU in my opinion.</p><p>Nevertheless, it does not seem to be very pratical to use Dolly in a personal PC instead of ChatGPT services for personal purposes only</p><h3>So why use Dolly?</h3><p>Key feature is its openness. You can provide you own dataset to train it on, and have a more customized chatGPT for specific purposes.</p><p>For instance, if I was a more frequent writer at Linkedin, I could gather all my posts, create inputs that I wish would generate them, and feed them to the model. This would tune the model to “write more like me”.</p><p>Enterprise applications are endless though. Feeding your internal knowledge base to have a better assistant for your employees, or your FAQ to train a better chatbot for customer service, your investments offerings to have an AI powered financial advisory to offer your clients.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fecb3436f93d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey on Scala ML pipeline — part 3of 3: Dealing with NAs and XGBoost]]></title>
            <link>https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-3of-3-dealing-with-nas-and-xgboost-bfd91b6e93c7?source=rss-9b1abe8b6091------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfd91b6e93c7</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[xgboost]]></category>
            <dc:creator><![CDATA[Caio Ishizaka Costa]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 10:40:05 GMT</pubDate>
            <atom:updated>2020-06-05T12:47:07.904Z</atom:updated>
            <content:encoded><![CDATA[<h3>A journey on Scala ML pipeline — part 3of 3: Dealing with NAs and XGBoost</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vbVM-nK-pwhqWOFWkpt0GA.jpeg" /><figcaption>Where are all my values? Photo: Wikimedia</figcaption></figure><p>Welcome back. If you just landed at this article, you want to take a look at <a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-1-of-3-my-first-ml-pipeline-21776fc13fb1">parts 1</a> and <a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-2-of-3-custom-transformers-b9826945b46">part 2</a></p><p>This is also available in <a href="https://github.com/caioishizaka/machinelearning/tree/master/scala/Custom%20Transformations%20and%20XGBoost">scala notebooks in my git</a>, with pretty much all the text explanation as well. Some code may have been omitted from this article (especially imported packages), the notebooks were tested in their entirety.</p><p>We are made promises that XGBoost deals with NAs by itself. <a href="https://xgboost.readthedocs.io/en/latest/faq.html#how-to-deal-with-missing-value">https://xgboost.readthedocs.io/en/latest/faq.html#how-to-deal-with-missing-value</a></p><p>Though it is true, <strong>XGBoost4J-Spark </strong>implementation is <strong>not 100% satisfactory </strong>in that sense. Here we have to discuss the <strong>different versions </strong>extensively, as they vary greatly on this regard. There is very few documentation on this, so most of my knowledge comes from <strong>forum discussions and source code reading.</strong></p><p>First let’s talk about the role of missing values in an XGBoost tree. A tree will make a decision on a variable. If it is <strong>categorical</strong>, it will choose <strong>which categories go to each leaf</strong>. If it is <strong>numerical</strong>, it will determine a <strong>threshold</strong>, numbers <strong>under the threshold go left</strong>, numbers<strong> above threshold goes right.</strong></p><figure><img alt="Survival of passengers of Titanic Decision Tree Model, showing different criteria in each branch" src="https://cdn-images-1.medium.com/max/457/1*3BkKVdBt-tnxKMa5NFajEg.jpeg" /></figure><p>Now, what role does <strong>missing values play in this decision</strong>? For categorical variables it should be pretty simple, it <strong>is another category</strong>, no big deal. For numerical variables it is a bit different. A <strong>missing value is neither above or below a threshold</strong> (nor equal). So it could go either way. XGBoost deals with it the proper way, it <strong>decides where it should go</strong> (based on data). If a missing value shows up in test but was not present in training, it will default to some of the nodes (left, if I am not mistaken).</p><h4>Missing values on XGBoost4j</h4><p>Okay, you are now an expert on decision trees. Let’s go to XGBoost4j now. The following code will work (i.e., it will run and produce results) in 1.0, but will fail in both 0.8 and 0.9</p><pre>import org.apache.spark.ml.Pipeline<br>import org.apache.spark.ml.feature.{VectorAssembler}</pre><pre>val na_assembler = new VectorAssembler().setInputCols((index_columns.map(name =&gt; s&quot;${name}_index&quot;) ++ numerical_columns.map(name =&gt; s&quot;${name}_norm&quot;)).toArray)<br>                                             .setOutputCol(&quot;features&quot;)<br>                                             .setHandleInvalid(&quot;keep&quot;)</pre><pre>val na_stages = indexerArray ++ normalizerArray ++ Array(labelIndexer, na_assembler, xgbClassifier, labelConverter)<br>val na_pipeline = new Pipeline().setStages(na_stages)</pre><pre>val na_model = na_pipeline.fit(training)</pre><p>Let’s talk about what is going on. Missing values are handled very similarly in all versions, so let’s start with the basic behaviour. Here is what the process looks like:</p><ol><li>XGBoost gets the data</li><li>If there are sparse vectors, missing values are considered, well, missing</li><li>Additionally, any values in the vectors that match the <em>“missing”</em> parameter will also be considered missing (the name of the parameter is “missing”, the parameter is not missing)</li></ol><p>On version 0.9 they added a validation step</p><p>4. If “missing” is different than 0, you have to have “allow_non_zero_for_missing” to be true, otherwise it will error</p><p>Fun fact, it is<strong> impossible to make 0.9 work </strong>without setting<strong> </strong>&quot;missing&quot; -&gt; 0<strong>. </strong>Even if you pass the &quot;allow_non_zero_for_missing_value&quot; -&gt; true&quot;it will fail. I don’t know what they did or how this made to release (there is a <a href="https://github.com/dmlc/xgboost/blob/master/jvm-packages/xgboost4j-spark/src/test/scala/ml/dmlc/xgboost4j/scala/spark/MissingValueHandlingSuite.scala">specific test for that</a> in the repo), but it is true, I tested with Databricks 6.6 ML (which includes XGBoost4j 0.9 by default). Maybe a different distribution of 0.9 has that fixed.</p><p>Another fun fact, from version 0.9 to 1.0 they renamed “<em>allow_non_zero_for_missing_value</em>” to “<em>allow_non_zero_for_missing</em>”, so for version 0.9, remember that. I believe such parameter did not exist on 0.8.</p><h4>How does it work in practice</h4><p>Enough of (not so) fun facts, and back to the algorithm. Remember from chapter 1 how <em>VectorAssembler</em> can generate sparse vectors? Well,<strong> by default it will omit 0 as value</strong>. But that does not mean that there won’t be dense vectors with 0s. It will <strong>decide whether to use dense or sparse</strong> by itself.</p><p>Now comes the problem. <strong>Missing values will be treated as missing, but they are in reality 0</strong>. Also, there are <strong>0s in the Dense vectors, which will be treated as zeroes</strong> unless you set “missing” -&gt; 0. Therefore you are in great risk of having some 0s (represented in sparse vectors as missing) treated as missing and some 0s (in dense vectors) treated as 0s. For example, these 2 identical vectors (from <em>VectorAssembler</em>):</p><ul><li>[0, 3, [0,1], [1,1]] -&gt; third feature will be treated as missing</li><li>[1, 3, [1,1,0]] -&gt; third feature will be treated as zero</li></ul><p>By default, XGBoost is set “<em>missing</em>” -&gt; <em>NaN</em> and “<em>allow_non_zero_for_missing</em>” -&gt; <em>false</em>. In version 0.8, we don’t have (4) check, so it will succeed, but it will treat the same original value differently, depending on the representation (as explained above). <strong>A huge flaw.</strong> By setting “<em>missing</em>” -&gt; 0, though it is <strong>not 100% correct</strong> to treat it as missing, at least you are <strong>consistently treating all 0s as missing</strong>.</p><p>All in all, I believe the additional check (4) is a good one. The only thing I would change is the default “<em>missing</em>” value. It is set to <em>NaN</em> by default, I would put 0 as default. If you don’t specify “<em>missing</em>” and you have sparse vectors, it will fail. Make something with default parameters fail seems weird to me.</p><h4>So, what to do?</h4><p>Finally, some practical advice:</p><ol><li>For 0.8 and 0.9, don’t sweat that much. <strong>Just set “missing” -&gt; 0,</strong> or your model will either fail or produce wrong results. If you have real 0s and real NAs in the dataset, your model will fail anyway and you will need one of the following approaches.</li><li>For 1.0 just keep in mind what you are doing. If you are using <em>VectorAssembler</em>, keep in mind it is omitting 0s, so the <strong>only way to have a consistent model is to have “<em>missing</em>” -&gt; 0.</strong> Nevertheless, if you have both 0s and <em>NaN</em>, the model will fail.</li></ol><p>Now you may ask, how can I treat 0s and missing values correctly? Great question, let me go through 2 different paths. Inspiration comes from <a href="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html#dealing-with-missing-values">their tutorial</a>, which gives us 3 options.</p><ol><li><strong>Converting all vectors to Dense</strong>. Looks easy, we create a new transformer (which we are expert at doing right now) after the assembler, to convert all feature vectors to <em>DenseVector</em>. Nevertheless if you have a lot of NAs, it will be very inefficient on memory. If you don’t have memory constraints, it is an okay path to go.<strong> I briefly tested it, with poor results</strong>. It will work on 1.0, but it will fail on 0.8, and to make it work on 0.9 you will have to set “<em>missing</em>” -&gt; 0, so you need to <strong>replace your actual 0 values by something else</strong> (0.001 maybe?). I will not pursue this option here, but rather focus on the next 2.</li><li><strong>Converting missing values to something else upstream</strong>. Very pragmatic approach. Just convert all the nulls/nans to something else, so you don’t have to deal with them. I will show this method and discuss some shortcomings.</li><li><strong>Replace the assembler by a custom transformer</strong>, so we can indicate the <strong>sparse value</strong> ourselves, <strong>instead of getting 0 for sparsity</strong>. This can be way more storage efficient than (1), depending on how many missing values you have.</li></ol><h3>Replacing missing values by something else.</h3><p><strong>We don’t have to worry about categorical variables</strong>. For <em>StringIndexer</em>, null is just another category, and will be treated as so, no confusion.</p><p>Risk is on <strong>Numerical Values</strong>. Remember how we made the <em>NumericNormalizer</em> replace missing values with NaN? Well, we could go back and make it replace it with something else, and leave it as a param for the user to decide. As we are <strong>normalising for the mean</strong>, we can leave the<strong> default replacement for 0.0</strong>, which equals to the mean. I will comment further about replacing missing values by the mean later on, but as of now, it is a value as any other.</p><p>Key risk here is that we <strong>have actual 0s in our set</strong>. As we are normalising, it is <strong>very unlikely to have a lot of values that are equal to the mean</strong>. Even in low cardinality columns, you have to be pretty unlucky to have a numerical value to be exactly the mean. But feel free to examine your data beforehand. If you rather set it to -999999, feel free. But understand the consequences:</p><p><strong>-999999 is a number</strong>. In versions 0.8 and 1.0 you can set the missing value as -999999, but if you are using VectorAssembler, remember it may omit a few 0s (if they exist), so you <strong>end up with the same issue described above</strong>.</p><p>If you have both 0s and missing values in your data, I strongly suggest you to replace the 0s with something else (0.0001), and set the NAs to 0. That way, your VectorAssembler will omit the 0s, and 0s in the Dense vectors will be correctly treated as missing as well.</p><p>This is what our new transformer looks like.</p><pre>class NumericNormalizerWithReplacement extends NumericNormalizer {<br>  <br>  val replacementValue: Param[Double] = new Param(this, &quot;replacementValue&quot;, &quot;&quot;&quot;Value to replace Null and NaNs&quot;&quot;&quot;)<br>  <br>  setDefault(replacementValue, 0.0)<br>  <br>  def setReplacementValue(value: Double): this.type = set(replacementValue, value)<br>  <br>  override def transform(dataset: Dataset[_]): DataFrame = {<br>    transformSchema(dataset.schema, logging = true)<br>    <br>    var result = dataset.withColumn(${outputCol}, (col(${inputCol}) - ${mean})/ ${stdDev})<br>    ${handleInvalid} match {<br>      case &quot;keep&quot;  =&gt; result = result.withColumn(${outputCol},  when(col(${outputCol}).isNull, lit(${replacementValue})).otherwise(col(${outputCol}).cast(&quot;Double&quot;))) //Now replacing Nulls with replacementValue<br>      case &quot;skip&quot;  =&gt; result = result.filter(col(${outputCol}).isNotNull).filter(col(${outputCol}) === col(${outputCol})) //This will remove both Nulls and NaN<br>      case &quot;error&quot; =&gt; throw new SparkException(&quot;&quot;&quot;Error mode not supported, sorry&quot;&quot;&quot;)<br>    }<br>    return result <br>  }<br>}</pre><p>And model will train without further problems if we remember to set “<em>missing</em>” -&gt; 0.</p><pre>val normalizerWithReplacementArray = numerical_columns.map(column_name =&gt; {<br>  new NumericNormalizerWithReplacement()<br>    .setInputCol(column_name)<br>    .setOutputCol(s&quot;${column_name}_norm&quot;)<br>    .setHandleInvalid(&quot;keep&quot;)<br>    .setMean(data.agg(mean(column_name)).head().getDouble(0))<br>    .setStdDev(data.agg(stddev(column_name)).head().getDouble(0))<br>})</pre><pre>val xgbParam = Map(&quot;eta&quot; -&gt; 0.3,<br>      &quot;missing&quot;   -&gt; 0,<br>      &quot;max_depth&quot; -&gt; 3,<br>      &quot;objective&quot; -&gt; &quot;multi:softprob&quot;,<br>      &quot;num_class&quot; -&gt; 2,       <br>      &quot;num_round&quot; -&gt; 100,<br>      &quot;num_workers&quot; -&gt; 3,<br>      &quot;seed&quot; -&gt; 123)</pre><pre>val xgbClassifier = new XGBoostClassifier(xgbParam).setFeaturesCol(&quot;features&quot;).setLabelCol(&quot;label&quot;)</pre><pre>val na_2_stages = indexerArray ++ normalizerWithReplacementArray ++ Array(labelIndexer, na_assembler, xgbClassifier, labelConverter)<br>val na_2_pipeline = new Pipeline().setStages(na_2_stages)<br>val na_2_model = na_2_pipeline.fit(training)</pre><h4>Best Option — Features as custom Sparse Vector</h4><p>High of hope on this one. Documentation says it is the best approach and I believe it. Not blindly, but it makes sense. If we take off NaNs out of the vectors, then they can’t cause an error anymore. And we have plenty of evidence that it will work with sparse vectors.</p><p>The idea here is <strong>replacing the assembler entirely</strong>. Our brand new custom made assembler will return only <strong>sparse vectors</strong>, and it <strong>will omit only NaNs</strong>. As we are running the normaliser beforehand, it is guaranteed all null values are now NaNs. If you want to make it more generic (to consider both NaNs and nulls), you can change the <em>vectorizeRow</em> function as you will. You may even create a parameter to tell which value should be considered as missing.</p><p>Nevertheless, being very pragmatic, we will just grab the data, and put it into a <em>SparseVector</em>, omitting NaN values. Here’s my approach:</p><ul><li><em>vectorizeRow</em> is where the magic happens. As we will iterate over all the rows, nothing better than make it <strong>take a row and return a SparseVector</strong>. The SparseVector constructor takes 3 inputs, the size of the vector, an Array with the indices of non-zero values, and an Array with those values. You can check on code that it is very straightforward, grab the Row, get it size, transform it into array, get the indices of not-NaN values, get those values, construct and return the <em>Vector</em></li><li><em>SparseVectorAssembler</em> is going to be very similar to VectorAssembler. So similar that I will just extend it and override the transform part (even the transformSchema can be the same). Crazy right? Not so much if you think that is just a small twist on original VectorAssembler.</li></ul><pre>def vectorizeRow(row: Row): org.apache.spark.ml.linalg.SparseVector = {<br>  val size = row.length<br>  val row_array = row.toSeq.asInstanceOf[Seq[Double]].toArray<br>  val indices = row_array.zipWithIndex.filter(!_._1.isNaN).map(_._2)<br>  val values = row_array.zipWithIndex.filter(!_._1.isNaN).map(_._1)<br>  return new SparseVector(size, indices, values)<br>}</pre><pre>def vectorizeRowUDF = udf(vectorizeRow _)</pre><pre>class SparseVectorAssembler extends VectorAssembler {<br>  <br>  override def transform(dataset: Dataset[_]): DataFrame = {<br>    transformSchema(dataset.schema, logging = true)</pre><pre>return dataset.withColumn(${outputCol}, vectorizeRowUDF(struct(${inputCols}.map(col):_*)))<br>  }</pre><pre>}</pre><p>Beautiful. Now we don’t have to worry about 0s being considered missing. But will it work? Only in 0.8 and 1.0, if you remember to set “<em>allow_non_zero_for_missing” -&gt; true.</em></p><pre>val na_3_xgbParam = Map(&quot;eta&quot; -&gt; 0.3,<br>      &quot;allow_non_zero_for_missing_value&quot; -&gt; true,<br>      &quot;allow_non_zero_for_missing&quot; -&gt; true,<br>      &quot;max_depth&quot; -&gt; 3,<br>      &quot;objective&quot; -&gt; &quot;multi:softprob&quot;,<br>      &quot;num_class&quot; -&gt; 2,       <br>      &quot;num_round&quot; -&gt; 100,<br>      &quot;num_workers&quot; -&gt; 3)</pre><pre>val na_3_xgbClassifier = new XGBoostClassifier(na_3_xgbParam).setFeaturesCol(&quot;features&quot;).setLabelCol(&quot;label&quot;)</pre><pre>val na_3_stages = indexerArray ++ normalizerArray ++ Array(labelIndexer, na_3_assembler, na_3_xgbClassifier, labelConverter)<br>val na_3_pipeline = new Pipeline().setStages(na_3_stages)<br>val na_3_model = na_3_pipeline.fit(training)</pre><blockquote>auROC: Double = 0.9270027614523798</blockquote><p>No improvement in AUC. But that <strong>was not the point</strong>. ML is not black and white, the best approach will depend on the problem and the data. Instead of 1% NAs, what if it were 10%, 30%, 80%. Filtering out NAs is probably not going to yield better results.</p><h4>Final remarks</h4><p>You were given <strong>2 approaches on dealing with NAs </strong>(instead of filtering them out). I would <strong>strongly recommend using the last one</strong>. As you saw, it is really easy to implement (no matter what the documentation said about extra work), and it treats NAs as what they are, missing values. Also it removes any confusion caused by sparse vectors omitting 0s.</p><p>The other approach was replacing NAs by a value. Let me briefly talk about that as promised earlier.</p><p>I am a <strong>big fan of replacing NAs by the mean</strong> of the distribution. But this may skew the model when you have a lot of NAs. You can do something more sophisticated, <strong>replacing by N(m,s)</strong>, which is a random number based on a normal distribution with mean m and std dev s. This would <strong>keep your distribution “unharmed”</strong>. If it is normally distributed, of course. If you know more about your feature distribution beforehand, I highly recommend a tailored made approach (including to the normalisation).</p><p>But you must always ask yourself: <strong>what does a missing value actually represents</strong>? Quick example, suppose you run a survey, and ask <strong>if the respondent owns a car</strong>. If they say yes, you ask how many cars. But if they say no, they won’t be shown that question. In a dataset, that may come as a <strong>missing value for all people that answered no the car question</strong>. Now, is it fair to fill the data with the mean of the answers?<strong> Definitely not,</strong> in this case it is pretty clear that those missing values <strong>should be replaced by zero.</strong></p><p>Another case that it may be a problem is that when you have a lot of missing values. If you have <strong>80% of missing values</strong>, for instance, filling them with the mean (with or without random noise) will certainly<strong> bias your model </strong>towards the mean. Adding the <strong>random noise</strong> won’t help much as well. If the <strong>variance of data is just random</strong>, you <strong>won’t be able to model it</strong> (as it is random). You may fool yourself, and overfit model, but in the end, it is just noise. Any good model will just tell you that the variables don’t really matter, and return you the mean of your output as an answer. Every <strong>machine learning model</strong> is just trying to do the same thing,<strong> find correlation between features and output</strong>, to build a predictive model. And if you remember correctly from your statistics books, what is <strong>correlation</strong> but the <strong>covariance of the variables </strong>divided by the product of each<strong> standard deviation</strong>? In other words, it is just a measure if the variance of two variables seems to be coupled together. When one goes above the average, does the other go too?</p><p>Therefore, when I said I am a big fan of replacing NAs by the mean, <strong>it is not without boundary conditions</strong>. For a <strong>small portion of NAs</strong> (like 1% in our example), it should be more than fine. For 80%+ of missing data, definitely don’t do it. And you may ask, <strong>what kind of problem have so many missing values</strong>? A lot of them actually. As we navigate through a world of unstructured data, you will find a lot of holes in your dataset. Imagine which kind of features <strong>Google search algorithm</strong> uses to show YOU (specifically) the best results for a specific search query. Or the features FB uses to determine w<strong>hich is the best Ad to display</strong> to you. Imagine all the data it has for the people, and all the holes.</p><p>But let me finish here leaving you with the <strong>mother of sparse data problems</strong>. The one that was made famous in 2009. <strong>The Netflix Challenge</strong>. Giving the ratings every single user gave to each individual movie, what is the best estimate of the rating of all other movies (the user didn’t rate) for each user? The core of Netflix recommendation system. I won’t talk about the solution itself, but just the sparsity aspect of each. Millions of users, tens of thousands of movies. Even if we make a very generous assumption that the average user rates 100 movies/tv shows, it would still leave us with <strong>(at least) 99% of missing data</strong>. How would you feel about filling the gaps with the mean?</p><p>Hope this helped give you some clarity on how XGBoost4j works, how it treats missing values, and what is the best way to treat missing values yourself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfd91b6e93c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-3of-3-dealing-with-nas-and-xgboost-bfd91b6e93c7">A journey on Scala ML pipeline — part 3of 3: Dealing with NAs and XGBoost</a> was originally published in <a href="https://medium.com/red-ventures-br-tech">Red Ventures Brasil - Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey on Scala ML pipeline — part 2 of 3: Custom transformers]]></title>
            <link>https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-2-of-3-custom-transformers-b9826945b46?source=rss-9b1abe8b6091------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9826945b46</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Caio Ishizaka Costa]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 10:39:55 GMT</pubDate>
            <atom:updated>2020-06-05T12:45:55.799Z</atom:updated>
            <content:encoded><![CDATA[<h3>A journey on Scala ML pipeline — part 2 of 3: Custom transformers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*ifTo1UMb-bZhTF1Oy9evbA.jpeg" /><figcaption>No, not that kind of transformer. Photo: <a href="https://www.syfy.pt/noticas/10-coisas-que-nao-sabias-sobre-a-saga-transformers">https://www.syfy.pt/noticas/10-coisas-que-nao-sabias-sobre-a-saga-transformers</a></figcaption></figure><p>Welcome back. If you just landed at this article, you want to take a look at the <a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-1-of-3-my-first-ml-pipeline-21776fc13fb1">part 1</a>.</p><p>This is also available in <a href="https://github.com/caioishizaka/machinelearning/tree/master/scala/Custom%20Transformations%20and%20XGBoost">scala notebooks in my git</a>, with pretty much all the text explanation as well. Some code may have been omitted from this article (especially imported packages), the notebooks were tested in their entirety.</p><p>As promised, let’s make our<strong> first custom transformer</strong>. This will be very simple. You will pass which column you want to be normalised, the mean and the standard deviation to be considered, and the other params we are used to (handleInvalid and outputCol). We will use standard score, which is the easiest to implement and will achieve what we need in most cases. <a href="https://en.wikipedia.org/wiki/Standard_score">https://en.wikipedia.org/wiki/Standard_score</a></p><p>This will basically do</p><figure><img alt="Z equals fraction, X minus mu on top, sigma on the bottom" src="https://cdn-images-1.medium.com/max/210/1*MVOCPW1KpmCKP8DkYbASjw.png" /></figure><p>Where µ is the mean and σ is the standard deviation of the population.</p><h4>Transformer requirements</h4><p>In order for your transformer to work in a pipeline it need a few things.</p><ol><li><strong><em>transform()</em> method</strong>, as it will be called by the pipeline. It has to return a <em>Dataset/Dataframe</em></li><li><strong><em>transformSchema()</em> method</strong>. I don’t quite understand why. My best guess is that it will assess the schema of the result beforehand, to understand if there are any conflicts in the pipeline before starting transforming (and figuring out in the last step it is broken because a column is missing)</li><li>It has to <strong>extend <em>Transformer</em>.</strong> Probably not true, but this is definitely the easiest way to achieve a lot of the needs, extend the class you are mimicking anyway.</li><li>It needs an <strong><em>Identifiable</em></strong>. Also not sure why, but it does. The code below should handle this. Also, initialising it with an uid.</li></ol><pre>def this() = this(Identifiable.randomUID(&quot;numNormalizer&quot;)) </pre><p>5. <strong>Copy method</strong>. I am just using the default one. Just remember it has to return an object of the same class you are implementing, in this case <em>NumericNormalizer.</em></p><pre>override def copy(extra: ParamMap): NumericNormalizer = defaultCopy(extra)</pre><p>Lots of things, but the bulk of the work will be at (1), a little bit at (2) if necessary. And some work getting params right. Let’s go right to them.</p><h4>How params work</h4><p>I would be lying if I told you I know. I just went down a rabbit hole to make NAs work with XGBoost, ended up in the source code of <em>Assemblers</em> and <em>Pipeline</em>, and emerged with a way to do it.</p><p>What I can tell is, <em>params</em> are a nice way to <strong>store information</strong>, <strong>cannot be accessed from the outside</strong> (unless you have a method for that), and can be <strong>easily managed via methods</strong>.</p><p>What is the difference from an <em>attribute</em>? I honestly don’t know. They are different in nature and properties, but at a fundamental level I couldn’t tell you why use one or the other. I am just going with the flow, those <em>transformers</em> are usually implemented with <em>Params</em>, let’s mimic them.</p><p>Now for the real deal. We can extend many classes with those params, such as <em>HasInputCol</em>, <em>HasOutputCol</em> and <em>HasHandleInvalid</em>. This will create the params for us (and some methods as well), so it will be very similar to out of shelf <em>StringIndexer</em> and <em>VectorAssembler</em>, for instance.</p><p>We then start our class extending all of those.</p><pre>class NumericNormalizer(override val uid: String) extends Transformer with HasInputCol with HasOutputCol with HasHandleInvalid</pre><p>But we can create our own params from scratch.</p><p>We will create the params mean and stdDev. I decided to override <em>handleInvalid</em>, basically because it was giving me trouble. Very simple syntax, <em>new Param(parent, name, documentation).</em></p><pre>val mean: Param[Double] = new Param(this, &quot;mean&quot;, &quot;&quot;&quot;Mean of the column&quot;&quot;&quot;)<br>  val stdDev: Param[Double] = new Param(this, &quot;stdDev&quot;, &quot;&quot;&quot;Standard Deviation of the column&quot;&quot;&quot;)<br>  override val handleInvalid: Param[String]  = new Param[String](this, &quot;handleInvalid&quot;,<br>    &quot;&quot;&quot; How to handle Null and Nan values<br>        keep: will convert Null to NaN<br>        skip: will filter out Null or NaN<br>    &quot;&quot;&quot;, ParamValidators.inArray(Array(&quot;keep&quot;,&quot;skip&quot;))) </pre><p>After defining our parameters, we can set default values, if the user does not want to set anything in particular, and define the methods to allow users to define the parameters. Those methods have to return the same class you are implementing, and this.type is the most practical way to do it.</p><pre>  setDefault(mean, 0.0)<br>  setDefault(stdDev, 1.0)<br>  <br>  def this() = this(Identifiable.randomUID(&quot;numNormalizer&quot;))<br>  <br>  override def copy(extra: ParamMap): NumericNormalizer = defaultCopy(extra)<br>  <br>  def setOutputCol(value: String): this.type = set(outputCol, value)<br>      <br>  def setInputCol(value: String): this.type = set(inputCol, value)<br>  <br>  def setHandleInvalid(value: String): this.type = set(handleInvalid, value)<br>  <br>  def setMean(value: Double): this.type = set(mean, value)<br>  <br>  def setStdDev(value: Double): this.type = set(stdDev, value)</pre><p>To access a param inside the class you use <em>${ParamName}. </em>If you want, you may implement a <em>getParam </em>method, if you want to allow the user to check some parameter afterwards, using the <em>get</em> method. All these params methods comes from <em>org.apache.spark.ml </em>interface <em>Params, </em>and you can <a href="https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/ml/param/Params.html">check all you can do in the documentation</a></p><h4>Why don’t we calculate the mean and std dev ourselves</h4><p>Great question. While 100% possible, it has a<strong> fundamental flaw</strong>. Mean and Standard Deviation are <strong>depending on the data</strong>. If your test data has a different distribution than your training data, values will be mapped differently. On the most extreme situation, when you pass a dataset with one element, you will not have any comparison to normalize.</p><p>By giving <strong>fixed numbers </strong>(based on your training set, whole set, whatever you want) you avoid that pitfall.</p><h4>The transform</h4><p>Now that we know all the requirements let’s go down to implementation. As you can see, our transform method does 2 things:</p><ol><li>Treats null/nan values accordingly to the handleInvalid parameter</li><li>Applies the standard score, creating a new column in the process</li></ol><pre>override def transform(dataset: Dataset[_]): DataFrame = {<br>    transformSchema(dataset.schema, logging = true)<br>    <br>    var result = dataset<br>    ${handleInvalid} match {<br>      case &quot;keep&quot;  =&gt; result = result.withColumn(${outputCol},  when(col(${inputCol}).isNull, lit(Double.NaN)).otherwise(col(${inputCol}).cast(&quot;Double&quot;))) // This will replace nulls by NaNs<br>      case &quot;skip&quot;  =&gt; result = result.withColumn(${outputCol},col(${inputCol})).filter(col(${inputCol}).isNotNull).filter(col(${inputCol}) === col(${inputCol})) //This will remove both Nulls and NaN<br>      case _ =&gt; throw new SparkException(&quot;&quot;&quot;Error mode not supported, sorry&quot;&quot;&quot;)<br>    }<br>    <br>    return result.withColumn(${outputCol}, (col(${outputCol}) - ${mean})/ ${stdDev})<br>  }</pre><p>Not rocket science here. <strong>But this is beautiful</strong>. All the magic happens in this method, and the sky is the limit. Just make sure you don’t do anything that is not serialisable (crappy vague advice, I know, but I also don’t quite understand what is and what is not serialisable)</p><p>Quick note on replacing <em>nulls</em> by <em>NaN</em> on the “<em>keep</em>” method. This is will have great impact in Chapter 3. As of now, just do it.</p><h4>The transformSchema</h4><p>Key thing, call it on .<em>transform()</em>. Why? I don’t know, everyone does it, and if you don’t it will fail.</p><p>Now, this is super easy. You should be able to know what is going to be the resulting schema based on the dataset schema that is coming in. Just write it on words. In scala language.</p><pre>override def transformSchema(schema: StructType): StructType = {StructType(schema.fields :+ new StructField(${outputCol}, DoubleType, true))}</pre><p><strong>Potential issue</strong>: If the user chooses the outputCol as an existing column, this is probably going to fail. It just occurred me now, and I’m too lazy to test. Try it yourself, and if necessary, put a check there.</p><h4>NumericNormalizer in action</h4><p>Let’s see if this is going to work.</p><pre>import org.apache.spark.sql.functions.{mean, stddev}</pre><pre>val age_mean = data.agg(mean(&quot;age&quot;)).head().getDouble(0)<br>val age_stdDev = data.agg(stddev(&quot;age&quot;)).head().getDouble(0)</pre><pre>val ageNormalizer = new NumericNormalizer()<br>    .setInputCol(&quot;age&quot;)<br>    .setOutputCol(&quot;age_norm&quot;)<br>    .setHandleInvalid(&quot;keep&quot;)<br>    .setMean(age_mean)<br>    .setStdDev(age_stdDev)</pre><pre>display(ageNormalizer.transform(data).select(&quot;age&quot;,&quot;age_norm&quot;))</pre><figure><img alt="Table with two columns, age and age_norm. Value pairs (age and age_norm) are like: 39 and 0.03, 50 and 0.83, null and NaN" src="https://cdn-images-1.medium.com/max/900/1*etmWEBGkLC839QPgsAo9gg.png" /></figure><p>Looks like success to me. <em>Null</em> being mapped to <em>NaN</em>. Normalized values around 0. Now let’s <strong>create <em>normalizer</em> for all numerical data</strong>, and put in the <em>pipeline</em>. We will create a map of mean and standard deviations as well, to makes things clearer. Also, I have a feeling if you just make the calculation inside the <em>normalizer</em> setMean and setStdDev params, it will calculate it every time the transformer is called (not confirmed, it is just a guess)</p><p>Again, you have an <strong>option to use only training data or entire data</strong> to calculate the mean and standard deviation. <strong>I prefer to use the entire data</strong> to avoid any skew/bias the sampling may have caused. Though unlikely, it may happen. Also, I am confident that just knowing the mean and standard deviation of the entire set won’t cause any bias/overfitting on the model itself.</p><pre>import org.apache.spark.ml.Pipeline<br>import org.apache.spark.ml.feature.{VectorAssembler}</pre><pre>val mean_map = numerical_columns.map(column =&gt; column -&gt; data.agg(mean(column)).head().getDouble(0)).toMap<br>val stdDev_map = numerical_columns.map(column =&gt; column -&gt; data.filter(col(column) &lt; 500*mean_map(column)).agg(stddev(column)).head().getDouble(0)).toMap</pre><pre>val normalizerArray = numerical_columns.map(column_name =&gt; {<br>  new NumericNormalizer()<br>    .setInputCol(column_name)<br>    .setOutputCol(s&quot;${column_name}_norm&quot;)<br>    .setHandleInvalid(&quot;keep&quot;)<br>    .setMean(mean_map(column_name))<br>    .setStdDev(stdDev_map(column_name))<br>}).toArray</pre><pre>val normalized_assembler = new VectorAssembler().setInputCols((index_columns.map(name =&gt; s&quot;${name}_index&quot;) ++ numerical_columns.map(name =&gt; s&quot;${name}_norm&quot;)).toArray)                                                .setOutputCol(&quot;features&quot;)                                                .setHandleInvalid(&quot;skip&quot;)</pre><pre>val normalized_stages = indexerArray ++ normalizerArray ++ Array(labelIndexer, normalized_assembler, xgbClassifier, labelConverter)<br>val normalized_pipeline = new Pipeline().setStages(normalized_stages)<br>val normalized_model = normalized_pipeline.fit(training)</pre><p>After testing (check previous article or notebook in repo for the code), we got pretty much the same AUC.</p><blockquote>auROC: Double = 0.9259361494824455</blockquote><p>But that was not the point. You learned <strong>how to create a transformer </strong>(not the giant robot cars unfortunately), isn’t that cool? You can use it to anything. <strong>Removing outliers, grouping categorical values, </strong>making<strong> </strong>fun<strong> feature interaction, applying a function</strong> (log, exp, x², sin, …) to a feature/label, <strong>making sense of some text</strong>. You can do anything, you are a powerful data scientist/engineer with the right tools in hand.</p><p>A quick note on normalising. Though close to useless on XGBoost tree model, it is very important on other models, such as linear and logistic regression, specially if you are using <strong>regularisation</strong>. Regularisation is nothing but keeping the coefficients small to avoid overfitting. <strong>If your data is not normalised, your coefficients may vary in magnitude.</strong> It is very simple to realize that. Imagine you have a model:</p><figure><img alt="Y equals Beta zero plus Beta one X 1 plus Beta two X 2" src="https://cdn-images-1.medium.com/max/444/1*32AF-dfSoOr1ye0INEs_RA.png" /></figure><p>It is very easy to see that if you multiply all X_1 by 100, the resulting model is going to be:</p><figure><img alt="Y equals Beta 0 plus Beta 1 X 1 plus Beta two over one hundred X two" src="https://cdn-images-1.medium.com/max/436/1*i9kOfKXs4z1w6PjkzFL_aA.png" /></figure><p>Now if you regularise that, guess which coefficient is going to suffer more?Normalising deals with that and greatly improve the efficacy of regularisation methods.</p><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-1-of-3-my-first-ml-pipeline-21776fc13fb1">Back to part 1</a></p><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-3of-3-dealing-with-nas-and-xgboost-bfd91b6e93c7">To part 3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9826945b46" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-2-of-3-custom-transformers-b9826945b46">A journey on Scala ML pipeline — part 2 of 3: Custom transformers</a> was originally published in <a href="https://medium.com/red-ventures-br-tech">Red Ventures Brasil - Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A journey on Scala ML pipeline — part 1 of 3: My first ML pipeline]]></title>
            <link>https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-1-of-3-my-first-ml-pipeline-21776fc13fb1?source=rss-9b1abe8b6091------2</link>
            <guid isPermaLink="false">https://medium.com/p/21776fc13fb1</guid>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[pipeline]]></category>
            <dc:creator><![CDATA[Caio Ishizaka Costa]]></dc:creator>
            <pubDate>Fri, 05 Jun 2020 10:39:09 GMT</pubDate>
            <atom:updated>2020-06-05T12:46:42.683Z</atom:updated>
            <content:encoded><![CDATA[<h3>A journey on Scala ML pipeline — part 1 of 3: My first ML pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9obiUlhqErs1bYE-jdETGQ.jpeg" /><figcaption>Photo credit: Siemens Pipelines 4.0</figcaption></figure><p>Welcome to the series of 3 chapters (so far) on the ML journey using scala and pipelines. Motivation to write this is I couldn’t find many tutorials or examples on ML pipelines using scala (in fact, I could only find the <a href="https://xgboost.readthedocs.io/en/latest/jvm/xgboost4j_spark_tutorial.html">same tutorial in different places</a>).</p><p>Also I had a lot of trouble to make XGBoost to work with missing values, and even fewer references out there. I hope by the end of this series you will:</p><ol><li>Have a good understanding on<strong> how a ML pipeline works</strong>, and how to use it</li><li>Be able to <strong>create your own custom made transformers</strong> to use in your pipeline</li><li>Understand how <strong>missing values (NAs) are treated in XGBoost</strong>, and how to <strong>properly deal with them</strong></li></ol><p>If you wish, I have <a href="https://github.com/caioishizaka/machinelearning/tree/master/scala/Custom%20Transformations%20and%20XGBoost">scala notebooks in my git </a>with all the code (and basically all this text as well, with minor edits). Some of the code was omitted here (and the next chapters). If you wish to replicate the code, I strongly suggest accessing the repo.</p><h4>Quick note on versions</h4><p>For this I am using Databricks, on a cluster with 2 workers, using DBR 6.6. You should be able to yield the same results, as long as you are using spark 2.4.5 and scala 2.11</p><p>As for XGBoost, be careful. I will comment on different behaviors of the different versions, but for the bulk of this series I used <a href="https://mvnrepository.com/artifact/ml.dmlc/xgboost4j_2.11/1.0.0">XGBoost4J 1.0.0</a>. You will need both XGBoost4j and XGBoost4j-spark installed in the cluster for this to work. Should you choose Databricks own fork of that project, XGBoost-linux64, it is based on the 0.82 version of XGBoost4J. It is more stable for Databricks, but lacks significant features (such as calculating the gain of each variable)</p><p>All comments on versions will be about the XGBoost version, and will be referred simply as 1.0, 0.9 and 0.8, and the 0.8 will refer specifically to XGBoost-linux64</p><p>Shall we begin?</p><h3>Chapter 1: Just getting our first ML pipeline working with out-of-shelf transformers and estimators</h3><p>I decided to write this chapter because, unlike the common (and basically only) <a href="https://docs.databricks.com/applications/machine-learning/third-party/index.html#install-xgboost-on-databricks-runtime-ml">tutorial on XGBoost</a> you can find for XGBoost4J, this will have <strong>both categorical and numerical variable as features</strong>, and this will lead to unexpected issues.<br>Before we dive in, let’s talk about pipelines. The concept is rather simple. You create objects (called stages) that can be called on a dataset to produce a certain result. Yeah, crappy explanation, let’s go to the specifics for clarity. There are 2 types of stages:</p><ol><li><strong>Transformers</strong>: They literally transform a dataset, and <strong>return a new dataset</strong> with whatever transformations were specified</li><li><strong>Estimator</strong>: They will get a dataset, and create a predictive model out of it. <strong>The outcome is a ML model</strong>, that can be used to estimate a measure in new datasets</li></ol><p>You may find out more about ML pipeline in this link: <a href="https://spark.apache.org/docs/latest/ml-pipeline.html#pipeline">https://spark.apache.org/docs/latest/ml-pipeline.html#pipeline</a></p><h4>Why use ML pipeline?</h4><p>I hope by the end of this series you realize they are <strong>super simple to use</strong> (once you get the grasp of it) and generates a very <strong>neat, streamlined final code</strong>. Instead of doing a lot of data manipulation outside of the model, then passing a final dataset to the model, you create all transformations, and those <strong>transformations are embedded in the model</strong>. <br>This is specially beneficial when applying the model to a new dataset. Instead of running a lot of code to prepare the data to be able to use model.fit(), the end result of a pipeline model will automatically run all those transformations. If you are used to pipes in R, it is similar in concept (though a bit more powerful, in my opinion).</p><h4>Getting our hands dirty</h4><p>We need some data. So let’s get it from: <a href="https://archive.ics.uci.edu/ml/datasets/Adult">https://archive.ics.uci.edu/ml/datasets/Adult</a></p><p>I like it because it has more than a few thousand lines, a small (but not too small) number of features, numeric and categorical features, and a clear output (also categorical).<br>What I didn’t like was how it handled NAs (with a ‘?’), and also the numeric variables didn’t have any NAs.<br>My manipulation was merely to make sure “?” would come in as Nulls, and also added 1% nulls to age, capital_gain and capital_loss. You can find the <a href="https://github.com/caioishizaka/machinelearning/raw/master/data/adult_data_treated.csv">treated file in my github repo</a> as well.</p><p>After loading the data, let’s perform the mandatory <strong>sampling.</strong></p><pre>val Array(training, test) = data.randomSplit(Array(0.8, 0.2), seed = 123)</pre><p>Now we have our data. Let’s separate our <strong>categorical data </strong>(<em>index_columns</em>) from our <strong>numerical data</strong>(<em>numerical_columns</em>), because we will treat them differently later on, applying different transformations. Also, let’s keep our label (<em>income</em>) outside, as it does not belong to the feature space.</p><p>First decision we must make is choosing between education and <em>education_num</em> features. They are obviously 100% correlated, as <em>education_num</em> is just an integer representation of the factor. For simplicity of the example, I will use just <em>education</em>, but will discuss a little further when we treat it, and argue how education_number might produce better results.</p><pre>val index_columns = Seq(&quot;workclass&quot;,&quot;education&quot;,&quot;marital_status&quot;,&quot;occupation&quot;,&quot;relationship&quot;,&quot;race&quot;,&quot;gender&quot;,&quot;native_country&quot;)<br>val numerical_columns = Seq(&quot;age&quot;,&quot;fnlwgt&quot;,&quot;capital_gain&quot;,&quot;capital_loss&quot;,&quot;hours_per_week&quot;)</pre><h4>VectorAssembler — putting all features into one vector</h4><p>We are going to use XGBoost, which, like most ML models in scala, will take <strong>one vector column as feature,</strong> and <strong>one column as label</strong>.<br>In order to get the feature columns, I introduce our first transformer: <em>VectorAssembler</em>.</p><p>It is super easy, <em>setInputCols</em> with an array with all the column names you want to consider as feature, and a <em>setOutputCol</em> with the name of the column you want it to generate<br>You can also <em>setHandleInvalid</em>, to determine what will happen to <em>null/nan/missing values</em>. If you choose “<em>skip</em>”, the entire row will be removed if any of the feature columns is null/na/missing. If you choose “<em>keep</em>”, it will not remove it.</p><p>For now, let’s not worry about missing values, and just remove them.</p><pre>import org.apache.spark.ml.feature.{VectorAssembler}</pre><pre>val simple_assembler = new VectorAssembler().setInputCols((index_columns ++ numerical_columns).toArray)<br>     .setOutputCol(&quot;features&quot;)<br>     .setHandleInvalid(&quot;skip&quot;)</pre><p>Now let’s take a look. Remember how I said a transformer will .<em>transform</em>() a dataset. Let’s see how it does.</p><pre>try {<br>    display(simple_assembler.transform(training))<br>} <br>catch {<br>    case error: Throwable =&gt; println(&quot;That didn&#39;t went well: \n&quot; + error)<br>}</pre><blockquote>That didn’t went well: <br>java.lang.IllegalArgumentException: Data type string of column workclass is not supported. <br>Data type string of column education is not supported. <br>Data type string of column marital_status is not supported. <br>Data type string of column occupation is not supported. <br>Data type string of column relationship is not supported. <br>Data type string of column race is not supported. <br>Data type string of column gender is not supported. <br>Data type string of column native_country is not supported.</blockquote><p>Boom, <strong>it won’t accept strings</strong>. So we need to convert the strings to something it will accept. That’s what <em>StringIndexer</em> is for, to convert categorical variables into numbers (representing the category).</p><h4>StringIndexer — properly handling categorical variables</h4><p>A StringIndexer will transform column of values [A,B,B,A,C,D] into [0,1,1,0,2,3]. It will also keep track of that match. It is also pretty smart. It will <strong>order the appearances</strong>, so the <strong>most recurrent category will be 0</strong>, then 1 and so forth.</p><p>As promised, let’s discuss education_num again. <em>StringAssembler</em> will order the factors by appearance. That is probably great for most cases, the rarer the value, the higher the number, so it becomes easy to separate common values from rare ones. Nevertheless, it is not always the best relation. Say we have not that many people with just basic education, a lot of people with high education, and very few people with doctorate, for instance. Our assembler will put basic education closer to doctorate.<br>Therefore, if you have any <strong>hierarchy that makes sense beforehand, use it</strong>. I will leave this as to you, how would you consider education_num instead of education?</p><p>Back to the problem at hand. You will notice the <em>StringIndexer</em> options are very similar to VectorAssembler, which is great. Here we won’t “skip” the missing values, as we will deal with them in the assembler anyway.<br>Quick trick, instead of creating an indexer for each column manually, I will leverage the index_columns array we defined previously to make this automatically.</p><pre>import org.apache.spark.ml.feature.{StringIndexer}</pre><pre>val indexerArray = index_columns.map(column_name =&gt; {<br>  new StringIndexer()<br>    .setInputCol(column_name)<br>    .setOutputCol(s&quot;${column_name}_index&quot;)<br>    .setHandleInvalid(&quot;keep&quot;)<br>    .fit(data)<br>}).toArray</pre><p>Let’s first take a look at what those indexers do. Remember that they have a .<em>transform</em>() method, that will be return a Dataset. So we can nest all the <em>StringIndexer.transform()</em> to get the final result (which is exactly what the pipeline will do).</p><p>One quick note on versions. If you use 0.8, this will work even without the .<em>fit()</em> at the end. On 0.9 and 1.0, this will throw an error when you try to train the model. I don’t know exactly why, but I have a guess on what .<em>fit</em> does. I believe it is used to <strong>asses which category will be mapped</strong> to each Int beforehand (to figure out the most occurring values). But I would have to take a look at the source code to be sure, and ain’t nobody got time fo that (already had my fair share of source code reading writing this series).</p><p>If I am correct, then I have an additional advise. You will see on tutorials that the indexer is <strong>fitted for the training data</strong>. Which is fine in most cases, but in the rare occasion that you have a categorical value in your test set that is not present in your training data, I actually don’t know what will happen, but it might give you some trouble. Also, if your training data has a different distribution than your overall data, the indices might differ a bit.</p><p>Not a lot of strong reasons not to fit it to training data, but exactly 0 reasons not to fit it to the whole data. <strong>Just use the entire data to fit the indexer</strong>.</p><p>Now back to our indexer, let’s see it in action.</p><pre>var temp_data = training<br>indexerArray.foreach{indexer =&gt;<br>  temp_data = indexer.transform(temp_data)<br>}</pre><pre>display(temp_data)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v2CYpL3o62JPZGdoxS3dew.png" /></figure><p>Pretty cool, right? Now all those strings are stored as integers (actually double). Super efficient storage. And it will make our assembler work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpjNkZGVsdmBqmVNlna1fg.png" /></figure><p>Let’s take a closer look at the feature column. You will see it is a vector with 4 elements. Let’s explain those very quickly (as they will be key down the road)</p><ol><li><strong>0 or 1</strong>, indicating if this vector is Dense (1) or Sparse(0). A Dense vector will store all values of all columns. A Sparse vector will store the indices of the non-zero values and those values. This is done to save on memory and some performance gains. Remember how StringIndexer assigns 0 to the most recurrent values? <strong>This helps having as many 0s as possible</strong>, making <strong>Sparse vectors be more efficient</strong></li><li><strong>Size of the vector</strong>. Particularly important for Sparse vectors, as many fields may be omitted</li><li><strong>Indices list</strong>. Will be empty for Dense vectors (no point of indicating it as all values will be stored). Will have the indices of the non-zero values for Sparse vector</li><li><strong>The values</strong>. All of them for Dense vectors, the corresponding values of the indices for Sparse.</li></ol><p>Wooh, that was a lot. But if you aim to understand what will happen when we introduce NAs to the model, it is good to understand this.</p><p>Last step before putting everything into the pipeline. We will create two last transformations. One that will convert our <strong>label (<em>income</em>) to an index,</strong> and one that will <strong>undo that afterwards</strong>. That way we can use XGBoost (or any other model) safely, and get back a label that makes sense (not some randomly mapped 0 or 1).</p><pre>import org.apache.spark.ml.feature.{IndexToString}</pre><pre>val labelIndexer = new StringIndexer()<br>                            .setInputCol(&quot;income&quot;)<br>                            .setOutputCol(&quot;label&quot;)<br>                            .setHandleInvalid(&quot;skip&quot;) //no point keeping data we have no label on<br>                            .fit(data)</pre><pre>val labelConverter = new IndexToString()<br>                            .setInputCol(&quot;prediction&quot;)<br>                            .setOutputCol(&quot;predicted_income&quot;)<br>                            .setLabels(labelIndexer.labels)</pre><h4>Finally, the pipeline</h4><p>As for the pipeline, look how easy it is. You define the stages in an Array, create a pipeline, set those stages, and you’re good to go</p><pre>import ml.dmlc.xgboost4j.scala.spark.{XGBoostClassifier}<br>import org.apache.spark.ml.Pipeline</pre><pre>val xgbParam = Map(&quot;eta&quot; -&gt; 0.3,<br>      &quot;allow_non_zero_for_missing&quot; -&gt; &quot;true&quot;,             <br>      &quot;max_depth&quot; -&gt; 3,<br>      &quot;objective&quot; -&gt; &quot;multi:softprob&quot;,<br>      &quot;num_class&quot; -&gt; 2,       <br>      &quot;num_round&quot; -&gt; 100,<br>      &quot;num_workers&quot; -&gt; 3,<br>      &quot;seed&quot; -&gt; 123)</pre><pre>val xgbClassifier = new XGBoostClassifier(xgbParam).setFeaturesCol(&quot;features&quot;).setLabelCol(&quot;label&quot;)</pre><pre>val indexed_stages = indexerArray ++ Array(labelIndexer, indexed_assembler, xgbClassifier, labelConverter)<br>val indexed_pipeline = new Pipeline().setStages(indexed_stages)<br>val indexed_model = indexed_pipeline.fit(training)</pre><p>Quick remark about versions:<br>0.8: This will work normally, you can even omit the “allow_non_zero_for_missing” parameter 0.9: You will need to add “missing” -&gt; 0 parameter, and can omit “allow_non_zero_for_missing” parameter 1.0: This will work normally, but you have to keep either the “allow_non_zero_for_missing” parameter or introduce “missing” -&gt; 0</p><p>Don’t worry quite so yet, as we will go into details in Chapter 3</p><h4>Using the model</h4><p>Okay, first win. Let’s see how our model is doing.</p><p>I won’t go over different metrics, as it is not the scope here. I will stick to AUC (Area under ROC) which requires a simple column with (estimated probability, actual classification). If you are not familiarised with AUC, stop right now, as you shouldn’t be worrying about ML pipeline transformations at the moment</p><p>But first, let’s take a look at our results dataframe. And stop to appreciate how simple it is to apply the model on our test set. All the transformations are embedded, no manipulations required whatsoever. <strong>One simple <em>model.transform(test)</em> does everything for us</strong>. Doesn’t this make you want to cry tears of joy? No? Okay, let’s move on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Nv3YAhVjWmD--tVRoURqg.png" /></figure><p>Wow, it looks like it is failing a lot, what is wrong with it? I don’t have an answer, it is just like it has flipped all predictions. I don’t know why exactly it is doing it. If you have an answer, please tell me.</p><p>But in this kind of model I am more interested about the probabilities, and we can easily get that and use it as we want. Without further due.</p><pre>import org.apache.spark.mllib.evaluation.BinaryClassificationMetrics  <br>import org.apache.spark.sql.functions.col</pre><pre>def getProb = udf((vector: org.apache.spark.ml.linalg.Vector) =&gt; vector.apply(0))</pre><pre>val predictionAndLabels = results.select(getProb(col(&quot;probability&quot;)),col(&quot;label&quot;)).as[(Double, Double)].rdd</pre><pre>val metrics = new BinaryClassificationMetrics(predictionAndLabels)</pre><pre>val auROC = metrics.areaUnderROC<br>println(&quot;Area under ROC = &quot; + auROC)</pre><blockquote>Area under ROC = 0.9265931919954883</blockquote><p>Not bad at all for our first ML model. In the next chapters we will go down the rabbit hole of data science and do a number of different things to improve the model (and most likely fail miserably).</p><p>Among the things we can do, we will explore 2 in the next chapters:</p><ol><li><strong>Can normalising numerical variables make a difference</strong>? For XGBoost, which is basically a boosted tree model, intuition says no, as normalising won’t impact the tree decisions. But it is a great way to learn how to build a custom transformer. Also this can be helpful for other (not tree-based) models, such as linear regression (specially with regularisation).</li><li><strong>How can we make most of the data win Null/NaN/missing values</strong>? It is just 1% of missing values, but what if it were 30%? 80%? We couldn’t afford filtering all out.</li></ol><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-2-of-3-custom-transformers-b9826945b46">To part 2</a></p><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-3of-3-dealing-with-nas-and-xgboost-bfd91b6e93c7">To part 3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21776fc13fb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/red-ventures-br-tech/a-journey-on-scala-ml-pipeline-part-1-of-3-my-first-ml-pipeline-21776fc13fb1">A journey on Scala ML pipeline — part 1 of 3: My first ML pipeline</a> was originally published in <a href="https://medium.com/red-ventures-br-tech">Red Ventures Brasil - Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>