<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anthony Cros on Medium]]></title>
        <description><![CDATA[Stories by Anthony Cros on Medium]]></description>
        <link>https://medium.com/@cros-anthony?source=rss-94c84ac498fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*i9p52mPcMu96rgCk9V8Gdw.png</url>
            <title>Stories by Anthony Cros on Medium</title>
            <link>https://medium.com/@cros-anthony?source=rss-94c84ac498fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:07:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cros-anthony/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Transformations in Scala with Gallia: Version 0.4.0 Is Out]]></title>
            <link>https://medium.com/data-science/data-transformations-in-scala-with-gallia-version-0-4-0-is-out-f0b8df3e48f3?source=rss-94c84ac498fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0b8df3e48f3</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Anthony Cros]]></dc:creator>
            <pubDate>Thu, 27 Oct 2022 14:40:04 GMT</pubDate>
            <atom:updated>2022-10-28T05:27:19.072Z</atom:updated>
            <content:encoded><![CDATA[<h4>Presenting new features available with this supercharged release</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-QiHSUWChN0V_vgS" /><figcaption>Photo by <a href="https://unsplash.com/@theshubhamdhage?utm_source=medium&amp;utm_medium=referral">Shubham Dhage</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This is a follow-up article to my previous <a href="https://towardsdatascience.com/gallia-a-library-for-data-transformation-3fafaaa2d8b9">introduction to Gallia</a>, a schema-aware data transformation library in Scala. This article will focus on the most important changes that were included in the latest release: <a href="https://github.com/galliaproject/gallia-core/releases/tag/v0.4.0">0.4.0</a> (available for Scala <a href="https://search.maven.org/artifact/io.github.galliaproject/gallia-core_2.12/0.4.0/jar">2.12</a> and <a href="https://search.maven.org/artifact/io.github.galliaproject/gallia-core_2.13/0.4.0/jar">2.13</a>).</p><p><strong><em>Table Of Contents</em></strong></p><ul><li><a href="#166d"><em>Reading/Writing from Apache Avro/Parquet</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#ff5d"><em>Apache Avro</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#beec"><em>Apache Parquet</em></a></li><li><a href="#d0c0"><em>Reading from case classes</em></a></li><li><a href="#102c"><em>Union Types</em></a></li><li><a href="#d600"><em>Metaschema</em></a></li><li><a href="#a75a"><em>Complex aggregations</em></a></li><li><a href="#e3eb"><em>Complex transformations/cotransformations</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#d204"><em>Transformation via data class (for nested entities)</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#9a3b"><em>Cotransformation via a data class (for current level)</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#41f6"><em>Cotransformation via custom processing</em></a></li><li><a href="#069f"><em>Newly supported types</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#4987"><em>Enums</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#8a0a"><em>Binary data</em></a><em><br></em><strong><em>–</em></strong><em> </em><a href="#0964"><em>Temporal types</em></a></li><li><a href="#e9d4"><em>Conclusion</em></a></li></ul><h3>Reading/Writing from Apache Avro/Parquet</h3><p>Recall from the previous article that a typical processing in Gallia looks something like:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1beea07bd1ab13a55f7176550465be27/href">https://medium.com/media/1beea07bd1ab13a55f7176550465be27/href</a></iframe><p>Which produces:</p><pre>&lt;root&gt;<br> title            _String<br> air_date         _String<br> doctor           _Int</pre><pre>title                   | air_date         | doctor<br>----------------------- | ---------------- | ------<br>THE ELEVENTH HOUR       | 3 April 2010     | 11    <br>THE DOCTOR&#39;S WIFE       | 14 May 2011      | 11 <br>...</pre><p><strong>Note: </strong>A <em>TSV</em> version of the episodes dataset is available as a gist <a href="https://gist.github.com/anthony-cros/3db86cca82cf27d0ea42a724bd78555c">here</a>.</p><h4>Apache Avro</h4><p>Similarly, an Avro file can now be consumed via the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93d08fd338d5b00fc3cea4a2f50ac6a9/href">https://medium.com/media/93d08fd338d5b00fc3cea4a2f50ac6a9/href</a></iframe><p>Which produces the exact same result.</p><p>Conversely, one can write results to an Avro file with .writeAvro(“./mydata.avro”)</p><p><strong>Notes:</strong></p><ul><li>It does not matter whether the origin was itself an Avro file, as inputs and outputs in Gallia are completely independent</li><li>A direct consequence of the above note is that one could use Gallia purely for conversion purposes: <em>Avro-&gt;JSON, Avro-&gt;TSV, Avro-&gt;Parquet, Parquet-&gt;Avro</em>, etc. One caveat however: the data must comply with Gallia’s <a href="https://github.com/galliaproject/gallia-docs/blob/master/modeling.md">data model</a> (for instance, no use of Avro’s <a href="https://avro.apache.org/docs/1.10.2/spec.html#Maps">map</a>, no 2+D arrays, etc).</li><li>In the future, the streamAvro/writeAvro methods will be replaced with genericstream/write methods, with the extension acting as reference (see <a href="https://github.com/galliaproject/gallia-core/blob/v0.4.0/README.md#io">I/O</a> section)</li></ul><h4>Apache Parquet</h4><p>To process Parquet files instead of Avro, the code is exactly the same except:</p><ul><li>The module name is gallia-<strong>parquet</strong> instead of gallia-avro</li><li>The import is import gallia.<strong>parquet</strong>._ instead of import gallia.avro._</li><li>The methods names are stream<strong>Parquet</strong>/write<strong>Parquet</strong> instead of streamAvro/writeAvro</li></ul><p><strong>Notes:</strong></p><ul><li>Gallia uses the Avro converter under the hood, namely <a href="https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-avro/src/main/java/org/apache/parquet/avro/AvroParquetReader.java"><em>AvroParquetReader</em></a> and <a href="https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-avro/src/main/java/org/apache/parquet/avro/AvroParquetWriter.java"><em>AvroParquetWriter</em></a></li><li>In future versions, we will offer direct processing instead, via the custom <a href="https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/api/ReadSupport.java"><em>ReadSupport</em></a>/<a href="https://github.com/apache/parquet-mr/blob/apache-parquet-1.12.3/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/api/WriteSupport.java"><em>WriteSupport</em></a> abstractions provided by Parquet</li></ul><h3>Reading from case classes</h3><p>Let’s consider the following case class:</p><pre>    case class <strong>Person</strong>(<strong>name</strong>: String, <strong>age</strong>: Int, <strong>income</strong>: Int)</pre><p>It can now be ingested as a single entity:</p><pre>  val peter = gallia.<strong><em>aobjFromDataClass</em></strong>(<br>      <strong>Person</strong>(<strong>name</strong> = &quot;Peter&quot; , <strong>age</strong> = 29, <strong>income</strong> = 80000))</pre><p>Meanwhile, a collection thereof can be ingested as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4bc6b4e50759363c6e2cd24257a5723/href">https://medium.com/media/f4bc6b4e50759363c6e2cd24257a5723/href</a></iframe><p>So for instance, the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22f9f5c92d94055c16ee0944b8ebe966/href">https://medium.com/media/22f9f5c92d94055c16ee0944b8ebe966/href</a></iframe><p>Will produce the expected:</p><pre>&lt;root&gt;<br>  name             _String<br>  age              _Int<br>  hourly_rate      _Double</pre><pre>name   | age | hourly_rate      <br>------ | --- | ------------------<br>Peter  | 29  | 38.31417624521073<br>Joanna | 29  | 14.367816091954023<br>Samir  | 28  | 38.31417624521073</pre><p><strong>Note</strong>: While Gallia does not allow <strong>writing</strong> to case classes yet, it is scheduled for the next release — see <a href="https://github.com/galliaproject/gallia-core/blob/2a5c81f/NEXT_RELEASES.md">NEXT_RELEASES</a> (will allow it both via reflection and macros)</p><h3>Union Types</h3><p>Partial support for union types was also added in <a href="https://github.com/galliaproject/gallia-core/releases/tag/v0.4.0">0.4.0</a>.</p><p>An example of usage is as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/953dee9fda7a4dedea595dfa9acdf094/href">https://medium.com/media/953dee9fda7a4dedea595dfa9acdf094/href</a></iframe><p>This results in the following output:</p><pre>&lt;root&gt; [... schema unchanged]</pre><pre>name  | age<br>----- | ------------<br>Peter | 29<br>Samir | TWENTY-EIGHT</pre><p>Since toUpperCase is a String-only operation, the entity with an integer value for <em>age</em> is left untouched.</p><p>The above example presents a trivial case, but there can of course be more complex cases. An example would be when multiple nested entities are part of an union:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2019dbab66898397d05cfd30ddfb84e0/href">https://medium.com/media/2019dbab66898397d05cfd30ddfb84e0/href</a></iframe><p>It produces the following output:</p><pre>&lt;root&gt; [... schema unchanged]</pre><pre>{ &quot;data&quot;:           &quot;Michael (27yo)&quot; }<br>{ &quot;data&quot;: { &quot;name&quot;: &quot;<strong>PETER</strong>&quot;, &quot;age&quot;: 29 }}<br>{ &quot;data&quot;:           &quot;Joanna (29 years old)&quot; }<br>{ &quot;data&quot;: { &quot;name&quot;: &quot;Samir&quot;, &quot;dob&quot;: &quot;March 14, 1971&quot; } }</pre><p>Only the value &quot;Peter&quot; was upper-cased, due to the fact that it’s the only nested entity with an <em>age</em> entry.</p><p><strong>Notes:</strong></p><ul><li>See <a href="https://github.com/galliaproject/gallia-docs/blob/v0.4.0/union_types.md">union_types.md</a></li><li>More examples can be seen in <a href="https://github.com/galliaproject/gallia-testing/blob/v0.4.0/src/main/scala/galliatest/suites/UnionTypeTest.scala">UnionTypeTest.scala</a></li><li>Union types in Gallia are still considered experimental at this point, and not all operations support them (the essential ones do however).</li><li>One of the main reasons for supporting union types is to help data cleaning/feature engineering efforts. Indeed cases where fields are captured with different types are sadly very common in legacy datasets (think true mixed with&quot;Yes&quot;and the likes)</li></ul><h3>Metaschema</h3><p>A result of the added support for union type is the ability for Gallia to provide it’s own <a href="https://github.com/galliaproject/gallia-core/blob/v0.4.0/src/main/scala/gallia/MetaSchema.scala">metaschema</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64031fea0b68d2348f1897a904aedabd/href">https://medium.com/media/64031fea0b68d2348f1897a904aedabd/href</a></iframe><p>This means Gallia could dogfood itself for schema transformations. For instance here’s what a nested field renaming would look like (from <em>f</em> to <em>F </em>here):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b761e98891f85ba8d7109751a11ab51c/href">https://medium.com/media/b761e98891f85ba8d7109751a11ab51c/href</a></iframe><p><strong>Notes:</strong></p><ul><li>Gallia does not actually use the metaschema in this manner internally</li><li>A <em>schema</em> being a special case of data, namely <em>metadata</em> (or “data about data”), Gallia’s <em>metachema</em> is therefore also <em>metametadata</em>. And since Gallia’s <em>metaschema</em> can also be used to model itself, it is also its own <em>metametaschema</em>. Hence it is also <em>metametametadata</em>. Obviously.</li></ul><h3><strong>Complex aggregations</strong></h3><p>Let’s reuse thepeoplehandle from <a href="https://gist.github.com/anthony-cros/2760c3dbb6f5d9d2f8c3cd41edbb0ba6">earlier</a>. A very simple way to aggregate data would be:</p><pre>people<em><br>  </em>.<strong>group</strong>(&quot;name&quot; ~&gt; &quot;names&quot;).<strong>by</strong>(&quot;age&quot;)<br>  .printJsonl()</pre><p>It produces the following:</p><pre>{ &quot;age&quot;: 29, &quot;names&quot;: [ &quot;Peter&quot;, &quot;Joanna&quot; ] }<br>{ &quot;age&quot;: 28, &quot;names&quot;: [ &quot;Samir&quot; ] }</pre><p>It could also (unnecessarily) be achieved via:</p><pre>people<em><br>  </em>.<strong>aggregateBy</strong>(&quot;age&quot;).as(&quot;names&quot;)<br>    .<strong>using</strong> { _.strings(&quot;name&quot;) }<br>  .printJsonl()</pre><p>Which produces the same result as above, but shows a trivial use of aggregateBy.</p><p>And while there are more built-in aggregators readily available (<em>SUM BY</em>, <em>MEAN BY</em>, …) in Gallia, the aggregateBy construct will be necessary to achieve processing like this in single action:</p><pre>people<em><br>  </em>.aggregateBy(&quot;age&quot;).as(&quot;names&quot;)<br>    .using { _.strings(&quot;name&quot;).<strong>concatenateStrings</strong> }<br>  .printJsonl()</pre><p>Which produces:</p><pre>{ &quot;age&quot;: 29, &quot;names&quot;: &quot;<strong>PeterJoanna</strong>&quot; }<br>{ &quot;age&quot;: 28, &quot;names&quot;: &quot;<strong>Samir</strong>&quot; }</pre><p>And is actually a shorthand for:</p><pre>people<em><br>  </em>.aggregateBy(&quot;age&quot;).as(&quot;names&quot;)<br>    .using { _.strings(&quot;name&quot;)<br>      .<strong>mapV</strong> { _.reduceLeft(_ + _) } }<br>  .printJsonl()</pre><p>It could therefore be customized as required, for instance:</p><pre>people<em><br>  </em>.aggregateBy(&quot;age&quot;).as(&quot;names&quot;)<br>    .using { _.strings(&quot;name&quot;)<br>      .mapV { _.reduceLeft(<br>        <strong>_.toUpperCase + &quot;|&quot; + _.toUpperCase</strong>) } }<br>  .printJsonl()</pre><p>To produce:</p><pre>{ &quot;age&quot;: 29, &quot;names&quot;: &quot;<strong>PETER|JOANNA</strong>&quot; }<br>{ &quot;age&quot;: 28, &quot;names&quot;: &quot;<strong>SAMIR</strong>&quot; }</pre><p>But the real power of theaggregateBy construct lies in the ability to do the following — more custom — type of aggregations:</p><pre>people<em><br>  </em>.aggregateBy(&quot;age&quot;).using { <strong>group</strong> =&gt;<br>    <strong>(</strong>&quot;names&quot;       <strong>-&gt;</strong> <strong>group</strong>.strings(&quot;name&quot;)<strong>,</strong><br>     &quot;mean_income&quot; <strong>-&gt;</strong> <strong>group</strong>.ints   (&quot;income&quot;).mean<strong>)</strong> }<br>  .printJsonl()</pre><p>Which results in:</p><pre>{&quot;age&quot;: 29, &quot;names&quot;: [ &quot;Peter&quot;, &quot;Joanna&quot; ], &quot;mean_income&quot;: 55000.0}<br>{&quot;age&quot;: 28, &quot;names&quot;: [ &quot;Samir&quot; ],           &quot;mean_income&quot;: 80000.0}</pre><p><strong>Note</strong>: The tuple-based entity creation used above is just a shorthand for the more explicit gallia.headO(&quot;names&quot; -&gt; ...), available for up to 5 entries</p><h3><strong>Complex transformations/cotransformations</strong></h3><p>Let’s switch over to a more serious domain to highlight these new features. Consider the following dataset:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71b7332de1ee420f3eeed945d5592554/href">https://medium.com/media/71b7332de1ee420f3eeed945d5592554/href</a></iframe><p><strong>Note</strong>: Reminder that the bobj and bobjs constructs are a convenience mechanism that allows the construction of entities whose schema can be easily inferred. Thereforebobj(&quot;f&quot; -&gt; 1) is equivalent to the more explicitaobj(&quot;f&quot;.int)(obj(&quot;f&quot; -&gt; 1)).</p><h4><strong>Transformation via data class (for nested entities)</strong></h4><p>Gallia now offers the ability to transform nested entities via case classes (“data classes”). Consider for instance:</p><pre>case class <strong>Change</strong>(<br>   <strong>chromosome</strong>: String,<br>   <strong>position</strong>  : Int   ,<br>   <strong>from</strong>      : String,<br>   <strong>to</strong>        : String) {<br><br>  def<strong> shorthand</strong>: String<strong> </strong>=<br>    s&quot;${<strong>chromosome</strong>}:${<strong>position</strong>};${<strong>from</strong>}&gt;${<strong>to</strong>}&quot;<br><br>}</pre><p>Which models the <em>change</em> nested entity in the mutations dataset above, and encapsulates an operation that produces a shorthand notation for the genetic change (e.g. &quot;3:14532127;C&gt;GG&quot;).</p><p>The following code will transform the <em>change</em> entity to its shorthand counterpart:</p><pre>mutations<br>  .transformDataClass[<strong>Change</strong>](&quot;<strong>change</strong>&quot;)<br>    .using(_.<strong>shorthand</strong>)<br>  .display()</pre><p>Which produces:</p><pre>[...]</pre><pre>patient_id | age | change<br>---------- | --- | ----------------<br>p1         | 23  | 3:14532127;C&gt;GG<br>p2         | 22  | 4:1554138;C&gt;T<br>p3         | 21  | Y:16552149;AA&gt;GT</pre><p>Notes:</p><ul><li>This would apply to any other combination of optional/required and single/multiple for the nested entity, via .transformDataClass[<strong>Option</strong>[Change]], .transformDataClass[<strong>Seq</strong>[Change]], and .transformDataClass[<strong>Option</strong>[<strong>Seq</strong>[Change]]</li><li>Subsequent versions of Gallia will leverage macros to render this mechanism more efficient (currently relies on reflection)</li></ul><h4><strong>Cotransformation via a data class (for current level)</strong></h4><p>Let’s now consider the following case class which models a subset of the fields at the <strong>current level</strong> (as opposed to a nested entity this time):</p><pre>import java.time.Year</pre><pre>case class <strong>Demographics</strong>(<br>    <strong>age</strong>: Int,<br>    <strong>sex</strong>: String) {<br><br>  def<strong> </strong>toNewDemographics =<br>    <strong>NewDemographics</strong>(<br>      <strong>year_of_birth</strong>    = Year.<em>now</em>().getValue - age,<br>      <strong>has_Y_chromosome</strong> = sex == &quot;male&quot;)<br><br>}</pre><p>Along with the following case class which models a desired model change:</p><pre>case class <strong>NewDemographics</strong>(<br>    <strong>year_of_birth</strong>   : Int,<br>    <strong>has_Y_chromosome</strong>: Boolean)</pre><p>The following code co-transforms the two fields by using the encapsulated method in the origin case class:</p><pre>mutations<br>  .cotransformViaDataClass[<strong>Demographics</strong>]<br>    .usingWithErasing(_.<strong>toNewDemographics</strong>)<br>  .display()</pre><p>Which results in the expected:</p><pre>&lt;root&gt;<br> patient_id       _String<br> year_of_birth    <strong>_Int</strong><br> has_Y_chromosome <strong>_Boolean</strong><br> change ...</pre><pre>{ &quot;patient_id&quot;      : &quot;p1&quot;,<br>  &quot;change&quot;          : { ... },<br>  &quot;year_of_birth&quot;   : <strong>1999</strong>,<br>  &quot;has_Y_chromosome&quot;: <strong>true</strong> }<br>...</pre><p><strong>Notes:</strong></p><ul><li>As it stands it would make more sense to transform <em>age</em> and <em>sex</em> separately, but the goal here is to highlight a co-transformation where the fields could be mixed and matched at will</li><li>.usingWithErasing removes the original entries, while .using would preserve them</li></ul><h4><strong>Cotransformation via custom processing</strong></h4><p>Gallia also now offers an improved mechanism for truly custom processing, although generally a bad idea (as we lose the guarantee that the schema/data will co-evolve accordingly). For instance to reproduce the co-transformation above, we could create the following object, extending gallia.ObjToObj:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d68b9dfd8790b2f363ea3110c2308a2/href">https://medium.com/media/1d68b9dfd8790b2f363ea3110c2308a2/href</a></iframe><p>And the following code would make use of it:</p><pre>mutations<br>  .custom(<strong>CustomDemographicsTransformation</strong>)<br>  .display()</pre><p>This would yield the same output.</p><h3><strong>Newly supported types</strong></h3><p>Gallia now supports additional types:</p><ul><li>Enums</li><li>Binary data</li><li>Temporal types</li></ul><h4>Enums</h4><p>Enums can be created/used like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b91803dc51c9c972a0f5c82ea311ae7/href">https://medium.com/media/9b91803dc51c9c972a0f5c82ea311ae7/href</a></iframe><p>Which produces:</p><pre>&lt;root&gt;<br>  choice           _Enm(List(rock, paper, scissors, <strong>spock</strong>))</pre><pre>choice<br>------<br><strong>paper</strong></pre><h4>Binary data</h4><p>Binary data can be created/used like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4f9b207d7057e860e1d9c0ccd0cd183/href">https://medium.com/media/b4f9b207d7057e860e1d9c0ccd0cd183/href</a></iframe><p>Which produces:</p><pre>bin<br>-----------<br>base64:<strong>Ym9v</strong></pre><p><strong>Notes:</strong></p><ul><li>&quot;Ym9v&quot; encodes&quot;boo&quot; in <em>Base64</em>, since we put a&#39;b&#39;byte (0x62) where the &#39;f&#39;byte was (0x66).</li><li>The <em>Base64</em> output is only used for serialization, the in-memory representation remainsjava.nio.ByteBuffer throughout processing</li></ul><h4>Temporal types</h4><p>The temporal types supported in Gallia match their Java counterparts: java.time.{LocalDate, LocalTime, LocalDateTime, OffsetDateTime, ZonedDateTime, Instant}</p><p>They can be created/used like so (for instance with LocalDateTime):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4da4ccf99cc7dad0f694e6d6895707ce/href">https://medium.com/media/4da4ccf99cc7dad0f694e6d6895707ce/href</a></iframe><p>Which produces:</p><pre>&lt;root&gt;<br>   hungover         _LocalDateTime</pre><pre><strong>hungover</strong><br>-------------------<br><strong>2022-01-01</strong>T00:00:00</pre><h3>Conclusion</h3><p>This concludes our tour of the main changes brought by <a href="https://github.com/galliaproject/gallia-core/releases/tag/v0.4.0">0.4.0</a>, at least the ones that will improve the client code experience.</p><p>Other noteworthy additions include:</p><ul><li>Experimental integration with the <strong>Python</strong> ecosystem:<br>– <strong>Pandas</strong>: see <a href="https://github.com/galliaproject/gallia-python-integration/blob/v0.4.0/gallia-pandas/src/main/scala/gallia/pandas/ScalaPyPandasTest.scala#L10">ScalaPyPandasTest.scala</a> and <a href="https://github.com/galliaproject/gallia-python-integration/blob/v0.4.0/gallia-python-viz/src/main/scala/gallia/pyviz/GalliaPandasTests.scala#L29">GalliaPandasTest.scala</a><br>– <strong>Seaborn</strong>: see <a href="https://github.com/galliaproject/gallia-python-integration/blob/v0.4.0/gallia-python-viz/src/main/scala/gallia/pyviz/GalliaVizTest.scala#L37-L40">GalliaVizTest.scala</a></li><li>Experimental memory optimization for <strong>dense data</strong>, captured as(size: Int, data: Array[Any])instead of Array((Symbol, Any)), with some corresponding operations. See <a href="https://github.com/galliaproject/gallia-core/blob/v0.4.0/src/main/scala/gallia/obg9/Obg9.scala">Obg9.scala</a> and example usage at <a href="https://github.com/galliaproject/gallia-testing/blob/v0.4.0/src/main/scala/galliatest/suites/Obg9Test.scala">Obg9Test.scala</a></li></ul><p>For a more complete list of changes and improvements that came with this release, see <a href="https://github.com/galliaproject/gallia-core/blob/2a5c81f/CHANGELOG.md">CHANGELOG.md</a>.</p><p>Feedback is always welcome!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0b8df3e48f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/data-transformations-in-scala-with-gallia-version-0-4-0-is-out-f0b8df3e48f3">Data Transformations in Scala with Gallia: Version 0.4.0 Is Out</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gallia: a library for data transformation]]></title>
            <link>https://medium.com/data-science/gallia-a-library-for-data-transformation-3fafaaa2d8b9?source=rss-94c84ac498fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fafaaa2d8b9</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Anthony Cros]]></dc:creator>
            <pubDate>Mon, 06 Dec 2021 19:38:17 GMT</pubDate>
            <atom:updated>2021-12-14T06:24:06.062Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Gallia: A Library for Data Transformation</strong></h3><h4><em>A schema-aware Scala library for practical data transformation: ETL, feature engineering, HTTP responses, etc</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FbM8E3xGluQ6Vp3B" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://github.com/galliaproject/gallia-core/blob/master/README.md">Gallia</a> is a Scala library for generic data transformation with a focus on <strong>practicality</strong>, <strong>readability</strong>, and <strong>scalability</strong> (if needed).</p><p>It is a personal project which I started after years of frustration with existing tools. It is meant to help data engineers get the job done without necessarily forgoing scalability if it is required. It is also intended to fill a gap between libraries like <em>pandas</em> (for those who value a powerful type system like Scala’s) and <em>Spark SQL</em> (for those who find SQL hard to fathom past a certain query complexity). More generally, it was created to offer a one-stop shop paradigm for most or all data transformations needs within one’s application.</p><p>Its execution happens in <strong>two phases</strong>, each traversing a dedicated execution <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graph (DAG)</a>:</p><ol><li>A initial <strong>meta</strong> phase which ignores the data entirely and ensures that transformation steps are consistent (schema-wise).</li><li>A subsequent <strong>data</strong> phase where the data is actually processed.</li></ol><figure><img alt="Gallia logo" src="https://cdn-images-1.medium.com/max/142/0*HLpNZD4B102UnmIU" /><figcaption>Image via LogoGround under license to Anthony Cros.</figcaption></figure><h3>Trivial example</h3><p>Here’s a very basic example of usage:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c7bc99363e21b09db1cb39ded460516/href">https://medium.com/media/6c7bc99363e21b09db1cb39ded460516/href</a></iframe><p>This will successfully print: <em>{“</em><strong><em>name</em></strong><em>”:”TONY”,”</em><strong><em>age</em></strong><em>”:40}</em></p><p>However, because Gallia is schema-aware, the following will fail before any data is actually processed:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/552165e0cc8f4b2386665c8d4789e83b/href">https://medium.com/media/552165e0cc8f4b2386665c8d4789e83b/href</a></iframe><p>Because incrementing a string is (typically) nonsensical, the initial “meta” phase will return an error complaining that a string cannot be incremented.</p><p>Notes:</p><ul><li><em>JSON</em> is used for all examples due to its ubiquity as an object notation, however Gallia is not <em>JSON</em>-specific for its data serialization or in-memory representation (see <a href="http://github.com/galliaproject/gallia-docs/blob/master/json.md">JSON flaws</a>)</li><li>The schema is actually inferred here, which is more succinct but typically not ideal (see <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#schema-metadata">providing schemas</a> instead)</li></ul><h3>More complex example</h3><p>Let’s walk through a more complex example of usage. Your boss, Bill, provides you with the following spreadsheets (graciously dumped as TSV):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4169fb99a3fe16849f3129e176ee0c1a/href">https://medium.com/media/4169fb99a3fe16849f3129e176ee0c1a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b657a4298af5ee70ed6183c49bac279/href">https://medium.com/media/3b657a4298af5ee70ed6183c49bac279/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/314a82cf139e34b2244ba8c48b767f1e/href">https://medium.com/media/314a82cf139e34b2244ba8c48b767f1e/href</a></iframe><p>He would like you to create a report for each employee at the company (<em>Initech</em>), and projects “W2K” and “W3K”, based on the following template:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2e9900856509d4667a74871b456f6a9/href">https://medium.com/media/b2e9900856509d4667a74871b456f6a9/href</a></iframe><h3>Code and data</h3><p>You can achieve the above result with Gallia and the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e1e00daf551c44c95d6e85e6a5d392c/href">https://medium.com/media/0e1e00daf551c44c95d6e85e6a5d392c/href</a></iframe><p>Runnable code for this example can be found <a href="https://github.com/galliaproject/gallia-medium-article">on github</a>. The repository also contains the input and output data, along with a dump of all intermediate top-level schema/data pairs, which should help clarify any ambiguity. If we consider line 31 above for instance (generate <em>is_manager</em> field), we can find all intermediates files like so:</p><ul><li><strong>input</strong> schema: <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/30.meta.txt">./data/intermediate/30.meta.txt</a></li><li><strong>input</strong> data: <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/30.data.json">./data/intermediate/30.data.json</a></li><li><strong>output</strong> schema: <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/31.meta.txt">./data/intermediate/31.meta.txt</a></li><li><strong>output</strong> data: <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/31.data.json">./data/intermediate/31.data.json</a></li></ul><p>Note that this is not a standard Gallia feature, just a convenience provided for the sake of the article. Note also that only the first data entity is provided for each step.</p><p><strong>EDIT: </strong>I also added a <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/src/main/scala/galliamedium/initech/InitechManual.scala">manual counterpart</a>, which can be compared easily with <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/src/main/scala/galliamedium/initech/InitechGallia.scala">Gallia processing</a>. The manual counterpart exclusively uses the standard library.</p><h3>Walkthrough</h3><p>The transformation steps above are meant to be as self-explanatory as possible. For instance .remove(“Employee ID”) means exactly that: the “Employee ID” field will no longer exist past this point.</p><p>Instead we will detail below a few of the more involved operations.</p><h4>Nesting operation</h4><p>The first nesting operation is straightforward:</p><pre>.nest(“first”, “middle”, “last”).under(“name”)</pre><p>It basically turns an entity:</p><p><em>{“</em><strong><em>first</em></strong><em>”: “Peter”, “</em><strong><em>middle</em></strong><em>”: “Ron”, “</em><strong><em>last</em></strong><em>”: “Gibbons”, …}</em></p><p>into</p><p><em>{“</em><strong><em>name</em></strong><em>”: { “</em><strong><em>first</em></strong><em>”: “Peter”, “</em><strong><em>middle</em></strong><em>”: “Ron”, “</em><strong><em>last</em></strong><em>”: “Gibbons”}, … }</em></p><p>However the second nesting operation is (intentionally) more complex:</p><pre>.renest{ <strong>_.filterKeys(_.startsWith(“Address”))</strong> }.usingSeparator(“ “)</pre><p>It takes advantage of two mechanisms:</p><ul><li>Selecting a subset of keys via a predicate, namely startsWith(“Address”) here</li><li>Re-nesting using a common separator, namely the space character here</li></ul><p>The target selection zeroes in on the only four fields with an “Address” prefix. Meanwhile the renesting mechanism uses the provided separator to reconstruct the implied nesting:</p><p>Therefore we go from:</p><pre>{<br>  ...<br>  &quot;Address number&quot;:  9,<br>  &quot;Address street&quot;: &quot;Channel Street&quot;,<br>  &quot;Address city&quot;  : &quot;Houston&quot;,<br>  &quot;Address zip&quot;   :  77001,<br>  ...<br>}</pre><p>to:</p><pre>{<br>  ...<br>  &quot;Address&quot;: {<br>    &quot;number&quot;:  9,<br>    &quot;street&quot;: &quot;Channel Street&quot;,<br>    &quot;city&quot;  : &quot;Houston&quot;,<br>    &quot;zip&quot;   :  77001<br>  },<br>  ...<br>}</pre><p>Note that this is very typical of processing data which has been “flattened” to fit a rectangular format.</p><p>More details on re-nesting with Gallia can be found <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#renesting-tables">here</a></p><h4>Bring operation</h4><pre>issues.bring(projects, target = &quot;Name&quot;, via = &quot;Project ID&quot; &lt;~&gt; &quot;ID&quot;)</pre><p>This statement can be read in plain English as &quot;bring field <em>Name </em>from <em>projects</em> into <em>issues</em> via their matching fields&quot;. Here, we have to explicitly name the matching fields because their keys differ (<em>&quot;Project ID&quot;</em> vs <em>&quot;ID&quot;</em>), otherwise they could be guessed¹.</p><p>&quot;<em>bring</em>&quot; is a special type of left join. It is used for convenience when one simply wants to augment data with a few fields from another source. This is in contrast with:</p><ul><li>A Gallia <em>join,</em> which would correspond to the namesake <em>SQL</em> operation, and therefore imply a potential denormalization².</li><li>A Gallia <em>co-group</em>, which would correspond to the namesake <em>Spark</em> operation: no denormalization but the grouped sides end up being nested under <em>_left</em> and <em>_right</em> fields respectively.</li></ul><h4>Pivot operation (nested)</h4><pre>.transformObjects(“issues”).using {<br>  _ .countBy(“Status”) // defaults to “_count”<br>    .pivot(_count).column(“Status”)<br>      .asNewKeys(“OPEN”, &quot;IN_PROGRESS&quot;, “RESOLVED”) }</pre><p>The first thing to note here is that we are applying a transformation to nested elements, as opposed to data elements at the root of each entity. The underscore, which has a special meaning in Scala, therefore corresponds to each entity nested under the <em>issues </em>field<em> </em>(from the preceding group-by Employee ID operation, see <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/50.meta.txt">intermediate meta</a> and <a href="https://github.com/galliaproject/gallia-medium-article/blob/master/data/intermediate/50.data.json">data</a>).</p><p>From here, for all such “issue” entities, we do a count by status (<em>OPEN</em>, <em>IN_PROGRESS</em>, or <em>RESOLVED</em>), followed by a pivot on that count. That is if these are the issues for a given assignee:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2038b0d0cfba2614dc9e4fb6f55d596c/href">https://medium.com/media/2038b0d0cfba2614dc9e4fb6f55d596c/href</a></iframe><p>then the counting operation results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f4cfaa270799f15a124c6d1a2223da5/href">https://medium.com/media/8f4cfaa270799f15a124c6d1a2223da5/href</a></iframe><p>and the pivot results in:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b564bcbe3e55b3a70ba3dab58de44fd7/href">https://medium.com/media/b564bcbe3e55b3a70ba3dab58de44fd7/href</a></iframe><p>It should be noted that the .asNewKeys(“OPEN”, “IN_PROGRESS”, “RESOLVED”) part is required due to the — very important — fact that Gallia maintains a schema throughout transformations. In the present situation, the values for <em>Status</em> can only be known <strong>after</strong> looking at the entire set of data³, and must therefore be provided ahead of time. More details on this matter can also be found in the <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#schema-metadata">schema</a> and <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#dag-heads">(DAG) heads</a> sections of the documentation.</p><h3>Future articles</h3><p>In future articles I’d like to discuss ways to achieve the same result using alternative technologies such as <em>SQL, Spark sql, pandas, </em>etc (relates to <a href="https://github.com/galliaproject/gallia-docs/blob/master/bounties.md">this idea</a>)</p><p>I’d also like to discuss how the above transformations can scale using <em>Spark RDDs</em>, support for which is built-in Gallia (see <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#spark">documentation</a>). The above example is intentionally trivial, but if the data contained billions of rows, the code would not be able to handle it as it is.</p><h3>Conclusion</h3><p>For a real life and significantly more complex example, <a href="https://github.com/galliaproject/gallia-dbnsfp">see this</a> repository. Also see the <a href="https://github.com/galliaproject/gallia-core/blob/master/README.md#examples">full list of examples</a>.</p><p>Gallia’s main <strong>strengths</strong> can be summed up like so:</p><ul><li>The most common/useful data operations are provided, or at least <a href="https://github.com/galliaproject/gallia-docs/blob/master/tasks.md">scheduled</a>.</li><li>Readable DSL that domain experts should be able to at least partially comprehend.</li><li>Scaling is not an afterthought and <em>Spark RDDs</em> can be leveraged when required.</li><li>Meta-awareness, meaning inconsistent transformations are rejected whenever possible (for instance, cannot use a field that’s been removed already).</li><li>Can process individual entities, not just collections thereof; that is, there’s no need to create “dummy” collections of one entity in order to operate on that entity.</li><li>Can process nested entities of any multiplicity in a natural way.</li><li>Macros are <a href="https://github.com/galliaproject/gallia-macros">available</a> for a smooth integration with case class hierarchies.</li><li>Provides flexible target selection — i.e. which field(s) to act on — which ranges from explicit reference to actual queries, including when nesting is involved.</li><li>The execution DAG is sufficiently abstracted that its optimization is a well-separated concern (e.g. predicate pushdowns, pruning, …); note however, that few such optimizations are in place at the moment.</li></ul><p>Gallia is still a fairly new project and as such I’m looking forward to any feedback on it! The main repository can be found <a href="https://github.com/galliaproject/gallia-core">on github</a>, and the <a href="https://github.com/galliaproject">parent organization</a> contains all related repositories.</p><figure><img alt="Gallia logo" src="https://cdn-images-1.medium.com/max/142/0*HLpNZD4B102UnmIU" /><figcaption><em>Image via LogoGround under license to Anthony Cros.</em></figcaption></figure><h3>Footnotes</h3><p>[1]: This would not be the case if relations between the classes were provided (for instance via <em>implicits</em>), see two related tasks:</p><ul><li><a href="https://github.com/galliaproject/gallia-docs/blob/master/tasks.md#t210426094707">providing relations (<em>t210426094707</em>)</a></li><li>more generally: <a href="https://github.com/galliaproject/gallia-docs/blob/master/tasks.md#t210124100546">semantics (<em>t210124100546</em>)</a></li></ul><p>Note also that <em>bring</em>/<em>join</em>/<em>co-group</em> operations on more than one key is not available yet: see task <a href="https://github.com/galliaproject/gallia-docs/blob/master/tasks.md#t210304115501">multiple keys (<em>t210304115501</em>)</a>. The workaround is to generate a temporary field.</p><p>[2]: <em>bring</em> and <em>left join</em> are functionally equivalent if there are never multiple matches on the right-side.</p><p>[3]: This may be circumvented if the field is explicitly set to be an <em>enum, </em>which is not the case in our example because we used schema inferring. Gallia may also offer “opaque objects” in the future: see corresponding <a href="https://github.com/galliaproject/gallia-docs/blob/master/tasks.md#t210202172304">task (<em>t210202172304</em>)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fafaaa2d8b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gallia-a-library-for-data-transformation-3fafaaa2d8b9">Gallia: a library for data transformation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>