<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Pyspark - Medium]]></title>
        <description><![CDATA[Pyspark tips methods - Medium]]></description>
        <link>https://medium.com/pyspark?source=rss----66cc7430d0e7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Pyspark - Medium</title>
            <link>https://medium.com/pyspark?source=rss----66cc7430d0e7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:45:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/pyspark" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Pyspark tips — Caste and filter dataframe by last modification]]></title>
            <link>https://medium.com/pyspark/pyspark-tips-caste-and-filter-dataframe-by-last-modification-484dbb3bb9b5?source=rss----66cc7430d0e7---4</link>
            <guid isPermaLink="false">https://medium.com/p/484dbb3bb9b5</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[dataframes]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Herculano Cunha]]></dc:creator>
            <pubDate>Mon, 20 Feb 2023 20:42:30 GMT</pubDate>
            <atom:updated>2023-02-28T22:58:03.087Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pyspark tips — Cast and filter dataframe by last modification</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Y-_vAtuSr4fY-lUrESoCUg.png" /><figcaption>PySpark</figcaption></figure><p>Hello guys, in this story I come t share two methods for reuse in yours spark pipelines.</p><p>The repository is <a href="https://github.com/herculanocm/pyspark-tips">here</a> and you’re wellcome to edit and fork.</p><p>Context:<br>Me and my team use these methods in ours pipelines in AWS Glue CDC, the AWS DMS deposit the files in S3 bucket, the glue pipeline consumes until AWS Redshift.</p><p>First method — cast_columns_types_by_schema</p><pre>def cast_columns_types_by_schema(df: DataFrame, list_schema: list, empty_columns_to_none: bool = False) -&gt; DataFrame:<br>    if (df is None or list_schema is None) or (len(list_schema) == 0):<br>        raise ValueError(&#39;The values could by not null or empty&#39;)<br>    else: <br>        for column in list_schema:<br>            fieldname = column[&#39;column_name&#39;]<br>            fieldtype = column[&#39;data_type&#39;].lower()<br><br>            if &#39;int&#39; in fieldtype and &#39;big&#39; not in fieldtype:<br>                df = df.withColumn(fieldname, F.col(fieldname).cast(IntegerType()))<br><br>            elif &#39;bool&#39; in fieldtype:<br>                df = df.withColumn(fieldname, F.col(fieldname).cast(BooleanType()))<br>            <br>            elif &#39;numeric&#39; in fieldtype or &#39;decimal&#39; in fieldtype or \<br>                &#39;double&#39; in fieldtype or &#39;float&#39; in fieldtype or \<br>                &#39;real&#39; in fieldtype or &#39;money&#39; in fieldtype or \<br>                &#39;currency&#39; in fieldtype:<br>                df = df.withColumn(fieldname, F.regexp_replace(F.col(fieldname), &quot;,&quot;, &#39;.&#39;))\<br>                    .withColumn(fieldname, F.col(fieldname).cast(DoubleType()))<br><br>            elif &#39;date&#39; == fieldtype:<br>                df = df.withColumn(fieldname,<br>                F.when(F.col(fieldname).like(&#39;%/%/%&#39;), F.to_date(F.col(fieldname), &#39;dd/MM/yyyy&#39;))<br>                .otherwise(F.date_format(F.col(fieldname), &quot;yyyy-MM-dd&quot;)).cast(DateType())<br>                )<br>                    <br>            elif &#39;datetime&#39; == fieldtype or &#39;timestamp&#39; in fieldtype:<br>                df = df.withColumn(fieldname, F.col(fieldname).cast(TimestampType()))<br><br>            else:<br>                df = df.withColumn(fieldname, F.col(fieldname).cast(StringType())) \<br>                .withColumn(fieldname, F.regexp_replace(F.col(fieldname),&quot;[\\r\\n]&quot;, &#39;&#39;)) \<br>                .withColumn(fieldname, F.trim(F.col(fieldname)))<br><br>                if empty_columns_to_none:<br>                    df = df.withColumn(fieldname, F.when(F.col(fieldname)==&quot;&quot;,None).otherwise(F.col(fieldname)))<br><br>        return df</pre><p>The purpose of this method is apply the correct datatype to each column according to your schema. Clone the code from Github and apply the tests to see it working.</p><p>Basically it receive the list of schema for conversion</p><pre>list_schema = [<br>        {&#39;column_name&#39;: &#39;id&#39;, &#39;data_type&#39;: &#39;INTEGER&#39;},<br>        {&#39;column_name&#39;: &#39;data&#39;, &#39;data_type&#39;: &#39;DATE&#39;},<br>        {&#39;column_name&#39;: &#39;ativo&#39;, &#39;data_type&#39;: &#39;BOOLEAN&#39;},<br>        {&#39;column_name&#39;: &#39;unidade&#39;, &#39;data_type&#39;: &#39;VARCHAR(200)&#39;},<br>        {&#39;column_name&#39;: &#39;sigla&#39;, &#39;data_type&#39;: &#39;VARCHAR(2)&#39;},<br>        {&#39;column_name&#39;: &#39;valor&#39;, &#39;data_type&#39;: &#39;NUMERIC(10,2)&#39;},<br>        {&#39;column_name&#39;: &#39;obs&#39;, &#39;data_type&#39;: &#39;VARCHAR(MAX)&#39;},<br>        {&#39;column_name&#39;: &#39;action&#39;, &#39;data_type&#39;: &#39;VARCHAR(1)&#39;},<br>        {&#39;column_name&#39;: &#39;data_transaction&#39;, &#39;data_type&#39;: &#39;TIMESTAMP&#39;}<br>]</pre><p>Second method — choose_last_row_modify_by_ids</p><pre>def choose_last_row_modify_by_ids(df: DataFrame, list_columns_primary_key: list, list_columns_order: list) -&gt; DataFrame:<br>    if (df is None or list_columns_primary_key is None or list_columns_order is None) or (len(list_columns_primary_key) == 0 or len(list_columns_order) == 0):<br>        raise ValueError(&#39;The values could by not null or empty&#39;)<br>    else:<br>        partition_order = Window.partitionBy(*list_columns_primary_key).orderBy(*[F.col(lc).desc() for lc in list_columns_order])<br>        df_filter = df.withColumn(&#39;rn&#39;, F.row_number().over(partition_order)).filter(&#39;rn = 1&#39;).drop(&#39;rn&#39;)<br>        return df_filter</pre><p>This method filter the last modify row by primary keys. It’s necessary for change data capture CDC.</p><p>Clone the code from Github and apply the tests to see it working.import de_pyspark_tips.df_transformations as ET</p><pre>import de_pyspark_tips.df_transformations as ET<br><br>from pyspark.sql import SparkSession <br><br>data_df1 = [<br>    (1,&quot;2020-08-28&quot;, &quot;true&quot;, &quot;G1 | ESCADA 1O&quot;,&quot;d&quot;, &quot;3.15868&quot;, &quot;&quot;,&quot;I&quot;,&quot;2020-08-27 23:59:59&quot;),<br>    (1,&quot;2020-08-28&quot;, &quot;true&quot;, &quot;G1 | ESCADA 1O&quot;,&quot;d&quot;, &quot;3.15868&quot;, &quot;&quot;,&quot;U&quot; ,&quot;2020-08-28 12:59:59&quot;),<br>    (2,&quot;2018-05-25&quot;, &quot;true&quot;,&quot;G1 | ESCADA 1O&quot;,&quot;A&quot;, &quot;5.5&quot;, &quot;NO EXIST&quot;, &quot;I&quot;,&quot;2020-08-27 23:59:59&quot;),<br>    (3,&quot;2018-05-25&quot;, &quot;true&quot;,&quot;&quot;&quot;G1 | ESCADA 1O P/    <br>    NEW |<br>    STUFF<br>    &quot;&quot;&quot;,&quot;d&quot;, &quot;389.390&quot;,&quot;&quot;, &quot;I&quot;,&quot;2020-08-27 23:59:59&quot;)<br>    ]<br>columns_df1 =[&quot;id&quot;,&quot;data&quot;, &quot;ativo&quot;, &quot;unidade&quot;, &quot;sigla&quot;, &quot;valor&quot;, &quot;obs&quot;, &quot;action&quot;, &quot;data_transaction&quot;]<br><br>if __name__ == &#39;__main__&#39;:<br>    spark=SparkSession.builder.appName(&quot;Pyspark_Tips&quot;).getOrCreate()<br>    df=spark.createDataFrame(data_df1,columns_df1)<br><br>    list_schema = [<br>        {&#39;column_name&#39;: &#39;id&#39;, &#39;data_type&#39;: &#39;INTEGER&#39;},<br>        {&#39;column_name&#39;: &#39;data&#39;, &#39;data_type&#39;: &#39;DATE&#39;},<br>        {&#39;column_name&#39;: &#39;ativo&#39;, &#39;data_type&#39;: &#39;BOOLEAN&#39;},<br>        {&#39;column_name&#39;: &#39;unidade&#39;, &#39;data_type&#39;: &#39;VARCHAR(200)&#39;},<br>        {&#39;column_name&#39;: &#39;sigla&#39;, &#39;data_type&#39;: &#39;VARCHAR(2)&#39;},<br>        {&#39;column_name&#39;: &#39;valor&#39;, &#39;data_type&#39;: &#39;NUMERIC(10,2)&#39;},<br>        {&#39;column_name&#39;: &#39;obs&#39;, &#39;data_type&#39;: &#39;VARCHAR(MAX)&#39;},<br>        {&#39;column_name&#39;: &#39;action&#39;, &#39;data_type&#39;: &#39;VARCHAR(1)&#39;},<br>        {&#39;column_name&#39;: &#39;data_transaction&#39;, &#39;data_type&#39;: &#39;TIMESTAMP&#39;},<br>    ]<br><br>    newDF = ET.cast_columns_types_by_schema(df, list_schema, True)<br><br>    newDF.printSchema()<br>    newDF.show(truncate=False)<br><br>    filterDF = ET.choose_last_row_modify_by_ids(newDF, [&#39;id&#39;], [&#39;data_transaction&#39;])<br><br>    filterDF.printSchema()<br>    filterDF.show(truncate=False)</pre><p>I hope this can help, any doubt I’m available.</p><p>Bye.<br>herculanocm@outlook.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=484dbb3bb9b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pyspark/pyspark-tips-caste-and-filter-dataframe-by-last-modification-484dbb3bb9b5">Pyspark tips — Caste and filter dataframe by last modification</a> was originally published in <a href="https://medium.com/pyspark">Pyspark</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>