<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Muthu Kumar S on Medium]]></title>
        <description><![CDATA[Stories by Muthu Kumar S on Medium]]></description>
        <link>https://medium.com/@muthukumar_s?source=rss-a1f40d25c3ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Jnz21Zu9o8l_AeK6</url>
            <title>Stories by Muthu Kumar S on Medium</title>
            <link>https://medium.com/@muthukumar_s?source=rss-a1f40d25c3ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:35:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@muthukumar_s/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PySpark — Getting Dynamic Schema from String]]></title>
            <link>https://medium.com/@muthukumar_s/pyspark-getting-dynamic-schema-from-string-88ab91c1e656?source=rss-a1f40d25c3ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/88ab91c1e656</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[schema]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Muthu Kumar S]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 15:44:04 GMT</pubDate>
            <atom:updated>2022-06-13T15:44:04.505Z</atom:updated>
            <content:encoded><![CDATA[<h3>PySpark — Getting Dynamic Schema from String</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*OtoFAt2kSCagi8-QgFDCOw.png" /></figure><p><strong>Introduction:</strong></p><p>We all know, PySpark is a Spark module used to provide a similar kind of Processing like spark using DataFrame. Schema is used to represent the structure of the data frame. It can be explicitly provided during the data frame creation or Spark itself infer schema from the data. But in few cases certain columns are in nested json strings format. To get the schema of nested json strings we need to do few workaround.</p><p><strong>Example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27d5ad6cbd6c44eccb5be63b6725f400/href">https://medium.com/media/27d5ad6cbd6c44eccb5be63b6725f400/href</a></iframe><p>In the above example, we have the nested json string. It is hard to get the inner data point without parsing it. In the larger data set with nested json string manual parsing will be a pain, so ideally we need to convert the data into the struct field, After converting to the struct field, data retrieval will be direct.</p><blockquote>Note: In this code, first element in the nested structure will be used as sample to find the schema</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71f9d2731c8faa8c9024e12dfa6153e9/href">https://medium.com/media/71f9d2731c8faa8c9024e12dfa6153e9/href</a></iframe><p>get_schema will get the input data and find the schema using the get_field_type . For the nested data the above function will be called recursively and schema created accordingly. All Set, now we need to do is convert the string to json and pass it to the get_schema function. Use the schema and update the data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb1e63b4e49a80d9ed3ea57f2b06e093/href">https://medium.com/media/cb1e63b4e49a80d9ed3ea57f2b06e093/href</a></iframe><p>That’s all in this article. Happy coding !!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88ab91c1e656" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use git pre-commit hooks in python codebase]]></title>
            <link>https://medium.com/@muthukumar_s/how-to-use-git-pre-commit-hooks-in-python-codebase-bb1702462c4?source=rss-a1f40d25c3ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb1702462c4</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pre-commit]]></category>
            <category><![CDATA[git-hooks]]></category>
            <dc:creator><![CDATA[Muthu Kumar S]]></dc:creator>
            <pubDate>Sun, 03 Oct 2021 13:42:27 GMT</pubDate>
            <atom:updated>2021-10-03T15:18:28.193Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*qaMy-8My8TEErVW3M6K9zQ.png" /><figcaption>Python pre-commit framework</figcaption></figure><h3><strong>Introduction</strong></h3><p>Pre commit is a framework which is used to identify the simple code issues before commit the code. It will run automatically before every commit made. It will point out the list of code issues such as indentation, trailing whitespace, missing docstring etc. It only allows to commit the code after fixing all these issues.</p><h3><strong>Usage</strong></h3><p>We can add collection of packages which will automatically checks different code standards and code stability by using tools like lints, tests, coverages etc. This will ensure the code and help to avoid simple mistakes. Pre-commit will only verify the modified files during every commit. By using pre-commit scripts as a developer we eventually start to follow the standards of the programming language</p><p>Pre commit framework is available in multiple languages. In this article we are going to see how to use pre commit in python project</p><h3><strong>Installation</strong></h3><p>Before using the pre-commit hooks, we need to install the pre-commit package manager.</p><p>To install pre-commit using the pip, use the below command</p><pre><strong>pip install pre-commit</strong></pre><h3><strong>Configuration</strong></h3><p>Pre commit can be configured through a yaml file. For that we need to create a yaml file in the root directory of the project with the name <strong>.pre-commit-config.yaml</strong></p><p><strong>Sample configuration file:</strong></p><pre>repos:<br>  - repo: https://github.com/pre-commit/pre-commit-hooks<br>      rev: v2.3.0<br>      hooks:<br>        - id: check-yaml<br>        - id: end-of-file-fixer<br>        - id: trailing-whitespace<br>  - repo: https://github.com/psf/black<br>      rev: 19.3b0<br>      hooks:<br>        - id: black</pre><p>Then run the below command, it will install all the mentioned hooks/packages in the <strong>.git/hooks/pre-commit</strong> location</p><pre><strong>pre-commit install</strong></pre><p>As pre-commit only verifies on modified files, when we are adding the pre-commit to the existing project, It is good to run the below command.</p><pre><strong>pre-commit run --all-files</strong></pre><h3><strong>Possible options available in pre-commit-config.yaml file</strong></h3><p><strong>repos: </strong>option for listing the different repos</p><p><strong>default_language_version: </strong>option for providing the language default versions. It will only override the packages which is not set the default version</p><p><strong>default_stages: </strong>option to provide the stages where it need to check</p><p><strong>files: </strong>option for global file include pattern</p><p><strong>exclude: </strong>option for global file exclude pattern</p><p><strong>fail_fast: </strong>option for stop running the hooks if first one is failed or not</p><p><strong>minimum_pre_commit_version: </strong>option to setting require a minimum version of pre-commit</p><h3>Some of most common repos used for python</h3><p><a href="https://github.com/pre-commit/pre-commit-hooks"><strong>pre-commit-hooks</strong></a><strong><br></strong>pre-commit-hooks provide many useful hooks which allows to reduce small mistakes made during coding. Some of them are check-json , end-of-file-fixer, requirements-txt-fixer, trailing-whitespace.</p><pre>- repo: https://github.com/pre-commit/pre-commit-hooks<br>    rev: v4.0.1<br>    hooks:<br>      - id: check-added-large-files<br>        args: [ &quot;--maxkb=700&quot; ]<br>      - id: check-json<br>      - id: end-of-file-fixer<br>      - id: requirements-txt-fixer<br>      - id: trailing-whitespace<br>      - id: check-yaml<br>        args:<br>          - --allow-multiple-documents<br>      - id: pretty-format-json<br>        args:<br>          - --autofix<br>        exclude: ^tests/.*</pre><p><a href="https://github.com/PyCQA/bandit"><strong>bandit</strong></a><strong><br></strong>Bandit is used to find the common security issues and vulnerabilities in the code.</p><pre>- repo: https://github.com/pycqa/bandit<br>    rev: 1.7.0<br>    hooks:<br>      - id: bandit<br>        args: [ --exclude, tests ]</pre><p><a href="https://github.com/PyCQA/flake8"><strong>flake8</strong></a></p><p>Flake8 is used to check code base against coding style (PEP8), programming errors like library imported but unused and Undefined name and to check cyclomatic complexity.</p><pre>- repo: https://gitlab.com/pycqa/flake8<br>    rev: 3.9.2<br>    hooks:<br>      - id: flake8</pre><p><a href="https://github.com/PyCQA/pydocstyle"><strong>pydocstyle</strong></a><strong><br></strong>pydocstyle is a static analysis tool for checking compliance with Python docstring conventions. It verifies the docstring of methods, class and files and complains if any missing.</p><pre>- repo: https://github.com/pycqa/pydocstyle<br>    rev: 6.1.1<br>    hooks:<br>      - id: pydocstyle</pre><p><a href="https://github.com/pycqa/pylint"><strong>pylint</strong></a><strong><br></strong>Pylint is a Python static code analysis tool which looks for programming errors, helps enforcing a coding standard, sniffs for code smells and offers simple refactoring suggestions.</p><pre>- repo: https://github.com/pycqa/pylint<br>    rev: v2.8.3<br>    hooks:<br>      - id: pylint<br><br>        types: [ python ]</pre><p><strong>Example .pre-commit-config.yaml:</strong></p><pre>default_language_version:<br>  python: python3.7<br>default_stages: [commit, push]<br>files: &#39;&#39;<br>exclude: &#39;^.txt$&#39;<br>fail_fast: false<br>minimum_pre_commit_version: &#39;0&#39;<br>repos:<br>  - repo: https://github.com/pre-commit/pre-commit-hooks<br>    rev: v2.3.0<br>    hooks:<br>      - id: check-yaml<br>      - id: end-of-file-fixer<br>      - id: trailing-whitespace<br>  - repo: https://github.com/psf/black<br>    rev: 19.3b0<br>    hooks:<br>      - id: black<br>  - repo: https://github.com/asottile/pyupgrade<br>    rev: v2.19.4<br>    hooks:<br>      - id: pyupgrade<br>        args:<br>          - --py3-plus<br>          - --py36-plus<br>  - repo: https://github.com/pre-commit/pre-commit-hooks<br>    rev: v4.0.1<br>    hooks:<br>      - id: check-added-large-files<br>        args: [ &quot;--maxkb=500&quot; ]<br>      - id: check-ast<br>      - id: check-builtin-literals<br>      - id: check-case-conflict<br>      - id: check-docstring-first<br>      - id: check-merge-conflict<br>      - id: check-symlinks<br>      - id: check-toml<br>      - id: debug-statements<br>      - id: end-of-file-fixer<br>      - id: mixed-line-ending<br>      - id: debug-statements<br>      - id: requirements-txt-fixer<br>      - id: trailing-whitespace<br>      - id: check-yaml<br>        args:<br>          - --allow-multiple-documents<br>  - repo: https://github.com/pycqa/bandit<br>    rev: 1.7.0<br>    hooks:<br>      - id: bandit<br>        args: [ --exclude, tests ]<br>  - repo: https://github.com/psf/black<br>    rev: 21.6b0<br>    hooks:<br>      - id: black<br>  - repo: https://gitlab.com/pycqa/flake8<br>    rev: 3.9.2<br>    hooks:<br>      - id: flake8<br>  - repo: https://github.com/pycqa/pydocstyle<br>    rev: 6.1.1<br>    hooks:<br>      - id: pydocstyle<br>  - repo: https://github.com/pycqa/pylint<br>      rev: v2.8.3<br>      hooks:<br>        - id: pylint<br>          args:<br>            [<br>                &quot;--max-line-length=120&quot;,<br>                &quot;-d duplicate-code&quot;,<br>                &quot;--disable=&quot;unnecessary-pass,broad-except&quot;,<br>                &quot;--ignore=old&quot;<br>            ]<br>          types: [ python ]<br>  - repo: https://github.com/asottile/reorder_python_imports<br>    rev: v2.5.0<br>    hooks:<br>      - id: reorder-python-imports<br>        name: Reorder Python imports</pre><p>Thanks for Reading. Happy Coding !</p><p><strong>References:</strong></p><ul><li><a href="https://pre-commit.com/">https://pre-commit.com/</a></li><li><a href="https://github.com/pre-commit/pre-commit-hooks">https://github.com/pre-commit/pre-commit-hooks</a></li><li><a href="https://github.com/PyCQA">https://github.com/PyCQA</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb1702462c4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>