<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ian Eaves on Medium]]></title>
        <description><![CDATA[Stories by Ian Eaves on Medium]]></description>
        <link>https://medium.com/@ianeaves?source=rss-c0e211fe3fc3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*-n157r5p8-JOY3clUqhC2Q.jpeg</url>
            <title>Stories by Ian Eaves on Medium</title>
            <link>https://medium.com/@ianeaves?source=rss-c0e211fe3fc3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 02:09:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ianeaves/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Customizing Pandas-Profiling Summaries]]></title>
            <link>https://medium.com/data-science/customizing-pandas-profiling-summaries-b16714d0dac9?source=rss-c0e211fe3fc3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b16714d0dac9</guid>
            <category><![CDATA[eda]]></category>
            <category><![CDATA[dylan-profiler]]></category>
            <category><![CDATA[tips-and-tricks]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas-profiling]]></category>
            <dc:creator><![CDATA[Ian Eaves]]></dc:creator>
            <pubDate>Fri, 05 Mar 2021 06:22:30 GMT</pubDate>
            <atom:updated>2021-03-18T00:26:26.513Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/tips-and-tricks"><em>Tips and Tricks</em></a></h4><h3>Bringing Customization to Pandas Profiling</h3><h4>Exploiting the visions typesystem for fun and profit</h4><p>If you’ve previously used <a href="https://github.com/pandas-profiling/pandas-profiling">pandas-profiling</a>, you might have observed that column summaries are unique to the data types of each feature in your data. However, until recently it wasn’t possible to customize those summaries, so, if you wanted to automatically compute the average surface area of a sequence of shapely geometries, or the set of domain names in a sequence of email addresses, you were out of luck — until now.</p><p>The recently completed migration of pandas-profiling to the <a href="https://github.com/dylan-profiler/visions">visions</a> type system brings fully customizable type detection logic, summary algorithms, and is the first step towards end-to-end report customization including customized renderings. Over the rest of blog post I’m going to show you how to get started with visions and easily customize your data summaries using pandas-profiling.</p><h3>A brief detour into typesets</h3><p>The default pandas-profiling type system consists of four basic data types:</p><ul><li>Categories</li><li>DateTimes</li><li>Boolean</li><li>Numeric</li></ul><p>Technically, there is a fifth catch-all type called Unsupported for anything that doesn’t fit in the other four (these are usually complex objects like lists). If you want to inspect it’s implementation check out the ProfilingTypeSet object found in pandas_profiling.model.typeset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/318d9a8a43e5c7136440d04bfd59a13e/href">https://medium.com/media/318d9a8a43e5c7136440d04bfd59a13e/href</a></iframe><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*Ca5JIV121PXJal8fJmfLmw.png" /><figcaption>Image by Author</figcaption></figure><p>NOTE: You’ll need to install pydot and pygraphviz to reproduce these plots</p><p>These typesets can be as complicated or simple as your usecase calls for. Want to distinguish between Integers and Floats? URLs from Strings? Images from Files? Something else specific to your data? You can find most of these types now available in pandas-profiling with many more available in <a href="https://dylan-profiler.github.io/visions/visions/api/types.html">visions</a>, and if not available, writing your own is usually just a few lines of code. A full walk-through of types and type systems can be found <a href="https://www.ianeaves.com/post/titanic-visions/">here</a>.</p><h3>Customizing Summaries</h3><p>To get started let’s download a copy of the Titanic dataset and convert the Survived field over to a boolean.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e4045288d10e838223bcc1d6691da0e/href">https://medium.com/media/6e4045288d10e838223bcc1d6691da0e/href</a></iframe><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*mE6lJtNOcA5lTiiXX9VBSg.png" /></figure><p>Now that we have some data we can use pandas-profiling to create a profile report.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1f005e3c4521f5cfa3e63f4bff884e4/href">https://medium.com/media/c1f005e3c4521f5cfa3e63f4bff884e4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNZ190jakaDw5lwLn8ytCw.png" /><figcaption>Image by Author</figcaption></figure><p>No surprises here, just the standard pandas-profiling workflow. In addition to the normal html report we can directly inspect the summarization values computed by pandas-profiling through the description_set attribute on the report. Let&#39;s check out our summary of the Survived attribute.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3546f22985e500175f5e076db7f24c9e/href">https://medium.com/media/3546f22985e500175f5e076db7f24c9e/href</a></iframe><pre>{&#39;n_distinct&#39;: 2,<br> &#39;p_distinct&#39;: 0.002244668911335578,<br> &#39;is_unique&#39;: False,<br> &#39;n_unique&#39;: 0,<br> &#39;p_unique&#39;: 0.0,<br> &#39;type&#39;: Boolean,<br> &#39;hashable&#39;: True,<br> &#39;value_counts_without_nan&#39;: False    549<br> True     342<br> Name: Survived, dtype: int64,<br> &#39;n_missing&#39;: 0,<br> &#39;n&#39;: 891,<br> &#39;p_missing&#39;: 0.0,<br> &#39;count&#39;: 891,<br> &#39;memory_size&#39;: 1019}</pre><p>There’s a lot of useful information here: memory usage, whether series values are hashable, missing value counts, etc… By default, every Boolean sequence receives an identical summary — and now, thanks to summarizers, we can customize these results!</p><h3>Summarizers</h3><p>Under the hood, pandas-profiling uses a PandasProfilingSummarizer object to generate summaries for the profile report. These summarizers map data types, like Boolean, to a list of summary functions used to create the final summary.</p><p>You can reproduce the summary above directly from this summarizer like so:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5a19c27036ebe8b6a7d0adf0a6ea398/href">https://medium.com/media/d5a19c27036ebe8b6a7d0adf0a6ea398/href</a></iframe><pre>{&#39;n_distinct&#39;: 2,<br> &#39;p_distinct&#39;: 0.002244668911335578,<br> &#39;is_unique&#39;: False,<br> &#39;n_unique&#39;: 0,<br> &#39;p_unique&#39;: 0.0,<br> &#39;type&#39;: Boolean,<br> &#39;hashable&#39;: True,<br> &#39;value_counts_without_nan&#39;: False    549<br> True     342<br> Name: Survived, dtype: int64,<br> &#39;n_missing&#39;: 0,<br> &#39;n&#39;: 891,<br> &#39;p_missing&#39;: 0.0,<br> &#39;count&#39;: 891,<br> &#39;memory_size&#39;: 1019}</pre><p>To make a customized summary we need to specify a sequence of function(s) to call for each data type in our typeset. Types support arithmetic in much the you would naturally expect such that adding two types together, like Boolean + Unsupported, yields a typeset containing Boolean and Unsupported.</p><p>Combining these together, let’s write a summarizer that computes length of each series for all data types and additionally, the probability of being True for boolean.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e30e2082c3f1d37d5103f70b4dd404c6/href">https://medium.com/media/e30e2082c3f1d37d5103f70b4dd404c6/href</a></iframe><p>Although this may look complicated, at heart it’s just keeping track of a bunch of functions to call on series of different data types. Behind the scenes, the summarizer object took care of composing the summary functions together to produce a final result tacking on a “type” value to indicate which summarization was used.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7110cbd5347362a46fd6a374e5cc85f1/href">https://medium.com/media/7110cbd5347362a46fd6a374e5cc85f1/href</a></iframe><pre>{&#39;type&#39;: Boolean, &#39;length&#39;: 891, &#39;probability_true&#39;: 0.3838383838383838}</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef8c085fd0352f2a188c0d96a802b564/href">https://medium.com/media/ef8c085fd0352f2a188c0d96a802b564/href</a></iframe><pre>{&#39;type&#39;: Unsupported, &#39;length&#39;: 891}</pre><p>Although summarizers are now fully customizable, the associated html renderings haven’t been updated to expose customized statistics so be on the lookout for additional updates in the future.</p><h3>Customizing ProfileReport Summaries</h3><p>Although you can now customize summaries in pandas-profiling pretty much to your hearts content, in most cases you’ll be modifying or adding to the default summarizer rather than building one entirely from scratch. This is also pretty easy, just append any new functionality you want onto it’s associated type! Let’s update the default PandasProfilingSummarizier to include the new_boolean_summary function and run a report with the new functionality</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e96670395cbc8000b0784526851882a5/href">https://medium.com/media/e96670395cbc8000b0784526851882a5/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3546f22985e500175f5e076db7f24c9e/href">https://medium.com/media/3546f22985e500175f5e076db7f24c9e/href</a></iframe><pre>{&#39;n_distinct&#39;: 2,<br> &#39;p_distinct&#39;: 0.002244668911335578,<br> &#39;is_unique&#39;: False,<br> &#39;n_unique&#39;: 0,<br> &#39;p_unique&#39;: 0.0,<br> &#39;type&#39;: Boolean,<br> &#39;hashable&#39;: True,<br> &#39;value_counts_without_nan&#39;: False    549<br> True     342<br> Name: Survived, dtype: int64,<br> &#39;n_missing&#39;: 0,<br> &#39;n&#39;: 891,<br> &#39;p_missing&#39;: 0.0,<br> &#39;count&#39;: 891,<br> &#39;memory_size&#39;: 1019,<br> &#39;probability_true&#39;: 0.3838383838383838}</pre><p>Voila, just like that we were able to insert the probability_true calculation into our profile report.</p><h3>Conclusion</h3><p>So far we’ve been quite happy with the flexibility afforded by visions to define advanced data types and support user defined functionality. We’re excited to begin rolling out more of these capabilities soon, including support for customized renderings, and are eager to gather feedback. How have you been using customized types and summaries or do you have interesting use-cases worth adding into pandas-profiling? If so, please comment below and share!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b16714d0dac9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/customizing-pandas-profiling-summaries-b16714d0dac9">Customizing Pandas-Profiling Summaries</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>