<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Govind Kanshi on Medium]]></title>
        <description><![CDATA[Stories by Govind Kanshi on Medium]]></description>
        <link>https://medium.com/@govindk?source=rss-7a485fd22c01------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ACq2Svtb_f-UiNfg.jpg</url>
            <title>Stories by Govind Kanshi on Medium</title>
            <link>https://medium.com/@govindk?source=rss-7a485fd22c01------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:35:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@govindk/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Troubleshooting 503 or service unavailable error in CosmosDB]]></title>
            <link>https://govindk.medium.com/troubleshooting-503-or-service-unavailable-error-in-cosmosdb-a10a346d0330?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/a10a346d0330</guid>
            <category><![CDATA[cosmosdb]]></category>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 01:50:35 GMT</pubDate>
            <atom:updated>2020-12-03T01:50:35.519Z</atom:updated>
            <content:encoded><![CDATA[<p>Troubleshooting 503 or service unavailable error in CosmosDB</p><p>CosmosDB status codes are published <a href="https://docs.microsoft.com/en-us/rest/api/cosmos-db/http-status-codes-for-cosmosdb">here</a> for reference. 503 server side errors can be detected on the CosmosDB backend for SQL API with help of Azure <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/monitor-server-side-latency">Monitor </a>(you need to split by status code the Total requests) and Azure <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/cosmosdb-monitor-resource-logs">Diagnostics</a>(need to look for status code in DataPlaneRequests for SQL API). You can create Azure Monitor alerts on their occurrence. If you receive multiple serverside 503 errors then definitely you should reach out to support.</p><p>SQL API’s client SDK too can throw the 503 errors in case it is unable to make connection with backend service. You should get full stack trace, status code, substatus code of this exception to debug <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-request-timeout-java-sdk-v4-sql#troubleshooting-steps">further</a>. It could be due to</p><ul><li>Transient <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-service-unavailable-java-sdk-v4-sql">Error</a></li><li>Lack of resource like CPU/memory/nw on the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-java-sdk-v4-sql#common-issues-workarounds">client side</a></li><li><a href="https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-request-timeout-java-sdk-v4-sql#troubleshooting-steps">Timeouts</a></li></ul><p>Resolution here is to retry the action again. On client side if you get too many of these errors —check the connectivity between the client and backend..</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a10a346d0330" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Story]]></title>
            <link>https://govindk.medium.com/story-2353b635a712?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/2353b635a712</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Fri, 24 Aug 2018 06:39:38 GMT</pubDate>
            <atom:updated>2018-08-24T07:34:26.685Z</atom:updated>
            <content:encoded><![CDATA[<p>How do you know you are valued? You know you work hard hoping people see the work you put without you showing yourself better than others or putting down others. And you don’t expect handouts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2353b635a712" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What: kind of json is expected in Capacity Estimation tool of Cosmos DB]]></title>
            <link>https://govindk.medium.com/what-kind-of-json-is-expected-in-capacity-estimation-tool-of-cosmos-db-1bfd8c9be725?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bfd8c9be725</guid>
            <category><![CDATA[cosmosdb]]></category>
            <category><![CDATA[capacity-estimation]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Wed, 20 Sep 2017 22:26:34 GMT</pubDate>
            <atom:updated>2017-09-20T22:26:34.436Z</atom:updated>
            <content:encoded><![CDATA[<p>With <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/introduction">Cosmos DB</a> intent of the designers is to help developers and DBA focus on domain thus bringing the <a href="https://azure.microsoft.com/en-us/support/legal/sla/cosmos-db/v1_0/">SLAs </a>for low latency, throughput, consistency and availability all with underlying powerful <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/distribute-data-globally">global distribution</a> available on tap.</p><p>The performance, throughput SLAs require defining required throughput. This throughput can be estimated using <a href="https://www.documentdb.com/capacityplanner#">capacity estimation tool</a>.</p><p>This tool requires a json document which represents one unit of entity that will be stored or retrieved. So an array like below will not work —</p><pre>[<br>{ “_id”:{  “$oid”:”5968dd23fc13ae04d9000001&quot;},<br>“product_name”:” citrate”,<br>“supplier”:”TEN”,“quantity”:261,<br>“unit_cost”:”$10.47&quot;},<br>{“_id”:{“$oid”:”5968dd23fc13ae04d9000002&quot;},<br>“product_name”:”  ashei”,<br>“supplier”:”XYZ”,“quantity”:292,<br>“unit_cost”:”$8.74&quot;},<br>{“_id”:{“$oid”:”5968dd23fc13ae04d9000003&quot;},<br>“product_name”:” HBr”,<br>“supplier”:”ABC”,<br>“quantity”:211,“unit_cost”:”$20.53&quot;}<br>]<br></pre><p>If you plan to store array like this — you need to convert this into full fledged json of that entity.</p><pre>{ &quot;o&quot; : <br>[<br>{ “_id”:{  “$oid”:”5968dd23fc13ae04d9000001&quot;},<br>“product_name”:” citrate”,<br>“supplier”:”TEN”,“quantity”:261,<br>“unit_cost”:”$10.47&quot;},<br>{“_id”:{“$oid”:”5968dd23fc13ae04d9000002&quot;},<br>“product_name”:”  ashei”,<br>“supplier”:”XYZ”,“quantity”:292,<br>“unit_cost”:”$8.74&quot;},<br>{“_id”:{“$oid”:”5968dd23fc13ae04d9000003&quot;},<br>“product_name”:” HBr”,<br>“supplier”:”ABC”,<br>“quantity”:211,“unit_cost”:”$20.53&quot;}<br>]<br>}</pre><p>Or otherwise if intent was to store individual document — for each product — just use that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bfd8c9be725" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo- Leverage Azure CosmosDB metrics to find issues ?]]></title>
            <link>https://govindk.medium.com/howto-leverage-azure-cosmosdb-metrics-to-find-issues-and-fix-them-d66d13603bb4?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/d66d13603bb4</guid>
            <category><![CDATA[azure-cosmos-db]]></category>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Mon, 04 Sep 2017 21:35:10 GMT</pubDate>
            <atom:updated>2019-11-23T16:37:13.542Z</atom:updated>
            <content:encoded><![CDATA[<h3>HowTo- Leverage Azure CosmosDB metrics via Azure Monitor &amp; analytic log to find issues ?</h3><ul><li>This article will help you understand Cosmos DB metrics — specially Storage and Throughput on the Cosmos DB portal.</li><li>We will find out how to tackle errors like “ Storage quota for ‘Document’ exceeded” or “Request rate too large”.</li><li>Reference for <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/partition-data">Partitioning</a></li><li>Reference for Throughput is <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">here </a>and <a href="https://medium.com/@th0maswe1ss/how-i-learned-to-stop-worrying-and-love-cosmos-dbs-request-units-92c68c62c938">here</a>(Thanks <a href="https://medium.com/u/a22e82772bd9">Thomas Weiss</a>).</li><li>Reference for Indexing is <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/indexing-policies">here</a>.</li><li>Reference for Financially backed <a href="https://azure.microsoft.com/en-us/support/legal/sla/cosmos-db/v1_0/">SLA </a>for throughput, latency, availability , consistency is <a href="https://azure.microsoft.com/en-us/support/legal/sla/cosmos-db/v1_0/">here</a>.</li><li>Big picture of Cosmos DB platform is <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/introduction">here</a>.</li></ul><p>Most of us can’t imagine life ending soon, <a href="https://medium.com/u/1e5b2bca5b5e">Scott Hanselman</a>’s <a href="http://keysleft.com/">keystrokes</a> left site makes end clearer in terms we understand — keystrokes left — what you want to do with rest of them. Getting more time to focus on important things is big help any day.</p><p>In IT industry Database folks spend lot of their time managing hardware, software details and learning every new trick to ensure end users have good SLAs for performance, throughput and availability. This requires them to learn and leverage expertise across clustering software, monitoring/management software and then all the gory implementation details of a particular data store. Never be surprised if DBA/developer tells you exactly how disk needs to be formatted, blocks laid out and then what jvm or memory/disk flush settings need to be done to get “started”. In majority of the cases database folks know lot more about how things work — indexes, statistics, execution plans, buffers, caches, disk issues. The other chunk of keystroke gobbler is capacity management/forecast of a database workloads.</p><p>With <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/introduction">Cosmos DB</a> intent of the designers is to help developers and DBA focus on domain thus bringing the <a href="https://azure.microsoft.com/en-us/support/legal/sla/cosmos-db/v1_0/">SLAs </a>for low latency, throughput, consistency and availability all with underlying powerful <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/distribute-data-globally">global distribution</a> available on tap.</p><p>It also makes it easier for developer to come in from direction they are comfortable — treat data store as <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-introduction">document store</a>(with option of <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/mongodb-introduction">MongoDB API</a>), <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/table-introduction">key value store</a> or a <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/graph-introduction">graph </a>with APIs which make sense. To make evolution of the data model easier system does not impose any specific schema. To take away the index management issues related to performance management— it is designed to be <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/indexing-policies">auto-indexing system</a> with low latency guarantee of durable commits with consistent index.</p><p>To ensure application provides low latency access and availability — system is designed to be <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/distribute-data-globally">globally available</a> at click of button or script. Client SDK is intelligent enough to be <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/regional-failover">auto-homing</a> to find “right” location every time in event of challenges. To provide easy capacity management — it helps you define the T<a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">hroughput </a>you need. Then it guarantees T<a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">hroughput </a>while providing unlimited elasticity for changing as per the needs of the workload.</p><p>All these features add up lot of <a href="http://keysleft.com/">clicks </a>end of the day.</p><p><strong>Metrics</strong></p><p>Cosmos DB provides metrics which cover SLA on the server side on the portal. These metrics can be pushed to Cosmos DB’s portal, Azure Monitor and via Diagnostic log to Storage/EventHub or Azure Monitor’s Log.</p><p><strong>Application Insights and Log Analytics</strong></p><p>Information about usage of throughput or query execution statistics is shared back to the client. You can take information about the Request Charge which is provided for every operation and push it to <a href="https://docs.microsoft.com/en-us/azure/application-insights/app-insights-analytics">Application Insights </a>and get instant alerts for extra RU or time spent at backend. Or you can also enable diagnostics push to log analytics which can collect this data for your analysis. Both systems provide good way to query the data to pinpoint issues. Some of the information you can stuff in from client side into AppInsight include full query (which is not available in log analytics today) or AD identity of execution principal. <a href="https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-log-search">Log Analytics</a> can <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/logging#interpret">log all interaction </a>irrespective of the origin which hit the database. This means log analytics will also have interaction started from portal for example. Azure Monitor for example can provide metrics for total requests which make up the operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yyplll9V0Z9LfLhuPn3TsA.png" /><figcaption>SQL API Monitoring</figcaption></figure><p><strong>Background</strong></p><p>In general with Azure Cosmos DB two things matter — distribution of data and Throughput.</p><p>Throughput is unlimited and elastic to scale up or down to meet the needs. Every operation against a container results in usage a portion of this throughput. This information is generally returned back as Request Charge as explained in the <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">documentation</a>.</p><p>Distribution of data depends on the chosen partition key and thus is important piece to ensure data can scale out while serving the requisite queries or ingesting the data.</p><p>Data storage and Throughput is practically unlimited. If you need more than what is available on the portal — always connect to us via support or askcosmosdb@microsoft.com.</p><p><strong>Portal metrics</strong></p><p>Today we will take a look at how to interpret these metrics shared via browser. They are collected every 5 minutes and have 7 days of history.</p><p><strong><em>Storage</em></strong></p><p>Storage metrics pane provides you 3 pieces of information — Number of physical partitions in a container, Total size occupied by data and index in a container, Total number of “entities” in a container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CKX681CancKf9nVdWFSTtQ.jpeg" /><figcaption>High level — view of storage pane metrics</figcaption></figure><p>1st — notice number of physical partitions and their size.</p><p>Cosmos DB provides you storage you require and increases it when your application needs it. Thus freeing you from the chore of capacity management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*WJi4sETv7j7EUQwosPzstQ.png" /><figcaption>Total number of physical partitions and their individual size</figcaption></figure><p>2nd — How much total space your container occupies including both data and index. Index generally is very small percentage of the actual data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*C_Ys7I27LnwB5_o4z4aJHA.png" /></figure><p>Index space can be controlled by looking at the data and queries — if you do not query certain data — you can remove that attribute from <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/indexing-policies">indexing</a>. Thus saving index space. By default all data/attributes are indexed by Cosmos DB today with performance guarantee. Index alteration is online process, your application continues operation without interruption. Thus change of schema — because of evolution of application needs — is a non-event for this indexing mechanism.</p><p>Below is another way to get similar information from the code.</p><pre>// Measure the document size usage (which includes the index size)  </pre><pre>ResourceResponse&lt;DocumentCollection&gt; collectionInfo = await client.ReadDocumentCollectionAsync(UriFactory.CreateDocumentCollectionUri(“db”, “coll”)); <br> Console.WriteLine(“Document size quota: {0}, usage: {1}”, collectionInfo.DocumentQuota, collectionInfo.DocumentUsage);</pre><p>3rd — How much data each physical partition has ? Which top 10 keys dominate the physical partition ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/859/1*TlaD0pLnWHnTmFq2YU9dXQ.png" /><figcaption>Amount of storage per physical partition and dominant keys in given partition</figcaption></figure><p>1st thing to notice is almost good distribution ranging from 4.4 GiB to 6.5 GiB. When you click on physical partition you will notice top partition keys.</p><p>At times you will see no dominant keys in a partition — do not worry — that just shows a good cardinality of the partition key. In that case no one key dominates the distribution. Or at times you will see only few keys and their size. That tells you these keys dominate in terms of logical size other keys in that physical partition.</p><p><strong>What would be a bad distribution ?</strong></p><p>Any container where distribution across all partitions does not look sort of equal — say +/- 10% variance is something that requires investigation. Very irregular or highly skewed data distribution points to a bad partition key. It is essential to look at the partition key and the kind of data which is coming in. It is always better to verify the loaded data data distribution follows your expectation.</p><p>Example — A customer who chose a date as partition key and only has data for few dates (may be he has just started) would have very skewed data and in worst case all queries/ingestion happening in that partition key.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*28BFd9gVZN6y-PDs6kpLxw.png" /><figcaption>Possible Bad distribution of data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*FtGsvkxTgKc7cKJ8dGSkwg.png" /><figcaption>Bad distribution of data</figcaption></figure><p><strong>Impact — </strong>You might see highly skewed throughput requests on one particular partition — you can try to co-relate this on Throughput metrics pane for Max consumed RU/second for each partition. If partition is hot in terms of all data ending up in one place and queries also hitting it — you have potential issue to fix.</p><p>As you can see below consumed per partition RUPS for 1st partition is pretty high compared to provisioned throughput. This will result in throttling and show up as latency (if sdk defaults are not overriden).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*rBo_5yc1BdRW8LUxEDwc6w.png" /><figcaption>Skewed throughput load on particular partition</figcaption></figure><p>c. How many documents exist in each physical partition ?</p><p>This data is present in the last pane — you should see correlation between this and the second pane. Again here you will see the dominant keys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*ZDqW4Kee0cDu1J5mh65OFg.png" /><figcaption>Total number of entities per physical partition</figcaption></figure><p><strong>What to take away — </strong>This pane provides you individual entity count in physical partition and ability to look up dominant partition key. This is important as you could have few large entities vs lot of small entities in the container. In general the same partition key should turn up for the size and count.</p><p><strong>When do I get error </strong>Storage quota for ‘Document’ exceeded ?</p><p>You will get this error if your physical partition is full and this is possible when your logical partition key size entities exceeds the physical partition. This indicates issue with partition key. To understand this how it can happen — say you want to store voter information and you choose gender as partition key. You start inserting data and you can end in situation where either of the gender data (assuming every entity representing voter data is 1 KB) exceeds about 10 M items or 10 GB of data. This implies you need to look at your queries and redesign the partition key and re-load the data.</p><p>Reference — <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/partition-data">Partitioning</a></p><p><strong><em>Throughput</em></strong></p><p>Throughput has been described well <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">here </a>and <a href="https://medium.com/@th0maswe1ss/how-i-learned-to-stop-worrying-and-love-cosmos-dbs-request-units-92c68c62c938">here</a>(Thanks <a href="https://medium.com/u/a22e82772bd9">Thomas Weiss</a>).</p><p>Key part of monitoring throughput is to see if you are getting throttled and then see if consumption is more than provisioned and dig into the actual time to see if it was related to one partition or many. This is usually followed by looking at client side telemetry (today) of RU charges for operations to find out culprit operations/queries.</p><p>So what you get to see on the throughput pane ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BnOB6wV22zza-iftp2o1Lw.jpeg" /><figcaption>Throughput metrics in one view</figcaption></figure><p>1st look at top line which might have details like below. Important line is latter one which provides you information about what throughput each partition gets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*xdvQnF9aoFcZMZY47sOH8A.jpeg" /><figcaption>Equal Throughput distribution across all partitions</figcaption></figure><p>Then you should focus on number of requests per minute</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*2zxFU1OWTSPnPhtxaGEmYg.png" /><figcaption>number of requests per minute</figcaption></figure><p>Here you see various http status codes — you should look for light blue for total number of successful requests. Most of the http status codes are described <a href="https://docs.microsoft.com/en-us/rest/api/documentdb/http-status-codes-for-documentdb">here</a>.</p><p>2nd focus on number of throttled operations per minute</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*72EFJeHomL1irU6_dEKK2Q.png" /><figcaption>Number of throttled requests per minute</figcaption></figure><p>This tells you have a problem, which needs resolution. So you need to get an idea about whether max consumed RU/second actually shows issue. In this case very clearly provisioned throughput is very small — around 1.5 K and actual usage touches upto 15 K RU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*E3tulJ36bT9tv8WhpJ0fIA.png" /></figure><p>If you want more finer details of this metrics — you can click on double arrow to get more details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/36/1*rZ49rC2uSxgO2pRIVYO3ig.png" /><figcaption>get-more details</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*GjWBILg6moV8yBdDD-vlnw.png" /><figcaption>Details of Max RU/second</figcaption></figure><p>3rd Now we need to see whether something specific happened in one partition or multiple partitions at particular time. You can hover on the metric above to get the time information when we hit the peak of about 15k — looks like 3.05 PM. So let us enter that information of time in Max consumed RU/second by each physical partition and hit apply</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*xa9-LgH9zTUxu3qsVg7JXQ.png" /><figcaption>Single Partition seeing heavy usage at 3.05 PM</figcaption></figure><p>There is obvious problem here. One partition is consuming lot of RU and others are silent. Indicates issue with either workload, data skew or both. With stream analytics + eventhub performance testing we have seen the skew of data, skew of workload resulting in this kind of behavior a lot. Fix here is to emulate real workload and fix the data skew.</p><p>In general in throughput pane — you will see either you have skewed workload in terms of workload happening on one partition or all partitions under provisioned. In either case you can get throttled and the error “request rate too large”.</p><p><strong>When do I get error </strong>Request rate too large?</p><p>When a throttle occurs, the server will preemptively end the request with RequestRateTooLargeException (HTTP status code 429) and return the x-ms-retry-after-ms header indicating the amount of time, in milliseconds, that the user must wait before reattempting the request.</p><pre>HTTP Status 429<br>Status Line: RequestRateTooLarge<br>x-ms-retry-after-ms :100</pre><p>If you are using the SDK , then most of the time you never have to deal with this exception, as the current version of the Client SDK implicitly catches this response, respects the server-specified retry-after header, and retries the request. Unless your account is being accessed concurrently by multiple clients, the next retry will succeed.</p><p>If you have more than one client simultaneously operating above the request rate, the default retry behavior may not suffice, and the client will throw a DocumentClientException with status code 429 to the application. In cases such as this, you may consider handling retry behavior and logic in your application’s error handling routines or increasing the reserved throughput for the container.</p><p>Reference for <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/request-units">Throughput</a> planning, handling issues.</p><p><strong>Examples</strong></p><p>Example below shows more patterns and details of the issue. Sometimes it helps to look at the data from 7 days or 24 hour perspective rather than 1 hr which is default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tak8VvhgmMJWkdAbWdJ8mA.png" /></figure><p>This was a condition where you can see periodicity of “some repeated job” which results in heavy throttled requests on Aug 17. In this case provisioned throughput was reduced — which was the culprit here resulting in heavy throttling on particular day. Most of the times customer increases/decreased the throttling during processing. In this case they did not increase throughput when needed.</p><p>Yet another example is this one where customer needs to increase throughput as he does not have any set pattern.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*gMvMPOpcmop-xVCIKws4dA.png" /><figcaption>Another example of more throughput required as all partitions are used</figcaption></figure><p>In this case all partitions are using up throughput more than provisioned. As in this case we tried to see at 10.19 am what was happening. You can see slight skew — but it is not significant. In this case we need to see why we got the spike.</p><p>So help comes in form of query statistics and also examining request charges for all operations at client side.</p><p><strong>Client Side metrics</strong></p><p>Other thing you need to do is look at <strong>request charge</strong> of each operation and keep them in some form of telemetry (<a href="https://azure.microsoft.com/en-us/services/application-insights/">Azure AppInsight</a>s) to find out which operations contributed to that spike in usage. Sometimes it is difficult to get one operation as you might be doing multiple of those operations from different client machines — so again telemetry is important for you to build.</p><pre>// Measure the performance (request units) of writes. <br> ResourceResponse&lt;Document&gt; response = await client.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri(“db”, “coll”), myDocument); <br> Console.WriteLine(“Insert of document consumed {0} request units”, <strong>response.RequestCharge</strong>);</pre><pre>// Measure the performance (request units) of queries. <br> IDocumentQuery&lt;dynamic&gt; queryable = client.CreateDocumentQuery(UriFactory.CreateDocumentCollectionUri(“db”, “coll”), queryString).AsDocumentQuery();</pre><pre>double totalRequestCharge = 0;<br> while (queryable.HasMoreResults)<br> {<br> FeedResponse&lt;dynamic&gt; queryResponse = await queryable.ExecuteNextAsync&lt;dynamic&gt;(); <br> Console.WriteLine(“Query batch consumed {0} request units”,<strong>queryResponse.RequestCharge</strong>);<br> totalRequestCharge += <strong>queryResponse.RequestCharge</strong>;<br> }</pre><pre>Console.WriteLine(“Query consumed {0} request units in total”, totalRequestCharge);</pre><p>Example for looking at all operation’s Request Charge is <a href="https://github.com/Azure/azure-cosmosdb-dotnet/blob/master/samples/request-units/Program.cs">here</a>.</p><p>Request Charge for an operation gives you how many of those operations can be done per second. You already know Throughput per partition which is possible — thus you can theoretical limit for operations per second.</p><p>For example</p><p>Let us say you have single partition with 10000 RU and single query operation taking up 100 RUs. So how many of these queries can execute simultaneously ? 10000/100 = 100 operations.</p><p>So now you get an idea of doing better job of your throughput and operations. You can get initial estimate of CRUD operations with help of<a href="https://www.documentdb.com/capacityplanner"> capacity estimator tool</a>. Once you have good enough data — and your queries laid out — you can see what could be possible every second.</p><p>Total Throughput/per second = Read Operation + Update operation + delete Operation + Create Operation + Query operation RU Charge (all at per second).</p><p>Reference for <a href="https://www.documentdb.com/capacityplanner">capacity estimation</a></p><p><strong>Client Query statistics </strong>— Cosmos DB provides query execution <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/documentdb-sql-query-metrics">statistics</a>. (focused today on DocumentDB API to be extended to other API in form it makes sense to the API)</p><pre>IDocumentQuery&lt;dynamic&gt; query = client.CreateDocumentQuery(<br> UriFactory.CreateDocumentCollectionUri(DatabaseName, CollectionName), <br> “SELECT * FROM c WHERE c.city = ‘Seattle’”, <br> new FeedOptions <br> { <br><strong> PopulateQueryMetrics = true,</strong> <br> MaxItemCount = -1, <br> MaxDegreeOfParallelism = -1, <br> EnableCrossPartitionQuery = true <br> }).AsDocumentQuery();</pre><pre>FeedResponse&lt;dynamic&gt; result = await query.ExecuteNextAsync();</pre><pre>// Returns metrics by partition key range Id </pre><pre>IReadOnlyDictionary&lt;string, QueryMetrics&gt; metrics = <strong>result.QueryMetrics</strong>;<br></pre><p>You need to execute queries with feed option of query statistics, this tells where system is spending time. Whether we are using index, whether we are retrieving lot of documents vs reading them. So what can you do here — when you find issues — generally if scans seem to be happening — provide partition key or better filter condition.</p><p>So you know the Request charge and where you spend time — you can take requisite action to modify index or the query.</p><p><strong>Client Diagnostics </strong>are important when you need details or support/product team will require you to get this for debugging. This over and above <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.resourceresponsebase.activityid?view=azure-dotnet">ActitivityId</a>, Exception, <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.resourceresponsebase.requestcharge?view=azure-dotnet">RequestCharge</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.resourceresponsebase.requestlatency?view=azure-dotnet">RequestLatency</a>, <a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.resourceresponsebase.statuscode?view=azure-dotnet">StatusCode</a>.</p><p>Client Diagnostics (.<a href="https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.documents.client.resourceresponsebase.requestdiagnosticsstring?view=azure-dotnet#Microsoft_Azure_Documents_Client_ResourceResponseBase_RequestDiagnosticsString">net</a>) , (<a href="https://docs.microsoft.com/en-us/java/api/com.microsoft.azure.cosmosdb.resourceresponse.getrequestdiagnosticsstring?view=azure-java-stable#com_microsoft_azure_cosmosdb_ResourceResponse_getRequestDiagnosticsString__">java</a>) can be retrieved on demand cases for direct mode of SQL API(As of 2019 May).</p><p><strong>Azure Monitor Metrics</strong></p><p>Azure monitor metrics provide way to get distribution of your operations across all requests. This helps to quickly get an idea about what kind of requests are coming in and what you need to to possibly do — increase throuhgput or otherwise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*bWZE8__2B05W5Ehy8L1epg.png" /><figcaption>Operation Types for requests</figcaption></figure><p>You can leverage Azure monitor metrics to create alerts and dashboards (look at workbooks feature — to monitor multiple cosmos db collections/accounts).</p><p><strong>Azure Monitor Log Analytics</strong></p><p>Cosmos DB provides immutable record of all interaction (CRUD on data plane and control plane activities) at via log analytics. All diagnostic information can either be pushed to “Blob/EventHub or Log Analytics”. The latter is what you can learn from <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/logging#view-logs-in-log-analytics">here</a> .</p><p>Log analytics provides easy way of finding out</p><ul><li>volume of operations which are coming in at within given time</li><li>operations which take more RU or</li><li>operations which take more time</li></ul><p><em>So how can you leverage it ?</em></p><p>First enable the log analytics for data plane requests and activity. Then use queries which help you to get information you are after.</p><pre>AzureActivity | where Caller == “<a href="/service/mailto:shireest@microsoft.com">x@x.com</a>”<br>| summarize count() by Resource</pre><pre>// which operations are happening <br>AzureDiagnostics | where ResourceProvider==”MICROSOFT.DOCUMENTDB” and Category==”DataPlaneRequests” | summarize count() by OperationName</pre><pre>// which operation is taking more than 3 ms<br>AzureDiagnostics <br>| where toint(duration_s) &gt; 3000 and ResourceProvider==”MICROSOFT.DOCUMENTDB” and Category==”DataPlaneRequests”<br>| summarize count() by clientIpAddress_s, TimeGenerated</pre><pre>// when long running operation ran — be careful these can pull in lot of data //so put a boundary box around them in terms of time<br> AzureDiagnostics | where ResourceProvider==”MICROSOFT.DOCUMENTDB” and Category==”DataPlaneRequests”| project TimeGenerated , toint(duration_s)/2000 | render timechart</pre><pre>// Which operation more than x RUs<br>AzureDiagnostics <br>| where ResourceProvider==”MICROSOFT.DOCUMENTDB” and Category==”DataPlaneRequests” and TimeGenerated between (datetime(2018–05–07) .. datetime(2018–05–09)) <br>and toreal(requestCharge_s) &gt; 2.0 <br>| project Resource , collectionRid_s , duration_s , OperationName, ResourceType</pre><pre>AzureDiagnostics<br>| where ResourceProvider==”MICROSOFT.DOCUMENTDB” and Category==”DataPlaneRequests” and toreal(requestCharge_s) &gt; 5 <br> and OperationName == “Query”</pre><p>If it is write — you need to see if you want to not filter all items in attributes in query. You can decide to drop index on selective attributes.</p><p>To summarize</p><ol><li>Verify Storage pane has information that your data is well distributed across the partitions.</li><li>Verify throttles are not happening with help if Throughput pane.</li><li>If throttles are present — verify if throttles are across all the partitions or specific partition.</li><li>Use client side metrics of Request charge and query statistics to find out where you spend RU /time— always focus on dominant queries/operations to focus on 80% of workload. Either increase the RU if needed or modify the query.</li><li>Use log analytics to deeper pinpointed investigation of issues and focus on queries or other operations.</li></ol><p>In one of the next articles we will focus on looking at query execution statistics and finding alternatives.</p><p>If you ever need help of Cosmos DB — please reach out to us on askcosmosdb@microsoft.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d66d13603bb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo-Tag existing DocumentDB accounts]]></title>
            <link>https://govindk.medium.com/howto-tag-existing-documentdb-accounts-295b2b864fb4?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/295b2b864fb4</guid>
            <category><![CDATA[tag]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[documentdb]]></category>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Tue, 08 Nov 2016 14:49:46 GMT</pubDate>
            <atom:updated>2016-11-08T14:49:46.982Z</atom:updated>
            <content:encoded><![CDATA[<p>Why would you want to do this ? Because you want to tag your DocumentDB assets.</p><pre>Get-AzureRmResource -Verbose | Where-Object {$_.ResourceType -like “*DocumentDB*”}</pre><pre>*** replace with your resourcegroupname and name</pre><pre>$tags = (Get-AzureRMResource -ResourceGroupName &lt;rgname&gt; -Name &lt;resname&gt;).Tags</pre><pre>*** make up your tags</pre><pre>$tags += @{Name=”sku”;Value=”mixed”}</pre><pre>Set-AzureRmResource -ResourceGroupName &lt;rg-name&gt; -Name &lt;resname&gt; -ResourceType “Microsoft.DocumentDB/databaseAccounts” -Tag $tags</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=295b2b864fb4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo-AzureML-create n-gram features using Feature Hashing for text data]]></title>
            <link>https://govindk.medium.com/howto-azureml-create-n-gram-features-using-feature-hashing-for-text-data-5e8c64d5e591?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e8c64d5e591</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Mon, 11 Aug 2014 10:23:51 GMT</pubDate>
            <atom:updated>2014-08-13T01:30:03.816Z</atom:updated>
            <content:encoded><![CDATA[<p>AzureML has famous Vowpal-Wabbit’s Hashing trick embedded in it. It allows to use low cost, low impact hashing of the features give hashing bit-size and # of n-grams.</p><p>Steps</p><ol><li>Upload the Text file or read it. I just took bunch of text from a news site and loaded it as text file.</li><li>Use the Feature hashing module — specify hashing bit size and n-grams.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OUx9-0PknA0Y3D2PeAkGeA.png" /></figure><p>Output is the Features as columns. (visualized as below). Mostly this will be sparse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YudTVrgAl-G0biwq_86vUw.png" /></figure><p>What is n-gram — <a href="http://en.wikipedia.org/wiki/N-gram">N-grams</a> are contiguous sequences of n items from a given sequence. Given “I like Star trek movie” — 2-gram output would be — I like, like star, star trek … I could not find a way to “print” those n-gram columns “values” in AzureML yet. Vowpal wabbit has the “ -invert_hash” option to print them out.</p><p>Background —</p><p><a href="http://alex.smola.org/papers/2009/Weinbergeretal09.pdf">http://alex.smola.org/papers/2009/Weinbergeretal09.pdf</a></p><iframe src="https://docs.google.com/viewer?url=http://www.cse.wustl.edu/~kilian/papers/ceas2009-paper-11.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a1a0b6e7b3335b41db8c405dc3f8929/href">https://medium.com/media/8a1a0b6e7b3335b41db8c405dc3f8929/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e8c64d5e591" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo-AzureML-Reduce Dimensionality or find important features using LDA or Fisher correlation]]></title>
            <link>https://govindk.medium.com/howto-azureml-reduce-dimensionality-or-find-important-features-using-lda-or-fisher-correlation-887a56c6b387?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/887a56c6b387</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Fri, 08 Aug 2014 18:27:21 GMT</pubDate>
            <atom:updated>2014-08-08T18:27:21.528Z</atom:updated>
            <content:encoded><![CDATA[<p>In this case we will use Steel-annealing data to just showcase how one can find important features by using oob Modules like “Filter based feature selection” and “LDA”. The former allows to choose “max” features and does not care about the data types of other features in the dataset. The latter has issues with presence of string datatypes and missing values. The latter also does not ask for a number of features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RZ-hpBvV0xj1MszaBJHirA.png" /><figcaption>LDA and Fisher score based Feature selection</figcaption></figure><p>If you really are interested — you should for the heck of it pass the LDA bad data and see what method it is using.</p><p>Background reading — <a href="http://web.engr.illinois.edu/~qgu3/ecmlpkdd11.pdf">http://web.engr.illinois.edu/~qgu3/ecmlpkdd11.pdf</a></p><p>In case you have question wrt why LDA since PCA already does this — LDA vs PCA — <a href="http://courses.cs.tamu.edu/rgutier/cs790_w02/l6.pdf">http://courses.cs.tamu.edu/rgutier/cs790_w02/l6.pdf</a></p><p>Simpler version — <a href="http://ceit.aut.ac.ir/~shiry/lecture/machine-learning/LDA%20Tutorial.pdf">http://ceit.aut.ac.ir/~shiry/lecture/machine-learning/LDA%20Tutorial.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=887a56c6b387" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why-AzureML-Deployed WebService model requires the dataset in experiment]]></title>
            <link>https://govindk.medium.com/why-azureml-deployed-webservice-model-requires-the-dataset-in-experiment-299d14cab137?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/299d14cab137</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Wed, 06 Aug 2014 16:42:08 GMT</pubDate>
            <atom:updated>2014-08-06T16:43:18.956Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why-AzureML-Is data-defn required in webservice experiment</h3><p>When you deploy a webservice with trained model for use in request response or batch mode, you will notice the initial dataset is still required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*yZ-vdyDW_HYuY6_5avACvA.png" /><figcaption>Webservice deployment experiment with input/output published and trained model + scoring module.</figcaption></figure><p>This model requires data schema for input — in this case we got it from the initial dataset modified by R script which added headers to columns. What is dataschema — this is the shape of data (types/names of data points required for the input of webservice). This is invoked from the published webservice endpoints for the workspace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*bZNFWbIEtBoEZJ5QusXDyw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=299d14cab137" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo-AzureML-Restrict # of rows in the dataset to be processed]]></title>
            <link>https://govindk.medium.com/howto-azureml-restrict-of-rows-in-the-dataset-to-be-processed-1ed17c6c3d07?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ed17c6c3d07</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Wed, 06 Aug 2014 12:58:23 GMT</pubDate>
            <atom:updated>2014-08-06T16:09:19.633Z</atom:updated>
            <content:encoded><![CDATA[<h3>HowTo-AzureML-Restrict number of rows in the dataset to be processed</h3><p>Let us say there is a dataset with lot of data and you want to restrict number of rows to smaller value. You can do that by using Split Module’s Relative expression statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*h63uK9ZEfb6vgr2dnpwCag.png" /><figcaption>Restrict # of rows by using /Solit module/s Relative expression technique</figcaption></figure><p>One has to use the column name and provide an expression in the split module.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ed17c6c3d07" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HowTo-Get summary/str/boxplot/scatterplot kind of information for a dataset in AzureML]]></title>
            <link>https://govindk.medium.com/howto-get-summary-str-boxplot-scatterplot-kind-of-information-for-a-dataset-in-azureml-37ad01c67725?source=rss-7a485fd22c01------2</link>
            <guid isPermaLink="false">https://medium.com/p/37ad01c67725</guid>
            <dc:creator><![CDATA[Govind Kanshi]]></dc:creator>
            <pubDate>Tue, 05 Aug 2014 07:06:55 GMT</pubDate>
            <atom:updated>2014-08-05T07:06:55.192Z</atom:updated>
            <content:encoded><![CDATA[<p>We will look at the fires dataset to see how we can explore it to get the distribution of data, unique or missing values.</p><p>By default AzureML dataset’s visualize option (obtained by clicking on the output port) gets generic idea about the dataset. It shows up nice information about the distribution about the dataset across 10 bins in simple histogram. It also shows “unique values”, “missing values”, min/max/sd/mean/median,missing values and data type of the feature.</p><p>Simple Data analysis is available as “visualize” off the dataset’s output port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6F0fukp8_PuPdRpoWGlfyA.png" /><figcaption>max/min/median/mean/sd/unique values/missing values/Data Type of the feature information</figcaption></figure><p>The above also provides at glance information of feature’s distribution of data in form of histogram. And as you would expect clicking on “view as” results in box-plots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a_EFgyo-vuqRMh4RDBr42w.png" /><figcaption>Boxplot of the data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hY5pR0l10ogFpUfG4lYajQ.png" /><figcaption>Scatter Plot comparing two items</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IP-tNVKLTqjI_HP_deCdPA.png" /><figcaption>Multiboxplot comparing two features</figcaption></figure><p>MultiBoxPlot is unique in terms of plot automatically deciding — we have of categorical items — so let us do a multi-box plot.</p><p>Taking it further there is also Cross Tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*LU2qu7JI1QYncr-LbNzkeQ.png" /><figcaption>CrossTab — two categorical data points</figcaption></figure><p>Histograms provide ability to add cumulative distribution &amp; probability density. All plots allow scaling of features. Histograms allow changing the # of bins to get more granular information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NYJeU5A66f91ZwV7BJ8lDg.png" /><figcaption>Histogram with probability density</figcaption></figure><p>Visualization of data also has ability to take snapshots of the visualization and remove them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8eDI8Ue1PivxyeEA9EJpxw.png" /><figcaption>Snapshots on the right of the MultiBoxPlot are persistent</figcaption></figure><p>More descriptive Data Analysis is available from “Descriptive Statistics module” and the “Co-relation Module”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fYqWFTsyOC60D6BvSWt4FA.png" /><figcaption>More Descriptive analysis including skewness,percentile distribution,range</figcaption></figure><p>Negative Skewness implies mean of the data values is less the median and vice versa.</p><p>In the detailed analysis <em>kurtosis </em>is a measure of the <em>peakedness </em>of the data distribution. Negative kurtosis would indicates a <em>flat </em>data distribution, aka <strong>platykurtic. </strong>Positive kurtosis would indicates a <em>peaked </em>distribution, aka <strong>leptokurtic. T</strong>he <a href="http://www.r-tutor.com/elementary-statistics/probability-distributions/normal-distribution">normal distribution</a> has zero kurtosis, and is said to be <strong>mesokurtic.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jiIOydmXoruwTM1QQVKR4A.png" /><figcaption>Correlation of data</figcaption></figure><p>Why is Nan present for Month/Day? — i.e because they are categorical data points.</p><p>Here is how modules are connected to get detailed analysis and correlation. First data anaylsis is available off the main dataset itself as visualize. Rest are also available as a click on visualize off the respective modules’s output port.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lwvBqOc9yEk-BH4WJdwzVA.png" /><figcaption>Modules for data exploration</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1V6RZNR8ehgvioNp4f2YiA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=37ad01c67725" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>