<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dimitris Krekoukias on Medium]]></title>
        <description><![CDATA[Stories by Dimitris Krekoukias on Medium]]></description>
        <link>https://medium.com/@dikrek?source=rss-7a980735fd77------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NdKD_pAnFBEOs1kEAJMKwQ.jpeg</url>
            <title>Stories by Dimitris Krekoukias on Medium</title>
            <link>https://medium.com/@dikrek?source=rss-7a980735fd77------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:38:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dikrek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nimble Cloud Volumes Removing Risk From Cloud Storage]]></title>
            <link>https://medium.com/recoverymonkey/nimble-cloud-volumes-removing-risk-from-cloud-storage-edfd5daca17?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/edfd5daca17</guid>
            <category><![CDATA[private-cloud]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 27 Feb 2017 17:03:58 GMT</pubDate>
            <atom:updated>2017-02-28T01:59:06.515Z</atom:updated>
            <content:encoded><![CDATA[<p>As of February 27th 2017, Nimble Storage announced Nimble Cloud Volumes (NCV), marking Nimble’s entry into the cloud space.</p><p>For those short on time: Nimble Cloud Volumes is <em>block</em> storage as a service, works with compute from AWS &amp; Azure, and avoids the significant block storage drawbacks of those two providers. In essence, it is <em>enterprise</em> storage as a service, at a competitive price, while retaining all the cloud conveniences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*yGrPCGae5kkmZgFS.png" /><figcaption>NCV works with AWS and Azure</figcaption></figure><h3>Why get in the Storage as a Service space?</h3><p>People like subscription-based cloud offerings for various reasons. Mostly due to convenience, agility and economies of scale. Plus, of course, converting all expenses into OPEX. But this is not a treatise on “why cloud”.</p><p>Currently, the two largest hyperscale cloud providers are AWS and Azure. While both are great offerings overall, their block storage component is <em>severely</em> lacking in several key aspects compared to enterprise-grade storage.</p><p>This is something that most people either are unaware of, or simply choose to accept the risk since they see the other benefits as compelling.</p><p>Knowledge is power.</p><h3>Here are some of the downsides of AWS EBS and Azure block storage:</h3><ul><li>Availability SLA: 3 nines (if an enterprise array vendor mentioned 3 nines, you’d kick them out immediately)</li><li>Data integrity: 0.1–0.2% Annualized Failure Rate (AFR). 1–2 out of 1,000 volumes will suffer from partial or complete data loss (this is for EBS, Azure numbers N/A). If an enterprise storage vendor said they’d lose your data with a 0.2% chance per year, you’d permanently and with extreme prejudice put them on the “banned” list…</li><li>Extremely slow (and costly) backups, clones and recoveries (for instance, many hours just to take a snapshot! And with high I/O performance variability while data is shuffled back and forth from S3)</li><li>Hard and expensive to switch providers (by design)</li><li>Hard and expensive to repatriate data back to a private cloud (by design)</li><li>Significant constraints in some services might force customers to change their application layouts (1TB volume limits in Azure are unacceptably small for many deployments)</li><li>Data mobility between cloud providers isn’t easy (by design)</li><li>Troubleshooting &amp; monitoring are hard. How does one do Root Cause Analysis for issues within a cloud provider? They’re designed to be black boxes. What happens when something goes wrong? How long does troubleshooting take? Can I see detailed volume statistics and trending? <em>Why</em> is my application suffering?</li></ul><p>Especially for the more critical applications, the above list is enough to give people pause.</p><p>Well — what if there <em>was</em> a better way? A way to significantly de-risk using cloud storage?<strong> This is the reason Nimble is entering this market. To provide a way to remove some of the more significant challenges of cloud storage while keeping the best aspects of what people like about the cloud.</strong></p><h3>How did Nimble Storage build the NCV STaaS offering?</h3><p>Nimble has built its own cloud to deliver the Nimble Cloud Volumes service.</p><p>The Nimble cloud is strategically placed around the world in locations in close proximity to AWS and Azure cloud infrastructure, ensuring low-latency access. The Nimble cloud is built using Nimble flash technology for storing data (i.e. it does not use the storage from either AWS or Azure). The service is managed through a simple portal and provides storage volumes that can be attached to Amazon EC2 or Azure Virtual Machine instances.</p><h3>The Benefits Customers get out of Nimble Cloud Volumes</h3><p>As previously mentioned, the entire goal of NCV is to keep what’s good about the cloud while at the same time removing some of the more significant risks and inconveniences. Here are the main benefits:</p><h3>Easy to Consume</h3><p>NCV is managed completely through an easy portal that lets users choose a capacity, performance and backup SLA, and attach to either AWS and/or Azure.</p><h3>Fully Managed STaaS Offering</h3><p>This is what customers expect out of the cloud. NCV is a fully managed service. The customer doesn’t have to ever touch and maintain a storage array, physical or virtualized.</p><h3>Extreme Reliability</h3><p>The NCV storage uptime is the same as what Nimble enterprise arrays provide: a measured 6+ nines.</p><p>The data integrity is even more impressive at millions of times more than what native cloud block storage provides. More details <a href="http://recoverymonkey.org/2016/12/01/uncompromising-resiliency/">here</a>.</p><h3>Enterprise Copy Data Management</h3><p><em>Instant</em> snapshots and clones. <em>Instant</em> recoveries. In addition, we only charge for delta usage when making clones, <em>not</em> full copies. Which means you can take many, many clones of something and the initial cost is almost zero. Only pay for the IOPS and additional data you consume as the clones deviate from the original. At scale, the savings can be significant.</p><h3>Provisioned IOPS, Low Latency plus Auto QoS</h3><p>Everyone expects an IOPS SLA from a STaaS offering. We do of course offer an IOPS SLA with NCV.</p><p>But we also take it to the next level by automatically prioritizing the <em>right</em> I/O using heuristics. That way, even if you are hitting your QoS limit, the right kind of I/O will still get prioritized automatically.</p><p>For instance, <em>automatically</em> prioritize latency-sensitive DB I/O vs latency-insensitive sequential I/O. Both because it makes sense and because nobody has the time to deal with micromanaging QoS. More <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">here</a>.</p><h3>Easy Mobility Eliminates Cloud Lock-In</h3><p>This is a big one. We make the following things easy and possible:</p><ol><li><strong>Cloud on-off ramp</strong>: easily and quickly send your data to the cloud. Want to stop using the cloud? That’s also easy and fast. And there are no egress charges.</li><li><strong>Multicloud without lock-in</strong>: easily use multiple public cloud offerings (and switch between them) without needing to migrate data.</li><li><strong>Hybrid Cloud Portability</strong>: Easy to mix and match private and public cloud. Easily move data back to on-premises private cloud if needed. Easily burst what you need to the cloud.</li></ol><h3>Global Visibility with Predictive Analytics</h3><p>Last, but by no means least: We are extending <a href="https://www.nimblestorage.com/technology-products/infosight/">InfoSight</a> to the cloud. The service that massively lowers business risk for Nimble private cloud customers will now also be available to public cloud consumers.</p><ul><li><strong>Predict and prevent issues across the stack</strong>: Because it’s really about the <em>applications</em>, not just the storage. Pinpoint difficult issues quickly, even if they have nothing whatsoever to do with storage.</li><li><strong>Visibility no matter where your data is</strong>: It shouldn’t matter whether your data is in the public or private cloud or both. Have detailed visibility into your data including exotic things like I/O histograms or more pedestrian (but still useful) things like detailed trending analysis.</li><li><strong>Predict, recommend, optimize</strong>: Use Predictive Analytics to go beyond reporting and use the information to optimize data placement — which could end up saving you money.</li></ul><p>Finally, the cloud stops being a black box when problems arise.</p><h3>In Summary</h3><p>The Nimble Cloud Volumes service was built to address some significant shortcomings AWS and Azure block storage services have, while retaining the benefits people expect from the cloud and providing much more flexibility to customers.</p><p>NCV provide benefits such as increased reliability, predictability, faster backups and clones, plus integration with the best infrastructure analytics platform, InfoSight.</p><h3>Some example methods of deployment:</h3><ul><li>Production in private cloud, DR in public cloud</li><li>Test/Dev in public cloud</li><li>“All In” public cloud</li></ul><p>We are accepting beta customers now (several big names already using it). Go <a href="https://www.nimblestorage.com/cloud/">here</a> to take a look and sign up.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edfd5daca17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/nimble-cloud-volumes-removing-risk-from-cloud-storage-edfd5daca17">Nimble Cloud Volumes Removing Risk From Cloud Storage</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Considerations for Implementing NVMe Storage]]></title>
            <link>https://medium.com/recoverymonkey/practical-considerations-for-implementing-nvme-storage-b5c30c32fd4?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5c30c32fd4</guid>
            <category><![CDATA[low-latency]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[nimble-storage]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[nvme]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:18:01 GMT</pubDate>
            <atom:updated>2017-01-30T03:18:01.696Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we begin, something needs to be clear: Although dual-ported NVMe drives are not yet cost effective, the architecture of Nimble Storage is NVMe-ready <em>today</em>. And always remember that in order to get good benefits from NVMe, one needs to implement it all the way from the client. Doing NVMe only at the array isn’t as effective.</p><p>In addition, Nimble <em>already</em> uses technology far faster than NVMe: Our write buffers use byte-addressable NVDIMM-N, instead of slower NVRAM HBAs or NVMe drives that other vendors use. Think about it: I/O happens at DDR4 RAM speeds, which makes even the fastest NVMe drive seem positively glacial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JNubCIgKILzPIe9N.png" /></figure><p>I did want to share my personal viewpoint of where storage technology in general may be headed if NVMe is to be mass-adopted in a realistic fashion and without making huge sacrifices.</p><h3>About NVMe</h3><p>Lately, a lot of noise is being made about NVMe technology. The idea being that NVMe will be the next step in storage technology evolution. And, as is the natural order of things, new vendors are popping up to take advantage of this perceived opening.</p><p>For the uninitiated: NVMe is a relatively new standard that was created specifically for devices connected over a PCI bus. It has certain nice advantages vs SCSI such as reduced latency and improved IOPS. Sequential throughput can be significantly higher. It can be more CPU-efficient. It needs a small and simple driver, the standard requires only 13 commands, and it can also be used over some FC or Ethernet networks (NVMe over Fabrics). Going through a fabric only adds a small amount of extra latency to the stack compared to DAS.</p><p>NVMe is strictly an optimized block protocol, and not applicable to NAS/object platforms unless one is talking about their internal drives.</p><p>Due to the additional performance, NVMe drives are a no brainer in systems like laptops and DASD/internal to servers. Usually there is only a small number (often just one device) and no fancy data services are running on something like a laptop… replacing the media with better media+interface is a good idea.</p><p>For enterprise arrays though, the considerations are different.</p><h3>NVMe Performance</h3><p>Marketing has managed to confuse people regarding NVMe’s true performance. It’s important to note that tests illustrating NVMe performance show a <em>single</em> NVMe device being faster than a <em>single</em> SAS or SATA SSD. But storage arrays usually don’t have a single device and so drive performance isn’t the bottleneck as it is with low media count systems.</p><p>In addition, most tests and research papers comparing NVMe to other technologies use <em>wildly</em> dissimilar SSD models. For instance, pitting a modern, ultra-high-end NVMe SSD against an older consumer SATA SSD with a <em>totally</em> different internal controller. This can make proper performance comparisons difficult. <em>How much of the performance boost is due to NVMe and how much because the expensive, fancy SSD is just a much better engineered device?</em></p><p>For instance, consider this chart of NVMe device latency, courtesy of Intel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uv7of5J_8xIGVywa.png" /></figure><p>As you can see, regarding latency, <strong>NVMe as a drive connection protocol <em>will</em> offer better latency than SAS or SATA but the difference is in the order of a few microseconds</strong>. The protocol differences become truly important only with next gen technologies like 3D Xpoint, which ideally needs a memory interconnect to shine (or, at a minimum, PCI) since the media is <em>so</em>much faster than the usual NAND. But such media will be prohibitively expensive to be used as the entire storage within an array in the foreseeable future, and would quickly be bottlenecked by the array CPUs at scale.</p><h3>NVMe over Fabrics</h3><p>Additional latency savings will come from connecting clients using NVMe over Fabrics. By doing I/O over an RDMA network, a latency reduction of around 100 microseconds is possible versus encapsulated SCSI protocols like iSCSI, assuming all the right gear is in place (HBAs, switches, host drivers). Doing NVMe at the client side also helps with lowering CPU utilization, which can make client processing overall more efficient.</p><h3>Where are the Bottlenecks?</h3><p>The reality is that the main bottleneck in today’s leading modern AFAs is the controller itself and <em>not</em>the SSDs (simply because there is enough performance in just a couple of dozen modern SAS/SATA SSDs to saturate most systems). Moving to competent NVMe SSDs will mean that those same controllers will now be saturated by maybe 10 NVMe SSDs. For example, a single NVMe drive may be able to read sequentially at 3GB/s, whereas a single SATA drive 500MB/s. Putting 24 NVMe drives in the controller doesn’t mean that magically the controller will now deliver 72GB/s. In the same way, a single SATA SSD might be able to do 100000 read small block random IOPS and an NVMe with better innards 400000 IOPS. Again, it doesn’t mean that same controller with 24 devices will all of a sudden now do 9.6 million IOPS!</p><h3>How Tech is Adopted</h3><p>Tech adoption comes in waves until a significant technology advancement is affordable and reliable enough to become pervasive. For instance, ABS brakes were first used in planes in 1929 and were too expensive and cumbersome to use in everyday cars. Today, most cars have ABS brakes and we take for granted the added safety they offer.</p><p>But consider this: What if someone told you that in order to get a <em>new</em> kind of car (that has several <em>great</em> benefits) you would have to utterly give up things like airbags, ABS brakes, all-wheel-drive, traction control, limited-slip differential? <em>Without</em> an equivalent replacement for these functions?</p><p>You would probably realize that you’re not <em>that</em> excited about the new car after all, no matter how much better than your existing car it might be in other key aspects.</p><p>Storage arrays follow a similar paradigm. There are several very important business reasons that make people ask for things like HA, very strong RAID, multi-level checksums, encryption, compression, data reduction, replication, snaps, clones, hot firmware updates. Or the ability to dynamically scale a system. Or comprehensive cross-stack analytics and automatic problem prevention.</p><p>Such features evolved over a long period of time, and help mitigate risk and accelerate business outcomes. They’re also <em>not</em>trivial to implement properly.</p><h3>NVMe Arrays Today</h3><p><strong>The challenge I see with the current crop of ultra-fast NVMe over Fabrics arrays is that they’re <em>so</em> focused on speed that they ignore the aforementioned enterprise features in lieu of sheer performance.</strong> I get it: it takes great skill, time and effort to <em>reliably</em> implement such features, especially in a way that they don’t strip the performance potential of a system.</p><p>There is also a significant cost challenge in order to <em>safely</em> utilize NVMe media <em>en masse</em>. Dual-ported SSDs are crucial in order to deliver proper HA. Current dual-ported NVMe SSDs tend to be very expensive per TB vs current SAS/SATA SSDs. In addition, due to the much higher speed of the NVMe interface, even with future CPUs that include FPGAs, <em>many</em> CPUs and PCI switches are needed to create a highly scalable system that can fully utilize such SSDs (and maintain enterprise features), which further explains why most NVMe solutions using the more interesting devices tend to be rather limited.</p><p>There are also client-side challenges: Using NVMe over Fabrics can often mean purchasing new HBAs and switches, plus dealing with some compromises. For instance, in the case of <a href="https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet">RoCE</a>, <a href="https://en.wikipedia.org/wiki/Data_center_bridging">DCB switches</a> are necessary, end-to-end congestion management is a challenge, and routability is not there until v2.</p><p>There’s a bright side: There actually exist some very practical ways to give customers the benefits of NVMe <em>without</em>taking away business-critical capabilities.</p><h3>Realistic Paths to NVMe Adoption</h3><p>We can divide the solution into two pieces, the direction chosen will then depend on customer readiness and component availability. All the following assumes <em>no</em> loss of important enterprise functionality (as we discussed, giving up on all the enterprise functionality is the easy way out when it comes to speed):</p><p><strong>Scenario 1: Most customers are <em>not</em> ready to adopt host-side NVMe connectivity:</strong></p><p>If this is the case, a good option would be to have something like a fast byte-addressable ultra-fast device inside the controller to massively augment the RAM buffers (like 3D Xpoint in a DIMM), or, if not available, some next-gen NVMe drives to act as cache. That would provide an overall speed boost to the clients and not need <em>any</em> client-side modifications. This approach would be the most friendly to an existing infrastructure (and a relatively economical enhancement for arrays) without needing all internal drives to be NVMe nor extensive array modifications.</p><p>You see, part of any competent array’s job is using intelligence to <em>hide</em> any underlying media issues from the end user. A good example: even super-fast SSDs can suffer from garbage collection latency incidents. A good system will smooth out the user experience so users won’t see extreme latency spikes. The chosen media and host interface are immaterial for this, but I bet if you were used to 100μs latencies and they suddenly spiked to 10ms for a while, it would be a bad day. Having an extra-large buffer in the array would help do this more easily, yet not need customers to change anything host-side.</p><p>An evolutionary second option would be to change <em>all</em> internal drives to NVMe, <em>but to make this practical would require wide availability of cost-effective dual-ported devices</em>. Note that with low SSD counts (less than 12) this would provide speed benefits even if the customer doesn’t adopt a host-side NVMe interface, but it will be a diminishing returns endeavor at larger scale, unless the controllers are significantly modified.</p><p><strong>Scenario 2: Large numbers of customers <em>are</em> ready and willing to adopt NVMe over Fabrics.</strong></p><p>In this case, the <em>first</em> thing that needs to change is the array connectivity to the outside world. That alone will boost speeds on modern systems even without major modifications. Of course, this will often mean client and networking changes to be most effective, and often such changes can be costly.</p><p>The next step depends on the availability of cost-effective dual-ported NVMe devices. But in order for <em>very</em>large performance benefits to be realized, pretty big boosts to CPU and PCI switch counts may be necessary, necessitating bigger changes to storage systems (and increased costs).</p><h3>Architecture Matters</h3><p>In the quest for ultra-low latency and high throughput without sacrificing enterprise features (yet remaining reasonably cost-effective), overall architecture becomes extremely important.</p><p>For instance, how will one do RAID? Even with NVMe over Fabrics, approaches like erasure coding and triple mirroring can be costly from an infrastructure perspective. Erasure coding remains CPU-hungry (even more so when trying to hit ultra-low latencies), and triple mirroring across an RDMA fabric would mean massive extra traffic on that fabric.</p><p>Localized CPU:RAID domains remain more efficient, and mechanisms such as Nimble NCM can fairly distribute the load across multiple storage nodes without relying on a cluster network for heavy I/O. This technology is available today.</p><h3>Next Steps</h3><p>In summary, I urge customers to carefully consider the overall business impact of their storage making decisions, especially when it comes to new technologies and protocols. Understand the true benefits first. Carefully balance risk with desired outcome, and consider the <em>overall</em> system and not just the components. Of course, one needs to understand the risks vs rewards first, hence this article. Just make sure that, in order to achieve a certain ideal, you don’t give up on <em>critical</em> functionality that you’ve been taking for granted.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5c30c32fd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/practical-considerations-for-implementing-nvme-storage-b5c30c32fd4">Practical Considerations for Implementing NVMe Storage</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uncompromising Resiliency]]></title>
            <link>https://medium.com/recoverymonkey/uncompromising-resiliency-db47105c35dd?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/db47105c35dd</guid>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[nimble-storage]]></category>
            <category><![CDATA[checksums]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[reliability]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:15:39 GMT</pubDate>
            <atom:updated>2017-01-30T03:15:39.620Z</atom:updated>
            <content:encoded><![CDATA[<p>(cross-posted at <a href="https://www.nimblestorage.com/blog/uncompromising-resiliency/">https://www.nimblestorage.com/blog/uncompromising-resiliency/</a>)</p><p>The cardinal rule for enterprise storage systems is to never compromise when it comes to data integrity and resiliency. Everything else, while important, is secondary.</p><p>Many storage consumers are not aware of what data integrity mechanisms are available or which ones are necessary to meet their protection expectations and requirements. It doesn’t help that a lot of the technologies and the errors they prevent are rather esoteric. However, if you want a storage system that safely stores your data and always returns it correctly, no measure is too extreme.</p><h3>The Golden Rules of Storage Engineering</h3><p>When architecting enterprise storage systems, there are three Golden Rules to follow.</p><p>In order of criticality:</p><ol><li><strong>Integrity</strong>: Don’t ever return incorrect data</li><li><strong>Durability</strong>: Don’t ever lose any data</li><li><strong>Availability</strong>: Don’t ever lose access to data</li></ol><p>To better understand the order, ask yourself, “what is preferred, temporary loss of <em>access to data </em>or the storage system returning the <em>wrong</em> data <em>without anyone even knowing it’s wrong</em>?”</p><p>Imagine life or death situations, where the wrong piece of information could have <em>catastrophic</em> consequences. Interestingly, vendors exist that focus a lot on Availability (even offering uptime “guarantees”) but are lacking in Integrity and Durability. Being able to access the array but have data corruption is almost entirely useless. Consider modern storage arrays with data deduplication and/or multi-petabyte storage pools. The effects are far more severe now that a single block represents the data for 1–100+ blocks and data is spread across 10’s — 100’s of drives instead of a few drives.</p><h3>The Nimble Storage Approach</h3><p>Nimble Storage has taken a multi-stage approach to satisfy the Golden Rules, and in some cases, the amount of protection offered verges on being paranoid (but the good kind of paranoid).</p><p>Simply, Nimble employs these mechanisms:</p><ol><li><strong>Integrity</strong>: Comprehensive multi-level checksums</li><li><strong>Durability</strong>: Hardened RAID protection and resilience upon power loss</li><li><strong>Availability</strong>: Redundant hardware coupled with predictive analytics</li></ol><p>We will primarily focus on the first two as they are often glossed over, assumed, or not well understood. Availability will be discussed in a separate blog, however it is good to mention a few details here.</p><p>To start, Nimble has achieved greater than 99.9997% measured uptime since 2014. This is measured across more than 9,000 customers using multiple generations of hardware and software. A key aspect of Nimble’s availability comes from InfoSight which continually improves and learns as more systems are used. Each week, trillions of data points are analyzed and processed with the goal of predicting and preventing issues, not just in the array, but across the entire infrastructure. 86% of issues are detected and automatically resolved before the customer is even aware of the problem. To further enhance this capability, Nimble’s Technical Support Engineers can resolve issues faster as they have all the data available when an issue arises. This bypasses the hours-days-weeks often required to collect data, send to support, analyze, repeat — until a solution can be found.</p><h3>Data Integrity Mechanisms in Detail</h3><p>The goal is simple: What is read must always match what was written. And, if it doesn’t, we fix it on the fly.</p><p>What many people don’t realize is there are occasions where storage media will lose a write, corrupt it or place it at the wrong location on the media. RAID (including 3-way mirroring) or Erasure Coding are <em>not</em> enough to protect against such issues. The older T10 PI employed by some systems is also not enough to protect against all eventualities.</p><p>The solution involves using checksums which get more computationally intensive the more paranoid one is. As checksums are computationally intensive, certain vendors don’t employ or minimally employ them to gain more performance or faster time to market. Unfortunately, the trade-off can lead to data corruption.</p><p>Broadly, Nimble creates a checksum <em>and</em> a “self-ID” for each piece of data. The checksum protects against data corruption. The self-ID protects against lost/misplaced writes and misdirected reads (incredible as it may seem, these things happen enough to warrant this level of protection).</p><p>For instance, if the written data has a checksum, and <em>corruption</em> occurs, when the data is read and checksummed <em>again</em>, the checksums will not match. However, if instead the data was <em>placed at an incorrect location on the media</em>, the checksums will match, but the self-IDs will <em>not</em> match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*eYx3Ytn41B0zVvZO.png" /></figure><p><strong>Where it gets interesting:</strong></p><p>Nimble doesn’t just do block-level checksums/IDs. These multi-level checksums are also performed:</p><ol><li>Per segment in each write stripe</li><li>Per block, before <em>and</em> after compression</li><li>Per snapshot (including all internal housekeeping snaps)</li><li>For replication</li><li>For all data movement within a cluster</li><li>All data and metadata in NVRAM</li></ol><p>This way, every likely data corruption event is covered, including metadata consistency and replication issues, which are often overlooked.</p><h3>Durability Mechanisms in Detail</h3><p>There are two kinds of data on a storage system and both need to be protected:</p><ol><li>Data in flight</li><li>Data on persistent storage</li></ol><p>One may differentiate between user data and metadata but we protect both with equal paranoid fervor. Some systems try to accelerate operations by not protecting metadata sufficiently, which greatly increases risk. <em>This is especially true with deduplicating systems, where metadata corruption can mean losing everything!</em></p><p><strong>Data in flight</strong> is data that is not yet committed to persistent storage. Nimble ensures all critical data in flight is checksummed and committed to both RAM and an ultra-fast byte-addressable NVDIMM-N memory module sitting right on the motherboard. The NVDIMM-N is mirrored to the partner controller and both controller NVDIMMs are protected against power loss via a supercapacitor. In the event of a power loss, the NVDIMMs simply flush their contents to flash storage. This approach is extremely reliable and doesn’t need inelegant solutions like a built-in UPS.</p><p><strong>Data on persistent storage</strong> is protected by what we call Triple+ Parity RAID. <em>Three orders of magnitude more resilient than RAID6</em>. For comparison, RAID6 is three orders of magnitude more resilient than RAID5. The “+” sign means that there is <em>extra</em> intra-drive parity that can safeguard against entire sectors being lost even if three whole drives fail in a single RAID group.</p><p>Some might say this is a bit much, however with drive sizes increasing rapidly (especially SSDs) and drive read error rates increasing as drives age, it was the architecturally correct choice to make.</p><h3>In Summary</h3><p>Users frequently assume that all storage systems will safely store their data. And they will, most of the time. But when it comes to your data, “most of the time” isn’t good enough. No measure should be considered too extreme. When looking for a storage system, it’s worth taking the time to understand all situations where your data could be compromised. And, if nothing else, it’s worth choosing a vendor who is paranoid and goes to extremes to keep your data safe.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db47105c35dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/uncompromising-resiliency-db47105c35dd">Uncompromising Resiliency</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Automated Headroom Management]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-automated-headroom-management-737602180327?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/737602180327</guid>
            <category><![CDATA[failover]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[headroom]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:14:22 GMT</pubDate>
            <atom:updated>2017-01-30T03:14:22.766Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we begin: This is another vendor-neutral post. I realize there may be no architecture that can do <em>everything</em>I’m proposing, but some may come closer to what you need than others. Whether you’re a vendor or a customer, see it as stuff you should be doing or be asking for respectively…</p><h3>Headroom!</h3><p><em>Headroom</em>is a term that applies to almost all technologies, and it’s crucially important for all of them. Some examples:</p><ul><li>Photography</li><li>Cars</li><li>Bridges</li><li>Storage arrays…</li></ul><h3>Why is Sufficient Headroom Important?</h3><p>Maintaining sufficient headroom in any solution is a way to <strong>ensure safety and predictability of operation under most conditions</strong> (<em>especially</em> under unfavorable ones).</p><p>For instance, if the maximum load for an evenly loaded bridge before it collapses is X, the overall recommended load will be a fraction of that. But even the weight/length/axle count of a <em>single</em>truck on a bridge will also be subject to certain strict limits in order to avoid excessive localized stress on the structure.</p><h3>Headroom in Storage Arrays</h3><p>Apologies to the seasoned storage pros for all the foundational material but it’s crucial to take this step by step.</p><p>It is important to note that headroom in arrays is not necessarily as simple as how busy the CPU is. Headroom is a multi-dimensional concept.</p><p>More factors than just CPU come into play, including how busy the underlying storage media are, how saturated various buses are, and how much of the CPU is spent on true workload vs opportunistic tasks (that could be deferred). Not to mention that in some systems, certain tasks are single-threaded and could pose an overall headroom bottleneck by maxing out a single CPU core, while the rest of the CPU is not busy at all.</p><p>Maintaining sufficient headroom in storage arrays is necessary in order to provide acceptable latency, especially in the event of high load during a controller failover. Depending on the underlying architecture of an array, different headroom approaches and calculations are necessary.</p><p>Some examples of different architectures:</p><ul><li>Active-Active controllers, per-controller pool</li><li>Active-Standby, single pool</li><li>Active-Active, single pool</li><li>Grid, single pool</li><li>Permutations thereof (it’s beyond the scope of this article to explore all possible options)</li></ul><p>The single vs multiple pool question complicates things a bit, plus things like disk ownership are also hugely important. This isn’t an argument about which architecture is better (it depends anyway), but rather about <em>headroom management in different architectures.</em></p><h3>Dual-Controller Headroom</h3><p>Dual-controller architectures need to be extremely careful with headroom management. After all, there are only two controllers in play. Here’s what sufficient headroom looks like in a dual-controller system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*kQ_CYILUzJ6kv52s.png" /></figure><p>There are not many options to keep things healthy in a dual-controller architecture. <strong>In an Active-Standby system</strong>, the Standby controller is ready to immediately take over. There is no danger in loading up the Active controller, aside from expected load-related latency.</p><p><strong>In an Active-Active HA system</strong>, maintaining a healthy amount of headroom has to be managed so that there is, overall, an <em>entire controller’s worth of free headroom available</em>.</p><h3>Headroom in a Cluster of HA Pairs Architecture</h3><p>There are several implementations that make use of a multiple HA Pair architecture. Often, the multiple HA pairs present a virtual pool to the outside world, even if, internally, there are multiple private pools. Some implementations just keep it to pools owned by each controller.</p><p>Here’s an example of healthy headroom in such a system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*tnhZAN8aFX-nUMuk.png" /></figure><p>Even though there are multiple controllers (at least 4 total), in order to maintain an overall healthy system, a total of 100% headroom needs to be maintained in <em>each</em> HA pair, otherwise the performance of an underlying private pool (in green) might suffer, making the overall virtual pool performance (light blue) unpredictable.</p><h3>Headroom in a True Grid Architecture</h3><p>Grid Architectures spread overall load among multiple nodes (often plain servers with some disks inside and connected via a network).</p><p>In such a scheme, overall headroom that needs to be maintained per node as a percentage is 100/N, where <em>N</em> is the number of nodes in the storage cluster.</p><p>So, in a 4-node cluster, 100/4=25% headroom per node needs to be maintained.</p><p>This doesn’t account for the significant work that rebalancing after a node failure takes in such architectures, nor the <em>capacity</em> headroom needed, but it’s roughly accurate enough for our purposes.</p><p>Schematically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*R6qrUMjvMA5wMfNf.png" /></figure><h3>How Headroom is Managed is Crucial</h3><p>In order to manage headroom, four things need to be able to happen first:</p><ol><li>Be able to calculate headroom</li><li>Be able to throttle workloads</li><li>Be able to prioritize between types of workload</li><li>Be able to move workloads around (architecture-dependent).</li></ol><p>The only architecture that inherently makes this a bit easier is Active-Standby since there is always a controller waiting to take over if anything bad happens. But even with a single active controller, headroom needs to be managed in order to avoid bad latency conditions during normal operation (see <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">here</a>for an example approach). Remember, headroom is a multi-dimensional thing.</p><h3>Example Problem Case: Imbalanced &amp; Overloaded Controllers</h3><p>Consider the following scenario: An Active-Active system has both controllers overloaded, and one of them is really busy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*1LPNDPS1WEGVlmfg.png" /></figure><p>Clearly, there are a few problems with this picture:</p><ol><li>It may be <em>impossible</em> to fail over in the event of a controller failure (total load is 165% of a single controller’s headroom)</li><li>The first controller may already be experiencing latency issues</li><li>Why did the system even get to this point in the first place?</li></ol><p>This is a commonplace occurrence unfortunately.</p><h3>Automation is Key in Managing Headroom</h3><p>The biggest problem in our example is actually the last point: <em>Why was the system allowed to get to that state to begin with?</em></p><p>Even if a system is able to calculate headroom, throttle workloads and move workloads around, if nothing is done <em>automatically</em> to prevent problems, it’s <em>extremely</em>easy for users to get into the problem situation depicted above. I’ve seen it affect critical production systems far too many times for comfort.</p><h3>Manual QoS is Not The Best Answer</h3><p>Being able to <em>manually</em> throttle workloads can obviously help in such a situation. The problems with the manual QoS approach are outlined in a past <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">article</a>, but, in summary, most users simply have absolutely <em>no</em> idea what the actual limits should be (nor should they be expected to). Most importantly, placing QoS limits up front doesn’t result in balanced controllers… and may even result in other kinds of performance problems.</p><p>Of course, using QoS limits <em>reactively</em> is not going to prevent the problem from occurring in the first place.</p><p>Some companies offer Data Classification as a Professional Services engagement, in order to try and figure out an IOPS/TB/Application metric. Even if that is done, it <em>doesn’t</em>result in balanced controllers… it’s also not very useful in dynamic environments. It’s more used as a guideline for setting up manual QoS.</p><h3>Automation Mechanisms to Consider for Managing Headroom</h3><p>Clearly, pervasive automation is needed in order to keep headroom at safe levels.</p><p>I will split up the proposed mechanisms per architecture. There is some common functionality needed regardless of architecture:</p><h3>Common Automation Needed</h3><p>Every architecture needs to have the ability to automatically achieve the following, <em>without</em>user intervention at any point:</p><ol><li>Conserve headroom per controller</li><li>Differentiate between different kinds of user workloads</li><li>Differentiate between different kinds of system workloads</li><li>Automatically prioritize between different workloads, especially under pressure</li><li>Automatically throttle different kinds of workloads, especially under pressure</li></ol><p>And now for the extra automation needed per architecture:</p><h3>Active-Standby Automation</h3><p>If in a single HA pair, nothing else is needed. If in a scale-out cluster of Active-Standby pairs:</p><ol><li>Automatically balance capacity and headroom utilization between HA pairs even if they’re different types</li><li>Be able to auto-migrate workloads to other cluster nodes (if using multiple pools instead of one)</li></ol><h3>Active-Active Automation</h3><ol><li>Automatically conserve one node’s worth of headroom across the HA pair (50/50, 60/40, 70/30 — all are OK)</li><li>When provisioning new workloads, auto-balance them by performance and capacity across the nodes</li><li>Be able to balance by auto-migrating workloads to the other node (if using multiple pools instead of one)</li></ol><h3>Active-Active with Multiple HA Pairs Automation</h3><ol><li>Automatically conserve one node’s worth of headroom per HA pair</li><li>Be able to auto-migrate workloads to any other node</li><li>Automatically balance workloads and capacity utilization in the underlying per-HA pools</li></ol><h3>Grid Automation</h3><ol><li>Automatically conserve at <em>least</em> one node’s worth of headroom across the grid</li><li>Automatically conserve enough capacity to be able to lose one node, rebalance, and have enough capacity left to lose another one (the more cautious may want the capability to lose <em>2–3 nodes simultaneously</em>)</li><li>Automatically take into account grid size and rebalancing effort in order to conserve the right amount of headroom</li></ol><h3>In Closing…</h3><p>If you’re a consumer of storage systems, always remember to be running your storage with sufficient headroom to be able to sustain a major failure without overly affecting your performance.</p><p>In addition, when looking to refresh your storage system, always ask the vendors about how they automate their headroom management.</p><p>Finally, if any vendor is quoting you performance numbers, <em>always</em> ask them how much headroom is left on the array at that performance level… (in addition to the extra questions about read/write percentages and latencies you should be <a href="http://recoverymonkey.org/2012/07/26/an-explanation-of-iops-and-latency/">asking already</a>).</p><p>The answer may surprise you.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=737602180327" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-automated-headroom-management-737602180327">The Importance of Automated Headroom Management</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of SSD Firmware Updates]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-ssd-firmware-updates-c6582ab29fe8?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/c6582ab29fe8</guid>
            <category><![CDATA[firmware]]></category>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[ssd]]></category>
            <category><![CDATA[sd]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:11:20 GMT</pubDate>
            <atom:updated>2017-01-30T03:11:39.802Z</atom:updated>
            <content:encoded><![CDATA[<p>I wanted to bring this crucial issue to light since I’m noticing several storage vendors being either cavalier about this or simply unaware.</p><p>I will explain why solutions that don’t offer some sort of automated, live SSD firmware update mechanism are potentially <em>extremely</em>risky propositions. Yes, this is another “vendor hat off, common sense hat on” type of post.</p><p><strong>Modern SSD Architecture is Complex</strong></p><p>The increased popularity and lower costs of fast SSD media are good things for storage users, but there is some inherent complexity within each SSD that many people are unaware of.</p><p>Each modern SSD is, in essence, <em>an entire pocket-sized storage array</em>, that includes, among other things:</p><ul><li>An I/O interface to the outside world (often two)</li><li>A CPU</li><li>An OS</li><li>Memory</li><li>Sometimes Compression and/or Encryption</li><li>What is, in essence, a log-structured filesystem, complete with complex load balancing and garbage collection algorithms</li><li>An array of flash chips driven in parallel through multiple channels</li><li>Some sort of RAID protection for the flash chips, including sparing, parity, error checking and correction…</li><li>A supercapacitor to safely flush cache to the flash chips in case of power failure.</li></ul><p>Sounds familiar?</p><p><strong>With Great Power and Complexity Come Bugs</strong></p><p>To make something clear: This discussion has <em>nothing</em> to do with overall SSD endurance &amp; hardware reliability. Only the software aspect of the devices.</p><p>All this extra complexity in modern SSDs means that an increased number of bugs compared to simpler storage media <em>is a statistical certainty.</em> There is just a <em>lot</em> going on in these devices.</p><p>Bugs aren’t necessarily the end of the world. They’re something understood, a fact of life, and there’s this magical thing engineers thought of called… Patching!</p><p>As a fun exercise, go to the firmware download pages of various popular SSDs and check the release notes for some of the bugs fixed. Many fixes address some rather abject gibbering horrors… :)</p><p>Even costlier enterprise SSDs have been afflicted by some <em>really</em> dangerous bugs — usually <em>latent defects</em> (as in: they don’t surface until you’ve been using something for a while, which may explain why these bugs were missed by QA).</p><p>I fondly remember a bug that hit some arrays at a previous place of employment: the SSDs would work great but after a certain number of hours of operation, if you shut your machine down, the SSDs would <em>never</em> come up again. Or, another bug that hit a very popular SSD that would downsize itself to an awesome 8MB of capacity (losing all existing data of course) once certain conditions were met.</p><p>Clearly, these are some pretty hairy situations. And, what’s more, <strong>RAID, checksums and node-level redundancy <em>wouldn’t</em> protect against all such bugs.</strong></p><p>For instance, think of the aforementioned power off bug — <em>all</em> SSDs of the same firmware vintage would be affected <em>simultaneously</em> and the entire array would have <em>zero</em> SSDs that functioned.<em> This actually happened</em>, I’m <em>not</em> talking about a theoretical possibility. You know, just in case someone starts saying “but SSDs are reliable, and think of all the RAID!”</p><p>It’s all about approaching correctness from a holistic point of view. Multiple lines of defense are necessary.</p><p><strong>The Rules: How True Enterprise Storage Deals with Firmware</strong></p><p>Just like with Fight Club, there are some basic rules storage systems need to follow when it comes to certain things.</p><ol><li>Any firmware patching should be a non-event. Doesn’t matter what you’re updating, there should be no downtime.</li><li>ANY firmware patching should be a NON-EVENT. Doesn’t matter what you’re updating, there should be NO downtime!</li><li>Firmware updates should be automated even when dealing with devices <em>en masse</em>.</li><li>The customer should automatically be notified of important updates they need to perform.</li><li>Different vintage and vendor component updates should be handled automatically and centrally. And, most importantly: <em>Safely</em>.</li></ol><p>If these rules are followed, bug risks are <em>significantly</em> mitigated and higher uptime is possible. Enterprise arrays typically will follow the above rules (but <em>always</em>ask the vendor).</p><p><strong>Why Firmware Updating is a Challenge with Some Storage Solutions</strong></p><p>Certain kinds of solutions make it inherently harder to manage critical tasks like component firmware updates.</p><p>You see, being able to hot-update different kinds of firmware in any given set of hardware means that <em>the mechanism doing the updating must be intimately familiar with the underlying hardware &amp; software combination, however complex.</em></p><p>Consider the following kind of solution, maybe for someone sold on the idea that white box approaches are the future:</p><ul><li>They buy a bunch of diskless server chassis from Vendor A</li><li>They buy a bunch of SSDs from Vendor B</li><li>They buy some Software Defined Storage offering from Vendor C</li><li>All running on the underlying OS of Vendor D…</li></ul><p>Now, let’s say Vendor B has an emergency SSD firmware fix they made available, easily downloadable on their website. Here are just <em>some</em>of the challenges:</p><ol><li>How will that customer be notified by Vendor B that such a critical fix is available?</li><li>Once they have the fix located, which Vendor will automate updating the firmware on the SSDs of Vendor B, and how?</li><li>How does the customer know that Vendor B’s firmware fix doesn’t violently clash with something from Vendor A, C or D?</li><li>How will all that affect the data-serving functionality of Vendor C?</li><li>Can any of Vendors A, B, C or D orchestrate <em>all</em> the above safely?</li><li>With no downtime?</li></ol><p>In most cases I’ve seen, the above chain of events will not even progress past #1. <em>The user will simply be unaware of any update</em>, simply because component vendors don’t usually have a mechanism that alerts individual customers regarding firmware.</p><p>You could inject a significant permutation here: <em>What if you buy the servers pre-built, including SSDs, from Vendor A, including full certification with Vendors C and D?</em></p><p>Sure — it still does not materially change the steps above. One of Vendors A, C or D still need to somehow:</p><ol><li>Automatically alert the customer about the critical SSD firmware fix being available</li><li>Be able to non-disruptively update the firmware…</li><li>…While not clashing with the other hardware <em>and</em> software from Vendors A, C and D</li></ol><p>I could expand this type of conversation to other things like overall environmental monitoring and checksums but let’s keep it simple for now and focus on just component firmware updates…</p><p><strong>Always Remember — Solve Business Problems &amp; Balance Risk</strong></p><p>Any solution is a compromise. Always make sure you are comfortable with the added risk certain areas of compromise bring (and that you are fully aware of said risk).</p><p>The allure of certain approaches can be significant (at the very least because of lower promised costs). It’s important to maintain a balance between increased risk and business benefit.</p><p>In the case of SSDs specifically, the utter criticality of certain firmware updates means that it’s crucially important for any given storage solution to be able to safely and automatically address the challenge of updating SSD firmware.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6582ab29fe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-ssd-firmware-updates-c6582ab29fe8">The Importance of SSD Firmware Updates</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of the Effective Capacity Ratio in Modern Storage Systems]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-the-effective-capacity-ratio-in-modern-storage-systems-e7bc62e36cd0?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7bc62e36cd0</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[fud]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[deduplication]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:10:02 GMT</pubDate>
            <atom:updated>2017-01-30T03:10:02.556Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern storage devices (especially All Flash Arrays), extensive data reduction techniques are commonplace and expected by customers.</p><p>This has, unavoidably, led to various marketing schemes that aim to make certain systems seem more appealing than the rest. Or at least not <em>less</em> appealing…</p><p>I will attempt to explain what customers should be looking for when trying to decipher capacity claims from a manufacturer.</p><p>In a nutshell — and for the ADD-afflicted — <strong>the most important number you should be looking for is the Effective Capacity Ratio, which is simply: <em>(Effective Capacity)/(Raw Capacity)</em></strong><em>. Ignore</em> the more common but far less useful Data Reduction Ratio, which is: (Effective Capacity)/(Usable Capacity).</p><p>Read on for more detail…</p><p><strong>The problem</strong></p><p>Most modern (and some not so modern) storage vendors claim a similar Data Reduction Ratio for their devices. Numbers like 5:1 seem commonplace these days. Which, to anyone with a modicum of common sense, simply means “I can store five times more stuff in the same space”. Unfortunately, the same 5:1 ratio does <em>not</em>mean the same end result for all vendors…</p><p><strong><em>Why</em> this is a problem</strong></p><p>Customers simply want to get a good deal for their money. The reality is that the exact same 5:1 Data Reduction Ratio may mean <em>very</em> different things between storage devices.</p><p>For instance: Not all vendors count space savings using the same math. Is the virtual size of snapshots calculated in the final savings ratio? (for example, if I take 5 snaps of a 1TB volume one after another, is that showing up as 6:1 savings?)</p><p>How about Thin Provisioning? Is that included? Such math can <em>wildly</em> alter the overall efficiency numbers.</p><p>Here’s a clear example of why counting thin provisioning in the overall ratio is probably misleading, and best used only for educational purposes… <em>Remember that overall savings ratios are multiplicative</em>. For example, 2:1 compression and 2:1 deduplication mean 4:1 overall savings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*7iwNqygRqCnd7ewbbGmg_g.png" /></figure><p>Does anyone <em>really</em> believe that such a system is actually providing <em>1000:1 </em>savings? Or even 10:1? <strong>Yet that’s what many vendors are counting towards savings ratios, <em>without</em> separating the thin provisioning savings from the overall savings number</strong>.</p><p>However, there is <em>another</em> dimension, and it has to do with <strong>how efficiently the raw capacity is actually utilized</strong>. It will be one of my many <em>Captain Obvious</em>moments for some of you, but I’ve seen enough people get confused, so it’s worth explaining.</p><p><strong>The Effective Capacity Ratio</strong></p><p>Different storage systems have different ways of utilizing their raw capacity. For example, a mirrored system can never have better than a 50% Usable:Raw ratio. By definition, it’s mathematically <em>impossible</em> since 2 copies are needed. That’s not even counting spares and other possible overheads.</p><p>Systems that do triple mirroring can’t do better than a theoretical 33.3% Usable:Raw etc.</p><p><strong>Some definitions are in order:</strong></p><ul><li><em>Usable Capacity</em>: How much data I can store in a system <em>after</em> overheads such as RAID, sparing etc. but <em>before</em> data reduction techniques.</li><li><em>Raw Capacity</em>: Add up the capacity of all the storage media in the system. Usually a Base 10 number (TB/GB, not Base 2, TiB/GiB)</li><li><em>Effective Capacity</em>: How much data I can store in a system <em>after</em> data reduction techniques like Deduplication and Compression but <em>not</em> Thin Provisioning</li><li><em>Data Reduction Ratio</em>: (Effective Capacity)/(Usable Capacity)</li><li><strong><em>Effective Capacity Ratio</em></strong>: (Effective Capacity)/(Raw Capacity)</li></ul><p><strong>Who is <em>More</em> Efficient?</strong></p><p>If every vendor is claiming 5:1 average savings, who is <em>truly</em> more efficient? A <em>Reductio ad Absurdum </em>example makes it pretty clear:</p><ol><li>A vendor that can do a Data Reduction Ratio of 5:1 but has a Usable Capacity of <strong>10%</strong> vs Raw or…</li><li>A vendor that can do a Data Reduction Ratio of 5:1 but has a Usable Capacity of <strong>70%</strong> vs Raw?</li></ol><p>Let’s put some numbers on a table. They roughly correspond to some existing storage vendors today (there may be some variation depending on whether the numbers for each line are TB vs TiB — everyone shows numbers differently — but the overall point remains the same):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*zNfryeWfzy3HpFHOVzSFIQ.png" /></figure><p>As you can see, the <em>same</em> Data Reduction Ratio, on the <em>same</em> amount of Raw Capacity, can have <em>wildly</em> different Effective Capacity results, depending on the system.</p><p>Clearly, a truly efficient system is one that can both:</p><ul><li>Provide a high Usable:Raw ratio <em>and</em></li><li>Provide a high Data Reduction Ratio (that does <em>not</em> include fluff like Thin Provisioning)</li></ul><p><strong>The Business Benefits of a High Effective Capacity Ratio</strong></p><p>There are multiple business reasons why chasing a high Effective Capacity Ratio is important:</p><ul><li>High rack density. Some vendors are <em>eight times</em> more dense than others (ironically, the ones claiming “white box economics” are the <em>least</em> dense and most expensive). This could lead to <em>significant</em> power/cooling and $/rack savings</li><li>Lower CapEx — if a vendor needs 2x the Raw Capacity vs another, guess who’s paying for the difference?</li><li>Less complexity — denser devices means <em>less</em> devices, less cables, less switching</li></ul><p><strong>What <em>you</em> can do as a Customer</strong></p><p><em>Level the playing field</em>. It’s actually pretty easy:</p><ul><li>Insist on seeing all capacity numbers expressed as <em>TiB</em> from every vendor — and if they don’t know what <em>that</em> means, <em>run</em>… (or at least subtract 9% from any capacity number you see and you’ll be safer)</li><li>Insist on seeing the Data Reduction Ratio <em>without</em> including Thin Provisioning or Snapshots. If they can’t break out the savings by category, <em>run</em>…</li><li>Insist on seeing the Usable:Raw percentage for various configurations. Is it better in larger configs vs smaller? Is it following all best practices? What <em>are</em> the best practices for space consumption? If they tell you to ignore the man behind the curtain, <em>run</em>… (there is a theme developing)</li><li><strong><em>Do your own Effective Capacity Ratio calculation</em></strong> and assign a $/Effective TiB to each vendor</li></ul><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7bc62e36cd0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-the-effective-capacity-ratio-in-modern-storage-systems-e7bc62e36cd0">The Importance of the Effective Capacity Ratio in Modern Storage Systems</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Processor scheduling and quanta in Windows (and a bit about Unix/Linux)]]></title>
            <link>https://medium.com/@dikrek/processor-scheduling-and-quanta-in-windows-and-a-bit-about-unix-linux-fb5ab02828e2?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb5ab02828e2</guid>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[windows-server]]></category>
            <category><![CDATA[scheduling]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[performance]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Sun, 29 Jan 2017 03:57:48 GMT</pubDate>
            <atom:updated>2017-01-29T03:57:48.979Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the more exotic and exciting IT subjects is the one of processor scheduling (if you’re not excited, read on, practical stuff to be seen later in the text). Multi-tasking OSes just give the illusion that they’re doing things in parallel — in reality, the CPUs rapidly skip from task to task using various algorithms and heuristics, making one think the processes truly are running simultaneously. The choice of scheduling algorithm can be immensely important.</p><p>Wikipedia has a nice article on schedulers in general: <a href="http://en.wikipedia.org/wiki/Scheduling_%28computing%29">en.wikipedia.org/wiki/Scheduling_%28computing%29</a>, good primer.</p><p>To cut a long story short: the processors are allowed to spend finite chunks of time (<em>quanta</em>) per process. <em>Note that the quantum has nothing to do with task priority, it’s simply the amount of time the CPU will spend on the task</em>. Every time the CPU switches to a new process, there’s what’s called a <em>context</em> <em>switch</em> (<a href="http://en.wikipedia.org/wiki/Context_switch">en.wikipedia.org/wiki/Context_switch</a>), which is computationally expensive. Obviously, we need to avoid excessive context switching but still maintain the illusion of concurrency.</p><p>In Windows Server (that uses a multi-level feedback queue algorithm, FYI), the default quantum is a fixed 120ms, close to many UNIX variants (100ms) and generally accepted as a reasonably short length of time that can fool humans into believing concurrency. Compare this to the workstation-level products (Windows Vista/XP/2000 Pro) that have a variable quantum that’s much shorter and also provide a quantum (<em>not</em> priority) boost to the foreground process (the process in the currently active window). In the workstation products, the quantum ranges from 20–60ms typically, with the background processes always relegated to the smallest possible quantum, ensuring that the application one is currently using “feels” responsive and that no background task hampers perceived performance too much. Typically, in a box that’s used as a busy terminal server this will be the better setting to use since it will ensure that the numerous “in-focus” user processes will all get a quantum sooner rather than later.</p><p>The longer, fixed quantum of Windows Server means that fewer system resources are wasted on context switching, and that all processes have the same quantum. More total system throughput can be realized with such a scheme, and it’s a more of a fair scheduler. It also explains the higher benchmark numbers when running the scheduler in “background services” mode. It’s obviously best for systems that are running a few intensive processes that can benefit from the longer quantum (and, believe it or not, games and pro audio apps run better like this).</p><p>Note that I/O-bound threads (processes waiting on disk, mouse, screen and keyboard I/O) are given priority over CPU-bound threads anyway, which explains why the longer quantum doesn’t harm interactivity much. Try it — have 4 winzip/winrar/7zip sessions running concurrently. You CAN still move your mouse :) Here’s a great primer on internal windows architecture: <a href="http://elqui.dcsc.utfsm.cl/apuntes/guias-free/Windows.pdf">elqui.dcsc.utfsm.cl/apuntes/guias-free/Windows.pdf</a>. Another, deeper dive: <a href="http://download.microsoft.com/download/5/b/3/5b38800c-ba6e-4023-9078-6e9ce2383e65/C06X1116607.pdf">download.microsoft.com/download/5/b/3/5b38800c-ba6e-4023–9078–6e9ce2383e65/C06X1116607.pdf</a>.</p><p>Of course, there are ways to tune the timeslice in a more fine-grained fashion. In the registry, check out <strong>HKLM\SYSTEM\CurrentControlSet\Control\PriorityControl\Win32PrioritySeparation</strong> . Here are some explanations about how it works: <a href="http://www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/29623.mspx?mfr=true">www.microsoft.com/technet/prodtechnol/windows2000serv/reskit/regentry/29623.mspx?mfr=true</a> and <a href="http://www.microsoft.com/mspress/books/sampchap/4354c.aspx">www.microsoft.com/mspress/books/sampchap/4354c.aspx</a> are great.</p><p>For instance — what if you don’t care to increase the quantum on the foreground window but, instead, just want short, fixed quanta (effectively around 60ms) for all processes to improve response time on a system with a lot of processes? Setting Win32PrioritySeparation to 0x28 will take care of that.</p><p>Here’s a useful Win32PrioritySeparation chart from <a href="http://forums.guru3d.com/showthread.php?p=1451631#post1451631">forums.guru3d.com/showthread.php?p=1451631#post1451631</a>:</p><p>2A Hex = Short, Fixed , High foreground boost.<br>29 Hex = Short, Fixed , Medium foreground boost.<br>28 Hex = Short, Fixed , No foreground boost.</p><p>26 Hex = Short, Variable , High foreground boost.<br>25 Hex = Short, Variable , Medium foreground boost.<br>24 Hex = Short, Variable , No foreground boost.</p><p>1A Hex = Long, Fixed, High foreground boost.<br>19 Hex = Long, Fixed, Medium foreground boost.<br>18 Hex = Long, Fixed, No foreground boost.</p><p>16 Hex = Long, Variable, High foreground boost.<br>15 Hex = Long, Variable, Medium foreground boost.<br>14 Hex = Long, Variable, No foreground boost.</p><p>Here are some other pages where others have figured out the effective quanta (and remember the numbers are <em>not</em> in ms): <a href="http://blogs.msdn.com/embedded/archive/2006/03/04/543141.aspx">blogs.msdn.com/embedded/archive/2006/03/04/543141.aspx</a> (for embedded Windows, I have doubts about the accuracy of his calculations regarding the effective quantum but still interesting), <a href="http://www.microsoft.com/technet/sysinternals/information/windows2000quantums.mspx">www.microsoft.com/technet/sysinternals/information/windows2000quantums.mspx</a> (for Windows 2000, probably still valid).</p><p>Here’s a really nice article on the effects of schedulers and I/O-bound processes on virtualization: <a href="http://regions.cmg.org/regions/mcmg/m102006_files/6187_Mark_Friedman_Virtualization.doc">regions.cmg.org/regions/mcmg/m102006_files/6187_Mark_Friedman_Virtualization.doc</a></p><p><strong>Linux, on the other hand</strong>, has not one but several totally different CPU schedulers and I/O elevators available. Just see this page, comparing 2.6.22 with Vista’s kernel, and note how many non-standard features are available as patches: <a href="http://widefox.pbwiki.com/Scheduler">widefox.pbwiki.com/Scheduler</a> . You can get schedulers with cool names such as genetic, anticipatory, etc. Linux used to suffer on the desktop, but with recent patches interactivity has improved tremendously, and is now far more viable as a desktop OS. Here’s some cool info on anticipatory schedulers: <a href="http://www.cs.rice.edu/~ssiyer/r/antsched/">www.cs.rice.edu/~ssiyer/r/antsched/</a>. Anticipatory schedulers can help systems with slower I/O (laptops and desktops, especially) feel more interactive, and was the default I/O elevator for a while (CFQ is the current default for I/O, though can have issues with desktop users, see <a href="http://ubuntuforums.org/showthread.php?t=456692">ubuntuforums.org/showthread.php?t=456692</a>). A list of all the I/O elevators in the kernel: <a href="http://ebergen.net/wordpress/2006/01/26/io-scheduling/">ebergen.net/wordpress/2006/01/26/io-scheduling/</a>. Whitepapers: <a href="http://www.cs.ccu.edu.tw/%7Elhr89/linux-kernel/Linux%20IO%20Schedulers.pdf">www.cs.ccu.edu.tw/%7Elhr89/linux-kernel/Linux%20IO%20Schedulers.pdf</a>, <a href="http://www.linuxinsight.com/files/ols2004/pratt-reprint.pdf">www.linuxinsight.com/files/ols2004/pratt-reprint.pdf</a>, <a href="http://www.linuxinsight.com/files/ols2005/seelam-reprint.pdf">www.linuxinsight.com/files/ols2005/seelam-reprint.pdf</a> .</p><p>Recently, Linux moved to the Completely Fair Scheduler model (<a href="http://www.osnews.com/story.php/18240/Linux-Switches-to-CFS-Scheduler-in-2.6.23">www.osnews.com/story.php/18240/Linux-Switches-to-CFS-Scheduler-in-2.6.23</a>), sparking a lot of controversy (<a href="http://www.osnews.com/story.php/18350/Linus-On-CFS-vs.-SD">www.osnews.com/story.php/18350/Linus-On-CFS-vs.-SD</a>) since it’s not quite done yet (<a href="http://kerneltrap.org/node/14055">kerneltrap.org/node/14055</a>). More info on CFS: <a href="http://immike.net/blog/2007/08/01/what-is-the-completely-fair-scheduler/">immike.net/blog/2007/08/01/what-is-the-completely-fair-scheduler/</a>.</p><p>Interesting benchmarks showing the effects of scheduling on Linux performance: <a href="http://developer.osdl.org/craiger/hackbench/">developer.osdl.org/craiger/hackbench/</a>, <a href="http://math.nmu.edu/~randy/Research/Speaches/Disk%20Scheduling%20In%20Linux.ppt">math.nmu.edu/~randy/Research/Speaches/Disk%20Scheduling%20In%20Linux.ppt</a>.</p><p>For anyone wishing to test the various Linux schedulers’ impact on interactivity, Con Kolivas has something: <a href="http://members.optusnet.com.au/ckolivas/interbench/">members.optusnet.com.au/ckolivas/interbench/</a>. Con’s Staircase/Deadline (SD) scheduler (<a href="http://lwn.net/Articles/224865/">lwn.net/Articles/224865/</a>) didn’t make it to the mainline kernel, unfortunately, and a miffed Con announced he’s dropping out of kernel development. Pity, since I think he single-handedly contributed more to the advancement of Linux interactivity on the desktop than anyone else. It’s great to have the choice of schedulers depending on how you’re planning to use your system — it’s already done with the I/O elevator, let it be done with the CPU scheduler. Instead, Linus invoked his Papal-like powers and made what I consider to be an unsound decision.</p><p>The real issue with Linux though is the userland. Here’s a great paper showing issues with the userland and how it robs us of speed: <a href="https://ols2006.108.redhat.com/reprints/jones-reprint.pdf">ols2006.108.redhat.com/reprints/jones-reprint.pdf</a> . A lot of the CPU and I/O scheduler design is workarounds for those issues. Unless one deliberately chooses a stripped-down Linux distribution, the amount of bloat in the current code is incredible.</p><p>Finally, <strong>Solaris</strong> <strong>10</strong> also comes with a bunch of different schedulers, which you can assign globally or on a per-process/project basis. Tons more info: <a href="http://www.princeton.edu/~unix/Solaris/troubleshoot/schedule.html">www.princeton.edu/~unix/Solaris/troubleshoot/schedule.html</a>, <a href="http://blogs.sun.com/andrei/date/20050131">blogs.sun.com/andrei/date/20050131</a>, <a href="http://wiki.its.queensu.ca/display/JES/Solaris+10+Containers+and+Fair+Share+Scheduling">wiki.its.queensu.ca/display/JES/Solaris+10+Containers+and+Fair+Share+Scheduling</a>, <a href="http://docs.sun.com/app/docs/doc/816-0222/6m6nmlsug?l=en&amp;a=view">docs.sun.com/app/docs/doc/816–0222/6m6nmlsug?l=en&amp;a=view</a>.</p><p>Heady reading, no?</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb5ab02828e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Just how much is your antivirus harming your I/O?]]></title>
            <link>https://medium.com/@dikrek/just-how-much-is-your-antivirus-harming-your-i-o-ea67923a670a?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea67923a670a</guid>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[computers]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[data-storage]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Sun, 29 Jan 2017 03:56:03 GMT</pubDate>
            <atom:updated>2017-01-29T03:56:03.666Z</atom:updated>
            <content:encoded><![CDATA[<p>(note: This article was originally posted on my blog back in ’07 — I’m moving the content to Medium. The principles are valid today, even if the gear listed is old).</p><p>I just got a new corporate laptop, a nice, shiny T60 (OK, it’s IBM black and therefore thoroughly incapable of reflecting on any part of the spectrum).</p><p>I noticed that doing disk-intensive work was much slower than I’ve been used to. I configured it as a server (see previous posts) and that helped a bit but not as much as I’d like to.</p><p>It seems the antivirus software is checking each and every file, and takes 100% of a CPU to do so. Were this not a dual-core box it would be begging for mercy.</p><p>Taking an entire CPU is unacceptable IMO. So I ran some benchmarks — the trusty postmark once more to the rescue:</p><p><strong>After tweaking as a server, antivirus running, 100% CPU utilization while bench running:</strong></p><p>Time:<br>344 seconds total<br>230 seconds of transactions (86 per second)</p><p>Files:<br>20092 created (58 per second)<br>Creation alone: 10000 files (95 per second)<br>Mixed with transactions: 10092 files (43 per second)<br>9935 read (43 per second)<br>10064 appended (43 per second)<br>20092 deleted (58 per second)<br>Deletion alone: 10184 files (1131 per second)<br>Mixed with transactions: 9908 files (43 per second)</p><p>Data:<br>548.25 megabytes read (1.59 megabytes per second)<br>1158.00 megabytes written (3.37 megabytes per second)</p><p><strong>With a more efficient antivirus program instead, variable CPU utilization (from 10%-100%):</strong></p><p>Time:<br>276 seconds total<br>174 seconds of transactions (114 per second)</p><p>Files:<br>20092 created (72 per second)<br>Creation alone: 10000 files (123 per second)<br>Mixed with transactions: 10092 files (58 per second)<br>9935 read (57 per second)<br>10064 appended (57 per second)<br>20092 deleted (72 per second)<br>Deletion alone: 10184 files (484 per second)<br>Mixed with transactions: 9908 files (56 per second)</p><p>Data:<br>548.25 megabytes read (1.99 megabytes per second)<br>1158.00 megabytes written (4.20 megabytes per second)</p><p><strong>Disabling the antivirus makes it way faster for transactions:</strong></p><p>Time:<br>174 seconds total<br>91 seconds of transactions (219 per second)</p><p>Files:<br>20092 created (115 per second)<br>Creation alone: 10000 files (222 per second)<br>Mixed with transactions: 10092 files (110 per second)<br>9935 read (109 per second)<br>10064 appended (110 per second)<br>20092 deleted (115 per second)<br>Deletion alone: 10184 files (268 per second)<br>Mixed with transactions: 9908 files (108 per second)</p><p>Data:<br>548.25 megabytes read (3.15 megabytes per second)<br>1158.00 megabytes written (6.66 megabytes per second)</p><p><strong>Caching with UpTempo for a nice 50% boost in performance:</strong></p><p>Time:<br>121 seconds total<br>65 seconds of transactions (307 per second)</p><p>Files:<br>20092 created (166 per second)<br>Creation alone: 10000 files (277 per second)<br>Mixed with transactions: 10092 files (155 per second)<br>9935 read (152 per second)<br>10064 appended (154 per second)<br>20092 deleted (166 per second)<br>Deletion alone: 10184 files (509 per second)<br>Mixed with transactions: 9908 files (152 per second)</p><p>Data:<br>548.25 megabytes read (4.53 megabytes per second)<br>1158.00 megabytes written (9.57 megabytes per second)</p><p>Not tweaking the laptop as a server resulted in &gt; 400s runtimes in the default config (sometimes 500s). FYI, the drive is a smaller, 5400 RPM jobbie, not the 200GB 7200 RPM SATA I have my eye on.</p><p>One could extrapolate these results. On a bigger box the end results will differ but everything will remain relatively similar.</p><p>Obviously, antivirus is sorely needed in this day and age, but if you’re planning on doing heavy I/O be careful what antivirus program you pick and how it’s configured. Depending on the server, I’d gladly trade some protection in exchange for a bunch more performance. Or you can go Unix/Linux and not really have to bother.</p><p>I’d say setting up an antivirus program to only scan extensions that can be infected and only scan on creates/modifies and not reads, can boost performance significantly.</p><p>Interestingly, caching didn’t help much with antivirus enabled — most of the bottleneck was the antivirus since everything had to go through it first. What if this was a database/email/fileserver with heavy activity?</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea67923a670a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Progress needs Trailblazers]]></title>
            <link>https://medium.com/recoverymonkey/progress-needs-trailblazers-39e8999eb982?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/39e8999eb982</guid>
            <category><![CDATA[progress]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Sun, 29 Jan 2017 03:34:59 GMT</pubDate>
            <atom:updated>2017-01-29T03:34:59.517Z</atom:updated>
            <content:encoded><![CDATA[<p>I got the idea for this stream-of-consciousness (and possibly Captain Obvious) post after reading some comments regarding new ways to do high speed I/O. Not something boring like faster media or protocols, but rather far more exotic approaches that require <em>drastically</em> rewriting applications to get the maximum benefits of radically different architectures.</p><p>The comments, in essence, stated that such advancements would be an <em>utter waste of time</em> since 99% of customers have absolutely no need for such exotica, but rather just need cheap, easy, reliable stuff, and can barely afford to buy infrastructure to begin with, let alone custom-code applications to get every last microsecond of latency out of gear.</p><h3>Why Trailblazers Matter</h3><p>It’s an interesting balancing game: There’s a group of people that want progress and innovation at all costs, and another group of people that think the status quo is OK, or maybe just give me cheaper/easier/more reliable and I’m happy.</p><p>Both groups are needed, like Yin and Yang.</p><p>The risk-averse are needed because without them it would all devolve into chaos.</p><p>But without people willing to take risks and do seemingly crazy and expensive things, progress would simply be much harder.</p><p>Imagine if the people that prevailed were the ones that thought trepanning and the balancing of humours was state-of-the-art medicine and needed no further advancements. Perchance the invention of MRI machines might have taken longer?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*E9Us55EkbeLJDh6u.jpg" /></figure><p>What if traditional planes were deemed sufficient, would a daring endeavor like the SR-71 have been undertaken?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/0*X8BFDux5LgErRfad.jpg" /></figure><h3>Advanced Tech Filters Down to the Masses</h3><p>It’s interesting how the crazy, expensive and “unnecessary” eventually manages to become part of everyday life. Think of such commonplace technologies like:</p><ul><li>ABS brakes</li><li>SSDs</li><li>4K TVs</li><li>High speed networking</li><li>Smartphones</li><li>Virtualization</li><li>Having more than 640KB of RAM in a PC</li></ul><p>They all initially cost a ton of money and/or were impractical for everyday use (the first ABS brakes were in planes!)</p><h3>What is Impractical or Expensive Now Will be Normal Tomorrow…</h3><p>…but not every new tech will make it. <em>Which is perfectly normal</em>.</p><ul><li>Rewriting apps to take advantage of things like containers, microservices and fast object storage? It will slowly happen. <em>Most</em> customers can simply wait for the app vendors to do it.</li><li>In-memory databases? Not a niche any more, when even the ubiquitous SQL Server is doing it…</li><li>Using advanced and crazy fast byte-addressable NVDIMM in storage solutions? Some mainstream vendors like Nimble and Microsoft are already doing it. Others are working on some really interesting implementations.</li><li>AI, predictive analytics, machine learning? Already happening and providing huge value (for example Nimble’s InfoSight).</li></ul><h3>Don’t Belittle the Risk Takers!</h3><p>Even if you think the crazy stuff some people are working on isn’t a fit for you, don’t belittle it. It can be challenging to hold your anger, I understand, especially when the people working on crazy stuff are pushing their viewpoint as if it will change the world.</p><p>Because you never know, it just might.</p><p>If you don’t agree — get out of the way, or, even better, help find a <em>better</em> way.</p><h3>Just Because it’s New to You Doesn’t Mean it’s Really New, or Risky</h3><p>Beware of turning into an IT Luddite.</p><p>Just because <em>you</em> aren’t using a certain technology, but many <em>others</em> are, and reaping far more benefits than you with your current tech, then maybe they’re onto something… :)</p><p><em>At what point does your resistance to change become a competitive disadvantage?</em></p><h3>Stagnation is Death</h3><p>…or, at a minimum, boring. Τα πάντα ρει, as Heraclitus said. I, for one, am looking forward to the amazing innovations coming our way, and actively participate in trying to make them come to fruition faster.</p><p>What will you do?</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39e8999eb982" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/progress-needs-trailblazers-39e8999eb982">Progress needs Trailblazers</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Well-Behaved Storage System: Automatic Noisy Neighbor Avoidance]]></title>
            <link>https://medium.com/recoverymonkey/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance-329388ddbb94?source=rss-7a980735fd77------2</link>
            <guid isPermaLink="false">https://medium.com/p/329388ddbb94</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[it-exotica]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Fri, 08 Apr 2016 08:57:13 GMT</pubDate>
            <atom:updated>2017-01-30T02:51:24.706Z</atom:updated>
            <content:encoded><![CDATA[<p>This topic is very near and dear to me, and is one of the big reasons I came over to Nimble Storage.</p><p>I’ve always believed that <strong>storage systems should behave gracefully and predictably under pressure. Automatically. Even under complex and difficult situations</strong>.</p><p>It sounds like a simple request and it makes a whole lot of sense, but <em>very</em> few storage systems out there actually behave this way. This creates business challenges and increases risk and OpEx.</p><h4>The Problem</h4><p>The simplest way to state the problem is that <strong>most storage systems can enter conditions where workloads can suffer from unfair and abrupt performance starvation under several circumstances.</strong></p><p>OK, maybe that wasn’t the simplest way.</p><p>Consider the following scenarios:</p><ol><li>A huge sequential I/O job (backup, analytics, data loads etc.) happening in the middle of latency-sensitive transaction processing</li><li>Heavy array-generated workloads (garbage collection, post-process dedupe, replication, big snapshot deletions etc.) happening at the same time as user I/O</li><li>Failed drives</li><li>Controller failover (due to an actual problem or simply a software update)</li></ol><p>#3 and #4 are more obvious — a well-behaved system will ensure high performance even during a drive failure (or three), and after a controller fails over. For instance, if total system headroom is <em>automatically</em> kept at 50% for a dual-controller system (or, simplistically, <em>100/n</em>, where <em>n</em> is the controller count for shared-everything architectures), even after a controller fails, performance should be fine.</p><p>#1 and #2 are a bit more complicated to deal with. Let’s look at this in more detail.</p><h4>The Case of Competing Workloads During Hard Times</h4><p>Inside every array, at any given moment, a balancing act occurs. Multiple things need to happen simultaneously.</p><p><strong>Several user-generated workloads</strong>, for instance:</p><ul><li>DB</li><li>VDI</li><li>File Services</li><li>Analytics</li></ul><p><strong>Various internal array processes</strong> — they <em>also</em> are workloads, just array-generated, and often critical:</p><ul><li>Data reduction (dedupe, compression)</li><li>Cleanup (object deletion, garbage collection)</li><li>Data protection (integrity-related)</li><li>Backups (snaps, replication)</li></ul><p>If the system has enough headroom, all these things will happen without performance problems.</p><p>If the system <em>runs out of headroom</em>, that’s where most arrays have challenges with <em>prioritizing</em> what happens when.</p><p>The most common way a system may run out of headroom is the sudden appearance of a hostile “bully” workload. This is also called a “noisy neighbor”. Here’s an example of system behavior in the presence of a bully workload:</p><figure><img alt="bully_vs_victim_2" src="https://cdn-images-1.medium.com/max/1024/0*tA7jxiRUpq020JYj.png" /></figure><p>In this example, the latency-sensitive workload will greatly and unfairly suffer after the “noisy neighbor” suddenly appears. If the latency-sensitive workload is a mission-critical application, this could cause a <em>serious</em> business problem (slow processing of financial transactions, for instance).</p><p>This is an <em>extremely</em> common scenario. A lot of the time it’s not even a new workload. Often, an <em>existing</em> workload changes behavior (possibly due to an application change — for instance a patch or a modified SQL query). <em>This stuff happens</em>.</p><h4>How some vendors have tried to fix the issue with Manual “QoS”</h4><p>As always, there is more than one way to skin a cat, if one is so inclined. Here are a couple of manual methods to fix workload contention:</p><ul><li>Some arrays have a simple IOPS or throughput <em>limit</em> that an administrator can <em>manually</em> adjust in order to fix a performance problem. This is an iterative and <em>reactive</em> method and hard to automate properly in real time. In addition, if the issue was caused by an <em>internal</em> array-generated workload, there is often no tooling available to throttle those processes.</li><li>Other arrays insist on the user setting up minimum, maximum and burst IOPS values <em>for every single volume in the system</em>, upon volume creation. This assumes the user knows <em>in advance</em> what performance envelope is required, in detail, <em>per volume</em>. The reality is that almost nobody knows these things beforehand, and getting the numbers wrong can itself cause a huge problem with latencies. Most people just want to get on with their lives and have their stuff work without babysitting.</li></ul><p><em>Manual</em> mechanisms for fixing the “bully” workload challenge result in systems that are hard to consume and complex to support while under performance pressure. Moreover, when a performance issue occurs, speed of resolution is <em>critical</em>. The issue needs to be resolved <em>immediately, </em>especially for latency-sensitive workloads. <em>Manual methods will simply not be fast enough</em>. Business <em>will</em> be impacted.</p><h4>How Nimble Storage Fixed the Noisy Neighbor Issue</h4><p>No cats were harmed in the process. Nimble engineers looked at the extensive telemetry in <a href="http://info.nimblestorage.com/rs/nimblestorage/images/nimblestorage-ds-infosight.pdf">InfoSight</a>, used data science, and neatly identified areas that could be massively automated in order to optimize system behavior under a wide variety of adverse conditions. Some of what was done:</p><ul><li>Highly advanced Fair Share disk scheduling (separate mechanisms that deal with different scenarios)</li><li>Fair Share CPU scheduling</li><li>Dynamic Weight Adjustment — automatically adjust priorities in various ways under different resource contention conditions, so that the system can always complete critical tasks and not fall dangerously behind</li></ul><p>The end result is a system that:</p><ul><li>Lets system latency increase gracefully and progressively as load increases</li><li>Carefully and <em>automatically</em> balances user and system workloads</li><li>Achieves I/O deadlines and preemption behavior</li><li>Eliminates the Noisy Neighbor problem without the need for <em>any</em> manual QoS adjustments</li><li>Allows latency-sensitive small-block I/O to proceed without interference from bully workloads</li></ul><p>Such automation achieves a better<em> business result</em>: less risk, less OpEx, easy supportability, simple and safe overall consumption even under difficult conditions.</p><h4>What Should Nimble Customers do to get this Capability?</h4><p>As is typical with Nimble systems and their impressive <a href="http://recoverymonkey.org/2016/03/28/going-green-why-i-joined-nimble-storage/">Ease of Consumption</a>, nothing fancy needs to be done apart from simply upgrading to a specific release of the code (in this case 3.1 and up — 2.3 did some of the magic but 3.1 is the fully realized vision).</p><p>A bit anticlimactic, apologies… if you like complexity, watching <a href="https://www.youtube.com/watch?v=qybUFnY7Y8w">this</a> instead is probably more fun than juggling QoS manually.</p><p>D</p><p>Technorati Tags: <a href="http://technorati.com/tag/Nimble">Nimble</a>, <a href="http://technorati.com/tag/Performance">Performance</a>, <a href="http://technorati.com/tag/QoS">QoS</a>, <a href="http://technorati.com/tag/Storage">Storage</a>, <a href="http://technorati.com/tag/Automation">Automation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=329388ddbb94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance-329388ddbb94">The Well-Behaved Storage System: Automatic Noisy Neighbor Avoidance</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>