<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[RecoveryMonkey - Medium]]></title>
        <description><![CDATA[Musings on storage, tuning, backups and more — delivered with a no-nonsense approach - Medium]]></description>
        <link>https://medium.com/recoverymonkey?source=rss----cfac87310593---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>RecoveryMonkey - Medium</title>
            <link>https://medium.com/recoverymonkey?source=rss----cfac87310593---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:58:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/recoverymonkey" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Nimble Cloud Volumes Removing Risk From Cloud Storage]]></title>
            <link>https://medium.com/recoverymonkey/nimble-cloud-volumes-removing-risk-from-cloud-storage-edfd5daca17?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/edfd5daca17</guid>
            <category><![CDATA[private-cloud]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 27 Feb 2017 17:03:59 GMT</pubDate>
            <atom:updated>2017-02-28T01:59:06.515Z</atom:updated>
            <content:encoded><![CDATA[<p>As of February 27th 2017, Nimble Storage announced Nimble Cloud Volumes (NCV), marking Nimble’s entry into the cloud space.</p><p>For those short on time: Nimble Cloud Volumes is <em>block</em> storage as a service, works with compute from AWS &amp; Azure, and avoids the significant block storage drawbacks of those two providers. In essence, it is <em>enterprise</em> storage as a service, at a competitive price, while retaining all the cloud conveniences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*yGrPCGae5kkmZgFS.png" /><figcaption>NCV works with AWS and Azure</figcaption></figure><h3>Why get in the Storage as a Service space?</h3><p>People like subscription-based cloud offerings for various reasons. Mostly due to convenience, agility and economies of scale. Plus, of course, converting all expenses into OPEX. But this is not a treatise on “why cloud”.</p><p>Currently, the two largest hyperscale cloud providers are AWS and Azure. While both are great offerings overall, their block storage component is <em>severely</em> lacking in several key aspects compared to enterprise-grade storage.</p><p>This is something that most people either are unaware of, or simply choose to accept the risk since they see the other benefits as compelling.</p><p>Knowledge is power.</p><h3>Here are some of the downsides of AWS EBS and Azure block storage:</h3><ul><li>Availability SLA: 3 nines (if an enterprise array vendor mentioned 3 nines, you’d kick them out immediately)</li><li>Data integrity: 0.1–0.2% Annualized Failure Rate (AFR). 1–2 out of 1,000 volumes will suffer from partial or complete data loss (this is for EBS, Azure numbers N/A). If an enterprise storage vendor said they’d lose your data with a 0.2% chance per year, you’d permanently and with extreme prejudice put them on the “banned” list…</li><li>Extremely slow (and costly) backups, clones and recoveries (for instance, many hours just to take a snapshot! And with high I/O performance variability while data is shuffled back and forth from S3)</li><li>Hard and expensive to switch providers (by design)</li><li>Hard and expensive to repatriate data back to a private cloud (by design)</li><li>Significant constraints in some services might force customers to change their application layouts (1TB volume limits in Azure are unacceptably small for many deployments)</li><li>Data mobility between cloud providers isn’t easy (by design)</li><li>Troubleshooting &amp; monitoring are hard. How does one do Root Cause Analysis for issues within a cloud provider? They’re designed to be black boxes. What happens when something goes wrong? How long does troubleshooting take? Can I see detailed volume statistics and trending? <em>Why</em> is my application suffering?</li></ul><p>Especially for the more critical applications, the above list is enough to give people pause.</p><p>Well — what if there <em>was</em> a better way? A way to significantly de-risk using cloud storage?<strong> This is the reason Nimble is entering this market. To provide a way to remove some of the more significant challenges of cloud storage while keeping the best aspects of what people like about the cloud.</strong></p><h3>How did Nimble Storage build the NCV STaaS offering?</h3><p>Nimble has built its own cloud to deliver the Nimble Cloud Volumes service.</p><p>The Nimble cloud is strategically placed around the world in locations in close proximity to AWS and Azure cloud infrastructure, ensuring low-latency access. The Nimble cloud is built using Nimble flash technology for storing data (i.e. it does not use the storage from either AWS or Azure). The service is managed through a simple portal and provides storage volumes that can be attached to Amazon EC2 or Azure Virtual Machine instances.</p><h3>The Benefits Customers get out of Nimble Cloud Volumes</h3><p>As previously mentioned, the entire goal of NCV is to keep what’s good about the cloud while at the same time removing some of the more significant risks and inconveniences. Here are the main benefits:</p><h3>Easy to Consume</h3><p>NCV is managed completely through an easy portal that lets users choose a capacity, performance and backup SLA, and attach to either AWS and/or Azure.</p><h3>Fully Managed STaaS Offering</h3><p>This is what customers expect out of the cloud. NCV is a fully managed service. The customer doesn’t have to ever touch and maintain a storage array, physical or virtualized.</p><h3>Extreme Reliability</h3><p>The NCV storage uptime is the same as what Nimble enterprise arrays provide: a measured 6+ nines.</p><p>The data integrity is even more impressive at millions of times more than what native cloud block storage provides. More details <a href="http://recoverymonkey.org/2016/12/01/uncompromising-resiliency/">here</a>.</p><h3>Enterprise Copy Data Management</h3><p><em>Instant</em> snapshots and clones. <em>Instant</em> recoveries. In addition, we only charge for delta usage when making clones, <em>not</em> full copies. Which means you can take many, many clones of something and the initial cost is almost zero. Only pay for the IOPS and additional data you consume as the clones deviate from the original. At scale, the savings can be significant.</p><h3>Provisioned IOPS, Low Latency plus Auto QoS</h3><p>Everyone expects an IOPS SLA from a STaaS offering. We do of course offer an IOPS SLA with NCV.</p><p>But we also take it to the next level by automatically prioritizing the <em>right</em> I/O using heuristics. That way, even if you are hitting your QoS limit, the right kind of I/O will still get prioritized automatically.</p><p>For instance, <em>automatically</em> prioritize latency-sensitive DB I/O vs latency-insensitive sequential I/O. Both because it makes sense and because nobody has the time to deal with micromanaging QoS. More <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">here</a>.</p><h3>Easy Mobility Eliminates Cloud Lock-In</h3><p>This is a big one. We make the following things easy and possible:</p><ol><li><strong>Cloud on-off ramp</strong>: easily and quickly send your data to the cloud. Want to stop using the cloud? That’s also easy and fast. And there are no egress charges.</li><li><strong>Multicloud without lock-in</strong>: easily use multiple public cloud offerings (and switch between them) without needing to migrate data.</li><li><strong>Hybrid Cloud Portability</strong>: Easy to mix and match private and public cloud. Easily move data back to on-premises private cloud if needed. Easily burst what you need to the cloud.</li></ol><h3>Global Visibility with Predictive Analytics</h3><p>Last, but by no means least: We are extending <a href="https://www.nimblestorage.com/technology-products/infosight/">InfoSight</a> to the cloud. The service that massively lowers business risk for Nimble private cloud customers will now also be available to public cloud consumers.</p><ul><li><strong>Predict and prevent issues across the stack</strong>: Because it’s really about the <em>applications</em>, not just the storage. Pinpoint difficult issues quickly, even if they have nothing whatsoever to do with storage.</li><li><strong>Visibility no matter where your data is</strong>: It shouldn’t matter whether your data is in the public or private cloud or both. Have detailed visibility into your data including exotic things like I/O histograms or more pedestrian (but still useful) things like detailed trending analysis.</li><li><strong>Predict, recommend, optimize</strong>: Use Predictive Analytics to go beyond reporting and use the information to optimize data placement — which could end up saving you money.</li></ul><p>Finally, the cloud stops being a black box when problems arise.</p><h3>In Summary</h3><p>The Nimble Cloud Volumes service was built to address some significant shortcomings AWS and Azure block storage services have, while retaining the benefits people expect from the cloud and providing much more flexibility to customers.</p><p>NCV provide benefits such as increased reliability, predictability, faster backups and clones, plus integration with the best infrastructure analytics platform, InfoSight.</p><h3>Some example methods of deployment:</h3><ul><li>Production in private cloud, DR in public cloud</li><li>Test/Dev in public cloud</li><li>“All In” public cloud</li></ul><p>We are accepting beta customers now (several big names already using it). Go <a href="https://www.nimblestorage.com/cloud/">here</a> to take a look and sign up.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edfd5daca17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/nimble-cloud-volumes-removing-risk-from-cloud-storage-edfd5daca17">Nimble Cloud Volumes Removing Risk From Cloud Storage</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Considerations for Implementing NVMe Storage]]></title>
            <link>https://medium.com/recoverymonkey/practical-considerations-for-implementing-nvme-storage-b5c30c32fd4?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/b5c30c32fd4</guid>
            <category><![CDATA[low-latency]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[nimble-storage]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[nvme]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:18:02 GMT</pubDate>
            <atom:updated>2017-01-30T03:18:01.696Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we begin, something needs to be clear: Although dual-ported NVMe drives are not yet cost effective, the architecture of Nimble Storage is NVMe-ready <em>today</em>. And always remember that in order to get good benefits from NVMe, one needs to implement it all the way from the client. Doing NVMe only at the array isn’t as effective.</p><p>In addition, Nimble <em>already</em> uses technology far faster than NVMe: Our write buffers use byte-addressable NVDIMM-N, instead of slower NVRAM HBAs or NVMe drives that other vendors use. Think about it: I/O happens at DDR4 RAM speeds, which makes even the fastest NVMe drive seem positively glacial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JNubCIgKILzPIe9N.png" /></figure><p>I did want to share my personal viewpoint of where storage technology in general may be headed if NVMe is to be mass-adopted in a realistic fashion and without making huge sacrifices.</p><h3>About NVMe</h3><p>Lately, a lot of noise is being made about NVMe technology. The idea being that NVMe will be the next step in storage technology evolution. And, as is the natural order of things, new vendors are popping up to take advantage of this perceived opening.</p><p>For the uninitiated: NVMe is a relatively new standard that was created specifically for devices connected over a PCI bus. It has certain nice advantages vs SCSI such as reduced latency and improved IOPS. Sequential throughput can be significantly higher. It can be more CPU-efficient. It needs a small and simple driver, the standard requires only 13 commands, and it can also be used over some FC or Ethernet networks (NVMe over Fabrics). Going through a fabric only adds a small amount of extra latency to the stack compared to DAS.</p><p>NVMe is strictly an optimized block protocol, and not applicable to NAS/object platforms unless one is talking about their internal drives.</p><p>Due to the additional performance, NVMe drives are a no brainer in systems like laptops and DASD/internal to servers. Usually there is only a small number (often just one device) and no fancy data services are running on something like a laptop… replacing the media with better media+interface is a good idea.</p><p>For enterprise arrays though, the considerations are different.</p><h3>NVMe Performance</h3><p>Marketing has managed to confuse people regarding NVMe’s true performance. It’s important to note that tests illustrating NVMe performance show a <em>single</em> NVMe device being faster than a <em>single</em> SAS or SATA SSD. But storage arrays usually don’t have a single device and so drive performance isn’t the bottleneck as it is with low media count systems.</p><p>In addition, most tests and research papers comparing NVMe to other technologies use <em>wildly</em> dissimilar SSD models. For instance, pitting a modern, ultra-high-end NVMe SSD against an older consumer SATA SSD with a <em>totally</em> different internal controller. This can make proper performance comparisons difficult. <em>How much of the performance boost is due to NVMe and how much because the expensive, fancy SSD is just a much better engineered device?</em></p><p>For instance, consider this chart of NVMe device latency, courtesy of Intel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uv7of5J_8xIGVywa.png" /></figure><p>As you can see, regarding latency, <strong>NVMe as a drive connection protocol <em>will</em> offer better latency than SAS or SATA but the difference is in the order of a few microseconds</strong>. The protocol differences become truly important only with next gen technologies like 3D Xpoint, which ideally needs a memory interconnect to shine (or, at a minimum, PCI) since the media is <em>so</em>much faster than the usual NAND. But such media will be prohibitively expensive to be used as the entire storage within an array in the foreseeable future, and would quickly be bottlenecked by the array CPUs at scale.</p><h3>NVMe over Fabrics</h3><p>Additional latency savings will come from connecting clients using NVMe over Fabrics. By doing I/O over an RDMA network, a latency reduction of around 100 microseconds is possible versus encapsulated SCSI protocols like iSCSI, assuming all the right gear is in place (HBAs, switches, host drivers). Doing NVMe at the client side also helps with lowering CPU utilization, which can make client processing overall more efficient.</p><h3>Where are the Bottlenecks?</h3><p>The reality is that the main bottleneck in today’s leading modern AFAs is the controller itself and <em>not</em>the SSDs (simply because there is enough performance in just a couple of dozen modern SAS/SATA SSDs to saturate most systems). Moving to competent NVMe SSDs will mean that those same controllers will now be saturated by maybe 10 NVMe SSDs. For example, a single NVMe drive may be able to read sequentially at 3GB/s, whereas a single SATA drive 500MB/s. Putting 24 NVMe drives in the controller doesn’t mean that magically the controller will now deliver 72GB/s. In the same way, a single SATA SSD might be able to do 100000 read small block random IOPS and an NVMe with better innards 400000 IOPS. Again, it doesn’t mean that same controller with 24 devices will all of a sudden now do 9.6 million IOPS!</p><h3>How Tech is Adopted</h3><p>Tech adoption comes in waves until a significant technology advancement is affordable and reliable enough to become pervasive. For instance, ABS brakes were first used in planes in 1929 and were too expensive and cumbersome to use in everyday cars. Today, most cars have ABS brakes and we take for granted the added safety they offer.</p><p>But consider this: What if someone told you that in order to get a <em>new</em> kind of car (that has several <em>great</em> benefits) you would have to utterly give up things like airbags, ABS brakes, all-wheel-drive, traction control, limited-slip differential? <em>Without</em> an equivalent replacement for these functions?</p><p>You would probably realize that you’re not <em>that</em> excited about the new car after all, no matter how much better than your existing car it might be in other key aspects.</p><p>Storage arrays follow a similar paradigm. There are several very important business reasons that make people ask for things like HA, very strong RAID, multi-level checksums, encryption, compression, data reduction, replication, snaps, clones, hot firmware updates. Or the ability to dynamically scale a system. Or comprehensive cross-stack analytics and automatic problem prevention.</p><p>Such features evolved over a long period of time, and help mitigate risk and accelerate business outcomes. They’re also <em>not</em>trivial to implement properly.</p><h3>NVMe Arrays Today</h3><p><strong>The challenge I see with the current crop of ultra-fast NVMe over Fabrics arrays is that they’re <em>so</em> focused on speed that they ignore the aforementioned enterprise features in lieu of sheer performance.</strong> I get it: it takes great skill, time and effort to <em>reliably</em> implement such features, especially in a way that they don’t strip the performance potential of a system.</p><p>There is also a significant cost challenge in order to <em>safely</em> utilize NVMe media <em>en masse</em>. Dual-ported SSDs are crucial in order to deliver proper HA. Current dual-ported NVMe SSDs tend to be very expensive per TB vs current SAS/SATA SSDs. In addition, due to the much higher speed of the NVMe interface, even with future CPUs that include FPGAs, <em>many</em> CPUs and PCI switches are needed to create a highly scalable system that can fully utilize such SSDs (and maintain enterprise features), which further explains why most NVMe solutions using the more interesting devices tend to be rather limited.</p><p>There are also client-side challenges: Using NVMe over Fabrics can often mean purchasing new HBAs and switches, plus dealing with some compromises. For instance, in the case of <a href="https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet">RoCE</a>, <a href="https://en.wikipedia.org/wiki/Data_center_bridging">DCB switches</a> are necessary, end-to-end congestion management is a challenge, and routability is not there until v2.</p><p>There’s a bright side: There actually exist some very practical ways to give customers the benefits of NVMe <em>without</em>taking away business-critical capabilities.</p><h3>Realistic Paths to NVMe Adoption</h3><p>We can divide the solution into two pieces, the direction chosen will then depend on customer readiness and component availability. All the following assumes <em>no</em> loss of important enterprise functionality (as we discussed, giving up on all the enterprise functionality is the easy way out when it comes to speed):</p><p><strong>Scenario 1: Most customers are <em>not</em> ready to adopt host-side NVMe connectivity:</strong></p><p>If this is the case, a good option would be to have something like a fast byte-addressable ultra-fast device inside the controller to massively augment the RAM buffers (like 3D Xpoint in a DIMM), or, if not available, some next-gen NVMe drives to act as cache. That would provide an overall speed boost to the clients and not need <em>any</em> client-side modifications. This approach would be the most friendly to an existing infrastructure (and a relatively economical enhancement for arrays) without needing all internal drives to be NVMe nor extensive array modifications.</p><p>You see, part of any competent array’s job is using intelligence to <em>hide</em> any underlying media issues from the end user. A good example: even super-fast SSDs can suffer from garbage collection latency incidents. A good system will smooth out the user experience so users won’t see extreme latency spikes. The chosen media and host interface are immaterial for this, but I bet if you were used to 100μs latencies and they suddenly spiked to 10ms for a while, it would be a bad day. Having an extra-large buffer in the array would help do this more easily, yet not need customers to change anything host-side.</p><p>An evolutionary second option would be to change <em>all</em> internal drives to NVMe, <em>but to make this practical would require wide availability of cost-effective dual-ported devices</em>. Note that with low SSD counts (less than 12) this would provide speed benefits even if the customer doesn’t adopt a host-side NVMe interface, but it will be a diminishing returns endeavor at larger scale, unless the controllers are significantly modified.</p><p><strong>Scenario 2: Large numbers of customers <em>are</em> ready and willing to adopt NVMe over Fabrics.</strong></p><p>In this case, the <em>first</em> thing that needs to change is the array connectivity to the outside world. That alone will boost speeds on modern systems even without major modifications. Of course, this will often mean client and networking changes to be most effective, and often such changes can be costly.</p><p>The next step depends on the availability of cost-effective dual-ported NVMe devices. But in order for <em>very</em>large performance benefits to be realized, pretty big boosts to CPU and PCI switch counts may be necessary, necessitating bigger changes to storage systems (and increased costs).</p><h3>Architecture Matters</h3><p>In the quest for ultra-low latency and high throughput without sacrificing enterprise features (yet remaining reasonably cost-effective), overall architecture becomes extremely important.</p><p>For instance, how will one do RAID? Even with NVMe over Fabrics, approaches like erasure coding and triple mirroring can be costly from an infrastructure perspective. Erasure coding remains CPU-hungry (even more so when trying to hit ultra-low latencies), and triple mirroring across an RDMA fabric would mean massive extra traffic on that fabric.</p><p>Localized CPU:RAID domains remain more efficient, and mechanisms such as Nimble NCM can fairly distribute the load across multiple storage nodes without relying on a cluster network for heavy I/O. This technology is available today.</p><h3>Next Steps</h3><p>In summary, I urge customers to carefully consider the overall business impact of their storage making decisions, especially when it comes to new technologies and protocols. Understand the true benefits first. Carefully balance risk with desired outcome, and consider the <em>overall</em> system and not just the components. Of course, one needs to understand the risks vs rewards first, hence this article. Just make sure that, in order to achieve a certain ideal, you don’t give up on <em>critical</em> functionality that you’ve been taking for granted.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5c30c32fd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/practical-considerations-for-implementing-nvme-storage-b5c30c32fd4">Practical Considerations for Implementing NVMe Storage</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Uncompromising Resiliency]]></title>
            <link>https://medium.com/recoverymonkey/uncompromising-resiliency-db47105c35dd?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/db47105c35dd</guid>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[nimble-storage]]></category>
            <category><![CDATA[checksums]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[reliability]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:15:40 GMT</pubDate>
            <atom:updated>2017-01-30T03:15:39.620Z</atom:updated>
            <content:encoded><![CDATA[<p>(cross-posted at <a href="https://www.nimblestorage.com/blog/uncompromising-resiliency/">https://www.nimblestorage.com/blog/uncompromising-resiliency/</a>)</p><p>The cardinal rule for enterprise storage systems is to never compromise when it comes to data integrity and resiliency. Everything else, while important, is secondary.</p><p>Many storage consumers are not aware of what data integrity mechanisms are available or which ones are necessary to meet their protection expectations and requirements. It doesn’t help that a lot of the technologies and the errors they prevent are rather esoteric. However, if you want a storage system that safely stores your data and always returns it correctly, no measure is too extreme.</p><h3>The Golden Rules of Storage Engineering</h3><p>When architecting enterprise storage systems, there are three Golden Rules to follow.</p><p>In order of criticality:</p><ol><li><strong>Integrity</strong>: Don’t ever return incorrect data</li><li><strong>Durability</strong>: Don’t ever lose any data</li><li><strong>Availability</strong>: Don’t ever lose access to data</li></ol><p>To better understand the order, ask yourself, “what is preferred, temporary loss of <em>access to data </em>or the storage system returning the <em>wrong</em> data <em>without anyone even knowing it’s wrong</em>?”</p><p>Imagine life or death situations, where the wrong piece of information could have <em>catastrophic</em> consequences. Interestingly, vendors exist that focus a lot on Availability (even offering uptime “guarantees”) but are lacking in Integrity and Durability. Being able to access the array but have data corruption is almost entirely useless. Consider modern storage arrays with data deduplication and/or multi-petabyte storage pools. The effects are far more severe now that a single block represents the data for 1–100+ blocks and data is spread across 10’s — 100’s of drives instead of a few drives.</p><h3>The Nimble Storage Approach</h3><p>Nimble Storage has taken a multi-stage approach to satisfy the Golden Rules, and in some cases, the amount of protection offered verges on being paranoid (but the good kind of paranoid).</p><p>Simply, Nimble employs these mechanisms:</p><ol><li><strong>Integrity</strong>: Comprehensive multi-level checksums</li><li><strong>Durability</strong>: Hardened RAID protection and resilience upon power loss</li><li><strong>Availability</strong>: Redundant hardware coupled with predictive analytics</li></ol><p>We will primarily focus on the first two as they are often glossed over, assumed, or not well understood. Availability will be discussed in a separate blog, however it is good to mention a few details here.</p><p>To start, Nimble has achieved greater than 99.9997% measured uptime since 2014. This is measured across more than 9,000 customers using multiple generations of hardware and software. A key aspect of Nimble’s availability comes from InfoSight which continually improves and learns as more systems are used. Each week, trillions of data points are analyzed and processed with the goal of predicting and preventing issues, not just in the array, but across the entire infrastructure. 86% of issues are detected and automatically resolved before the customer is even aware of the problem. To further enhance this capability, Nimble’s Technical Support Engineers can resolve issues faster as they have all the data available when an issue arises. This bypasses the hours-days-weeks often required to collect data, send to support, analyze, repeat — until a solution can be found.</p><h3>Data Integrity Mechanisms in Detail</h3><p>The goal is simple: What is read must always match what was written. And, if it doesn’t, we fix it on the fly.</p><p>What many people don’t realize is there are occasions where storage media will lose a write, corrupt it or place it at the wrong location on the media. RAID (including 3-way mirroring) or Erasure Coding are <em>not</em> enough to protect against such issues. The older T10 PI employed by some systems is also not enough to protect against all eventualities.</p><p>The solution involves using checksums which get more computationally intensive the more paranoid one is. As checksums are computationally intensive, certain vendors don’t employ or minimally employ them to gain more performance or faster time to market. Unfortunately, the trade-off can lead to data corruption.</p><p>Broadly, Nimble creates a checksum <em>and</em> a “self-ID” for each piece of data. The checksum protects against data corruption. The self-ID protects against lost/misplaced writes and misdirected reads (incredible as it may seem, these things happen enough to warrant this level of protection).</p><p>For instance, if the written data has a checksum, and <em>corruption</em> occurs, when the data is read and checksummed <em>again</em>, the checksums will not match. However, if instead the data was <em>placed at an incorrect location on the media</em>, the checksums will match, but the self-IDs will <em>not</em> match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*eYx3Ytn41B0zVvZO.png" /></figure><p><strong>Where it gets interesting:</strong></p><p>Nimble doesn’t just do block-level checksums/IDs. These multi-level checksums are also performed:</p><ol><li>Per segment in each write stripe</li><li>Per block, before <em>and</em> after compression</li><li>Per snapshot (including all internal housekeeping snaps)</li><li>For replication</li><li>For all data movement within a cluster</li><li>All data and metadata in NVRAM</li></ol><p>This way, every likely data corruption event is covered, including metadata consistency and replication issues, which are often overlooked.</p><h3>Durability Mechanisms in Detail</h3><p>There are two kinds of data on a storage system and both need to be protected:</p><ol><li>Data in flight</li><li>Data on persistent storage</li></ol><p>One may differentiate between user data and metadata but we protect both with equal paranoid fervor. Some systems try to accelerate operations by not protecting metadata sufficiently, which greatly increases risk. <em>This is especially true with deduplicating systems, where metadata corruption can mean losing everything!</em></p><p><strong>Data in flight</strong> is data that is not yet committed to persistent storage. Nimble ensures all critical data in flight is checksummed and committed to both RAM and an ultra-fast byte-addressable NVDIMM-N memory module sitting right on the motherboard. The NVDIMM-N is mirrored to the partner controller and both controller NVDIMMs are protected against power loss via a supercapacitor. In the event of a power loss, the NVDIMMs simply flush their contents to flash storage. This approach is extremely reliable and doesn’t need inelegant solutions like a built-in UPS.</p><p><strong>Data on persistent storage</strong> is protected by what we call Triple+ Parity RAID. <em>Three orders of magnitude more resilient than RAID6</em>. For comparison, RAID6 is three orders of magnitude more resilient than RAID5. The “+” sign means that there is <em>extra</em> intra-drive parity that can safeguard against entire sectors being lost even if three whole drives fail in a single RAID group.</p><p>Some might say this is a bit much, however with drive sizes increasing rapidly (especially SSDs) and drive read error rates increasing as drives age, it was the architecturally correct choice to make.</p><h3>In Summary</h3><p>Users frequently assume that all storage systems will safely store their data. And they will, most of the time. But when it comes to your data, “most of the time” isn’t good enough. No measure should be considered too extreme. When looking for a storage system, it’s worth taking the time to understand all situations where your data could be compromised. And, if nothing else, it’s worth choosing a vendor who is paranoid and goes to extremes to keep your data safe.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db47105c35dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/uncompromising-resiliency-db47105c35dd">Uncompromising Resiliency</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of Automated Headroom Management]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-automated-headroom-management-737602180327?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/737602180327</guid>
            <category><![CDATA[failover]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[headroom]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:14:23 GMT</pubDate>
            <atom:updated>2017-01-30T03:14:22.766Z</atom:updated>
            <content:encoded><![CDATA[<p>Before we begin: This is another vendor-neutral post. I realize there may be no architecture that can do <em>everything</em>I’m proposing, but some may come closer to what you need than others. Whether you’re a vendor or a customer, see it as stuff you should be doing or be asking for respectively…</p><h3>Headroom!</h3><p><em>Headroom</em>is a term that applies to almost all technologies, and it’s crucially important for all of them. Some examples:</p><ul><li>Photography</li><li>Cars</li><li>Bridges</li><li>Storage arrays…</li></ul><h3>Why is Sufficient Headroom Important?</h3><p>Maintaining sufficient headroom in any solution is a way to <strong>ensure safety and predictability of operation under most conditions</strong> (<em>especially</em> under unfavorable ones).</p><p>For instance, if the maximum load for an evenly loaded bridge before it collapses is X, the overall recommended load will be a fraction of that. But even the weight/length/axle count of a <em>single</em>truck on a bridge will also be subject to certain strict limits in order to avoid excessive localized stress on the structure.</p><h3>Headroom in Storage Arrays</h3><p>Apologies to the seasoned storage pros for all the foundational material but it’s crucial to take this step by step.</p><p>It is important to note that headroom in arrays is not necessarily as simple as how busy the CPU is. Headroom is a multi-dimensional concept.</p><p>More factors than just CPU come into play, including how busy the underlying storage media are, how saturated various buses are, and how much of the CPU is spent on true workload vs opportunistic tasks (that could be deferred). Not to mention that in some systems, certain tasks are single-threaded and could pose an overall headroom bottleneck by maxing out a single CPU core, while the rest of the CPU is not busy at all.</p><p>Maintaining sufficient headroom in storage arrays is necessary in order to provide acceptable latency, especially in the event of high load during a controller failover. Depending on the underlying architecture of an array, different headroom approaches and calculations are necessary.</p><p>Some examples of different architectures:</p><ul><li>Active-Active controllers, per-controller pool</li><li>Active-Standby, single pool</li><li>Active-Active, single pool</li><li>Grid, single pool</li><li>Permutations thereof (it’s beyond the scope of this article to explore all possible options)</li></ul><p>The single vs multiple pool question complicates things a bit, plus things like disk ownership are also hugely important. This isn’t an argument about which architecture is better (it depends anyway), but rather about <em>headroom management in different architectures.</em></p><h3>Dual-Controller Headroom</h3><p>Dual-controller architectures need to be extremely careful with headroom management. After all, there are only two controllers in play. Here’s what sufficient headroom looks like in a dual-controller system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/0*kQ_CYILUzJ6kv52s.png" /></figure><p>There are not many options to keep things healthy in a dual-controller architecture. <strong>In an Active-Standby system</strong>, the Standby controller is ready to immediately take over. There is no danger in loading up the Active controller, aside from expected load-related latency.</p><p><strong>In an Active-Active HA system</strong>, maintaining a healthy amount of headroom has to be managed so that there is, overall, an <em>entire controller’s worth of free headroom available</em>.</p><h3>Headroom in a Cluster of HA Pairs Architecture</h3><p>There are several implementations that make use of a multiple HA Pair architecture. Often, the multiple HA pairs present a virtual pool to the outside world, even if, internally, there are multiple private pools. Some implementations just keep it to pools owned by each controller.</p><p>Here’s an example of healthy headroom in such a system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*tnhZAN8aFX-nUMuk.png" /></figure><p>Even though there are multiple controllers (at least 4 total), in order to maintain an overall healthy system, a total of 100% headroom needs to be maintained in <em>each</em> HA pair, otherwise the performance of an underlying private pool (in green) might suffer, making the overall virtual pool performance (light blue) unpredictable.</p><h3>Headroom in a True Grid Architecture</h3><p>Grid Architectures spread overall load among multiple nodes (often plain servers with some disks inside and connected via a network).</p><p>In such a scheme, overall headroom that needs to be maintained per node as a percentage is 100/N, where <em>N</em> is the number of nodes in the storage cluster.</p><p>So, in a 4-node cluster, 100/4=25% headroom per node needs to be maintained.</p><p>This doesn’t account for the significant work that rebalancing after a node failure takes in such architectures, nor the <em>capacity</em> headroom needed, but it’s roughly accurate enough for our purposes.</p><p>Schematically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*R6qrUMjvMA5wMfNf.png" /></figure><h3>How Headroom is Managed is Crucial</h3><p>In order to manage headroom, four things need to be able to happen first:</p><ol><li>Be able to calculate headroom</li><li>Be able to throttle workloads</li><li>Be able to prioritize between types of workload</li><li>Be able to move workloads around (architecture-dependent).</li></ol><p>The only architecture that inherently makes this a bit easier is Active-Standby since there is always a controller waiting to take over if anything bad happens. But even with a single active controller, headroom needs to be managed in order to avoid bad latency conditions during normal operation (see <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">here</a>for an example approach). Remember, headroom is a multi-dimensional thing.</p><h3>Example Problem Case: Imbalanced &amp; Overloaded Controllers</h3><p>Consider the following scenario: An Active-Active system has both controllers overloaded, and one of them is really busy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/0*1LPNDPS1WEGVlmfg.png" /></figure><p>Clearly, there are a few problems with this picture:</p><ol><li>It may be <em>impossible</em> to fail over in the event of a controller failure (total load is 165% of a single controller’s headroom)</li><li>The first controller may already be experiencing latency issues</li><li>Why did the system even get to this point in the first place?</li></ol><p>This is a commonplace occurrence unfortunately.</p><h3>Automation is Key in Managing Headroom</h3><p>The biggest problem in our example is actually the last point: <em>Why was the system allowed to get to that state to begin with?</em></p><p>Even if a system is able to calculate headroom, throttle workloads and move workloads around, if nothing is done <em>automatically</em> to prevent problems, it’s <em>extremely</em>easy for users to get into the problem situation depicted above. I’ve seen it affect critical production systems far too many times for comfort.</p><h3>Manual QoS is Not The Best Answer</h3><p>Being able to <em>manually</em> throttle workloads can obviously help in such a situation. The problems with the manual QoS approach are outlined in a past <a href="http://recoverymonkey.org/2016/04/08/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance/">article</a>, but, in summary, most users simply have absolutely <em>no</em> idea what the actual limits should be (nor should they be expected to). Most importantly, placing QoS limits up front doesn’t result in balanced controllers… and may even result in other kinds of performance problems.</p><p>Of course, using QoS limits <em>reactively</em> is not going to prevent the problem from occurring in the first place.</p><p>Some companies offer Data Classification as a Professional Services engagement, in order to try and figure out an IOPS/TB/Application metric. Even if that is done, it <em>doesn’t</em>result in balanced controllers… it’s also not very useful in dynamic environments. It’s more used as a guideline for setting up manual QoS.</p><h3>Automation Mechanisms to Consider for Managing Headroom</h3><p>Clearly, pervasive automation is needed in order to keep headroom at safe levels.</p><p>I will split up the proposed mechanisms per architecture. There is some common functionality needed regardless of architecture:</p><h3>Common Automation Needed</h3><p>Every architecture needs to have the ability to automatically achieve the following, <em>without</em>user intervention at any point:</p><ol><li>Conserve headroom per controller</li><li>Differentiate between different kinds of user workloads</li><li>Differentiate between different kinds of system workloads</li><li>Automatically prioritize between different workloads, especially under pressure</li><li>Automatically throttle different kinds of workloads, especially under pressure</li></ol><p>And now for the extra automation needed per architecture:</p><h3>Active-Standby Automation</h3><p>If in a single HA pair, nothing else is needed. If in a scale-out cluster of Active-Standby pairs:</p><ol><li>Automatically balance capacity and headroom utilization between HA pairs even if they’re different types</li><li>Be able to auto-migrate workloads to other cluster nodes (if using multiple pools instead of one)</li></ol><h3>Active-Active Automation</h3><ol><li>Automatically conserve one node’s worth of headroom across the HA pair (50/50, 60/40, 70/30 — all are OK)</li><li>When provisioning new workloads, auto-balance them by performance and capacity across the nodes</li><li>Be able to balance by auto-migrating workloads to the other node (if using multiple pools instead of one)</li></ol><h3>Active-Active with Multiple HA Pairs Automation</h3><ol><li>Automatically conserve one node’s worth of headroom per HA pair</li><li>Be able to auto-migrate workloads to any other node</li><li>Automatically balance workloads and capacity utilization in the underlying per-HA pools</li></ol><h3>Grid Automation</h3><ol><li>Automatically conserve at <em>least</em> one node’s worth of headroom across the grid</li><li>Automatically conserve enough capacity to be able to lose one node, rebalance, and have enough capacity left to lose another one (the more cautious may want the capability to lose <em>2–3 nodes simultaneously</em>)</li><li>Automatically take into account grid size and rebalancing effort in order to conserve the right amount of headroom</li></ol><h3>In Closing…</h3><p>If you’re a consumer of storage systems, always remember to be running your storage with sufficient headroom to be able to sustain a major failure without overly affecting your performance.</p><p>In addition, when looking to refresh your storage system, always ask the vendors about how they automate their headroom management.</p><p>Finally, if any vendor is quoting you performance numbers, <em>always</em> ask them how much headroom is left on the array at that performance level… (in addition to the extra questions about read/write percentages and latencies you should be <a href="http://recoverymonkey.org/2012/07/26/an-explanation-of-iops-and-latency/">asking already</a>).</p><p>The answer may surprise you.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=737602180327" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-automated-headroom-management-737602180327">The Importance of Automated Headroom Management</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of SSD Firmware Updates]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-ssd-firmware-updates-c6582ab29fe8?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/c6582ab29fe8</guid>
            <category><![CDATA[firmware]]></category>
            <category><![CDATA[hci]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[ssd]]></category>
            <category><![CDATA[sd]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:11:21 GMT</pubDate>
            <atom:updated>2017-01-30T03:11:39.802Z</atom:updated>
            <content:encoded><![CDATA[<p>I wanted to bring this crucial issue to light since I’m noticing several storage vendors being either cavalier about this or simply unaware.</p><p>I will explain why solutions that don’t offer some sort of automated, live SSD firmware update mechanism are potentially <em>extremely</em>risky propositions. Yes, this is another “vendor hat off, common sense hat on” type of post.</p><p><strong>Modern SSD Architecture is Complex</strong></p><p>The increased popularity and lower costs of fast SSD media are good things for storage users, but there is some inherent complexity within each SSD that many people are unaware of.</p><p>Each modern SSD is, in essence, <em>an entire pocket-sized storage array</em>, that includes, among other things:</p><ul><li>An I/O interface to the outside world (often two)</li><li>A CPU</li><li>An OS</li><li>Memory</li><li>Sometimes Compression and/or Encryption</li><li>What is, in essence, a log-structured filesystem, complete with complex load balancing and garbage collection algorithms</li><li>An array of flash chips driven in parallel through multiple channels</li><li>Some sort of RAID protection for the flash chips, including sparing, parity, error checking and correction…</li><li>A supercapacitor to safely flush cache to the flash chips in case of power failure.</li></ul><p>Sounds familiar?</p><p><strong>With Great Power and Complexity Come Bugs</strong></p><p>To make something clear: This discussion has <em>nothing</em> to do with overall SSD endurance &amp; hardware reliability. Only the software aspect of the devices.</p><p>All this extra complexity in modern SSDs means that an increased number of bugs compared to simpler storage media <em>is a statistical certainty.</em> There is just a <em>lot</em> going on in these devices.</p><p>Bugs aren’t necessarily the end of the world. They’re something understood, a fact of life, and there’s this magical thing engineers thought of called… Patching!</p><p>As a fun exercise, go to the firmware download pages of various popular SSDs and check the release notes for some of the bugs fixed. Many fixes address some rather abject gibbering horrors… :)</p><p>Even costlier enterprise SSDs have been afflicted by some <em>really</em> dangerous bugs — usually <em>latent defects</em> (as in: they don’t surface until you’ve been using something for a while, which may explain why these bugs were missed by QA).</p><p>I fondly remember a bug that hit some arrays at a previous place of employment: the SSDs would work great but after a certain number of hours of operation, if you shut your machine down, the SSDs would <em>never</em> come up again. Or, another bug that hit a very popular SSD that would downsize itself to an awesome 8MB of capacity (losing all existing data of course) once certain conditions were met.</p><p>Clearly, these are some pretty hairy situations. And, what’s more, <strong>RAID, checksums and node-level redundancy <em>wouldn’t</em> protect against all such bugs.</strong></p><p>For instance, think of the aforementioned power off bug — <em>all</em> SSDs of the same firmware vintage would be affected <em>simultaneously</em> and the entire array would have <em>zero</em> SSDs that functioned.<em> This actually happened</em>, I’m <em>not</em> talking about a theoretical possibility. You know, just in case someone starts saying “but SSDs are reliable, and think of all the RAID!”</p><p>It’s all about approaching correctness from a holistic point of view. Multiple lines of defense are necessary.</p><p><strong>The Rules: How True Enterprise Storage Deals with Firmware</strong></p><p>Just like with Fight Club, there are some basic rules storage systems need to follow when it comes to certain things.</p><ol><li>Any firmware patching should be a non-event. Doesn’t matter what you’re updating, there should be no downtime.</li><li>ANY firmware patching should be a NON-EVENT. Doesn’t matter what you’re updating, there should be NO downtime!</li><li>Firmware updates should be automated even when dealing with devices <em>en masse</em>.</li><li>The customer should automatically be notified of important updates they need to perform.</li><li>Different vintage and vendor component updates should be handled automatically and centrally. And, most importantly: <em>Safely</em>.</li></ol><p>If these rules are followed, bug risks are <em>significantly</em> mitigated and higher uptime is possible. Enterprise arrays typically will follow the above rules (but <em>always</em>ask the vendor).</p><p><strong>Why Firmware Updating is a Challenge with Some Storage Solutions</strong></p><p>Certain kinds of solutions make it inherently harder to manage critical tasks like component firmware updates.</p><p>You see, being able to hot-update different kinds of firmware in any given set of hardware means that <em>the mechanism doing the updating must be intimately familiar with the underlying hardware &amp; software combination, however complex.</em></p><p>Consider the following kind of solution, maybe for someone sold on the idea that white box approaches are the future:</p><ul><li>They buy a bunch of diskless server chassis from Vendor A</li><li>They buy a bunch of SSDs from Vendor B</li><li>They buy some Software Defined Storage offering from Vendor C</li><li>All running on the underlying OS of Vendor D…</li></ul><p>Now, let’s say Vendor B has an emergency SSD firmware fix they made available, easily downloadable on their website. Here are just <em>some</em>of the challenges:</p><ol><li>How will that customer be notified by Vendor B that such a critical fix is available?</li><li>Once they have the fix located, which Vendor will automate updating the firmware on the SSDs of Vendor B, and how?</li><li>How does the customer know that Vendor B’s firmware fix doesn’t violently clash with something from Vendor A, C or D?</li><li>How will all that affect the data-serving functionality of Vendor C?</li><li>Can any of Vendors A, B, C or D orchestrate <em>all</em> the above safely?</li><li>With no downtime?</li></ol><p>In most cases I’ve seen, the above chain of events will not even progress past #1. <em>The user will simply be unaware of any update</em>, simply because component vendors don’t usually have a mechanism that alerts individual customers regarding firmware.</p><p>You could inject a significant permutation here: <em>What if you buy the servers pre-built, including SSDs, from Vendor A, including full certification with Vendors C and D?</em></p><p>Sure — it still does not materially change the steps above. One of Vendors A, C or D still need to somehow:</p><ol><li>Automatically alert the customer about the critical SSD firmware fix being available</li><li>Be able to non-disruptively update the firmware…</li><li>…While not clashing with the other hardware <em>and</em> software from Vendors A, C and D</li></ol><p>I could expand this type of conversation to other things like overall environmental monitoring and checksums but let’s keep it simple for now and focus on just component firmware updates…</p><p><strong>Always Remember — Solve Business Problems &amp; Balance Risk</strong></p><p>Any solution is a compromise. Always make sure you are comfortable with the added risk certain areas of compromise bring (and that you are fully aware of said risk).</p><p>The allure of certain approaches can be significant (at the very least because of lower promised costs). It’s important to maintain a balance between increased risk and business benefit.</p><p>In the case of SSDs specifically, the utter criticality of certain firmware updates means that it’s crucially important for any given storage solution to be able to safely and automatically address the challenge of updating SSD firmware.</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6582ab29fe8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-ssd-firmware-updates-c6582ab29fe8">The Importance of SSD Firmware Updates</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Importance of the Effective Capacity Ratio in Modern Storage Systems]]></title>
            <link>https://medium.com/recoverymonkey/the-importance-of-the-effective-capacity-ratio-in-modern-storage-systems-e7bc62e36cd0?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/e7bc62e36cd0</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[fud]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[deduplication]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:10:03 GMT</pubDate>
            <atom:updated>2017-01-30T03:10:02.556Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern storage devices (especially All Flash Arrays), extensive data reduction techniques are commonplace and expected by customers.</p><p>This has, unavoidably, led to various marketing schemes that aim to make certain systems seem more appealing than the rest. Or at least not <em>less</em> appealing…</p><p>I will attempt to explain what customers should be looking for when trying to decipher capacity claims from a manufacturer.</p><p>In a nutshell — and for the ADD-afflicted — <strong>the most important number you should be looking for is the Effective Capacity Ratio, which is simply: <em>(Effective Capacity)/(Raw Capacity)</em></strong><em>. Ignore</em> the more common but far less useful Data Reduction Ratio, which is: (Effective Capacity)/(Usable Capacity).</p><p>Read on for more detail…</p><p><strong>The problem</strong></p><p>Most modern (and some not so modern) storage vendors claim a similar Data Reduction Ratio for their devices. Numbers like 5:1 seem commonplace these days. Which, to anyone with a modicum of common sense, simply means “I can store five times more stuff in the same space”. Unfortunately, the same 5:1 ratio does <em>not</em>mean the same end result for all vendors…</p><p><strong><em>Why</em> this is a problem</strong></p><p>Customers simply want to get a good deal for their money. The reality is that the exact same 5:1 Data Reduction Ratio may mean <em>very</em> different things between storage devices.</p><p>For instance: Not all vendors count space savings using the same math. Is the virtual size of snapshots calculated in the final savings ratio? (for example, if I take 5 snaps of a 1TB volume one after another, is that showing up as 6:1 savings?)</p><p>How about Thin Provisioning? Is that included? Such math can <em>wildly</em> alter the overall efficiency numbers.</p><p>Here’s a clear example of why counting thin provisioning in the overall ratio is probably misleading, and best used only for educational purposes… <em>Remember that overall savings ratios are multiplicative</em>. For example, 2:1 compression and 2:1 deduplication mean 4:1 overall savings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*7iwNqygRqCnd7ewbbGmg_g.png" /></figure><p>Does anyone <em>really</em> believe that such a system is actually providing <em>1000:1 </em>savings? Or even 10:1? <strong>Yet that’s what many vendors are counting towards savings ratios, <em>without</em> separating the thin provisioning savings from the overall savings number</strong>.</p><p>However, there is <em>another</em> dimension, and it has to do with <strong>how efficiently the raw capacity is actually utilized</strong>. It will be one of my many <em>Captain Obvious</em>moments for some of you, but I’ve seen enough people get confused, so it’s worth explaining.</p><p><strong>The Effective Capacity Ratio</strong></p><p>Different storage systems have different ways of utilizing their raw capacity. For example, a mirrored system can never have better than a 50% Usable:Raw ratio. By definition, it’s mathematically <em>impossible</em> since 2 copies are needed. That’s not even counting spares and other possible overheads.</p><p>Systems that do triple mirroring can’t do better than a theoretical 33.3% Usable:Raw etc.</p><p><strong>Some definitions are in order:</strong></p><ul><li><em>Usable Capacity</em>: How much data I can store in a system <em>after</em> overheads such as RAID, sparing etc. but <em>before</em> data reduction techniques.</li><li><em>Raw Capacity</em>: Add up the capacity of all the storage media in the system. Usually a Base 10 number (TB/GB, not Base 2, TiB/GiB)</li><li><em>Effective Capacity</em>: How much data I can store in a system <em>after</em> data reduction techniques like Deduplication and Compression but <em>not</em> Thin Provisioning</li><li><em>Data Reduction Ratio</em>: (Effective Capacity)/(Usable Capacity)</li><li><strong><em>Effective Capacity Ratio</em></strong>: (Effective Capacity)/(Raw Capacity)</li></ul><p><strong>Who is <em>More</em> Efficient?</strong></p><p>If every vendor is claiming 5:1 average savings, who is <em>truly</em> more efficient? A <em>Reductio ad Absurdum </em>example makes it pretty clear:</p><ol><li>A vendor that can do a Data Reduction Ratio of 5:1 but has a Usable Capacity of <strong>10%</strong> vs Raw or…</li><li>A vendor that can do a Data Reduction Ratio of 5:1 but has a Usable Capacity of <strong>70%</strong> vs Raw?</li></ol><p>Let’s put some numbers on a table. They roughly correspond to some existing storage vendors today (there may be some variation depending on whether the numbers for each line are TB vs TiB — everyone shows numbers differently — but the overall point remains the same):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*zNfryeWfzy3HpFHOVzSFIQ.png" /></figure><p>As you can see, the <em>same</em> Data Reduction Ratio, on the <em>same</em> amount of Raw Capacity, can have <em>wildly</em> different Effective Capacity results, depending on the system.</p><p>Clearly, a truly efficient system is one that can both:</p><ul><li>Provide a high Usable:Raw ratio <em>and</em></li><li>Provide a high Data Reduction Ratio (that does <em>not</em> include fluff like Thin Provisioning)</li></ul><p><strong>The Business Benefits of a High Effective Capacity Ratio</strong></p><p>There are multiple business reasons why chasing a high Effective Capacity Ratio is important:</p><ul><li>High rack density. Some vendors are <em>eight times</em> more dense than others (ironically, the ones claiming “white box economics” are the <em>least</em> dense and most expensive). This could lead to <em>significant</em> power/cooling and $/rack savings</li><li>Lower CapEx — if a vendor needs 2x the Raw Capacity vs another, guess who’s paying for the difference?</li><li>Less complexity — denser devices means <em>less</em> devices, less cables, less switching</li></ul><p><strong>What <em>you</em> can do as a Customer</strong></p><p><em>Level the playing field</em>. It’s actually pretty easy:</p><ul><li>Insist on seeing all capacity numbers expressed as <em>TiB</em> from every vendor — and if they don’t know what <em>that</em> means, <em>run</em>… (or at least subtract 9% from any capacity number you see and you’ll be safer)</li><li>Insist on seeing the Data Reduction Ratio <em>without</em> including Thin Provisioning or Snapshots. If they can’t break out the savings by category, <em>run</em>…</li><li>Insist on seeing the Usable:Raw percentage for various configurations. Is it better in larger configs vs smaller? Is it following all best practices? What <em>are</em> the best practices for space consumption? If they tell you to ignore the man behind the curtain, <em>run</em>… (there is a theme developing)</li><li><strong><em>Do your own Effective Capacity Ratio calculation</em></strong> and assign a $/Effective TiB to each vendor</li></ul><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7bc62e36cd0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-importance-of-the-effective-capacity-ratio-in-modern-storage-systems-e7bc62e36cd0">The Importance of the Effective Capacity Ratio in Modern Storage Systems</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Progress needs Trailblazers]]></title>
            <link>https://medium.com/recoverymonkey/progress-needs-trailblazers-39e8999eb982?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/39e8999eb982</guid>
            <category><![CDATA[progress]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 30 Jan 2017 03:00:05 GMT</pubDate>
            <atom:updated>2017-01-29T03:34:59.517Z</atom:updated>
            <content:encoded><![CDATA[<p>I got the idea for this stream-of-consciousness (and possibly Captain Obvious) post after reading some comments regarding new ways to do high speed I/O. Not something boring like faster media or protocols, but rather far more exotic approaches that require <em>drastically</em> rewriting applications to get the maximum benefits of radically different architectures.</p><p>The comments, in essence, stated that such advancements would be an <em>utter waste of time</em> since 99% of customers have absolutely no need for such exotica, but rather just need cheap, easy, reliable stuff, and can barely afford to buy infrastructure to begin with, let alone custom-code applications to get every last microsecond of latency out of gear.</p><h3>Why Trailblazers Matter</h3><p>It’s an interesting balancing game: There’s a group of people that want progress and innovation at all costs, and another group of people that think the status quo is OK, or maybe just give me cheaper/easier/more reliable and I’m happy.</p><p>Both groups are needed, like Yin and Yang.</p><p>The risk-averse are needed because without them it would all devolve into chaos.</p><p>But without people willing to take risks and do seemingly crazy and expensive things, progress would simply be much harder.</p><p>Imagine if the people that prevailed were the ones that thought trepanning and the balancing of humours was state-of-the-art medicine and needed no further advancements. Perchance the invention of MRI machines might have taken longer?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*E9Us55EkbeLJDh6u.jpg" /></figure><p>What if traditional planes were deemed sufficient, would a daring endeavor like the SR-71 have been undertaken?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/0*X8BFDux5LgErRfad.jpg" /></figure><h3>Advanced Tech Filters Down to the Masses</h3><p>It’s interesting how the crazy, expensive and “unnecessary” eventually manages to become part of everyday life. Think of such commonplace technologies like:</p><ul><li>ABS brakes</li><li>SSDs</li><li>4K TVs</li><li>High speed networking</li><li>Smartphones</li><li>Virtualization</li><li>Having more than 640KB of RAM in a PC</li></ul><p>They all initially cost a ton of money and/or were impractical for everyday use (the first ABS brakes were in planes!)</p><h3>What is Impractical or Expensive Now Will be Normal Tomorrow…</h3><p>…but not every new tech will make it. <em>Which is perfectly normal</em>.</p><ul><li>Rewriting apps to take advantage of things like containers, microservices and fast object storage? It will slowly happen. <em>Most</em> customers can simply wait for the app vendors to do it.</li><li>In-memory databases? Not a niche any more, when even the ubiquitous SQL Server is doing it…</li><li>Using advanced and crazy fast byte-addressable NVDIMM in storage solutions? Some mainstream vendors like Nimble and Microsoft are already doing it. Others are working on some really interesting implementations.</li><li>AI, predictive analytics, machine learning? Already happening and providing huge value (for example Nimble’s InfoSight).</li></ul><h3>Don’t Belittle the Risk Takers!</h3><p>Even if you think the crazy stuff some people are working on isn’t a fit for you, don’t belittle it. It can be challenging to hold your anger, I understand, especially when the people working on crazy stuff are pushing their viewpoint as if it will change the world.</p><p>Because you never know, it just might.</p><p>If you don’t agree — get out of the way, or, even better, help find a <em>better</em> way.</p><h3>Just Because it’s New to You Doesn’t Mean it’s Really New, or Risky</h3><p>Beware of turning into an IT Luddite.</p><p>Just because <em>you</em> aren’t using a certain technology, but many <em>others</em> are, and reaping far more benefits than you with your current tech, then maybe they’re onto something… :)</p><p><em>At what point does your resistance to change become a competitive disadvantage?</em></p><h3>Stagnation is Death</h3><p>…or, at a minimum, boring. Τα πάντα ρει, as Heraclitus said. I, for one, am looking forward to the amazing innovations coming our way, and actively participate in trying to make them come to fruition faster.</p><p>What will you do?</p><p>D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39e8999eb982" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/progress-needs-trailblazers-39e8999eb982">Progress needs Trailblazers</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Well-Behaved Storage System: Automatic Noisy Neighbor Avoidance]]></title>
            <link>https://medium.com/recoverymonkey/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance-329388ddbb94?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/329388ddbb94</guid>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[storage]]></category>
            <category><![CDATA[it-exotica]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Fri, 08 Apr 2016 08:57:13 GMT</pubDate>
            <atom:updated>2017-01-30T02:51:24.706Z</atom:updated>
            <content:encoded><![CDATA[<p>This topic is very near and dear to me, and is one of the big reasons I came over to Nimble Storage.</p><p>I’ve always believed that <strong>storage systems should behave gracefully and predictably under pressure. Automatically. Even under complex and difficult situations</strong>.</p><p>It sounds like a simple request and it makes a whole lot of sense, but <em>very</em> few storage systems out there actually behave this way. This creates business challenges and increases risk and OpEx.</p><h4>The Problem</h4><p>The simplest way to state the problem is that <strong>most storage systems can enter conditions where workloads can suffer from unfair and abrupt performance starvation under several circumstances.</strong></p><p>OK, maybe that wasn’t the simplest way.</p><p>Consider the following scenarios:</p><ol><li>A huge sequential I/O job (backup, analytics, data loads etc.) happening in the middle of latency-sensitive transaction processing</li><li>Heavy array-generated workloads (garbage collection, post-process dedupe, replication, big snapshot deletions etc.) happening at the same time as user I/O</li><li>Failed drives</li><li>Controller failover (due to an actual problem or simply a software update)</li></ol><p>#3 and #4 are more obvious — a well-behaved system will ensure high performance even during a drive failure (or three), and after a controller fails over. For instance, if total system headroom is <em>automatically</em> kept at 50% for a dual-controller system (or, simplistically, <em>100/n</em>, where <em>n</em> is the controller count for shared-everything architectures), even after a controller fails, performance should be fine.</p><p>#1 and #2 are a bit more complicated to deal with. Let’s look at this in more detail.</p><h4>The Case of Competing Workloads During Hard Times</h4><p>Inside every array, at any given moment, a balancing act occurs. Multiple things need to happen simultaneously.</p><p><strong>Several user-generated workloads</strong>, for instance:</p><ul><li>DB</li><li>VDI</li><li>File Services</li><li>Analytics</li></ul><p><strong>Various internal array processes</strong> — they <em>also</em> are workloads, just array-generated, and often critical:</p><ul><li>Data reduction (dedupe, compression)</li><li>Cleanup (object deletion, garbage collection)</li><li>Data protection (integrity-related)</li><li>Backups (snaps, replication)</li></ul><p>If the system has enough headroom, all these things will happen without performance problems.</p><p>If the system <em>runs out of headroom</em>, that’s where most arrays have challenges with <em>prioritizing</em> what happens when.</p><p>The most common way a system may run out of headroom is the sudden appearance of a hostile “bully” workload. This is also called a “noisy neighbor”. Here’s an example of system behavior in the presence of a bully workload:</p><figure><img alt="bully_vs_victim_2" src="https://cdn-images-1.medium.com/max/1024/0*tA7jxiRUpq020JYj.png" /></figure><p>In this example, the latency-sensitive workload will greatly and unfairly suffer after the “noisy neighbor” suddenly appears. If the latency-sensitive workload is a mission-critical application, this could cause a <em>serious</em> business problem (slow processing of financial transactions, for instance).</p><p>This is an <em>extremely</em> common scenario. A lot of the time it’s not even a new workload. Often, an <em>existing</em> workload changes behavior (possibly due to an application change — for instance a patch or a modified SQL query). <em>This stuff happens</em>.</p><h4>How some vendors have tried to fix the issue with Manual “QoS”</h4><p>As always, there is more than one way to skin a cat, if one is so inclined. Here are a couple of manual methods to fix workload contention:</p><ul><li>Some arrays have a simple IOPS or throughput <em>limit</em> that an administrator can <em>manually</em> adjust in order to fix a performance problem. This is an iterative and <em>reactive</em> method and hard to automate properly in real time. In addition, if the issue was caused by an <em>internal</em> array-generated workload, there is often no tooling available to throttle those processes.</li><li>Other arrays insist on the user setting up minimum, maximum and burst IOPS values <em>for every single volume in the system</em>, upon volume creation. This assumes the user knows <em>in advance</em> what performance envelope is required, in detail, <em>per volume</em>. The reality is that almost nobody knows these things beforehand, and getting the numbers wrong can itself cause a huge problem with latencies. Most people just want to get on with their lives and have their stuff work without babysitting.</li></ul><p><em>Manual</em> mechanisms for fixing the “bully” workload challenge result in systems that are hard to consume and complex to support while under performance pressure. Moreover, when a performance issue occurs, speed of resolution is <em>critical</em>. The issue needs to be resolved <em>immediately, </em>especially for latency-sensitive workloads. <em>Manual methods will simply not be fast enough</em>. Business <em>will</em> be impacted.</p><h4>How Nimble Storage Fixed the Noisy Neighbor Issue</h4><p>No cats were harmed in the process. Nimble engineers looked at the extensive telemetry in <a href="http://info.nimblestorage.com/rs/nimblestorage/images/nimblestorage-ds-infosight.pdf">InfoSight</a>, used data science, and neatly identified areas that could be massively automated in order to optimize system behavior under a wide variety of adverse conditions. Some of what was done:</p><ul><li>Highly advanced Fair Share disk scheduling (separate mechanisms that deal with different scenarios)</li><li>Fair Share CPU scheduling</li><li>Dynamic Weight Adjustment — automatically adjust priorities in various ways under different resource contention conditions, so that the system can always complete critical tasks and not fall dangerously behind</li></ul><p>The end result is a system that:</p><ul><li>Lets system latency increase gracefully and progressively as load increases</li><li>Carefully and <em>automatically</em> balances user and system workloads</li><li>Achieves I/O deadlines and preemption behavior</li><li>Eliminates the Noisy Neighbor problem without the need for <em>any</em> manual QoS adjustments</li><li>Allows latency-sensitive small-block I/O to proceed without interference from bully workloads</li></ul><p>Such automation achieves a better<em> business result</em>: less risk, less OpEx, easy supportability, simple and safe overall consumption even under difficult conditions.</p><h4>What Should Nimble Customers do to get this Capability?</h4><p>As is typical with Nimble systems and their impressive <a href="http://recoverymonkey.org/2016/03/28/going-green-why-i-joined-nimble-storage/">Ease of Consumption</a>, nothing fancy needs to be done apart from simply upgrading to a specific release of the code (in this case 3.1 and up — 2.3 did some of the magic but 3.1 is the fully realized vision).</p><p>A bit anticlimactic, apologies… if you like complexity, watching <a href="https://www.youtube.com/watch?v=qybUFnY7Y8w">this</a> instead is probably more fun than juggling QoS manually.</p><p>D</p><p>Technorati Tags: <a href="http://technorati.com/tag/Nimble">Nimble</a>, <a href="http://technorati.com/tag/Performance">Performance</a>, <a href="http://technorati.com/tag/QoS">QoS</a>, <a href="http://technorati.com/tag/Storage">Storage</a>, <a href="http://technorati.com/tag/Automation">Automation</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=329388ddbb94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/the-well-behaved-storage-system-automatic-noisy-neighbor-avoidance-329388ddbb94">The Well-Behaved Storage System: Automatic Noisy Neighbor Avoidance</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going Green: Why I Joined Nimble Storage]]></title>
            <link>https://medium.com/recoverymonkey/going-green-why-i-joined-nimble-storage-a87f9153bb42?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/a87f9153bb42</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[general]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[technology-news]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Mon, 28 Mar 2016 12:15:51 GMT</pubDate>
            <atom:updated>2017-01-30T02:51:27.705Z</atom:updated>
            <content:encoded><![CDATA[<p>I am proud to announce that, as of today, I am a member of the Nimble Storage team.</p><figure><img alt="Nimble Logo" src="https://cdn-images-1.medium.com/max/1016/0*D7pf6s0JaaWozGXc.png" /></figure><p>This marks the end of an era — I spent quite a bit of time at NetApp: learned a lot, did a lot — by the end I had my hands in all kinds of sausage making… :)</p><p>I wish my friends at NetApp the best of luck for the future. The storage industry is a very tough arena, and one that will be increasingly harder and with less tolerance than ever before.</p><h4>Why?</h4><p>I compared Nimble Storage with many competitors before making my decision. Quite simply, Nimble’s core values agree with mine. It goes without saying that I wouldn’t choose to move to a company unless I believed they had the best technology (and the best support), but the core values is where it all starts. The product is built upon those core values.</p><p><strong>I firmly believe that modern storage should be <em>easy to consume</em></strong>. Indeed, it should be a <em>joy</em> to consume, even for complex multi-site environments. It should not be a burden to deal with. <em>Nor should it be a burden to sell.</em></p><p>Systems that are holistically easy to consume have several business benefits, some of which are: lower OPEX and CAPEX, increased productivity, less risk, easier planning, faster problem resolution.</p><p>It’s important to understand that <em>easy to consume is not at all the same as</em> <strong><em>easy to use</em></strong><em> — </em>that is but a very small subset of <em>easy consumption</em>.</p><p>The core value of <em>easy consumption</em> encompasses several aspects, many of which are ignored by most storage vendors. Most modern players will focus on <em>ease of use</em>, show demos of a pretty GUI and suchlike. <em>“Look how easy it is to install”</em> or <em>“look how easy it is to create a LUN”</em>. Well — there’s a <em>lot</em> more to worry about in real life.</p><h4>The lifecycle of a storage system</h4><p>Beyond initial installation and simple element creation, there is a multitude of things users and vendors need to be concerned with. Here’s a partial list:</p><ul><li>Installation</li><li>Migration to/from</li><li>Provisioning</li><li>Host/fabric configuration</li><li>Backups, restores, replication</li><li>Scaling up/out</li><li>Upgrading from a smaller/older version</li><li>Firmware updates for all components (including drives)</li><li>Tech refresh</li><li>Support</li></ul><h4>What about more advanced topics?</h4><p>A storage solution cannot exist in a vacuum. There are several ancillary (but <em>extremely</em> important) services needed in order to help consume storage, especially at scale. Services that typically cannot (and many that <em>should not</em>) reside on the storage system itself. How about…</p><ul><li>Initial and future sizing</li><li>Capacity planning based on long-term usage data</li><li>Performance analysis and profiling</li><li>Performance issue resolution/recommendations</li><li>Root cause analysis</li><li>What-if scenario modeling</li><li>Support case resolution</li><li>Comprehensive end-to-end monitoring and alerting</li><li>Comprehensive reporting (including auditing)</li><li>Security (including RBAC and delegation)</li><li>Upgrade planning</li><li>Pervasive automation (including host-side)</li><li>Ensuring adherence to best practices</li></ul><p><em>If a storage solution doesn’t make all or most of the above straightforward, then it is not truly easy to consume.</em></p><h4>The problem:</h4><p>Storage vendors will typically either be lacking in many of the above areas, or may need <em>many</em> different tools, and significant manual effort, in order to provide even <em>some</em> of these services.</p><p><em>Not</em> having the tools creates an obvious problem — the customer and vendor simply can’t perform these functions, or the implementation is too basic. <em>Most smaller vendors are in this camp</em>. Not much functionality beyond what’s inside the storage device itself. Long-term consumption, especially at scale, becomes challenging.</p><p>On the other hand, having a <em>multitude</em> of tools to help with these areas <em>also</em> makes the solution hard to consume overall. <em>Larger vendors fall into this category</em>.</p><p>For instance: <em>Customers</em> may need to access many different tools just to monitor and alert on various metrics. One tool may provide certain information, another tool provides different metrics (often with <em>significant</em> overlap with the first tool), and so on. And not all tools work with all versions of the product. This increases administrative complexity and overall time and money spent. And the end result is often compromised and incredibly hard to support.</p><p><em>Vendors</em> that need many different tools also create a problem for <em>themselves</em>: Almost nobody on staff will have the expertise to deal with the plethora of tools necessary to do certain things like sizing, performance troubleshooting or even a tech refresh. Or optimizing a product for specific workloads. Deep expertise is often needed to interpret the results of the tools. This causes interminable delays in problem resolution, lengthens sales cycles, complicates product development, creates staffing challenges, increases costs, and in general makes life miserable.</p><figure><img alt="RG autojack" src="https://cdn-images-1.medium.com/max/887/0*6GW7H5bdIkHlb0Wz.jpg" /></figure><h4>How?</h4><p>What always fascinated me about Nimble Storage is that not only did they recognize these challenges, they actually built an entire infrastructure and innovative approach in order to solve the problem.</p><p><strong><em>Nimble recognized the value of Predictive Analytics</em>.</strong></p><p>The challenge: How to use Big Data to solve the challenges faced by storage customers <em>and</em> storage vendors. And how to do this in a way that achieves a <em>dramatically</em> better end result.</p><p>While most vendors have call-home features, and some even have rudimentary capacity, configuration and maybe even performance telemetry being sent to some central repository (usually <em>very</em> infrequently), Nimble elected instead to send <em>extremely</em> comprehensive sensor telemetry to a huge analytics engine. A difficult undertaking, but one that would define the company in the years to come.</p><p>Nimble also recognized the need to do this <em>from the very beginning</em>. Each Nimble array sends 30–70 million data points back to Nimble every day. Trying to retrofit telemetry of this scope would be extremely difficult if not impossible to achieve effectively.</p><p>This wealth of data (the <em>largest</em> storage-related analytics engine in the world, by far) is used to help customers with the challenges mentioned previously, while at the same time lowering complexity.</p><p>It also, crucially, helps Nimble better support customers <em>and</em> design better products without having to bother customers for data dumps.</p><p>For example: What if a Nimble engineer trying to optimize SQL I/O performance wants to see <em>detailed</em> I/O statistics <em>only </em>for SQL workloads on all Nimble arrays in the world? Or on one array? Or on all arrays at a certain customer? It’s only a simple query away… and that’s just scratching the surface of what’s possible. <em>It certainly beats trying to design storage based on arbitrary synthetic benchmarks, or manually grabbing performance traces from customer gear…</em></p><h4>What?</h4><p>Enter <a href="http://info.nimblestorage.com/rs/nimblestorage/images/nimblestorage-ds-infosight.pdf">InfoSight</a>. That’s the name of the gigantic analytics engine currently ingesting trillions of anonymized sensor data points every week. And growing. Check some numbers <a href="http://www.nimblestorage.com/blog/you-call-that-big-data/">here</a>…</p><p>Nimble Storage customers <em>do not</em> need to install custom monitoring tools to perform highly advanced storage analytics, performance troubleshooting, and even hardware upgrade recommendations based on automated performance analysis heuristics.</p><p>No need to use the CLI, no need to manually send data dumps to the vendor, no need to use 10 different tools.</p><p>All the information customers need is available through a browser GUI. Even the vast majority of support cases are <em>automatically</em> handled by InfoSight, and I’m <em>not</em> talking about simply sending replacement hardware (that’s trivial).</p><p>I always saw InfoSight as the core offering of Nimble Storage, the huge differentiator that works hand in hand with the hardware and helps customers consume storage easily. Yes, Nimble Storage arrays are fast, reliable, easy to use, have impressive data reduction abilities, scale nicely, have great features, are cost-effective etc. But other vendors can claim they can satisfy at least some of those attributes.</p><p><em>Nobody</em> else has anything even<em> remotely</em> the depth and capability of InfoSight. This is why Nimble calls their offering the <em>Predictive Flash Platform</em>. InfoSight Predictive Analytics + great hardware = Predictive Flash.</p><p>I will be covering this fascinating topic in a lot more depth in the future. An AI Expert System powered by a behemoth analytics engine, helping reduce complexity and making the solution <strong><em>Easy To Consume</em></strong> is a pretty impressive piece of engineering.</p><p>Watch this space…</p><p>D</p><p>Technorati Tags: <a href="http://technorati.com/tag/NetApp">NetApp</a>, <a href="http://technorati.com/tag/Nimble">Nimble</a>, <a href="http://technorati.com/tag/Big%20Data">Big Data</a>, <a href="http://technorati.com/tag/Analytics">Analytics</a>, <a href="http://technorati.com/tag/InfoSight">InfoSight</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a87f9153bb42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/going-green-why-i-joined-nimble-storage-a87f9153bb42">Going Green: Why I Joined Nimble Storage</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7-Mode to Clustered ONTAP Transition]]></title>
            <link>https://medium.com/recoverymonkey/7-mode-to-clustered-ontap-transition-19ceb0be8804?source=rss----cfac87310593---4</link>
            <guid isPermaLink="false">https://medium.com/p/19ceb0be8804</guid>
            <category><![CDATA[fud]]></category>
            <category><![CDATA[7mtt]]></category>
            <category><![CDATA[rant]]></category>
            <category><![CDATA[it-exotica]]></category>
            <category><![CDATA[storage]]></category>
            <dc:creator><![CDATA[Dimitris Krekoukias]]></dc:creator>
            <pubDate>Sat, 06 Feb 2016 00:38:44 GMT</pubDate>
            <atom:updated>2017-01-30T02:51:21.997Z</atom:updated>
            <content:encoded><![CDATA[<p>I normally deal with different aspects of storage (arguably far more exciting) but I thought I would write something to provide some common sense perspective on the current state of 7-Mode to cDOT adoption.</p><p>I will tackle the following topics:</p><ol><li>cDOT vs 7-Mode capabilities</li><li>Claims that not enough customers are moving to cDOT</li><li>7-Mode to cDOT transition is seen by some as difficult and expensive</li><li>Some argue it might make sense to look at competitors and move to those instead</li><li>What programs and tools are offered by NetApp to make transition easy and quick</li><li>Migrating from competitors to cDOT</li></ol><h4>cDOT vs 7-Mode capabilities</h4><p>I don’t want to make this into a dissertation or take a trip down memory lane. Suffice it to say that while cDOT has most of the 7-Mode features, it is <em>internally</em> very different but also much, <em>much</em> more powerful than 7-Mode — cDOT is a <em>far</em> more capable and scalable storage OS in almost every possible way (and the roadmap is utterly <em>insane</em>).</p><p>For instance, cDOT is able to nondisruptively do anything, including crazy stuff like moving SMB shares around cluster nodes (moving LUNs around is <em>much</em> easier than dealing with the far more quirky SMB protocol). Some reasons to move stuff around the cluster could be node balancing, node evacuation and replacement… all done on the fly in cDOT, <em>regardless of protocol</em>.</p><p>cDOT also handles Flash much better (cDOT 8.3.x+ can be several <em>times</em> faster than 7-Mode for Flash on the <em>exact same</em> hardware). Even things like block I/O (FC and iSCSI) are <em>completely</em> written from the ground up in cDOT. Cloud integration. Automation. How failover is done. Or how CPU cores are used, how difficult edge conditions are handled… I could continue but then the ADD-afflicted would move on, if they haven’t already…</p><p>In a nutshell, cDOT is a more flexible, forward-looking architecture that respects the features that made 7-Mode so popular with customers, but goes incredibly further. There is no competitor with the breadth of features available in cDOT, let alone the features coming soon.</p><p>cDOT is quite simply the next logical step for a 7-Mode customer.</p><h4>Not enough customers moving to cDOT?</h4><p>The reality is actually pretty straightforward.</p><ul><li>Most new customers simply go with cDOT, naturally. 7-Mode still has a couple of features cDOT doesn’t, and if those features are <em>really</em> critical to a customer, that’s when someone might go with 7-Mode today. With each cDOT release the feature delta list gets smaller and smaller. Plus, as mentioned earlier, cDOT has a plethora of features and huge enhancements that will never make it to 7-Mode, with much more coming soon.</li><li>The things still missing in cDOT (like WORM) aren’t even offered by the majority of storage vendors… many large customers use our WORM technology.</li><li>Large existing customers, especially ones running critical applications, naturally take longer to cycle technologies. <strong>The average time to switch major technologies (irrespective of vendor) is around 5 years</strong>. The big wave of cDOT transitions hasn’t even hit yet!</li><li>Given that cDOT 8.2 with SnapVault was the appropriate release for many of our customers and 8.3 the release for most of our customers, a huge number of systems are still within that 5-year window prior to converting to cDOT, given when those releases came out.</li><li>Customers with mission-critical systems will typically <em>not</em> convert an existing system — they will wait for the next major refresh cycle. Paranoia rules in those environments (that’s a general statement regardless of vendor). And we have <em>many</em> such customers.</li></ul><h4>7-Mode to cDOT transition is seen by some as difficult and expensive</h4><p>This is a fun one, and a favorite FUD item for competitors and so-called “analysts”. I sometimes think we confused people by calling cDOT “ONTAP”. I bet expectations would be different if we’d called it <em>“SuperDuper ClusterFrame OS”</em>.</p><p>You see, cDOT is <em>radically</em> different in its internals versus 7-Mode — however, it’s still officially <em>also</em> called “ONTAP”. As such, customers are conditioned to super-easy upgrades between <em>ONTAP</em> releases (just load the new code and you’re done). cDOT is different enough that we can’t just do that.</p><p>I lobbied for the “ClusterFrame” name but was turned down BTW. I still think it rocks.</p><p>The fact that you can run either 7-Mode or cDOT on the <em>same</em> physical hardware confuses people even further. It’s a <em>good</em> thing to be able to reuse hardware (software-defined and all that). Some vendors like to make each new rev of the <em>same</em> family line (and its code) <em>utterly incompatible</em> with the last one… we don’t do that.</p><p>And for the startup champions: <strong>Startups haven’t been around <em>long</em> enough to have seen a <em>truly</em> <em>major</em> hardware and/or software change!</strong> (another thing conveniently ignored by many). Nor do they have the sheer amount of features and ancillary software ONTAP does. And of course, some vendors forget to mention what even a normal tech refresh looks like for their fancy new “built from the ground up” box with the <em>extremely</em> exciting name.</p><p>We truly <em>know</em> how to do upgrades… probably better than any vendor out there. For instance: What most people don’t know is that WAFL (ONTAP’s underlying block layout abstraction layer) has been quietly upgraded many, many times over the years. On the fly. In major ways. <em>With</em> a backout option. Another vendor’s product (again the one with the <em>extremely</em> exciting name) needed to be wiped <em>twice</em> by as many “upgrades” in <em>one</em> year in order to have its block layout changed.</p><p>Here’s the rub:</p><p><strong>Transition complexity really depends on how complex your current deployment is, your appetite for change and tolerance of risk. But transition <em>urgency</em> depends on how much you need the fully nondisruptive nature of cDOT and all the other features it has vs 7-Mode.</strong></p><p>What I mean by that:</p><p>We have some customers that lose upwards of $4m/hour of downtime. The long-term benefits of a truly nondisruptive architecture make any arguments regarding migration efforts effectively moot.</p><p>If you are using a <em>lot</em> of the 7-Mode features and companion software (and it has more features than almost any other storage OS), specific tools written only for 7-Mode, older OS clients only supported on 7-Mode, tons of snaps and clones going back to several years’ retention etc…</p><p>Then, in order to retain that kind of similar elaborate deployment in cDOT, the migration effort will also naturally be a bit more complex. But still doable. <strong>And we can automate most of it. Including moving over all the snapshots and archives!</strong></p><p>On the other hand, if you are using the system like an old-fashioned device and aren’t taking advantage of all the cool stuff, then moving to <em>anything</em> is relatively easy. And especially if you’re close to 100% virtualized, migration can be downright trivial (though simply moving VMs around storage systems ignores any snapshot history — the big wrinkle with VM migrations).</p><h4>Some argue it might make sense to look at competitors and move to those instead</h4><p>Looking at options is something that makes business sense in general. It would be very disingenuous of me to say it’s foolish to look at options.</p><p>But this holds firmly true: If you want to move to a <em>competitor</em> platform, <em>and</em> use a lot of the 7-Mode features, it would arguably be impossible to do cleanly <em>and</em> maintain full functionality (at a bare <em>minimum</em> you’d lose all your snaps and clones — and some customers have several <em>years’</em> worth of backup data in SnapVault — try asking them to give <em>that</em> up).</p><p>This is true for all competitor platforms: Someone using specific features, scripts, tools, snaps, clones etc. on <em>any</em> platform, will find it almost <em>impossible</em> to cleanly migrate to a <em>different</em> platform. I don’t care who makes it. Doesn’t matter. Can you cleanly move from VMware + VMware snaps to Hyper-V <em>and retain the snaps</em>?</p><p><strong>Backup/clone retention is really the major challenge here — for <em>other</em> vendors. Do some research and see how frequently customers switch <em>backup</em> platforms… :) We can move snaps etc. from 7-Mode to cDOT just fine :)</strong></p><p>The less features you use, the easier the migration and acclimatization to new stuff becomes, but the less value you are getting out of any given product.</p><p>Call it vendor lock-in if you must, but it’s merely a side effect of using any given device to its full potential.</p><p>The reality: <strong>it is incredibly <em>easier</em> to move from 7-Mode to cDOT than from 7-Mode to other vendor products</strong>. Here’s why…</p><h4>What programs and tools are offered by NetApp to make transition easy and quick?</h4><p>Initially, migration of a <em>complex</em> installation <em>was</em> harder. But we’ve been doing this a while now, and can do the following to make things much easier:</p><ol><li>The very cool 7MTT (7-Mode Transition Tool). This is an automation tool we keep rapidly enhancing that <em>dramatically</em> simplifies migrations of complex environments from 7-Mode to cDOT. Any time and effort analysis that ignores how this tool works is quite simply a flawed and incomplete analysis.</li><li>After migrating to a new cDOT system, you <em>can</em> take your old 7-Mode gear and convert it to cDOT (another thing that’s impossible with a competitor — you can’t move from, say, a VNX to a VMAX and then <em>convert</em> the VNX to a VMAX).</li><li>As of cDOT 8.3.0: We made SnapMirror replication work for <em>all</em> protocols from 7-Mode to cDOT! This is the fundamental way we can easily move over not just the baseline data but also all the snaps, clones etc. Extremely important, and something that moving to a competitor would simply be <em>impossible</em> to carry forward.</li><li>As of cDOT 8.3.2 we are allowing something pretty amazing: CFT (Copy Free Transition). Which does exactly what the name suggests: <em>Allows not having to move any data over to cDOT</em>! It’s a combination ONTAP and 7MTT feature, and allows disconnecting the shelves from 7-Mode controllers and re-attaching them to cDOT controllers, and thereby converting even a gigantic system in practically no time. See <a href="http://www.netapp.com/us/media/7mtt_cft_cablingguide.pdf">here</a> for a quick guide, <a href="https://whyistheinternetbroken.wordpress.com/2015/11/25/introducing-copy-free-transition/">here</a> for a great blog post.</li></ol><p>And before I forget…</p><h4>What about migrating from <em>other</em> vendors to cDOT?</h4><p>It cuts both ways — anything less would be unfair. Not only is it <em>far</em> easier to move from 7-Mode to cDOT than to a competitor, it’s also <strong>easy to move from a <em>competitor</em> to cDOT! Since it’s all about growth, and that’s the only way real growth happens.</strong></p><p>As of version 8.3.1 we have what’s called <strong>Online Foreign LUN Import (FLI)</strong>. See link <a href="http://www.netapp.com/us/media/tr-4380.pdf">here</a>. <em>It’s all included</em> — no special licenses needed.</p><p>With Online FLI we can migrate LUNs from other arrays <em>and</em> maintain maximum uptime during the migration (a cutover is needed at some point of course but that’s quick).</p><p>And all this we do <em>without</em> external “helper” gear or special software tools.</p><p>In the case of NAS migrations, we have the free and incredibly cool XCP software that can migrate things like high file count environments <strong>25–100x</strong> <strong>faster</strong> than traditional methods. Check it out <a href="https://xcp.netapp.com/">here</a>.</p><h4>In summary</h4><p>I hardly expect to change the minds of people suffering from acute confirmation bias (I <em>wish</em> I could say “you know who you are”, not knowing you are afflicted is the major problem), but hopefully the more level-headed among us should recognize by now that:</p><ul><li>7-Mode to cDOT migrations are extremely straightforward in all but the most complex and custom environments</li><li>Those same complex environments would find it impossible to transparently migrate to anything else anyway</li><li>Backups/clones is one of those things that complicates migrations for any vendor — ONTAP happens to be used by a lot of customers to handle backups as part of its core value prop</li><li>NetApp provides <em>extremely</em> powerful tools to help with migrations from 7-Mode to cDOT <em>and</em> from competitors to cDOT (with amazing tools for both SAN <em>and</em> NAS!) that will <em>also</em> handle the backups/clones/archives!</li><li>The grass isn’t always greener on the other side — The transition from 7-Mode to cDOT is the first time NetApp has asked customers to do anything that major in over 20 years. Other, especially younger, vendors haven’t even seen a <em>truly major</em> code change yet. How will they react to such a thing? NetApp is handling it just fine :)</li></ul><p>D</p><p>Technorati Tags: <a href="http://technorati.com/tag/ONTAP">ONTAP</a>, <a href="http://technorati.com/tag/cDOT">cDOT</a>, <a href="http://technorati.com/tag/7-mode">7-mode</a>, <a href="http://technorati.com/tag/Transition">Transition</a>, <a href="http://technorati.com/tag/Migration">Migration</a>, <a href="http://technorati.com/tag/FLI">FLI</a>, <a href="http://technorati.com/tag/7MTT">7MTT</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19ceb0be8804" width="1" height="1" alt=""><hr><p><a href="https://medium.com/recoverymonkey/7-mode-to-clustered-ontap-transition-19ceb0be8804">7-Mode to Clustered ONTAP Transition</a> was originally published in <a href="https://medium.com/recoverymonkey">RecoveryMonkey</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>