Based on the number of environments I’ve seen in the last couple of months, I can definitely confirm that the DACH region and especially Germany is still the land of synchronously mirrored storage systems.
While this kind of design offers a couple of valid advantages, it on the same time introduces a couple of challenges. In my option the biggest challenge is the price tag that comes with a synchronously mirrored storage.
In the end you have to pay not only for the virtualization layer and its features that runs on top of the actual storage, but also for twice the number of controllers, spinning disks and Flash devices. Not to speak of other things like maintenance, power & cooling, etc.
But there is another important factor I would like to cover in this post, the performance penalty.
Obviously nobody will go all-flash throughout the whole datacenter as nobody can really afford that. So the solution is to use a rather small fraction of Flash compared to the total storage capacity. And it actually doesn’t matter if we talk about a handful of Flash devices for existing arrays or All Flash Arrays, the problem is pretty much the same.
There will always be a gap between the individual device performance and the net performance the storage system is able to deliver. This could have multiple reasons like limited CPU performance, data services, RAID overhead, etc. Let’s take a closer look at it in the scenario of a synchronously mirrored storage.
The first performance loss comes into play when protecting the local SSDs with some form of RAID. Depending on the number of devices this is often a RAID1 or RAID10, rather rarely a RAID5. So you add like four SSDs to the local system and what do you get out of it? Just 50% of the performance & Flash capacity. So instead of getting 100k write IOPS you get like 50k and just 1.6TB of Flash capacity.
And it won’t get better, even if you add the exact same number of SSDs to the array at the other site, the usable capacity will still be just 1.6TB, simply because of the synchronous mirror.
When it comes to read performance, things look a bit better but the capacity is still just 1.6TB. Depending on the storage controller and storage virtualization layer the system is maybe able to read from multiple device simultaneously and can read independently on both sites to get some more IOPS out of the system.
But will you be able to really consume all those IOPS?
To be able to utilize all those IOPS it requires consumers in form of application that could request all those IOPS. But what happens when you move a 1TB database onto a Flash volume? The capacity is gone and you can think about what other workloads you could move onto the remaining space. But is it likely that those workloads really request all those IOPS? Probably not. So there is a bunch of IOPS left on the road that can’t be used by other applications.
Because of the costs for Flash in storage arrays, many companies consider using features like Auto-Tiering to use Flash within a bigger pool of storage. This way you could keep just the really hot data on Flash, so that every application could get a small piece of the cake. Sounds good at first. But do you know all of your apps good enough to know if the Auto-Tiering should consider only reads? Maybe writes? Or both? In which intervals will blocks be moved around? Are the blocks still hot when they got moved hours later? You probably get the point I’m making here. And let’s not forget those features are usually not free of charge at all.
Is there a way to do it more efficiently?
Using FVP as acceleration layer across multiple ESXi hosts that could leverage those SSD drives to scale-out the performance with every single device you add. This would give you about 6.4TB of Flash capacity for an acceleration at raw device performance to accelerate a wide range of applications.
The storage system wouldn’t be replaced in this design and would still provide the pooled SAS & SATA capacity, in conjunction with some other data services.
With FVP in the picture, there would be even an offloading of IOPS from the spinning disks up to the Flash devices within the hosts.
Another advantage of having the low latency devices as close to the applications as possible is that this will remove a couple of shared resources and obviously the distance between the virtual machine and the storage system.
I want to wrap up this post with a question: Is a synchronously mirrored storage, at the end of the line, the right place for Flash?