My move to pernixdata plus FVP 2.0 is here!

As announced earlier on Twitter I’m currently in a rather short transition phase of moving away from SHE Informationstechnologie AG over to


I’ll join an awesome team at pernixdata as System Engineer to support the DACH region.

The first question that might come up why pernixdata?

I was following pernixdata and FVP right from the start because I was of the opinion that it could be really a game changer. Decoupling storage performance from capacity in a transparent way sounded really awesome. And now I’m a part of the team and I think I hit a pretty good timing. Just today we announced the GA of FVP 2.0 with a lot of new features:

  • Distributed Fault Tolerant Memory (DFTM)
  • Optimize any storage device—NFS, Block storage like iSCSI, Fibre Channel, FCoE as well as local datastores (DAS)
  • Fault Domains
  • Adaptive network compression

FVP 2.0 is now also available in different editions, FVP Enterprise (including Subscription), Standard, VDI and Essentials Plus. More details can be found here. Frank already covered some of the features on his blog in more detail. So I’m not only excited about the move but also about FVP 2.0!

What about my blog?

I will definitely continue my blog but no doubt about that I’m going to cover primarily FVP as logical consequence of working with the product on a daily basis. But I also have some more cool ideas in mind which I won’t spoil at this point. I plan to continue writing in English but if you would like to see some German content or information, don’t hesitate and let me know.

VMworld 2014 – Barcelona2014NoLimits

In case you will be in Barcelona to attend VMworld 2014 make sure to stop by at our booth, I would be happy to meet you there and to provide you with some more details about FVP.


My take on All Flash Arrays vs. Server Side Caching

This post is about my personal view as System Engineer to the question “All Flash Arrays (AFA) vs. Server Side Caching (SSC)”. A while ago somebody asked me exactly this question and to be honest I had no proper answer back then, simply because I have never thought about it! So I’ve spent some time thinking about it and this is the result.

One of the first factors I’ve had in mind were costs. No matter how cool, innovative or trendy a technology might be, it always comes down to costs. Even if I think AFA in general are a cool stuff, they still have their price. You basically pay for hardware (SSDs + controllers), for the software (intelligence) that powers the array and of course for a maintenance contract. The point is how much capacity & performance you will get for this price? You usually get rather low physical capacity and you have to rely on data reduction technologies and a good reduction ratio to get as much as possible logical space out of that box. Don’t get me wrong, that’s the way it is because otherwise All Flash Arrays would be even more expensive.

In my opinion the SSC approach sets the lower hurdles to get access to the technology. Some reasons might be:

  • No need for dedicated controllers  (even tough for local RAID controllers in the hypervisor if not already in place) which reduces initial investment costs
  • SSDs of your choice and I think this one can make a big difference. Unfortunately I’ve seen vendors charging 5 times more for a single SSD than the actual street price!
  • A single SSD (not even for all hosts) is sufficient to start with
  • Less rack space, power consumption and cooling

In turn you could argue against a SSC approach because it doesn’t provide any persistent storage. But because a caching layer can reduce the total number of IOPS my primary storage has to process, I can scale down I/O requirements which again saves money. And it definitely scales down better to SMB customers which have rather small environments and tight budgets.

Licensing is an interesting point, many all Flash vendors offer an all-inclusive bundle compared to annual, capacity or host based licensing offered by the SSC provider. So with a growing number of hosts the AFA pricing gets more interesting.

Since both solutions can accelerate reads AND writes, especially writes need some special attention to avoid data loss. Server Side Caching “simply” replicates IOs to other Flash devices within the cluster to withstand host and device failures. So this could be seen as network RAID10, assuming two copies of a particular block. Whereas AFAs usually use some form of dual parity RAID to distribute data across SSDs and to protect your data. This is pretty efficient if you ask me, as long we are talking about non-stretched clusters.

For some businesses 10GbE could be a problem, but from what I’ve seen there are also solutions available which optimize replication traffic (when using write back caching) to be able to run even on 1GbE. Going with an AFA will also require having a proper 10GbE or 8Gbs + Fibre Channel SAN in place to avoid bottle necks. A plus for AFAs is that 8GBs FC SANs are more common than 10GbE Ethernet networks, at least in the German SMB market.

When it comes to data placement the SSC can help to accelerate a broad range of workloads; basically all virtual workloads could be speed up. Depending on the number of VMs per host, this of course would require multiple SSDs or PCIe based Flash devices to provide sufficient caching space and performance.

An All Flash Array on the other hand needs a decision which workload should be moved over to the new array, because I assume not many people will be able to replace their current SAN completely with an All Flash solution anytime soon. Another option for AFAs would be to be placed underneath the umbrella of a storage virtualization layer like DataCore’s SANsymphony-V with the Automated Storage Tiering feature. Even if SSV has pretty good storage tiering implementation which can move blocks within minutes rather than just once a day, a block accessed on a Tier > 1 will be considerably slower than from flash. And to be a moved up to a faster tier, a block needs a certain number read/write hits to heat up and to be finally moved. That sounds probably a bit too negative, but it’s just the way a storage tiering approach works.

When expanding your infrastructure scalability is an important factor. There are all Flash solutions with shared nothing architectures which scale out very well or other approaches which allow transparently scaling up without any impact. But that’s why I’ve said in my previous post, AFA =! AFA details matter!

The SSC approach causes less overhead like zoning, LUN masking when scaling out and doesn’t require any rack space and less power and cooling. But it could require downtime of the host depending if hardware changes like a new RAID controller are required. But in times of vMotion this isn’t really a problem. Scaling the SSC layer offers more granularity since I can slowly scale by purchasing a certain number SSDs of the shelf and licenses, whereas scaling out the AFA means buying a completely new array/block.

And what about the impact of a failure? Both approaches will be able to handle drive and host/controller failures. An outage of a dual controller array is rather unlikely but can happen; I’ve seen things like that before. But also kernel driver/server side software can go crazy but this will most likely only impact the host it’s running on and not all connected hosts. So SSC compared to a dual controller AFA is less vulnerable to impact big portions of the production workload. Compared to a scale-out (shared nothing) design it’s quite similar.

Host overhead is there but rather minimal. When comparing an AFA, which of course causes no overhead, to a SSC solution based on a kernel module. Especially people here in Germany tend to be more conservative when it comes to overprovisioning, so many environments I work with are usually < 20% CPU utilization. Usually RAM is the number one resource than runs out very quickly. So if you ask me it’s important to have a solution which lets you chose between RAM or Flash based devices as caching medium.

As always I’m sure there are folks that won’t agree with some of my arguments, so feel free to comment and share your thoughts!

My thoughts about Server Side Caching

To follow up my last post about different storage technologies and approaches, I want to dive deeper into one of those. This time it’s all about server side caching which is rather new in the way it now can be implemented in virtual environments and how it can accelerate a broad range of applications.

Server side caching technologies leverage local resources right inside the hypervisor like SSDs, PCIe Flash cards or even RAM to drive down application (VM) latency which in the end is the key performance metric that really matters. The idea is to transparently plug into the virtual machines I/O path and to intercept their I/Os, write them to Flash or RAM and acknowledge the I/O right back to the VM. So virtual machines see real Flash/RAM performance and latency.

Since this technology hasn’t gained as much as awareness it actually deserves, I think it’s worth it to point out some of the key benefits:

Decouples virtual machine performance (IOPS/Latency) from primary storage

I have no clue how much time I’ve spent just playing with spindle counts, RAID variants, vendor discussions just to hit the requires IOPS for new storage systems. Even with a Storage Tiering approach and Flash as Tier 1 it still can be a challenging because all of a sudden the IOPS/GB ratio between different storage tiers comes into play. How much Tier 1 do I need? Should I go with just two Tiers or will my pool be big enough to create a third Tier 3 with slow SATA disks and so forth.

This is especially important for SMB customers, who often have to use basic block storage systems with rather low spindle counts. I’ve seen those system to become a bottle neck very fast because people tend to underestimate the I/O demand of applications like SQL, Exchange, etc. even in smaller environments.

Allows to freely choose which Flash devices to use

I’ve seen storage vendors using of the shelve SSDs but charging 5 times more than the actual street price. No doubt that people still hesitate to pay this price! But this approach lets you choose what to plug into your host. You can choose the SSD vendor, capacity with the best $/GB or IOPS/GB. This can significantly drive down the costs to implement flash. If you go with devices from the OEM of your existing server equipment the price will certainly be higher, but you don’t need to worry about compatibility, RMA processes etc. basically you get the services/SLAs you are used to.

Efficient & Fault Tolerant

Since the local device don’t need to be RAIDed you don’t lose as much capacity as in external storage systems. Only in case the write back cache is enabled, the cache software has to replicate every I/O to a second (or even third) host within the cluster to ensure fault tolerance against host and device failures. If you use the layer as read cache only, there is basically no loss and the full capacity can be used to cache I/Os.

It can scale out

If you add additional hosts to a virtual environment you CAN add them with or without local flash devices. If your current hosts are sufficient to run your most critical enterprise applications and they are already accelerated you don’t necessarily have to add flash to those new hosts but you can of course! Just keep failover scenarios in mind. And in case you want every VM to be accelerated you can easily scale out by adding new hosts with local flash devices (or additional RAM).

Management & Ease of Use

All solutions I’ve seen so far did a very good job in terms of usability and integration into VMware’s vSphere & Web Client. So setup and management don’t cause a big overhead and don’t require a special training to be able to install and use the product.

However you may ask yourself “What about my primary storage system and the cache software, will they get along? And how about all the dirty data inside the cache?”

Of course this in turn means that the local caching devices are full of “dirty data” and the caching software is responsible to take care of it in terms of fault redundancy (I/O replication between nodes) and to commit (de-stage) those I/Os over time to a persistent storage system.

This has can have a huge positive impact to your primary storage, simply because it can significantly reduce the total number of IOPs your SAN or NAS has to process. Even if the caching layer acts just as read cache, due to the size of Flash devices the data (individual blocks or content) can reside there way longer than in just some GB of NVRAM inside the storage array.

If also the write back cache functionality is being used the caching software can eliminate redundant writes of the same block and only commit the latest version of a block down to the array. Both scenarios can significantly reduce the total number of IOPS which allows some conclusions:

  • Bring primary storage arrays down to a healthy state in case it’s already at its limits
  • Get more out of existing assets
  • And even if some folks don’t like this one, consider smaller arrays due to the reduced number of IOPS it has to process

But nothing comes for free. The caching software needs to be purchased, no doubt about the Flash devices and probably, if your hosts are just equipped with SD cards (diskless config.), a proper RAID controller will also be required. RAM can really help here, so don’t rule it out too quickly.

In my opinion Server Side Caching is a smart way to speed things up and to create room when it comes to choosing a new primary storage, since IOPS & latency is not longer your primary concern. Guess the storage folks don’t like to hear this but if SSC gets enough awareness and a growing install base it can really steal a big peace of the storage market cake.

That’s it for now. I’m planning to follow this one up with two additional posts. A hands-on deep dive on one of the current solutions available on the market as well as a post about the question “server side caching vs. All Flash Arrays” so stay tuned.