A few weeks ago VMware released a blast at Amazon's recently released total cost of ownership (TCO) bake-off model that compared AWS cloud services with the on-premise vSphere suite of virtualization products. VMware called the AWS online tool "biased and inaccurate."
Oh my! That's sort of like Manet criticizing Seurat because his dots are too small.
The truth is, as a rule vendor-based TCO, ROI, and Value models are biased and inaccurate. Every major and many minor IT providers have focused on TCO as a key function in their marketing and sales strategies. They often become the basis for ROI projections and business case justifications. And virtually all of their models are biased and inaccurate "out of the box."
I have reviewed dozens of vendor models over the years. I have also used the VMware model extensively in the field and as a developer. My team and I have also dug into the AWS model. Biased and inaccurate, every one of them.
So where do the biases and inaccuracies lie? (No pun intended.)
All of these models are based on spreadsheets, some of these are also modified to work online. Errors can be in the logic, the arithmetic, the output, the complexity or the assumptions.
The assumptions are a likely spot for inaccuracies and one that VMware targets in it criticism of Amazon. There is a litany of allegations ranging from inaccurate server numbers to useful life of the installed base to having any installed base at all. These assumptions come from "industry data", "field experience" and "expert knowledge." Very few of these assumptions are documented. Proceed with caution when accepting these numbers as valid for your environment.
Many of these models are very complex with multiple tabs, many links between tabs, scalars, hidden tables and factors that can easily be broken during development or in field use. The more complex, the greater potential for arithmetic or logic errors that might be difficult to isolate.
So what does an IT decision maker to do?
Here are five absolutely necessary steps for buyers that will make a vendor-based TCO model and your business justification case to senior management more defensible:
1. Know your own numbers and take ownership of them. Vendor assumptions are really just placeholders for customer specific costs. Every one of them should be replaced with your numbers if possible. I gained some notoriety at Gartner for two things. First I created the first IT TCO model for PCs. Then, on stage in front of 100s of clients, I stated that the Gartner numbers were B.S. -- that we just made them up! The lesson here was that any cost or time estimates are biased and inaccurate unless you own them.
2. Challenge the models. The definition of IT TCO is: A holistic view of IT costs across the enterprise over time. If the model does not consider relevant line items, such as your installed base, your current skill sets, or change management maturity, ask where they are and work with the vendor to provide a complete picture.
3. Determine if an objective third party has vetted the models. Most models are created by marketing, sales or field engineering that may introduce biases and inaccuracies. Models developed by a contracted specialist are still subject to biasing by the paying party.
4. Spend the time needed to have a defensible model that can be sold up the line as part of a business case justification. Ask for the spreadsheet model to internally review and test. These are also great for what-if analysis that might not be part of the original presentation. If the vendor won't leave the model, lock the vendor's value modeler in a room and do a full cavity inspection of the tool.
5. Follow up on the costs and benefits through the projected life of the business plan, or at least through the projected payback period. This is both validating the model and a valuable tracking tool for the implementation where corrections can be made to keep the project on track.
Most, if not all, large IT investments are scrutinized at the highest levels and even smaller spending requests are challenged by finance, procurement or other stakeholders. Regretably, most IT departments do not have the in-house resources to develop TCO or ROI analyses for their projects. Nor do they often bring in outside consultants to create business cases. Therefore, the vendor is charged with running the numbers, which they will be happy to do. I am amazed at how often the actual numbers are unknown by the customer, vendor defaults are taken as gospel, and they often fail to pass muster in the enterprise sign-off process.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.