Very amazing what you can sell to most customers today. Even some of the large users have (once again?) lost the view and is just reading the marketing / advertising numbers or getting them blindly from a vendor rep.

The real problem is, seen this before, companies don’t realize that a system is a sum of many things and especially the cost/benefit is even more, much more complicated than just a benchmark result. Anyone remember those MIPS fights (even long before PC’s existed), it just comes and goes, again and again?

It is a catch 22 for smaller companies which don’t have the resources or can’t influence the vendors, should be no problem for large corporations but.. My advice for smaller companies would be, read the reports with (big) grain of salt, do your own homework – even a little helps, but the best, be active in computer user groups and communities, etc – maybe the companies don’t share their information very easily but the developers, analysts, whatever are not so shy. For a large company it is no brainer, any vendor arranges a benchmark environment for your application when they hear words as 10K txs/second or nnn number of nodes/clusters/whatever. Unfortunately (not again!) even large companies seem to have forgotten that your infrastructure has to match, otherwise it doesn’t matter how fast you execute something or how much throughput one of the components have. IMHO capacity planning has gone worse, not better lately and already causing problems everywhere.