The manager of the accounting office calls up and gets the young energetic programmer on the phone. What can I do for you today the programmer says? Well the manager says I do not see the correct figures in the xxx report. The number should be ten times what is reported. Yes I see what you are saying I will have it straightened out in a few minutes. The programmer gets to work and in no time the figures in the production / development system are 10 times what they where earlier.

It took us a week to straighten up the production data. Had the programmer run the script in a development and test system first and proofed the concept we would have not spent 3 programmers time for a week getting the data straightened out.

Moral of the story is that nothing is put into test unless it has been proofed on the development machine. Nothing goes into production unless the user has run the process on the test machine. For your sanity create a development and test environment.

Ultimately, you can’t run high availability production infrastructure unless it is a controlled environment. Change is the cause of most problems, and well run production environments control change, and are careful in the introduction of each change. Development, by its very nature, is a changing environment. Testing environments begin at the lowest level (ie unit testing) as dynamic, changing environments, and end up at the highest level (ie regression testing) as production mirrored / stable baseline environments.

Ok, all nice stuff…summary: if you need high availability in production (or anything remotely approaching that) then you need a stable environment – network; storage; processors; OS DBMS and app software stack rev levels – all controlled and managed. If you can map development into that environment and provide (guarantee) a logically separate environment – then it can work. If not, then you need to ask whether the risk is worth the cost saving, and if not, show the risk exposure in dollars…cost of downtime; cost to recover the environment and data to a known and stable point; cost to bring the data up to date…
All the best…

Find or figure out how much production network downtime will cost (give best / worst case scenarios) per minute. Typically, the figure ranges from the thousands to the tens of thousands of dollars (assuming of course your location is in the US – but location is irrelevant).

Next, again giving best and worst cases, figure out how long it would reasonably take to address a given failure/issue/conflict in the test/development environment which adversely affects the production environment.

Once you have this information, analyse how much it would cost to implement the separation you desire (hardware cost, software cost, human capital, maintenance)… For example, one DC at 10,000.00 (include operating system and required software, but itemize your list), salary or hourly rate of any time needed to configure the system) and the same hourly or salary information for system maintenance (include time spent fior patching, hardware replacement and scheduled general maintenance).

Using these figures, (ie: production downtime at $2,000 – 3,500.00 per minute times 30 – 40 minutes gives an estimated downtime cost of between $40,000.00 and $140,000.00; the cost of separating out the production domin from the test domain (do I have that backward???)) present this cost analysis to the CIO/CEO/Owner or whomever is the party who’s responsibility it is to answer the the company stakeholders for lost income…..

When presented with hard facts in a non-technical manner (ie: in language the “bean counters” would understand – my apologies to all you bean counters out there), you stand a much better chance of achieving the results you desire…

Of course, the figures will vary for each environment.