In my last dispatch I had spoken about the recent trend of organizations hiring CIOs as part time advisors to help put their IT program on track. This concept is certainly catching up and some organizations have already started experimenting with this service. While this seems to be working, there exist quite a few rough edges which need to be evened out.
The service can come in various flavors and companies define the scope depending on the complexity of their problem and their level of dissatisfaction with their current IT set up. Based on my experience and my conversations with fellow professionals, I have found the variations in scope which lets me classify the service into the following types:
- As an assessor
Here the CEO wants the CIO to come and review the entire operations, point out things that are in order and those that are not, prepare a report, and also specify the budget for the next one year. The need arises because the CEO may often be irked by frequent requests for spends on IT and is not sure if the IT is headed in the right direction. If CEO is satisfied with the CIO-engagement, he may ask the CIO to come over again after six months and assess if the IT tasks are running as per plan and to make a revised budget based on progress till then.
- As an advisor
Here the CEO may not be dissatisfied with IT, but is still not happy enough and wants IT to acquire a proper direction. He therefore asks his CIO friend to act as an advisor and be a member of the IT steering committee which may meet once in a month or so. The CIO is expected to either present a status report or join in the review of various projects and present his opinion as an expert. Wherever necessary, he can ask the advisor CIO to undertake special study or seek some involvement in a specific project. This can work well with the part time CIO devoting only as much time as is required by the organization.
- As an advisor cum reviewer
The need of the organization is somewhat different here. There is no IT manager in play and therefore the whole IT program has no direction and the function has the limited role of maintaining the infrastructure and developing a few miscellaneous applications. The CEO wants the external CIO to advise the company on the road map for IT and also regularly review the working of the function to ensure the processes and delivery mechanisms are properly managed. For this the CIO may have to spend a fixed number of days, say two days a week to monitor the progress on various projects. He is not expected to manage the day-to-day operations but is expected to put systems in place, define performance parameters, and monitor them every week.
- As a CIO, almost
Some companies have very little or no confidence in their IT team and therefore want someone to take over the entire responsibility. They look for a senior professional who has been a CIO to advise them on IT and also help manage the show. The CIO then spends some time to assess the requirements and then posts his own person to devote two or three days in a week to ensure that operations are smooth. The CIO does a periodic review and reports to the management and the company staff reports to him.
Which option is the best?
This depends entirely on the company’s requirement and plans. The objective here is to use the right resource and in optimal measure so that the plans of the organization are met. Being a new facility which has recently gathered momentum, I am not quite sure which model is setting the trend but from my recent interactions it seems the option 3, ‘advisor cum reviewer’ is getting some traction. It is up to the CIOs and IT service providers to develop this market and make the offering more attractive.
This is a new concept that has been doing rounds for the last couple of years. The idea looks good and sounds logical too. There are several other phrases by which this concept is referred to, such as ‘CIO as a service’, ‘Shared CIO’, ‘Virtual CIO’, ‘Outsourced CIO’, etc.
‘CIO on Demand’ refers specifically to the services that can be made available to a host of small and medium organizations who struggle with their IT program. These organizations have difficulty in getting good IT managers and even if they manage to grab a worthy professional, he leaves them soon as he spots greener pastures. They cannot afford to appoint consulting houses to run their IT programs as it gets a little too expensive. In such circumstances they make do with a low level IT officer who just operates and manages the IT set up.
Many of these companies have ambitious plans of growth and also have the next generation of owners/ directors at the helm who are not shy of technology. The managements here want more out of IT and feel terribly constrained in the absence of right people required to run and manage the show. These organizations may be at different levels of IT maturity but most of them wish that their IT had better direction and wonder if the money spent on IT is right.
How can this be addressed?
These companies cannot afford consulting houses neither are they sure about the complete outsourcing of the IT function as advocated by some IT service organizations. They cannot also recruit and retain senior IT managers. They can however look for senior IT professionals (ex-CIOs), those who have either retired and are available for consulting or CIOs who have left their regular job to become freelance consultants. Instead of a full time engagement, they can devote time as much as is necessary to guide the IT program to fulfill management needs & expectations. Such a service has not yet gained ground yet and many of the users organizations are not aware nor have not examined this option. Such services are however affordable.
Who is talking about this concept?
I have seen some hardware and software vendors exploring this option in an effort to place one of their chosen persons there, a few experienced CIOs who have a contract or two, a few CIOs who want to kick their jobs and want to start off on their own, or IT service firms who suddenly have sensed this as an opportunity and are building their portfolio by trying to string in a few CIOs available.
Is the going easy?
It is perhaps not that easy. Many have spoken to me wondering how to tap such opportunities. This kind of service is new and no one seems to have clear answers. Some say they want to create a portal offering free counseling to attract customers, some speak about building a team or a panel of CIOs and then publicize through mailers or through a special site, some plan to add this as a strength on offer to the firm’s profile and I have also heard of a few CIOs coming together to form a loose association thus offering varied expertise related to specific industry segments or technology areas.
If we are confident that this service has potential, then, in my opinion, companies that fall into this segment should be educated and made aware of such a service being available. One way could be to tie-up with industry associations and spread awareness through them, or one can arranging seminars where business heads from these participate. Advantages could be two-fold; these companies can derive benefit from such as a service and at the same time useful resource available in the community can be utilized.
All of us like attending seminars or short term courses once in a while, don’t we? This gives us a break from mundane activities and a chance to be away to network with fellow professionals. The lucky ones get an outing every now and then but some are not as fortunate. I am reminded of a senior official in an organization that I worked for, and he was one of the fortunate ones. He was intelligent, but wasn’t one of those efficient and effective managers who are sought after, nor did he get along well with this peers and seniors. So away from routine work, he was assigned as an officer on special duty with no clue of what his responsibilities were. The organization heads were often flooded with requests from industry associations, management institutes and vendors asking them to nominate someone for their seminars and short term courses. Out of obligation, the directors looked around and finding no one free for the purpose would send our man on Fridays to attend the seminars. That fortunate soul spent half of his time attending such sessions and gained knowledge which was never put to use by the business. This speaks of the mis-utilization of learning opportunities in many corporates.
The CIO faces this situation often, either with respect to his own training or for the officers and staff reporting to him. He is torn between alternatives and tries to balance the factors in question. Let us look at some of these situations.
The HR intervention
The human resources function has their task cut out clearly. They have their own budgets and targets to meet, for example, the number of training programs to hold, total training hours achieved, average training hours per employee met etc. So they fix up seminars & courses and then look around for candidates to fill in for the batches. Quite often, without my knowledge, I have been nominated for management development programs, soft skill development (like communication skills), leadership development, negotiation skills or as a representative to an industry forum. Similar pressure is applied when they ask our officers to be spared for training seminars they have arranged. Constrained by the work in hand, CIOs may nominate someone who can easily be spared. This story thus gets repeated more often than we are aware of.
The CIO too has his own sphere of influence. He has friends in the industry, whether it is vendors, service providers or media agencies. Seminars and events take place at a monotonous frequency and the CIO cannot possibly attend all of them. He is however obligated and cannot afford to neglect the invite that may affect his relationships and therefore assigns one or two of his associates to attend the events. He therefore plays the same game with his associates that his management plays on him.
Candidate’s own needs
In most organizations there is a process of asking the employee of his training needs so that steps can be taken during the year to meet those expectations. Some term it as ‘development need’ for the employee so as to help him function more effectively and is directly linked to the role that he has to play. The needs may be genuine but at other times it has more to do with the candidate’s desire to learn something which could make him marketable to seek opportunities elsewhere. The CIO has to play a balancing act here by striking down a few requests and agreeing to others.
Is there a right way ?
Yes, I would think so. We are normally consumed by the conventional practices followed by the functions and toe the line faithfully. It may sometimes be prudent to turn down requests for attending events if they do not seem to benefit us and at the same time pick up courage to ask for specific programs or seminars which would benefit the department. Breaking traditions is always a difficult task but is worth attempting.
It is true that information technology, as a discipline, has made steady progress over the years. From being a back-end operation, IT today has come on to the forefront, impacting work in all organizations. While companies are embracing IT like never before, IT as a function still struggles to stay relevant in many organizations.
Beyond the jargon
As the debate lingers on, CIOs speak out about their achievements and about how they bring value to their organizations. We often hear terms like business alignment, IT strategy, talking business language, business orientation, reforms / process re-engineering, leadership qualities, cross-functional orientation, etc. They sound great, no doubt; but different workplaces require different solutions and the CIO has to do what is relevant for the situation.
For instance, large organizations may require a different focus than a medium sized one; similarly banking companies may have different priorities than conventional manufacturing companies. I have come across quite a few of them during days of my regular employment and subsequently as a consultant. I would like to classify companies and their requirements in the following four types:
• Companies that want a ‘good’ IT budget:
These are essentially the small and medium companies who have IT facilities but have no grand ambitions. The proprietor or director is troubled with the IT manager’s regular demand for more PCs, servers, switches, and the likes, and wants the spends on IT to be regulated. He looks for some advisor who would tell him how much of IT he should have and help make a proper IT budget for the company so that he can manage his meager finances better.
• Companies that want a streamlined IT:
These are medium-sized organizations and have much better intentions with respect to IT. They are serious about IT and want a proper direction for their IT program but are often plagued with their inability to get a good, senior IT professional to manage the function. They look for external assistance in the form of consultants or advisors who can draw an IT Plan for the next year and also set a direction for IT over a 3-year period. They may not have exalted ideas of using IT as a competitive weapon or making IT a driver for their business, but have down-to-earth views of using IT efficiently.
• Companies that understand the strategic value of IT:
These are from the medium and large segment of companies belonging to the conventional manufacturing and distribution, whether in the discrete or process manufacturing. They usually operate in a competitive environment and look for ways and means of increasing or retaining their market share. Efficiencies in operations, effectiveness in delivery, responding to vagaries of the market, and being alive to changing customer preferences are areas that they want to use IT for. They have a senior person to head their IT function and also seek external consulting assistance. They largely consider IT a strategic tool and insist on an IT strategy document and one that aligns with business goals and priorities. The CIO here does not really have an easy time.
• Companies that use IT as a differentiator:
Companies belonging to the BFSI segment, telecom, retail, airlines, rail / road travel and related service companies, have a different take on IT. For them IT is the lifeline and most of their offerings ride on their IT systems. They create differentiated products and services and make as their winning formula. They also offer personalized services to important customers and use IT to retain and enhance their customer base. CIOs in such industries have to be on their toes and create a flexible architecture to meet ever changing business environment.
No simple solution
A single solution or approach therefore is not apt for dealing with all situations. Importance of IT differs from one organization to another. I have often heard some people speaking of the winning formulas and advocating solutions to others based on their own successes. We don’t have to talk of an IT strategy or IT as a differentiator in all cases and IT solutions therefore need to be relevant to the situation the company is in.
I have often wondered why users clamor for more reports. It is as if they have been denied a fundamental right and would starve without them. When I look at some of these demands dispassionately I don’t see clear answers to the usefulness of many of these reports.
Let me share a story here which will explain the point. During a consulting engagement several years ago, I got lists of reports needed by various departments in the organization. These reports totaled to over 200 and I had a problem accommodating them. I then had to approach the CEO to find a way out. He agreed with me and clearly said that his managers do not know what they ask of and gave me permission to question them on the reports desired. I thought of a simple way, I added an additional column and asked the departmental heads to indicate the decisions or action that they take on each of these reports. Surprisingly, the number of reports dropped to half and on questioning further a good number from the remaining ones got withdrawn too. This experience taught me two lessons:
How much is enough
A report is meaningful only if it gives information that prompts one to take decision or action to correct a position, to build up further if the situation is favorable or not to do anything if the going is good. The decision could be either to do or not to do something but is a conscious action.
There is a difference between what users want and what they need. It is therefore important for us to understand the purpose for which users want the information. It is possible that we end up giving him more than what he really asked for.
A few suggestions
It is amazing to know how little information is actually used for decision making. A lot of information in the form of reports floats around and I come across only a few who ask for targeted information for dealing with a specific problem. I have generally adopted the following approach when addressing users’ demand for reports:
Use of query screens: I encourage users to make use of query screens and draw out information for any period or based on various other parameters and take a print-out only where necessary, instead of insisting on static reports.
Not printing large reports: I discourage them from printing of periodic large statements like the General Ledger, Party Ledger, Stock Ledger, etc. for the purposes of records. I ask them to take selective print-outs of individual accounts as and when necessary for review, meeting with customers, etc.
Exception reports: Rather than going in for routine reports I ask of them to seek information that will allow them to cut costs, review efficiencies, reduce shortages, etc. For example they can get reports on material due but not received, POs pending for authorization, customer outstanding beyond ‘x’ number of days, material shortage, etc. Such information can help them act to plug inefficiencies, reduce shortages and enhance profitability.
Analytics: There are immense possibilities with information in our hands. A lot of meaning stays hidden in the data which can be uncovered through analytics. Various trends not visible otherwise can be unearthed through BI tools and can help in drawing out information to help in decision making and for charting out path for further planning.
Extracting valuable information on business activities is crucial for improving business efficiency and to stay competitive in the markets. A lot of effort that goes waste in generating irrelevant information through routine reports can easily be channelized for providing intelligent and targeted information which can assist in improving performance in specific areas of business. Information is a useful resource and need to be harnessed well.
The one user demand that has never ebbed has been the need for more reports. In every organization that I have been in, this phenomenon has raised its head and claimed attention. Users always demand reports either as a specific need or on a perception that they have not got enough out of the system. Some ask for those reports that have been in vogue for years even though it may be in a state of disuse, while for some it is a feeling of deprivation as they perceive their getting less number of reports as compared with other users at the same level. They are but our customers and therefore have a right to be heard.
The term ‘MIS Reports’ (Management Information System reports) has been in use for many years now for describing statements that gives out information. As systems grew, more and more data got stored and linked and managers sought greater detail as well as greater abstraction to generate meaning from the raw, stored data. ‘MIS’ came into being as systems that provided managers with information about sales, inventories, and other data that would help in managing the enterprise. This has continued as a legacy and in these days of instant reporting, online queries and intense analytics demand for more and more of these reports look a bit out of place.
Types of users:
It has been interesting dealing with users with respect to information reports. Frankly, this requires patience, persuasion, understanding, humor and several other skills to keep the users happy and satisfied with the reports they get. I would classify users into the following types:
- Users who have no further demands: These users are a satisfied lot and think they have everything to take care of their requirements. When approached, they give a look of askance wondering why they are being asked for more. Life for them is stable and uncluttered and they certainly dislike anyone disturbing the serene environment.
- Users who have a laundry list of reports they want: These ever demanding users strongly feel that the number of reports they have should keep increasing as they progress in their work. Sometimes I liken them to tribal head of yore whose power was gauged by the number of skulls displayed in his hut. Some of the reports may not be of current relevance but you wouldn’t dare to take them off their list.
Many years ago, I came across a report on manpower numbers which was being sent to the CEO every month. On enquiry I found out that the CEO wanted this report at the beginning, during the project stage when he monitored the progress personally. Later as the organization grew, this report was not really needed but his personal assistant continued to file this faithfully.
I stopped the report and no one raised the flag for six months and then as the PA discovered the lapse he made noise full throttle questioning as to how a report for the CEO could be discontinued. I had then to seek audience of the CEO to have the report formally withdrawn. Usually, legacy practices run deep into the work practices and users hold these reports as their lifeline. Persuading them to let go of these reports often require us to dig deep into our skill reservoir to find out the best tools to win the user over.
- Users who have demands but know not what they need: These users are the most difficult to handle. They are evasive when it comes to defining requirements but still demand more out of the system. This is commonly experienced in organizations who implement ERP systems. Some such users make loud noises about ERP not delivering information reports for them. When asked to define their information needs they put up a counter demand asking us to showcase what the ERP can give them. They ask about the best practices we claim ERP brings and want those practices to be implemented rather than being asked what they want. Explaining to them that ERP has a large repository but that reports have to be tailored to address specific business pains faced by the organization, does not cut ice with them. In my opinion, the reason for such reaction, in most cases, is the fear of exposure or ignorance.
So here we are, trying to help users get targeted information so that they are able to bring in more efficiency and effectiveness in their areas of operation, but users keep raising their level of resistance or putting up a wall trying to ward off prying eyes from invading their domain. Tackling this needs our push, diplomacy and perseverance in order to succeed in our mission.
This is a familiar scene each day. I walk into my office in the morning with a vague sense of what I want to accomplish. Then I sit down, turn on my computer, and check my email. Time moves on and a few hours later, after fire-fighting and troubleshooting, solving various problems, and dealing with whatever is flung at me through my computer and phone, I can hardly recall what I had set out to do when I first turned on my computer. I sit back, tired and forlorn, looking for a solution for such challenging situations.
This scene repeats each day and I am so consumed by these routine tasks that I start cursing the company, the environment, the people, the work assigned and so on, but lay no blame on myself. I try to reason and find justification for my actions. I retire for the day with a resolve to complete some of the tasks next day but do no better in the day that follows.
After reading a book, ‘First Things First’ by Stephen Covey, and a couple of workshops later, I realized the folly of falling into this do-loop. Starting the day with a vague sense of purpose, I understood, doesn’t get me anywhere as I start reacting to whatever scene that unfolds in front of me. I am therefore led by the events that occur rather than I determining how the day should proceed for me.
The task list
The best thing therefore is to start the day with a clear sense of purpose and lay down for yourself matters that you think are important to attend to. In simple terms what you need to do is to write out a ‘to-do’ list. Set a plan for the day. Spend five minutes before turning on your computer in the morning to write down what you want to accomplish that day. Be realistic. Schedule time in your calendar to get each thing done, putting the harder tasks at the beginning of the day.
Just preparing the list is not enough; the tasks have to be executed too. Every hour, take a minute to stop what you’re doing, look at your list, and reflect on your last hour. Was it productive? What can you do to make the next hour productive? At the end of the day, as you are closing down your work, review your day and ask yourself what you were able to accomplish. If you have been unable to achieve much of what you set out to do, ask yourself – what will you do differently tomorrow to be more productive?
Prioritize your chores
How we spend our time usefully during the day is the key strategic decision. You may end up doing a lot of unimportant tasks and leave out key tasks because you did not have time enough to execute them. It is natural to pick up simple and miscellaneous tasks thinking that once you get them out of the way, you can concentrate on the important ones that need time.
These include disposing off all pending papers, going through the inbox to read/delete/forward mails to keep the inbox clean and stopping at your colleague’s desk to exchange pleasantries. The truth is that before you realize, most of the day gets over leaving very little time to work on larger tasks that have been waiting for your attention. These tasks therefore are put off for the next day and the same routine gets carried forward.
The task list therefore needs to be split into those tasks that are large and critical and others that could be important but not urgent. It is better to address the critical jobs first and the smaller tasks can fill in during the time available in the day. Towards the end, as you prepare to wind up for the day and are relaxed, you could clear your desk and the mailbox and get back home with a good sense of having accomplished what you had targeted and come back the next day with new targets.
I have often found my CIO friends unhappy, worried and agitated about the apparent lack of recognition of their efforts in their organizations. They feel ignored and side-stepped when not considered for promotion to the next level. I too have felt grumpy many times during my long career and I therefore empathize with their feelings.
Every employee has his career ambitions and works towards his goals. He strives hard and sincerely pursues his objectives but sometimes doesn’t get what he aspires for. There could be several reasons for his disappointment and some of them may be genuine. His expectations may be realistic at times and not so at other times and that determines his reaction during such situations. Recognition of CIOs work can come through higher increments, express appreciation of their work or being given enhanced roles and not necessarily through promotions alone. Promotions are often times governed by factors such as organization’s priorities, the thrust and focus and moderation across the enterprise. However, promotions are also denied because of unfair reasons such as biases, political factors, superior’s incompetence, etc.
A few difficult situations
I will describe a few situations from my experience, some in which I was directly affected and others where my colleagues got trapped.
Of differing perceptions: CIOs feel good about their work and think they have done enough to warrant recognition. They express this in the self-assessment form that organizations have as a part of the process. Perceptions of the boss or other assessors, however, could be different and herein lies the great disconnect that troubles the CIO. Wonderful work carried out in installing and implementing new software or new network or security device may appeal less to the superiors who may want to know the benefit that has accrued to business. Irrespective of who is right, the CIO would do well to look at the other side of the argument and learn from it where appropriate. He could examine his orientation to business needs and goals and work in alignment of the needs.
Of relationships: Implementing systems and bringing in changes affect people and their work routine. The CIO therefore has to deal with people and get their buy-in. Often good systems fall short of their promise because of inadequate cooperation from the end users. These negative vibes then act against the CIO though the CIO may sometimes not be aware of this factor and would feel victimized.
Uneasy relationship with the boss: The old saying ‘the boss is always right’ cannot sometimes be overlooked. Notwithstanding the fact that the boss is less competent, we still have to carry him along. We need not necessarily bend backwards to please him, but we could do without picking up a fight with him. Criticizing him may not get us far and we could do better to stick to our work. Some situations may really be difficult and CIO has to exercise his discretion in dealing with such situations.
The status of IT: Conventional organizations or traditional CEOs sometimes never grow up and continue to view IT as a sub-function not giving IT the desired status in the organization. I had once faced this situation and was not promoted in spite of having done well. My boss couldn’t explain but then I had a heart-to-heart chap with the CEO and he admitted that he was not in favor of giving IT a status equal to those of Finance, Sales and Manufacturing. I then had to look for greener pastures.
Dealing with the situation
I know such situations could be unfair on us but this is a reality and happens sometime or the other during our careers. However, on being denied a promotion it would be wrong of us to sulk and recede into a corner. I have seen people doubling their efforts in the following year, forcing the hands of superiors to give them the promotion they richly deserve. The opposite case was in my last organization, where the aggrieved person reacted negatively and hence has been languishing at the same level for the last two years.
It may be staid throwing in the towel and start seeking a change. We can’t always run away from a problem, can we? We have to face and fight the battle. I have often found CIOs jumping out in a hurry but regretting later akin to the popular phrase ‘from frying pan to fire’. It is better live to fight another day.
I realized the importance of documentation many years ago when I joined an organization to head its IT function. The previous IT head had left the organization a couple of months ago. The managing director called me over and voiced his expectation. He told me that all ground work had been done for ordering new set of servers and application packages and that I should act upon it soon. I promised to take a look at the situation and revert with plans.
However, when I sat in my department and rummaged through papers, I could not find much except notes on discussions with the vendor and details of configuration. For instance, there was no document showing an IT plan, applications to be developed / bought, functional areas to be covered, priority of tasks and justification for the equipment and software to be bought. When I went back to the boss expressing my helplessness in the absence of documentation, he was visibly annoyed and refused to discuss further. I then spent the next three months drawing out fresh plans and submitted them for approval.
Having learnt the lesson, I made it a point to always submit a handing over report to the management whenever I left any organization, which carried details on the IT set up, current status on various tasks, pending work, and the matters that would need attention in the following six months.
Consequences of poor documentation
Proper documentation is essential at every stage of our working and it doesn’t have to wait for a specific occasion. Documentation is simply a habit and a discipline and contrary to what many think, it does not require great effort. Good practices speak of creating documentation alongside and not to wait for the entire work to be over. I have faced several embarrassing situations due to lack of documentation. In one case, my assistant misplaced documents relating to approvals and vendor negotiation thus attracting auditor’s comments for loss of control.
Failure to record discussions and agreement with users has often lead to damning arguments during implementation stages. Not documenting system specs has got many of us in trouble. Absence of software specifications or incomplete specs makes many a developer code again rather than trying to rectify an earlier program. Lack of proper recording of approvals for user rights in the user profile can be a serious lapse and may cede ground for frauds to be committed.
What constitutes documentation
Documentation involves creating documents to record details / specification / events or storing and preserving documents that are relevant. They are important for the purposes of recording details of various activities, for retaining as evidence, for documenting policies & rules, for exercising control, preserving for posterity or as instruction for work to be done. Documentation can be on any medium including paper, on disks, DVDs, or on common repositories accessible on the net.
Types of documentation
I would classify documents in the following few categories:
General: Information systems plan, IT strategy, yearly budget, and the spends against the budgets, proposals for projects and their approvals, minutes of important meetings, periodic reports to management, etc.
Commercial: Request for Proposal (RFP), Record of talks / negotiation with vendors, PO copies, correspondence, etc.
Project documents: Requirements, statement of work, scope of work (SOW), software design and functional specification, system design and functional specifications, change management, error and enhancement tracking, user test and acceptance (UTA), and end-user manuals.
Others: List of IT assets, network diagram, security policy and other policies, disaster recovery policies, and action steps, etc.
When to start
I am told by many, that Indians are poor in documentation because of the legacy of the old Vedic period when thousands of lines of verses were carried through generations just by word of mouth. I do not however believe on that theory and strongly suggest that it is matter of habit. All IT service companies and consulting houses are immaculate in their documentation as it is a requirement dictated by the contract of services they have with their customers. It is however in the end-user companies that we face this problem.
The question, when to start: well, it is ‘now’. It doesn’t matter where we begin but we have to make a start and slowly build up our repository and should not be found wanting when any crisis strikes.
Over the last few years we have seen phenomenal changes in the technology landscape. New technology breakthroughs keep taking place every now and then; either in the hardware space or in new software and process ideas and these have been altering our approach to computing. I am now talking about the latest move to exploit computer memory to store and process data.
In the conventional form, we typically store data on the hard disk, and when we want to perform a task, we pull out the relevant data and applications for the purpose on to the computer’s main memory, which is where computations happen. The time taken for the entire process therefore depends on the access time for reading data on the disk and the speed with which the data is loaded on to the processor and memory. Though disk and channel speeds have been enhanced over time, they still are much lower than today’s processor and memory speeds. With computing needs of enterprises on the increase, accessing data from disks has become a bottleneck.
Going down the memory lane, I remember storing data on tapes and cartridges. Then as disk costs (cost per GB) reduced, came the concept of use of disks as tapes. Now as memory costs go down further, it has become possible to store larger chunks of data in the memory.
The concept at work
With the emergence of multi-core processors and the sharp decline in prices of processors and memory, a few vendors have developed a technology that makes possible for even large enterprises to dispense with hard disks and store and perform all operations in the main memory. This boosts performance enormously as compared to systems based on retrieving data from hard drives. The pioneers in this area have been German Software vendor SAP and TIBCO.
Curious to know more about this approach, we, at our CIO Association, invited SAP to hold a seminar explaining the concept and their offerings. The seminar was well-received by the audience as SAP gave an introduction to its new BI product, HANA (high-performance analytic appliance). HANA is a composite solution comprising the software solution and an appliance. SAP has tied up with various hardware vendors including IBM, HP and others, and such servers come to customers pre-configured. With complete data residing in the main memory, computing gets very fast; SAP claims increase of speed by a factor of 10 on data loading and by a factor of 100 on reporting. That makes real time reporting and ability to take faster decisions possible. The data is stored in a compressed form using a horizontal format (compression ratio of 6+) and the data split and stored with check bits to ensure recovery in case of failure of the system.
While it is always good to hear about new technologies and their breakthrough capabilities, it is the questions of its relevance and usefulness to user organizations that come to fore. So let us take a look at the possibilities where a product like this could be of help:
- Significantly faster processing: This could be useful when you want to do a quick profitability analysis in a complex organization with multiple products – you can process in seconds which would otherwise take a few hours. You could analyze the sales performance during the day and take appropriate action before the competition wakes up. Similar use cases can be for production scheduling, logistics, etc.
- Replacing complex data warehouses with in-memory data storage: Data warehouses aggregate data in various different aggregates in order to have the answer ready when the question comes. With in-memory computing, you don’t need to do the aggregates but can just calculate on the fly. That means less cost for infrastructure to run large-scale analytics systems.
- Run features that you could not do before: You may have been constrained in the past from running complex iterations including linear programming models and other optimization solutions as they took enormous time and resources. Various what-if analysis and future scenarios calculations can also find a use here.
Should we take a plunge?
Well, the technology is here and it works. However as in the case of all new introductions, organizations would do well to assess their requirements and work out a justification for adopting in-memory computing. The technology will get further fine-tuned as it matures and we would see more applications getting on to the memory. The technology is truly transformational and need to be watched closely and adopted if the situation so demands.