Here’s an example of what we mean. Back in 2010, the Corporate Executive Board Co. (CEB) pulled out its crystal ball to summon a five-year forecast (it was the big crystal ball) for corporate IT. What it saw for 2015 was that IT organizations would be radically different from those at the dawn of the decade. Good for CEB: Those predictions were pretty right-on. Bad for those involved in tech hiring: They’re already coming to pass, whether you’re ready or not.  Here’s what CEB identified as “radical” shifts in IT way back then:

• Information over process: Information management skills will rise in importance as competitive advantage from IT shifts from process automation to customer experience, data analytics and knowledge worker enablement.
• IT embedded in business services: Centrally provided applications and infrastructure will be embedded in business services and delivered by a multifunctional shared services organization.
• Externalized service delivery: Delivery will be predominantly externalized as vendors expand service provision and internal resources become brokers instead of providers.
• Greater business partner responsibility: Business unit leaders and end users will play a greater role in obtaining and managing technology for themselves in areas where differentiation has more value than standardization.
• Unbundled IT: Some IT roles will be embedded in business services, evolve into business roles or be externalized. Remaining IT roles will be housed in a business shared services group, and IT staffing needs will change significantly.

With all this flux, it’s no wonder that, as CEB research director Shalini Das puts it, tech hiring “is like chasing a moving target.” IT is facing rising demand for business intelligence, mobility, and restructuring to deliver end-to-end services for higher speed and efficiency. And it must partner with increasingly tech-savvy business users willing to self-manage IT projects, she said.

So what to do? Leaving it up to HR doesn’t work; you have to do the opposite. As many of the experts we spoke with for our latest piece on help with tech hiring agreed, the CIO who wants the best people — the right people — must partner up with HR.

“CIOs must clarify and communicate new or changing definitions and skills requirements to hiring managers and recruiters so they can update job descriptions, résumé screening filters and interview questions as needed,” Das said. “In addition, they must work with their HR peers to correspondingly update staff competency models, training and development programs.”

“If you think of big data as a belief that there is going to be a single giant data warehouse just as there is a single instance of ERP, and that this is going to be the source of intelligence and the driver of business decisions, then that attitude is on the wane,” McDonald said. “What we found in this year’s surveys is a shift to a much more operational or tactical application of analytics apps.”

“I’m not quite ready to say big data is dead,” he added. “But the interest we’re seeing is in disaggregated BI, as opposed to a unified, over-mined version of the truth.”

Cost is one of the reasons big data is disaggregating, McDonald said. Like the economics of ERP, the economics of big data BI — where the data is unified and clean –is prohibitive, especially as data increases.

The way his clients describe analytics is in combination with other business functions: analytics apps and social media, analytics apps and supply chain, analytics and mobile, for example. That’s in distinction, McDonald said, to the traditional view of analytics as a “corporate shared services capability.”

In this worldview of BI, the job of CIOs is to think about analytics apps just as they would about other applications, with an emphasis on providing practical business value. “A CIO who can demonstrate the use of data in formulating business cases, in sizing up business opportunities and in making fact-based decisions is providing business value,” McDonald said.

Sounds good, but that’s kind of the problem. I’m left wondering — as with so much in IT — whether the shift away from traditional “big data” BI to analytics apps for specific business functions is more semantics than “qualitative change.” And I can’t wait to hear the hue and cry from big data experts about how big data cannot by any definition be equated with traditional BI — not to mention emails from all those big-name BI vendors selling something called big data analytics.

In 2009, more than half of organizations surveyed suffered budget cuts. In 2011, however, 56% of IT budgets increased, a healthy percentage compared with 2010, when 34% of organizations saw their IT budgets go up, and to 2009, when 25% of organizations reported IT budget increases. These results are based on SIM interviews with CIOs at 275 organizations in late June.

“It’s probably the biggest jump I have ever seen, and puts us back at pre-recession levels,” said Jerry Luftman, distinguished professor at the Stevens Institute of Technology, who conducts the research for SIM’s annual benchmark.

IT leaders expect the positive trend to continue into next year. Despite talk of a double-dip recession, 84% of the CIOs surveyed expect 2012 budgets to equal or exceed 2011 levels. In one area, IT budgets did decline in the 2011 CIO survey: The percentage of corporate revenue allocated to IT dropped from 3.8% in 2010 to 3.5% in 2011. Luftman has attributed the decrease to a rise in corporate revenue last year and to the historically high percentage of corporate revenue allocated to IT over the past three years — which, at nearly 4%, was well above the average 3.6% of the past six years.

On the hiring front, turnover remains quite low, at 7%, partly because retirement-age boomers can’t afford to retire and partly because there are fewer job openings for senior-level positions, Luftman said. CIOs tell him that when an experienced staff member does retire, they are using that senior-level salary to hire two “newbies,” who cost less and often come in with the newer skills and technology expertise CIOs need. On the bright side, however, overall spending on salaries is trending up:

• IT staff salaries increased at 66% of organizations in 2011 compared with 2010.
• 67% of organizations expect staff salaries will go up again in 2012.

BI a hard nut to crack

Given their plushier budgets, what are CIOs spending money on? Business intelligence (BI) outstripped cloud computing; ERP systems; mobile and wireless apps; and customer relationship management, or CRM, systems as the top technology investment by CIOs in 2011, according to the survey — and by a long shot.

“BI was a standout — it was 50% higher in the rankings than all the others, which were relatively close in ranking,” Luftman said.

But it appears the upstarts are poised to give BI a run for its money. Mobile and wireless apps took fourth place, up from ninth last year and 13th in 2009. Cloud computing occupies second place, up from fifth place a year ago and 17th place in 2009, the year it made its debut on the SIM survey. The wide disparities in the amount companies are investing in cloud, however, show how nebulous this new computing model remains, Luftman said:

• 20% spend more than 10% of their IT budgets on cloud.
• 21% spend between 1% and 10%.
• 43% are doing nothing with cloud.

In one respect, BI’s top standing in the SIM survey is no surprise. The technology has ranked first or second on the SIM list of the top five CIO investments since 2003, Luftman said. The reasons for the heavy investment in BI, however, keep changing, he added — a mark of just how hard it is to extract potentially valuable insight from the reams of data collected by businesses . “Initially, BI ranked high because of the complexity of getting your databases in order,” he said.

As organizations have mastered the technical challenges of their BI investments, they have recognized they don’t have the talent to support the technology, Luftman said. “You can’t throw a tool up and expect magic to come out.” The portfolio of required skills goes beyond understanding databases and the way the technology works (important as that is) to include statistical and in-depth business knowledge. People with that combination of skills are “few and far between,” he said. The large volume and the velocity of data generated by companies — Big Data — adds to the challenge. “It is one of the more complicated technologies that we have been engaged in in perhaps in 50 years,” Luftman said — and SearchCIO.com can attest to that in our coverage of Big Data.

CIOs still have serious worries. Of the Top 10 IT management concerns of 2011, the first four focus on using technology to help the business compete. IT and business alignment claimed the top spot in 2011, followed by business agility and speed to market. Reducing business expenses through business process management and re-engineering took the third spot; and increasing business productivity and cost reduction came in fourth. Rounding out the Top 10 management concerns, in order, are these:

5. IT strategic planning.
6. IT reliability and efficiency.
7. Enterprise architecture/infrastructure capability.
8. Security and privacy.
9. Revenue generating IT innovations.
10. IT cost reduction.

You’re not alone.

According to Forrester Research Inc., an RDBMS has always been an awkward fit for BI. When you need to find relationships that require analyzing many-to-many correspondences; when the variables themselves aren’t all of the same kind; or when you don’t know, going in, exactly which relationships you’re looking for, traditional spreadsheets and their more sophisticated relational-database progeny come up short. Even if you know what you’re looking for, a traditional RDBMS requires time-consuming tuning to get the job done. That’s just not practical in the modern business landscape. When the questions are changing faster than the BI answers can be provided, it’s time for something new.

In fact, in the search for BI agility, most companies will jettison their current RDBMS over the next decade for BI needs, Forrester BI expert Boris Evelson predicted.

Last week, I spoke to David Gallaher, IT services manager at the National Snow and Ice Data Center, who went to an object-oriented database because a traditional RDBMS was of no use.

“We have tried to shoehorn all kinds of data into these constructs, and now Big Data is where we have really run into the limitation of what you can do with these old constructs, where everything has to fit into a table,” Gallaher told me. “Well, what if my data doesn’t really fit into a table?”

In a report published in May, Evelson discussed several new strategies for extracting relationships out of ever-more-complex data sets, and reviewed four relevant BI database management system technologies that have already arrived or at least are on their way. Here’s the skinny:

Columnar DBMS: Although traditional spreadsheets — still the most popular BI tool — can always analyze a row or a column, the emphasis in some new DBMSes is shifting to the power and flexibility of columnar analysis. Evelson believes there are distinct advantages with a columnar DBMS. It compresses data better than a row-based RDBMS, because everything in a column is of the same type. Indexing is an easier task than it would be in a row-based RDBMS because each column “already represents its own index,” he said. “It can keep the database size roughly equal to that of the raw data set — or sometimes cut it in half,” he added.

Many DBMS vendors already offer columnar or hybrid row-based and columnar systems. They range from such mainstream vendors as IBM (Netezza), Microsoft (PowerPivot), and EMC Corp. (Greenplum) to such pure-play columnar RDBMS vendors as Hewlett-Packard Co. (Vertica), SAP AG, Sybase Inc. (IQ), Infobright Inc., and 1010data Inc.

In-memory index DBMS: This is the most agile and flexible of the four technologies because the entire relational database is either in memory or can be swapped rapidly into memory. That flexibility and agility, however, add risk. One risk is that business users could arrive at a wrong answer because they’re no longer constrained by the rigid data models typical of an RDBMS.

It should also be kept in mind that the functions offered by in-memory vendors vary widely, Evelson warned. Among other questions, business pros should inquire whether an in-memory DBMS can be accessed by their other BI tools. Another issue is that if Big Data is being used, the entire data model might not fit into a single memory space.

When sizing applications for a single memory space, users should consider the size of the raw data set, compression ratios and the number of concurrent users, he advised. If the total exceeds a few hundred gigabytes, he suggested picking a vendor that can “dynamically swap chunks of your model in and out of [random-access memory],” or one of the hybrid in-memory databases. The vendor list includes Tibco Software Inc. (Spotfire), Tableau Software Inc., SAP (HANA), and MicroStrategy Inc., among others.

Inverted-index DBMS: According to Evelson, this is a useful database technology when data is complex, content is unstructured and the user’s hypothesis is vague. By building indexes, an inverted-index BI DBMS upends the RDBMS practice of putting the database first and worrying about tuning it later. “This approach builds one big index, but instead of just pointing to data sources — as traditional search engines like Google and Yahoo do — it embeds data in the index itself,” he explained.

The inverted index works well for applications that use data from a variety of sources and that incorporate structured as well as unstructured content. BI pros should consider an inverted index when a project requires numerous data marts to get around the limitations of traditional and even multidimensional DBMSes. An RDBMS assumes you know what you’re looking for, “but BI end users often don’t,” he noted. This searching allows BI users to navigate through the data in order to zero in on what they want by subtracting what they know they don’t want. Attivio Inc. and Endeca Technologies Inc. offer an inverted-index DBMS.

Associative DBMS: It’s tough to make predictions, especially about the future, as Yogi Berra is said to have noted. That’s why some business users are insisting that everything gets filed away in the data warehouse because who knows when it might come in handy. An associative DBMS attempts to link everything together, allowing any trend to be pulled out at any time. “Imagine a data warehouse that can store all-to-all relationships — associations or vectors — between every entity and every attribute in your domain, with counters, aggregates and indexes for every intersection,” Evelson said. Oh, my! But it will cost you. The factor used to calculate the size of an associative DBMS as a multiple of the raw data set is as high as 10 in the associative databases used in academia, he said.

An associative DBMS also requires purpose-built graphical user interfaces, and is not easily accessed by queries based on the Structured Query Language and the Multidimensional eXpressions language. Rather than think in traditional “where clauses,” associative DBMSes let their imaginations run wild, finding connections and analogies that — you guessed it — don’t necessarily line up neatly by rows. Saffron Technology Inc., Ingres Corp. (VectorWise), Illuminate Solutions Inc. (iLuminate), LazySoft Ltd. (Sentences), and Splunk Inc. (a variation on an associative DBMS) are in the vanguard.

“One thing we do not yet know how to do well is agility,” Boris Evelson told me at the recent Forrester IT Forum in Las Vegas.

Evelson, a principal analyst at the Cambridge, Mass.-based Forrester Research, has been covering BI for some 30 years. Over that time, scalable, powerful, stable BI has become a reality at companies with enough money and know-how. “I don’t want to say it is a commodity, but we know how to do that,” he said.

On the other hand, agile business intelligence — the ability to react faster to the ever-increasing speed of business change — remains “challenge No. 1,” Evelson said. It’s the subject of every conversation he has with clients these days, he told me, and it was the centerpiece of his talks at the conference.

Guidelines for an agile organizational structure

One of the reasons agile business intelligence remains elusive for most CIOs, Evelson said, is that BI software is different from almost any other enterprise application. With ERP or CRM, for example, once the requirements are defined and the software either procured or developed in-house, IT can expect a shelf life for that software of 12 to 36 months, with minor modifications. “With BI, if you do that, when you roll out the first iteration, it is already too late. The world changes way too fast,” he said.

Given that CIOs can’t do much about the pace of change, how do they get to agile business intelligence? In Evelson’s view, it’s a combination of using an agile software development methodology — which relies on prototyping rather than specifications — and on an agile organizational structure. Not that either is easy to do, especially the organizational-structure part. CIOs and their BI experts understand that silos are bad for BI, he said. But so is centralization, because “shared services are anything but agile.” What’s needed is a middle ground. Not middle as in wishy-washy, but as in a nuanced set of guidelines for handling BI. That requires a hard-nosed discussion about which apps need to be in a central area (mission-critical ones, for example) and which nice-to-have, ad hoc apps should stay where they are.

In-memory analytics, mobility

There are also plenty of technologies that can make a BI environment more agile. Forrester has a list of about 20, Evelson said, from cloud and mobility for BI infrastructure and delivery to inverted indexes and in-memory analytics, an approach he believes is suitable for as much as 90% of BI efforts.

“Think of in-memory as Excel on steroids. It has all the flexibility of Excel but also the power of traditional BI tools, like virtualization,” Evelson said. QlikTech International’s QlikView and Microsoft’s PowerPivot take an in-memory approach to BI.

Of course, the flexibility these tools provide also represents a “huge danger,” Evelson hastens to add. IT cannot control what users do in Excel, and the same is true for in-memory tools: One person’s analysis of customer profitability is not going to be the same as another’s. “You have to be smart. If it is a mission-critical app where nothing less than 100% accuracy is good enough, then in-memory analytics is not the right choice.”

Not surprisingly, at companies where there is more business ownership of BI, in-memory analytics are being adopted “left and right,” Evelson said. At IT-centric companies, not so much, he said.

The quick fix

Technologies help facilitate agile business intelligence, but for CIOs, finding the organizational structures and methodology is the tough nut to crack, in Evelson’s view. In the meantime? Users will gravitate toward instant gratification.

“Traditional BI is why spreadsheets are still the most ubiquitous, best BI tool out there. You have a question, and as long as you have spreadsheets, you can get your answers,” Evelson said.

Is agile business intelligence as hard as Evelson makes it sound? If you have blazed a path to agility and are willing to talk about it, I’d love to hear from you.

According to Burton Group analyst Lyn Robison, one reason CIOs are struggling to deliver business insight to the business — as opposed to information — is technology’s misguided relationship with data. IT professionals of a certain age, he said, tend to view data as “sawdust,” a byproduct of the processes that information systems so brilliantly automate.

“Many IT professionals still haven’t realized that we actually store this data and can do useful things with it,” said Robison, who presented his views at last week’s Catalyst conference in San Diego.

For process-oriented IT pros, data is an interchangeable commodity, to be shoveled into databases just as oil is pumped into steel barrels — or at best, organized by type like cut lumber in a warehouse, one plank as good as another.

“The real world is filled with unique things that we must uniquely identify, if we are going to capture those aspects of reality that are important to us,” Robison said. To be useful, data needs to be a snapshot of reality. Nonfungible assets, unlike fungible commodities, need to be identified individually. And the IT department needs to manage those identifiers so the business can zero in on the data that matters. Fungibility matters.

So, what’s fungible? Currency, for example, usually is considered fungible. One $5 bill is as good as another. Buildings are nonfungible. Transactions are nonfungible. Customers are nonfungible. When nonfungible assets are treated like fungible commodities, the consequence is “distortion and incomplete information,” Robison said.

A large university Robison worked with recently discovered it was paying costly insurance premiums for five buildings it no longer owned, because its information systems managed the university’s buildings as interchangeable, he said. A Florida utility company paid out millions of dollars to the families of a couple tragically killed by a downed pole’s power line — only to discover afterwards that another entity owned the pole. “The liable entity got off, because the utility poles around that metro area were not uniquely identified,” he said.

It turns out, however, that discerning the difference between fungible commodities and nonfungible assets is not as clear-cut a task as it might appear, Robison conceded. “Defining fungibility is something of an art,” he said. Just like in life, context is everything.

However, the bigger problem in managing data to deliver business insight, according to Robison, is that today’s enterprise systems do not identify nonfungible data assets “beyond silo boundaries.”

“Primary keys are used as identifiers, but are not meant to be used beyond the boundaries of any particular database silo,” he said.

After his presentation, I learned that Robison has developed something he calls the methodology for overcoming data silos (MODS), “a groundbreaking project structure for bridging data silos and delivering integrated information from decentralized systems,” according to his recent paper on the topic. You can hear Robison talk about using MODS here. Let me know what you think.

Oh, and how you distinguish between the fungible and the nonfungible.

“Adoption is still the weak link,” said Gary Gallant, VP of Coty Inc.’s global applications center of expertise, as well as the perfume manufacturer’s BI point man. ”What we are trying to do with BI now is build some prototypes to give to leadership so they can get a better feel for BI, because what we have now is, ‘Well, what do I do with a dashboard?’”

Indeed, several  BI pros I spoke with at the show intimated that the widespread adoption of BI tools by the business–the BI revolution–awaits the rise of the digital natives in corporate management: in other words, the people who grew up with electronic data and are comfortable manipulating it.

In the meantime, Gallant had an interesting suggestion for bringing the “cool” factor to BI reporting for the C-suite: Forget the laptop and get your CEO an Apple iPad.

“I do think the iPad has the ability to change things,” Gallant said. Part of what prevents CEOs and other C-suite execs from really living with BI, in his view, is the physical barrier: having to reach into the computer bag, lug out the laptop and wait for it to light up.

“The time it takes you to get to productivity — they just don’t do it,” he said. The BlackBerry is too small to see the results. But the iPad? “It gives you landscape to look and drill down. Plus, anything connected with Steve Jobs has sex appeal,” he said.

Given the complexity of making an organization’s store of information actionable, as they say in the BI biz, I’d like to hear your insights and best practices for dealing with this daunting task. Email me at ltucci@techtarget.com.

One thing we haven’t lost is our love of puzzles. Test your analytics IQ with four brainteasers, offered up in a session on BI analytics by researcher and BI author Wayne Eckerson of TDWI Research. (The smarties in the room were quick to solve all but one.)

1. 1. What is the missing number in the following layout?

37   10    82
29   11    47
96   25    87
42    ?     15

2. 2. What does this equal?

12 = DD + 11 = PP + 10 = LL + 9 = LD + 8 = MM + 7 = SS + 6 = GL = 5 = GR + 4 = CB + 3 = FH + 2 = TD + 1 = PPT = ?

3. 3. What do these words have in common?

Vermont
Statuesque
Swedish
Arthur’s
Africa
Sensation
Misunderstood

4. 4. Which word does not belong?

Binoculars
Eyeglasses
Goggles
Handlebars
Jeans
Pliers
Scissors
Shoes
Tweezers

Answers: 6 (sum of the two digits of the number flanking the middle column); Twelve Days of Christmas; each contains the abbreviation for a day of the week; shoes (the only pair on the list that is not connected.)

LucidEra is in part a victim of a down economy, just as application service providers (ASPs) were in the late ’90s/early 2000s when the dot-com bust happened and VC funding started to dry up.

Like ASPs USinternetworking and Corio, LucidEra was one of the first to the SaaS BI parade. It had to lay new ground in many ways: The Web technologies that today’s SaaS vendors tap into weren’t around when LucidEra got started, so the company had a bigger learning curve and had to do lot of the development itself.

LucidEra told ThinkStrategies’ Jeff Kaplan that newer kids on the block learned from LucidEra’s mistakes and could skip many of the development cycles and bumps in the road that the company had to go through.

Back when next-generation ASPs such as Salesforce.com were getting started, they certainly didn’t try to go out and buy large data centers to essentially foot the infrastructure bill for enterprise customers, or try to retrofit Oracle’s or SAP’s licensing model to fit a multi-tenant one like first-generation ASPs had.

No, they, and other ASPs — now called SaaS vendors — learned from the mistakes of first-to-market ASPs like USinternetworking (USi), now part of IBM, and Corio, also now part of IBM.

USi and Corio came out the other side, but there are others that simply disappeared. Like ASP FutureLink, a company that many, including Microsoft — which sank $10 million into it — had high hopes for.

But all the buzz around so many of these players didn’t bring in enough customers to support them all.

Similarly, today there is a lot of interest in business intelligence and in the SaaS model. But is there enough interest to support all of the SaaS BI vendors?

Economy and customer adoption aside, LucidEra had a unique set of circumstances, including hooking its wagon to Salesforce.com. It is always risky to ride the coattails of another company, as USinternetworking and Corio found out by relying so heavily on Oracle and SAP.

And LucidEra did choose a niche in sales analytics. “One problem that [LucidEra] ran into was that not a lot of Salesforce.com customers saw the value-add of what they had to offer,” Kaplan said. “And to a greater extent, a lot of folks today think having analytics is a luxury they can do without.”

Some competitors believe that LucidEra’s downfall was its older code, developed in the late 1990s by Broadbase Software, the argument being that such code was not designed for the SaaS model. “I believe that it is difficult to retrofit a SaaS approach to an existing architecture and, unless designed as a SaaS application – multi-tenant, SOA, layered architecture than can scale horizontally – cost-effectively scaling the solution is incredibly hard,” said Wayne Morris, CEO of SaaS business intelligence vendor myDials. Morris expands on what went wrong at LucidEra on his company’s blog post.

Meanwhile, Brad Peters, CEO of SaaS business intelligence vendor Birst, chalks up LucidEra’s expected shutdown to the company’s standalone analytic software approach, as opposed to most companies’ need to analyze data from multiple sources, something that LucidEra’s software wasn’t set up for, he said.

All in all, the comments I’ve seen on blogs say this is not a sign of the on-demand model’s going away — not by a long shot — but a demise that happens naturally when a lot of companies crop up in one space. There are bound to be some that just don’t cross the chasm, as Geoffrey Moore would say.