Data science image via Shutterstock

By James Kobielus (@jameskobielus)

The best scientists speak with authority that is grounded in their mastery of empirical observations and of the tools and methods needed to find powerful truths. In modern society, we’d like to think that scientists of any stripe are unimpeachable authorities, because, after all, isn’t science a noble calling? Or, if individual scientists are fallible and occasionally dishonest human beings, isn’t the scientific process supposed to expose their lies, sanction them severely, and possibly end their careers.

So it’s especially disturbing when we find that some scientists abuse our trust by falsifying the data and models at the heart of their work. This recent Information Week article (http://www.informationweek.com/big-data/news/big-data-analytics/big-data-fakers-5-warning-signs/240152921) discusses several scientific researchers who were caught fabricating data, models, and experiments.

None of the cited examples specifically involves data scientists doing work with big data in commercial organizations, but it makes you wonder.  Data scientists are the rockstars of the big-data revolution and they carry an increasing amount of perceived authority. This category includes statistical analysts, data miners, predictive modelers, computational linguists, and other smart people whose job is to find deep insights in large, complex data sets.

Data scientists are like any skilled person in any esteemed profession. Most are honest, have professional integrity, and stand behind their work. But there’s always the opportunity for an unscrupulous data scientist, in any context, to fake their work. To the extent that a secretly dishonest data scientist operates autonomously, without independent oversight or peer-vetting of their work, they can do incalculable damage to your big-data initiatives. If every other data scientist in your organization uses their (falsified) data and their (bogus) models that were trained to that data, it might take a long time (if ever) before you realize you’ve been had.

Trustworthy data science demands trustworthy data scientists. But trustworthiness, of course, requires continual independent verification. Where big data is concerned, do you have full lineage, access and version controls, and audit trails of every record stored in your data science sandboxes, and also a equivalent governance process applying to all models built on that data?

If you don’t, your big-data initiatives may be operating on a single version of a lie. And that can expose your business to significant legal, operational, and strategic risks.

Big data image via Shutterstock

By James Kobielus (@jameskobielus)

Big data’s primary executive-level sponsors are chief marketing officers (CMOs). Consequently, it’s no surprise that big-data repositories are primarily populated with information on customers. To the extent that the CMO’s IT organization has linked diverse customer records to positive identifiers–a process often known as “identity resolution”–they can drive finely targeted marketing efforts.

These massive stores of customer identity information–let’s call them “big identity”–are the lifeblood of modern commerce. Social media, other Internet sites, and enterprise repositories manage a sprawling, heterogeneous, disconnected variety of identity information. Rolling up any particular individual’s various identities across these sites demands the massively parallel horsepower, specialized analytic tools, and high-capacity storage of a robust big-data infrastructure. Identity resolution leverages advanced algorithms to uniquely match the disparate identities that an individual or group might be using.

Identity resolution is the missing link between two important use cases of big data: social media analytics on the one hand and multichannel customer relationship management on the other. High-performance identity resolution is already a substantial application in customer data integration, data quality, master data management, and anti-fraud applications.

Social media monitoring, however, has yet to tap its potential. How else can we match the disparate identities that people go under in Twitter, Facebook, and other socials, both against each other and against the system-of-record identifiers that we keep on customers in our CRM, data warehousing, and other operational platforms? Without the ability to resolve some prospective customer’s social-sourced identities, how can we determine whether or not they’re an existing customer or a hot prospect?

The flip side of the “big identity” dream is the potential for high-powered violations of personal privacy. Some say big data is Big Brother’s chief tool for mass surveillance. Others say it opens a Pandora’s box for any grass-roots peeping-Tom to pry into other people’s affairs with the most powerful telescope ever invented. A cynic might say that social business–one of the hottest new focus areas in multichannel marketing and engagement–is all about everybody minding–and mining–everybody else’s business.

Privacy concerns are rooted deep in the heart of the online experience, which thrives on freewheeling give-and-take but can easily slip into oversharing, surveillance, cyberstalking, and intrusive targeting. Businesses should put privacy considerations at the core of our big-data strategies before customers demand it or the courts, regulatory bodies, and legislators decide to force our hands.

CIO image via Shutterstock

This week’s roundup is full of news from the CIO, big data and healthcare industry. Read on to find out the big headlines from these industries!

1. Hauwei security issues are result of ‘rumors’ says Hauwei executive – Valery Marchive (SearchSecurity)

Several months after the U.S. House of Representatives Permanent Select Committee on Intelligence labeled the Chinese networking vendors, Huawei Technologies and ZTE Corp. as a threat to U.S. national security, Huawei’s Chairman Francois Quentin said the company has become a victim of ‘rumors’.

2. Beyond the hype, CIOs can generate business value from big data tools – Editorial team (SearchCIO)

In this video interview, Sears Holdings CTO Phil Shelley discusses how embracing big data tools can lead to new business and how CIOs can use them to add value to their business.

3. Telecommuting technology a double-edged sword for SMBs – Nicole Laskowski (SearchCIO-Midmarket)

Following the reaction to Yahoo’s ban on telecommunication, Nicole Laskowski says most small business owners continue to rely on telecommuting for day-to-day operations but believe customer interaction will lead to success.

4. Look out, Big Data: In-memory data grids start to go mainstream – Stephanie Mann (SearchSOA)

As in-memory data grids become more popular (Gartner called in-memory computing one of the top 10 strategic technology trends of 2013), Stephanie Mann looks at why it has finally gone mainstream.

5. Podcast: What you missed at HIMSS 2013 – Ed Burns and Don Fluckinger (SearchHealthIT)

SearchHealthIT reporters Ed Burns and Don Fluckinger break down the big stories and hot health IT themes that emerged from HIMSS 2013 in New Orleans.

Big data image via Shutterstock

By James Kobielus (@jameskobielus)

Big data has its discontents. The discontent is taking the form of a backlash against the incessant hype of big data’s most noteworthy rock star approach: Hadoop. The backlash is a necessary reality-check in an otherwise fast-growing space. Often in this industry, when a technology is super-hot, the hype can get in the way of rational decision making, both among users and among solution providers.

Backlashes against hype are inevitable. At times, it almost feels like people discuss big data with the assumption that bigger is necessarily better and that throwing more data at your problems will automatically produce insights. You should be searching for those special problems, often of a scientific nature, that can be solved best through petabyte-scale analytics. You don’t need a data center full of maxed-out storage arrays to derive powerful insights. Gut feel is free, and it often thrives on the scantiest information.

This viewpoint, which is now in the mainstream, is often referred to as “small data,” or as the need for traditional lower-scale analytics platforms to balance the hype for all things “big.” Of course, keeping your BI, performance management, and statistical modeling deployments small and simple can be a challenge. This is especially true as your user base grows; the range of data sources, structured reports, metrics, dashboards, models, visualizations, and downstream applications grows. Throughout the life of your “small data” environment, it will probably continue to grow into “big data” territory on the volume, velocity, and/or variety dimensions.

As your needs scale from small to big, you should have a data platform that you can grow modularly and cost-effectively to keep pace. Big data and small data should be joined at the hip in your strategy–simply a seamless spectrum of analytical platforms, tools, and approaches serving your business.

Big data, small data. Big insights, small epiphanies. It’s all important. And none of it is mutually exclusive.

Big data image via Shutterstock

By James Kobielus (@jameskobielus)

Big data thrives on data scientists who use their expertise in statistical modeling and business analytics to find meaningful patterns in data sets that can grow extraordinarily complex.

Organizations everywhere are stepping up their hiring, recruitment, and training of data scientists. But there seems to be a growing alarm that we won’t have enough data scientists to go around. Will the big data revolution screech to a halt due to a shortage of data scientists?

Don’t worry. Any talent gaps that may exist are likely to be short term and ephemeral. Data science is not an elite priesthood of quants, geeks, and Ph.Ds, but rather, a set of skills and practices that are being nurtured and cultivated everywhere in the business world, not just in academia. The core data-scientist aptitudes–curiosity, intellectual agility, statistical fluency, research stamina, scientific rigor, skeptical nature–are widely distributed throughout the population.

The following trends explain why the supply of data science talent in the business world will continue to expand and the productivity of data scientists will continue to improve over the foreseeable future:

• Data scientists are automating more of their tedious data discovery, acquisition, and preparation tasks through sophisticated tools (e.g., as IBM InfoSphere Server, IBM SPSS Modeler).
• Data scientists are developing fewer models from scratch, because more big data projects run on application-embedded analytic models integrated into commercial solutions, such as IBM Unica.
• Data scientists are being trained within enterprise centers of excellence, which provide forums and resources for long-time business analytics professionals to enhance their skills in hot new areas such as text mining, graph modeling, and behavioral analytics.
• Data scientists are offering their services to the business market through professional services organizations such as IBM’s Business Analytics and Optimization group; through specialized, boutique consulting firms; and through the open-source communities.
• Data scientists are taking advantage of increasingly low-cost, self-service analytics power tools to teach themselves the core skills and become productive developers of sophisticated models

Data science is a highly skilled profession with a significant learning curve. But throngs of smart people are flocking to it in ever greater numbers to advance their careers in the age of big data. Rather than lose sleep over some overblown threat of a chronic “talent gap,” you should simply cultivate the right data scientists for your specific needs.

Big Data image via Shutterstock

By James Kobielus (@jameskobielus)

All things considered, big-data platforms are not any more or less secure than established, smaller-scale databases.

Where security is concerned, the sheer volume of your data is not the key factor to lose sleep over. Instead, the chief security vulnerabilities in your big-data strategy may have more to do with the unfamiliarity of the platforms and the need to harmonize disparate legacy data-security systems.

Platform heterogeneity is potentially a big-data security vulnerability. If you’re simply scaling out your an existing DBMS, your current security tools and practices probably continue to work well.  However, many big-data deployments involve deploying a new platform–such as Hadoop, NoSQL, and in-memory databases–that you have never used before and for which your existing security tools and practices are either useless or ill-suited. If the big-data platform is new to the market, you may have difficulty finding a sufficient range of commercial security tools, or, for that matter, anybody with experience using them in a high-stakes production setting.

Platform heterogeneity is also a potential big-data security vulnerability. Some organizations implement big data as a consequence of consolidating inconsistent, siloed data sets. In the process of amassing your big-data repository from heterogeneous precursors, you will need to focus on the thorny issue of harmonizing disparate legacy security tools. These tools may support a wide range of security functions, including authentication, access control, encryption, intrusion detection and response, event logging and monitoring, and perimeter and application access control.

At the same time as you’re consolidating into a big-data platform, you will need to harmonize the disparate security policies and practices associated with the legacy platforms. Your big-data consolidation plans must be overseen and vetted by security professionals at every step. And, ultimately, your consolidated big-data platform must be certified according to all controlling enterprise, industry, and government mandates.

And if you want to approach big-data security holistically, your strategy should also include tools and procedures for unified (hopefully real-time) monitoring of security events on disparate data sets. You should consider implementing a logical big-data mart for security incident and event monitoring (SIEM) to identify threats across your disparate big-data platforms, consolidated and otherwise.

To the extent that you’re managing customer, finance, and other system-of-record data on your big-data clusters, you should certainly consider the need for strong SIEM. If you don’t, your enterprise’s chief information security officer will almost certainly, at some point, ask why you haven’t.

Twitter image via Shutterstock

Want to know the latest news and updates surrounding big data? We’ve built a list of the top experts and professionals across the Twitter world who update and share their data gossip. Ten of our favorites are listed below; tell us in the comments below if we left anyone off!

• Manish Bhatt (@imbigdata): Big data enthusiast
• Michael Driscoll (@medriscoll): CEO of Metamarkets
• Peter Skomoroch (@peteskomoroch): Data scientist at LinkedIn
• James Kobielus (@jameskobielus): Big data evangelist at IBM
• Roger Ehrenberg (@infoarbitrage): Big data VC at IA Ventures
• Ben Lorica (@bigdata): Chief data scientist at OReilly Media
• Andrew Brust (@andrewbrust): CEO of Blue Badge Insights
• Paul Miller (@PaulMiller): Analyst for GigaOM
  
• Jeff Kelly (@jeffreyfkelly): Technology market analyst for The Wikibon Project
• Dan Vesset (@DanVesset): Vice president at IDC

Disclaimer: All videos presented in the “YouTube IT Video of the Week” series are subjectively selected by ITKnowledgeExchange.com community managers and staff for entertainment purposes only. They are not sponsored or influenced by outside sources.

Tell us in the comments below on how you think big data will evolve in the future.

Please Include Attribution to OnlineBusinessDegree.org with this Graphic

Big Data image via Shutterstock

By James Kobielus (@jameskobielus)

Big data is a complex, tricky thing to govern. Often, it’s an unholy siloed mess of disparate databases under various business units, on various data platforms, and managed by various “stewards” with various tools and workflows.

Consolidation of your big-data assets must be an ongoing initiative, both to reduce overhead and to free up the insights that come from correlating disparate data sets. But you can scarcely consolidate such a mission-critical resource without addressing the administrative issue of big-data governance head-on. Presumably, you already have some level of governance–aka data stewardship or master data management–in your data warehousing and business intelligence practices.

Smart big-data consolidation demands the following double-barreled approach to governing the assets that matter:

• Governing analytic data: Keeping your big data under control means, among other things, determining what small subset of it should be managed with tight stewardship. Usually, those are the system-of-record relational data you’ve long managed within the master tables of your enterprise data warehouse. In other words, your official records on customer, finances, human resources, the supply chain will still be governed tightly in the era of big data, and probably on your scaled-up enterprise data warehouse. But the larger volume of unstructured data–such as social marketing intelligence, real-time sensor data feeds, browser clickstream sessions, and IT system logs–can remain outside your governance practice until such time as it is linked to systems of record.

• Governing analytic models: Big-data applications ride on a never-ending stream of new statistical, predictive, segmentation, behavioral, and other advanced analytic models. As you ramp up your data scientist teams and give them more powerful modeling tools, you will soon be swamped with models. Big data analytics demands governance of analytic models, if they’re to be deployed into production business applications. Key governance features include check in/check-out, change tracking, version control, and collaborative development and validation. Your big-data sandboxing platforms and modeling tools should ensure consistent governance automation, and managed collaboration across multidisciplinary teams working on your most challenging big data analytics initiatives.

No, governance is not the sexy side of big data. It’s often an afterthought in big-data projects. But it’s absolutely essential if you wish to keep your data clean, your models fit, and your big-data applications delivering reliable insights throughout the business.

James Kobielus is an IBM Big Data evangelist.