Big data analytics image via Shutterstock
By James Kobielus (@jameskobielus)
To truly see deep into the future, you need to see just as far into the past. If you have a large enough sample of data on how those populations behaved in the past, you can predict their behaviors with a reasonable degree of confidence.
But if your historical sample is very small, you’ll have a tough time explaining why your statistical predictions of future events is better than the proverbial coin flip. And if the event you’re trying to predict is rare, you may not have enough historical occurrences to overcome the statistical biases inherent in small samples. In those latter cases, you may be able to see far into the deep past, but the events you’re searching for are as sparse as Earth-like exoplanets in the vastness of space.
Data scientists are never content to throw up their hands and say we can’t at least have a statistical best guess on rare events. Indeed, decision makers demand clarity on “black swans” and other once-in-a-lifetime events with significant downside risks. We would all like to believe that the probability of these showstoppers is far from random, though the shape of their distribution curves is anybody’s guess.
Nevertheless, data scientists try various approaches to wrap their models around rare events. In this recent blog, Tavish Srivastava provides guidance for building logistic regression models to predict rare events with confidence in spite of small-sample bias. Logistic regression is for predicting the outcome of a categorical dependent variable (e.g., the binary outcomes of “event will occur” vs. “event won’t occur) based on one or more predictive independent variables. Here’s a more general discussion of the issues involved in applying logistical regression to prediction of rare events.
The take-away is that, to use this approach with confidence, a data scientist should have both a large enough sample size of the events being predicted and a large enough number of occurrences, within the sample, of the least-frequent event (e.g., “will they churn?” vs. “will they not churn”?) being predicted.
This is a modeling challenge where big data provides undoubted value. To the extent that you can collect, store, and analyze the entire population of event data (or at least a very large sample) in a Hadoop or other big-data cluster, the more likely you are to find enough occurrences.
Even a limited sample from a whole-population big-data store may contain more rare occurrences than would a smaller-population database that was culled from the same event-data source.
Microsoft Windows image via Shutterstock
Wait…so the death of Windows XP will actually bring the PC industry back to life? Well, in this week’s roundup, find out how the “elder” operating system will bring a resurrection to the market.
1. Windows XP death expected to resuscitate the PC industry – Diana Hwang (SearchEnterpriseDesktop)
Even the the Windows XP death looms large over many IT organizations, there’s good news that the aging operating system will actually stabilize the PC market.
2. Is Lenovo making an enterprise smartphone play with its Motorola buy? – Jake O’Donnell (SearchConsumerization)
According to industry watchers, Lenovo’s massive Motorola Mobility purchase could lead to a big opportunity for the tech giant by integrating with its ThinkPad brand.
3. Alleged SpyEye mastermind pleads guilty to fraud – Warwick Ashford (ComputerWeekly)
The alleged mastermind of the notorious SpyEye banking Trojan toolkit has pleaded guilty to conspiracy to commit fraud in a United States court.
4. Amazon could be working on Kindle-based retail checkout system – Jeff Dunn (TabletPCReview)
If recent reports hold true, Amazon will soon be going from virtual stores to physical ones with the help of its Kindle tablets.
5. Consumer tech enters the workplace: What are CIOs to do? – Emily McLaughlin (SearchCIO)
The adoption of consumer tech in business is trending toward wearable devices. In this tweet recap, learn how to prep for wearables in the workplace.
NFL image via Shutterstock
Now you may be wondering why there’s a picture of the NFL here. As it turns out, the league is teaming up with a major company to look at its in-stadium Wi-Fi. Find out which company it is and more in this week’s roundup.
1. IBM cuts bait on Intel server line, Lenovo bites – Ed Scannel (SearchDataCenter)
With speculation mounting, IBM finally decided to cut loose its Intel-based System X servers and sell it to Lenovo for $2.3 billion. IBM has already said it will begin focusing more on big data, cloud computing and software.
2. VMware leaps into enterprise mobility management with AirWatch buy – Jake O’Donnell (SearchConsumerization)
The consolidation of the EMM world continued with the biggest move yet: VMware will purchase AirWatch for a record-breaking $1.54 billion to shore up its EUC division.
3. FBI warns of memory-scraping malware in wake of Target breach – Brandan Blevins (SearchSecurity)
According to a report by Reuters, the FBI is providing retailers a a confidential, three-page document warning them that more cyberattacks, like the recent Target data breach, could happen in the near future.
4. Microsoft might be tweaking Windows 8.1.1 to use less memory – Ed Hardy (TabletPCReview)
With mid-range Windows tablets such a hot item right now, an unconfirmed report has emerged that an upcoming update to Microsoft’s operating system will make life easier for organizations by using less RAM and storage space.
5. NFL uses Extreme Networks for wireless analytics in its stadiums – Gina Narcisi (SearchNetworking)
The NFL will begin using Extreme Network’s wireless analytics to give its IT organizations insight into how fans are using in-stadium Wi-Fi.
Microsoft Windows image via Shutterstock
Are you getting nervous about migrating off Windows XP? Well, in this week’s roundup, you’ll find out why you can breathe a sigh of relief for a little longer.
1. Windows XP anti-malware support extended until 2015, buys IT time – Diana Hwang (SearchEnterpriseDesktop)
If you or your organization are scrambling to migrate off of Windows XP by its April end-of-life date, you can breathe a bit easier as Microsoft will extend anti-malware support until July 2015.
2. Could the end of Nirvanix storage mean the end of cloud nirvana? – Jonathan Eunice (SearchCloudComputing)
With Nirvanix’s collapse along with recent data privacy news, many enterprise IT organizations are becoming wary of going all-in with cloud computing services.
3. Major retail breaches highlight point-of-sale security weaknesses – Brandan Blevins (SearchSecurity)
After the recent data breaches at Target and Neiman Marcus, several experts say point-of-sale security is just one of many payment-processing infrastructure problems.
4. Mid-range tablets will turn Microsoft around – Ed Hardy (TabletPCReview)
Even though Microsoft may be going through a rough time, there’s room for optimism for the company: A new crop of Windows tablets offer good featuresets for low prices.
5. The IT master plan for 2014: IT managers look ahead – Mark Schlack (SearchCIO)
TechTarget’s annual IT priorities survey finds that the IT master plan for 2014 includes heavy doses of business intelligence, mobility and Windows 8.
By James Kobielus (@jameskobielus)
Logs of all sorts–web logs, application logs, database logs, system logs, etc.–are fundamental to the promise of the Internet of Things (IoT).
Without continuous logging of relevant events, the IoT can’t fulfill its core role as the real-time event-notification bus of the online world. Machine-readable event logging is fundamental to all the core applications of IoT, including real-time sensor grids, remote telemetry, self-healing network computing, medical monitoring, traffic management, emergency response, and security incident and event monitoring. Ubiquitous IoT will depend on the ability to support continuous real-time ingest, analysis, correlation, handling, and any-to-any routing of machine-generated information.
IoT’s development depends on implementation of a ubiquitous, general-purpose event-logging infrastructure. This global logging infrastructure must be able to support disparate relational and non relational logged data types; execution of advanced analytics against myriad logged data objects; agility to work in batch and streaming environments; scalability to support growing volumes of in-flight log data replication without choking or slowing down. Individual event logs need not be peta-scale; in fact, most IoT devices will support local logs that are constrained to their increasingly tight storage constraints and disparate form factors.
I recently came across a great article on the untapped potential for general-purpose logging infrastructure in the IoT age. Though the author, LinkedIn software engineer Jay Kreps, doesn’t specifically connect his discussion to IoT, the affinity is obvious. The two trends that he highlights as increasing the need for distributed data logging–“event data firehose” and “explosion of specialized data systems”–are at the very heart of the IoT revolution.
Kreps lays out a real-time pub-sub architecture reminiscent of the time-honored concept of an “enterprise service bus” (ESB). “[M]any of the things we were building,” he says, “had a very simple concept at their heart: the log. Sometimes called write-ahead logs or commit logs or transaction logs, logs have been around almost as long as computers and are at the heart of many distributed data systems and real-time application architectures.”
To the extent that we intend for IoT to evolve into an ESB-like infrastructure for big data applications, we must grapple with the central role of distributed logs and with the protocols that support that support distributed log-data consistency, replication, and concurrency.
It seems to me that this loosely-coupled ESB-like approach for data integration is the best infrastructure for truly flexible, increasingly heterogeneous big data, IoT, and cloud infrastructures. The log will be the common denominator data-storage and integration abstraction.
Target image via Shutterstock
Between CES 2014 and Target’s data breach mess, there was plenty to talk about in the IT industry last week. And, as always, the TechTarget writers covered it all in this week’s roundup.
1. New Samsung enterprise mobility deal may give Knox a kick start – Jake O’Donnell (SearchConsumerization)
Samsung’s new deal with the U.S. government MDM partner DMI has analysts thinking the company’s enterprise offerings could get the needed jolt it needs in the New Year.
2. Target breach update: Information on up to 70 million customers stolen – Brandan Blevins (SearchSecurity)
More bad news for Target as the company has disclosed that over 70 million customers were affected in it’s recent credit card data breach, a much bigger estimate of the previous 40 million.
3. For IBM Watson, no easy answers on commercial cognitive computing – Jack Vaughan (SearchDataManagement)
When IBM’s Watson analytics system bested ‘Jeopardy!’ champs in 2011, all of the world cheered. Now the company must up its game to make Watson a commercial success.
4. Fun tech finds of CES 2014 – Michael Epstein (NotebookReview)
During CES 2014, many quirky gadgets and devices were unveiled including a gravity-defying window washer and edible (yes I said edible) 3D-printed art. What was your favorite gadget from CES 2014?
5. CES to CIOs: Watch out for wearable technology – Karen Goulart (SearchCIO)
Speaking of CES 2014, the big story throughout the conference was “wearables” as wearable technology is here to stay.
Technology image via Shutterstock
With the New Year’s celebrations behind us, we should start looking ahead right? Well, the TechTarget writers don’t see it that way as they are still uncovering top stories from 2013 as well as looking ahead to 2014.
1. DaaS will grow when Microsoft blows up Windows licensing rules – Alyssa Wood (SearchVirtualDesktop)
One DaaS supporter, tuCloud Inc’s CEO Guise Bule, sounds off on unfair Microsoft Windows licensing rules for cloud service providers and the cloud desktop industry in 2014.
2. SDN, hyperscale data center technologies to impact strategies in 2014 – SearchDataCenter Editorial Staff (SearchDataCenter)
With many IT professionals spending a lot of time fighting fires and focusing on strategic improvements, software-defined everything, real-world implementations of converged infrastructure and a hybrid cloud strategy will be on their agenda in 2014.
3. The great eight: Popular enterprise technology stories in 2013 – Sonia Groff (SearchNetworking)
From faster switching to ambitious wireless initiatives, the networking industry went under major changes in all facets. To keep you caught up, here are some of the most popular stories that appeared on SearchNetworking.
4. HTC posts quarterly losses for the ninth time running – Brighthand Editorial Staff (Brighthand)
Even with several successful innovations lately (particularly the HTC One Max), the company has suffered another decline in sales after posting its fourth quarter earnings.
5. Searchlight list of lists: The best, the worst and what’s next in tech – Karen Goulart (SearchCIO)
Was 2013 another flop for the technology industry? Well to end this year’s Searchlight articles, SearchCIO took a look at the best and worst of technology in 2013. What should we look forward to in 2014?
Happy Friday! Now before you go off enjoying the weekend, we thought to bring you a little “IT humor” (courtesy of Imgur). Enjoy!
Christmas gift image via Shutterstock
Are you feeling the after Christmas blues? Perk up as the TechTarget writers continue to share their top stories from the past year. There may also be a few gift ideas in there too.
1. Top 10 data center trends that shaped 2013 – Sharon Zaharoff (SearchDataCenter)
Whether it has the evolution of servers, storage and mainframes or more affordable and accessible technology, the SearchDataCenter’s editors share their top 10 data center stories of 2013.
2. Businesses must cut through the big data hype in 2014 – Ed Burns (SearchBusinessAnalytics)
While there’s a ton of hype surrounding analytics and big data these days, it’s very unlikely that every new technology or strategy will pan out. Businesses will need to be able to differentiate the trends in order to be successful in 2014.
3. After Christmas gift ideas – Jerry Jackson (NotebookReview)
If you’re still trying to figure out what you should get with your new gift cards (maybe a new laptop or some nice accessories), here are some great-after Christmas presents for the technology lover.
4. The cloud looms large over IT career paths – Christina Torode (SearchCIO-Midmarket)
In the recent TechTarget “IT Salary and Careers Survey”, IT professionals believe their profession is on the cusp of significant change due to the much-hyped cloud computing industry.
5. Top 10 UK cyber security stories of 2013 – Warwick Ashford (ComputerWeekly)
As cyber security has moved up the national agenda in the past year, with the UK government paying increasing attention to bolstering UK cyber defenses, here are the top 10 UK cyber security stories of 2013.
2013 image via Shutterstock
As we prepare for 2014, it’s always great to take a look back at some of the most popular stories, quotes and news from the past year. And that’s exactly what the TechTarget writers did in this week’s roundup. What was your favorite story / quote from the IT industry in 2013?
1. SDN market predictions for New Year: NFV, OpenFlow, Open vSwitch boom – Shamus McGillicuddy (SearchSDN)
While the SDN market remained more hype than reality in 2013, SDN technology will take a big step forward in 2014 as overlays will rule the early part of the New Year, but OpenFlow and other hardware-based SDN will start to gain ground.
2. Massive Target data breach: Retailer says 40 million cards compromised – Brandan Blevins (SearchSecurity)
Target Corp. confirmed last week that its aware of a data breach impacting approximately 40 million credit and data debit cards, making it one of the largest breaches of 2013 and one of the largest retail breaches in history.
3. They said what?! This year’s most notable cloud computing quotes – Adam Hughes (SearchCloudComputing)
“Everyone needs a backup plan.” “The Achilles’ heel for Amazon is the noisy neighbor.” These were just some of the best cloud computing quotes that stuck out in our minds in 2013.
4. The top Exchange Server content of the year – SearchExchange Editorial Staff (SearchExchange)
Take a look at why these five pieces of Exchange Server content struck a chord with our readers this year.
5. Top 10 VDI news stories of 2013 – Margaret Jones (SearchVirtualDesktop)
As we saw big VDI news around popular products from VMware, Citrix and Amazon, here are some of the biggest stories from the VDI landscape in 2013.