Systems Management

Brave new LinuxCon

I’ve been in Portland, Ore., this week attending the inaugural LinuxCon, hosted by the Linux Foundation. The event was a bigger draw than the organizers had anticipated, with about 600 attendees registered, making the WiFi in the rooms a bit spotty and the keynote hall a bit crowded, but all in all more interest in Linux is a good thing. The sessions have been a mix of big picture Linux evangelism to detailed technical sessions for developers with the weakest area being sessions that were designed to attract the systems administrator set.

Linux from the kernel to the big picture
Highlights so far have included a kernel maintainer panel discussion featuring Linux kernel founder Linus Torvalds, Jon Corbet of LWN.net, Chris Wright from Red Hat, IBM’s Ted Ts’o, Novell’s Greg Kroah-Hartmann and moderator James Bottomley. While many positive things were said about the panel discussion, one sentence uttered by Torvalds got the most attention.

“We’re getting bloated and huge, and yes it’s a problem,” said Torvalds in reference to the size of the kernel. “I would love to say that we had a plan. Our icache footprint is scary.”
Continued »

Surveillance tools beat hidden malware at its own game

Just as surveillance tools have flourished in the physical world because they can monitor systems in hiding – think nanny cam – such seemingly invisible monitoring systems have flourished in the digital domain.

Rootkits, a form of malware designed to take control of a system without the authorization or knowledge of an administrator, can wreak havoc on a system and compromise everything it does by infecting code, nestling within it and becoming the malevolent phantom of an OS.

If security plays second fiddle to other system administration duties, your system may be just as much at risk as if you didn’t monitor it at all. But a new crop of rootkit detection tools is designed to detect these malware breaches and, in some cases, beat malware at its own game.

Products such as F-Secure Blacklight and OSSEC help protect system information from being used against you by making it inaccessible to nonadministrators. Unlike traditional antivirus scanners, these tools examine the system at a deep level to detect active rootkits and rout them out. Tools like Blacklight also tout themselves as user-friendly and nontechnical .

The new security tool ProcL goes a step further by hiding information about which version of software a system uses. As a result, malware attempting to gain system access cannot tell whether the system has software from 2008 or 1988 and will likely move on to an easier target.

For more on ProcL, see the Scanit website. And to check out an advanced security “hiding” tactic involving virtualization, click here.

Varonis explains data governance, product DatAdvantage

During the week of LinuxWorld, Johnnie Konstantas, a marketing VP at Varonis Systems, a data governance software provider, talks about the company’s release of DatAdvantage and approaches to data governance within a company. 

What does DatAdvantage do?                                                                                                                     
Johnnie Konstantas: The focus … was to automate user-to-data mapping such that only the right users have access to only the data they need at all times. A sophisticated mathematical engine computes permissions revocations so that user access to data is always warranted by business need. DatAdvantage also ensures that data use is business warranted by providing the means to continuously monitor what users are doing with the permissions they have. DatAdvantage logs every user’s every “file touch” (i.e., open, delete, create, rename) and provides this information as part of a consolidated and searchable record. 

Why is data governance so important, even more so than it was five years ago?                               
Konstantas: IT managers are currently responsible for controlling access to business-critical and sensitive data — 80% of which takes an unstructured form (i.e., documents, spreadsheets, presentations, image and multimedia files, source code). A system of data governance that includes people, processes and technology for ensuring that access is warranted is the only way to implement access controls that are consistently applied and enforced through data growth and change. The rate at which unstructured data is created outpaces that of five years ago. Digital images from scanners and cameras, portable audio files, podcasting and Web content are being added to the scores of documents and spreadsheets that are produced for business communication. All of this now “business relevant” data must be protected, and access to it controlled as it is for documents and software code. In the next three years, the rate of data creation will increase still.  

What trends has Varonis seen in data access rights within a company?                                            
Konstantas: As a general trend, companies are seeing the need for unstructured data management become more acute as data growth explodes. IT operations are turning to Varonis to automate a process which is largely manual and quite costly. The steps taken for data management auditing and control for unstructured data are being rolled into the thinking, models and projects for overall data governance. 

How should a company think about data governance to make the most of DatAdvantage?   
Konstantas: Since most enterprises have not reviewed their data entitlement settings in some time (the process is almost impossible without Varonis which automates it), step one is to have IT review and clean up unwanted access controls. Then the data management shift from IT staff to data stewards can take place. A good governance environment has a specialized team in charge of entitlement management, but also involves IT in auditing the process.

How does data governance overlap with data security? How does DatAdvantage differ from data center security software?                                                                                                                   
Konstantas: Data governance as a model and framework includes the safeguarding of data that is business-sensitive. IT greatly reduces the risk of data loss and misuse by revoking unwarranted permissions and limiting control according to a business’s need-to-know. Data governance comprises much more than security, however; it ensures that data stewards rather than IT staff manage entitlements to data and determine which data is worthy of archiving, deleting, preserving and protecting. The two products in Varonis’ data governance suite provide the means to both manage entitlement to and determine treatment of data. DatAdvantage helps IT remove excess permissions and identify data business owners. Varonis DataPrivilege puts data stewards in charge of their data by giving them the means to manage all entitlement requests and to audit data use.

Splunk highlights data management maturity at LinuxWorld

Software company Splunk creates products that aid companies primarily in log file management - collecting information about the data in their systems and continuously reporting it back. At this year’s LinuxWorld Conference & Expo, Splunk will highlight several further-reaching data management products: Splunk for Virtual Server Management, Splunk for Change Management and Splunk for Server Management.

The products, in providing fuller access to information about what and how your system is doing, promise to make system management more practical and security maintenance more immediate.

The products being released this week at LinuxWorld integrate log file management with a variety of other tasks. They can simultaneously manage log files and collect and manage messages, traps and alerts as well as statistics from all system areas.

As one administrator commented on the blog of Splunk CEO Michael Baum, “Log file management is DEAD.” It is becoming just one side of the larger task of system management. For help on configuring Splunk, check out this tip.

Trusted Computer Solutions intros security automation tools at LinuxWorld

San Antonio, Texas-based Trusted Computer Solutions will release a group of security management features next week at LinuxWorld Conference & Expo that will perform pre-packaged assessments and configuration procedures so that IT managers won’t have to do so manually.

Called “lockdown profiles,” these features enable IT managers to quickly assess systems for security and compliance with four distinct sets of security standards: PCI DSS (credit card security standards), JAFAN (Joint Air Force Army Navy), DCID (Director of Central Intelligence Directive) 6/3, and CIP (Critical Infrastructure Protection). The four profiles are an addition to the company’s product Security Blanket Enterprise Edition.

They are also the most recent in a series of releases from TCS in the past year. The LinuxWorld release of Security Blanket Enterprise Edition will also be able to take snapshots of system security configuration and then provide those snapshots for simplified comparison to previous configurations.

Security Blanket Enterprise Edition supports Red Hat Enterprise Linux versions 4 and 5, CentOS versions 4 and 5 and Oracle Enterprise Linux versions 4 and 5.

Package managers: Downloaders beware

Package management—the process of determining which update packages should be installed on a host and then downloading and installing those package—invites a dilemma : OSes need to updated, but the process of updating them can invite security breaches.

A recent study at the University of Arizona explored nine feasible attacks on the popular package managers APT and YUM. As part of their research, the study’s conductors posed as a group of administrators from a nonexistent company and leased a server from a hosting provider. Thousands of clients, including government agencies, downloaded upgrades, which prompted their operating systems to endlessly replicate data, misidentify dependencies, and install unnecessary software. It also left these clients vulnerable to other attacks on their systems, including hackers gaining root access to OSes, system crashes and erased files . Researchers concluded that many public storage spaces for upgrade downloads are in fact maliciously established “mirrors,” or software repositories , that have become infected with sources of attack. You can prevent most of these issues by downloading from only signed metadata repositories, the study counseled. A signature verifies that the repository was created benevolently.

Protecting against mirror threats
In response, readers suggested a number of additional ways to protect a package manager from such threats.

• An OpenSUSE page suggested its internally developed tool, download redirector.
• One blogger wrote that the risks posed by infected repositories are not great enough to merit changes to package manager security.
• Another acknowledged the risk and argued that simply allowing the number of open source package manager products available to increase will maintain or improve current open source package manager security.
• A Gentoo administrator promoted rotating mirrors to ensure security.

Package manager security, as pointed out by this report, is crucial to the success of your operating system. With the present drive for continuous upgrades for your data center, you may feel pressure to download from the most accessible source available. Don’t: the risk of downloading insecure software is greater than the time it will take to check out the above links.

For more on package managers, check out these links: How to manage software on Ubuntu Server with “aptitude” and “apt-get”

Managing Software on Ubuntu Server Edition

Construction firm turns to open source for systems management

When Sam Lamonica first joined Rudolph and Sletten as CIO in 2003, one of his first orders of business was to stabilize the IT infrastructure at the construction firm. “There were all sorts of IS problems affecting the network and the applications,” Lamonica recalled. And with no uniform systems management tools in place, the IT group was essentially flying blind. “We knew we had a problem when one of two things happened,” Lamonica said. “We’d get calls from end users telling us an application was down, or we’d look at the servers and see that the lights were not blinking.”

A 600-employee commercial construction firm with four offices in California, Rudolph and Sletten relies heavily on its IT infrastructure to conduct business. At any given time, the company operates 50 or 60 construction projects at job sites that last anywhere from one to five years; each job site essentially operates as a temporary regional office that requires all the connectivity and applications as a permanent office. So keeping tabs on applications and the network from the company’s data center in Redwood City, Calif., is critical to keeping remote operations running smoothly.

In a previous job, Lamonica got to know Bob Fanini and Dave Lilly, the two entrepreneurs who went on to start GroundWork Open Source Inc., a provider of open source IT and network monitoring software. “The previous company I worked for – Phoenix Technologies – was actually GroundWork’s first customer,” Lamonica said.

In addition to GroundWork, Lamonica was also familiar with HP OpenView, another monitoring and systems management tool. “We really needed something we could implement quickly, so we went with GroundWork,” Lamonica says.

Within six weeks, Rudolph and Sletten had its first set of diagnostics and assessments, courtesy of the GroundWork Monitor Professional tool. Today Lamonica uses the tool to monitor the entire infrastructure, from enterprise business applications and email to servers and network devices. Reliability and stability have improved markedly, from about 80% when Lamonica arrived to 99.99%.

Gone are the days when IT was the last to know when a problem occurred. “We set thresholds, so that we know well before issues arise,” Lamonica said.

Lamonica said that GroundWork has changed some minds regarding the use of open source at the primarily Windows-based Rudolph and Sletten. “We don’t really care if something is open source anymore,” he said. “We just want a solution that fits our needs.”

UPDATE REMINDER: Product of the Year nominations are going on now!

Working with vendors is tough. You need their help, they want your money. Hopefully, whatever it is they help you install works and the price meets you both somewhere in the middle (as in your side of the middle, right?).

Sometimes this process is a headache, but sometimes a project can really surprise you—things just work and upper management is just peachy keen with how the whole thing looks on the balance sheet.

In that vein, SearchEnterpriseLinux.com wants to help its readers discover the best of the best in Linux products for the enterprise in our prestigious SearchEnterpriseLinux.com 2007 Products of the Year awards. We’ve been asking readers and vendors over at SearchEnterpriseLinux.com to nominate a favorite product they’ve used or to nominate their own new product, and now we’ve opened it up to the Intertubes here at the Enterprise Linux Log. Regardless of where you fall — vendor, user or general Linux guru –the deadline is drawing near!

Our editorial team and a select panel of industry experts and analysts are currently accepting submissions online until 5 p.m. PST on Nov. 9, 2007 in a range of categories, including: Server Linux platform product (either a distribution release or a new, integrated server Linux offering); Security applications/tools for Linux on the server; Virtualization product for Linux on the server; and Linux administration tools. You can access the 2007 POY submission page in the link above.

To qualify, new or significantly upgraded products must have been shipped after October 31, 2006, and before November 1, 2007. Submit your entry today and let us know what you think are the top data center products on the market!

Red Hat’s big day

Yesterday, Red Hat offered a smörgåsbord to the press and the analyst community. Their main announcement: Red Hat is the greatest Linux and open source vendor on the face of the planet.

Actually, there were several announcements in yesterday’s conference call and webcast: within the typical sales and marketing noise was talk of virtualization at almost every level of the discussion, hosted by a trio of Red Hat executives.

The first of the announcements, regarding the official release of Red Hat Enterprise Linux 5.1, was made by Scott Crenshaw, Red Hat’s vice president of enterprise Linux business. In some prepared remarks, Crenshaw went after proprietary virtualization technologies, saying RHEL 5.1’s virtualization delivers broader server support and up to twice the performance that the competition.

The skinny on 5.1

There were no real surprises in this announcement, especially if you’re a regular reader of SearchEnterpriseLinux.com. Back in September we filed a preview article on 5.1 (RHEL 5.1 update tweaks virtualization, Windows interoperability), where we discussed the virtualizaiton updates with a few experts. Jan Stafford, our Senior Site Editor at SEL, had a 5.1 preview up as far back as May from the Red Hat Summit.

RHEL 5.0 was a success when it launched in March. The inclusion of Xen support was almost a full year behind Novell, which had baked in Xen paravirtualization back in June 2006, but it worked as advertised, albeit with a few tweaks here and there. “It’s not half-baked,” Illuminata analyst Gordon Haff told me at the time, “but it certainly doesn’t have the fit and finish we see with VMware.” Not many things do these days, as VMware loves to point out during their quarterly “ESX Server prints money!!!” press conferences. With 5.1 officially avaialble to Red Hat customers via the Red Hat Network, however, the consensus was that the gap got a little smaller.

Also back in September, Jim Klein, director of information services and technology at the Saugus Union School District in Saugus, Calif., told me that RHEL 5.1 is a “significant improvement over version 5 on the management side of things.”

In this regard, the Windows functionality in 5.1 is critical: IT managers are making decisions now about which platform to base their virtualized infrastructure on, Klein said. “If Red Hat can get their Windows drivers out soon, I think they will be well positioned to pick up significant market share in the coming year,” he said (Read Jim Klein’s Enterprise Linux Log guest blog post on Xen and Fedora 7 – J.L.).

Chance of clouds

Moving on, things got a bit cloudy during the press conference as Crenshaw and company (Paul Cormier, v.p engineering; and Brian Stevens, CTO) announced that beta availability of Red Hat Enterprise Linux on Amazon Elastic Compute Cloud (Amazon EC2), a web service that provides re-sizeable compute capacity in the cloud.

In a statement that accompanied the press call, Red Hat said the combination of RHEL and Amazon EC2 “changes the economics of computing by allowing customers to pay only for the infrastructure software services and capacity that they actually use. Red Hat Enterprise Linux on Amazon EC2 enables customers to increase or decrease capacity within minutes, removing the need to over-buy software and hardware capacity as a set of resources to handle periodic spikes in demand.”

As part of this partnership, Red Hat Network will offer a common set of management and automation tools across on-premises deployments and the Amazon EC2 cloud computing environment. Red Hat will provide technical support and maintenance of Red Hat Enterprise Linux on Amazon EC2. This is the first commercially supported operating system available on Amazon EC2.

As far as pricing and availability are concerned, RHEL on Amazon EC2 is available as a private beta today, with public beta availability planned for the fourth calendar quarter of 2007. Base prices are $19 per month, per user and $0.21, $0.53 or $0.94 for every compute hour used on Amazon’s EC2 service, depending on whether customers choose a small, large or extra-large compute instance size, plus bandwidth and storage fees.

Red Hat Appliance OS, HO!

The final piece of the pie was the pending release of Red Hat Appliance Operating System, or AOS for short. This ISV-themed OS means that in the very near future (first half of 2008, execs told me), ISVs will be assembling appliances for their customers that run on AOS and work with every certified RHEL application under the sun. Hint: That’s a lot, and was exactly the angle Red Hat executives took on the Wednesday call.

“The Red Hat Appliance Operating System will allow applications that are certified on Red Hat Enterprise Linux to be deployed as software appliances on the broadest range of servers in the industry, including those running Red Hat Enterprise Linux, VMware ESX and Microsoft Windows Viridian. Red Hat’s Linux Automation strategy, also announced today, delivers a standardized development, deployment and management infrastructure for the entire Red Hat Enterprise Linux ecosystem,” a statement said. Look for an industry reaction piece from us on SearchEnterpriseLinux.com later in the day.

The Red Hat Appliance Operating System (AOS) is built from Red Hat Enterprise Linux, with which it shares full ABI and API compatibility. It includes the Virtual Appliance Development Kit (vADK) that will allow ISVs to configure the operating system along with their middleware and applications to produce a complete system image.

Red Hat also announced that a range of software solutions on Red Hat Exchange are available for trial and purchase as pre-configured software appliances. Customers can now purchase and deploy an integrated solution consisting of third-party software, JBoss middleware and Red Hat Enterprise Linux. The total time necessary to purchase, install and use these solutions is “just minutes,” Crenshaw said.

A lot of PR in this announcement, so we’ll have to see where it goes in 2008. Stay tuned.

10 IT system monitoring best practices

Here are 10 best practices for system monitoring that Javier Soltero, CTO of Hyperic Inc., has seen succeed in his years in IT.

1. Define what it means for a given resource — a server, an application or a service — to be labeled “production”.
2. Figure out what monitoring you need to satisfy the production requirement.
3. Implement the monitoring capability, either manually or with open source tools like Nagios or commercial tools.
4. Define what it means for something to be “broken/unavailable/on fire” — also referred to as WARN/ERROR/CRITICAL.
5. Implement alerts in your monitoring system to capture these thresholds.
6. Define what process is to be followed for each alarm level.
7. Make sure your alerting process follows that notification process.
8. Create roles/responsibilities for groups of people to share alerts, control and detailed access to relevant their job function.Focusing individuals generally means better performance for their area.
9. Designate a small number of super-users that architect your entire system of alerts, monitoring protocols, roles, etc., to ensure they follow a single blueprint.
10. Lather, rinse, and repeat if necessary.

I pulled these tips from a LinuxWorld 2007 preview interview with Javier Soltero. In another excerpt from that interview — Virtualization boosts Linux adoption big-time — he talks about the synergy between Linux and virtualization and challenges posed in managing multiple-operating system environments and identifying and tracking virtual machines. Javier also offered some great comments on other subjects, which can be found in articles from our LinuxWorld and Next Generation Data Center Conference 2007 coverage here.