Software Quality Insights

A blog

October 29, 2008  7:00 PM

More from the e-voting front

Posted by: Colin Smith
Application security, Software Quality, Software testing

Two more stories about e-voting machines were reported this week. The first is about a report from Princeton University that says an e-voting machine in New Jersey can be hacked in seven minutes.

In its report, the university says it is possible to hack the Sequoia AVC Advantage 9.00H DRE (direct-recording electronic) voting machine by loading fraudulent firmware.

Sequoia has responded to the Princeton study with a report of its own, rebutting many of the claims in the Princeton report.

Princeton’s report, which was conducted during the summer as part of a lawsuit in New Jersey, was allowed to be released just a couple weeks ago. The lateness of the report — and the examination of the e-voting systems — is because of the time it has taken a 2004 lawsuit against the state for using DRE machines to progress.

In 2004 a group of public-interest plaintiffs sued the State of New Jersey over the State’s use of DRE voting machines. The plaintiffs argued that the use of DRE voting machines is illegal and unconstitutional.

The case was dismissed in January 2005 by a trial court, but then appealed. While the appeal was pending, the state legislature passed — and the governor signed — a bill requiring that no later than January 1, 2008, any voting system in New Jersey must produce a voter-verified paper ballot.

In 2006 the Appellate Court reinstated the lawsuit and instructed the trial judge to monitor the progress of State election officials in meeting the legislature’s deadline. In 2008 the executive branch twice requested delays to the deadline and the legislature obliged.

Based on concern that the state would not meet the deadline, the lawsuit was allowed to continue and the judge ordered that the state provide to the plaintiffs’ expert witnesses the voting machines complete with their source code. The witnesses, who are authors of the Princeton report, examined the voting machines and their source code during July and August 2008 and delivered their report to the court on Sept. 2. A court order permitted them to make their findings available to the public 30 days later.

So, the state of New Jersey had four years to improve its e-voting systems and prevent a lawsuit, yet it did not. And now voters in that state once again are using machines that can be tampered with and don’t produce paper ballots — and once again face the possibility that their votes may not count.

E-voting problems in Finland
The other story being reported is that usability problems in Finland’s pilot e-voting system caused 2% of votes cast to be lost.

With that system, voters were required to insert a smart card to identify the voter, type their selected candidate number, press “OK”, check the candidate details on the screen and then press “OK” again. Some voters did not press “OK” a second time and instead removed their smart card prematurely, causing their ballots not to be cast.

October 27, 2008  5:25 PM

Security review of Florida voting system

Posted by: Colin Smith
Application security, Software Quality, Software testing

Since writing about the Florida voting experience, it was brought to my attention how the state of Florida commissioned an independent expert review of the remote voting software that is being used in Okaloosa County. A team from the Florida State University’s (FSU) Security and Assurance in Information Technology (SAIT) Laboratory reviewed the Pnyx.core ODBP 1.0 remote voting software developed by Scytl.

The software is for use in the Okaloosa Distance Balloting Pilot (ODBP), which will test remote e-voting for about 1,000 overseas voters whose permanent residence is in Okaloosa County. It replaces other absentee voting mechanisms for participating overseas voters.

Under this pilot, voters will enter their votes electronically, those votes will be transmitted over the Internet, and the votes will be tabulated electronically.

The state of Florida, which certified this system at the end of September, always certifies its voting technology and processes. And in the past an independent review was done of the then-named Diebold systems. What makes this review stand out is the vendor’s willingness to cooperate and provide a full build environment for the source code.

“Scytl provided VMWare virtual machine images containing a full build environment, scripts to drive the build process, and step-by-step documentation describing how to initiate the build process,” according to the team’s report.

Doing that saved the team “significant” time and made it possible to apply static analysis tools to the software. The team used reports from two static analysis tools:

  • Fortify SCA, which Fortify donated, was used by the team.

  • Klocwork Insight was used by the Florida Division of Elections.

Additionally, the team participated with the vendor in an online question-and-answer exchange that “proved invaluable to the study.”

The team’s final report was mixed; it reported some good things, but it also found some bad things. In general, it passed review and was certified by the state.

But the important thing to take from this was the process and the cooperation of the vendor. This is hopefully the start of how things are done for the 2012 election.

“There are very few developers engaging with vendors such as [Klocwork] or state-sponsored programs to make their code usable in four years time or eight years time,” said Gwyn Fisher, Klocwork’s CTO.

Brendan Harrison, director of marketing at Klocwork, said it’s hoped that this review is used as a model going forward.

“The e-voting marketplace burst on the scene, and what we see happening is that the e-voting vendors are going to have to change how they develop software and work more cooperatively with the authorities,” he said.

The e-voting market needs to transition to one that is regulated in order to enforce good standards, a high-quality process, and a secure development lifecycle.

October 27, 2008  12:41 AM

Voting machines — what do you think?

Posted by: Colin Smith
Software Quality

On we’re asking readers if they think voting machines (their usability, bugs, or security concerns) will cause problems for next month’s U.S. elections. What do you think? Take the poll and let us know.

October 23, 2008  11:38 PM

What are your software requirements headaches?

Posted by: Colin Smith
Requirements gathering, Requirements management, Software Quality, Software requirements validation

Understanding what your stakeholders want in an application can be challenging, to say the least. You need to know what questions to ask and get your stakeholders to explain their needs and wants. It requires not just eliciting requirements but also validating that what you’re ready to send to the developers is in fact what your stakeholders want. Communication with the stakeholders and with the developers is essential.

That’s just one headache business analysts often have. Others that I’ve heard people talk about:

  • Transitioning from legacy requirements or documents (Word, Excel files) to use cases
  • Transitioning from per-project requirements to per-system requirements
  • Being asked to specify many of the user interface details as requirements (Too much UI detail in the requirements constrains the designers and takes away from the functional requirements)
  • Managing requirements across the enterprise
  • Managing requirements for reusable components (How to achieve effective reuse across the enterprise)

Additionally, more people are asking about how to manage and define software requirements in agile environments. How do you handle changing software requirements?

Do any of those issues cause headaches for you? Are there other things related to software requirements that create problems for you or you need more information about? Tell me about your pains — be as specific as you want.

Think of me as your doctor: Tell me where it hurts, and I’ll try to help you get rid of the pain. Only I won’t charge you for an office visit. :-)

October 22, 2008  7:27 PM

West Virginia has e-voting concerns

Posted by: Colin Smith
Software Quality

We’re a few days into early voting for the U.S. presidential election, and some West Virginia voters are voicing concerns with e-voting machines from Election Systems and Software. In Putnam County and Jackson County, voters reported that their electronic votes for Democrats were switched to Republicans.

Clerks in both counties say the incidents are isolated, and they blamed voters for not being more careful. Additionally, West Virginia Secretary of State Betty Ireland says she is confident the machines are up to the task. This despite the numerous reports detailing problems with the company’s machines.

The machines being used, however, do not provide paper receipts. It’s important that voters double check their votes before leaving the voting booth. If a voter notices a discrepancy, he needs to tell a poll worker so the mistake can hopefully be corrected. Once a voter’s ballot is cast, he will not be allowed to vote again.

October 22, 2008  1:53 PM

One solution to software requirements challenges

Posted by: Colin Smith
Requirements gathering, Requirements management, Software Quality, Software requirements validation

If you’re responsible for making sure stakeholders get the software that they want, then you’re probably all-too familiar with the four aspects of software requirements — elicitation, elaboration, validation, and acceptance.

Increasingly I hear how an iterative approach is best and how tools can help. One tool that sounds like it could help is Blueprint’s Requirements Center. It provides a single environment for everything — there’s no need to leave the environment.

The elicitation tool provides “rapid requirements capture.” You can use it to identify and capture the relationships between the different requirements, you can capture images to include, you can capture data definitions, and you can import requirements from Excel spreadsheets.

The elaboration tool helps you start to make sense of everything. You use it to start to model the business process and the applications. It provides a GUI center to show interfaces of the software. And because it records the traceability of requirements, you can see what is impacted if a requirement is changed.

When it comes to validating what you have with stakeholders, you can create an end-to-end workflow diagram. You create a simulation to review with the stakeholders, and then gather their feedback. That feedback is entered directly into the center. You also have the option of passing the simulation around, and stakeholders can enter their own comments.

When the stakeholders give their OK, signaling that you’ve got it right, you can then generate the standard documents required for signoff. And those signoffs can be recorded on the server. When all is said and done, you’re giving the designers “a very comprehensive and complete diagram of what stakeholders want,” said Tony Higgins, vice president of products at Blueprint.

Additionally, the requirements center can generate all of the functional tests that correspond to the requirements. These are “ready-to-run” tests, Higgins said.

Last week Blueprint released new features for the requirements center. Blueprint Requirements Center 2009 Feature Pack Two introduces the Blueprint Resource Center. It provides analysts with instructional materials such as videos, samples, and best practices; company-specific templates and guides; advice from Blueprint experts; and syndicated articles and tips from Web communities and blogs.

Feature Pack Two also enhances integration with HP Quality Center. Now, requirements definition meta data (including visual requirements, GUI prototypes, security requirements, and data elements) are seamlessly integrated with HP Quality Center’s Requirements Management and Test Management modules. In addition, HP Quality Center users also have the ability to import and leverage assets within Blueprint’s elicitation module to provide early visibility and to speed IT development and quality assurance teams.

Want to see how the various modules work? Blueprint provides online demonstrations of its Blueprint’s Requirements Center.

October 16, 2008  3:40 PM

The value of certification

Posted by: Colin Smith
Certification, Software Quality, Training/education

Certification — this controversial topic continues to pop up and be discussed in columns and blogs.

James Bach, long an opponent of certification, recently had an interesting experience with a representative of ASTQB (American Software Testing Qualifications Board). He asked her about what it takes to receive a testing certification from her organization and the benefits of certification. The dialog, which James provides in his blog, is hysterical and illustrates the ignorance people in that organization have about testers and testing. Here’s a sample:

James: Do you need any experience to get certified?

Lois: No, you just have to pass the exam.

James: What are the benefits of certification?

Lois: JB.

James: JB?

Lois: Just Because. There are almost 90,000 certified testers. It’s fast becoming the norm. In some countries you can’t get a job unless you have our certification.

Project managers have had similar complaints about certifications for their profession. Bas de Baar wrote about his experiences in his column “Finding work as a PM: Value of certification debatable.” He says experience counts for far more than what a person is able to remember from a book, and he finds it unfair that job applicants aren’t given a second look if they don’t have the certification.

In Kevin Beaver’s recent column “Does certification really matter?” Kevin agrees that it’s wrong to simply memorize a book in order to get certified. Certification should mean more than that. Certified professionals should be able to “execute in real-world scenarios,” he says.

And while getting certified is a marketing tactic, it’s often necessary to move ahead in your career, Kevin says. You can disagree all you want, but employers look for those certifications when making hiring decisions.

If certifications are necessary, then the certification bodies need to make sure the tests cover actual experience. And make sure what is being taught in certification training sessions is at the appropriate level.

Some other thoughts: If company and personal budgets get tighter in light of the economic problems we’re dealing with, will certification and training get pushed to the back burner? Will fewer certified people mean employers will have to look more carefully at job applicants? Will certifications have more value because they’ll be given to people who are really committed to their profession?

October 16, 2008  1:23 PM

Agile tool tracks app changes

Posted by: Colin Smith
Agile software development, Requirements gathering, Software change management, Software Quality, Software requirements validation

Even when you think you’ve elicited and validated all of your stakeholders’ requirements, you’re still bound to have users who are unhappy with an application or a feature within an application.

Determining what the problem is and what users would prefer, however, can be challenging. But a feature in OutSystems’ All-In-One Agile Suite can help.

In its recently announced suite, OutSystems includes Embedded Change Technology (ECT), an automated mechanism for collecting business users’ feedback directly from a running Web application. Users point and click on the area in the applications where they want a change to be made and write their comments in the ECT pop-up window. The feedback is then made available to project managers and developers for review from the Agile Network’s Projects component.

More than that, members on the development team can use ECT. Business analysts can use it to validate requirements, QA engineers and testers can use it to flag problems and post comments, and programmers can use it to communicate with testers.

OutSystems is targeting Web 2.0 development teams who follow agile methodologies. The suite, which will be available in early November, includes the following:

  • ECT

  • New Agile Network — a portal for accessing OutSystems’ purpose-built Agile project management tools, online training, and knowledgebase of expertise

  • Agile Platform 4.2 — an enhanced version that simplifies the creation of Web 2.0 applications leveraging Ajax

Visit OutSystems site to see demonstrations of its All-In-One Agile Suite.

If you’re struggling to give your stakeholders what they want in an application or if communication on your team is lacking, it’s worth checking this tool suite out.

October 14, 2008  4:42 PM

Florida voting machine update

Posted by: Colin Smith
Software Quality, Software testing

I experienced a bit of relief today after reading that a mock presidential election in Palm Beach County resulted in perfect performance from the voting equipment. County officials tested voting equipment using 1,332 ballots marked by election workers. “The votes were tallied perfectly and quickly,” according to the Sun Sentinel article.

The Obama and McCain campaigns are keeping a close eye on the county’s voting equipment and processes should there be another close vote and they decide to challenge the election results. Let’s hope that doesn’t happen. We don’t need another election like the 2000 presidential election.

October 10, 2008  4:01 PM

Human, voting system errors a concern in Florida

Posted by: Colin Smith
Software Quality, Software testing

No matter how much or how well you test a system, human error can still bring the whole thing down. The voting system in now-infamous Palm Beach County in Florida is one example. Since the 2000 presidential election (at least), the county has dealt with significant system and human errors in its voting system.

The 2000 presidential election, with Al Gore (D) and George W. Bush (R) on the ticket, brought us the punch-card ballot that was designed in a butterfly format. The candidates were listed on both sides of the ballot rather than just on one side in order to save space — and printing costs. What the designers did not consider was the confusion this would cause for its senior citizen voters — of which there are many in the county. Even I — still a young-ish person — was thrown by the layout and rechecked my voting several times before leaving the voting booth.

What resulted was several votes cast for Pat Buchanan rather than Gore. Usability testing was either overlooked or done very badly. So, we have design errors combined with human error.

More human error came to light when the vote was so close that it required a recount. Several things happened: some districts could not find the election machines — volunteers did not return them to the district and had left them in the polling locations — the electronic tally came out with a different result but still too close to determine a winner.

That led to the dimpled and hanging chads in the ballots’ hand count. (If you don’t know, a chad is the tiny piece of paper that is punched through on a ballot.) Lawmakers had to decide if such chads represented the will of the voters. More than two weeks after the Nov. 7 election, a Florida Circuit Court Judge decided that they do. How do you get dimpled chads? You get them when people don’t push the pin hard enough. And you get them when the voting machine hasn’t been cleaned out and hundreds of other chads jam the machine — preventing voters from pushing their chads out completely. The entire chad thing could have been prevented if the machines were maintained properly.

As a result of that voting fiasco, the county decided to switch to electronic voting. It will solve all those problems, supervisors were told, and they rushed out to spend $56 million to implement e-voting machines. They saw demonstrations of the machines, but they did not ask about nor were they told about possible security problems. It wasn’t until the machines were purchased that the security issues came to light. Those machines were easy to hack, and by decision of the state they did not have a paper trail. That meant recounts were impossible. (Read’s story on e-voting flaws.)

The 2004 presidential election did not have as much drama or controversy as the 2000 election, but throughout it and local elections thereafter, legislators fought for some kind of paper trail. That did not happen and so the county has a whole new voting system for 2008. Unfortunately, it too has problems.

The new voting system is now a paper ballot on which voters are asked to fill in the missing gap of an arrow that points to the candidate they are voting for. And then feed the ballot into an optical scanner. During the recent primary election, again there was voter confusion. People did not know what to do, so they drew a thin line or circled the person’s name. Neither technique counts.

That election, too, had a controversy — this time for the Palm Beach County Circuit Judge race. It was another close vote, which requires a recount. However, for the first recount some ballots were missing and those results were thrown out. They eventually found the ballots, did a second recount but experienced a problem with the tabulation machines. Then an additional 156 ballots turned up. A third recount finally resulted in a winner.

Since then, a test of the high-speed counting machines, requested by the lawyers for the person who lost the election, found the machines couldn’t count the same ballots the same way twice. That’s just great (she said sarcastically).

Here we are just weeks away from the next presidential election, with Florida considered a swing state, and system and human errors again threaten to disrupt election proceedings — if not create chaos.

Only 4,093 people voted in the recent primary election; for the Nov. 4 election, more than 800,000 people have registered to vote. Just voting is expected to be painful, with long lines expected. But even if you get through that, adding up the final tallies from all the precincts is expected to take significantly longer, and there’s a chance your vote won’t be counted.

Needless to say, people here are concerned.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: