Uncharted Waters

Jan 11 2016   12:11PM GMT

The Princeton University Con

Justin Rohrman Justin Rohrman Profile: Justin Rohrman

Tags:
Application testing
Automation
University Study

Princeton University led by Andrew Appel just received a 10 million dollar grant from the National Science Foundation to explore tools and processes to completely eliminate the software bug. The basic idea is that all bugs originate from differences between the software specification, and how the code is written. The researchers hope to do all this through a tool called DeepSpec.

The premise of this research is a scam, anyone with a few years of experience in software could tell you that. That is a pretty hard statement, let me explain.

I haven’t seen a project that used a detailed, heavy-weight specification in a few years. Those are still around of course, you can find them in government projects and medical devices. But for the most part, those specifications have given way to lighter methods like the agile user story (or a conversation) that help us get to the point and deliver software faster. Over the last decade of change, mostly through experimentation with agile, we’ve learned about the failings of both of these methods.

deepspec-logo-300dpi

DeepSpec is an attempt to remove bugs by formalizing the specification in the academic sense like proofs you did while taking high school geometry. Based on what I know right now, this feels like a combination two things that are already very popular —  test driven development (unit tests) where the developer writes a ‘test’ and then builds the production code that will satisfy that test, and business facing tests where the product manager specifies behavior like valid date ranges and business workflows and then the developer writes the code to test those scenarios. Even then though, there is still the possibility of a bug creeping in. That’s just how software works. These tests completely depend the programmer and testers ability to anticipate where they might live.

Scope And Definition

Any sort of planning or specification falls apart the minute a person first encounters software. The testing community I associate with defined the term bug in a social way that involves people and their relationships to computers. A bug is anything that threatens the value of the product. That could come from the specification as the DeepSpec team suggest.

But what happens if the specification is wrong, what happens if a new tool needs to integrate, or if a workflow is misunderstood or a new one introduced?  Every bit of DeepSpecs claims rest on limiting the definition of what a bug is enough that anything outside the spec is not labeled a bug. Princeton computer scientist Andrew Appel sums it up well with this quote.

Even if the engineer who builds one component writes in English ‘this is what it does and here is how to use it,’ the engineer of another component might interpret the English-language description the wrong way, and their two components won’t work properly together.

And, Edsger Dijkstra knew all of this a long time ago.

Program testing can be used to show the presence of bugs, but never to show their absence!

Attending a routine bug triage at any large software company will show the complexity of how we talk about software bugs. More than once, I have had things I would describe as bugs — parts of functionality that just wouldn’t work for a user, exceptions from special characters in a name field, value too high in a drug dose field — changed to feature, no fix, or very low importance in the back log. When is a bug not really a bug? When the product manager and dev manager say so, that’s when.

The Princeton team are smart people, hopefully they’ll figure it out soon. Meanwhile, the rest of us will go on working.

6  Comments on this Post

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Kevin Beaver
    Thanks for this write-up, Justin. Yet another example of genius academicians using tax dollars as leverage to further their own job security. Like many politicians, it's their only leverage for survival in the market. Perhaps next they can figure out how to design the flawless human being.
    27,480 pointsBadges:
    report
  • Justin Rohrman
    @Kevin

    If they changed the goal to something like 'figuring out how to improve code quality' or even figuring out some ways to define code quality that could extend into the life of people that actually work in software...well I'd have written something completely different. 
    2,130 pointsBadges:
    report
  • 625weps
    Justin:

    Thank you the discussion.  The security of software is no better than the software engineers who wrote it.  While Deepspec is a possible scam by Princeton of NSF, the bigger issue start with adequately defining the functionality of the software for the developers.  The functionality must include security measures that balance the functionality requirements and risks generated by achieving the requisite functionality.
    20 pointsBadges:
    report
  • Kevin Beaver
    Agreed, Justin.
    27,480 pointsBadges:
    report
  • Justin Rohrman
    @625weps

    One lesson I have learned is that the amount of detail in a specification isn't really related to how well we understand what the customer wants. Language is hard, and English in particular is a riddled with ambiguity. We can surely make an effort to understand each other better, but I think there is more value in figuring out how to iterate faster to recoup our mistakes than to create the 'perfect specification'. 
    2,130 pointsBadges:
    report
  • 625weps
    Agreed. The perfect specification does not exist. 

    If the developer and user engage in an evolutionary dialogue, the closer the final functional specification meets the goals of both user functionality and managed risk.

    I recognize the statement above is somewhat idealist. It takes breaking communications boundaries have existed for 40+ years. One can only push the two parties to meet and communicate.

    20 pointsBadges:
    report

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: