Customer

Ultra Electronics Maritime Systems, headquartered in Dartmouth, Canada, is a leading supplier of naval defence systems, including sonar systems, sonobuoys, and communication systems.

Challenge

Ultra Electronics Maritime Systems wanted to:

• Standardize requirements authoring practices across large, multidisciplinary teams
• Reduce manhours spent correcting vague or ambiguous requirements
• Mitigate potential risks in customer requirements earlier in the bidding process

Solution

QVscribe from QRA Corp, a software tool for analysis and improvement of requirements written in natural language. Results and Benefits With QVscribe, Ultra Electronics Maritime Systems: • Streamlined requirements engineering workflow • Reduced time and cost to correct requirements errors by 75% • Unified application of RE best practices across large teams • Bolstered their RE quality assurance procedures

Introduction

To reduce engineering risk, Ultra Electronics Maritime Systems (Ultra) is always looking for ways to identify and eliminate requirements errors – and to do so as early as possible.

Requirements definition is one of the most critical phases of engineering development. This is especially true for complex, cyber-physical systems like aircraft, automobiles and military defence systems, where requirements typically number in the thousands.

This earliest phase of development is critical for two reasons. First, this is where, according to the famed computer scientist James Martin, more than half of all engineering errors are introducedⅰ. Second, it’s urgent to correct requirements errors in this phase. Numerous studies (Jonetteⅰⅰ, Boehmⅰⅰⅰ, Rothmanⅳ, McGibbonⅴ, Chigitalⅵ) have shown that specification and design errors become exponentially more expensive to fix the longer they go undetected through subsequent phases of development, production, and operation. In requirements engineering (RE), continuous improvement is a must.

Ultra Electronics Maritime Systems recently adopted QVscribe, a tool designed to automate the analysis and improvement of requirements written in natural (spoken) language. Using QVscribe, they learned how they could streamline the RE workflow, automate detection of requirements errors, accelerate error correction, and reduce the costs and risks associated with requirements definition.

ⅰ Martin, James, An Information Systems Manifesto, Prentice Hall, January 1984.

ⅰⅰ Jonette M.,et al, Error Cost Escalation Through the Project Life Cycle, NASA Johnson Space Center and INCOSE, June 2004.

ⅰⅰⅰ Boehm, B. W., Software Engineering Economics, Prentice-Hall, 1981.

ⅳ Rothman, J., What does it cost to fix a defect?, Stickyminds.com, August 2002.

ⅴ McGibbon, T, Return on Investment from Software Process Improvement, The DoD Software Tech News, Volume 5, Number 4, November 2002.

ⅵ Cigital, Case study: Finding defects earlier yields enormous savings, Cigital, 2003

Clear Requirements are Critical for Ultra Electronics Maritime Systems

Ultra Electronics Maritime Systems is a leader in the design, development and manufacture of both passive and active sonar systems, sonobuoys and through-the-earth communication systems. For over 70 years, they have supplied Canada and its allies with advanced, robust technologies, systems, and support services for undersea surveillance and anti-submarine warfare.

As a supplier to government defence departments, Ultra Electronics Maritime Systems receives frequent requests for proposal (RFP) from its customers. The specifications in these RFPs typically run hundreds of pages and contain thousands of requirements. It is essential that Ultra engineers fully understand those customer requirements. Only when they do can they properly constrain and negotiate the scope of their task and price it accurately. Only then can they accurately derive the more detailed requirements that will define their systems for their developers and suppliers. It is also important they write those derived requirements clearly, so they can assure correct functionality and on-time delivery.

Ultra has a mature RE process with ISO 9001 quality control procedures in place. With all they have riding on the accuracy and clarity of their requirements, they’re always looking to improve.

An Array of Requirements Engineering Challenges

One of Maritime System’s biggest challenges has been standardizing requirements writing practices across large, multidisciplinary teams. On most programs, they have many engineers writing requirements simultaneously. These engineers come from a variety of disciplines – systems, electrical, mechanical, software – and vary in experience from less than one year to more than five. Maritime Systems wanted to ensure consistent use of accepted RE best practices, as well as company standards. They also wanted to train inexperienced requirements authors faster and more efficiently, without significant added cost. They recognized that relying solely on written standards makes application of best practices laborious and problematic for inexperienced engineers.

Another area of concern was the amount of effort being spent correcting requirements in review meetings. Requirement review meetings are Maritime Systems’ primary means of RE quality control and the primary source of feedback for requirements authors.

Every requirement of every Maritime Systems project is reviewed and approved in a requirements review meeting. Four to six people and sometimes more – including the requirements author, lead systems engineer, project manager, and quality assurance manager – attend each of these meetings. They thus represent a significant cost on every program.

Finally, Maritime Systems wanted to identify and mitigate potential program risks from poorly or loosely defined customer requirements earlier in the bidding process. Doing so would allow them to correct or constrain such requirements through negotiation at the very start of the program, and price projects more accurately. Earlier and more precise correction and constraint of customer requirements would, in turn, help Maritime Systems (1) avoid unnecessary non-recurring engineering (NRE) which drives up costs, (2) minimize costly scrap and rework due to requirement inaccuracy, and (3) avoid conflict with customers by minimizing cost and schedule overruns.

A New Analysis Tool for Natural-Language Requirements

As it turned out, Ultra Electronics Maritime Systems found a solution to many of these problems right in their own hometown. They connected with QRA Corp, a fellow Nova Scotian technology developer. QRA invited Maritime Systems to try their new requirements analysis tool, QVscribe.

QVscribe has been designed to help engineers, business analysts and project managers improve requirements written in natural language. It analyses requirements against accepted RE best practices and highlights the problems it finds, directly within the application in which the requirements are written.

QVscribe helps streamline RE workflow with simple, intuitive, actionable scoring of individual requirement quality, usage consistency of terminology and units of measurement, and overall document quality. It also analyses requirements for similarity – a feature that’s useful in finding duplications and conflicts between requirements. And QVscribe can be quickly and easily customized for program-specific practices and terminology. I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Requirements Correction Time and Cost Reduced by 75%

An important consideration for Ultra Electronics Maritime Systems was how well and how easily QVscribe would integrate with their existing tools and workflow. They wanted to retain their current requirements authoring tool, Microsoft Word, and their requirements management (RM) tool, but add QVscribe’s more extensive analysis features.

Fortunately, as soon as they began their evaluation, Ultra found QVscribe integrated with their existing toolset and processes. QVscribe works directly within the Microsoft Office apps (Word and Excel) Ultra uses in creating specifications, and it exports requirements to industry standard formats. Ultra had no trouble transferring QVscribe-analysed requirements into their RM tool.

QVscribe also meshed well with Ultra’s RE process. “We have a very mature ISO 9001-compliant requirements development process based on INCOSE-recommended practices, and I can’t recall any instances where there were conflicts between QVscribe and our existing standards,” said Abigail LeFrank, a mechanical engineer who has used QVscribe extensively at Ultra.

The most important result for Ultra was a significant savings in the time and cost of correcting errors during review.

As an example, they cited a QVscribe analysis of one batch of 37 requirements written by a young engineer with less than a year’s experience. QVscribe flagged eight of those 37 requirements for vague wording that could have led to misinterpretation. Under Ultra’s existing procedure – before QVscribe – correcting those eight requirements in a review meeting could easily have consumed more than two manhours of effort (5 reviewers x 8 requirements x 3 minutes/requirement = 120 minutes), even if rapid agreement was achieved on all eight corrections.

With QVscribe, however, the relatively inexperienced author, using the best-practice guidance the tool displayed, was able to satisfactorily correct all eight of the flagged requirements in a total of less than 30 minutes: a time and cost reduction over Maritime Systems’ existing process of at least 75%.

Ultra realized that scaling such results over just a single program would result in large savings totalling hundreds of manhours and tens of thousands of dollars.

It is also important to note that QVscribe’s automated detection of potential requirements problems greatly reduces the chances that errors go undetected in the requirements definition phase. Detecting and correcting requirements errors in the definition phase – before they can propagate to later phases of development where they become increasingly costly to fix – is the greatest potential savings factor QVscribe offers.

Requirements Engineering Made Faster, Easier and More Consistent

Ultra cited several QVscribe features as being especially useful to them. Chief amongst these was QVscribe’s Quality Analysis, which accelerates identification and correction of requirements errors, but that also provided Ultra with an unexpected benefit: on-the-job training. “I was relatively new to writing requirements, and I was happy to have a tool that would not only identify deficiencies in my requirements, but also help me learn the big do’s and don’ts of requirement writing,” said Abigail LeFrank. “We have references on good requirement writing, but QVscribe is faster.”

Ultra Electronics Maritime System’s engineers also liked QVscribe’s Unit Consistency Analysis and Configuration window. The Unit Consistency Analysis helped the engineers quickly verify that all requirements were expressed using the correct units of measurement. The Configuration window allowed them to specify which measurement units should be used, and to add or exclude trigger words from the quality analysis. “In one instance, we were dealing with requirements that referred to various standards documents, so we temporarily removed the word ‘standard’ from the vague words list,” Abigail LeFrank noted. “Doing so eliminated numerous false detections and made it easier to find real problems.”

Program and quality assurance managers were highly appreciative of QVscribe’s facilities for exporting and importing configuration settings and for generating reports. “The Configuration Settings Export/Import feature lets us configure QVscribe to company and program-specific practices on one computer and then transfer those settings to all our other licensed work stations,” said Marc Charbonneau, Director of Program Management at Ultra Electronics Maritime Systems. “This is especially useful for ensuring consistency of requirements when they are being written by a large team.”

QVscribe’s Report Generation capabilities, on the other hand, provides a detailed, on-demand report of the analysis performed enabling users to share analyses with colleagues and managers, and creating an artefact that can be used for quality assurance purposes.

A Process Improvement

Not long after they had begun using QVscribe, Ultra Electronics Maritime Systems decided to add a “QVscribe Check” to their requirements engineering process.

Now, all new requirements must be analysed with QVscribe prior to being submitted for formal review. Once the author has addressed all the issues identified by QVscribe, he or she generates a QVscribe report and presents it with the requirements package at the review meeting.

“Having our requirements authors run a QVscribe analysis and address all the issues QVscribe finds – before submitting their requirements for review – is saving us a lot of time and labour cost in review meetings,” said Marc Charbonneau.

Maritime Systems Continues To Improve With QVscribe

Ultra Electronics Maritime Systems has begun installing QVscribe on their network and adjusting their RE processes to leverage it. Their plan, now, is to get all their requirements engineering team members trained in QVscribe and using it consistently. They are convinced that QVscribe will not only help new requirements engineers learn company standards and industry best practices faster. It will also significantly reduce their requirements development costs, timescales, and risks.

Speaking of risk reduction, Maritime Systems also plans to use QVscribe to help them assess and reduce risk during their project bidding process. “We fully intend on using QVscribe on RFPs to identify vague or ill-defined requirements, ensure we respond with suitable compliance matrices, and properly estimate the associated risks,” said Marc Charbonneau.

Learn how QVscribe can transform your design process.

Download a PDF copy of the Case Study.