How the Royal Canadian Air Force is Reducing Requirements Review Time by Over 50% with QVscribe
Customer
Royal Canadian Air Force Directorate of Air Programmes (D Air Prog) provides quality assurance of RCAF Statements of Operational Requirements.
Challenge
The RCAF needed to reduce the man-hours they were spending on requirements review. They also wanted to improve requirement quality to de-risk RCAF procurement.
Solution
D Air Prog conducted a trial evaluation of QVscribe, QRA Corp’s tool for analysing and improving natural language requirements. Thanks to that trial, D Air Prog modified their Statement of Requirements processes and specification template to align with the best practices QVscribe enforces. Results Thanks to QVscribe, the D Air Prog has streamlined Statement of Requirements workflow, improved requirements clarity, testability and traceability, and reduced specification review and update time by 50 to 75 percent.
Introduction
The Directorate of Air Programmes at the Royal Canadian Air Force (D Air Prog) had realized that ambiguous and vague requirements represent a significant risk to the Department of National Defence’s (DND) procurement process.
They needed to reduce the large number of manhours they were spending on review of their natural language requirements documents for clarity, consistency, and accuracy. And they wanted to lessen the risk that a vendor might contest a contract award over an unclear requirement.
D Air Prog was presented with an opportunity to test QVscribe – a solution that helped them transform their Statement of Requirements (SOR) processes, decrease their specification review time by 50 to 75 percent, and raise the quality of their requirements and specifications to a level that gives them a high degree of confidence.
A Requirements Development Bottleneck
The D Air Prog conducts quality assurance of engineering SORs for the RCAF. The D Air Prog reports to the Director General of Air Force Development and reviews SORs developed by several RCAF organizations.
Until recently, their process consisted of laborious, line-by-line reviews of every submitted SOR – most of which number hundreds of pages in length – combing them for anything that might be incorrect or misleading: such as ambiguities, lack of clarity, lack of testability, and inconsistencies in use of terms or units of measurement. RCAF reviewers had nothing cueing them to problems or indicating what those problems might be. Review of SORs was a tedious, error-prone operation, and a bottleneck in the RCAF’s vice D Air Prog’s requirements development process.
Procurement risk was also a major concern. D Air Prog and their Director General wanted to improve requirement quality and thus reduce the possibility of misinterpretation due to ambiguity.
A New Tool for Improving Natural Language Requirements
A possible solution presented itself in late 2017. The Defence Research and Development Canada (DRDC) – a division within the Canadian DND which looks for cutting-edge solutions that might help DND organizations – approached D Air Prog about evaluating a new requirements analysis tool for the DND. The tool, called QVscribe, promised to address the issues D Air Prog was concerned about.
QVscribe, from QRA Corp, was designed to analyse and help improve requirements written in natural language. It performs analysis and highlights the problems directly in the document and within the application in which the requirements are written. There is no need to export or reformat requirements for analysis in a separate application. QVscribe helps streamline RE workflow by presenting its analysis results through simple, intuitive scoring of individual requirement quality and overall document quality. Along with evaluating requirements against accepted RE best practices, QVscribe also reports on consistency in use of terminology and units of measurement, and similarity between requirements.
Evaluating QVscribe
D Air Prog and QRA Corp agreed to an evaluation process which would be conducted over a period of about four months. This extended trial was designed to allow D Air Prog to perform thorough due diligence prior to purchase while also providing QRA with extensive feedback that might be used to improve QVscribe. The evaluation was divided into two stages, with the D Air Prog team briefing QRA at the end of each stage.
In the first stage (Stage 1), D Air Prog worked with the tool on their own, using it to analyse several of their own SORs. They evaluated QVscribe for three key desired qualities:
- Speed of analysis
- Clarity of reporting
- Ease of error correction
When QRA received the feedback from the evaluation, D Air Prog gave QVscribe full marks (100%) in all three categories. D Air Prog stated that RCAF users had found QVscribe’s user interface uncomplicated, intuitive and easy to learn.
D Air Prog also briefed QRA on issues they encountered, which provided QRA with valuable insights on user situations QVscribe might encounter. While QVscribe for Word requires the source specifications it analyses to be in “.docx” (Office 2003 and later) format, for example, it was found that many RCAF SORs were in “.doc” (pre-Office 2003) format, as RCAF requirements authors often use previously approved SORs as baseline templates for new specifications.
Also, the SORs lacked unique requirement identifier tags and often contained multiple requirements within the same paragraph. Sections rarely made use of Word header styles in section headers. There were no clear indications of where one requirement ended, and another began. This made it impossible for D Air Prog to take advantage of QVscribe’s Requirements Finder feature during Stage 1.
“Air Programmes recognized these issues were caused by the formatting of their own documents,” said Jordan Kyriakidis. “Still, we listened carefully to their feedback, and we modified the software so QVscribe will provide better assistance to users who may face similar circumstances in the future.” Meanwhile, as QRA was updating QVscribe, D Air Prog was applying what they had learned from QRA and their Stage 1 evaluation. Two weeks later, when QRA returned with their updates, they were very happy with what they found.
The RCAF Transforms its Statement of Requirements Process
When QRA brought the revised software to Ottawa for installation, they discovered the RCAF was already using QVscribe in the development of new SORs. What’s more, they were modifying their Statement of Requirements processes to align with QVscribe’s best practice expectations and better leverage the tool’s power.
Through their use of QVscribe in their Stage 1 evaluation, the D Air Prog had recognized their SOR processes were less than optimal. The language they had been using to express requirements did not follow a solid standard across all authoring teams. Rules had not been clearly defined or strictly enforced. Specification authors had been writing to make paragraphs flow, rather than crafting discrete, concise requirements that are clear, unambiguous and testable.
Now, having observed and applied the best practices QVscribe enforces through its analysis, the RCAF could see a clear path to process improvement. To that end they had:
- Added unique alphanumeric identifiers for each requirement
- Isolated individual requirements in separate paragraphs
- Separated requirements from explanatory context • Imposed unified rules across groups on the use of imperatives and other language
They were also updating legacy documents before using them as templates, converting each to .docx format and converting legacy section headers to Word heading styles. These changes helped the D Air Prog fully unlock the benefits QVscribe offers, including streamlined workflow, improved requirements quality, and significant savings of time and effort in specification development and review.
D Air Prog worked with the updated versions of QVscribe for Word and QVscribe for Excel for another few weeks. When they presented their final, Stage 2 evaluation to QRA, they again gave QVscribe full marks in all evaluation categories. And they were extremely pleased with the way QVscribe supports their new RE process and with the new efficiencies they’ve gained.
QVscribe Automates and Expedites RCAF Requirements Engineering
QVscribe streamlined the RCAF’s SOR workflow and speeded document turnaround by providing near-instant analysis and clear, intuitive reporting of all requirement weaknesses – nonconformities with best practices – found within each SOR analysed.
Once the RCAF upgraded its SOR template, QVscribe’s Requirement Finder easily found all requirements within each SOR in just seconds, eliminating the need to manually identify individual requirements for the tool. Plus, QVscribe fully analyses most specifications in a matter of seconds. SOR turnaround was greatly accelerated.
The RCAF found QVscribe’s color-coded Quality Analysis Scorecard and other reporting features very simple and easy to understand. These reporting features proved very useful in prioritizing review time. It was easy to see which areas of the specification required attention and which could safely be left alone.
QVscribe’s Analysis Report document proved very valuable to RCAF Directors and their Director General, as well. The Analysis Report gives an objective, at-a-glance evaluation of overall document quality, which has simplified and accelerated their final review and sign-off process.
Requirements Quality Improved Immediately
One thing that surprised the D Air Prog was that QVscribe improved their requirement syntax right from the start. With QVscribe, team members were able to quickly turn long paragraphs of prose into lists of clear, concise requirement statements. This has also improved requirement traceability, as SOR requirements are now discrete and easy to number.
As an example, the D Air Prog vice cited the case of a 700-page aircrew training specification. When this document was initially analysed, QVscribe found only 35% of its requirements were of acceptable quality. But in less than two weeks, after just a few iterations of correction and rechecking with QVscribe, everyone involved in the project was very pleased with the specification.
QVscribe quickly changed the mindset of the RCAF teams who write requirements. They’re no longer concerned with the paragraph flow or making SORs read like a book. They’re focused on delivering complete sets of low ambiguity, testable requirements.
So are their management. The D Air Prog now runs all new or modified SORs through QVscribe for analysis. Every SOR submitted for signature must be accompanied by a QVscribe report. And the Director General of Force Development has mandated that no SOR be submitted for his signature until 75% to 85% of the requirements in the document have received a score of 5, QVscribe’s highest rating.
“QVscribe flagged some SOR requirements because of military terminology the tool did not recognize, but the RCAF determined it was safe to ignore such instances,” said Jordan Kyriakidis. Hence the sub-100% quality expectation. “QVscribe can be configured to account for program-specific terms, but the RCAF had not yet agreed upon a list of such terms at the time they concluded their evaluation. Thanks to RCAF feedback, however, QVscribe is now being updated to provide additional user-configurable terminology lists, making it even easier for users to customize QVscribe’s analyses to their specific environment.”
The improved requirement quality the D Air Prog has achieved with QVscribe has also helped de-risk their procurement process. QVscribe cues users to requirements which might be misinterpreted and provides guidance on how those requirements could be improved. This helps users focus their time on requirements that truly need work. Thanks to the high scores for requirements quality the D Air Prog now achieves with QVscribe, they feel much more confident their SORs will lead to successful procurement.
Significant Savings in Time and Effort
The D Air Prog cited time savings – compared to manually reading and analysing large documents line by line – as the biggest advantage QVscribe gave them. D Air Prog now gets involved much later in the development process – when specifications are at a much higher level of quality. The project team sends an SOR to D Air Prog to run through QVscribe, and D Air Prog sends back the report. The project team then corrects their document and the process repeats until the document achieves a sufficiently high score. Only then does D Air Prog make a final review, cued by the QVscribe report, before the specification is submitted for signature.
Another way QVscribe is saving the RCAF time is by allowing them to automate the process of creating compliance matrices found at the end of each SOR. Previously, these were created by manually copying requirement from the main document and pasting them into the compliance table. Not only was the process labour-intensive, but keeping these two sets of requirements identical was difficult. Table entries did not always match precisely the text in the body of the specification.
With QVscribe’s Requirement Similarity analysis, however, D Air Prog can discover and correct these discrepancies quite easily. And now, using QVscribe’s Requirements Finder and Requirements Export features, the D Air Prog is creating their compliance matrices with just a few mouse clicks. Requirements Finder finds the requirements and Requirements Export builds an Excel spreadsheet of them, which can then be embedded in the SOR as a table.
The Director of Air Programmes noted QVscribe also accelerates that final review process, because reviewers are now reading for big picture issues rather than syntax problems. Reviewers can review documents much more quickly when they don’t have to look for the subtle issues QVscribe finds automatically.
Finally, QVscribe saves time for the Director General and directors, because they only need to read the reports QVscribe generates. They see a 75% to 85% quality score, they confirm the remaining flags are non-issues, and they’re ready to sign off the document.
“Overall, Air Programmes estimates QVscribe has reduced the time for review and update of RCAF requirements documents by 50 to 75 percent,” said Jordan Kyriakidis.
The RCAF Directorate of Air Programmes Fully Integrates QVscribe
Following their successful evaluation, D Air Prog will train all incoming team members in QVscribe as part of their entry orientation. QRA has taken the next step to install QVscribe for Teams on the Defence Wide Area Network, so all personnel can use it at their own workstations. QVscribe for Teams provides centralized configuration, reporting, and user management to better fit in larger organizations like the RCAF. RCAF requirements authors will then have immediate access to QVscribe as they write and edit requirements in Word and Excel. Plus, they’ll be sure they have the latest updates to terminology lists and other RCAF-specific configuration details, thanks to QVscribe for Teams’ unified configuration management.
