7 Crucial Reports for Quality Assurance

Quality Assurance is not simply measuring the quality of an application.  It’s also measuring the quality of your development.  More times than not, development teams become so focused on “fixing bugs” that we “miss the forest for the trees” – we don’t stop to assess the quality of the development process.  What good are our bug fixes if we simply introduce new bugs into the system?  How often are we introducing new bugs?  Is there a feature set or an area of the application that seems to have more bugs than other areas?  Is a developer more prone to creating bugs?  With all of the tools that we have at our disposal, it amazes me how little quality assurance tracks the actual development quality.  Instead, we merely have a backlog of bugs which we keep increasing with each new iteration.

Measuring the Quality of a Product

If I was ask you what you would consider as a successful product, some of your answers maybe:

• Completed under/around deadline or budget.
• Very few bugs.
• Good amount of features incorporated.
• Unit tests passing.

In reality, a “successful” product maybe nothing more than our Definition of Done.  On the other hand, if I was to ask you what you would classify as a failed product, this list is typically longer (much longer).  Taking this one step further, answers to the above two questions also differ based on your role (stakeholder, developer, client, etc.).

You see, measuring the success of the product – measuring good quality – is often very subjective.  Everyone has their own idea of “success” or “good quality.”  But, it’s very easy to determine poor quality and, typically, everyone has the same answers.  And, most of the time, we talk about the quality of a product based on its issues.  Since this is the case – many of us judge quality from a perspective of current issues – we can create reports to provide insight and help us examine the quality of our development process.  As we improve our development process, the quality of our product also increases.

“The products, systems, and services you build will mirror your organizational structure.” ~ Conway’s Law

Decreasing complexity in organizational structure increases the value of software.  Improving our organizations and development processes will, in turn, make our software more valuable and stable.  Inversely, if our organizations and development processes are a train wreck, then so shall be the software we deliver.

“You don’t have to have good software.  Just good marketing.” ~ Unknown

Don’t let this be the mantra of your organization.

A Baseline – The 5 Why’s

The 5 Why’s is an iterative process examining cause and effect in order to determine the root cause of a problem.  A root cause of a problem is  determined by continuing to ask Why? to each answer of the previous question.  Typically, it takes at least five iterations of asking Why? for the root cause to be determined.  Examine the scenario below:

The Problem: The vehicle is out of gas.
Exploration:

1. Why? –  I didn’t pump any gas.
2. Why? – I was running late this morning.
3. Why? – I overslept.
4. Why? – I stayed up too late working last night.
5. Why? – I had too many tasks to work on last night for this week’s deployment.

Obviously, we could drill down into this much further.  But, by continuing to ask Why?, we, again, can attempt to uncover the root cause of the problem and make adjustments to, possibly, prevent the situation from happening again.  In this case, we could maybe reduce the number of tasks or add additional resources to complete outstanding tasks.

Empirical Process

Think back to your days of middle school, high school and college chemistry class.  You were taught to ask questions, make observations and create hypotheses.  This was the empirical process.  It goes a little something like this:

1. Ask a question
2. Analyze
3. Make a hypothesis
4. Test
5. Analyze
6. Gather input from others
7. Repeat, if necessary

If you were like me, you were required to memorize this process and write down your notes (while, of course, wearing your big goggles – safety first!) on graph paper.

When it comes to development, we are so unorganized.  Much of the time, quality assurance analysts are following simple scripts to regression test applications in order to find bugs.  When a bug is found, they log it and move on.  In this, QA’s are only performing the first step of the empirical process – asking the question, “Does the application perform as expected?”  Occasionally, a good QA will attempt to perform step 2.  A QA that has some development experience could possibly even get to step 3 and make suggestions to developers on how to fix the problem.  But, the development team never follows through the entire process.

If we were to follow the entire empirical process, then reporting could assist us in proactively reducing the amount of bugs introduced into the system by revealing repetitive problem areas and factors that are attributing to ongoing issues.

“Good development is scientific.  Great development is an art.”

If we don’t consider our development great enough to be an art form, we can at least follow the scientific process in order to produce good quality.  Speaking as a developer, developers, on average, are very undisciplined.  We eat too much pizza, drink a lot of Mt. Dew’s, never shave and stay up all hours of the night.  We need accountability.

Reporting directs quality assurance and creates accountability for development.

Conversation Starters

Following the empirical process and asking the right questions allows the development team to have dialogue.  The following reports are not an “end-all-be-all.”  They are merely tools to help us ask questions and examine our development efforts in order to find opportunities for improvement.

Remember, better organization produces better software.

Sample Reports

1. Ishikawa (“fishbone”) Diagram
2. Check Sheet
3. Stratification (alternatively, flowchart or run chart)
4. Control Chart
5. Histogram
6. Pareto Chart
7. Scatter Diagram