Software Testing and Risk Management
(Master Thesis)
Tagalaki Vasiliki
M.Sc. In Information Systems
Department of Informatics
Athens University of Economics And Business
76 Patission Ave, Athens, GR 10434
E-mail: vasotag@aueb.gr
February, 2000
Executive Summary
Software
failure is the nightmare of the Information Age. Vast software engineering resources are
ubiquitously spent throughout all industries to avert this nightmare. However, in the end
it is impossible to guarantee that software is perfect, nor is it possible to predict and
eliminate every possible gliche that could attack software from the outside world as it
executes.
Although, much
of the failure could be avoided by dealing with various risk factors than waiting for
problems to occur and then trying to react. In other words, an adequate and systematic
risk management could provide insight to potential problem areas and identify, address and
eliminate them before they derail the project.
Risk management
has long been recognized as an important part of project management in all the traditional
engineering disciplines. Software risk management has received increasing attention over
the last decade, as new opportunities have opened and worldwide competition for software
business has become keener.
Most of the
approaches that have been developed suggest that risk management is a process that must be
applied throughout all the stages of the software development life cycle, in order to
track, manage and control effectively the several risks that threaten the software.
Even though the
afore mentioned view has turned out to be true, it is important to highlight the fact
that, while in all the other phases of the software life cycle there is a great chance of
risk elimination, in the testing phase there is always some risk remaining after the end
of the phase.
On top of that,
software testing, whose principal objective is to gain confidence in the software, is one
of the most important activities in the development process, where a great amount of the
available resources is spent. Therefore, it is essential for the success of the project;
to take into account the risks involved, in order to make effective testing decisions.
Having in mind
the afore mentioned observations, the present work focuses on the software testing phase
and inquire into its relationship with the risk management process. It also introduces a
number of factors that affect the total risk remaining after the completion of testing,
and it proposes an overall risk management process in relation with the testing that
taking place.
According to
David Cluch, a risk is a combination of an abnormal event or failure and the consequences
of that event or failure to a system’s operators, users, or environment.
Like any other
engineering project, software development is a risky business. However, the risks involved
in software engineering is much more complicated due to mainly three broad reasons: (i)
software is an extremely complex end creative artifact and software development could not
be rigorously defined as engineering process yet, (ii) it is much more human-oriented, and
(iii) sound risk analysis and reliability testing methods are in lacking.
If
we accept that risk includes technical and managerial factors, which threaten
the success of the project, risk management is the process of identifying these
threats, analyzing them, quantifying their effects, and implementing plans that
counteract their negative effects.
In
other words, risk management asks, “What exactly could go wrong, and what do
we do when it does?” The first sub question looks for anything that could fail
and trigger a string of failures; the second one produces alternative courses
of action if a problem occur.
The
practice of risk management involves the following four main steps:
1)
Risk Analysis which consisting of:
2)
Risk Management Planning
3)
Risk Resolution
4)
Risk Monitoring
Software
testing is often associated with two basic terms: verification, which
refers to ensuring correctness from phase to phase of the software development
cycle, and validation, which involves checking the software against the
requirements.
Even
though testing can prove the presence of vulnerabilities in software by finding
faults in it, it is not able to guarantee its perfection by demonstrating that
there are no faults in it. Moreover, during the testing phase, as the errors
are identified, they may result in changes to the code, which can lead to additional
problems and ripple effect errors, especially in code that is high risk due
to size and complexity.
If
we consider that the main objective for a testing team is to do the most cost
effective testing which can ensure that the product is reliable enough, safe
enough and satisfies the user/customer requirements, we establish the fact that
something like that is extremely difficult, if not impossible, to accomplish,
since there is never enough time to test everything completely. What is required,
is understanding of the risks that relate with software faults, risks for the
user or the customer, the developer or the supplier, or even the responsible
for the maintenance of the software. In other words, testing is nothing more
than risk management, whose aim is to achieve confidence in the software.
After
a close examination, we found that the risk remaining after the end of the testing
phase is determined by the following set of factors:
1)
Software
Criticality (CRIT)
Software
criticality can be defined as the function of the losses caused by unsatisfactory
outcomes {CRIT = f(L(UO) (1)} who can range from catastrophic (loss of life
or permanent disability) to negligible (no damage, no injury). Criticality characterises
the software and categorises it to safety-critical, semi-critical and non-critical.
2)
Software
Size (SIZE)
Three
sizing quantities are used, so as to comprise both conventional and object-oriented
software: (i) Object Points, including screens, reports and 3GL modules, (ii) Function Points
which measure a software project by quantifying the information processing functionality
associated with major external data or control input, output, or file types,
and (iii) Source Lines of Code.
3)
Testing
Method (TESM)
There
are two prominent strategy dimensions in testing techniques: function (black-box)/structured
(white-box) and static/dynamic. Depending on the testing method used, different levels of coverage are
achieved and consequently different level of risk remains after the completion
of the phase.
4)
Experience
of the Testing Team (TEXP)
The
effectiveness of software testing depends on the capability and the experience
of the testing team; the more experienced the team is, particularly in similar
with the one under development projects, the higher the probability of uncovering
faults in the software. One way to categorize this factor is on the basis of
time, even though it is not easy to set strict borders between the categories.
5)
Software
Complexity (CPLX)
The
complexity of the software, like its size, is something that affects the testing
process and its categorization is based on the Constructive Cost Model (COCOMO),
which was developed for software cost estimation.
6)
Frequency
of Use (FREQ)
Frequency
of use is a factor that must not be ignored, as the more a software application
is used, the higher the probability of an unsatisfactory outcome to occur. Frequency
of use can vary even between the various areas of the same software, as there
may be, for example, some basic blocks that are being executed more often than
others.
7)
Time
for Testing (TIME)
For
every software project there are certain time limitations and deadlines in product
delivery. Sometimes this fact is one of the reasons for a non-effective testing.
But time shouldn’t be regarded as a critical factor by itself; for instance,
there is a great chance to have enough time for testing but choose an inappropriate
method.
8)
Available
Resources (RESO)
Except
the time limitations, in every case, there are certain resources available for
the software development, meaning basically the financial resources and the
budget that must not be exceeded.
The
last seven factors affect the probability of an unsatisfactory outcome to occur,
so we have the following relation:
P(UO)
= f(SIZE,TEXP,TESM,CPLX,FREQ,TIME,RESO)
(2)
Where:
P(UO) is the probability of an unsatisfactory outcome
If
we define risk as the product of the probability of an unsatisfactory outcome
and the loss experienced if the outcome occurs (which from relation (1) is criticality)
we have:
R =
P(UO) x CRIT
(3)
In
order to make various testing decisions; a policy must be determined from the
Development Company about the tolerable risk level of the software. This determination
will be based on the available time and resources, and of course on the criticality
of the project.
After
making the appropriate decisions, testing is being executed, and after that
an assessment of the risk level remaining is taking place, based on the abone
mentioned factors.
If
the risk level satisfies the requirements of the company, other quality measures
are used in order to decide whether the software system is ready for delivery
or not. If the risk level exceeds the tolerable one, new decisions must be made
in order to do more testing. Of course, that presupposes that there is time
and resources left for more testing, because if that is not the case, the company
must request them from the customer and then decide the next step.
Risk Management is a critical part of any software-intensive project, and is
particularly critical in large projects, projects where there is high uncertainty,
or projects which are on the leading age of technology.
Our
claim is that although risk must be recognized and managed throughout all the
stages of the software development life cycle, it is necessary to pay great
attention to the risks involved in the testing phase. And that is because during
the testing phase, we are not able to eliminate the risks but only minimize
them.
Attempting
to determine the total risk remainimg after the end of testing, we found a set
of factors that affect the probability of an unwanted outcome and together with
the criticality of the software, which represents the loss experienced if the
outcome occurs, we managed to define this risk.
Of
course, the set of factors mentioned is not closed; there may be more factors
involved, which we did not manage to find.
The
present work, showing that there is always some risk remaining in the software
after the testing phase, sets a solid base for further research in that field.
The remaining risk can be estimated by quantifying the factors that were introduced
and by defining a proper mathematical relation.
Furthermore,
on the basis of the present work, a model for risk management, in relation of
course with software testing can be developed.
Moreover,
a risk management tool can be developed whose use would definitely help to reduce
the uncertainty involved in any software project. A tool like that would be
extremely useful for insurance companies, whose job is to indemnify or guarantee
a software user against loss caused by a software failure (software insurability).
Finally, an interesting target would be the combination of risk with other software quality criteria, like security and reliability.