Software Testing and Risk Management
(Master Thesis)

Tagalaki Vasiliki
M.Sc. In Information Systems
Department of Informatics
Athens University of Economics And Business
76 Patission Ave, Athens, GR 10434

February, 2000

Executive Summary


Software failure is the nightmare of the Information Age. Vast software engineering resources are ubiquitously spent throughout all industries to avert this nightmare. However, in the end it is impossible to guarantee that software is perfect, nor is it possible to predict and eliminate every possible gliche that could attack software from the outside world as it executes.

Although, much of the failure could be avoided by dealing with various risk factors than waiting for problems to occur and then trying to react. In other words, an adequate and systematic risk management could provide insight to potential problem areas and identify, address and eliminate them before they derail the project.

Risk management has long been recognized as an important part of project management in all the traditional engineering disciplines. Software risk management has received increasing attention over the last decade, as new opportunities have opened and worldwide competition for software business has become keener.

Most of the approaches that have been developed suggest that risk management is a process that must be applied throughout all the stages of the software development life cycle, in order to track, manage and control effectively the several risks that threaten the software.

Even though the afore mentioned view has turned out to be true, it is important to highlight the fact that, while in all the other phases of the software life cycle there is a great chance of risk elimination, in the testing phase there is always some risk remaining after the end of the phase.

On top of that, software testing, whose principal objective is to gain confidence in the software, is one of the most important activities in the development process, where a great amount of the available resources is spent. Therefore, it is essential for the success of the project; to take into account the risks involved, in order to make effective testing decisions.

Having in mind the afore mentioned observations, the present work focuses on the software testing phase and inquire into its relationship with the risk management process. It also introduces a number of factors that affect the total risk remaining after the completion of testing, and it proposes an overall risk management process in relation with the testing that taking place.

Software Risk

According to David Cluch, a risk is a combination of an abnormal event or failure and the consequences of that event or failure to a system’s operators, users, or environment.

Like any other engineering project, software development is a risky business. However, the risks involved in software engineering is much more complicated due to mainly three broad reasons: (i) software is an extremely complex end creative artifact and software development could not be rigorously defined as engineering process yet, (ii) it is much more human-oriented, and (iii) sound risk analysis and reliability testing methods are in lacking.

Software Risk Management

If we accept that risk includes technical and managerial factors, which threaten the success of the project, risk management is the process of identifying these threats, analyzing them, quantifying their effects, and implementing plans that counteract their negative effects.

In other words, risk management asks, “What exactly could go wrong, and what do we do when it does?” The first sub question looks for anything that could fail and trigger a string of failures; the second one produces alternative courses of action if a problem occur.

The practice of risk management involves the following four main steps:

1)      Risk Analysis which consisting of:

2)      Risk Management Planning

3)      Risk Resolution

4)      Risk Monitoring

Correlation between Risk Management and Software Testing

Software testing is often associated with two basic terms: verification, which refers to ensuring correctness from phase to phase of the software development cycle, and validation, which involves checking the software against the requirements.

Even though testing can prove the presence of vulnerabilities in software by finding faults in it, it is not able to guarantee its perfection by demonstrating that there are no faults in it. Moreover, during the testing phase, as the errors are identified, they may result in changes to the code, which can lead to additional problems and ripple effect errors, especially in code that is high risk due to size and complexity.

If we consider that the main objective for a testing team is to do the most cost effective testing which can ensure that the product is reliable enough, safe enough and satisfies the user/customer requirements, we establish the fact that something like that is extremely difficult, if not impossible, to accomplish, since there is never enough time to test everything completely. What is required, is understanding of the risks that relate with software faults, risks for the user or the customer, the developer or the supplier, or even the responsible for the maintenance of the software. In other words, testing is nothing more than risk management, whose aim is to achieve confidence in the software.

Software Risk Factors

After a close examination, we found that the risk remaining after the end of the testing phase is determined by the following set of factors:

1)      Software Criticality (CRIT) 

Software criticality can be defined as the function of the losses caused by unsatisfactory outcomes {CRIT = f(L(UO) (1)} who can range from catastrophic (loss of life or permanent disability) to negligible (no damage, no injury). Criticality characterises the software and categorises it to safety-critical, semi-critical and non-critical.

2)      Software Size (SIZE)

Three sizing quantities are used, so as to comprise both conventional and object-oriented software: (i) Object Points, including screens, reports and 3GL modules,    (ii) Function Points which measure a software project by quantifying the information processing functionality associated with major external data or control input, output, or file types, and (iii) Source Lines of Code.

3)      Testing Method (TESM)

There are two prominent strategy dimensions in testing techniques: function (black-box)/structured (white-box) and static/dynamic. Depending on the testing method  used, different levels of coverage are achieved and consequently different level of risk remains after the completion of the phase.

4)      Experience of the Testing Team (TEXP)

The effectiveness of software testing depends on the capability and the experience of the testing team; the more experienced the team is, particularly in similar with the one under development projects, the higher the probability of uncovering faults in the software. One way to categorize this factor is on the basis of time, even though it is not easy to set strict borders between the categories.

5)      Software Complexity (CPLX)

The complexity of the software, like its size, is something that affects the testing process and its categorization is based on the Constructive Cost Model (COCOMO), which was developed for software cost estimation.

6)      Frequency of Use (FREQ)

Frequency of use is a factor that must not be ignored, as the more a software application is used, the higher the probability of an unsatisfactory outcome to occur. Frequency of use can vary even between the various areas of the same software, as there may be, for example, some basic blocks that are being executed more often than others.

7)      Time for Testing (TIME)

For every software project there are certain time limitations and deadlines in product delivery. Sometimes this fact is one of the reasons for a non-effective testing. But time shouldn’t be regarded as a critical factor by itself; for instance, there is a great chance to have enough time for testing but choose an inappropriate method.

8)      Available Resources (RESO)

Except the time limitations, in every case, there are certain resources available for the software development, meaning basically the financial resources and the budget that must not be exceeded.


The last seven factors affect the probability of an unsatisfactory outcome to occur, so we have the following relation:

P(UO) = f(SIZE,TEXP,TESM,CPLX,FREQ,TIME,RESO)                                          (2)

Where: P(UO) is the probability of an unsatisfactory outcome    

If we define risk as the product of the probability of an unsatisfactory outcome and the loss experienced if the outcome occurs (which from relation (1) is criticality) we have:

R = P(UO) x CRIT                                                                                                      (3)

Risk Management Procedure in relation with Software Testing

In order to make various testing decisions; a policy must be determined from the Development Company about the tolerable risk level of the software. This determination will be based on the available time and resources, and of course on the criticality of the project.

After making the appropriate decisions, testing is being executed, and after that an assessment of the risk level remaining is taking place, based on the abone mentioned factors.

If the risk level satisfies the requirements of the company, other quality measures are used in order to decide whether the software system is ready for delivery or not. If the risk level exceeds the tolerable one, new decisions must be made in order to do more testing. Of course, that presupposes that there is time and resources left for more testing, because if that is not the case, the company must request them from the customer and then decide the next step.


Risk Management is a critical part of any software-intensive project, and is particularly critical in large projects, projects where there is high uncertainty, or projects which are on the leading age of technology.

Our claim is that although risk must be recognized and managed throughout all the stages of the software development life cycle, it is necessary to pay great attention to the risks involved in the testing phase. And that is because during the testing phase, we are not able to eliminate the risks but only minimize them.

Attempting to determine the total risk remainimg after the end of testing, we found a set of factors that affect the probability of an unwanted outcome and together with the criticality of the software, which represents the loss experienced if the outcome occurs, we managed to define this risk.

Of course, the set of factors mentioned is not closed; there may be more factors involved, which we did not manage to find.

Future Work

The present work, showing that there is always some risk remaining in the software after the testing phase, sets a solid base for further research in that field. The remaining risk can be estimated by quantifying the factors that were introduced and by defining a proper mathematical relation.

Furthermore, on the basis of the present work, a model for risk management, in relation of course with software testing can be developed.

Moreover, a risk management tool can be developed whose use would definitely help to reduce the uncertainty involved in any software project. A tool like that would be extremely useful for insurance companies, whose job is to indemnify or guarantee a software user against loss caused by a software failure (software insurability).

Finally, an interesting target would be the combination of risk with other software quality criteria, like security and reliability.

Bibliography - References