Software Testing in Safety Critical Systems

Abstract Today, many safety-critical applications are controlled by computer software. Therefore effective testing tools are required to provide a high degree of safety and to reduce severe failures too minimum. The paper examines existing regulating standards in safety-critical systems. By comparing different software testing methods the requirements and challenges in safety-critical software testing are being evaluated. The QUICKIES standard serves as the mall regulatory framework for all separately systems and provides the basis for the creation of application- and Interdependently tankards.

Moreover it defines certain safety integrity levels depending on the field of application and recommends testing methods according to these levels. In model- based safety testing a usage model with restricted space state domain is used to generate representative test cases. Statistical testing is a mathematical approach that uses a high number of test cases to reach a significant result. The main challenge of all safety-related testing methods Is to reduce testing time and complexity without distorting the significance of the test.

These can for example be transportation systems, power plants, and medical applications. As people’s lives depend on the correct function of such control systems and their software, thorough testing is required before they can be admitted to operation. There are many different software testing methods. Most of them only analyze the probability of a failure but do not value its severity. However, in safety-critical systems a failure that has severe consequences, even if it is extremely rare, can not be accepted. Therefore testing in this field has to be adopted accordingly.

The purpose of this paper is to find and compare the latest methods for safety-critical footwear testing and to identify the most common industry standard in this field. Moreover the requirements and challenges in safety-critical software testing will be elaborated. At the beginning the paper will provide definitions that are required for the understanding of the subsequent chapters. After that, an introduction to the JUICE 508 safety standard, which serves as a basis for most industry-specific standards, is given.

The chapter “Testing Methods” will address some of the latest safety-related software testing methods in detail. 5 Definitions 2 Definitions 2. 1 Reliability and Safety In safety critical systems both, reliability and safety are required to achieve the goals of dependability. However, reliability and safety are two different attributes of dependability. The reliability, R(t) , of a system is a function of time. It is defined as the conditional probability that the system will perform its intended function in a defined way over a given time period and under certain specified and assumed conditions.

The most used parameter to characterize reliability is the Mean Time To Failure (MATT). The safety, S(t), of a system is defined as the probability that a system ill either perform its functions correctly or will discontinue its functions in a way that does not interrupt the operation of other systems or Jeopardize the safety of any people associated with the system [1]. Based on these definitions, in reliability testing all failures are weighted equally, whereas in safety testing the failures are weighted according to their severity.

Therefore, a reliable system may be quite unsafe and a safe system may be very unreliable. 2. 2 Safety-critical System States very complex to generate. As many states are unreachable or very difficult to reach hey can be reduced to a relatively small number of representative system states. These states are grouped in three subsets: Normal State Subset (NUNS), Fail-Safe State subset (FPS) and Risky state subset (IRS). Their relationships are: s=Unusualness; 6 Their inter-dependability is described as a Markova chain (see figure 1) [2]. Figure 1 : Three-state Markova Model for Safety-critical Systems(Source: 2. Markova Chain Usage Model The Markova chain usage model describes the possible usage of a software based on a predicted environment. It can be used to generate statistical test cases and to estimate the software reliability. In an Markova model the transition from operation I to operation J can be denoted by an ordered pair . Let be the transition probability from operation I to operation J, with and EX=I .. N p(is)=1, where n is the number of operations. The transitions and transition probabilities can be represented in the form of a matrix [3].

Each specific usage of the program corresponds to a path X=(XI, XX,… Xi) in the Markova chain where Xi corresponds to the I-the operation. P(Xi, X]) determines the next executed operation J after execution of operation I. Since the operations are random rabbles, each path X=(XI, XX,… ) forms a stochastic process. For a particular path x=(ox, XSL ,… ), the corresponding path execution probability is [3]: 7 pox pop , x 3 Standards There exist both national and international standards and guidelines at different depths and classifications which define requirements for safety-related technologies. Yester and provides the basis for the creation of application- and underspecified standards. It includes more than 500 pages of normative and informative specifications and proposals. Nowadays most safety-related standards are based on he JUICE 508 in combination with the previously applicable requirements [4]. The JUICE 508 defines so called Safety Integrity Levels (Sills) which serve as a measure for the safety requirements on a certain system. The following table shows the different SILLS as well as the corresponding probability of failure and application examples.

Probability of Failure One Failure in x Years Consequences Application Example < 10-8 110000 years Potential for fatalities in the community Nuclear power plant control 2 < 10_7 1 1100 years Potential for multiple on-site fatalities Hazardous area laser curtain sensor 1100 years Potential for major on-site injuries and fatalities Hazardous liquid flow meter < 10-5 110 years Potential for minor on-site injuries Thermal meter Table 1 : Safety Integrity Levels (Source: 8 Standards The ‘EC 61 508 is divided into seven parts. Parts one through four are normative and are used as a guide.

The last three parts, are informative and include practical examples which should help to simplify the application of the standard. The ‘CE 61 508 describes the complete life cycle of safety-related systems from planning to decommissioning and refers to all aspects related to the use and requirements for electrical / electronic / programmable electronic systems (E / E / PEE) for separately functions [4]. According to the focus of this paper only the parts relating to software testing are mentioned in the following paragraph. Figure 2 shows the verification and validation process in software development according to the JUICE 508 standard.

The E/E/PEE system safety requirements are applied both on the system architecture and the software specifications. Every level in the system architecture verifies if it meets the requirements of the next higher layer (I. E. Coding fulfills module design requirements, module design fulfills software yester design requirements etc. ). Moreover each system architecture layer is tested by a specific test. As soon as the test circuit is closed successfully the software can be validated. The standard also recommends and rates certain test methods according to the required SILL. In order to meet the requirements of the ‘CE standard a series.

Test methods comprised in the ‘CE 61 508 are categorized as follows [6]: Failure analysis (I. E. Cause consequence programs) Dynamic analysis and testing (I. E. Test case execution from model-based test case generation) Functional and black box testing (I. . Equivalence classes and input partition testing, including boundary value analysis) Performance testing (I. E. Response timings and memory constraints) Static analysis (I. E. Static analysis of run time error behavior) 9 Figure 2: ‘CE 61 508-3 Verification and Validation Process(Source: 10 Testing Methods 4 Testing Methods There are many different software testing methods.

A detailed introduction to all different methods would be far beyond the scope of this paper. Therefore the author will only mention two methods he deems most relevant in the field of safety-related software testing. Finally both methods are compared and their possible application areas are evaluated. 4. 1 Model-based Safety Testing In model-based testing explicit behavior models that encode the intended behavior of a system and its environment are used. These models generate pairs of inputs and outputs. The output of such a model represents the expected output of the system under test (SOT). Mineral model-based testing method. The system safety-related behavior is defined in the safety requirements specification. Test cases are derived from a safety model that is extracted from the SHUT and from formal safety requirements. This model encodes the intended behavior and maps each possible input to the corresponding output. Safety test selection criteria relate to the functional safety of the safety- critical system, to the structure of the model (state coverage, transition coverage), and also to a well defined set of system faults.

Safety test case specifications are used to formalize the safety test selection criteria and render them operational. For the given safety model and the safety test case specification, an automatic safety test case generator and optimizer generates the safety test case suite. Finally, the concreted input part of a test case is submitted to the SHUT and the SOT’s output is recorded. The concentration of the input part of a test case is performed by a safety test engine. Besides executing the safety case, it can also compare the output of the SHUT with the expected output as provided by the safety test case [6]. 1 Figure 3: Model-based Safety Testing according Gang You et al. (Source: Test Case Generation One of the most commonly tools for test case generation are model checking techniques. The main purpose of model checking is to verify a formal safety property (given as a logic formula) on a system model. In test case generation, model checking is used in order to find violations of certain formal safety properties. Safety models of safety-critical software systems may have a huge number of states. Therefore the greatest challenge when using a model checker is to cope with the state space explosion.

As a countermeasure, Gang You et al. ‘s approach applies the safety model, which is derived from SHUT and certain safety requirements. The model 12 limits the number of states by splitting them into three subsets (NUNS, FPS, IRS) containing only representative states (see 2. X). Moreover the safety model encodes he intended behavior, and from its structure, safety test cases can be derived. It thereby restricts the possible inputs into the SHUT and the set of possible separately behaviors of the SOT.

Hence, to reduce the amount of testing and guarantee the quality of testing the model checker will search those most frequently entered states and generate the corresponding safety test cases without searching the whole state spaces. The selection of states is based on the safety requirements (Sills). Generally speaking, the safety model can be seen as a test selection criterion generate safety-related test cases. Figure 4 shows the corresponding flow chart. 1 . The system safety model in the form of a finite state machine (FSML) is transformed into the input language of the model checker tool (SPIN) 2.

Each test requirement of a given safety criterion is formulated as a temporal logic expression (LET). 3. Based on the Markova model of a system, the state space is divided into three subsets. 4. In term of these subsets, the negation of each expression of the formula is verified by the model checker. If there is an execution path in the model that does not satisfy the negated formula then it is presented by the model checker as a counter-example. This path becomes a test sequence that satisfies the original test requirement. 5.

The inputs and outputs that form the executable test case are extracted from the counter-example or are derived by a corresponding guided simulation of the model. 13 Figure 4: Test Case Generation Framework according Gang You et al. (Source: 4. 2 Statistical Testing As already mentioned in 2. 1 reliability is defined as the conditional probability that the system will perform its intended function. This chapter will link the reliability of a system with the Markova usage model (see 2. 3). Let f: be a function that shows the failure probability of a software. The argument D represents the possible usage set of the software.

Each element AXED is a usage path from quo (initial operation) to send (final operation) The relation between software reliability R and failure probability F is: R=l -F (2). In the assumed model the failure behavior of the software only depends on its usage path X and not on the input. This means that the input domain corresponding to the used X is homogeneous. The simplest way of obtaining unbiased reliability estimation of the software is to select N test paths XSL, XX, … , CNN according to the usage model. The exult of the function f(Xi) is 1 if the path fails and O otherwise.

Then the arithmetic 14 mean of f(Xi) is an unbiased estimate PEP(f(X)), which is the mathematical expectation of the software failure probability under transition matrix P. Hence, the software reliability can be expressed as R=l -PEP(f(X)) [3]. Critical operations are infrequently executed in real applications. This generates the problem that development organizations have to spend too much time when performing adequate statistical testing. Although one can overcome these drawbacks by increasing the execution probabilities of critical operations during statistical entire software under test. Yang Going et al. 3] found a possible approach to overcome this problem: Importance Sampling (IS) Based Safety-critical Software Statistical Testing Acceleration. IS Based Safety-critical Software Statistical Testing Acceleration This chapter presents the Is-based software statistical testing acceleration method. It ensures that the critical operations tested adequately by adjusting the transition probabilities in the matrix of the usage model, and at the same time, produces the unbiased reliability of the software under test. The IS technique reduces simulation run times hen estimating the probabilities of rare events by Monte Carlo simulations [3].

For complex software with a large model matrix, the simulation procedure is often extremely time consuming. To overcome this problem, Yang Going et al. ‘s approach adopts a simulated annealing algorithm to calculate the optimal matrix Q. This widely used optimization method employs stochastic techniques to avoid being trapped in local optimal solution. The 16 exact mathematical explanation of this algorithm is complex and would be out of the scope of this paper. [3] 4. 3 Method Comparison Although model-based and statistical testing follow completely different approaches, the challenges are very similar.

Both methods have to limit the extent and complexity of testing. Model-based testing reduces the number of test cases by restricting the state space domain of the Markova chain usage model. Whereas statistical testing reduces the number by changing the relation between critical and normal test cases with help off likelihood ratio. 5 Conclusion Today an increasing number of safety-critical applications are controlled by computer software. Therefore effective testing tools are required to provide a high degree of safety and to reduce severe failures to a minimum. The paper focused on