Test application generation

Test applications can be generated from interface and component models. A test application generates on-the-fly test sequences of events based on the used model (interface or component) and sends these events to a system under test. An adapter is used as intermediary between the test application and the system under test. The adapter receives events from the test application and translates them to calls understandable by the system under test. It also receives the responses from the system under test and translates them to events understandable by the test application.

If the used interfaces contain triggers with parameters, replies and notifications with variables or any (*) value then one or more parameters file are required.

In order to generate a test application, add a task to the .prj file similarly to the following example:

import "Camera.interface"

Project Camera {

	Generate TestApplication {
        // For components, use "for component" instead
        Testapplication_ICamera for interface ICamera
	    {
            params: "ICamera.params"
	    }
	}
}

The params option is only required when the interface has commands/signals with input parameters and replies/notifications with variables or any (*) expression.

Path to adapter

The path to the executable file for the adapter can optionally be given in the generator task as follows:

Testapplication_ICamera for interface ICamera
        {
            params: "ICamera.params"
            adapter: "C:/ComMA/Adapter/bin/adapter.exe"
        }

The path to the adapter has to be given with forward slashes or if backward slashes are used they have to be escaped in the following way: C:\\\\ComMA\\\\Adapter\\\\bin\\\\adapter.exe

Timeouts

In some cases the test application can receive events from the system under test, for example, a reply to a command or a notification. The time for receiving such events can vary and is generally unknown in advance. The user can specify how long the test application should wait for events before taking other actions or stopping. See the timeouts option in the following example:

Testapplication_ICamera for interface ICamera
        {
            params: "ICamera.params"
            adapter: "C:/ComMA/Adapter/bin/adapter.exe"
            timeouts {stop when no events observed: 10.0 sec default: 0.2 sec}
        }

The first timeout value indicates how long the test application will wait for events before stopping in case no other actions are possible according to the model. The second timeout value (the default) indicates the waiting period if other actions are possible. When this timeout expires and no events are received from the system under test, the test application will take an action admitted by the model. Usually the first value is larger than the second one.

Multiple clients or servers on a single port

Test application can be generated from a component model. By default, the test application will have one client for each provided port and one server for each required port. The user can change this default behavior by defining the clients and servers for each port. It is also possible that each client or server has its own parameters file. This is illustrated in the following example .prj file.

Testapplication_Camera for component CameraComponent
        {
            params: "ICamera.params"

            adapter: "C:/ComMA/Adapter/bin/adapter.exe"

            client c1 ports controlPort default_params
            client c2 ports controlPort "ICamera1.params"

            server l ports lensPort "ILens.params"
        }

In this example, there will be two clients (named c1 and c2) for controlPort. The first client will use the default parameters file given in the params section. The second client will use the parameters file ICamera1.params. There is also one server named l that will be used at the hypothetical required port lensPort.

It should be noted that if parameters are not required for a given port, 'default_params' still needs to be given.

If no clients or servers are specified for a given port, there will be no default client/server generated for it.

It is possible to use the same component name for both the client and server roles. Furthermore, a client component can be a client for more than one port, for example:

client c1 ports controlPort default_params servicePort "IService.params"

See also the help page on the generation of a simulator for another example.

Generate and run test application

To generate the test application, run the .prj (right click on the .prj file → Run As → Run generators). The client can be started by executing the generated .bat/.sh in the src-gen/testapplication folder.

Once the test application is started you will see a window like below:

image

The path to the adapter will be filled in the field to the left of the Start button.

Pressing this button will start the adapter and the test application. The label of the button changes to Stop. The events sent to and received from the system under test are shown in the window. The test process can be stopped by pressing the button Stop. The process may also stop of its own if no further events can be generated based on the model or if an error is detected.

Pressing Save Coverage Info creates files with information about the visited model states and the transitions that were traversed. Details of the structure of transition coverage file can be found at the end of this page.

Apart from the basic Start/Stop functionality, the test application has the following options:

  • Saving an execution to a file. Selecting the check box Save to file will lead to serialization of the exchanged events in a file with extension .recording.

  • A recording file can be used by pressing Rerun from file button. This will show dialog window to load a recording file.

  • Using a recording file comes with two alternative options. Continue recorded run will continue the run after replaying the events in the recording. The option Continue and save recorded run will save the run to a recording file.

Break

A break statement can be added to recordings (at the and of a line) and to parameter files.

These break statements are used when the Break check box in the bottom part of the user interface has been selected. Then the test application will stop when a statement with a break can be selected and the user interface presents the options for the next statement. The user can select the next action, continue in step mode, or let the test application continue until the next statement with a break.

Behavior of the test application

The test application works as follows:

  • In case of an interface model, the test application selects commands and signals that are allowed at the current state of the interface model and sends them to the system under test via the adapter. The system under test may respond with reply or notifications. The test application checks if the received events are allowed in the current state and if the parameter values conform to the ones given in the model.

  • In case of a component model, the test application selects one of the allowed commands or signals on a provided port or one of the allowed reply or notifications on a required port. The chosen event is sent to the system under test. It in turn may respond with reply/notifications on a provided port or a command/signal on a required port. The received event is checked against the component model.

The adapter plays a crucial role in the described communication. It is usually implemented manually, the implementation reflects the specifics of the system under test.

The interaction between the test application and the adapter is as follows:

  • The test application starts the adapter, which in turn may need to initialize and start the system under test.

  • The test application and the adapter communicate via the standard input/output. The events are in JSON format following the structure described on the JSON events file page.

  • The adapter reads events from the standard input, translates the event and the parameter data (in JSON) to a call with parameter values specific to the system under test.

  • The adapter converts the input from the system under test to a JSON event and sends it to the test application.

The values in parameters files are used by the test application depending on the scenario:

  • In case of interface model, the parameters for commands and signals are used.

  • In case of component model, the parameters for commands and signals on provided ports are used as well as the parameters for replies and notifications on required ports.

Limitations:

  • Test application for component models:

    • Component models with parts are not supported (compound components)

  • Time and data constraints are not supported

Structure of transition coverage files

Transition coverage files contain comma-separated value data. For example:

Component,Port,Interface,Clause,SourceLine
c1,vmUserPort,IUser,C_AcceptUserCommands_0_InsertCoin()_0_0,18
c1,vmUserPort,IUser,C_AcceptUserCommands_4_OrderProduct()_1_0,56
c1,vmServicePort,IService,C_Operational_ias0.0_NonTriggered_0_0,11

Clauses in the transitions are identified by the following naming scheme. For easier tracing back to the original source file, the line number is given.

For transitions with a trigger:

C_<stateName>_<indexOfTransitionInState>_<TriggerName>_<indexOfClause>_0

For transitions without a trigger:

C_<stateName>_<indexOfTransitionInState>_NonTriggered_<indexOfClause>_0

Observe 'NonTriggered' in the example.

The following example shows that the transition is defined in 'in all states' block (observe the 'ias' part).

C_<stateName>_ias<indexInAllStates>.<indexTransition>_<TriggerName>_<indexClause>_0