A runtime test tool for applications

The LSB specifies 17 shared libraries, which contains the 6600 interfaces. The interfaces in each library are grouped into logical units called a LibGroup. The LibGroups help to organize the interfaces, which is very useful in the written specification, but isn't used much elsewhere.

Figure 1.  Relationship between Library, LibGroup and Interface

An Interface represents a globally visible symbol, such a function, or piece of data. Interfaces have a Type, which is either the type of the global data or the return type of the function. If the Interface is a function, then it will have zero or more Parameters, which form a set of Types ordered by their position in the parameter list.

Figure 2.  Relationship between Interface, Type and Parameter

As mentioned above, the database contains enough information to be able to generate header files which are a part of the LSB development tools. This means that the database must be able to represent C-language types. The Type and TypeMember tables provide these. These tables are used recursively. If a Type is defined in terms of another type, then it will have a base type that points to that other type.

For structs and unions, the TypeMemeber table is used to hold the ordered list of members. Entries in the TypeMember table point back to the Type table to describe the type of each member. For enums, the TypeMember table is also used to hold the ordered list of values.

struct foo {
	int	a;
	int	*b;
	}
        

Headers, like Libraries, have their contents arranged into logical groupings known a HeaderGroups. Unlike Libraries, these HeaderGroups are ordered so that the proper sequence of definitions within a header file can be maintained. HeaderGroups contain Constant definitions (ie #define statements) and Type definitions. If you examine a few well designed header files, you will notice a pattern of a comment followed by related constant definitions and type definitions. The entire header file can be viewed as a repeating sequence of this pattern. This pattern is the basis for the HeaderGroup concept.

Figure 5.  Organization of Headers

One last construct in our database should be mentioned. While we are able to represent a syntactic description of interfaces and types in the database, this is not enough to automatically generate meaningful test cases. We need to add some semantic information that better describes how the types in structures and parameters are used. As an example, struct sockaddr contains a member, sa_family , of type unsigned short . The compiler will of course ensure that only values between 0 and 2^16-1 will be used, but only a few of those values have any meaning in this context. By adding the semantic information that this member holds a socket family value, the test generator can cause the value found in sa_family to be tested against the legal socket families values ( AF_INET , AF_INET6 , etc), instead of just ensuring the value falls between 0 and 2^16-1, which is really just a noop test.

As mentioned above, the data in the database is used to generate different portions of the LSB project. This strategy was adopted to ensure these different parts would always be in sync, without having to depend on human intervention.

The written specification contains tables of interfaces, and data definitions (constants and types). These are all generated from the database.

The LSB development environment consists of stub libraries and header files that contain only the interfaces defined by the LSB. This development environment helps catch the use of non-LSB interfaces during the development or porting of an application instead of being surprised by later test results. Both the stub libraries and headers are produced by scripts pulling data from the database ^[1] .

Some of the test suites described previously have components which are generated from the database. Cmdchk and libchk have lists of commands and interfaces respectively which are extracted from the database. The static application test tool, appchk, also has a list of interfaces that comes from the database. The dynamic application test tool, dynchk, has the majority of its code generated from information in the database.

The mechanism for interacting with the application must be transparent and non-interfering to the application. We considered the approach used by 3 different tools: abc , ltrace , and fakeroot .

We chose to use the LD_PRELOAD mechanism because we felt it was the simplest to use. Based on this mechanism, a sample test case looks like this:

	ssize_t read (int arg0 , void * arg1 , size_t arg2 )
	{
        if(!funcptr)
                funcptr = dlsym(RTLD_NEXT, "read");
        validate_filedescriptor(arg0, "read");
        validate_RWaddress(arg1, "read");
        validate_size_t(arg2, "read");
        return funcptr(arg0, arg1, arg2);
	}
      

One problem that must be avoided when using this mechanism is recursion. If the above function just called read() at the end, it would end up calling itself again. Instead, the RTLD_NEXT flag passed to dlsym() tells the dynamic linker to look up the symbol on one of the libraries loaded after the current library. This will get the original version of the function.

The test set functions are organized into 3 layers. The top layer contains the functions that are test stubs for the LSB interfaces. These functions are implemented by calling the functions in layers 2 and 3. An example of a function in the first layer was given above.

The second layer contains the functions that test data structures and types which are passed in as parameters. These functions are also implemented by calling the functions in layer 3 and other functions in layer 2. A function in the second layer looks like the following.

	void validate_struct_sockaddr_in(struct sockaddr_in  * input, char *name)
	{
        validate_socketfamily(input->sin_family,name );
        validate_socketport(input->sin_port,name );
        validate_IPv4Address( &(input->sin_addr),name );
	}
      

The third layer contains functions that test the types which have been annotated with additional semantic information. These functions often have to perform non-trivial operations to test the assertion required for these supplemental types. Here is an example of a layer 3 function.

	void validate_filedescriptor(const int fd, const char *name)
	{
        if (fd >= lsb_sysconf(_SC_OPEN_MAX))
		ERROR("fd too big");
	else if ( fd < 0 )
		ERROR("fd negative");
	}
      

Presently, there are 3056 functions in layer 1, 106 functions in layer 2, and just a few in layer 3. We estimate that the total number of functions in layer 3 upon completion of the test tool will be on the order of several dozen. The functions in the first two layers are automatically generated based on the information in the database. Functions in layer 3 are hand coded.

So far, we have come up with an overall architecture for the test tool, selected a mechanism that allows us to hook the tests into the running application, discovered the pattern in the test functions so that we could create a template for automatically generating the code, and implemented the scripts to generate all of the tests cases. The only problem is that now we run into the real world, where things don't always follow the rules.

Here are a few of the interesting situations we have encountered

No doubt, there will be more interesting situations to have to deal with before this tool is completed.