Title:   MultiGenTest

 

Date:    04 April 2002

 

Revision History:           25 March 2002, Jim Trainor, Initial version.

                                    04 April 2002, Jim Trainor, Added test coverage section.  Several minor edits.

 

MultiGenTest is a tool for testing cross platform, and cross version, AAF compatibility.  It is comprised of three parts: i) a set of test modules in the MultGenTests directory, ii) a test harness, MultiGenTest.cpp, that is used to execute any set of individual test modules, and iii) a Perl script, MultiGenTest.pl, that automates the execution of the MultiGenTest program across a set of platforms, COM library versions, and individual tests.

 

MultiGenTest Program

 

Each test module in the MultiGenTests module implements a simple test protocol (C++ virtual base class).  The test modules make their existence known by registering a factory object with a registry.  The test module factories are static objects that register themselves when they are constructed at program init time.

 

Each test takes it own set of command line arguments.  The MultiGenTest program parses the command line and pulls out the command line arguments for each tests.  It then executes, in order, each test that is found on the command line.  A typical MultiGenTest command line looks like this:

 

./MultiGenTest –r aTest anArg anArg –r anotherTest anArg anArg anArg …..

 

All useful processing is done by test modules.  The role of the code in MultiGenTest.cpp is simply to execute test modules. Even loading the COM library is done within a test module.

 

A small amount of state information can be passes from one test to the next.  Currently the only state information is the currently open AAF file.  The “FileOp” test module will open files, and save the resulting IAAFFile pointer in the state object.  Subsequent tests use this to access the open file.  Other state information can be added as required.

 

There is a noop’ed test implementation that is useful as a template to write new tests.

 

A typical MultiGenTest command is:

 

./MultiGenTest –r LoadLib /my/aaf/path/libcom-api.so –FileOp write /tmp/test.aaf \

-r aTest anArg anArg anArg –r FileOp save_and_close –r UnloadLib

 

            If no arguments are passed to MultiGenTest it will dump the list of registered tests.

 

MultiGenTest Script

 

            MultGenTest.pl is a Perl script that will execute a set of tests across a set of platforms and COM library versions.

It takes a single argument that is another Perl script that is responsible for initializing numerous configuration parameters.  A sample configuration file is present in the samples directory (samples/sample_cfg.pl). That sample file documents the configuration parameters.

 

            The test script implements a simple test procedure that executes every test on every platform using every version of the COM library.  The result of each test (an .aaf file) is then verified on each platform using each version of the COM library.

 

            A test that modifies a file, will modify the result of a previous test that created that file.   The original file is copied prior to the execution of each modify test (per platform and version).  Modify tests operate on the copied file, not the original. The result of each modify test is then verified on each platform and version.

 

            The algorithm looks like this:

 

            foreach test T {

                        foreach platform Pc {

                                    foreach version Vc {

 

                                                if ( test creates a new file ) {

                                                            // We need to test that we create the file on this platform

                                                            // using this version.  Then, we need to verify that we can

                                                            // read the resulting file on every platform/version combo.

                                                            execute test on Pc using Vc

                                                            foreach platform and version {

                                                                        verify contents of created file

                                                            }

                                                }

 

                                                else if ( test modifies the result of a previous test ) {

 

                                                            // We need to test that every platform/version combo

// can modify a file created on any other platform/version

// combo.  This means that for *every* version of a test file we

// created above, we must run the modify test using *every*

// possible platform/version comb.

 

                                                            // We also need to verify the every platform/version combo

                                                            // can read the modified file.

 

                                                            foreach platform Pm and version Vm {

                                                                        copy result of test (T,Pc,Vc) to a new file named (T,Pm,Vm)

                                                                        modify the new file

                                                                        foreach platform and version {

                                                                                    verify contents of modified file

                                                                        }

}

}


} end foreach version

                        } end foreach platform

            } end foreach test

 

The MultiGenTest program is used to create, modify, and verify files.  The command line arguments required to perform these operation are part of the configuration.

 

The Perl script that implements this algorithm will execute MultGenTest this many times:

 

            Nct = num create tests

            Nmt = num modify tests

Np = num platforms

Nv = num versions

 

            Total MultGenTest executions = (1 + NpNv)( Nct + NmtNpNv)NpNv

 

 The sample configuration file (samples/sample_cfg.pl) has:  Nct = 1, Nmt = 1, Np = 3, Nv = 2.  This works out to

294 MultiGenTest executions (some are excluded – see note below).

 

  The output of MultiGenTest.pl may be found in samples/sample_test_log.txt.gz.  Note, there is an additional test, not discussed here, that boosts the total number of MultiGenTest executions to 300.  This count can be seen at the bottom of the log.

 

  It takes about three minutes to execute the tests in sample/sample_cfg.pl – not unreasonable.

 

  The total number of MultiGenTest executions varies O(1) with Nct and Nmt, hence, the total execution time will not grow unreasonably as more tests are added.

 

  However, the total number of MultiGenTest executions varies O(3) with Np and Nv.  Adding too many platforms or versions could lead to unreasonable total execution times.

 

   With Nct = Nmt = 8 (just a guess of a typical test vector size), Np = 4 (Linux,Mac,Irix,Win2K), and Nv =4 (V1,V101,V15), 16224 executions would be required.  A rough time estimate is: 16224/294 * 3 minutes = 165 minutes.  That is not too bad considering how unusual it would be to run a test with such complete platform and version coverage.

 

  Some simple optimizations could reduce that.  For example, caching the result of previous executions and only re-executing the test vectors that failed (vector meaning the failed test and everything that depended on it).  That may be useful for a developer requiring a fast test cycle.

 

  One final minor detail:  some tests are known not to work.  For example, AAF COM library V1 cannot modify files.  These are handled using some ad-hoc test exclusion rules.  Excluded rules are counted as such and reported along with tests that passed and failed.  The total number of tests reported by MultiGenTest.pl should be constant regardless of the presence of excluded tests.

 

Coverage Requirements/Analysis

 

 The test implementations (in the MultiGenTests directory) attempt to cover, as a minimum requirement, each stored format defined by the Object Manager.  These are:

 

            SF_DATA

            SF_DATA_STREAM

            SF_STRONG_OBJECT_REFERENCE

            SF_STRONG_OBJECT_REFERENCE_SET

            SF_STRONG_OBJECT_REFERENCE_VECTOR

            SF_WEAK_OBJECT_REFERENCE

            SF_WEAK_OBJECT_REFERENCE_SET

            SF_WEAK_OBJECT_REFERENCE_VECTOR

 

  These stored format types are reflected in the COM interface implementations in ref-impl/src/impl by the OM types that appear in the implementation class declarations.  An ImplAAF* class may have an OMStrongReference data member.  This data member has the stored format: SF_STRONG_OBJECT_REFERENCE.   The MultiGenTest modules attempt to implement at least one operation that writes/modifies/read each stored format type.

 

            The following operations are (will be) implemented to cover each of the stored formats:

 

            [*]  = implemented

            [+] = partially implemented

 

            SF_DATA

 

- Mob name set/get  (variable size property). [*]

 

- MobSlot physical track number set/get (fixed size property) .

 

            SF_DATA_STREAM

 

                        - Create/append/verify essence data (created as a result of IAAFMasterMob::CreateEssence()). [+]

                          (Append essence test currently fails, also, a test is still required to verify the actual essence data.).

 

            SF_STRONG_OBJECT_REFERENCE

 

                        - Set/get _rendering parameter of ImplAAFOperationGroup.

 

                                    and/or

 

                        - Set/get _selected parameter of a Selector object.

 

            SF_STRONG_OBJECT_REFERENCE_SET

 

                        - Add/remove mobs and essence data from the ContentStorage object. [*]

 

            SF_STRONG_OBJECT_REFERENCE_VECTOR

 

- Add/remove comments (TaggedValues) from mobs. [*]

 

- Add MobSlots (as a result of IAAFMasterMob::CreateEssence()) [*]

 

                        - Add/remove Segments to the _alternatives property of the Selector object.

 

            SF_WEAK_OBJECT_REFERENCE

 

                        - Set/get _dataDef of an OperationDef or the _operationDef of an OperationGroup.                 

 

            SF_WEAK_OBJECT_REFERENCE_SET

 

                        - append/get/count/lookup parameter definitions of an OperationDef.

 

            SF_WEAK_OBJECT_REFERENCE_VECTOR

 

                        - Append/remove/get values in the “degrade to” operation vector of an OperationDef.

 

   Tests selected based on stored format coverage map quite well to the test list provided by Oliver Morgan:

 

            1. Modify scalar property

2. Append/insert/delete/modify strong ref vector

3. Append/delete/modify strong ref set (bundle)

4. Append/insert/delete/modify weak ref vector

5. Append/delete/modify weak ref set (bundle)

6. Modify stream property (e.g. WAVE essence data)

7. Insert/delete Package and traverse mob chain

8. Register new property and type on existing class

9. Register new subclass and add new property

10. Verify polymorphic traversal

   

END a: Trainor, rsion