Migration of Parameter files to the XML format

This page explains the migration problems and plans with the simulation parameter files in Simthetic, its relation to class parameters and constructor arguments, and our goals for the final solution of these issues.

(Updated 2005-09-20) A general explanation of the current and recommended approach for parameterfiles and XML can be found here: The XML file format of the Parameter files

Overall goal

Every class that is used in Simthetic and its libraries usually has several parameters that are different in different simulation setups (e.g.: the number of output bits in the simthlib::BitGenerator class; the number of reflection paths in the simthlib::WssusChannel radio channel). These parameters should therefore be read from a configuration file at simulation startup time (the file is called "the parameter file"), and the values should somehow passed on to the respective objects.

New approach

We now invented the simth::PropertyList. Every property has a name, a type and a value. A Device stores a list of such properties, i.e. a PropertyList. The lookup of values is performed with template functions PropertyList::getProperty<T>() (which check whether a Property with that name exists and also for the correct type) so that the return value is already type-correct. The Device class has the shortcut functions Device::getProperty<T>() which directly work on the device's PropertyList. The constructor no longer declares all parameters as separate arguments, but instead the constructor only gets passed one PropertyList.

Previous (old) approach

Originally, all of these parameters have been programmed as arguments of the constructor. The objects are created inside some static factory method (e.g. initChannel() or now Channel::init()). In that factory method, the desired parameters are extracted from the parameter files and then passed as arguments to the constructor when the respective object is created. (The factory method then returns an upcast pointer to the newly created object.) The extraction of the parameters from the parameter file is done by extraction functions like simth::getDouble(). These extraction functions check whether the parameter actually exists in the parameter file and throw an Exception if it doesn't (if there is no default value), they perform a (rough) type check, and they return the given value (or the default value if one was specified).

Biggest Problems here:

• Default values might get specified twice -- first in the declaration of the constructor, secondly in the extraction function in the factory function. This duplication of information is very confusing and bad.

• The spelling of the parameters in the parameter file depends only on the spelling in the extraction function. This is documented nowhere else. Users who need to check the spelling have to look it up in the factory method, which is hidden somewhere in some .cpp file. This is not good documentation.

• If one wants to add a new parameter to a class, then one has to add a lot of things: additional constructor argument, additional member variable, additional call to the extraction function in the factory method, additional auxiliary variable in the factory method. This is a lot of work which often is really overkill.

Previous Interim Implementation without XML

The PropertyList is defined inside the factory method by calling PropertyList::addProperty() for each Property. With addProperty(), the Property's name and type is defined as well as the default value (or the empty string if it doesn't have one) and an explanatory documentation string. These four things are already used for the documentation: They are extracted from the cpp file automatically by the script extractProperties and transformed into some appropriate file api-doc/properties.h so that the doxygen documentation of a class includes its properties. (However, this script can't always find out automatically which properties belong to which class, but nevertheless it already is some nice documentation.) The PropertyList as just defined is then populated with values by the function simth::readProperties(), which reads the value of each defined Property from the conventional parameter file. After that, the now populated PropertyList is passed as argument to the constructor when the object is newly created.

Previous Interim approach with XML-based PropertyLists

(updated 2004-01-30) As a next step, the definition of the PropertyList for each individual Device are no longer being done "by hand" inside the factory method. Instead, the definition of a Device's PropertyList resides in an additional XML file. The file format is specified with a DTD which can be found in api-doc/devicelist.dtd which is replicated below. These additional XML file(s) are installed in a known directory which is specified at installation time of simtheticbase, and the class simth::DeviceManager can be used to retrieve information from these files. In the factory method, the code block which previously hold all the PropertyList::addProperty() method calls now is replaced by a call to the simth::DeviceManager that will: 1. read the XML file for that Device, and 2. return the defined PropertyList.

The DTD for the devicelist XML files is as follows:

<!-- ............................................................ -->
<!-- 
Typical Invocation:
#  <!DOCTYPE devicelist PUBLIC 
#   "-//TUHH//Simthetic Devicelist V1.0//EN" 
#   "devicelist.dtd">

Purpose:

  This DTD was developed to specify the properties of a Device in the
  simulations system Simthetic. Basically, an XML file adhering to
  this XML should specify everything about a device that is needed by
  any GUI.
-->
<!-- ............................................................ -->

<!-- A devicelist file can include many device definitions. The
     include element (see explanation there) can appear file-wide here, or
     can appear in particular devices. -->
<!ELEMENT devicelist (include*, device*) >

<!-- Each device element describes one C++ class. The C++ typename has
     to appear in the type attribute. The libraryname attribute contains
     the name of the shared library this device is implemented in. The
     extends attribute contains the name of the C++ base class, if this
     device is derived from another one. The abstract attribute is true if
     the C++ class is an abstract class.  -->
<!ELEMENT device (include*, description, inputinterfaces?, 
                  outputinterfaces?, overriddenproperties?, propertylist?) >
<!ATTLIST device
   name CDATA #REQUIRED
   type ID #REQUIRED
   libraryname CDATA #REQUIRED
   extends CDATA #IMPLIED
   abstract (true|false) "false">

<!-- The include element should point to the file name that contains
     the devicelist of the base class. This is just for helping the
     DeviceManager finding all necessary xml files. -->
<!ELEMENT include (#PCDATA)>
<!ELEMENT description (#PCDATA)>

<!-- The input- and outputinterfaces describe all the interfaces of a
     device. The attribute "multiplied_by" contains the name of a
     property, which, if it is set, gives the number of multiple
     instances of this interface that will actually exist.
-->
<!ELEMENT inputinterfaces (interface*)>
<!ELEMENT outputinterfaces (interface*)>
<!ELEMENT interface (description?)>
<!ATTLIST interface 
   type CDATA #REQUIRED
   multiplied_by CDATA #IMPLIED>

<!-- Sometimes, a derived class will override properties of the base
     class by its own values as defined in the C++ code. The names of these
     properties are listed in this element. -->
<!ELEMENT overriddenproperties (value*)>

<!-- Most important for the definition of a useful simulation
     scenario: The propertylist describes all properties that are available
     in this device. The propertylist defined here is also directly used
     inside simthetic when the objects are created. -->
<!ELEMENT propertylist (property*)>

<!ELEMENT property (default?, description, option*)>
<!ATTLIST property
   name CDATA #REQUIRED
   type (bool | int | unsigned | double | complex | string | 
         list_string | enum | sequencelength | vector_octal_int |
         vector_double | octal_int ) #REQUIRED
   empty_default (true|false) "false"
   obsolete (true|false) "false"
   grouping CDATA #IMPLIED>

<!-- For most property types, the option element describes possible
     values that particularly make sense for this property (please add
     appropriate descriptions). But for properties of type 'enum' the
     semantics are different: For 'enum' properties, the property is only
     allowed to have values that are mentioned here as option elements. -->
<!ELEMENT option (value, description?)>

<!-- The default element simply contains a default value for a
     property. If it is empty and the property attribute empty_default is
     false, then this property has to be set in the simulation parameter
     file. -->
<!ELEMENT default (#PCDATA)>
<!ELEMENT value (#PCDATA)>

This interim approach fortunately does not require any further changed in Simthetic's structure for those devices that are already PropertyList-based. It can therefore be implemented step-by-step.

Current (ultimate) approach with XML-capable Simthetic

(updated 2004-04-30) Eventually, not only the definition of the PropertyLists is done in some XML file, but also the parameter files for a particular simulation is switched over to some (other) XML format. The previous (legacy) parameter file format is clumsy and has some very ugly syntax limitations inside. Instead, the new XML file format for the simulation parameter file is being described below, and it can very easily also hold information that is used by some GUI.

This is done as follows: The class simth::ParameterDescription reads the XML PropertyList file. The StreamDrivenSimulationSystem will read all PropertyList, populate it with values from the simth::ParameterDescription, and then pass it as argument to the constructor when the object is newly created. The device objects are created through the factory function from simth::DeviceFactory::allDevices().

The current DTD for the XML-based parameter file is the following:

<!-- ............................................................ -->
<!-- 
Typical Invocation:
#  <!DOCTYPE parameterfile PUBLIC 
#   "-//TUHH//Simthetic Parameterfile V1.0//EN" 
#   "parameterfile.dtd">

Purpose:

  Proposed DTD for specifying the parameters of one concrete
  simulation run of the simthetic simulation system. This file is
  meant for the final solution where every Device of Simthetic is
  solely configured via a PropertyList. In this final case, this
  parameter file would simply "fill in the values" for all properties
  of the used devices.
-->
<!-- ............................................................ -->


<!ELEMENT parameterfile (simulationparameters*)>
<!ATTLIST parameterfile
   xmlns:gui CDATA #FIXED "http://simthetic.sourceforge.net/dtd/gui/1.0">
<!-- Note: The URL for the namespace does not have to point to any
     valid location. It is simply used to distinguish from possible 
     other namespaces. -->

<!ELEMENT simulationparameters (title?, devicevalues?, variables?,
                       devicesvalues, loopvariables, connectionlist,
                       gui:objects?) >
<!-- title = (unimplemented) Short descriptive title of this simulation
     devicevalues = This holds the global properties of the simulation 
                    system. The name is (admittantly) misleading, but this
                    is by far the easiest implementation, hence we stick 
                    with this element here.
     variables = A list of string variables (name and value) that 
                 can be re-used in the values of other properties 
                 by writing $(variable_name).
     devicesvalues = The list of devices and their properties 
     loopvariables = The list of loop variables and their settings
     connectionlist = The list of connections between the devices
     gui:objects = List of additional GUI objects (text, lines) which 
                   might be useful for explaining the system.
-->

<!ELEMENT title (#PCDATA)>
<!ELEMENT description (#PCDATA)>

<!ELEMENT variables (variabledef*)>
<!ELEMENT variabledef (description?, value)>
<!ATTLIST variabledef 
   name ID #REQUIRED>

<!ELEMENT value (#PCDATA)>

<!ELEMENT propertyvalues (propertyvalue*)>
<!ELEMENT propertyvalue (value)>
<!ATTLIST propertyvalue
   name CDATA #REQUIRED>

<!ELEMENT devicesvalues (devicevalues*)>
<!ELEMENT devicevalues (description | propertyvalues | gui:point)*>
<!ATTLIST devicevalues
   id ID #REQUIRED
   type CDATA #REQUIRED
   gui:rotation CDATA #IMPLIED>

<!ELEMENT loopvariables (loopvariable*)>
<!ELEMENT loopvariable (description?, value+)>
<!ATTLIST loopvariable
   id ID #REQUIRED
   name CDATA #REQUIRED
   type CDATA #REQUIRED>

<!ELEMENT connectionlist (connection*)>
<!ELEMENT connection (description?, gui:point*)>
<!ATTLIST connection 
   fromid IDREF #REQUIRED
   fromintf CDATA #REQUIRED
   toid IDREF #REQUIRED
   tointf CDATA #REQUIRED
   active (true|false) "true">

<!ELEMENT gui:point EMPTY>
<!ATTLIST gui:point    
   x CDATA #REQUIRED
   y CDATA #REQUIRED>

<!ELEMENT gui:objects (gui:text | gui:polygon)*>
<!ELEMENT gui:text (gui:point, value)>
<!ATTLIST gui:text
   gui:rotation CDATA #IMPLIED>
<!ELEMENT gui:polygon (gui:point*)>

Note:

Backward compatibility is maintained by simply falling back to the old parameter file format with the old initialization procedure, if the filename does not end with .xml.

Current Status

Some of the classes in Simtheticlib (approx. 50 out of the ~150) are still in old-approach implementation. These cannot currently be used from any XML parameterfile. Some more of them (the next approx. 50 out of the ~150) are at least converted to the interim approach, and the last third (e.g. BitGenerator, TransmissionMonitor) are already converted to the ultimate XML-based Propertylists. In Mimolib approx. 15 out of the 30 are in the ultimate approach.

The object creation itself does already fully support the PropertyList-based object creation.

TODO

Already completed

a. Define XML format for device's PropertyList XML file -- completed, see devicelist.dtd.

b. Implement infrastructure for generating a PropertyList from XML file (depends on a.) -- completed.

c. Implement auto-generation of documentation from a device's XML file (depends on a.) (For this XML -> HTML conversion XSLT is probably appropriate?) -- completed for doxygen by propertyheader.xsl stylesheet.

f. Define XML format for simulation parameter XML files. -- almost completed, see parameterfile.dtd.

g. Change simulation setup procedures and functions in StreamdrivenSystem so that PropertyLists are populated with values from XML simulation parameter files. (depends on a, b, c, e, f)

Still to be done

d. Convert factory methods to use the PropertyList generated from XML file (depends on a. and b.) -- about 50% completed.

e. Convert existing class code to get their parameters from a PropertyList instead from constructor arguments. about 50% completed.

Estimated work time for these tasks

a. completed

b. completed (30 hours)

c. completed (10 hours)

d. 10 hours

e. 50 hours

f. completed

g. completed (8 hours)

---