[NAME] ALL.daovm.interface.extending [TITLE] Extending Dao VM [DESCRIPTION] In the help entry daovm.interface.embedding, it has been demonstrated that embedding Dao is extremely simple. Here we will demonstrate that extending Dao is also extremely simple. Since Dao supports explicit type specification in function parameter lists, you will not need to write a lot of boilerplate codes to check and convert function parameters from Dao data types to C/C++ data types. This means writing wrapping functions (Dao-callable C functions) is significantly simpler than writing wrapping functions for other languages such as Python or even Lua. 1 The First Simple Extending Module All Dao-callable C functions must have prototype similar to the following example, 1 void MyCFunction( DaoProcess *proc, DaoValue *param[], int nparam ) 2 { 3 printf( "Hello Dao!\n" ); 4 } Each Dao extending module must provide an entry function, 1 // Entry function for each C/C++ module: 2 int DaoOnLoad( DaoVmSpace *vmspace, DaoNamespace *ns ); This function will be called automatically to allow the module to register its functions and types etc. The first parameter is the DaoVmSpace instance which is responsible for loading and managing the module. And the second parameter is the namespace object that will represent this module, so all the functions and types etc. should be registered to this namespace. This function can also be named as DaoXXX_OnLoad, where XXX is the module name that will appear in a load statement such as load XXX or load path.XXX. To register functions to a namespace, one can use one of the following interface functions of DaoNamespace, 1 // Function to register a single function: 2 DaoRoutine* DaoNamespace_WrapFunction( DaoNamespace *self, 3 DaoCFunction fp, 4 const char *proto ); 5 6 // Function to register multiple functions: 7 int DaoNamespace_WrapFunctions( DaoNamespace *self, DaoFuncItem *items ); We will come to the second function later. For the first function, the first parameter is the namespace to which the function is registering to; the second is the function pointer to the function that needs to be registered ( MyCFunction in this case); and the last parameter is the Dao function prototype for the registered function. So you can register the above function MyCFunction as the following, 1 // Register function: 2 DaoNamespace_WrapFunction( nspace, MyCFunction, "HelloDao()" ); So that this function can be called in Dao by name HelloDao without any parameter. To sum it up, the simplest Dao extending module could be the following, 1 #include "dao.h" 2 #include "stdio.h" 3 static void MyCFunction( DaoProcess *proc, DaoValue *param[], int nparam ) 4 { 5 printf( "Hello Dao!\n" ); 6 } 7 int DaoOnLoad( DaoVmSpace *vmspace, DaoNamespace *nspace ) 8 { 9 DaoNamespace_WrapFunction( nspace, MyCFunction, "HelloDao()" ); 10 return 0; 11 } To compile it, you will need to add the Dao header file path to your compiling option. And you will also need to add the following preprocessor definitions: * On Win32: WIN32; * On Unix: UNIX; * On Mac OSX: MAC_OSX; For linking, on Windows you will need to link the module against the Dao library. But on the other platforms, you can simply use the following flags, * On Unix: -rdynamic; * On Mac OSX: -undefined dynamic_lookup; 2 The Second Simple Extending Module Now we will demonstrate how to create a function that can accept parameters and return a value. Suppose we want to create C function with the following Dao function prototype, 1 # Dao function prototype 2 MyTest( id :int, name :string, extra = 0 ) => float So this function will take an integer parameter, a string parameter and an extra integer parameter with default value. This prototype also indicates that it will return a float. In the C function, it is very easy to convert Dao data type to C data type, and to return C data type to Dao, 1 void MyTestInC( DaoProcess *proc, DaoValue *param[], int nparam ) 2 { 3 daoint id = DaoValue_TryGetInteger( param[0] ); 4 char *name = DaoValue_TryGetMBString( param[1] ); 5 daoint extra = DaoValue_TryGetInteger( param[2] ); 6 printf( "MyTest: %i %s %i\n", id, name, extra ); 7 DaoProcess_PutFloat( proc, 0.5*(id + extra) ); 8 } As you can see, there are no boilerplate codes to check the number of parameters or the type of parameters. When the execution reaches the function body of MyTestInC(), it is guaranteed that all the parameter values are available with correct types. Actually the DaoValue_TryGetXXX() methods did check the value type, but in this case, they will always succeed. So if you include the header file daoValue.h in your source file, you can simple write the function as the following to completely (and safely) remove the type checking, 1 void MyTestInC( DaoProcess *proc, DaoValue *param[], int nparam ) 2 { 3 daoint id = param[0]->xInteger.value; 4 char *name = DString_GetMBS( param[1]->xString.data ); 5 daoint extra = param[2]->xInteger.value; 6 printf( "MyTest: %i %s %i\n", id, name, extra ); 7 DaoProcess_PutFloat( proc, 0.5*(id + extra) ); 8 } But the downside of this is that you need to be familiar with the internal Dao data structures. These DaoValue_TryGetXXX() functions will convert the Dao values to the requested C values. And DaoProcess_PutFloat() will put a float value at proper location as the returned value of the C function. Please see the following section(s) for more details. Now this function can be registered as: 1 // Register a function with parameters and returned value: 2 DaoNamespace_WrapFunction( nspace, MyTestInC, "MyTest(id:int,name:string,extra=0)=>float" ); 3 Basic wrapping C/C++ type There are two ways to extend Dao with user defined C/C++ types. One is to wrap it around by a DaoCdata object, and access it as an opaque pointer. This is the standard way to wrap existing C/C++ types. The other is to define a customized C type, and use it in the same way as the first. A customized C type is a C structure sharing the same header fields as DaoCdata, which can make sharing types between Dao and C simpler (especially for garbage collection). Wrapped C/C++ types and customized C types can be added to Dao in almost the identical way, so I will introduce the wrapped types first, and then the customized types should be very easy to understand. 3.1 Type information structure A C/C++ type can be used in Dao if only if it is registered in a Dao namespace with a type information structure through, 1 DaoType* DaoNamespace_WrapType( DaoNamespace *self, DaoTypeBase *typer, int opaque ); 2 int DaoNamespace_WrapTypes( DaoNamespace *self, DaoTypeBase *typer[] ); The opaque parameter must be set to 1 for wrapped types, and 0 for customized types. 3.1.1 Structure definition Here DaoTypeBase is defined as the following, 1 /* Type information structure for creating Dao types for C/C++ types: */ 2 struct DaoTypeBase 3 { 4 const char *name; /* type name; */ 5 DaoTypeCore *core; /* data used internally; */ 6 DaoNumItem *numItems; /* constant number list: should end with a null item; */ 7 DaoFuncItem *funcItems; /* method list: should end with a null item; */ 8 9 /* typers for super types, to create c type hierarchy: */ 10 DaoTypeBase *supers[ DAO_MAX_CDATA_SUPER ]; 11 12 /* function(s) to cast a C/C++ type to and from one of its parent type: */ 13 FuncPtrCast casts[ DAO_MAX_CDATA_SUPER ]; 14 15 /* function to free data: */ 16 void (*Delete)( void *self ); 17 18 /* Get garbage collectable fields (Dao data types with refCount by the type): */ 19 void (*GetGCFields)( void *self, DArray *values, DArray *arrays, DArray *maps, int remove ); 20 }; This structure defines the set of type information needed to create a corresponding Dao type. Obviously each type needs a name, which can be set in the first field of this structure. The second field is reserved for internal use for storing wrapped member constants and methods. 3.1.2 Member constant numbers and methods The third field numItems can be used to specify a list of member constant numbers, which are defined in an array of the following structure: 1 struct DaoNumItem 2 { 3 const char *name; /* contant name; */ 4 int type; /* number type; */ 5 double value; /* number value; */ 6 }; The number type should be one of DAO_INTEGER, DAO_FLOAT and DAO_DOUBLE. The array should be terminated with a item with null name: 1 static DaoNumItem myTypeNumbers[] = 2 { 3 { "MODE_ONE", DAO_INTEGER, MODE_ONE }, 4 { "MODE_TWO", DAO_INTEGER, MODE_TWO }, 5 { NULL, 0, 0 } 6 }; If the type has no member constant numbers, it can be simply set to NULL. The fourth field funcItems can be used to specify a list of member methods, which are defined in an array of the following structure: 1 struct DaoFuncItem 2 { 3 DaoCFunction fpter; /* C function pointer; */ 4 const char *proto; /* function prototype: name( parlist ) => return_type */ 5 }; where the two fields fpter and proto should be the same as they would in the parameter list of: 1 DaoRoutine* DaoNamespace_WrapFunction( DaoNamespace *self, 2 DaoCFunction fp, 3 const char *proto ); 3.1.3 Base types and casting functions The fifth field supers can be use to expose the inheritance structure of C/C++ types to Dao. To do this, one just need to set the supers array to the type information structures of the parent types, and terminate the array with a NULL pointer. If the wrapped type is a C++ class with virtual method(s) or virtual base(s), Dao will need to know how to cast this type to and from its parent type properly. This can be done by specifying a list of cast functions in the sixth field casts of the type structure. If the C++ class has virtual method(s) but no virtual base(s), the cast function should be provided in the following form, 1 void* cast_Sub_Base( void *data, int down_casting ) 2 { 3 if( down_casting ) return static_cast<Sub*>( (Base*)data ); 4 return dynamic_cast<Base*>( (Sub*)data ); 5 } And if the C++ class has virtual base(s), the cast function should be like, 1 void* cast_Sub_Base( void *data, int down_casting ) 2 { 3 if( down_casting ) return dynamic_cast<Sub*>( (Base*)data ); 4 return dynamic_cast<Base*>( (Sub*)data ); 5 } 3.1.4 Deallocation and GC handling If the C/C++ type should not be deallocated by the standard C function free(), a customized deallocation function must be specified for the type structure. For wrapped C/C++ types, the self parameter passed to the deallocation function will be the opaque C/C++ pointer, and for customized C types, it will be the whole objects that have the same header fields as DaoCdata. The last field GetGCFields can be normally set to NULL. But if the C/C++ type may retain reference(s) to Dao data structures that can form potentially cyclic referencing relationships, this field must be set to a proper function. The main purpose of this function is to expose the references retained by the C/C++ type to Dao garbage collector. In this function, the pointers of Dao data that are directly referenced by the C/C++type should be pushed to the values parameters; and for Dao data that are referenced through DArray structures, these DArray structures should be pushed into the arrays; and for those that are referenced through DMap structures, these DMap structures should be pushed into the maps. This function is called with a zero remove parameter automatically by the garbage collector (GC) when an object of the C/C++ type needs to be scanned to determine if it is no longer reachable from the program. It is also called before the object is deallocated by the GC, this time the remove parameter is set to one, and in this case, the GetGCFields must break the references that are pushed into the values, arrays and maps arrays. This is necessary, because some of the objects referenced by the C/C++ object can be deallocated by the GC, breaking references to them will avoid posible double deletion. 3.2 A simple example Given the following C++ class, 1 class ClassOne 2 { 3 public: 4 int value; 5 6 enum{ CLASSONE_AA, CLASSONE_BB }; 7 8 ClassOne( int v ); 9 10 int Method( const char *s ); 11 }; It can be wrapped in the following way, 1 // Declare the wrapper functions first: 2 static void dao_ClassOne_ClassOne( DaoProcess *proc, DaoValue *p[], int n ); 3 static void dao_ClassOne_Method( DaoProcess *proc, DaoValue *p[], int n ); 4 // List of constant member numbers for the enums: 5 static DaoNumItem ClassOneNumbers[] = 6 { 7 { "CLASSONE_AA", DAO_INTEGER, CLASSONE_AA }, 8 { "CLASSONE_BB", DAO_INTEGER, CLASSONE_BB }, 9 { NULL, 0, 0 } 10 }; 11 // List of member constructors or methods of ClassOne: 12 static DaoFuncItem ClassOneMethods[] = 13 { 14 // Methods with the same name as the type name are constructors: 15 { dao_ClassOne_ClassOne, "ClassOne( v :int )" }, 16 { dao_ClassOne_Method, "Method( self :ClassOne, s :string ) => int" }, 17 { NULL, NULL } 18 }; 19 static void ClassOne_Delete( void *self ) 20 { 21 delete (ClassOne*) self; 22 } 23 // The type information structure for ClassOne: 24 static DaoTypeBase ClassOne_Typer = 25 { 26 "ClassOne", NULL, ClassOneNumbers, ClassOneMethods, 27 {NULL}, {NULL}, ClassOne_Delete, NULL 28 }; 29 // The Dao type structure for ClassOne: 30 DaoType *dao_type_ClassOne = NULL; 31 32 static void dao_ClassOne_ClassOne( DaoProcess *proc, DaoValue *p[], int n ) 33 { 34 // Get the integer parameter; 35 daoint v = DaoValue_TryGetInteger( p[0] ); 36 // Create a ClassOne instance: 37 ClassOne *self = new ClassOne( v ); 38 // Wrap the instance with Dao type structure: 39 DaoProcess_PutCdata( proc, self, dao_type_ClassOne ); 40 } 41 static void dao_ClassOne_Method( DaoProcess *proc, DaoValue *p[], int n ) 42 { 43 // Get the ClassOne instance: 44 ClassOne *self = (ClassOne*) DaoValue_TryCastCdata( p[0], dao_type_ClassOne ); 45 // Get the string parameter: 46 char *s = DaoValue_TryGetMBString( p[1] ); 47 int res = self->Method( s ); 48 // Return the integer result: 49 DaoProcess_PutInteger( proc, res ); 50 } 51 int DaoOnLoad( DaoVmSpace *vmSpace, DaoNamespace *nspace ) 52 { 53 // Wrap ClassOne as an opaque C/C++ type: 54 dao_type_ClassOne = DaoNamespace_WrapType( nspace, & ClassOne_Typer, 1 ); 55 return 0; 56 } Since the value member of ClassOne is a public member, it is reasonable to add a getter and a setter method to wrapped ClassOne type. To add a getter, one only needs to register a method with name .field and no extra parameter. And for a setter, the method name must be .field=, and it must also accept a parameter with type the same as the value that can be assigned. For example, for the value member, one can added the following to the ClassOneMethods list, 1 // the getter and setter: 2 { dao_ClassOne_GETF_value, ".value( self :ClassOne ) => int" }, 3 { dao_ClassOne_SETF_value, ".value=( self :ClassOne, value :int )" }, Here the name dao_ClassOne_GETF_value and dao_ClassOne_GETF_value are completely arbitrary. They can be implemented in the following way, 1 static void dao_ClassOne_GETF_value( DaoProcess *proc, DaoValue *p[], int n ) 2 { 3 ClassOne *self = (ClassOne*) DaoValue_TryCastCdata( p[0], dao_type_ClassOne ); 4 DaoProcess_PutInteger( proc, self->value ); 5 } 6 static void dao_ClassOne_SETF_value( DaoProcess *proc, DaoValue *p[], int n ) 7 { 8 ClassOne *self = (ClassOne*) DaoValue_TryCastCdata( p[0], dao_type_ClassOne ); 9 self->value = DaoValue_TryGetInteger( p[1] ); 10 } 3.3 An advanced example Now given the following class that is derived from ClassTwo, 1 class ClassTwo : public ClassOne 2 { 3 public: 4 virtual void VirtualMethod( int i, float f ); 5 }; Because this class has a virtual method, if we want Dao classes can be derived from ClassTwo and reimplement its virtual functions, the wrapping will be a bit more sophisticated. First, we will need to define a "proxy" class that is derived from ClassTwo and reimplements its virtual function such that this reimplemented function can check for a Dao reimplementation of the function and invoke it if it exists. When an instance of ClassTwo needs to be created, an instance of this proxy class will be created and returned instead of the original ClassTwo. Here is an example of such proxy class, 1 class Dao_ClassTwo : public ClassTwo 2 { 3 public: 4 5 DaoCdata *dao_cdata; 6 7 Dao_ClassTwo(); 8 ~Dao_ClassTwo(); 9 10 int VirtualMethod( int i, float f ); 11 }; This proxy class will need to maintain a reference to the wrapper object, so an extra field dao_cdata is declared in the class. This wrapper object can be pre-allocated in the constructor of Dao_ClassTwo, 1 Dao_ClassTwo::Dao_ClassTwo() 2 { 3 dao_cdata = DaoCdata_New( dao_type_ClassTwo, this ); 4 DaoGC_IncRC( (DaoValue*)dao_cdata ); 5 } Here the dao_type_ClassTwo is the Dao type object for ClassTwo, and can be obtained in the same way as dao_type_ClassOne. Now that Dao_ClassTwo has a reference to a DaoCdata object, the GetGCFields field of the type information structure for ClassTwo should be set to a proper function, which will be provided later. The destructor should also be handled this reference properly by, 1 Dao_ClassTwo::~Dao_ClassTwo() 2 { 3 if( dao_cdata ){ // Could have been set to NULL by the GC: 4 // Set the opaque pointer of dao_cdata to NULL, so that the deallocator 5 // of DaoCdata will not attempt to call the deallocator of the opaque pointer: 6 DaoCdata_SetData( dao_cdata, NULL ); 7 DaoGC_DecRC( (DaoValue*) dao_cdata ); 8 } 9 } Then the VirtualMethod() could be implemented in the following way, 1 int Dao_ClassTwo::VirtualMethod( int i, float f ) 2 { 3 DaoVmSpace *vmspace = DaoVmSpace_MainVmSpace(); 4 DaoProcess *proc = NULL; 5 6 // Try to get the instance of a derived Dao class: 7 DaoObject *object = DaoCdata_GetObject( dao_cdata ); 8 if( object == NULL ) goto CallDefault; 9 10 // Try to get a method named "VirtualMethod": 11 DaoRoutine *method = DaoObject_GetMethod( object, "VirtualMethod" ); 12 if( method == NULL ) goto CallDefault; 13 14 // Check if the method is a C/C++ wrapper function: 15 if( DaoRoutine_IsWrapper( method ) ) goto CallDefault; 16 17 // Acquire a process object to execute the re-implemented virtual function: 18 proc = DaoVmSpace_AcquireProcess( vmspace ); 19 20 // Prepare function call parameters: 21 DaoProcess_NewInteger( proc, i ); 22 DaoProcess_NewFloat( proc, f ); 23 DaoValue **params = DaoProcess_GetLastValues( proc, 2 ); 24 25 // Resolve possible overloading using the parameters: 26 // This can be merged with DaoProcess_Call(), if the error code returned 27 // by DaoProcess_Call() is checked an properly handled. 28 method = DaoRoutine_Resolve( method, object, params, 2 ); 29 if( method == NULL ) goto CallDefault; 30 31 // Run the re-implemented function: 32 if( DaoProcess_Call( proc, method, object, params, 2 ) ) goto ErrorCall; 33 34 // Check the returned value: 35 DaoValue *res = DaoProcess_GetReturned( proc ); 36 if( DaoValue_CastInteger( res ) ) goto ErrorCall; 37 38 int ires = DaoValue_TryGetInteger( res ); 39 40 // Release the process object: 41 DaoProcess_Release( vmspace, proc ); 42 43 return ires; 44 45 CallDefault: 46 if( proc ) DaoProcess_Release( vmspace, proc ); 47 return ClassTwo::VirtualMethod( i, f ); 48 ErrorCall: 49 DaoProcess_Release( vmspace, proc ); 50 // Do something; 51 return 0; 52 } Now we will define a function that can be set to the GetGCFields field of the type information structure of ClassTwo. 1 static void Dao_ClassTwo_GetGCFields( void *self0, 2 DArray *values, DArray *arrays, DArray *maps, int remove ) 3 { 4 Dao_ClassTwo *self = (Dao_ClassTwo*) self0; 5 if( self->dao_cdata == NULL ) return; 6 DArray_Append( values, self->dao_cdata ); 7 if( removeĀ ){ 8 // If this object cannot be deallocated because some other C/C++ objects 9 // are using it, here is the right place to set the wrapped pointer (namely 10 // this object) of dao_cdata to NULL, so that no deallocation will be 11 // invoked for this object. 12 // 13 // For example, in some GUI library, if a widget has a parent, the parent 14 // will be responsible for deleting the widget, so something like the 15 // following can be used to support this: 16 // if( self->parent() ) DaoCdata_SetData( self->dao_cdata, NULL ); 17 18 // To avoid the deallocator of Dao_ClassTwo being called by the GC: 19 self->dao_cdata = NULL; 20 } 21 } The remaining part for wrapping ClassTwo should be something like, 1 static void dao_ClassTwo_ClassTwo( DaoProcess *proc, DaoValue *p[], int n ) 2 { 3 Dao_ClassTwo *self = new Dao_ClassTwo(); 4 DaoProcess_PutValue( proc, (DaoValue*) self->dao_cdata ); 5 } 6 static DaoFuncItem ClassTwoMethods[] = 7 { 8 { dao_ClassTwo_ClassTwo, "ClassTwo()" }, 9 { NULL, NULL } 10 }; 11 static void Dao_ClassTwo_Delete( void *self ) 12 { 13 delete (Dao_ClassTwo*) self; 14 } 15 static void* Dao_ClassTwo_Cast_ClassOne( void *data, int down ) 16 { 17 if( down ) return static_cast<ClassTwo*>((ClassOne*)data); 18 return dynamic_cast<ClassOne*>((ClassTwo*)data); 19 } 20 // The type information structure for ClassTwo: 21 static DaoTypeBase ClassTwo_Typer = 22 { 23 "ClassTwo", NULL, NULL, ClassTwoMethods, 24 { & ClassOne_Typer, NULL }, 25 { Dao_ClassTwo_Cast_ClassOne, NULL }, 26 Dao_ClassTwo_Delete, NULL 27 }; 28 // The Dao type structure for ClassTwo: 29 DaoType *dao_type_ClassTwo = NULL; 30 31 int DaoOnLoad( DaoVmSpace *vmSpace, DaoNamespace *nspace ) 32 { 33 ... 34 // Wrap ClassTwo as an opaque C/C++ type: 35 dao_type_ClassTwo = DaoNamespace_WrapType( nspace, & ClassTwo_Typer, 1 ); 36 return 0; 37 } 4 Data Conversion between Dao and C/C++ Dao provides various C interface functions to make data conversion between Dao and C/C++ simple. For simple data types, one can use the one of the following functions to convert Dao values to C values, 1 daoint DaoValue_TryGetInteger( DaoValue *self ); 2 float DaoValue_TryGetFloat( DaoValue *self ); 3 double DaoValue_TryGetDouble( DaoValue *self ); 4 complex16 DaoValue_TryGetComplex( DaoValue *self ); 5 char* DaoValue_TryGetMBString( DaoValue *self ); 6 wchar_t* DaoValue_TryGetWCString( DaoValue *self ); 7 DString* DaoValue_TryGetString( DaoValue *self ); 8 int DaoValue_TryGetEnum( DaoValue *self ); 9 10 void* DaoValue_TryGetCdata( DaoValue *self ); 11 void** DaoValue_TryGetCdata2( DaoValue *self ); 12 void* DaoValue_TryCastCdata( DaoValue *self, DaoType *totype ); If the DaoValue object is of the requested type, the correct data will be returned, otherwise zero or a null pointer is return. The last three functions are execlusively for wrapped C/C++ types, we come to this later. For other data types, you may need to cast DaoValue objects to proper types, and then use proper methods to retrieve C data values. There are two ways to cast from DaoValue to other types, one is to use DaoValue_Type() to check its type and than do C casting, the other is to use one of the DaoValue_CastXXX() series of methods. For example, the following are the two ways to cast value from DaoValue to DaoTuple, 1 DaoTuple *tup1 = DaoValue_Type( value ) == DAO_TUPLE ? (DaoTuple*) value : NULL; 2 DaoTuple *tup2 = DaoValue_CastTuple( value ); DaoValue_CastXXX() methods will return NULL, if the value is not the correct type.