Merging QofBook structures
[Query Object Framework]


Detailed Description

Collision handling principles.

  1. Always check for a GUID first and compare. qofbookmerge only accepts valid QofBook data and therefore ALL objects in the import book will include valid GUID's.
  2. If the original import data did not contain a GUID (e.g. an external non-GnuCash source) the GUID values will have been created during the import and will not match any existing GUID's in the target book so objects that do not have a GUID match cannot be assumed to be MERGE_NEW - parameter values must be checked.
  3. If import contains data from closed books, store the data from the closed books in the current book as active. i.e. re-open the books.

More information is at http://code.neil.williamsleesmill.me.uk/

Each foreach function uses g_return_if_fail checks to protect the target book. If any essential data is missing, the loop returns without changing the target book. Note that this will not set or return an error value. However, g_return is only used for critical errors that arise from programming errors, not for invalid import data which should be cleaned up before creating the import QofBook.

Only qof_book_merge_update_result and qof_book_merge_commit return any error values to the calling process. qof_book_merge_init returns a pointer to the QofBookMergeData struct - the calling process needs to make sure this is non-NULL to know that the Init has been successful.

Todo:
Prior to libqof2 rationalise internal variable names.


Files

file  qofbookmerge.h
 API for merging two QofBook structures with collision handling.

Data Structures

struct  QofBookMergeRule
 One rule per entity, built into a single GList for the entire merge. More...
struct  QofBookMergeData
 mergeData contains the essential context data for any merge. More...

qof_book_merge API

typedef void(*) QofBookMergeRuleForeachCB (QofBookMergeData *, QofBookMergeRule *, guint)
 Definition of the dialogue control callback routine.
QofBookMergeDataqof_book_merge_init (QofBook *importBook, QofBook *targetBook)
 Initialise the QofBookMerge process.
void qof_book_merge_rule_foreach (QofBookMergeData *mergeData, QofBookMergeRuleForeachCB callback, QofBookMergeResult mergeResult)
 Dialogue Control Callback.
gchar * qof_book_merge_param_as_string (QofParam *qtparam, QofEntity *qtEnt)
 provides easy string access to parameter data for dialogue use
QofBookMergeDataqof_book_merge_update_result (QofBookMergeData *mergeData, QofBookMergeResult tag)
 called by dialogue callback to set the result of user intervention
gint qof_book_merge_commit (QofBookMergeData *mergeData)
 Commits the import data to the target book.
void qof_book_merge_abort (QofBookMergeData *mergeData)
 Abort the merge and free all memory allocated by the merge.

Enumerations

enum  QofBookMergeResult {
  MERGE_UNDEF, MERGE_ABSOLUTE, MERGE_NEW, MERGE_REPORT,
  MERGE_DUPLICATE, MERGE_UPDATE, MERGE_INVALID
}
 Results of collisions and user resolution. More...


Typedef Documentation

typedef void(*) QofBookMergeRuleForeachCB(QofBookMergeData *, QofBookMergeRule *, guint)

Definition of the dialogue control callback routine.

All MERGE_REPORT rules must be offered for user intervention using this template.
Commit will fail if any rules are still tagged as MERGE_REPORT.

Calling processes are free to also offer MERGE_NEW, MERGE_UPDATE, MERGE_DUPLICATE and MERGE_ABSOLUTE for user intervention. Attempting to query MERGE_INVALID rules will cause an error.

For an example, consider test_rule_loop, declared as:

void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder);
void test_rule_loop(QofBookMergeData *mergeData, QofBookMergeRule *rule, guint remainder)
{
g_return_if_fail(rule != NULL);
g_return_if_fail(mergeData != NULL); printf("Rule Result %s", rule->mergeType);
qof_book_merge_update_result(mergeData, rule, MERGE_UPDATE);
}

The dialogue is free to call qof_book_merge_update_result in the loop or at the end as long as the link between the rule and the result is maintained, e.g. by using a GHashTable.
The parameters are:

If the dialogue sets any rule result to MERGE_INVALID, the import will abort when qof_book_merge_commit is called. It is the responsibility of the calling function to handle the error code from qof_book_merge_commit, close the dialogue and return. The merge routines in these files will already have halted the merge operation and freed any memory allocated to merge structures before returning the error code. There is no need for the dialogue process to report back to QofBookMerge in this situation.

Definition at line 321 of file qofbookmerge.h.


Enumeration Type Documentation

enum QofBookMergeResult

Results of collisions and user resolution.

All rules are initialised as MERGE_UNDEF. Once the comparison is complete, each object within the import will be updated.

MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE can be reported to the user along with all MERGE_REPORT objects for confirmation. It may be useful later to allow MERGE_ABSOLUTE, MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE to not be reported, if the user sets a preferences option for each result. (Always accept new items: Y/N default NO, ignores all MERGE_NEW if set to Y etc.) This option would not require any changes to qofbookmerge.

MERGE_NEW, MERGE_DUPLICATE and MERGE_UPDATE are only actioned after conflicts are resolved by the user using a dialog and all MERGE_REPORT objects are re-assigned to one of MERGE_NEW, MERGE_DUPLICATE or MERGE_UPDATE. There is no automatic merge, even if no entities are tagged as MERGE_REPORT, the calling process must still check for REPORT items using qof_book_merge_rule_foreach and call qof_book_merge_commit.

MERGE_INVALID data should be rare and allows for user-abort - the imported file/source may be corrupted and the prescence of invalid data should raise concerns that the rest of the data may be corrupted, damaged or otherwise altered. If any entity is tagged as MERGE_INVALID, the merge operation will abort and leave the target book completely unchanged.

MERGE_ABSOLUTE is only used for a complete match. The import object contains the same data in the same parameters with no omissions or amendments. If any data is missing, amended or added, the data is labelled MERGE_UPDATE.

Every piece of data has a corresponding result. Only when the count of items labelled MERGE_REPORT is equal to zero are MERGE_NEW and MERGE_UPDATE items added to the existing book.
MERGE_DUPLICATE items are silently ignored. Aborting the dialogue/process (by the user or in a program crash) at any point before the final commit leaves the existing book completely untouched.

Enumerator:
MERGE_UNDEF  default value before comparison is made.
MERGE_ABSOLUTE  GUID exact match, no new data - ignore
MERGE_NEW  import object does not exist in the target book - add
MERGE_REPORT  import object needs user intervention - report
MERGE_DUPLICATE  import object with different GUID exactly matches existing GUID - ignore
MERGE_UPDATE  import object matches an existing entity but includes new or modified parameter data - update
MERGE_INVALID  import object didn't match registered object or parameter types or user decided to abort - abort

Definition at line 125 of file qofbookmerge.h.

00126 {
00127     MERGE_UNDEF,     
00128     MERGE_ABSOLUTE,  
00129     MERGE_NEW,       
00131     MERGE_REPORT,    
00132     MERGE_DUPLICATE, 
00134     MERGE_UPDATE,    
00136     MERGE_INVALID    
00138 } QofBookMergeResult;


Function Documentation

void qof_book_merge_abort ( QofBookMergeData mergeData  ) 

Abort the merge and free all memory allocated by the merge.

Sometimes, setting MERGE_INVALID is insufficient: e.g. if the user aborts the merge from outside the functions dealing with the merge ruleset. This function causes an immediate abort - the calling process must start again at Init if a new merge is required.

Definition at line 1024 of file qofbookmerge.c.

01031     {
01032         param_string = qtparam->param_getfcn (qtEnt, qtparam);
01033         if (param_string == NULL)
01034             param_string = "";
01035         return param_string;
01036     }
01037     if (safe_strcmp (paramType, QOF_TYPE_TIME) == 0)
01038     {
01039         QofDate *qd;
01040 
01041         param_qt = qof_time_copy (
01042             qtparam->param_getfcn (qtEnt, qtparam));
01043         if (!param_qt)
01044             return NULL;
01045         qd = qof_date_from_qtime (param_qt);
01046         param_string = qof_date_print (qd, QOF_DATE_FORMAT_UTC);
01047         qof_date_free (qd);
01048         qof_time_free (param_qt);
01049         return param_string;
01050     }

gint qof_book_merge_commit ( QofBookMergeData mergeData  ) 

Commits the import data to the target book.

The last function in the API and the final part of any QofBookMerge operation.

qof_book_merge_commit will abort the entire merge operation if any rule is set to MERGE_INVALID. It is the responsibility of the calling function to handle the error code from qof_book_mergeCommit, close the dialogue and return. qof_book_merge_commit will already have halted the merge operation and freed any memory allocated to all merge structures before returning the error code. There is no way for the dialogue process to report back to qof_book_merge in this situation.

qof_book_merge_commit checks for any entities still tagged as MERGE_REPORT and then proceeds to import all entities tagged as MERGE_UPDATE or MERGE_NEW into the target book.
This final process cannot be UNDONE!

Parameters:
mergeData the merge context, QofBookMergeData*
Returns:
  • -2 if any rules are tagged as MERGE_INVALID
    • mergeData will have been g_free'd).
    • note that this will be before any operations are done on the target QofBook.
  • -1 if mergeData is invalid or no merge has been initialised with qof_book_merge_init - the calling process must check the value of mergeData
  • +1 if some entities are still tagged as MERGE_REPORT - use qof_book_merge_update_rule and try again (mergeData is retained).
  • 0 on success - mergeData will have been freed.

Definition at line 1222 of file qofbookmerge.c.

01222 {
01223     struct QofBookMergeRuleIterate qiter;
01224     QofBookMergeRule *currentRule;
01225     GList *matching_rules, *node;
01226 
01227     g_return_if_fail (cb != NULL);
01228     g_return_if_fail (mergeData != NULL);
01229     currentRule = mergeData->currentRule;
01230     g_return_if_fail (mergeResult > 0);
01231     g_return_if_fail (mergeResult != MERGE_INVALID);
01232     g_return_if_fail (mergeData->abort == FALSE);
01233     qiter.fcn = cb;
01234     qiter.data = mergeData;
01235     matching_rules = NULL;
01236     for (node = mergeData->mergeList; node != NULL; node = node->next)
01237     {
01238         currentRule = node->data;
01239         if (currentRule->mergeResult == mergeResult)
01240             matching_rules = g_list_prepend (matching_rules, 
01241                 currentRule);
01242     }
01243     qiter.remainder = g_list_length (matching_rules);
01244     g_list_foreach (matching_rules, qof_book_merge_rule_cb, &qiter);
01245     g_list_free (matching_rules);
01246 }
01247 
01248 /* ============================================================== */
01249 /* ============================================================== */

QofBookMergeData* qof_book_merge_init ( QofBook importBook,
QofBook targetBook 
)

Initialise the QofBookMerge process.

First function of the QofBookMerge API. Every merge must begin with init.

Requires the book to import (QofBook *) and the book to receive the import, the target book (QofBook *). Returns a pointer to QofBookMergeData which must be checked for a NULL before continuing.
Process:

  1. Invoke the callback qof_book_merge_foreach_type on every registered object class definition.
  2. Callback obtains the registered parameter list for each object type. This provides run time access to all registered objects and all object parameters without any changes to QofBookMerge - no registered object or parameter is omitted from any merge operation.
  3. Use qof_object_foreach to invoke the callback qof_book_merge_foreach, one object at a time on every instance stored in mergeBook. This is the first point where real data from the import book is accessed.
  4. qof_book_merge_foreach obtains the GUID for the object from the import book and runs the first check on the original book, checking for any exact GUID match. With the full parameter list, the rules for this object can be created. If there is a GUID match, the data in each parameter of the import object is compared with the same semantic object in the original book. If there is no GUID in the import object or no GUID match with the original book, the original book is searched to find a parameter match - checking for a MERGE_DUPLICATE result.
  5. qof_book_merge_compare sets the QofBookMergeResult of the comparison.
  6. Inserts the completed rule into QofBookMergeData::mergeList GSList.

Returns:
NULL in case of error, otherwise a QofBookMergeData* metadata context.

Definition at line 987 of file qofbookmerge.c.

01000                           : 2005-01-01T10:55:23Z
01001 If you change QOF_UTC_DATE_FORMAT, change 
01002 backend/file/qsf-xml.c : qsf_entity_foreach to
01003 reformat to QSF_XSD_TIME or the QSF XML will
01004 FAIL the schema validation and QSF exports will become invalid.
01005 
01006 The QOF_TYPE_BOOLEAN is lowercase for the same reason.
01007 
01008 \todo deprecate and replace with
01009 gchar* qof_instance_param_as_string(const QofParam*, QofInstance*);
01010 and then add
01011 gchar* qof_class_get_param_as_string(QofIdTypeConst, QofInstance*); ?
01012 */
01013 gchar *
01014 qof_book_merge_param_as_string (QofParam * qtparam, QofEntity * qtEnt)
01015 {
01016     gchar *param_string;
01017     gchar param_sa[GUID_ENCODING_LENGTH + 1];
01018     QofType paramType;
01019     const GUID *param_guid;
01020     QofTime *param_qt;
01021     gnc_numeric param_numeric, (*numeric_getter) (QofEntity *, QofParam *);

gchar* qof_book_merge_param_as_string ( QofParam qtparam,
QofEntity qtEnt 
)

provides easy string access to parameter data for dialogue use

Uses the param_getfcn to retrieve the parameter value as a string, suitable for display in dialogues and user intervention output. Within a QofBookMerge context, only the parameters used in the merge are available, i.e. parameters where both param_getfcn and param_setfcn are not NULL.

Note that the object type description (a full text version of the object name) is also available to the dialogue as QofBookMergeRule::mergeLabel.

This allows the dialog to display the description of the object and all parameter data.

Definition at line 1068 of file qofbookmerge.c.

01070     {
01071         numeric_getter =
01072             (gnc_numeric (*)(QofEntity *,
01073                 QofParam *)) qtparam->param_getfcn;
01074         param_numeric = numeric_getter (qtEnt, qtparam);
01075         param_string = g_strdup (gnc_numeric_to_string (param_numeric));
01076         return param_string;
01077     }
01078     if (safe_strcmp (paramType, QOF_TYPE_GUID) == 0)
01079     {
01080         param_guid = qtparam->param_getfcn (qtEnt, qtparam);
01081         guid_to_string_buff (param_guid, param_sa);
01082         param_string = g_strdup (param_sa);
01083         return param_string;
01084     }
01085     if (safe_strcmp (paramType, QOF_TYPE_INT32) == 0)
01086     {
01087         int32_getter =
01088             (gint32 (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01089         param_i32 = int32_getter (qtEnt, qtparam);
01090         param_string = g_strdup_printf ("%d", param_i32);
01091         return param_string;
01092     }
01093     if (safe_strcmp (paramType, QOF_TYPE_INT64) == 0)
01094     {
01095         int64_getter =
01096             (gint64 (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01097         param_i64 = int64_getter (qtEnt, qtparam);
01098         param_string = g_strdup_printf ("%" G_GINT64_FORMAT, param_i64);
01099         return param_string;
01100     }
01101     if (safe_strcmp (paramType, QOF_TYPE_DOUBLE) == 0)
01102     {
01103         double_getter =
01104             (double (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01105         param_double = double_getter (qtEnt, qtparam);
01106         param_string = g_strdup_printf ("%f", param_double);
01107         return param_string;
01108     }
01109     if (safe_strcmp (paramType, QOF_TYPE_BOOLEAN) == 0)
01110     {
01111         boolean_getter =
01112             (gboolean (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01113         param_boolean = boolean_getter (qtEnt, qtparam);
01114         /* Boolean values need to be lowercase for QSF validation. */
01115         if (param_boolean == TRUE)
01116             param_string = g_strdup ("true");
01117         else
01118             param_string = g_strdup ("false");
01119         return param_string;
01120     }
01121     /* "kvp" contains repeating values, cannot be a single string for the frame. */
01122     if (safe_strcmp (paramType, QOF_TYPE_KVP) == 0)
01123         return param_string;
01124     if (safe_strcmp (paramType, QOF_TYPE_CHAR) == 0)
01125     {
01126         char_getter =
01127             (gchar (*)(QofEntity *, QofParam *)) qtparam->param_getfcn;
01128         param_char = char_getter (qtEnt, qtparam);
01129         param_string = g_strdup_printf ("%c", param_char);
01130         return param_string;
01131     }
01132     return NULL;
01133 }
01134 
01135 QofBookMergeData *
01136 qof_book_merge_update_result (QofBookMergeData * mergeData,
01137     QofBookMergeResult tag)
01138 {
01139     QofBookMergeRule *resolved;
01140 
01141     g_return_val_if_fail ((mergeData != NULL), NULL);
01142     g_return_val_if_fail ((tag > 0), NULL);
01143     g_return_val_if_fail ((tag != MERGE_REPORT), NULL);
01144     resolved = mergeData->currentRule;
01145     g_return_val_if_fail ((resolved != NULL), NULL);
01146     if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_DUPLICATE))
01147         tag = MERGE_ABSOLUTE;
01148     if ((resolved->mergeAbsolute == TRUE) && (tag == MERGE_NEW))
01149         tag = MERGE_UPDATE;
01150     if ((resolved->mergeAbsolute == FALSE) && (tag == MERGE_ABSOLUTE))
01151         tag = MERGE_DUPLICATE;
01152     if ((resolved->mergeResult == MERGE_NEW) && (tag == MERGE_UPDATE))
01153         tag = MERGE_NEW;
01154     if (resolved->updated == FALSE)
01155         resolved->mergeResult = tag;
01156     resolved->updated = TRUE;
01157     if (tag >= MERGE_INVALID)
01158     {
01159         mergeData->abort = TRUE;
01160         mergeData->currentRule = resolved;
01161         return NULL;
01162     }
01163     mergeData->currentRule = resolved;
01164     return mergeData;
01165 }
01166 
01167 gint
01168 qof_book_merge_commit (QofBookMergeData * mergeData)
01169 {
01170     QofBookMergeRule *currentRule;
01171     GList *check, *node;
01172 
01173     g_return_val_if_fail (mergeData != NULL, -1);
01174     g_return_val_if_fail (mergeData->mergeList != NULL, -1);
01175     g_return_val_if_fail (mergeData->targetBook != NULL, -1);
01176     if (mergeData->abort == TRUE)
01177         return -1;
01178     check = g_list_copy (mergeData->mergeList);
01179     g_return_val_if_fail (check != NULL, -1);
01180     for (node = check; node != NULL; node = node->next)
01181     {
01182         currentRule = node->data;
01183         if (currentRule->mergeResult == MERGE_INVALID)
01184         {
01185             qof_book_merge_abort (mergeData);
01186             g_list_free (check);
01187             return (-2);

void qof_book_merge_rule_foreach ( QofBookMergeData mergeData,
QofBookMergeRuleForeachCB  callback,
QofBookMergeResult  mergeResult 
)

Dialogue Control Callback.

This function is designed to be used to iterate over all rules tagged with a specific QofBookMergeResult value.

Parameters:
callback external loop of type QofBookMergeRuleForeachCB
mergeResult QofBookMergeResult value to look up.
mergeData QofBookMergeData merge context.
Note : MERGE_NEW causes a new entity to be created in the target book at commit which is then assigned as the targetEnt of that rule. If mergeResult == MERGE_NEW, the rules returned by qof_book_merge_rule_foreach will have a NULL set for the targetEnt. This is because commit has not yet been called and no changes can be made to the target book. The calling process must handle the NULL targetEnt and NOT call any param_getfcn routines for the target entity. The import entity is available for display.

Uses qof_book_get_collection with the QofBookMergeRule::mergeType object type to return a collection of QofEntity entities from either the QofBookMergeData::mergeBook or QofBookMergeData::targetBook. Then uses qof_collection_lookup_entity to lookup the QofBookMergeRule::importEnt and again the qof_book_mergeRule::targetEnt to return the two specific entities.

Definition at line 1274 of file qofbookmerge.c.

QofBookMergeData* qof_book_merge_update_result ( QofBookMergeData mergeData,
QofBookMergeResult  tag 
)

called by dialogue callback to set the result of user intervention

Set any rule result to MERGE_INVALID to abort the import when qof_book_merge_commit is called, without changing the target book.

The calling process should make it absolutely clear that a merge operation cannot be undone and that a backup copy should always be available before a merge is initialised.

Recommended method: Only offer three options to the user per rule:

  1. Allow import data to be merged into target data
    • change MERGE_REPORT to MERGE_UPDATE
  2. Allow import data without an exact match to be added as new
    • change MERGE_REPORT to MERGE_NEW IF mergeAbsolute = FALSE
  3. Ignore import data and leave target data unchanged
    • change MERGE_REPORT to MERGE_ABSOLUTE or MERGE_DUPLICATE

Handle the required result changes in code: Check the value of qof_book_mergeRule::mergeAbsolute and use these principles:

To ignore entities tagged as:

To merge entities that are not pre-set to MERGE_NEW, set MERGE_UPDATE.
Attempting to merge an entity when the pre-set value was MERGE_NEW will force a change back to MERGE_NEW because no suitable target exists for the merge.

To add entities, check mergeAbsolute is FALSE and set MERGE_NEW.
An entity only be added if mergeAbsolute is FALSE. Attempting to add an entity when mergeAbsolute is TRUE will always force a MERGE_UPDATE.

It is not possible to update the same rule more than once.

  1. MERGE_NEW is reserved for new objects and is only pre-set if all parameters, including GUID, have already failed to match any relevant object. qof_book_mergeCommit will create new entities for all rules tagged as MERGE_NEW.
    • if mergeAbsolute is TRUE and the user wants to import the data, requests to set MERGE_NEW will be forced to MERGE_UPDATE because an entity with that GUID already exists in the target book.
    • if MERGE_NEW is pre-set, requests to change to MERGE_UPDATE will be ignored because a new entity is needed.
  2. MERGE_UPDATE is reserved for existing objects - qof_book_mergeCommit will require a matching entity to update and will force a change to back to MERGE_NEW if none is known to exist, using the principle above.
  3. MERGE_INVALID will cause an abort of the merge process.
  4. MERGE_UNDEF and MERGE_REPORT cannot be set - the entity result will be unchanged.
  5. MERGE_DUPLICATE and MERGE_ABSOLUTE are handled identically but are semantically different - QofBookMergeRule::mergeAbsolute is used to dictate which to set:
    • if mergeAbsolute is TRUE but MERGE_DUPLICATE is requested, force a change to MERGE_ABSOLUTE.
    • if mergeAbsolute is FALSE but MERGE_ABSOLUTE is requested, force a change to MERGE_DUPLICATE.

qof_book_merge_commit only commits entities tagged with MERGE_NEW and MERGE_UPDATE results.
Entities tagged with MERGE_ABSOLUTE and MERGE_DUPLICATE results are ignored.

The calling process must check the return value and call qof_book_merge_abort(mergeData) if non-zero.

Parameters:
mergeData the merge context, QofBookMergeData*
tag the result to attempt to set, QofBookMergeResult
Returns:
-1 if supplied parameters are invalid or NULL, 0 on success.

Definition at line 1190 of file qofbookmerge.c.

01190         {
01191             g_list_free (check);
01192             return 1;
01193         }
01194     }
01195     g_list_free (check);
01196     qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
01197         MERGE_NEW, mergeData);
01198     qof_book_merge_commit_foreach (qof_book_merge_commit_rule_loop,
01199         MERGE_UPDATE, mergeData);
01200     /* Placeholder for QofObject merge_helper_cb - all objects
01201        and all parameters set */
01202     while (mergeData->mergeList != NULL)
01203     {
01204         currentRule = mergeData->mergeList->data;
01205         g_slist_free (currentRule->mergeParam);
01206         g_slist_free (currentRule->linkedEntList);
01207         mergeData->mergeList = g_list_next (mergeData->mergeList);
01208     }
01209     g_list_free (mergeData->mergeList);
01210     g_slist_free (mergeData->mergeObjectParams);
01211     g_slist_free (mergeData->targetList);
01212     if (mergeData->orphan_list != NULL)
01213         g_slist_free (mergeData->orphan_list);
01214     g_hash_table_destroy (mergeData->target_table);
01215     g_free (mergeData);
01216     return 0;
01217 }
01218 
01219 void


Generated on Mon May 21 17:42:23 2007 for QOF by  doxygen 1.5.1