In this post I will explain how I created a very useful reusable Mendix module for the following user story:
As an admin I want to export some of the application’s data, so that I can easily import it in other environments of the same application.
In Mendix, you can create specific export functionality for your entities, using Domain-to-XML-mappings. However, this is a lot of work for and you would need to develop this specifically to the domain model for each application (or even multiple exports within a single application). Another drawback of this approach is that you can only export a simple tree structure. More complex reference structures, like circular references, are not easy to do: to get an idea of what I mean, look at how the export/import functionality of the DBReplication module in the Mendix app store is implemented.
I wanted to create a generic way to export data. Simply selecting the entities and associations of the objects that you want to export should be enough.
The (meta-)domain model
I created the following domain model that will be used to contain the data for exporting and importing. It is basically an abstraction of the Mendix object model:
- An Export contains 0 or more ExportedObjects. An ExportedObject has a name (EntityName) and an identifying number (ObjectID, unique within the Export. The boolean SearchExistingObject is used to indicate if this ExportedObject needs to be created as a new Mendix object on import (if false), or (if true) that the import should try to find an existing Mendix object of this type with the given attributes where IsKey = true.
- An ExportedObject contains 0 or more ExportedAttributes. We save the name and the string representation of the value of the attribute in the ExportedAttributes. The boolean IsKey is only used when searching for existing objects. Even for empty attributes an object is creates (vit AttributeValue=empty), because attirbutes can have a default value, which needs to be cleared when importing, if empty.
- An ExportedObject also can have references to other objects. This is done via 0 or more ExportedAssociation objects. The name of the association is saved in this object. For each reference (the number can be 0 or 1 for references, and 0 or more for reference sets), an ExportedReference objects is created. An ExportedReference saves the ID of the ExportedObject that it refers to (also if the object has SearchExistingObject set to true).
Export definitions
I will allow the user to create “Export Definitions”. An ExportDefinition defines the entities of which the objects (optionally constrained by XPath) and associated objects, will be exported when the user clicks the Export-button. For example, you could have 1 ExportDefinition to export all your database synchronization settings from the DBReplication-module (database connection settings, table mappings, import calls, etc.).
These settings must be persistant, so I’ve created the following domain model (the self-references are only used to to provide dropdowns in the user interface):
For each association an entity is the parent of, the user can set the way referred objects are exported via the attribute ReferenceHandling, which has 3 possibilities:
- Don’t export: The association is disregarded and won’t be in the export.
- Export, create objects on import: The association + the referred objects will be fully exported.
- Export, on import search for existing object with key(s): The association will be exported, but of the referred objects only key attributes will be exported (the user can select which attributes). On import, these key attributes will be used the look up an existing object in the database, outside of the exported XML.
An Export-object is based on a ExportDefinition, so combined, the domain looks like this (the XMLFile entity is used for the xml file upload/downloads):
The user interface
to-do
Creating the XML on export
When the user click the export-button, the following microflow is executed.
The selected ExportDefinition is passed to a Java action that will return an Export-object. After the Export-object has been generated by the Java-magic, it is converted to XML by Mendix, using a Domain-to-XML-mapping. This results in a file, which the user downloads.
|
1 2 3 |
// BEGIN USER CODE return new Exporter(getContext(), exportDefinition).createExport(); // END USER CODE |
For more complex Mendix/Java-combo’s, I always like to use the design pattern where we create a seperate Java class to do the processing. This is the class Exporter, which is called by the above Java action. The method createExport() does the actual processing and creates and returns an Export-MendixObject. Click to see the source code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
package xmlimportexport.implementation; import java.util.HashMap; import java.util.List; import java.util.Map; import xmlimportexport.proxies.Association; import xmlimportexport.proxies.Entity; import xmlimportexport.proxies.Export; import xmlimportexport.proxies.ExportDefinition; import xmlimportexport.proxies.ExportedAssociation; import xmlimportexport.proxies.ExportedAttribute; import xmlimportexport.proxies.ExportedObject; import xmlimportexport.proxies.ExportedReference; import xmlimportexport.proxies.KeyMember; import xmlimportexport.proxies.ReferenceHandling; import com.mendix.core.Core; import com.mendix.core.CoreException; import com.mendix.core.objectmanagement.member.MendixAutoNumber; import com.mendix.core.objectmanagement.member.MendixObjectReference; import com.mendix.core.objectmanagement.member.MendixObjectReferenceSet; import com.mendix.logging.ILogNode; import com.mendix.systemwideinterfaces.core.IContext; import com.mendix.systemwideinterfaces.core.IMendixIdentifier; import com.mendix.systemwideinterfaces.core.IMendixObject; import com.mendix.systemwideinterfaces.core.IMendixObjectMember; public class Exporter { private static final String XML_EXPORT = "XMLExport"; private IContext context; private Map<Long, IMendixObject> exportedObjects, exportedLookupObjects; private IMendixObject export; private ILogNode logger; private ExportDefinition exportDefinition2; public Exporter(IContext c, ExportDefinition exportDefinition2) { context = c; this.exportDefinition2 = exportDefinition2; exportedObjects = new HashMap<Long, IMendixObject>(); //keep track of already exported objects, to avoid exporting objects double exportedLookupObjects = new HashMap<Long, IMendixObject>(); //keep track of already exported objects that won't be created but looked up with key logger = Core.getLogger(XML_EXPORT); } public IMendixObject createExport() throws CoreException { export = Core.instantiate(context, Export.getType()); List<IMendixObject> lstEntities = Core.retrieveByPath(context, exportDefinition2.getMendixObject(), Entity.MemberNames.Entity_ExportDefinition.toString()); for (IMendixObject entity: lstEntities) { String entityName = entity.getValue(context, Entity.MemberNames.CompleteName.name()); String xpath = entity.getValue(context, Entity.MemberNames.XPath.name()); List<IMendixObject> lstAssocs = Core.retrieveByPath(context, entity, Association.MemberNames.Association_Entity.toString()); List<IMendixObject> lstObjects = Core.retrieveXPathQueryEscaped(context, "//"+entityName+(xpath==null?"":xpath)); for (IMendixObject source: lstObjects) { exportObject(source, lstAssocs); } } logger.info("Exported "+exportedObjects.size()+" full objects and "+exportedLookupObjects.size()+" objects by key."); return export; } private void exportReferredObject(IMendixObject assoc, IMendixIdentifier refId, String refHandling) throws CoreException { if (refHandling.equals(ReferenceHandling.Create_object.name())) { if (!exportedLookupObjects.containsKey(refId.toLong())) { //remove from the lookup list: the object will be fully exported instead exportedLookupObjects.remove(refId.toLong()); } if (!exportedObjects.containsKey(refId.toLong())) { //avoid double exporting of objects IMendixObject o = Core.retrieveId(context, refId); exportObject(o, null); } } else if (refHandling.equals(ReferenceHandling.Search_for_object.name())) { if (!exportedObjects.containsKey(refId.toLong()) && !exportedLookupObjects.containsKey(refId.toLong())) { // if already exported then skip IMendixObject o = Core.retrieveId(context, refId); List<IMendixObject> lstKeyMembers = Core.retrieveByPath(context, assoc, KeyMember.MemberNames.KeyMember_Association.toString()); if (lstKeyMembers.size()==0) throw new CoreException("Exporting association "+assoc.getValue(context, Association.MemberNames.CompleteName.toString())+" set to 'search for object' but no key attributes defined."); //create ExportedObject object IMendixObject exportedObject = Core.instantiate(context, ExportedObject.getType()); exportedObject.setValue(context, ExportedObject.MemberNames.ExportedObject_Export.toString(), export.getId()); exportedObject.setValue(context, ExportedObject.MemberNames.EntityName.toString(), o.getType()); exportedObject.setValue(context, ExportedObject.MemberNames.SearchExistingObject.toString(), true); exportedObject.setValue(context, ExportedObject.MemberNames.ObjectID.toString(), o.getId().toLong()); exportedLookupObjects.put(o.getId().toLong(), exportedObject); //create ExportedAttribute objects for all key members for (IMendixObject keymember: lstKeyMembers) { KeyMember km = KeyMember.initialize(context, keymember); IMendixObject exportedAttribute = Core.instantiate(context, ExportedAttribute.getType()); exportedAttribute.setValue(context, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString(), exportedObject.getId()); exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeName.toString(), km.getAttributeName()); //get the value of the key member on the actual object IMendixObjectMember<?> m = o.getMember(context, km.getAttributeName()); Object value = m.getValue(context); if (value != null) //export empty attributes, but leave the value empty exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeValue.toString(), m.parseValueToString(context)); } } } } private void exportObject(IMendixObject source, List<IMendixObject> lstAssocs) throws CoreException { IMendixObject exportedObject = Core.instantiate(context, ExportedObject.getType()); exportedObject.setValue(context, ExportedObject.MemberNames.ExportedObject_Export.toString(), export.getId()); exportedObject.setValue(context, ExportedObject.MemberNames.EntityName.toString(), source.getType()); exportedObject.setValue(context, ExportedObject.MemberNames.ObjectID.toString(), source.getId().toLong()); exportedObjects.put(source.getId().toLong(), exportedObject); Map<String, ? extends IMendixObjectMember<?>> members = source.getMembers(context); for(String key : members.keySet()) { IMendixObjectMember<?> m = members.get(key); if (m.isVirtual() || (m instanceof MendixAutoNumber)) continue; Object value = m.getValue(context); if ((m instanceof MendixObjectReference) || (m instanceof MendixObjectReferenceSet)) { if (value != null && lstAssocs != null) { //skip empty associations for (IMendixObject assoc: lstAssocs) { if (assoc.getValue(context, Association.MemberNames.CompleteName.toString()).equals(m.getName())) { String refHandling = assoc.getValue(context, Association.MemberNames.ReferenceHandling.toString()); if (refHandling.equals(ReferenceHandling.Create_object.name()) || refHandling.equals(ReferenceHandling.Search_for_object.name())) { // create an ExportAssociation object for the exported association IMendixObject exportedAssociation = Core.instantiate(context, ExportedAssociation.getType()); exportedAssociation.setValue(context, ExportedAssociation.MemberNames.ExportedAssociation_ExportedObject.toString(), exportedObject.getId()); exportedAssociation.setValue(context, ExportedAssociation.MemberNames.CompleteName.toString(), m.getName()); // create ExportedReference objects for each reference if (m instanceof MendixObjectReference) { IMendixIdentifier refId = ((MendixObjectReference)m).getValue(context); IMendixObject exportedReference = Core.instantiate(context, ExportedReference.getType()); exportedReference.setValue(context, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString(), exportedAssociation.getId()); exportedReference.setValue(context, ExportedReference.MemberNames.ReferredObjectID.toString(), refId.toLong()); exportReferredObject(assoc, refId, refHandling); } else if (m instanceof MendixObjectReferenceSet) { MendixObjectReferenceSet rs = (MendixObjectReferenceSet) m; for(IMendixIdentifier refId : rs.getValue(context)) { IMendixObject exportedReference = Core.instantiate(context, ExportedReference.getType()); exportedReference.setValue(context, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString(), exportedAssociation.getId()); exportedReference.setValue(context, ExportedReference.MemberNames.ReferredObjectID.toString(), refId.toLong()); exportReferredObject(assoc, refId, refHandling); } } } } } } } else { //attributes IMendixObject exportedAttribute = Core.instantiate(context, ExportedAttribute.getType()); exportedAttribute.setValue(context, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString(), exportedObject.getId()); exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeName.toString(), m.getName()); if (value != null) //export empty attributes, but leave the value empty exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeValue.toString(), m.parseValueToString(context)); } } } } |
Processing the XML on import
When an XML-file is uploaded, the user does not need to have the ExportDefinition in his database, which is a great advantage. This way, you can import your data on any environment, which has the same Mendix model as the one where the data was exported. After selecting an XML-file, the following microflow is executed when the user clicks ‘Upload’:
First, the XML file is translated to an Export object, including all related objects, using standard Mendix functionality, with this XML-to-domain mapping:
Next, the Export object is given to a Java action, which creates all the objects and sets links between them:
|
1 2 3 |
// BEGIN USER CODE return new Importer(getContext(), export).handleImport(); // END USER CODE |
The method handleImport() of the java class Importer does all the work here. Its source is displayed below (click to view):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
package xmlimportexport.implementation; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import xmlimportexport.proxies.Export; import xmlimportexport.proxies.ExportedAssociation; import xmlimportexport.proxies.ExportedAttribute; import xmlimportexport.proxies.ExportedObject; import xmlimportexport.proxies.ExportedReference; import com.mendix.core.Core; import com.mendix.core.CoreException; import com.mendix.core.objectmanagement.member.MendixObjectReference; import com.mendix.core.objectmanagement.member.MendixObjectReferenceSet; import com.mendix.logging.ILogNode; import com.mendix.systemwideinterfaces.core.IContext; import com.mendix.systemwideinterfaces.core.IMendixIdentifier; import com.mendix.systemwideinterfaces.core.IMendixObject; import com.mendix.systemwideinterfaces.core.IMendixObjectMember; import com.mendix.systemwideinterfaces.core.meta.IMetaPrimitive; import com.mendix.systemwideinterfaces.core.meta.IMetaPrimitive.PrimitiveType; public class Importer { private IContext context; private Map<Long, IMendixObject> newObjects, existingObjects; private ILogNode logger; private Export export; public Importer(IContext c, Export export) { context = c; this.export = export; logger = Core.getLogger("XMLImport"); newObjects = new HashMap<Long, IMendixObject>(); //will keep track of the created objects, with their original exported id's existingObjects = new HashMap<Long, IMendixObject>(); //will keep track of the objects that imported objects refer to, with their exported id } public boolean handleImport() throws CoreException { List<IMendixObject> exportedObjects = Core.retrieveByPath(context, export.getMendixObject(), ExportedObject.MemberNames.ExportedObject_Export.toString()); for (IMendixObject exportedObject: exportedObjects) { ExportedObject o = ExportedObject.initialize(context, exportedObject); if (newObjects.containsKey(o.getObjectID())) continue; // skip, because object was already created earlier in the import if (o.getSearchExistingObject() && existingObjects.containsKey(o.getObjectID())) continue; //skip, because referenced object was already searched for and found earlier in the import List<IMendixObject> exportedAttributes = Core.retrieveByPath(context, exportedObject, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString()); logger.debug(exportedAttributes.size()+": " + o.toString()); if (!o.getSearchExistingObject()) { // create new Mendix object IMendixObject actualObject = Core.instantiate(context, o.getEntityName()); newObjects.put(o.getObjectID(), actualObject); for (IMendixObject exportedAttribute: exportedAttributes) { ExportedAttribute a = ExportedAttribute.initialize(context, exportedAttribute); IMendixObjectMember<?> m = actualObject.getMember(context, a.getAttributeName()); actualObject.setValue(context, a.getAttributeName(), m.getValueFromString(a.getAttributeValue())); } } else { //todo: look for existing object using key members if (exportedAttributes.size()==0) throw new CoreException("Object of type "+o.getEntityName()+" is set to search for object, but has no key attributes defined."); String xpath = ""; //build xpath to search for object with the key attributes for (IMendixObject exportedAttribute: exportedAttributes) { ExportedAttribute a = ExportedAttribute.initialize(context, exportedAttribute); IMetaPrimitive mp = Core.getMetaObject(o.getEntityName()).getMetaPrimitive(a.getAttributeName()); String xpathValue = a.getAttributeValue(); if (mp.getType().equals(PrimitiveType.String) || mp.getType().equals(PrimitiveType.Enum) || mp.getType().equals(PrimitiveType.HashString)) xpathValue = "'"+xpathValue+"'"; xpath += "["+a.getAttributeName()+"="+xpathValue+"]"; } List<IMendixObject> foundObjects = Core.retrieveXPathQueryEscaped(context, "//"+o.getEntityName()+xpath); if (foundObjects.size()==0) throw new CoreException("No objects found for xpath-query "+xpath); if (foundObjects.size()>1) logger.warn(foundObjects.size()+" objects found for xpath-query "+xpath+". Taking the first one"); existingObjects.put(o.getObjectID(), foundObjects.get(0)); } } for (IMendixObject exportedObject: exportedObjects) { //now set the associations between the objects for the newly created objects ExportedObject o = ExportedObject.initialize(context, exportedObject); if (!o.getSearchExistingObject()) { IMendixObject actualObject = newObjects.get(o.getObjectID()); List<IMendixObject> exportedAssocs = Core.retrieveByPath(context, exportedObject, ExportedAssociation.MemberNames.ExportedAssociation_ExportedObject.toString()); for (IMendixObject exportedAssoc: exportedAssocs) { ExportedAssociation as = ExportedAssociation.initialize(context, exportedAssoc); List<IMendixObject> exportedReferences = Core.retrieveByPath(context, exportedAssoc, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString()); IMendixObjectMember<?> m = actualObject.getMember(context, as.getCompleteName()); if (m==null) throw new CoreException("Association "+as.getCompleteName()+" not found in entity "+o.getType()); if (m instanceof MendixObjectReference) { if (exportedReferences.size()>0) { MendixObjectReference r = ((MendixObjectReference)m); r.setValue(context, getIdInThisEnvironmentForImportedReference(ExportedReference.initialize(context, exportedReferences.get(0)))); //actualObject.setValue(context, as.getCompleteName(), i); } } else if (m instanceof MendixObjectReferenceSet) { MendixObjectReferenceSet rs = (MendixObjectReferenceSet) m; List<IMendixIdentifier> refIds = new ArrayList<IMendixIdentifier>(); for (IMendixObject exportedReference: exportedReferences) refIds.add(getIdInThisEnvironmentForImportedReference(ExportedReference.initialize(context, exportedReference))); rs.setValue(context, refIds); } } } } logger.info(exportedObjects); // commit created objects logger.info(newObjects.toString()); List<IMendixObject> objectsToCommit = new ArrayList<IMendixObject>(newObjects.values()); Core.commitWithoutEvents(context, objectsToCommit); logger.info("Commited "+objectsToCommit.size()+" objects: "+objectsToCommit.toString()); return true; } private IMendixIdentifier getIdInThisEnvironmentForImportedReference(ExportedReference ref) throws CoreException { long id = ref.getReferredObjectID(); IMendixObject objectInThisEnvironment = newObjects.get(id); //first look for the ObjectID in the list of newly created objects if (objectInThisEnvironment != null) return objectInThisEnvironment.getId(); else { objectInThisEnvironment = existingObjects.get(id); //if not found, look in the list of objects found by key if (objectInThisEnvironment != null) return objectInThisEnvironment.getId(); else throw new CoreException("Object with id "+id+" not found in xml file"); } } } |
The Java code uses Mendix’s core functions to create and fill the objects by name. Objects that need to be search by key are retrieved from the database. Finally, the associations between all objects created and found are set, and the objects are committed. That’s it! Now to try it out: I’ll leave that to you.
This was my final Mendix blog entry. Of course, you can contact me by email if you need help on your Mendix projects.