In this post I will explain how I created a very useful reusable Mendix module for the following user story:
As an admin I want to export some of the application’s data, so that I can easily import it in other environments of the same application.
In Mendix, you can create specific export functionality for your entities, using Domain-to-XML-mappings. However, this is a lot of work for and you would need to develop this specifically to the domain model for each application (or even multiple exports within a single application). Another drawback of this approach is that you can only export a simple tree structure. More complex reference structures, like circular references, are not easy to do: to get an idea of what I mean, look at how the export/import functionality of the DBReplication module in the Mendix app store is implemented.
I wanted to create a generic way to export data. Simply selecting the entities and associations of the objects that you want to export should be enough.
The (meta-)domain model
I created the following domain model that will be used to contain the data for exporting and importing. It is basically an abstraction of the Mendix object model:
- An Export contains 0 or more ExportedObjects. An ExportedObject has a name (EntityName) and an identifying number (ObjectID, unique within the Export. The boolean SearchExistingObject is used to indicate if this ExportedObject needs to be created as a new Mendix object on import (if false), or (if true) that the import should try to find an existing Mendix object of this type with the given attributes where IsKey = true.
- An ExportedObject contains 0 or more ExportedAttributes. We save the name and the string representation of the value of the attribute in the ExportedAttributes. The boolean IsKey is only used when searching for existing objects. Even for empty attributes an object is creates (vit AttributeValue=empty), because attirbutes can have a default value, which needs to be cleared when importing, if empty.
- An ExportedObject also can have references to other objects. This is done via 0 or more ExportedAssociation objects. The name of the association is saved in this object. For each reference (the number can be 0 or 1 for references, and 0 or more for reference sets), an ExportedReference objects is created. An ExportedReference saves the ID of the ExportedObject that it refers to (also if the object has SearchExistingObject set to true).
Export definitions
I will allow the user to create “Export Definitions”. An ExportDefinition defines the entities of which the objects (optionally constrained by XPath) and associated objects, will be exported when the user clicks the Export-button. For example, you could have 1 ExportDefinition to export all your database synchronization settings from the DBReplication-module (database connection settings, table mappings, import calls, etc.).
These settings must be persistant, so I’ve created the following domain model (the self-references are only used to to provide dropdowns in the user interface):
For each association an entity is the parent of, the user can set the way referred objects are exported via the attribute ReferenceHandling, which has 3 possibilities:
- Don’t export: The association is disregarded and won’t be in the export.
- Export, create objects on import: The association + the referred objects will be fully exported.
- Export, on import search for existing object with key(s): The association will be exported, but of the referred objects only key attributes will be exported (the user can select which attributes). On import, these key attributes will be used the look up an existing object in the database, outside of the exported XML.
An Export-object is based on a ExportDefinition, so combined, the domain looks like this (the XMLFile entity is used for the xml file upload/downloads):
The user interface
to-do
Creating the XML on export
When the user click the export-button, the following microflow is executed.
The selected ExportDefinition is passed to a Java action that will return an Export-object. After the Export-object has been generated by the Java-magic, it is converted to XML by Mendix, using a Domain-to-XML-mapping. This results in a file, which the user downloads.
// BEGIN USER CODE return new Exporter(getContext(), exportDefinition).createExport(); // END USER CODE
For more complex Mendix/Java-combo’s, I always like to use the design pattern where we create a seperate Java class to do the processing. This is the class Exporter, which is called by the above Java action. The method createExport() does the actual processing and creates and returns an Export-MendixObject. Click to see the source code:
package xmlimportexport.implementation;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import xmlimportexport.proxies.Association;
import xmlimportexport.proxies.Entity;
import xmlimportexport.proxies.Export;
import xmlimportexport.proxies.ExportDefinition;
import xmlimportexport.proxies.ExportedAssociation;
import xmlimportexport.proxies.ExportedAttribute;
import xmlimportexport.proxies.ExportedObject;
import xmlimportexport.proxies.ExportedReference;
import xmlimportexport.proxies.KeyMember;
import xmlimportexport.proxies.ReferenceHandling;
import com.mendix.core.Core;
import com.mendix.core.CoreException;
import com.mendix.core.objectmanagement.member.MendixAutoNumber;
import com.mendix.core.objectmanagement.member.MendixObjectReference;
import com.mendix.core.objectmanagement.member.MendixObjectReferenceSet;
import com.mendix.logging.ILogNode;
import com.mendix.systemwideinterfaces.core.IContext;
import com.mendix.systemwideinterfaces.core.IMendixIdentifier;
import com.mendix.systemwideinterfaces.core.IMendixObject;
import com.mendix.systemwideinterfaces.core.IMendixObjectMember;
public class Exporter {
private static final String XML_EXPORT = "XMLExport";
private IContext context;
private Map<Long, IMendixObject> exportedObjects, exportedLookupObjects;
private IMendixObject export;
private ILogNode logger;
private ExportDefinition exportDefinition2;
public Exporter(IContext c, ExportDefinition exportDefinition2) {
context = c;
this.exportDefinition2 = exportDefinition2;
exportedObjects = new HashMap<Long, IMendixObject>(); //keep track of already exported objects, to avoid exporting objects double
exportedLookupObjects = new HashMap<Long, IMendixObject>(); //keep track of already exported objects that won't be created but looked up with key
logger = Core.getLogger(XML_EXPORT);
}
public IMendixObject createExport() throws CoreException {
export = Core.instantiate(context, Export.getType());
List<IMendixObject> lstEntities = Core.retrieveByPath(context, exportDefinition2.getMendixObject(), Entity.MemberNames.Entity_ExportDefinition.toString());
for (IMendixObject entity: lstEntities) {
String entityName = entity.getValue(context, Entity.MemberNames.CompleteName.name());
String xpath = entity.getValue(context, Entity.MemberNames.XPath.name());
List<IMendixObject> lstAssocs = Core.retrieveByPath(context, entity, Association.MemberNames.Association_Entity.toString());
List<IMendixObject> lstObjects = Core.retrieveXPathQueryEscaped(context, "//"+entityName+(xpath==null?"":xpath));
for (IMendixObject source: lstObjects) {
exportObject(source, lstAssocs);
}
}
logger.info("Exported "+exportedObjects.size()+" full objects and "+exportedLookupObjects.size()+" objects by key.");
return export;
}
private void exportReferredObject(IMendixObject assoc, IMendixIdentifier refId, String refHandling) throws CoreException {
if (refHandling.equals(ReferenceHandling.Create_object.name())) {
if (!exportedLookupObjects.containsKey(refId.toLong())) {
//remove from the lookup list: the object will be fully exported instead
exportedLookupObjects.remove(refId.toLong());
}
if (!exportedObjects.containsKey(refId.toLong())) { //avoid double exporting of objects
IMendixObject o = Core.retrieveId(context, refId);
exportObject(o, null);
}
}
else if (refHandling.equals(ReferenceHandling.Search_for_object.name())) {
if (!exportedObjects.containsKey(refId.toLong()) && !exportedLookupObjects.containsKey(refId.toLong())) { // if already exported then skip
IMendixObject o = Core.retrieveId(context, refId);
List<IMendixObject> lstKeyMembers = Core.retrieveByPath(context, assoc, KeyMember.MemberNames.KeyMember_Association.toString());
if (lstKeyMembers.size()==0)
throw new CoreException("Exporting association "+assoc.getValue(context, Association.MemberNames.CompleteName.toString())+" set to 'search for object' but no key attributes defined.");
//create ExportedObject object
IMendixObject exportedObject = Core.instantiate(context, ExportedObject.getType());
exportedObject.setValue(context, ExportedObject.MemberNames.ExportedObject_Export.toString(), export.getId());
exportedObject.setValue(context, ExportedObject.MemberNames.EntityName.toString(), o.getType());
exportedObject.setValue(context, ExportedObject.MemberNames.SearchExistingObject.toString(), true);
exportedObject.setValue(context, ExportedObject.MemberNames.ObjectID.toString(), o.getId().toLong());
exportedLookupObjects.put(o.getId().toLong(), exportedObject);
//create ExportedAttribute objects for all key members
for (IMendixObject keymember: lstKeyMembers) {
KeyMember km = KeyMember.initialize(context, keymember);
IMendixObject exportedAttribute = Core.instantiate(context, ExportedAttribute.getType());
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString(), exportedObject.getId());
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeName.toString(), km.getAttributeName());
//get the value of the key member on the actual object
IMendixObjectMember<?> m = o.getMember(context, km.getAttributeName());
Object value = m.getValue(context);
if (value != null) //export empty attributes, but leave the value empty
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeValue.toString(), m.parseValueToString(context));
}
}
}
}
private void exportObject(IMendixObject source, List<IMendixObject> lstAssocs) throws CoreException {
IMendixObject exportedObject = Core.instantiate(context, ExportedObject.getType());
exportedObject.setValue(context, ExportedObject.MemberNames.ExportedObject_Export.toString(), export.getId());
exportedObject.setValue(context, ExportedObject.MemberNames.EntityName.toString(), source.getType());
exportedObject.setValue(context, ExportedObject.MemberNames.ObjectID.toString(), source.getId().toLong());
exportedObjects.put(source.getId().toLong(), exportedObject);
Map<String, ? extends IMendixObjectMember<?>> members = source.getMembers(context);
for(String key : members.keySet()) {
IMendixObjectMember<?> m = members.get(key);
if (m.isVirtual() || (m instanceof MendixAutoNumber))
continue;
Object value = m.getValue(context);
if ((m instanceof MendixObjectReference) || (m instanceof MendixObjectReferenceSet)) {
if (value != null && lstAssocs != null) { //skip empty associations
for (IMendixObject assoc: lstAssocs) {
if (assoc.getValue(context, Association.MemberNames.CompleteName.toString()).equals(m.getName())) {
String refHandling = assoc.getValue(context, Association.MemberNames.ReferenceHandling.toString());
if (refHandling.equals(ReferenceHandling.Create_object.name()) ||
refHandling.equals(ReferenceHandling.Search_for_object.name()))
{ // create an ExportAssociation object for the exported association
IMendixObject exportedAssociation = Core.instantiate(context, ExportedAssociation.getType());
exportedAssociation.setValue(context, ExportedAssociation.MemberNames.ExportedAssociation_ExportedObject.toString(), exportedObject.getId());
exportedAssociation.setValue(context, ExportedAssociation.MemberNames.CompleteName.toString(), m.getName());
// create ExportedReference objects for each reference
if (m instanceof MendixObjectReference) {
IMendixIdentifier refId = ((MendixObjectReference)m).getValue(context);
IMendixObject exportedReference = Core.instantiate(context, ExportedReference.getType());
exportedReference.setValue(context, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString(), exportedAssociation.getId());
exportedReference.setValue(context, ExportedReference.MemberNames.ReferredObjectID.toString(), refId.toLong());
exportReferredObject(assoc, refId, refHandling);
}
else if (m instanceof MendixObjectReferenceSet) {
MendixObjectReferenceSet rs = (MendixObjectReferenceSet) m;
for(IMendixIdentifier refId : rs.getValue(context)) {
IMendixObject exportedReference = Core.instantiate(context, ExportedReference.getType());
exportedReference.setValue(context, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString(), exportedAssociation.getId());
exportedReference.setValue(context, ExportedReference.MemberNames.ReferredObjectID.toString(), refId.toLong());
exportReferredObject(assoc, refId, refHandling);
}
}
}
}
}
}
}
else { //attributes
IMendixObject exportedAttribute = Core.instantiate(context, ExportedAttribute.getType());
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString(), exportedObject.getId());
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeName.toString(), m.getName());
if (value != null) //export empty attributes, but leave the value empty
exportedAttribute.setValue(context, ExportedAttribute.MemberNames.AttributeValue.toString(), m.parseValueToString(context));
}
}
}
}
Processing the XML on import
When an XML-file is uploaded, the user does not need to have the ExportDefinition in his database, which is a great advantage. This way, you can import your data on any environment, which has the same Mendix model as the one where the data was exported. After selecting an XML-file, the following microflow is executed when the user clicks ‘Upload’:
First, the XML file is translated to an Export object, including all related objects, using standard Mendix functionality, with this XML-to-domain mapping:
Next, the Export object is given to a Java action, which creates all the objects and sets links between them:
// BEGIN USER CODE return new Importer(getContext(), export).handleImport(); // END USER CODE
The method handleImport() of the java class Importer does all the work here. Its source is displayed below (click to view):
package xmlimportexport.implementation;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import xmlimportexport.proxies.Export;
import xmlimportexport.proxies.ExportedAssociation;
import xmlimportexport.proxies.ExportedAttribute;
import xmlimportexport.proxies.ExportedObject;
import xmlimportexport.proxies.ExportedReference;
import com.mendix.core.Core;
import com.mendix.core.CoreException;
import com.mendix.core.objectmanagement.member.MendixObjectReference;
import com.mendix.core.objectmanagement.member.MendixObjectReferenceSet;
import com.mendix.logging.ILogNode;
import com.mendix.systemwideinterfaces.core.IContext;
import com.mendix.systemwideinterfaces.core.IMendixIdentifier;
import com.mendix.systemwideinterfaces.core.IMendixObject;
import com.mendix.systemwideinterfaces.core.IMendixObjectMember;
import com.mendix.systemwideinterfaces.core.meta.IMetaPrimitive;
import com.mendix.systemwideinterfaces.core.meta.IMetaPrimitive.PrimitiveType;
public class Importer {
private IContext context;
private Map<Long, IMendixObject> newObjects, existingObjects;
private ILogNode logger;
private Export export;
public Importer(IContext c, Export export) {
context = c;
this.export = export;
logger = Core.getLogger("XMLImport");
newObjects = new HashMap<Long, IMendixObject>(); //will keep track of the created objects, with their original exported id's
existingObjects = new HashMap<Long, IMendixObject>(); //will keep track of the objects that imported objects refer to, with their exported id
}
public boolean handleImport() throws CoreException {
List<IMendixObject> exportedObjects = Core.retrieveByPath(context, export.getMendixObject(), ExportedObject.MemberNames.ExportedObject_Export.toString());
for (IMendixObject exportedObject: exportedObjects) {
ExportedObject o = ExportedObject.initialize(context, exportedObject);
if (newObjects.containsKey(o.getObjectID())) continue; // skip, because object was already created earlier in the import
if (o.getSearchExistingObject() && existingObjects.containsKey(o.getObjectID())) continue; //skip, because referenced object was already searched for and found earlier in the import
List<IMendixObject> exportedAttributes = Core.retrieveByPath(context, exportedObject, ExportedAttribute.MemberNames.ExportedAttribute_ExportedObject.toString());
logger.debug(exportedAttributes.size()+": " + o.toString());
if (!o.getSearchExistingObject()) { // create new Mendix object
IMendixObject actualObject = Core.instantiate(context, o.getEntityName());
newObjects.put(o.getObjectID(), actualObject);
for (IMendixObject exportedAttribute: exportedAttributes) {
ExportedAttribute a = ExportedAttribute.initialize(context, exportedAttribute);
IMendixObjectMember<?> m = actualObject.getMember(context, a.getAttributeName());
actualObject.setValue(context, a.getAttributeName(), m.getValueFromString(a.getAttributeValue()));
}
}
else { //todo: look for existing object using key members
if (exportedAttributes.size()==0)
throw new CoreException("Object of type "+o.getEntityName()+" is set to search for object, but has no key attributes defined.");
String xpath = ""; //build xpath to search for object with the key attributes
for (IMendixObject exportedAttribute: exportedAttributes) {
ExportedAttribute a = ExportedAttribute.initialize(context, exportedAttribute);
IMetaPrimitive mp = Core.getMetaObject(o.getEntityName()).getMetaPrimitive(a.getAttributeName());
String xpathValue = a.getAttributeValue();
if (mp.getType().equals(PrimitiveType.String) || mp.getType().equals(PrimitiveType.Enum) || mp.getType().equals(PrimitiveType.HashString))
xpathValue = "'"+xpathValue+"'";
xpath += "["+a.getAttributeName()+"="+xpathValue+"]";
}
List<IMendixObject> foundObjects = Core.retrieveXPathQueryEscaped(context, "//"+o.getEntityName()+xpath);
if (foundObjects.size()==0)
throw new CoreException("No objects found for xpath-query "+xpath);
if (foundObjects.size()>1)
logger.warn(foundObjects.size()+" objects found for xpath-query "+xpath+". Taking the first one");
existingObjects.put(o.getObjectID(), foundObjects.get(0));
}
}
for (IMendixObject exportedObject: exportedObjects) { //now set the associations between the objects for the newly created objects
ExportedObject o = ExportedObject.initialize(context, exportedObject);
if (!o.getSearchExistingObject()) {
IMendixObject actualObject = newObjects.get(o.getObjectID());
List<IMendixObject> exportedAssocs = Core.retrieveByPath(context, exportedObject, ExportedAssociation.MemberNames.ExportedAssociation_ExportedObject.toString());
for (IMendixObject exportedAssoc: exportedAssocs) {
ExportedAssociation as = ExportedAssociation.initialize(context, exportedAssoc);
List<IMendixObject> exportedReferences = Core.retrieveByPath(context, exportedAssoc, ExportedReference.MemberNames.ExportedReference_ExportedAssociation.toString());
IMendixObjectMember<?> m = actualObject.getMember(context, as.getCompleteName());
if (m==null) throw new CoreException("Association "+as.getCompleteName()+" not found in entity "+o.getType());
if (m instanceof MendixObjectReference) {
if (exportedReferences.size()>0) {
MendixObjectReference r = ((MendixObjectReference)m);
r.setValue(context, getIdInThisEnvironmentForImportedReference(ExportedReference.initialize(context, exportedReferences.get(0))));
//actualObject.setValue(context, as.getCompleteName(), i);
}
}
else if (m instanceof MendixObjectReferenceSet) {
MendixObjectReferenceSet rs = (MendixObjectReferenceSet) m;
List<IMendixIdentifier> refIds = new ArrayList<IMendixIdentifier>();
for (IMendixObject exportedReference: exportedReferences)
refIds.add(getIdInThisEnvironmentForImportedReference(ExportedReference.initialize(context, exportedReference)));
rs.setValue(context, refIds);
}
}
}
}
logger.info(exportedObjects);
// commit created objects
logger.info(newObjects.toString());
List<IMendixObject> objectsToCommit = new ArrayList<IMendixObject>(newObjects.values());
Core.commitWithoutEvents(context, objectsToCommit);
logger.info("Commited "+objectsToCommit.size()+" objects: "+objectsToCommit.toString());
return true;
}
private IMendixIdentifier getIdInThisEnvironmentForImportedReference(ExportedReference ref) throws CoreException {
long id = ref.getReferredObjectID();
IMendixObject objectInThisEnvironment = newObjects.get(id); //first look for the ObjectID in the list of newly created objects
if (objectInThisEnvironment != null) return objectInThisEnvironment.getId();
else {
objectInThisEnvironment = existingObjects.get(id); //if not found, look in the list of objects found by key
if (objectInThisEnvironment != null) return objectInThisEnvironment.getId();
else throw new CoreException("Object with id "+id+" not found in xml file");
}
}
}
The Java code uses Mendix’s core functions to create and fill the objects by name. Objects that need to be search by key are retrieved from the database. Finally, the associations between all objects created and found are set, and the objects are committed. That’s it! Now to try it out: I’ll leave that to you.
This was my final Mendix blog entry. Of course, you can contact me by email if you need help on your Mendix projects.
Leave a Reply