Don't Directly Serialize Your Classes To JSON or XML

23-Jul-2013 Like this? Dislike this? Let me know
As a way to get the ball rolling, consider this assertion:

If you have a need to produce or consume data at the edge of a system and you cannot switch select between JSON and XML formats, or be able to implement a new format converter in less than 1 day that can be immediately leveraged by the existing footprint, then you have not properly designed your externalization strategy.

Web services, AJAX backends, file vending and receipt, messaging, and many other use cases call for the creation and consumption of structured data, i.e. data with rich shape and not necessarily a flat collection of scalar values. There are many tools and techniques (like JAXB) in practically every popular language to somehow go from a JSON or XML document to a hydrated object. Yet every day, new tools and techniques (for example, those using Java annotations to drive externalization behavior) arrive on the scene and we constantly seem to spend more time than we wish in this area of development.

The challenges aren't in the tools. "Edge conversion" of content continues to be a time-drain for these reasons:

  1. There is only one, single compile-time version of code but there could be many externalized forms.
    It's not the simple, strict-rules conversions that cause the trouble; it's when the rules have to be broken to accommodate special object transformation and/or encodings, consumer demands, representations that combine other objects and potentially dynamic value generation logic, and other variations. Simply put, you cannot closely couple the object with forms of externalization. If you do, the first conversion pathway (from object to a particular output shape in a particular format) might be easy to produce, but as you add more, you quickly run into these problems:

  2. Objects and most languages have strong definitions of types, names, and values but externalized forms have different interpretations of these or sometimes lack them altogether.
    Objects have methods and state and there is no guarantee of one-to-one correspondence between a method and an underlying private piece of data.

    JSON has names and values but a limited type system; in particular, since it has no record definitions (schema), this means that to properly round-trip data to JSON back to data again, something has to know somewhere that a collection should be bound to a particular language-specific type.

    XML is simply a pile of nested tags and no intrinsic definition of type, name, or value. All semantics for the consumption of the tags as structure are implied and realized by a particular implementation of parsing/processing software. The XML specification says nothing about name and value, only the rules on what characters can be used in tags. In particular, because XML does not natively understand the concept of a collection, the distinction between name and type gets very distorted and in kind, so does conversion to- and from-object representation. Consider this simple XML data:

        <data>
            <name>Buzz</name>
        </data>
    
    Now, let's add a hobby:
        <data>
            <name>Buzz</name>
            <hobby>hacking</hobby>
        </data>
    
    How about ANOTHER hobby? We have two choices. The first is unfriendly XML:
        <data>
            <name>Buzz</name>
            <hobby>hacking</hobby>
            <hobby>cars</hobby>
        </data>
    
    It is unfriendly because until the second hobby is set up, a consumer does not know if hobby is modeled to be an entity occuring 0 or 1 times.

    The second choice is friendly XML where it is made structurally clear that hobby can occur more than once:

        <data>
            <name>Buzz</name>
    	<hobbies>
              <hobby>hacking</hobby>
    	  <!-- does not matter if "cars" is present or not.
    	       We "know" that there can be more than 1 hobby
    	       -->
    	</hobbies>
        </data>
    
    This is clearer, yes, but now look what happens: What ends up happening is the contained element (in this case hobby) takes on a blended, implicit role of both name and type. It is a name because it is used as a key to find it's value but it is also a type in the context of wrapper that implies it lives in a collection.

  3. JSON and XML have very different handling of collections, and a single environment for creating/consuming these two formats from a common set of objects is difficult.

  4. XML complicates the issue by supporting attributes which are not in the same namespace as nested elements. This means the following XML is perfectly legal:
        <data type="mineral">
            <type>vegetable</type>
        </data>
    
    and there is no default way to represent both type datums in a single JSON structure, although one could do somthing like this:
        {
            "@type":  "mineral",
            "type":  "vegetable"
        }
    
  5. JSON has no "enclosingTag" but in XML it is required. For example:.
        {
            "msg": "hello"
        }
    
    The JSON above will parse into (typically) a Map, e.g.:
        Map m = parseJSON(stream);
        String s = (String) m.get("msg"); // s = "hello"
    
    But in XML, an equivalent structure would be:
        <data>
            <msg>hello</msg>
        </data>
    
    Parsing this XML introduces an extra level into hierarchy.

The Solution

The solution I have found that offers the greatest flexibility and Day 2 creation of both new externalized shapes and format converters (e.g. adding an Avro converter as a "peer" in the toolkit to the XML and JSON converters) is to create a three-level chain for conversions: bespoke objects, Maps, and Map-to-format/format-to-Map converters. We will use Java as the example but the concept extends to any language. Even languages that have more in the way of runtime features and flexibility (perl, python, lisp, etc.) benefit from the three-level approach if for no other reason than a simplified and lower-risk test and release process. Also, for now, we will set aside the mechanics of web services, http, security and entitlements, REST v. SOAP, etc., and focus on the business of turning an object into an externalizable form.

The Bespoke Objects

Let's create a non-trivial class, acknowledging that on Day 2 your information and object design is not just a simple class with 3 get/set of Strings. If your classes are generally characterized as simple get/set of {strings,int,double,date}, you may have a different design problem altogether and likely your software design model has been overly influenced by an RDBMS persistor.
    package com.whatever.bespoke;

    public class MyClass {
        private List<Transitions> tlist;
        private Actor actor;
        private Map<String,Money> paymentThreshold;

	// ... various high-value methods
    }
MyClass is a regular, simple, plain object. No annotations. No special marker classes. No concerns about reflection. No special frameworks or dependencies. The object is, for once, simply allowed to do exactly what it needs to do as an object without being distorted by requirements for externalization.

The Map

The Map layer is the bridge to object data (de)serialization. Maps are perfectly capable of carrying strings, ints, doubles, dates, and other objects (notably BigDecimal) as well as nested Maps and Lists. Thus, they are well-suited to carry the data of a bespoke object in a form "just before it is stringified." Because there can be many external representations of MyClass, the to- and from-Map conversion (the "mapper") does not live in MyClass but rather in one or more utility classes that are segregated based on physical dependency, performance, security, and other dimensions. For our example here, we will examine only MyClass2Shape1. understanding that many other shapes of Maps could be constructed by the package housing MyClass2Shape1 or any other package.
    package com.whatever.maputils;

    import com.whatever.bespoke.MyClass;
    import java.util.Map;
    import java.util.HashMap;
    import java.util.List;
    import java.util.ArrayList;

    public class MyClass2Shape1 {
        public static Map toMap(MyClass c) {
            Map m = new HashMap();
            List l = new ArrayList();
            for(Transition t : c.getTransitions()) {
              l.add(TransitionMapUtils.toMap(t));
            }
            m.put("transitions", l);

            Map m2 = new HashMap();
	    for(Map.Entry me : c.getPaymentThreshold()) {
                m2.put(me.getKey(), MoneyMapUtils.toMap(me.getValue()));
	    }
            m.put("thresholds", m2);
            // ...
        }
    }
Important things to note:
  1. MyClass2Shape1 is in the maputils package, NOT in the bespoke package
  2. MyClass2Shape1 contains only static methods to do conversions
  3. Some of the other bespoke objects like Transition have ther own mapping utilities (TransitionMapUtils, MoneyMapUtils) that can be used to piece-wise contruct an outbound Map shape
  4. The Map may only contain those types that can be reasonably converted to a target externalizable format. String, Date, Integer, Long, and BigDecimal figure prominently here (plus Map and List). It largely becomes a Map of scalars for which toString() is very well defined.
  5. The to/fromMap functionality depends only on the bespoke objects and basic Map and List functionality which is clearly available in Java and pretty much any other language. No third party dependencies or special code of any kind is required here; it's only about understanding what a pre-stringified representation of the object might look like.
  6. Development of a mapper is admittedly tedious but straightforward. The good news is incremental additions to the mapper are very easy because there is likely already an existing snippet that can be copied and modified.

The Converters

By the time the pipeline gets to the converters, the job is relatively easy. The Map is passed to the third tier which draws upon any amount of third party libs (including zero) to actually render the Map to a stream in the desired format. The converters are shape neutral; thus, there is only one converter required to- and from-Maps for each format
    package com.whatever.jsonutils;

    import whatever.json.implementation.is.required;

    public class JSONUtils {
        public static void writeJSON(Map m, OutputStream os) {
            // whatever implementation is required
        }
        public static Map parseJSON(InputStream is) {
            // ...   
        }
    }

and

    package com.whatever.xmlutils;

    import whatever.xml.implementation.is.required;

    public class XMLUtils {
        public static void writeXML(Map m, OutputStream os) {
            // whatever implementation is required
        }
        public static Map parseXML(InputStream is) {
            // ...   
        }
    }
Important things to note:
  1. The converters live in package namespaces discrete from the the Map layer. This is important to prevent any third party physical dependency from creeping into the otherwise technology-neutral middle tier.
  2. As with the Map utils, the output converters are static functions that take in a stream and produce a Map or take a Map and write it to a stream.

The Consequences

  1. At a high level, the dependency/variability spectrum is polarized so that the bespoke objects and converter levels are relatively UNchanging, and the Map layer is as extendible as necessary. There is "no limit" to the number of MyClass2ShapeN variations that can be crafted. Remember, it is not unreasonable to have more than version of XML or JSON in production to satisfy consumers -- but there is only one version of the production software!
     Layer
     BespokeMapConverters
    Depends On Nothing except other bespoke objects
    • Bespoke objects
    • Basic scalar types, Map, and List
    • Basic scalar types, Map, and List
    • Third-party libs for parsing and/or emission, if necessary
    Changes Only when methods critical to the business function of the object are added/modified As often as necessary to create new shapes of externalizable data Rarely after the first set of converters formatters is created
    Variations None As many as necessary to create new shapes of externalizable data Few

  2. Adding a new converter is immediately leveragable to converting many existing Map shapes. Making the Map the externalized data form keeps the complexity to O(N) where N is the number of converters instead of O(N2) (recoding the formatter for each bespoke type).

  3. Objects organize their internal data on a single axis; that is, there is a single namespace of data within the object (sidestepping composition of other classes for the moment). Specifically, objects do not have a data section separate from an attributes section. So, when the MyClass2ShapeN mapper is run, it yields a very straightforward Map, also with a single axis. And since the XML converter is merely walking the Map, you end up with very friendly XML. There are no attributes and all data is structured in exactly the same way. Some might balk at the idea of XML with no attributes but if you're serious about consistent and straightforward deserializtion into a Map, it quickly becomes clear that attributes only get in your way.

  4. Clearly, lists of data do not exist in software in a form similar to the unfriendly XML form described earlier. You cannot have two items named hobby in the same key-value collection.
    A reasonable strategy for list emission in the XML converter is to take the name of the list, append _LIST to it, and use the result as the wrapper tag. For example, given this:
        Map m1 = new HashMap(); 
        List l2 = new ArrayList();
        while(conditions) {
            Map m2 = new HashMap();
    	m2.put("name", value); // value changes from item to item
    	l2.add(m2);
        }
        m1.put("things", l2); // the main Map has a single entry: things
    
    we will end up with a main map m1 that has a name things which references a List. Note that the elements of the List are Maps and are essentially untyped and unnamed; they are just integer offsets. Just for grins, here's what the JSON generator would emit:
        {
            "things": [
              { "name": "value1" },
              { "name": "value2" }
             ]
        }
    
    JSON exactly follows the in-memory Map represention. XML isn't so lucky, but using the strategy described above, an XML converter would emit:
        <someEnclosingTag>
          <things_LIST>
    	<things>
    	  <name>value1</name>
    	</things>
    	<things>
    	  <name>value2</name>
    	</things>
          </things_LIST>
        </someEnclosingTag>
    
    The same converter module could also be sensitive to _LIST when asked to parse XML and upon detecting it, would "promote" the resulting List into the parent map and then remove the _LIST element. So instead of this:
        Map m1 = result.get("someEnclosingTag");
        Map m2 = m1.get("things_LIST");  // m2 is a "wasted step"
        List l2 = m2.get("things");
    
    we would have:
        Map m1 = result.get("someEnclosingTag");
        // things_LIST has been removed 
        List l2 = m1.get("things");  // ah!  things is now in parent (m1); no wasted step
    
    Of course, we probably wouldn't hard code _LIST in the handler but instead would code a "list strategy" class and pass that in, something akin to this:
        public class MyXMLListStrategy implements XMLListStrategy {
            public void produceList(Context ctx) {
                // Upon production of list, candidate wrapperName
                // is the name of the item in the Map, in the example
                // above, "things" 
                ctx.wrapperName = ctx.wrapperName + "_LIST";
                ctx.containedName = ctx.wrapperName;
            }
            public void consumeList(Context ctx) {
                int zn = ctx.wrapperName.indexOf("_LIST");
                if(zn > 0) {
                    // Shorten the wrapper name:
                    ctx.wrapperName =  ctx.wrapperName.substring(0, zn);
                    ctx.collectNestedElements = true;
                } // else not one of our constructs; consume it unchanged
            }
        }
    
        // In use:
        XMLUtils.writeXML(out, map, "nameOfEnclosingTag", new MyXMLListStrategy());
    
    In the example above, the list strategy is "Map-centric" in that the emitted XML is named with a bias toward the Map structure. Some consumers might want something more "XML-centric" where the wrapper tag has a more familiar plural or similar feel to it and the contained items are named so as to represent a single item:
    
        // This turns:
        //     people: [ "A", "B" ] 
        // into the familiar:
        // <people>
        //   <person>A</person>
        //   <person>B</person>
        // </people>
        public class MyXMLListStrategy implements XMLListStrategy {
            public void produceList(Context ctx) {
                if(ctx.wrapperName.equals("people")) {
                    // wrapper remains people; instead we change the contained name
                    ctx.containedName = "person";
                }
            }
            public void consumeList(Context ctx) {
                if(ctx.wrapperName.equals("people")) {
                    // wrapper remains people; instruct the engine to collect
                    ctx.collectNestedElements = true;
                }
            }
        }
    
    In general, the XML list strategy logic for emission takes a list name and returns two things: a) the name to use for the list    b) the name to use for each of the contained items. The logic for list parsing takes a list name and returns the new name to use for the list into which all the contained items will be collected.

  5. It is acceptable for the map created from a bespoke object to have different (and probably richer) types than a map hydrated from XML or JSON or other format. For example, a mapper might take a date in a bespoke object and place it in a Map as a Date, but after conversion to JSON and round-tripping to a new Map, the date will be carried as a String (mainly because JSON does not have a Date type). The reason this is not a showstopper is that the mapper class knows that the target field in the bespoke class is a Date, so if not presented with a Date, it can do whatever it needs to try to construct a Date from the input material. In essentially all cases, some form of a String can be used to round-trip a richer type (Date, BigDecimal, etc.) for a particular conversion.

    Note that both JSON and XML have schema facilities that live outside the core definition of the formats. JSON has json-schema (see json-schema.org and XML has many mature options including XSD. More sophisticated mappers and converters can make use of these facilities as desired, and when used, proper and consistent types can be generated on "both sides" of the process.

Like this? Dislike this? Let me know


Site copyright © 2013-2017 Buzz Moschetti. All rights reserved