Home » Delphi » Delphi 2010 » Programming » RTTI » Custom Marshalling/UnMarshalling in Delphi 2010

Custom Marshalling/UnMarshalling in Delphi 2010

Introduction
Some days ago, Embarcadero has presented the new version of RAD Studio, 2010.
The are many new features, but you can find in a lot places around the web, so
I won’t repeat them here.

One of the things widely requested from all Delphi programmers all over the world over the past few years, including myself, is
certainly a new and more powerful RTTI.

The new system of RTTI has finally arrived, and pave the way for a large number of applications.
One area that has benefited from the new RTTI is for sure the marshaled objects.

Marshaling is defined as follows:

“In computer science, marshalling (similar to serialization) is the process of
transforming the memory representation of an object to a data format suitable for
storage or transmission. It is typically used when data must be moved between
different parts of a computer program or from one program to another.
The opposite, or reverse, of marshalling is called unmarshalling (demarshalling) (similar to deserialization).”
–WikiPedia

In Delphi 2010 the process of serialization and deserialization is handled respectively by a Marshaller and an Unmarshaller.

The built-in format for the serialization of any Delphi object is JSON.
There are 2 main classes responsible for serializing objects into JSON, both present in the unit DBXJSONReflect:
– TJSONMarshal
– TJSONUnMarshal

Let’s say you have an object defined as follow:

type
  TKid = class
    FirstName: String;
    LastName: String;
    Age: Integer;
  end;

To serialize and deserialize an instance of TKid it requires the following steps:

var
  Mar: TJSONMarshal;  //Serializer
  UnMar: TJSONUnMarshal;  //UnSerializer
  Kid: TKid;  //The Object to serialize
  SerializedKid: TJSONObject;  //Serialized for of object
begin
  Mar := TJSONMarshal.Create(TJSONConverter.Create);
  try
    Kid := TKid.Create;
    try
      Kid.FirstName := 'Daniele';
      Kid.LastName := 'Teti';      
      Kid.Age := 29;      
      SerializedKid := Mar.Marshal(Kid) as TJSONObject;
    finally
      FreeAndNil(Kid);
    end;
  finally
    Mar.Free;
  end;
  //Output the JSON version of the Kid object
  WriteLn(SerializedKid.ToString);   
  // UnMarshalling Kid
  UnMar := TJSONUnMarshal.Create;
  try
    Kid := UnMar.UnMarshal(SerializedKid) as TKid;
    try
      //now kid is the same as before marshalling
      Assert(Kid.FirstName = 'Daniele');
      Assert(Kid.LastName = 'Teti');
      Assert(Kid.Age = 29);
    finally
      Kid.Free;
    end;
  finally
    UnMar.Free;
  end;
end;

Simple, isn’t it?
To access the JSON string that is our object, we must call the method ToString.
The JSON representation of this object SerializedKid can be saved to file,
sent to a remote server, used by a Web page from a web service, stored on a database or sent into space (!!!).
The Delphi application re-read the JSON string, you can recreate the object as it was at the time of serialization.
But anyone with a JSON parser can still read the data in our object, even non Delphi client.
These are the advantages of having used an open format and standard.

So far the simple part …
How serialize a field differently from the default?

Suppose we add the date of birth to our TKid:

type
  TKid = class
    FirstName: String;
    LastName: String;
    Age: Integer;
    BornDate: TDateTime;
  end; 

Serialize a TDateTime, localized and that I have in JSON string is a float, because for Delphi TDateTime is a decimal number.
If I read the data from another program Delphi, no problem, but if I wanted to read a script in JavaScript? or. NET? or Ruby?
Then I use a format “DATA” to understand, even for these languages.
The new engine provides the serialization too.
Is needed, however, to tell the Marshaller and UnMarsheller how to represent and reconstruct a particular
object field by two statements like the following:

//marshaller
Marshaller.RegisterConverter(TKid, 'BornDate', 
  function(Data: TObject; Field: string): string
  var
    ctx: TRttiContext; date : TDateTime;
  begin
    date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
    Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
  end);
  
//UnMarshaller
UnMarshaller.RegisterReverter(TKid, 'BornDate', 
  procedure(Data: TObject; Field: string; Arg: string)
  var
    ctx: TRttiContext;
    datetime:TDateTime;
  begin
    datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)), 
                               StrToInt(Copy(Arg, 6, 2)), 
                               StrToInt(Copy(Arg, 9, 2)), 
                               StrToInt(Copy(Arg, 12, 2)), 
                               StrToInt(Copy(Arg, 15, 2)), 
                               StrToInt(Copy(Arg, 18, 2)), 0);
    ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
  end);

The anonymous method is called when the marshaller serializes the field ‘BornDate’ is called “Converter” while Unmarshaller anonymous method that calls when he has to reconstruct the object from the JSON string is the “Reverter”.
Thus serializing a TKid assure you that my object is readable both by Delphi from another language without loss of information.

But what happens when I have to serialize a complex type?

Suppose we extend TKid this:

type
  TTeenager = class(TKid)
    Phones: TStringList;
    constructor Create; virtual;
    destructor Destroy; virtual;
  end; 

We must define a Converter and a Reverter for the TStringList class.
We can do it this way:

var
  Marshaller: TJSONMarshal;
  UnMarshaller: TJSONUnMarshal;
  Teenager: TTeenager;
  Value, JSONTeenager: TJSONObject;
begin
  Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
  try
    Marshaller.RegisterConverter(TTeenager, 'BornDate', 
      function(Data: TObject; Field: string): string
      var
        ctx: TRttiContext; date : TDateTime;
      begin
        date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
        Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
      end);
      
    Marshaller.RegisterConverter(TStringList, function(Data: TObject): TListOfStrings
                                              var
                                                i, count: integer;
                                              begin
                                                count := TStringList(Data).count;
                                                SetLength(Result, count);
                                                for i := 0 to count - 1 do
                                                  Result[i] := TStringList(Data)[i];
                                              end);  //TStringList Converter 
    Teenager := TTeenager.CreateAndInitialize;
    try
      Value := Marshaller.Marshal(Teenager) as TJSONObject;
    finally
      Teenager.Free;
    end;
  finally
    Marshaller.Free;
  end;
  // UnMarshalling Teenager
  UnMarshaller := TJSONUnMarshal.Create;
  try
    UnMarshaller.RegisterReverter(TTeenager, 'BornDate', 
      procedure(Data: TObject; Field: string; Arg: string)
      var
        ctx: TRttiContext;
        datetime: TDateTime;
      begin
        datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)), 
                                   StrToInt(Copy(Arg, 6, 2)), 
                                   StrToInt(Copy(Arg, 9, 2)), 
                                   StrToInt(Copy(Arg, 12, 2)), 
                                   StrToInt(Copy(Arg, 15, 2)), 
                                   StrToInt(Copy(Arg, 18, 2)), 0);
        ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
      end);
    UnMarshaller.RegisterReverter(TStringList, function(Data: TListOfStrings): TObject
                                               var
                                                 StrList: TStringList;
                                                 Str: string;
                                               begin
                                                 StrList := TStringList.Create;
                                                 for Str in Data do
                                                   StrList.Add(Str);
                                                 Result := StrList;
                                               end);  //TStringList Reverter

    Teenager := UnMarshaller.Unmarshal(Value) as TTeenager;
    try
      Assert('Daniele' = Teenager.FirstName);
      Assert('Teti' = Teenager.LastName);
      Assert(29 = Teenager.Age);
      Assert(EncodeDate(1979, 11, 4) = Teenager.BornDate);
      Assert(3 = Teenager.Phones.Count);
      Assert('NUMBER01'=Teenager.Phones[0]);
      Assert('NUMBER02'=Teenager.Phones[1]);
      Assert('NUMBER03'=Teenager.Phones[2]);
    finally
      Teenager.Free;
    end;
  finally
    UnMarshaller.Free;
  end;
end;

There are different types of Converter and Reverter.
In the the DBXJSONReflect there are 8 types of converters:

  //Convert a field in an object array
  TObjectsConverter = reference to function(Data: TObject; Field: String): TListOfObjects;
  //Convert a field in a strings array
  TStringsConverter = reference to function(Data: TObject; Field: string): TListOfStrings;

  //Convert a type in an objects array
  TTypeObjectsConverter = reference to function(Data: TObject): TListOfObjects;
  //Convert a type in a strings array  
  TTypeStringsConverter = reference to function(Data: TObject): TListOfStrings;

  //Convert a field in an object
  TObjectConverter = reference to function(Data: TObject; Field: String): TObject;
  //Convert a field in a string  
  TStringConverter = reference to function(Data: TObject; Field: string): string;

  //Convert specified type in an object
  TTypeObjectConverter = reference to function(Data: TObject): TObject;
  //Convert specified type in a string  
  TTypeStringConverter = reference to function(Data: TObject): string;

Each of them deals with a particular conversion object representation in the final serialization, in our case we will use them to convert to JSON.

Also in the DBXJSONReflect unit are defined many “Reverter” dealing with retrieving
the serialized version of the data and use it to reconstruct the object previously serialized.
Because they are complementary to the Converter, I will not copy them here.

As a final example, we derive from TProgrammer by TTeenager adding a list of Laptops in the properties.

Is therefore necessary to introduce a new pair of Converter / Reverter.
In this example I have defined all the converter and reverter in another unit in
order to have more readable code:

type
  TLaptop = class
    Model: String;
    Price: Currency;
    constructor Create(AModel: String; APrice: Currency);
  end;
  TLaptops = TObjectList<TLaptop>;
  TProgrammer = class(TTeenager)
    Laptops: TLaptops;
    constructor Create; override;
    destructor Destroy; override;
    class function CreateAndInitialize: TProgrammer;
  end;
// Implementation code...
var
  Marshaller: TJSONMarshal;
  UnMarshaller: TJSONUnMarshal;
  Programmer: TProgrammer;
  Value, JSONProgrammer: TJSONObject;
begin
  Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
  try
    Marshaller.RegisterConverter(TProgrammer, 'BornDate', ISODateTimeConverter);
    Marshaller.RegisterConverter(TStringList, StringListConverter);
    Marshaller.RegisterConverter(TProgrammer, 'Laptops', LaptopListConverter);
    Programmer := TProgrammer.CreateAndInitialize;
    try
      Value := Marshaller.Marshal(Programmer) as TJSONObject;
    finally
      Programmer.Free;
    end;

    // UnMarshalling Programmer
    UnMarshaller := TJSONUnMarshal.Create;
    try
      UnMarshaller.RegisterReverter(TProgrammer, 'BornDate', ISODateTimeReverter);
      UnMarshaller.RegisterReverter(TStringList, StringListReverter);
      UnMarshaller.RegisterReverter(TProgrammer, 'Laptops', LaptopListReverter);

      Programmer := UnMarshaller.Unmarshal(Value) as TProgrammer;
      try
        Assert('Daniele' = Programmer.FirstName);
        Assert('Teti' = Programmer.LastName);
        Assert(29 = Programmer.Age);
        Assert(EncodeDate(1979, 11, 4) = Programmer.BornDate);
        Assert(3 = Programmer.Phones.Count);
        Assert('NUMBER01' = Programmer.Phones[0]);
        Assert('NUMBER02' = Programmer.Phones[1]);
        Assert('NUMBER03' = Programmer.Phones[2]);
        Assert('HP Presario C700' = Programmer.Laptops[0].Model);
        Assert(1000 = Programmer.Laptops[0].Price);
        Assert('Toshiba Satellite Pro' = Programmer.Laptops[1].Model);
        Assert(800 = Programmer.Laptops[1].Price);
        Assert('IBM Travelmate 500' = Programmer.Laptops[2].Model);
        Assert(1300 = Programmer.Laptops[2].Price);
      finally
        Programmer.Free;
      end;
    finally
      UnMarshaller.Free;
    end;
  finally
    Marshaller.Free;
  end;
end;


Unit CustomConverter.pas contains all needed Converters/Reverts as anon methods.

unit CustomConverter;

interface

uses
  DBXJSONReflect,
  MyObjects; //Needed by converter and reverter for TLaptops

var
  ISODateTimeConverter: TStringConverter;
  ISODateTimeReverter: TStringReverter;

  StringListConverter: TTypeStringsConverter;
  StringListReverter: TTypeStringsReverter;

  LaptopListConverter: TObjectsConverter;
  LaptopListReverter: TObjectsReverter;

implementation

uses
  SysUtils, RTTI, DateUtils, Classes;

initialization

LaptopListConverter := function(Data: TObject; Field: String): TListOfObjects
var
  Laptops: TLaptops;
  i: integer;
begin
  Laptops := TProgrammer(Data).Laptops;
  SetLength(Result, Laptops.Count);
  if Laptops.Count > 0 then
    for I := 0 to Laptops.Count - 1 do
      Result[I] := Laptops[i];
end;


LaptopListReverter := procedure(Data: TObject; Field: String; Args: TListOfObjects)
var
  obj: TObject;
  Laptops: TLaptops;
  Laptop: TLaptop;
  i: integer;
begin
  Laptops := TProgrammer(Data).Laptops;
  Laptops.Clear;
  for obj in Args do
  begin
    laptop := obj as TLaptop;
    Laptops.Add(TLaptop.Create(laptop.Model, laptop.Price));
  end;
end;

StringListConverter := function(Data: TObject): TListOfStrings
var
  i, count: integer;
begin
  count := TStringList(Data).count;
  SetLength(Result, count);
  for i := 0 to count - 1 do
    Result[i] := TStringList(Data)[i];
end;


StringListReverter := function(Data: TListOfStrings): TObject
var
  StrList: TStringList;
  Str: string;
begin
  StrList := TStringList.Create;
  for Str in Data do
    StrList.Add(Str);
  Result := StrList;
end;

ISODateTimeConverter := function(Data: TObject; Field: string): string
var
  ctx: TRttiContext; date : TDateTime;
begin
  date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
  Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
end;

ISODateTimeReverter := procedure(Data: TObject; Field: string; Arg: string)
var
  ctx: TRttiContext;
  datetime :
  TDateTime;
begin
  datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)), StrToInt(Copy(Arg, 6, 2)), StrToInt(Copy(Arg, 9, 2)), StrToInt
      (Copy(Arg, 12, 2)), StrToInt(Copy(Arg, 15, 2)), StrToInt(Copy(Arg, 18, 2)), 0);
  ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
end;

end.

Last hint…
Every serialization/unserialization process can create “warnings”.
Those warnings are collected into the “Warnings” property of the Ser/UnSer Object.

Conclusions
In this post I tried to introduce the basics of the new serialization engine in Delphi 2010.
During the next ITDevCon to be held in Italy next November 11.12, I’ll have a talk in which I will extensively talk about serialization and RTTI.
All interested smart developers are invited :-)

ITALIAN P.S.
Se qualche programmatore italiano volesse avere la versione in italiano di questo post può lasciare un commento e vedrò di accontentarlo :-)

You can find the DUnit project Source Code

20 commenti a “Custom Marshalling/UnMarshalling in Delphi 2010

  1. This example shows a basic flaw of JSON as used by Delphi: it is very easy to sniff and tamper with data. And Datasnap does nothing to protect the data and identify who sent them. Sure, one can use IPSec (which may require a fairly complex network setup, and can identify a machine, not a user) or TLS/SSL (and it requires certificate issuing and management for all parties). Would you transport your sensitive data via a string protocol with no protection?
    Also, all those strings manipulations reminds me of PHP, not a compiled language very apt to manage binary streams. But one day all data types will go away and programmers will be able to use strings only…

  2. @LDS:
    From JSON website:
    “JSON is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate.”
    Those are all the JSON benefits. JSON is not related with security issues like XML (as data-interchange protocol) is not directly related with SOAP web services security issues. Security is not a responsability of the data-interchage format, but is a responsability for the transport protocol.

    Wikipedia about transport protocol says:
    “In computer networking, the Transport Layer is a group of methods and protocols within a layered architecture of network components within which it is responsible for encapsulating application data blocks into data units (datagrams, segments) suitable for transfer to the network infrastructure for transmission to the destination host”

    This is definitively not a JSON responsability.

    However, your security needs (very similar to my needs for some projects) are very important… so I’m preparing some talk about datasnap and datasnap filters. I’m developing a cryptography filter for secure comunication over DataSnap (without using “TCP security” tecnologies like IPSec). Probably I’ll talk about this filters in the italian Delphi2010 presentation in Milan next september-24-2010 and, in a more deeper manner, at the ITDevCon in Verona 11-12 sept 2010.

  3. Great article!!

    in the first example (serialize and deserialize an instance of TKid) if I save the SerializedKid.ToString into a file when I reload the file how can I convert the string into a TJsonValue (SerializedKid) to call UnMar.UnMarshal(SerializedKid) as TKid;

    I’ve tried SerializedKid:= TJSONString.Create(Memo1.Lines.LoadfFromFile(‘xxx.txt’);
    but I get an access violation

    There are very few examples on the web about Json in delphi 2010
    Thanks for your support!!!

  4. those converters/reverts need to be free/released or something ?
    Because i get a memory leak as long as they are defined. Just open/close the application, without using the converters/reverts will result in a memory leak.

    thank you

  5. StringListReverter causes a memory leak!
    Everytime the callback is called during the unmarshal process it replaces the existing field with a new instance without freeing the old one!
    If I understood all, this is a design problem with all TTypeXXXReverter callbacks.
    Did I understood?

    Thank you.

    #
    StringListReverter := function(Data: TListOfStrings): TObject
    #
    var
    #
    StrList: TStringList;
    #
    Str: string;
    #
    begin
    #
    StrList := TStringList.Create;
    #
    for Str in Data do
    #
    StrList.Add(Str);
    #
    Result := StrList;
    #
    end;

  6. I investigated and I must say that “Delphi JSON Marshaling/Unmarshaling” is full of bugs like this.
    Look in QualityCentral: #QC#81869, QC#81862 and QC#82857.
    I’m very disappointed.
    How can I use a feature with bugs like those?

    Thank you.

  7. Pingback: Alex Gordon
  8. Excellent post!

    How can I marshall a TDictionary in Delphi 2010? I can’t seem to get it to call the registered converter at all. Is this because Delphi 2010’s RTTI doesn’t know about the instantiated generics? Otherwise i’ll be stuck using customized xml marshalling for now.

Lascia una risposta

L'indirizzo email non verrà pubblicato. I campi obbligatori sono contrassegnati *

È possibile utilizzare questi tag ed attributi XHTML: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>