Aug 20

How to serialize a TList of objects with Delphi

Delphi XE2, Embarcadero, Programming, RTTI, Uncategorized 5 Comments »

Some weeks ago a customer asked to me if it is possibile serialize a TList of objects. “Hey, you should use a TObjectList for this”, I said, but he absolutely needs (I dont know why) of a TList.

This is the (simple) sample code tested with Delphi XE2 Update4. Enjoy.

  1.  
  2. unit Unit4;
  3.  
  4. interface
  5.  
  6. uses
  7.   Winapi.Windows,
  8.   Winapi.Messages,
  9.   System.SysUtils,
  10.   System.Variants,
  11.   System.Classes,
  12.   Vcl.Graphics,
  13.   Vcl.Controls,
  14.   Vcl.Forms,
  15.   Vcl.Dialogs,
  16.   Vcl.StdCtrls;
  17.  
  18. type
  19.   TForm4 = class(TForm)
  20.     Button1: TButton;
  21.     Memo1: TMemo;
  22.     procedure Button1Click(Sender: TObject);
  23.   private
  24.     { Private declarations }
  25.   public
  26.     { Public declarations }
  27.   end;
  28.  
  29.   TPerson = class
  30.   private
  31.     FName: String;
  32.     procedure SetName(const Value: String);
  33.   published
  34.     property Name: String read FName write SetName;
  35.   end;
  36.  
  37. var
  38.   Form4: TForm4;
  39.  
  40. implementation
  41.  
  42.  
  43. uses
  44.   Contnrs,
  45.   dbxjson,
  46.   dbxjsonreflect;
  47.  
  48. {$R *.dfm}
  49.  
  50. procedure TForm4.Button1Click(Sender: TObject);
  51. var
  52.   list: TList;
  53.   m: TJSONMarshal;
  54.   json: TJSONObject;
  55.   p1: TPerson;
  56.   p2: TPerson;
  57. begin
  58.   p1 := TPerson.Create;
  59.   p2 := TPerson.Create;
  60.   try
  61.     p1.Name := 'Daniele Teti';
  62.     p2.Name := 'Peter Parker';
  63.     list := TList.Create;
  64.     try
  65.       list.Add(p1);
  66.       list.Add(p2);
  67.  
  68.       m := TJSONMarshal.Create;
  69.       try
  70.         // Register a specific converter for field FList
  71.         m.RegisterConverter(TList, 'FList', function(Data: TObject; Field: String): TListOfObjects
  72.           var
  73.             l: TList;
  74.             j: integer;
  75.           begin
  76.             l := Data as TList;
  77.             SetLength(Result, l.Count);
  78.             for j := 0 to l.Count - 1 do
  79.               Result[j] := TObject(l[j]); // HardCast from pointer
  80.           end);
  81.  
  82.         json := m.Marshal(list) as TJSONObject;
  83.         try
  84.           Memo1.Lines.Text := json.tostring;
  85.         finally
  86.           json.free;
  87.         end;
  88.       finally
  89.         m.free;
  90.       end;
  91.     finally
  92.       list.free;
  93.     end;
  94.   finally
  95.     p1.free;
  96.     p2.free;
  97.   end;
  98. end;
  99.  
  100. { TPerson }
  101.  
  102. procedure TPerson.SetName(const Value: String);
  103. begin
  104.   FName := Value;
  105. end;
  106.  
  107. end.

The output is, as expected, the following:

  1. {"type":"System.Classes.TList","id":1,"fields":{"FList":[{"type":"Unit4.TPerson","id":2,"fields":{"FName":"Daniele Teti"}},{"type":"Unit4.TPerson","id":3,"fields":{"FName":"Peter Parker"}}],"FCount":2,"FCapacity":4}}
Nov 08

Duck Typing in Delphi

"The Delphi ORM", Delphi XE2, Programming, Uncategorized 9 Comments »

During a new dorm feature development, I’m faced a nice problem:

I want to have a generic access to a “kind” of list

Let’s say:

  1. procedure DoSomething(Obj: TMyListType);
  2. begin
  3. …
  4. end;

But, I want to have that generic access without a Layer Supertype object, because I need to be able to use objects from other libraries or 3rd party. In this case traditional polimorphism is not usable, so I’ve opted for an interface…

  1. procedure DoSomething(MyIntf: IMyListInterface);
  2. begin
  3. …
  4. end;

Cool, but I want to have that access without any change to that object. So implement an interface is not a viable solution. This is particulary true because the generics data structure in Delphi do not implement an interface. Will be nice to have a fully interfaced list and dictionaries in a future Delphi version.

So, how I could implement a generic access to a generic list?

  1. procedure DoSomething(MySomething: ???);
  2. begin
  3. …
  4. end;

However, operations on that list are few and well known, so I’ve opted for a “DuckTyping“.

Introducing The Duck

Introducing The Duck

The name of the concept refers to the duck test, attributed to James Whitcomb Riley, which may be phrased as follows:

“When I see a bird that walks like a duck and swims like a duck and quacks like a duck, I call that bird a duck.”

So, in my case, I’ve created an Adapter object wich adapts the external interface (few, well know operations) to the “duck” inside it.

The adapter is useful because I can use the compiler checking that the RAW RTTI access doesn’t allow.

Write an “RTTI adapter” using the new Delphi RTTI, is very simple. This is the code of the class TDuckTypedList that allow to use any object as a “list”. How I can define that the object is actually a list?

The criteria is:

If the object has
- An “Add” method with one parameter of type TObject (or descendant);
- A “Count” property;
- A “GetItem” method that have an integer parameter and return an object;
- A “Clear” method;
then, that object is a list.

So I can write the following:

  1. DuckObject := TDuckTypedList.Create(Coll);  //the adapter
  2. for x := 0 to DuckObject.Count - 1 do
  3.   DoSomething(DuckObject.GetItem(x));

I’ve done some speed tests comparing this way to the classic static way, and the speed is almost the same because the RTTI lookup is cached in the constructor of the adapter. So, so far so good.

This solution is already in use inside the dorm code in a feature branch.

Full code is available in the dorm SVN

Any comments?

Sep 01

Custom Marshalling/UnMarshalling in Delphi 2010

Delphi 2010, Programming, RTTI, Uncategorized 19 Comments »

Introduction
Some days ago, Embarcadero has presented the new version of RAD Studio, 2010.
The are many new features, but you can find in a lot places around the web, so
I won’t repeat them here.

One of the things widely requested from all Delphi programmers all over the world over the past few years, including myself, is
certainly a new and more powerful RTTI.

The new system of RTTI has finally arrived, and pave the way for a large number of applications.
One area that has benefited from the new RTTI is for sure the marshaled objects.

Marshaling is defined as follows:

“In computer science, marshalling (similar to serialization) is the process of
transforming the memory representation of an object to a data format suitable for
storage or transmission. It is typically used when data must be moved between
different parts of a computer program or from one program to another.
The opposite, or reverse, of marshalling is called unmarshalling (demarshalling) (similar to deserialization).”
–WikiPedia

In Delphi 2010 the process of serialization and deserialization is handled respectively by a Marshaller and an Unmarshaller.

The built-in format for the serialization of any Delphi object is JSON.
There are 2 main classes responsible for serializing objects into JSON, both present in the unit DBXJSONReflect:
- TJSONMarshal
- TJSONUnMarshal

Let’s say you have an object defined as follow:

type
  1.   TKid = class
  2.     FirstName: String;
  3.     LastName: String;
  4.     Age: Integer;
  5.   end;

To serialize and deserialize an instance of TKid it requires the following steps:

var
  1.   Mar: TJSONMarshal;  //Serializer
  2.   UnMar: TJSONUnMarshal;  //UnSerializer
  3.   Kid: TKid;  //The Object to serialize
  4.   SerializedKid: TJSONObject;  //Serialized for of object
  5. begin
  6.   Mar := TJSONMarshal.Create(TJSONConverter.Create);
  7.   try
  8.     Kid := TKid.Create;
  9.     try
  10.       Kid.FirstName := 'Daniele';
  11.       Kid.LastName := 'Teti';      
  12.       Kid.Age := 29;      
  13.       SerializedKid := Mar.Marshal(Kid) as TJSONObject;
  14.     finally
  15.       FreeAndNil(Kid);
  16.     end;
  17.   finally
  18.     Mar.Free;
  19.   end;
  20.   //Output the JSON version of the Kid object
  21.   WriteLn(SerializedKid.ToString);  
  22.   // UnMarshalling Kid
  23.   UnMar := TJSONUnMarshal.Create;
  24.   try
  25.     Kid := UnMar.UnMarshal(SerializedKid) as TKid;
  26.     try
  27.       //now kid is the same as before marshalling
  28.       Assert(Kid.FirstName = 'Daniele');
  29.       Assert(Kid.LastName = 'Teti');
  30.       Assert(Kid.Age = 29);
  31.     finally
  32.       Kid.Free;
  33.     end;
  34.   finally
  35.     UnMar.Free;
  36.   end;
  37. end;

Simple, isn’t it?
To access the JSON string that is our object, we must call the method ToString.
The JSON representation of this object SerializedKid can be saved to file,
sent to a remote server, used by a Web page from a web service, stored on a database or sent into space (!!!).
The Delphi application re-read the JSON string, you can recreate the object as it was at the time of serialization.
But anyone with a JSON parser can still read the data in our object, even non Delphi client.
These are the advantages of having used an open format and standard.

So far the simple part …
How serialize a field differently from the default?

Suppose we add the date of birth to our TKid:

type
  1.   TKid = class
  2.     FirstName: String;
  3.     LastName: String;
  4.     Age: Integer;
  5.     BornDate: TDateTime;
  6.   end;

Serialize a TDateTime, localized and that I have in JSON string is a float, because for Delphi TDateTime is a decimal number.
If I read the data from another program Delphi, no problem, but if I wanted to read a script in JavaScript? or. NET? or Ruby?
Then I use a format “DATA” to understand, even for these languages.
The new engine provides the serialization too.
Is needed, however, to tell the Marshaller and UnMarsheller how to represent and reconstruct a particular
object field by two statements like the following:

//marshaller
  1. Marshaller.RegisterConverter(TKid, 'BornDate',
  2.   function(Data: TObject; Field: string): string
  3.   var
  4.     ctx: TRttiContext; date : TDateTime;
  5.   begin
  6.     date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
  7.     Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
  8.   end);
  9.  
  10. //UnMarshaller
  11. UnMarshaller.RegisterReverter(TKid, 'BornDate',
  12.   procedure(Data: TObject; Field: string; Arg: string)
  13.   var
  14.     ctx: TRttiContext;
  15.     datetime:TDateTime;
  16.   begin
  17.     datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)),
  18.                                StrToInt(Copy(Arg, 6, 2)),
  19.                                StrToInt(Copy(Arg, 9, 2)),
  20.                                StrToInt(Copy(Arg, 12, 2)),
  21.                                StrToInt(Copy(Arg, 15, 2)),
  22.                                StrToInt(Copy(Arg, 18, 2)), 0);
  23.     ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
  24.   end);

The anonymous method is called when the marshaller serializes the field ‘BornDate’ is called “Converter” while Unmarshaller anonymous method that calls when he has to reconstruct the object from the JSON string is the “Reverter”.
Thus serializing a TKid assure you that my object is readable both by Delphi from another language without loss of information.

But what happens when I have to serialize a complex type?

Suppose we extend TKid this:

type
  1.   TTeenager = class(TKid)
  2.     Phones: TStringList;
  3.     constructor Create; virtual;
  4.     destructor Destroy; virtual;
  5.   end;

We must define a Converter and a Reverter for the TStringList class.
We can do it this way:

var
  1.   Marshaller: TJSONMarshal;
  2.   UnMarshaller: TJSONUnMarshal;
  3.   Teenager: TTeenager;
  4.   Value, JSONTeenager: TJSONObject;
  5. begin
  6.   Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
  7.   try
  8.     Marshaller.RegisterConverter(TTeenager, 'BornDate',
  9.       function(Data: TObject; Field: string): string
  10.       var
  11.         ctx: TRttiContext; date : TDateTime;
  12.       begin
  13.         date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
  14.         Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
  15.       end);
  16.      
  17.     Marshaller.RegisterConverter(TStringList, function(Data: TObject): TListOfStrings
  18.                                               var
  19.                                                 i, count: integer;
  20.                                               begin
  21.                                                 count := TStringList(Data).count;
  22.                                                 SetLength(Result, count);
  23.                                                 for i := 0 to count - 1 do
  24.                                                   Result[i] := TStringList(Data)[i];
  25.                                               end);  //TStringList Converter
  26.     Teenager := TTeenager.CreateAndInitialize;
  27.     try
  28.       Value := Marshaller.Marshal(Teenager) as TJSONObject;
  29.     finally
  30.       Teenager.Free;
  31.     end;
  32.   finally
  33.     Marshaller.Free;
  34.   end;
  35.   // UnMarshalling Teenager
  36.   UnMarshaller := TJSONUnMarshal.Create;
  37.   try
  38.     UnMarshaller.RegisterReverter(TTeenager, 'BornDate',
  39.       procedure(Data: TObject; Field: string; Arg: string)
  40.       var
  41.         ctx: TRttiContext;
  42.         datetime: TDateTime;
  43.       begin
  44.         datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)),
  45.                                    StrToInt(Copy(Arg, 6, 2)),
  46.                                    StrToInt(Copy(Arg, 9, 2)),
  47.                                    StrToInt(Copy(Arg, 12, 2)),
  48.                                    StrToInt(Copy(Arg, 15, 2)),
  49.                                    StrToInt(Copy(Arg, 18, 2)), 0);
  50.         ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
  51.       end);
  52.     UnMarshaller.RegisterReverter(TStringList, function(Data: TListOfStrings): TObject
  53.                                                var
  54.                                                  StrList: TStringList;
  55.                                                  Str: string;
  56.                                                begin
  57.                                                  StrList := TStringList.Create;
  58.                                                  for Str in Data do
  59.                                                    StrList.Add(Str);
  60.                                                  Result := StrList;
  61.                                                end);  //TStringList Reverter
  62.  
  63.     Teenager := UnMarshaller.Unmarshal(Value) as TTeenager;
  64.     try
  65.       Assert('Daniele' = Teenager.FirstName);
  66.       Assert('Teti' = Teenager.LastName);
  67.       Assert(29 = Teenager.Age);
  68.       Assert(EncodeDate(1979, 11, 4) = Teenager.BornDate);
  69.       Assert(3 = Teenager.Phones.Count);
  70.       Assert('NUMBER01'=Teenager.Phones[0]);
  71.       Assert('NUMBER02'=Teenager.Phones[1]);
  72.       Assert('NUMBER03'=Teenager.Phones[2]);
  73.     finally
  74.       Teenager.Free;
  75.     end;
  76.   finally
  77.     UnMarshaller.Free;
  78.   end;
  79. end;

There are different types of Converter and Reverter.
In the the DBXJSONReflect there are 8 types of converters:

  1.   //Convert a field in an object array
  2.   TObjectsConverter = reference to function(Data: TObject; Field: String): TListOfObjects;
  3.   //Convert a field in a strings array
  4.   TStringsConverter = reference to function(Data: TObject; Field: string): TListOfStrings;
  5.  
  6.   //Convert a type in an objects array
  7.   TTypeObjectsConverter = reference to function(Data: TObject): TListOfObjects;
  8.   //Convert a type in a strings array  
  9.   TTypeStringsConverter = reference to function(Data: TObject): TListOfStrings;
  10.  
  11.   //Convert a field in an object
  12.   TObjectConverter = reference to function(Data: TObject; Field: String): TObject;
  13.   //Convert a field in a string  
  14.   TStringConverter = reference to function(Data: TObject; Field: string): string;
  15.  
  16.   //Convert specified type in an object
  17.   TTypeObjectConverter = reference to function(Data: TObject): TObject;
  18.   //Convert specified type in a string  
  19.   TTypeStringConverter = reference to function(Data: TObject): string;

Each of them deals with a particular conversion object representation in the final serialization, in our case we will use them to convert to JSON.

Also in the DBXJSONReflect unit are defined many “Reverter” dealing with retrieving
the serialized version of the data and use it to reconstruct the object previously serialized.
Because they are complementary to the Converter, I will not copy them here.

As a final example, we derive from TProgrammer by TTeenager adding a list of Laptops in the properties.

Is therefore necessary to introduce a new pair of Converter / Reverter.
In this example I have defined all the converter and reverter in another unit in
order to have more readable code:

type
  1.   TLaptop = class
  2.     Model: String;
  3.     Price: Currency;
  4.     constructor Create(AModel: String; APrice: Currency);
  5.   end;
  6.   TLaptops = TObjectList<TLaptop>;
  7.   TProgrammer = class(TTeenager)
  8.     Laptops: TLaptops;
  9.     constructor Create; override;
  10.     destructor Destroy; override;
  11.     class function CreateAndInitialize: TProgrammer;
  12.   end;
  13. // Implementation code…
  14. var
  15.   Marshaller: TJSONMarshal;
  16.   UnMarshaller: TJSONUnMarshal;
  17.   Programmer: TProgrammer;
  18.   Value, JSONProgrammer: TJSONObject;
  19. begin
  20.   Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
  21.   try
  22.     Marshaller.RegisterConverter(TProgrammer, 'BornDate', ISODateTimeConverter);
  23.     Marshaller.RegisterConverter(TStringList, StringListConverter);
  24.     Marshaller.RegisterConverter(TProgrammer, 'Laptops', LaptopListConverter);
  25.     Programmer := TProgrammer.CreateAndInitialize;
  26.     try
  27.       Value := Marshaller.Marshal(Programmer) as TJSONObject;
  28.     finally
  29.       Programmer.Free;
  30.     end;
  31.  
  32.     // UnMarshalling Programmer
  33.     UnMarshaller := TJSONUnMarshal.Create;
  34.     try
  35.       UnMarshaller.RegisterReverter(TProgrammer, 'BornDate', ISODateTimeReverter);
  36.       UnMarshaller.RegisterReverter(TStringList, StringListReverter);
  37.       UnMarshaller.RegisterReverter(TProgrammer, 'Laptops', LaptopListReverter);
  38.  
  39.       Programmer := UnMarshaller.Unmarshal(Value) as TProgrammer;
  40.       try
  41.         Assert('Daniele' = Programmer.FirstName);
  42.         Assert('Teti' = Programmer.LastName);
  43.         Assert(29 = Programmer.Age);
  44.         Assert(EncodeDate(1979, 11, 4) = Programmer.BornDate);
  45.         Assert(3 = Programmer.Phones.Count);
  46.         Assert('NUMBER01' = Programmer.Phones[0]);
  47.         Assert('NUMBER02' = Programmer.Phones[1]);
  48.         Assert('NUMBER03' = Programmer.Phones[2]);
  49.         Assert('HP Presario C700' = Programmer.Laptops[0].Model);
  50.         Assert(1000 = Programmer.Laptops[0].Price);
  51.         Assert('Toshiba Satellite Pro' = Programmer.Laptops[1].Model);
  52.         Assert(800 = Programmer.Laptops[1].Price);
  53.         Assert('IBM Travelmate 500' = Programmer.Laptops[2].Model);
  54.         Assert(1300 = Programmer.Laptops[2].Price);
  55.       finally
  56.         Programmer.Free;
  57.       end;
  58.     finally
  59.       UnMarshaller.Free;
  60.     end;
  61.   finally
  62.     Marshaller.Free;
  63.   end;
  64. end;

Unit CustomConverter.pas contains all needed Converters/Reverts as anon methods.

unit CustomConverter;
  1.  
  2. interface
  3.  
  4. uses
  5.   DBXJSONReflect,
  6.   MyObjects; //Needed by converter and reverter for TLaptops
  7.  
  8. var
  9.   ISODateTimeConverter: TStringConverter;
  10.   ISODateTimeReverter: TStringReverter;
  11.  
  12.   StringListConverter: TTypeStringsConverter;
  13.   StringListReverter: TTypeStringsReverter;
  14.  
  15.   LaptopListConverter: TObjectsConverter;
  16.   LaptopListReverter: TObjectsReverter;
  17.  
  18. implementation
  19.  
  20. uses
  21.   SysUtils, RTTI, DateUtils, Classes;
  22.  
  23. initialization
  24.  
  25. LaptopListConverter := function(Data: TObject; Field: String): TListOfObjects
  26. var
  27.   Laptops: TLaptops;
  28.   i: integer;
  29. begin
  30.   Laptops := TProgrammer(Data).Laptops;
  31.   SetLength(Result, Laptops.Count);
  32.   if Laptops.Count > 0 then
  33.     for I := 0 to Laptops.Count - 1 do
  34.       Result[I] := Laptops[i];
  35. end;
  36.  
  37.  
  38. LaptopListReverter := procedure(Data: TObject; Field: String; Args: TListOfObjects)
  39. var
  40.   obj: TObject;
  41.   Laptops: TLaptops;
  42.   Laptop: TLaptop;
  43.   i: integer;
  44. begin
  45.   Laptops := TProgrammer(Data).Laptops;
  46.   Laptops.Clear;
  47.   for obj in Args do
  48.   begin
  49.     laptop := obj as TLaptop;
  50.     Laptops.Add(TLaptop.Create(laptop.Model, laptop.Price));
  51.   end;
  52. end;
  53.  
  54. StringListConverter := function(Data: TObject): TListOfStrings
  55. var
  56.   i, count: integer;
  57. begin
  58.   count := TStringList(Data).count;
  59.   SetLength(Result, count);
  60.   for i := 0 to count - 1 do
  61.     Result[i] := TStringList(Data)[i];
  62. end;
  63.  
  64.  
  65. StringListReverter := function(Data: TListOfStrings): TObject
  66. var
  67.   StrList: TStringList;
  68.   Str: string;
  69. begin
  70.   StrList := TStringList.Create;
  71.   for Str in Data do
  72.     StrList.Add(Str);
  73.   Result := StrList;
  74. end;
  75.  
  76. ISODateTimeConverter := function(Data: TObject; Field: string): string
  77. var
  78.   ctx: TRttiContext; date : TDateTime;
  79. begin
  80.   date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
  81.   Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
  82. end;
  83.  
  84. ISODateTimeReverter := procedure(Data: TObject; Field: string; Arg: string)
  85. var
  86.   ctx: TRttiContext;
  87.   datetime :
  88.   TDateTime;
  89. begin
  90.   datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)), StrToInt(Copy(Arg, 6, 2)), StrToInt(Copy(Arg, 9, 2)), StrToInt
  91.       (Copy(Arg, 12, 2)), StrToInt(Copy(Arg, 15, 2)), StrToInt(Copy(Arg, 18, 2)), 0);
  92.   ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
  93. end;
  94.  
  95. end.

Last hint…
Every serialization/unserialization process can create “warnings”.
Those warnings are collected into the “Warnings” property of the Ser/UnSer Object.

Conclusions
In this post I tried to introduce the basics of the new serialization engine in Delphi 2010.
During the next ITDevCon to be held in Italy next November 11.12, I’ll have a talk in which I will extensively talk about serialization and RTTI.
All interested smart developers are invited :-)

ITALIAN P.S.
Se qualche programmatore italiano volesse avere la versione in italiano di questo post può lasciare un commento e vedrò di accontentarlo :-)

You can find the DUnit project Source Code

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in