Oct 01

DataSnap Filters Compendium

DataSnap Filters Compendium, Delphi 2010, Delphi for Win32, Uncategorized Add comments

What’s DataSnap Filters Compendium

DataSnap Filters Compendium (DSFC) is a compendium of 9 filters for DataSnap 2010.
The filters are divided into 3 groups:

HASH
MD5
MD4
SHA1
SHA512

CIPHER
Blowfish
Rijndael
3TDES
3DES

COMPRESS
LZO

HASH filters
The HASH filters helps avoid to any spiteful person to modify datasnap message through an “Man in the middle” attack (http://en.wikipedia.org/wiki/Man-in-the-middle_attack).
Functioning is basing on an easy principle. After sending the message, the filter calculates the hash of the message and tags along it to the message. When the message gets to destination, the filter recovers the hash calculated by the client and recalculates it on the remaining part of the message.
If the part of the extrated hash (calculated at the beginning) and the hash recalculated to the end are equal, the message hasn’t change.
To avoid someone could modify the message and also recalculates the hash, after calculating the hash, a GUID is tagged along to the message, which just the sender and the receiver know.
This kind of filters DOES NOT AVOID THE UNAVOIDED READING OF DATA,it avoids just the modification.

CIPHER filters
The CIPHER filters are the most interesting filters. Many datasnap users have requested a built-in system to get the data transmission safe. This set of filters colud be the answer.
In the actual version I’ve implemented Symmetric-key algorithms. Maybe I’m going to develop of Asymmetric-key algorithms filters.
Implemented algorithms features are well known, I list them as follows just to be completed:

Blowfish
Blowfish has a 64-bit block size and a variable key length from 32 up to 448 bits.
The filter version has a keysize = 56 byte.

Rijndael
AES Round 2.
AES has a fixed block size of 128 bits and a key size of 128, 192, or 256 bits, whereas Rijndael can be specified with block and key sizes in any multiple of 32 bits, with a minimum of 128 bits and a maximum of 256 bits.
The filter version has a keysize = 32 byte.

3TDES
Triple DES with 24 byte Blocksize, 24 byte Keysize 168 bits relevant
The filter version has a keysize = 24 byte.

3DES
Triple DES with 8 byte Blocksize, 24 byte Keysize 168 bits relevant
The filter version has a keysize = 24 byte.

COMPRESS filters
Actually the LZO compression is the only one that exists, and is one of the faster compression algorithms.
The compression ratio compared to the ZLib is worse but about 3 times faster (as the table in next paragraph shows).

Tests
DSFC has a huge suite of unit tests and speed tests.
The speed tests show how filters are fast and how the data stream size is affected by their work. Follow table contains results of speed test execution (DSFCSpeedTest) on my workstation. If you want give a meaning to those numbers, see the code of the speed test :-)

  1. == HASH FILTERS == 1000 iterations
  2. MD5             =  161ms (stream size:  8304 byte, filtered stream size:  4184 byte)
  3. MD4             =   99ms (stream size:  8304 byte, filtered stream size:  4184 byte)
  4. SHA1            =  145ms (stream size:  8304 byte, filtered stream size:  4192 byte)
  5. SHA512          =  344ms (stream size:  8304 byte, filtered stream size:  4280 byte)
  6.  
  7. == CIPHER FILTERS == 1000 iterations
  8. Blowfish        =  898ms (stream size:  8304 byte, filtered stream size:  8304 byte)
  9. Rijndael        =  941ms (stream size:  8304 byte, filtered stream size:  8304 byte)
  10. 3TDES           = 1729ms (stream size:  8304 byte, filtered stream size:  8304 byte)
  11. 3DES            = 1757ms (stream size:  8304 byte, filtered stream size:  8304 byte)
  12.  
  13. == COMPRESS FILTERS == 1000 iterations
  14. LZO             =   79ms (stream size:  8304 byte, filtered stream size:  1113 byte)
  15. ZLibCompression =  295ms (stream size:  8304 byte, filtered stream size:   799 byte)

Last speed test is for the only filter included in Delphi2010. It’s included only for compare times and data stream size with the other filters.

After you install DataSnapFiltersCompendium.bpl you will see following filters into the “Filters” property

New filters registered by DSFC

New filters registered by DSFC

Cipher filters have only one property for the encription key

Encryption key for the cipher filters

Encryption key for the cipher filters

Project Source Contains

  • Full Filters Code
  • Unit Test
  • SpeedTest

Download DataSnap Filters Compendium

P.S. I’ll be at ITDevCon… and you? :-)

27 Responses to “DataSnap Filters Compendium”

  1. Compendium – Filtros “Open Source” para DataSnap 2010 | Cesar Romero Says:
    October 2nd, 2009 at 10:49 am

    [...] Página do Projeto [...]

  2. Bruce McGee Says:
    October 2nd, 2009 at 2:08 pm

    Thanks.

    You mentioned that you might implement some asymmetric cipher filters. I’d love to see these, and maybe a built in key-exchange.

  3. Bruce McGee Says:
    October 2nd, 2009 at 2:13 pm

    Thank you for making this collection open source. What it the license?

  4. Daniele Teti Says:
    October 2nd, 2009 at 2:32 pm

    @Bruce
    License is “Mozilla Public License Version 1.1″.
    You can find the actual license text inside every unit (on top of it).

  5. David Taylor Says:
    October 3rd, 2009 at 5:18 am

    Nice bit of work!

    Is the LZO compressor native Delphi or linked with object files? I was also wondering about LZO licensing and compatibility with the MPL.

  6. Daniele Teti Says:
    October 5th, 2009 at 6:34 pm

    @David
    LZO compressor is pure Delphi+ASM code.
    You can find the author of LZO library at http://bouchez.info/lzo.html

  7. LDS Says:
    October 5th, 2009 at 11:06 pm

    Good. How keys are exchanged? How keys are stored safely? How do you ensure keys comes from a trusted endpoint? How do you ensure the MAC received is the same that was computed on the other side?
    If proper security was so easy, we wouldn’t see so many breeches.

  8. LD Says:
    October 6th, 2009 at 9:20 am

    “GUID is tagged along to the message”. Look for HMAC and see why it’s not enough - especially if the key is prepended to the message (and embedded in the code…)
    Cryptography is not a matter for sorcerer’s apprentices, sorry, it requires proper knowledge to implement it the correct way, and avoid pitfalls.
    And it *must* work even if the attacker has access to the executable or even the source code.
    The more Embarcadero and its representatives understimate security needs and deliver flawed implementations, the more Delphi appears a non-professional tool, sorry.

  9. Daniele Teti Says:
    October 6th, 2009 at 9:53 am

    @LD/LDS
    Cipher filter doesn’t have GUID tagged along with message.

    This kind of filters (cipher) work even if the attacker has access to the executable.

    Key is not embedded into the executable (if you want). You can read the key wherever want, but safely storing of keys is not a responsibility of the datasnap filters!

    Besides, there is no key exchange.
    Wikipedia says: “The encryption key is trivially related to the decryption key, in that they may be identical or there is a simple transformation to go between the two keys. The keys, in practice, represent a shared secret between two or more parties that can be used to maintain a private information link.”

    Have you saw some “sorcerer’s apprentices” here? :-)

  10. LDS Says:
    October 6th, 2009 at 4:09 pm

    “Cipher filter doesn’t have GUID tagged along with message”. In fact I was speaking about the hash functions. HMAC = Hash Message Authentication Code. The way you implemented hash they are just useful as a “check” to ensure data aren’t changed by the communication link - not that they were not tampered with.

    “The keys, in practice, represent a shared secret between”. Right - how are you going to share that secret? With any ciphering algorithm, if the algorithm has no known flaws and is well know, it’s only the key to warrant data security. Thereby you have to ensure both parties have the keys and they cannot be easily get. And how do you that? Embedding the key in the executable is a risk. Storing keys in accessible files is a risk - there are not outsiders only, insiders may be a risk too. Using the same key over and over may be a risk - but if you use session keys you have to be able to transmit them to the other end securely.
    That’s why I was talking about “sorcerer’s apprentices” - security is a “all or nothing” approach, “half securiry” usually means “no security”, encrypting a buffer is easy, but if you can’t handle keys properly it’s useless. Ok, storing the keys is not a Datasnap responsibility - but the key exchange?

  11. Ali Says:
    October 6th, 2009 at 5:05 pm

    Thank you, that’s really a nice share.

  12. Atmapuri Says:
    October 15th, 2009 at 7:23 pm

    The package installs, the tests pass, and the filters work for tables with few records. For one table which 300 records and 13MBytes it fails and for another which has 10 000 records and 400Kbytes also. The client side of the Datasnap gives error here, or returns a “Read error”: (lzo.inc)

    assert(result=destB[0]);

    In fact, there appears to be no filter working as soon as the table is large enough. (using TDSProviderConnection on the client side and tested server and client on local computer).

  13. Bob Swart Says:
    October 20th, 2009 at 10:49 am

    Thanks for the filters - I’ve mentioned them in my DataSnap 2010 white paper, as well as my Delphi 2010 DataSnap Enhancements session (in a few hours here at SDC 2009).

  14. Charles Says:
    November 22nd, 2009 at 1:20 pm

    is the LZO compressor has problems with blob fields (BMP images)?

  15. Daniele Teti Says:
    November 23rd, 2009 at 3:55 pm

    @Charles & @Atmapuri
    No, there are not *known* bugs in DSFC.
    Owever, there are some bugs in DataSnap that may affect DSFC. With Delphi 2010 Update Pack 2 those bugs should be resolved. I’ll write a post about this asap.

  16. Franco Says:
    December 16th, 2009 at 11:23 am

    I’ve just installed Delphi 2010 Update Pack 4&5 (superseders of updates 2&3) and still I have no luck working with DSFC with “big” tables (100-200 records or more).
    I’ve seen something mentioned in the release notes for updates 4&5: “DataSnap Filters is raising exception some times, more frequently in debug mode. After some test with Adrian, he knows where is the problem and look likes is easy to fix.”. Is this related to what makes DSFC working bad? If so, why it’s still not working properly after installing the updates?

  17. Bruno Says:
    February 3rd, 2010 at 8:58 pm

    I didn’t see in your comparison the time without the filters. I would like to use some filter to maximize the performance from my datasnap server (Delphi 2010). Could you say something about that? Thanks.

  18. Daniele Teti Says:
    February 3rd, 2010 at 9:52 pm

    @Bruno
    The speed test program test the speed of filters only. It’s not a fully datasnap system, so cannot test the speed without filters. You can create a small server for this kind of test. And, if you want, I’ll publish result here.

  19. Bruno Says:
    February 4th, 2010 at 1:27 pm

    Hi Daniele I’m using RAD Studio 2010 Update 1. I’m trying to use the filter LZO in the component DSTCPServerTransport, but when I start my application an exception ocurred:
    “Exception EReadError in module rtl140.bpl at 0006B8EA.
    Error reading DSTCPServerTransport1.Filters: Error reading TTransportFilterItem.FilterId: Illegal argument”

    Am I doing something wrong?

    Thanks.

    Best Regards from Brazil!

  20. Bruno Says:
    February 10th, 2010 at 8:04 pm

    Hi Daniele, after I upgrade my RAD Studio 2010 (Update 4 and 5) the filters does not works, yet. What can I do to fix that? Thanks.

  21. Bruno Says:
    February 12th, 2010 at 7:13 pm

    Daniele, I’ve already done the test required last week. Well, the performance of DataSnap decreased a lot when I used the LZO filter over a VPN, so I decided not use filters. Thanks!

  22. Daniele Teti Says:
    February 13th, 2010 at 12:01 pm

    @Bruno
    There is a QC issue related to the speed of datasnap filters.
    This slowness is not related to the LZO filter itself (it is faster that the default ZLib filter), but is related to a generic slowness in filter using

  23. Jon Scarlett Says:
    March 10th, 2010 at 12:14 am

    I am having to same problem as Bruno.

    “Exception EReadError in module rtl140.bpl at 0006B8EA.
    Error reading DSTCPServerTransport1.Filters: Error reading TTransportFilterItem.FilterId: Illegal argument”

    What is the solution?

  24. -dj- Says:
    May 19th, 2010 at 1:36 pm

    I have the same problem as Jon and Bruno!!
    The error raise in TTransportFilterFactory.CreateFilter function, because the TDBXObjectStore.GetObjectFromName returns NIL. Only the ZLibCompression is correct.

  25. Juliano Says:
    May 24th, 2010 at 6:52 pm

    I have the same problem to.
    Illegal argument, on the server.

    Have solution ?

  26. Daniele Teti Says:
    May 24th, 2010 at 11:02 pm

    @Juliano et All
    Please, create an issue in the googlecode project and attach a simple project to reproduce the error.

  27. Murat Ak Says:
    June 21st, 2010 at 12:40 pm

    Hi,
    I make same test for lzo. And i think there is a problem in
    TLZO.DeCompress function.

    I Compress a file sized 1.500 KB. And compressed memory is 98 KB. and again i decompress same memory and i got access violation.

    In TLZO.Decompress
    OutData := GetMemory(100 + Length(Data) * 2);
    Code is not correct, because Data is 98 KB and * 2 = 196 KB but output memory is 1.500 KB.
    and GetMemory for outdata is not correct. and i got access violation error. I Change the code that
    OutData := GetMemory(100 + Length(Data) * 40);
    and now it is working, but sure GetMemory can not be correct again. And it can be use more memory.
    I think we need to correct getmemory size.

    Best Regards

Leave a Reply

WP Theme & Icons by N.Design Studio
Entries RSS Comments RSS Log in