DataSnap Filters Compendium

What’s DataSnap Filters Compendium

DataSnap Filters Compendium (DSFC) is a compendium of 9 filters for DataSnap 2010.
The filters are divided into 3 groups:

HASH
MD5
MD4
SHA1
SHA512

CIPHER
Blowfish
Rijndael
3TDES
3DES

COMPRESS
LZO

HASH filters
The HASH filters helps avoid to any spiteful person to modify datasnap message through an “Man in the middle” attack (http://en.wikipedia.org/wiki/Man-in-the-middle_attack).
Functioning is basing on an easy principle. After sending the message, the filter calculates the hash of the message and tags along it to the message. When the message gets to destination, the filter recovers the hash calculated by the client and recalculates it on the remaining part of the message.
If the part of the extrated hash (calculated at the beginning) and the hash recalculated to the end are equal, the message hasn’t change.
To avoid someone could modify the message and also recalculates the hash, after calculating the hash, a GUID is tagged along to the message, which just the sender and the receiver know.
This kind of filters DOES NOT AVOID THE UNAVOIDED READING OF DATA,it avoids just the modification.

CIPHER filters
The CIPHER filters are the most interesting filters. Many datasnap users have requested a built-in system to get the data transmission safe. This set of filters colud be the answer.
In the actual version I’ve implemented Symmetric-key algorithms. Maybe I’m going to develop of Asymmetric-key algorithms filters.
Implemented algorithms features are well known, I list them as follows just to be completed:

Blowfish
Blowfish has a 64-bit block size and a variable key length from 32 up to 448 bits.
The filter version has a keysize = 56 byte.

Rijndael
AES Round 2.
AES has a fixed block size of 128 bits and a key size of 128, 192, or 256 bits, whereas Rijndael can be specified with block and key sizes in any multiple of 32 bits, with a minimum of 128 bits and a maximum of 256 bits.
The filter version has a keysize = 32 byte.

3TDES
Triple DES with 24 byte Blocksize, 24 byte Keysize 168 bits relevant
The filter version has a keysize = 24 byte.

3DES
Triple DES with 8 byte Blocksize, 24 byte Keysize 168 bits relevant
The filter version has a keysize = 24 byte.

COMPRESS filters
Actually the LZO compression is the only one that exists, and is one of the faster compression algorithms.
The compression ratio compared to the ZLib is worse but about 3 times faster (as the table in next paragraph shows).

Tests
DSFC has a huge suite of unit tests and speed tests.
The speed tests show how filters are fast and how the data stream size is affected by their work. Follow table contains results of speed test execution (DSFCSpeedTest) on my workstation. If you want give a meaning to those numbers, see the code of the speed test 🙂

== HASH FILTERS == 1000 iterations
MD5             =  161ms (stream size:  8304 byte, filtered stream size:  4184 byte)
MD4             =   99ms (stream size:  8304 byte, filtered stream size:  4184 byte)
SHA1            =  145ms (stream size:  8304 byte, filtered stream size:  4192 byte)
SHA512          =  344ms (stream size:  8304 byte, filtered stream size:  4280 byte)

== CIPHER FILTERS == 1000 iterations
Blowfish        =  898ms (stream size:  8304 byte, filtered stream size:  8304 byte)
Rijndael        =  941ms (stream size:  8304 byte, filtered stream size:  8304 byte)
3TDES           = 1729ms (stream size:  8304 byte, filtered stream size:  8304 byte)
3DES            = 1757ms (stream size:  8304 byte, filtered stream size:  8304 byte)

== COMPRESS FILTERS == 1000 iterations
LZO             =   79ms (stream size:  8304 byte, filtered stream size:  1113 byte)
ZLibCompression =  295ms (stream size:  8304 byte, filtered stream size:   799 byte)

Last speed test is for the only filter included in Delphi2010. It’s included only for compare times and data stream size with the other filters.

After you install DataSnapFiltersCompendium.bpl you will see following filters into the “Filters” property

New filters registered by DSFC
New filters registered by DSFC

Cipher filters have only one property for the encription key

Encryption key for the cipher filters
Encryption key for the cipher filters

Project Source Contains

  • Full Filters Code
  • Unit Test
  • SpeedTest

Download DataSnap Filters Compendium

P.S. I’ll be at ITDevCon… and you? 🙂

33 thoughts on “DataSnap Filters Compendium

  1. Thanks.

    You mentioned that you might implement some asymmetric cipher filters. I’d love to see these, and maybe a built in key-exchange.

  2. Nice bit of work!

    Is the LZO compressor native Delphi or linked with object files? I was also wondering about LZO licensing and compatibility with the MPL.

  3. Good. How keys are exchanged? How keys are stored safely? How do you ensure keys comes from a trusted endpoint? How do you ensure the MAC received is the same that was computed on the other side?
    If proper security was so easy, we wouldn’t see so many breeches.

  4. “GUID is tagged along to the message”. Look for HMAC and see why it’s not enough – especially if the key is prepended to the message (and embedded in the code…)
    Cryptography is not a matter for sorcerer’s apprentices, sorry, it requires proper knowledge to implement it the correct way, and avoid pitfalls.
    And it *must* work even if the attacker has access to the executable or even the source code.
    The more Embarcadero and its representatives understimate security needs and deliver flawed implementations, the more Delphi appears a non-professional tool, sorry.

  5. @LD/LDS
    Cipher filter doesn’t have GUID tagged along with message.

    This kind of filters (cipher) work even if the attacker has access to the executable.

    Key is not embedded into the executable (if you want). You can read the key wherever want, but safely storing of keys is not a responsibility of the datasnap filters!

    Besides, there is no key exchange.
    Wikipedia says: “The encryption key is trivially related to the decryption key, in that they may be identical or there is a simple transformation to go between the two keys. The keys, in practice, represent a shared secret between two or more parties that can be used to maintain a private information link.”

    Have you saw some “sorcerer’s apprentices” here? 🙂

  6. “Cipher filter doesn’t have GUID tagged along with message”. In fact I was speaking about the hash functions. HMAC = Hash Message Authentication Code. The way you implemented hash they are just useful as a “check” to ensure data aren’t changed by the communication link – not that they were not tampered with.

    “The keys, in practice, represent a shared secret between”. Right – how are you going to share that secret? With any ciphering algorithm, if the algorithm has no known flaws and is well know, it’s only the key to warrant data security. Thereby you have to ensure both parties have the keys and they cannot be easily get. And how do you that? Embedding the key in the executable is a risk. Storing keys in accessible files is a risk – there are not outsiders only, insiders may be a risk too. Using the same key over and over may be a risk – but if you use session keys you have to be able to transmit them to the other end securely.
    That’s why I was talking about “sorcerer’s apprentices” – security is a “all or nothing” approach, “half securiry” usually means “no security”, encrypting a buffer is easy, but if you can’t handle keys properly it’s useless. Ok, storing the keys is not a Datasnap responsibility – but the key exchange?

  7. The package installs, the tests pass, and the filters work for tables with few records. For one table which 300 records and 13MBytes it fails and for another which has 10 000 records and 400Kbytes also. The client side of the Datasnap gives error here, or returns a “Read error”: (lzo.inc)

    assert(result=destB[0]);

    In fact, there appears to be no filter working as soon as the table is large enough. (using TDSProviderConnection on the client side and tested server and client on local computer).

  8. Thanks for the filters – I’ve mentioned them in my DataSnap 2010 white paper, as well as my Delphi 2010 DataSnap Enhancements session (in a few hours here at SDC 2009).

  9. @Charles & @Atmapuri
    No, there are not *known* bugs in DSFC.
    Owever, there are some bugs in DataSnap that may affect DSFC. With Delphi 2010 Update Pack 2 those bugs should be resolved. I’ll write a post about this asap.

  10. I’ve just installed Delphi 2010 Update Pack 4&5 (superseders of updates 2&3) and still I have no luck working with DSFC with “big” tables (100-200 records or more).
    I’ve seen something mentioned in the release notes for updates 4&5: “DataSnap Filters is raising exception some times, more frequently in debug mode. After some test with Adrian, he knows where is the problem and look likes is easy to fix.”. Is this related to what makes DSFC working bad? If so, why it’s still not working properly after installing the updates?

  11. I didn’t see in your comparison the time without the filters. I would like to use some filter to maximize the performance from my datasnap server (Delphi 2010). Could you say something about that? Thanks.

  12. @Bruno
    The speed test program test the speed of filters only. It’s not a fully datasnap system, so cannot test the speed without filters. You can create a small server for this kind of test. And, if you want, I’ll publish result here.

  13. Hi Daniele I’m using RAD Studio 2010 Update 1. I’m trying to use the filter LZO in the component DSTCPServerTransport, but when I start my application an exception ocurred:
    “Exception EReadError in module rtl140.bpl at 0006B8EA.
    Error reading DSTCPServerTransport1.Filters: Error reading TTransportFilterItem.FilterId: Illegal argument”

    Am I doing something wrong?

    Thanks.

    Best Regards from Brazil!

  14. Hi Daniele, after I upgrade my RAD Studio 2010 (Update 4 and 5) the filters does not works, yet. What can I do to fix that? Thanks.

  15. Daniele, I’ve already done the test required last week. Well, the performance of DataSnap decreased a lot when I used the LZO filter over a VPN, so I decided not use filters. Thanks!

  16. @Bruno
    There is a QC issue related to the speed of datasnap filters.
    This slowness is not related to the LZO filter itself (it is faster that the default ZLib filter), but is related to a generic slowness in filter using

  17. I am having to same problem as Bruno.

    “Exception EReadError in module rtl140.bpl at 0006B8EA.
    Error reading DSTCPServerTransport1.Filters: Error reading TTransportFilterItem.FilterId: Illegal argument”

    What is the solution?

  18. I have the same problem as Jon and Bruno!!
    The error raise in TTransportFilterFactory.CreateFilter function, because the TDBXObjectStore.GetObjectFromName returns NIL. Only the ZLibCompression is correct.

  19. Hi,
    I make same test for lzo. And i think there is a problem in
    TLZO.DeCompress function.

    I Compress a file sized 1.500 KB. And compressed memory is 98 KB. and again i decompress same memory and i got access violation.

    In TLZO.Decompress
    OutData := GetMemory(100 + Length(Data) * 2);
    Code is not correct, because Data is 98 KB and * 2 = 196 KB but output memory is 1.500 KB.
    and GetMemory for outdata is not correct. and i got access violation error. I Change the code that
    OutData := GetMemory(100 + Length(Data) * 40);
    and now it is working, but sure GetMemory can not be correct again. And it can be use more memory.
    I think we need to correct getmemory size.

    Best Regards

  20. Aumentando o BufferKBSize de 32 para 64 o filtro lzo funcionou perfeitamente para grandes quantidades de dados

  21. Hello Daniele! Sorry to resume this topic, but I need help!

    It is possible to use data compression (zLib or LZO) with DSHTTPService? I’m with difficulties because the Filters are not triggered using HTTP. Data compression only works using TCP, HTTP not had success.

    Do you have any suggestions for using data compression using HTTP (Rest)?

  22. @eurides.baptistella
    You cannot use filters with http ds services. However, if you need http compression, you could create an isapi and publish it behind a webserver gzip enabled.

  23. Fiquei com algumas duviads. Sobre o FireMonkey uma curiosidade e9 componentes como o INDY, synapse, acesso ao Spool do Windows e apesar de ne3o ter conhecimento algum de Mac acredito que o mesmo deve ter um tipo Spool para controle de impresse3o, como isso sere1 feito no FireMonkey ?? ou seja quando parte para gerencimento de Hardware, impresse3o, internet como o FireMonkey vai trabalhar a esse respeito ou a forma que je1 usamos continuara para as outras plataformas com os Componentes je1 usados. Sobre o Fast Report 5 sere1 que nessa verse3o quando abrirmos o Designer dele teremos acesso aos eventos e objetos pelo Delphi ou vai continuar em modo Modaltendo que ser feito o acesso atraves de FindObjecteu em particular tinha muitos relatorios no QuickReport usando muitos eventos e a migrae7e3o para o Fast Report 3 na epoca era muito complicada pois teria que mexer muito no codigo devido essaforma diferente de acesso que o Fast Report tem assim optei por migrar para o Report Builder que apesar de ser muito semelhante ao Fast Report tem o acesso simplificado aos Objetos e eventos atraves do delphi assim como o QuickReport.

Comments are closed.