Storage files

Dec 22, 2010 at 2:19 AM

Hi,

Very cool library, thanks a million. I'm thinking of using this in a project, and was wondering how to get rid of residual files left behind by the library. All my storage requirements are temporary and as such need a way for my project to 'clean up' after itself by removing these storage files when done. How can I do that?

Gideon

Coordinator
Dec 22, 2010 at 11:17 AM

In theory it should clean up itself with finalizers. Could you post or e-mail me a sample  program where this is not working and I will look into why it fails.

Regards,
Mikael 

Dec 22, 2010 at 11:54 PM

Hi,

Thanks yes, I figured that out after letting the app run through. I need some advice though, I am looking for a fast and efficient way to store 5 million records (8 fields) “in-memory”. I’ve managed to develop a compressed data table control which keeps memory consumption down to about 900MB, compared to almost 3GB using the standard .NET tools of the trade (DataTable – what a mess that is). However, I noticed my memory usage is under 20MB using your library, except that loading time into the MMF file is extremely slow. My current compressed table control loads all 5m records into memory in about 70 seconds, but I find the serialization you are using is extremely slow. Any chance of plugging in another serialization library instead? I know it is always a trade-off between memory and speed, but it would be great for .NET developers to have a fast and efficient alternative to the DataTable (unusable for large record sets). Something that can handle a DataRow, store it on-disk locally, but really fast.

Any ideas?

Regards,

Gideon

From: Wobba [email removed]
Sent: Wednesday, December 22, 2010 7:18 PM
To: gideon.vos@cxcglobal.com
Subject: Re: Storage files [mmf:239193]

From: Wobba

In theory it should clean up itself with finalizers. Could you post or e-mail me a sample program where this is not working and I will look into why it fails.

Regards,
Mikael

Coordinator
Dec 23, 2010 at 11:26 AM

"Serialization" is slow ;) My code includes several serializers, and it benchmarks which one to use for your data type. Can you tell me which serializer it uses, and perhaps say something about your data structure?

The by far fastest way to serialize/deserialize is to have your data in a defined struct. Then the conversion is done by memcopy. But using a struct takes some work if you have strings and lists in your object.

It´s also possible to add your own serializer/deserializer to the code by inheriting from the ISerializeDeserialize interface.

If you give me a copy of the data structure you are using I can check which serializer would be best, or if there´s is something else which slows down the building. And is it a Dictionary<T,V> you are using? The protobuf-net serializer should be fairly quick, but I have not upgraded it to the latest version.

Regards,
Mikael