Dictionary Loading Issues

Jul 28, 2012 at 11:53 AM

I am using build 67418 development version on 32-bit Windows OS. The server for my deployment is using Windows 64-bit OS. I am using .NET 4.0.

I have the Dictionary running in a Windows service so that it is persisted long term. I am using a string as the Key, and the Value is a class that contains getters/setters for string, bool, ushort, and IEnumerable<string> types.

If I load the Dictionary with many instances of keyed values, I can stop/restart my Windows service and the Dictionary reloads the correct values from the file system. I can also power cycle the computer and everything is properly loaded from the file system to dictionary.

If I change values in the dictionary, the Dictionary seems to function properly as long as I do not stop/restart my Windows service. When I stop/restart my Windows service, I encounter various problems when I try to reload the Dictionary after modifying the data set. The issues range from a loss of data to a corruption of one or more of the memory-mapped files.

Am I trying to use the Dictionary in a way that was not intended or is this a bug?

Thanks...

Jul 28, 2012 at 12:38 PM

I have been thinking about my issue, which is likely related to the changes in string length for modied value properties. If I pad all strings so that they are fixed length and eliminate the IEnumerable<string> type, which seems to be problematic, will this work in all cases?

Jul 28, 2012 at 10:19 PM

I modified my code so that all properties are fixed length for the Dictionary Values. It seemed to work for a limited number of cases when I power cycled or stopped/restarted my Window service. However, after running more thorough tests, some of the data was lost after retarting the service and then adding and modifying more items. I have not yet expeienced corrupted files that caused my Windows service to throw an exception this go around. I will try to create some additional automated tests to try to discover what is happening.

Thanks...

Coordinator
Jul 29, 2012 at 6:41 PM

Hi,

In theory it should work, but I haven't tested it too much with reloading of dictionaries. Could you provide a copy of the class you are using?

I think corruption could occur if the files are not flushed/closed properly. The Dictionary<T,U> has a finalizer, but it could be that your service shuts down before it is being closed (just guessing until I can test and reproduce). I could make it inherit IDisposable as well so that you can dispose it properly yourself.

Are you using the dictionary as a cache service?

Thanks,
Mikael 

Jul 30, 2012 at 10:37 AM

Mikael,

After moving all the code to a small Windows Form application where testing is a lot easier, I think I found what caused the issue. I had ensured that the client process waited until the four mapping files were created, but was not actually checking to see that the ' create new' dictionary operation had completed. I believe there was a race condition. I am now testing that the IsInitialized flag is true before allowing any client operations to occur. This appears to work.

IsInitialized = false;

MemoryMappedDictionary = new mAdcOW.DataStructures.Dictionary<string, Member>(@"C:\MyPath\", 100000, true, "FileName");

IsInitialized = true;

I am doing some extensive testing now to confirm that the dictionary functions properly with power cycling/process restarts. It appears to work with the 'padded' fixed field version. I am repeating the tests for the variable field length strings in the dictionary.

My plan is to use your mmf dictionary, and maybe some of the other collections, for a wide range of in-memory cache services for items that need to be persisted long-term and that are frequently accessed. I am using SSD storage for improved write performace when the data is updated, which is relatively low frequency relative to the reads.

Thanks,

Warren

 

Jul 31, 2012 at 11:58 AM
Edited Jul 31, 2012 at 12:23 PM

Mikael,

I have completed some charaterization testing and still encounter a variety of errors when I try to reload the dictionary. I will send a detailed decription and the class file that I am using to your other email address. I also tried reducing the number of properties in the class to one string and two unshort types. The error results are similar.

Thanks...

Warren

 

Aug 13, 2012 at 8:56 PM

Mikael,

I downloaded build version 68550 and all of the issues that I had observed appear to be fixed. I have only tested the new update with my small test application, but I will create some extensive test cases scenarios over the next couple of week to verify very large volumes of actual data.

Thanks again for your support.

Warren

Aug 13, 2012 at 9:50 PM
Edited Aug 14, 2012 at 11:30 AM

After running my test application and closing it, an exception was raised in Array.cs in the following method:

public bool KeepFile {
get{ return ViewManager.KeepFile; } 
set{ ViewManager.KeepFile = value; }
}

ViewManager is null.

I had to manually clean up the mmf in the target directory before I could rerun the application. I will try to do some more characterization testing to see if I can isolate the issue(s).

Regards...

Warren

Aug 14, 2012 at 11:39 AM

The ViewManager code appears to be referenced only in the List.cs and Array.cs files. I have not yet determined how this code in the Array.cs files is being called when I close my test application. I am not using the Array.cs functionality. The problem is reproducible by deleting any mmf's that exist and opening the application, waiting until the mmf's are created, and then closing the test application.

When the test application is first opened, all the functionality that I have tested appears to be working properly. I have not observed any data overwriting and the clear seems to work properly.

Regards...

Coordinator
Aug 15, 2012 at 9:50 AM

Hi,

When loading up existing files, you should either use the Dictionary like:

using( var dict = new mAdcOW.DataStructures.Dictionary<stringMember>(@"C:\MyPath\", 100000, true"FileName") )
{
   ...
}

or call: dict.Dispose() at the end.

This ensures that everything is closed and cleaned up. I added IDisposable in order to file resources to get cleaned up. Let me know if this helps. If you look at the unit tests for reloading existing data you can see how this is done.

Thanks,
Mikael 

Aug 16, 2012 at 7:38 PM

Mikael,

I tried both techniques. The Dispose() method works properly, and the dictionary is loaded properly (previous data sets) when the application starts. However, the using(...){...} form did not work. Althought the dictionary appears to be created properly, it threw an exception and was not perperly initialized. I tried both techniques in the test application that I sent you. Anyway, thanks for your assistance since the dict.Dispose() method seem to do the trick. I really appreaciate your willingness to help.

Best regards,

Warren

Coordinator
Aug 17, 2012 at 4:17 AM
Hi,
I'll test "using" myself, and add a unit-test for it. Should be the same as it calls dispose at the end. Glad I could enrich the project :D

thanks,
-m
On Fri, Aug 17, 2012 at 1:13 AM, highdownts <notifications@codeplex.com> wrote:

From: highdownts

Mikael,

I tried both techniques. The Dispose() method works properly, and the dictionary is loaded properly (previous data sets) when the application starts. However, the using(...){...} form did not work. Althought the dictionary appears to be created properly, it threw an exception and was not perperly initialized. I tried both techniques in the test application that I sent you. Anyway, thanks for your assistance since the dict.Dispose() method seem to do the trick. I really appreaciate your willingness to help.

Best regards,

Warren

Read the full discussion online.

To add a post to this discussion, reply to this email (mmf@discussions.codeplex.com)

To start a new discussion for this project, email mmf@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com