Introducing The Dynamic Data Store
Ok, so maybe “Introducing” was a bit optimistic in this case as the cat has been out of the bag for a few weeks on this subject.
The Dynamic Data Store is a new component which will be shipped with EPiServer CMS 6 but forms part of a framework we are building up here at EPiServer so it should also be available in the future with other EPiServer products such as EPiServer Community.
A typical conversation about the Dynamic Data Store normally goes something like this:
Fred: So what is the Dynamic Data Store?
Jim: It’s a store for storing dynamic data obviously!
Fred: Hmm, isn’t most data dynamic in some way?
Jim: Yeah I guess.
Fred: So what makes your store so special?
Jim: Well, the dynamic in Dynamic Data Store actually refers to the structure or shape of the data.
Fred: What are you talking about?
Jim: For example, storing data in a database using Entity Framework or NHibernate requires you to design and compile a class when developing your application. This works really well when you know the shape or structure of your data at compile time. EPiServer CMS has a few features where the shape or structure of the data isn’t actually known until runtime.
Fred: Examples please!
Jim: XForms and Page Types for starters!
Fred: Ok, so you’ve cracked the mind boggling problem of storing name/value pairs in a database. Congratulations!
Jim: Basically yes. Well….there is a bit more to it that than.
Fred: Go on.
Jim: We’ve sprinkled a little bit of magic powder on top of it.
Fred: Have you been drinking again?
Jim: That would be telling. Anyways….what I’m trying to say is that, yes, the Dynamic Data Store stores names/value pairs or property bags as we like to call them, BUT the magic is that they are treated exactly the same way internally as compile time objects.
Fred: Compile time objects?
Jim: .NET classes to you my friend. Actually to put it more correctly, compile time objects are treated exactly the same as property bags internally.
Fred: How many have you had?
Jim: One or two but the point is now we have a way of storing our dynamic data structures such as XForms in a relational database in relational form.
Fred: And that is good because?
Jim: Well, searching for one. For example, an XForm post suddenly becomes a first class artifact in the EPiServer CMS database. Think of all the useful reports and queries you could do with them.
Jim: Then there’s LINQ support.
Fred: LINQ?
Jim: Yeah you know, Language INtegrated Query. With the Dynamic Data Store you can can make queries against property bags using LINQ.
Fred: Sounds interesting. If I wanted, could I store my own data in the Dynamic Data Store?
Jim: Yes you can my friend. Both dynamic / property bags / name-value pairs (whatever you want to call them) and normal .NET class instances can be saved in the Dynamic Data Store.
Fred: Sounds like a configuration nightmare. I bet there are 50 XML files to edit every time you want to save a new type right?
Jim: Nope. It’s a code only solution. No XML files (web.config apart), just create your property bag or .NET class instance at runtime, create a store and save.
Fred: Got any code examples?
Jim: Sure. Here’s how you save a property bag:
using EPiServer.Data;
using EPiServer.Data.Dynamic;
PropertyBag pb = new PropertyBag();
pb.Add("FirstName", "Jack");
pb.Add("LastName", "Williams");
pb.Add("DateOfBirth", new DateTime(1973, 05, 25));
pb.Add("Gender", 'm');
DynamicDataStore store = DynamicDataStore.CreateStore("People", true, pb.GenerateTypeBag());
Identity id = store.Save(pb);
Fred: How about reading it back?
Jim: There are 3 ways. Directly by the Id returned from the Save method:
PropertyBag pb2 = store.Load(id);
or using Find:
var propertyBags = store.Find("FirstName", "Jack");
or using LINQ
var propertyBags = from pb in store where pb["FirstName"] == "Jack" select pb;
Fred: Cool. Can I do more complex stuff with the LINQ support?
Jim: What like:
var propertyBags = from pb in store
where
((DateTime)pb["DateOfBirth"]) > new DateTime(1976, 01, 01) &&
((string)pb["Gender"]) == "m"
select pb;
Fred: Yeah
Jim: Nah!
Fred: What?
Jim: Just kidding. Actually the LINQ support is quite extensive. In many cases better than a well know LINQ toolkit you can download from CodePlex.
Fred: Ok, you said I could also store normal .NET class instances in it like I can with Entity Framework or NHibernate.
Jim: Sure. Like this:
Person p = new Person()
{
FirstName = "Jack",
LastName = "Williams",
DateOfBirth = new DateTime(1973, 05, 25)
};
DynamicDataStore<Person> store = DynamicDataStore<Person>.CreateStore("People", true);
Identity id = store.Save(p);
So there you go. That’s the introduction to the Dynamic Data Store or DDS as we call it internally. I will post more detailed information over the coming days and weeks.
Code samples can be found here and documentation here
Have fun!
I saw this at the EPiServer Partner day (as ably demonstrated by Mats and Peter). Count me impressed..
Its a really nice feature which if used sensibly will come in handy in a lot of scenarios.
However I can't help but worry that such power brings great responsibility. Despite the promised performance you wouldn't want your DynamicDataStore becoming a DynamicDataDumpingGround.
/ Mark Everard
Quite agree. We are not trying to replace Entity Framework or NHibernate with this. We built it initially to solve our own problems with XForms but quickly came to the conclusion that typed support (i.e. .NET classes) came almost free as it is a thin wrapper around the property bag support.
I would say that for complex solutions where you are storing .NET classes and require very high performance then go with EF2 or similar.
That isn't to say that we expect the performance of the DDS to suck. We are already testing with high volumes (1 million plus rows) plus we will have caching in the final solution.
Speaking as a technical architect I can see the convenience, but I can't help thinking that this might turn out to be a weapon of mass destruction...
...btw you're performance testing this to more than 1m rows, but how wide are the rows in the test sample? My understanding of the DDS is that you're using a "BigTable"-style structure for the data storage that adds columns on the fly...
Yes we are using a big table approach but we don't add columns on the fly. The default big table will ship with a set amount of columns (currently 50 columns). Objects spill onto more than one row if they need to (e.g. they have more string properties than string columns in the big table). As a site owner / web master / dba you can add more columns to the big table yourself to ensure that objects are more likely to fit on one row.
You can also define your own big table and map your objects to that instead. Obviously you lose a little of the convenience as you will need to create your own table.
There is no default indexing in the CTP but we are looking at this. Obviously it's hard to know what columns to index by default, too many and inserts will be slow, too few and queries will be too slow.
Code samples project is now available here:
http://world.episerver.com/Download/Items/EPiServer-CMS/Version-6/EPiServer-CMS-6-CTP-2/Dynamic-Data-Store-Examples/
That link redirects me to the access denied page, even though I'm logged in. Was this supposed to be available to all?
Nice post! Informative AND entertaining. :)
Following is the code I got from DDS sample to save dynamic data
DynamicDataStore store = DynamicDataStoreFactory.Instance.CreateStore("People", storeMappings);
// Now we can save PropertyBags without having to have values for all the properties
PropertyBag pb = new PropertyBag();
pb.Add("FirstName", "Jack");
pb.Add("LastName", "Williams");
pb.Add("Gender", "m");
// Don't know his date of birth yet
// If we hadn't mapped the store explicity first then only the FirstName, LastName and Gender
// would be mapped
store.Save(pb);
PropertyBag pb2 = new PropertyBag();
pb2.Add("FirstName", "Anne");
pb2.Add("LastName", "Jackson");
pb2.Add("Gender", "f");
pb2.Add("DateOfBirth", new DateTime(1953, 01, 16));
store.Save(pb2);
When program program is executed, data couldn't be traced in tblBigTable, tblBigTableStoreInfo
Can you please help into it?