|Number of votes:||3|
Updated for R2 SP1 and R3 2013 02 13. Notes for each version are added inline.
EPiServer Commerce exposes three distinct carriers of catalog subsystem data: DTOs, Objects and MetaObject. Different abilities, usage, approach and implementation govern which way to go and what parts should be used in certain situations when populating a CMS page. This article solely discusses the Catalog subsystem, relying on the MetaDataPlus metadata engine. There are a number of ways of loading catalog-specific data to front-end web pages and back-end processing.
A couple of the main objectives of this article are to describe the usage of catalog entry ResponseGroups and various ways of building a targeted set of data for different purposes.
The system defines a wide range of predefined chunks of metadata that support e-commerce and web shop management. For sellable and non-sellable products, along with other catalog entities, this set of metadata is predefined, but extensible. Custom data can also easily be attached to the different catalog entities by custom MetaClasses and MetaFields. Parts of the catalog system can also be extended by dictionaries for different purposes. Both the system-defined data and the custom defined data can be retrieved in several ways as this article explains.
The platform uses custom-designed MetaClasses in the catalog namespace to describe a specialized part of the complete information structure for various types of catalog elements. A variety of MetaClasses designed for entities as product categories (nodes), product packages and variations, etc. are the basis for custom information storage. An example is that a vacuum cleaner differs from a computer in the set of custom metadata the item should carry when exposed or used with business logic. Processor type and network interface are, for instance, not applicable to vacuum cleaners. A large part of the system-defined metadata is however common, like units in stock, cross sell products and preorder quantity that would be defined for both types of products.
A MetaClass endures a collection of custom-defined fields/properties together with partial system metadata as fields/properties. This is done by specifying a set of MetaFields of different data types for each type of entry (like a computer or a vacuum cleaner). Multiple catalog entities are supposed to, and most likely will, share the same MetaClass definition and information scheme.
System-defined and custom metadata can be retrieved in several ways and different formats. Information can be retrieved as strings and arrays of strings, regardless of database-defined type, targeted the front-end site or strong-typed data for back-end processing or even fast loading HashTables from serialized data. Some metadata is always retrieved strong-typed whereas the format for portions of the metadata can be chosen for when retrieved.
The general idea with MetaDataPlus metadata engine is to allow for customized extensions of system-defined objects as well as the creation of new definitions by Meta Classes and Meta Fields. At the bottom-line it is also notable that MetaDataPlus is designed to give the resulting database structure the fastest possible performance.
MetaDataPlus supports a number of common Database types as well as additional structures essential for an e-commerce platform. Available types can be divided into two major categories; MSSQL Common Types and product-specific MetaDataTypes. Examples of product-specific types are Date, Email, URL, File, ImageFile, LongHtmlString, different types of dictionaries and more.
MetaFields are used to describe different types of columns of a few tables in an EPiServer Commerce database, without tampering directly with the database. Upon MetaClass and MetaField modeling the container data tables, language-specific tables and field value change history tables are managed, as well as a corresponding set of stored procedures and full text indexes.
An Instrument for granularity in data loading is the ResponseGroup. The functionality gained from Response Groups instructs the platform which parts of the complete set of system-defined data to load in a given situation. In other words, the Response Groups govern partial data loading behavior performed in one single call to retrieve information for catalog items.
As the name implies, the Response Group articulates how to build a response from cache or database with a group (set) of conditions. It is possible to form a group of circumstances as the Response Groups are bitwise and therefore can be used to load multiple parts of the complete set of entry data in one single database hit. This is used to achieve a fine coarse load of data for a targeted operation.
The pattern is similar in data loading for catalogs, nodes and entries in the catalog subsystem. The following examples and discussion adhere to entry specific (leaf level) data loading as this incorporates more than one approach for EPiServer Commerce.
The DTOs (Data Transfer Objects) are somewhat cumbersome to use in the beginning but rather straightforward when you get used to them. This part of the API mainly contains strong-typed system-defined fields for different objects like catalog entries, nodes and catalogs. Custom fields added to the entry MetaClass can be reached as serialized data with DTOs. Custom metadata can by this be fast and easily retrieved as HashTables. MetaField type checks may have to be performed as the system defines two types of MetaFields, complex types (ComplexFields) and simple types (ElementaryFields). Examples of complex types are different kinds of dictionaries and files stored within MetaDataPlus (not the Asset system). Examples of simple types are strings, integers and DateTime. It is the complex types that may need extra processing to be handled correctly.
When a DTO-row (or several) is retrieved there is a column at the [dbo]. [CatalogEntry] table named [SerializedData] that holds custom added metadata in all languages defined for the catalog in focus. This column can be used for fast retrieval of stored custom product attributes to populate web pages.
DTOs are basically typed datasets. For example, the CatalogEntryDto (leaf level data carrier) consists of nine tables that will be filled with data differently, depending on the Response Groups used. Each of these tables is typed and exposes system-defined metadata for entry level items like tiered pricing, inventory, assets (for example high resolution images and downloads) and associations (for example cross sell and upsell).
Objects ("the Object-API" – merely terminology) are leaner to work with as a lot of system-defined and custom-defined information is assembled and accessible for various uses. When loaded, gathered and converted to a format for ease of use, an Entry (or Entries) contains all data needed for read-only actions. The data is exposed in this way for both front-end consumption and back end use.
The "object API" does not depend on, but is more efficient when serialized data is stored in the [SerializedData] column of the [dbo].[CatalogEntry] table mention before. Perhaps developers could argue that in some areas of what an Entry contain, the notation used and format of the retrieved data is not as strong-typed as it could be. Strings or arrays of strings come in certain sections of an Entry as ItemAttributes (collection of ItemAttribute), but this arrangement is meant for convenient population of the front-end. Laid out this way, it makes development faster as no hesitation exists about the type loaded and how to cast.
Loading of data by the object-API is more time consuming, compared to DTO-rows, as it incorporates more platform work in terms of assembling a larger set of data and exposing it in well-known formats. The set of Entry metadata is however more "complete" (subjective statement) and more convenient to use than DTOs or MetaObjects metadata.
With MetaObjects (MetaDataContext) EPiServer Commerce points to a small set of system-defined metadata fields together with all the custom metadata fields attached to a MetaClass. The set of custom-defined attributes is loaded separately by the MetaObjects, in its own manner, distinguished from most of the system-defined metadata. Data retrieved in this way is suitable for read-write and back-end operations.
The number of custom fields attached to a MetaClass can vary from a few to more than a hundred depending on the kind of products exposed, catalog modeling and the potential use of surrounding information systems.
While discussing the loading of catalog data, thoughts might arise regarding the file-based search indexes that an e-commerce site relies on for quick search, faceted navigation and sometimes partial analysis. A balanced padding of custom fields in the database and in the search indexes can be the path to success as selections of data can be loaded to the front end site by both file based indexes and database/cache. Out of the box EPiServer Commerce comes with configuration and libraries for Solr and a fully functional Lucene.net. The later can be used for smaller shops and catalogs. Solr is relying on Java and Tomcat but is much more an enterprise-like search utility. If neither Solr nor Lucene is appropriate then a third-party search engine should be connected to the e-commerce store.
The bottom line is the well-known fact that loading data should be as finely granular as possible to avoid wasting CPU cycles, memory and I/O reads.
As stated previously in this article we use Entry-level loading as an example. For Objects and DTOs as well as for different entry level types (like SKUs or Packages), behavior differs somewhat when using the same Response Group. All ResponseGroups are not applicable at all times. The following table shows the available options for a varied amount of data retrieved at entry level within a given call. As indicated above, several Response Groups can be combined in a single operation for data retrieval.
Public enum ResponseGroup
Note: ResponseGroups are further improved in future versions.
The CatalogEntryDto, used in this example, contains the following dataset tables. All tables are found in the Mediachase.Commerce.Catalog.Dto namespace:
|Name||Dataset table name||Description|
|Inventory||InventoryDataTable||Holds Backorder, Preorder information and stores stock keeping data|
|CatalogItemSeo||CatalogItemSeoDataTable||Stores URL, Title, Keywords and Description for the languages chosen at Catalog level|
|Variation||VariationDataTable||Data regarding Weight, Fallback price, max and min orderable amount and related information|
|CatalogAssociation||CatalogAssociationDataTable||Associations like CrossSell, Accessories, Upsell etc. Whatever names and use can be specified|
|Merchant||MerchantDataTable||No UI is present for adding merchants, direct DB inserts of Merchants must be done. UI exist for the use of Merchants when they are added|
|CatalogEntry||CatalogEntryDataTable||Basic system properties, like dates for availability, friendly name etc. Correspond to the system defined part of the MetaClass information|
|SalePrice||SalePriceDataTable||Tiered pricing (differential pricing) defined for individual Entries and specified customers. Note: In R3 the pricing storage and API are changed. This table will remain for compatibility.|
|CatalogItemAsset||CatalogItemAssetDataTable||Usage, and linking, of digital assets in the Asset Management Subsystem. Also includes asset type and asset group name|
|NodeEntryRelation||NodeEntryRelationDataTable||Not currently used for DTOs. Relations between nodes and entries are stored in a CatalogRelationDto in the NodeEntryRelationRow table|
All of the ResponseGroups have relatively straightforward names to describe the data they will load, as the following list shows. If the cumulative set of ResponseGroups used contains CatalogEntryFull, all of the applicable data present is loaded to the entry/DTO, or to an array of entries or as several rows in DTO-tables. Exceptions to the straightforward rules are outlined in the list below.
Because data loading is on the agenda in this article we should touch upon a couple of different ways of bringing data from the cache or database. The following summary of data formats will probably be useful as a guide, even if it is a short version in this article.
The usage of DTOs is a quick way of loading a lot of data, even when the CatalogEntryResponseGroup.ResponseGroup.CatalogEntryFull is used. Strong-typed data along with different kinds of binary data are features of DTOs.
Objects are slightly slower to load as a lot of information will be brought up and put together, mostly as strong-typed properties and collections. Assets, for instance, come in the form of a Mediachase.Commerce.Catalog.Objects.ItemAsset array. Another example is the Associations as a Mediachase.Commerce.Catalog.Objects.EntryAssociation array, with ease of use and well-known data types and structures. The following screenshot from Visual Studio shows Assets and Associations along with a few other properties.
Another example is when a Package is loaded a Mediachase.Commerce.Catalog.Objects.Entries type of an Entry array is assembled with a lot of data for each Entry. Each Entry has, for example, SalePrices in the form of Mediachase.Commerce.Catalog.Objects.SalePrices with a SalePrice array and the rest that adheres to the ResponseGroup chosen. This is outlined in the following image.
When loading Entries by the Object-API, defined custom properties are found in an ItemAttributes collection of Mediachase.Commerce.Catalog.Objects.ItemAttribute exposed as string or string at the individual item's level. Also note in the image below the Images and Files properties added by MetaDataPlus (not Assets) in the ItemAttributes collection.
The files and images stored as array of ItemFile and array of Image also come in a handy well-known format, which means that they are easy to work with. The rest of the properties in the ItemAttributes collection are, as expected, int32 & System.DateTime and alike.
MetaObjects is a fast way of loading custom metadata where strong-typed fields or HashTables can be retrieved. As stated previously the MetaObject contains basic system-defined info and custom metadata. MetaObjects can load differentiated HashTables for complex and simple field types along with different sets of custom and system fields on a MetaClass. The following example is for a single catalog entry in the MetaDataContext of the fallback language.
// Specific load, a single row. "EntryLabCode" is an Entry codeNote: MetaClasses also exist in the context of BusinessFoundation metadata engine, with another way of working.
CatalogEntryDto dto = CatalogContext.Current.GetCatalogEntryDto
// The only row available
CatalogEntryDto.CatalogEntryRow catRow =
// Get a grip of the metadata object in the chosen language (MetaDataContext)
MetaObject mo = MetaObject.Load
(CatalogContext.MetaDataContext, catRow.CatalogEntryId, catRow.MetaClassId);
// Get Complex vs. Simple fields as HashTables
Hashtable simple = mo.GetElementaryFieldValues();
Hashtable complex = mo.GetComplexFieldValues();
// Get specific types
int myInt = mo.GetInt32(mc.MetaFields["LabIntegerField"].Name);
MetaStringDictionary myDict = mo.GetStringDictionary(mc.MetaFields["LabStringDictField"].Name);
// Can load custom and system fields separately
MetaClass mc = MetaClass.Load(CatalogContext.MetaDataContext, catRow.MetaClassId);
foreach (MetaField mf in mc.UserMetaFields)
foreach (MetaField mf in mc.SystemMetaFields)
HashTables created from SerializedData is a very fast loadable chunk of data. A single column of a DTO-row with the Info/Request ResponseGroup is sufficient. The drawback is the rather weakly typed HashTable used for custom metadata. Note below how the different languages are loaded by sending the language string to the method GetValues().
IFormatter formatter = new BinaryFormatter();Note: The GetMetaFieldValues method in Mediachase.Commerce.Catalog.Managers.ObjectHelper fetches the SerializedData if it´s not null and cast it to appropriate types. If SerializedData is null the method loads the actual MetaObject in a standard way and values are retrieved from that object.
MetaObjectSerialized serialized = null;
if (catRow["SerializedData"] != DBNull.Value)
serialized = (MetaObjectSerialized)formatter.Deserialize
Hashtable values = null;
Hashtable values2 = null;
// Gets strings in a HashTable
values = serialized.GetValues("en-us");
values2 = serialized.GetValues("es-es");
As the following image shows, complex fields prompt for more coding when different types are retrieved. This is easiest done by a switch construct with the enum MetaDataType. Simple values are retrieved as easy-to-use strings.
The method chosen to load data in a given situation can be expressed in many ways as this article shows. It is hard to come up with general recommendations for a data loading scheme in a given situation, too many parameters can influence in any situation. However, a couple of tips can be expressed as the following:
The message to bring back at the development computer keyboard for a reader of this article is that several measures and paths are achievable for high performance data loading with EPiServer Commerce.