Catalog Entry loading and ResponseGroups in EPiServer Commerce

Updated for R2 SP1 and R3 2013 02 13. Notes for each version are added inline.

EPiServer Commerce exposes three distinct carriers of catalog subsystem data: DTOs, Objects and MetaObject. Different abilities, usage, approach and implementation govern which way to go and what parts should be used in certain situations when populating a CMS page. This article solely discusses the Catalog subsystem, relying on the MetaDataPlus metadata engine. There are a number of ways of loading catalog-specific data to front-end web pages and back-end processing.

A couple of the main objectives of this article are to describe the usage of catalog entry ResponseGroups and various ways of building a targeted set of data for different purposes.

Background of EPiServer Commerce metadata

The system defines a wide range of predefined chunks of metadata that support e-commerce and web shop management. For sellable and non-sellable products, along with other catalog entities, this set of metadata is predefined, but extensible. Custom data can also easily be attached to the different catalog entities by custom MetaClasses and MetaFields. Parts of the catalog system can also be extended by dictionaries for different purposes. Both the system-defined data and the custom defined data can be retrieved in several ways as this article explains.

The platform uses custom-designed MetaClasses in the catalog namespace to describe a specialized part of the complete information structure for various types of catalog elements. A variety of MetaClasses designed for entities as product categories (nodes), product packages and variations, etc. are the basis for custom information storage. An example is that a vacuum cleaner differs from a computer in the set of custom metadata the item should carry when exposed or used with business logic. Processor type and network interface are, for instance, not applicable to vacuum cleaners. A large part of the system-defined metadata is however common, like units in stock, cross sell products and preorder quantity that would be defined for both types of products.

A MetaClass endures a collection of custom-defined fields/properties together with partial system metadata as fields/properties. This is done by specifying a set of MetaFields of different data types for each type of entry (like a computer or a vacuum cleaner). Multiple catalog entities are supposed to, and most likely will, share the same MetaClass definition and information scheme.

System-defined and custom metadata can be retrieved in several ways and different formats. Information can be retrieved as strings and arrays of strings, regardless of database-defined type, targeted the front-end site or strong-typed data for back-end processing or even fast loading HashTables from serialized data. Some metadata is always retrieved strong-typed whereas the format for portions of the metadata can be chosen for when retrieved.

The general idea with MetaDataPlus metadata engine is to allow for customized extensions of system-defined objects as well as the creation of new definitions by Meta Classes and Meta Fields. At the bottom-line it is also notable that MetaDataPlus is designed to give the resulting database structure the fastest possible performance.

Meta Fields and Data Types

MetaDataPlus supports a number of common Database types as well as additional structures essential for an e-commerce platform. Available types can be divided into two major categories; MSSQL Common Types and product-specific MetaDataTypes. Examples of product-specific types are Date, Email, URL, File, ImageFile, LongHtmlString, different types of dictionaries and more.

MetaFields are used to describe different types of columns of a few tables in an EPiServer Commerce database, without tampering directly with the database. Upon MetaClass and MetaField modeling the container data tables, language-specific tables and field value change history tables are managed, as well as a corresponding set of stored procedures and full text indexes.

Response Groups in general

An Instrument for granularity in data loading is the ResponseGroup. The functionality gained from Response Groups instructs the platform which parts of the complete set of system-defined data to load in a given situation. In other words, the Response Groups govern partial data loading behavior performed in one single call to retrieve information for catalog items.

As the name implies, the Response Group articulates how to build a response from cache or database with a group (set) of conditions. It is possible to form a group of circumstances as the Response Groups are bitwise and therefore can be used to load multiple parts of the complete set of entry data in one single database hit. This is used to achieve a fine coarse load of data for a targeted operation.

The pattern is similar in data loading for catalogs, nodes and entries in the catalog subsystem. The following examples and discussion adhere to entry specific (leaf level) data loading as this incorporates more than one approach for EPiServer Commerce.

Catalog data carriers

DTO-based

The DTOs (Data Transfer Objects) are somewhat cumbersome to use in the beginning but rather straightforward when you get used to them. This part of the API mainly contains strong-typed system-defined fields for different objects like catalog entries, nodes and catalogs. Custom fields added to the entry MetaClass can be reached as serialized data with DTOs. Custom metadata can by this be fast and easily retrieved as HashTables. MetaField type checks may have to be performed as the system defines two types of MetaFields, complex types (ComplexFields) and simple types (ElementaryFields). Examples of complex types are different kinds of dictionaries and files stored within MetaDataPlus (not the Asset system). Examples of simple types are strings, integers and DateTime. It is the complex types that may need extra processing to be handled correctly.

When a DTO-row (or several) is retrieved there is a column at the [dbo]. [CatalogEntry] table named [SerializedData] that holds custom added metadata in all languages defined for the catalog in focus. This column can be used for fast retrieval of stored custom product attributes to populate web pages.

DTOs are basically typed datasets. For example, the CatalogEntryDto (leaf level data carrier) consists of nine tables that will be filled with data differently, depending on the Response Groups used. Each of these tables is typed and exposes system-defined metadata for entry level items like tiered pricing, inventory, assets (for example high resolution images and downloads) and associations (for example cross sell and upsell).

Object-based

Objects ("the Object-API" – merely terminology) are leaner to work with as a lot of system-defined and custom-defined information is assembled and accessible for various uses. When loaded, gathered and converted to a format for ease of use, an Entry (or Entries) contains all data needed for read-only actions. The data is exposed in this way for both front-end consumption and back end use.

The "object API" does not depend on, but is more efficient when serialized data is stored in the [SerializedData] column of the [dbo].[CatalogEntry] table mention before. Perhaps developers could argue that in some areas of what an Entry contain, the notation used and format of the retrieved data is not as strong-typed as it could be. Strings or arrays of strings come in certain sections of an Entry as ItemAttributes (collection of ItemAttribute), but this arrangement is meant for convenient population of the front-end. Laid out this way, it makes development faster as no hesitation exists about the type loaded and how to cast.

Loading of data by the object-API is more time consuming, compared to DTO-rows, as it incorporates more platform work in terms of assembling a larger set of data and exposing it in well-known formats. The set of Entry metadata is however more "complete" (subjective statement) and more convenient to use than DTOs or MetaObjects metadata.

MetaData-based (MetaObjects)

With MetaObjects (MetaDataContext) EPiServer Commerce points to a small set of system-defined metadata fields together with all the custom metadata fields attached to a MetaClass. The set of custom-defined attributes is loaded separately by the MetaObjects, in its own manner, distinguished from most of the system-defined metadata. Data retrieved in this way is suitable for read-write and back-end operations.

The number of custom fields attached to a MetaClass can vary from a few to more than a hundred depending on the kind of products exposed, catalog modeling and the potential use of surrounding information systems.

Additional options in data loading

While discussing the loading of catalog data, thoughts might arise regarding the file-based search indexes that an e-commerce site relies on for quick search, faceted navigation and sometimes partial analysis. A balanced padding of custom fields in the database and in the search indexes can be the path to success as selections of data can be loaded to the front end site by both file based indexes and database/cache. Out of the box EPiServer Commerce comes with configuration and libraries for Solr and a fully functional Lucene.net. The later can be used for smaller shops and catalogs. Solr is relying on Java and Tomcat but is much more an enterprise-like search utility. If neither Solr nor Lucene is appropriate then a third-party search engine should be connected to the e-commerce store.

The bottom line is the well-known fact that loading data should be as finely granular as possible to avoid wasting CPU cycles, memory and I/O reads.

The current Response Groups available

As stated previously in this article we use Entry-level loading as an example. For Objects and DTOs as well as for different entry level types (like SKUs or Packages), behavior differs somewhat when using the same Response Group. All ResponseGroups are not applicable at all times. The following table shows the available options for a varied amount of data retrieved at entry level within a given call. As indicated above, several Response Groups can be combined in a single operation for data retrieval.

Note: ResponseGroups are further improved in future versions.

The CatalogEntryDto, used in this example, contains the following dataset tables. All tables are found in the Mediachase.Commerce.Catalog.Dto namespace:

ResponseGroup usage

All of the ResponseGroups have relatively straightforward names to describe the data they will load, as the following list shows. If the cumulative set of ResponseGroups used contains CatalogEntryFull, all of the applicable data present is loaded to the entry/DTO, or to an array of entries or as several rows in DTO-tables. Exceptions to the straightforward rules are outlined in the list below.

• Assets
  - A straightforward database query loads data from the table [dbo].[CatalogItemAsset]
  - For Entries (object-API) the GroupName (for example Image, Downloads or Specifications) can be used to differentiate at the front-end site. The Click+Talk sample site has several examples of this.
• Associations
  - Data is loaded from the table [dbo].[CatalogAssociation]The SortOrder field in this data table is a useful property not exposed in UI
  - The AssociationName, for example Accessories or CrossSell, can be used to differentiate between distinct uses
  - For Entry (the object-API) an array of Entries along with metadata regarding the associations are assembled for use. DTOs load raw data rows
• ResponsGroups CatalogEntryInfo and Request loads the same set of data in this version
  - It is basic information where DTO and "Objects" differ to a smaller extent
  - For the Objects an ItemAttributes type is loaded with most of the custom metadata in the shape of an ItemAttribute array
  - For the DTO the custom metadata is loaded as serialized data that can be retrieved as a HashTable
• The Children ResponseGroup is somewhat special. It is used to load variations for a package/bundle as Entry objects. This ResponseGroup does not apply to DTOs, the actual loading is performed separately by an CatalogRelationDto where a CatalogEntryRelation table will serve the information
  - In other words, if the variations of a package are looked for in context of DTOs, the RelationDTO must be used. The CatalogRelationResponseGroup.ResponseGroup has three different enumerations for specific loading
• Nodes (applicable with Entry (object-API))
  A RelationDTO is used in the background to create a Nodes object (array of Node)
• Inventory loads basic data for inventory properties
• Variation loads standard variation data along with sales prices from tiered pricing and the fallback price (ListPrice)
• The ResponseGroup CatalogEntryFull loads all data available. Note: Entry pricing retrieval with R3 builds on a new API and can be retrieved separately.

Bringing up data in different formats

Because data loading is on the agenda in this article we should touch upon a couple of different ways of bringing data from the cache or database. The following summary of data formats will probably be useful as a guide, even if it is a short version in this article.

DTOs

The usage of DTOs is a quick way of loading a lot of data, even when the CatalogEntryResponseGroup.ResponseGroup.CatalogEntryFull is used. Strong-typed data along with different kinds of binary data are features of DTOs.

Object API

Objects are slightly slower to load as a lot of information will be brought up and put together, mostly as strong-typed properties and collections. Assets, for instance, come in the form of a Mediachase.Commerce.Catalog.Objects.ItemAsset array. Another example is the Associations as a Mediachase.Commerce.Catalog.Objects.EntryAssociation array, with ease of use and well-known data types and structures. The following screenshot from Visual Studio shows Assets and Associations along with a few other properties.

Another example is when a Package is loaded a Mediachase.Commerce.Catalog.Objects.Entries type of an Entry array is assembled with a lot of data for each Entry. Each Entry has, for example, SalePrices in the form of Mediachase.Commerce.Catalog.Objects.SalePrices with a SalePrice array and the rest that adheres to the ResponseGroup chosen. This is outlined in the following image.

When loading Entries by the Object-API, defined custom properties are found in an ItemAttributes collection of Mediachase.Commerce.Catalog.Objects.ItemAttribute exposed as string or string[] at the individual item's level. Also note in the image below the Images and Files properties added by MetaDataPlus (not Assets) in the ItemAttributes collection.

The files and images stored as array of ItemFile and array of Image also come in a handy well-known format, which means that they are easy to work with. The rest of the properties in the ItemAttributes collection are, as expected, int32 & System.DateTime and alike.

MetaObjects (MetaDataPlus)

MetaObjects is a fast way of loading custom metadata where strong-typed fields or HashTables can be retrieved. As stated previously the MetaObject contains basic system-defined info and custom metadata. MetaObjects can load differentiated HashTables for complex and simple field types along with different sets of custom and system fields on a MetaClass. The following example is for a single catalog entry in the MetaDataContext of the fallback language.

// Specific load, a single row. "EntryLabCode" is an Entry code
CatalogEntryDto dto = CatalogContext.Current.GetCatalogEntryDto
    ("EntryLabCode", new
    CatalogEntryResponseGroup(CatalogEntryResponseGroup.ResponseGroup.Request));

// The only row available
CatalogEntryDto.CatalogEntryRow catRow =    
    (CatalogEntryDto.CatalogEntryRow)dto.CatalogEntry.Rows[0];

// Get a grip of the metadata object in the chosen language (MetaDataContext)
MetaObject mo = MetaObject.Load
      (CatalogContext.MetaDataContext, catRow.CatalogEntryId, catRow.MetaClassId);

// Get Complex vs. Simple fields as HashTables
Hashtable simple = mo.GetElementaryFieldValues();
Hashtable complex = mo.GetComplexFieldValues();

// Get specific types
int myInt = mo.GetInt32(mc.MetaFields["LabIntegerField"].Name);
MetaStringDictionary myDict = mo.GetStringDictionary(mc.MetaFields["LabStringDictField"].Name);

// Can load custom and system fields separately
MetaClass mc = MetaClass.Load(CatalogContext.MetaDataContext, catRow.MetaClassId);

foreach (MetaField mf in mc.UserMetaFields)
{
    Console.WriteLine(mf.Name);
}
foreach (MetaField mf in mc.SystemMetaFields)
{
    Console.WriteLine(mf.Name);
}Note: MetaClasses also exist in the context of BusinessFoundation metadata engine, with another way of working.

HashTables

HashTables created from SerializedData is a very fast loadable chunk of data. A single column of a DTO-row with the Info/Request ResponseGroup is sufficient. The drawback is the rather weakly typed HashTable used for custom metadata. Note below how the different languages are loaded by sending the language string to the method GetValues().

IFormatter formatter = new BinaryFormatter();
MetaObjectSerialized serialized = null;

if (catRow["SerializedData"] != DBNull.Value)
{
    serialized = (MetaObjectSerialized)formatter.Deserialize
        (new MemoryStream((byte[])catRow["SerializedData"]));
    Hashtable values = null;
    Hashtable values2 = null;

    // Gets strings in a HashTable
    values = serialized.GetValues("en-us");
    values2 = serialized.GetValues("es-es");
...
...
} Note: The GetMetaFieldValues method in Mediachase.Commerce.Catalog.Managers.ObjectHelper fetches the SerializedData if it´s not null and cast it to appropriate types. If SerializedData is null the method loads the actual MetaObject in a standard way and values are retrieved from that object.

As the following image shows, complex fields prompt for more coding when different types are retrieved. This is easiest done by a switch construct with the enum MetaDataType. Simple values are retrieved as easy-to-use strings.

EPiLogue

The method chosen to load data in a given situation can be expressed in many ways as this article shows. It is hard to come up with general recommendations for a data loading scheme in a given situation, too many parameters can influence in any situation. However, a couple of tips can be expressed as the following:

• The granularity ResponseGroups offer should be considered and used
• Depending on how the platform and public downloadable EPiServer Commerce code is used and an Entry is loaded for addition to a cart, the lack of pricing information gives a non-descriptive null reference
• The public downloadable Visual Studio project that contains the CartHelper class should be altered for every specific customer. One example of this is that the CartHelper.AddEntry(MyEntry) method will throw a Null exception out of the box if custom MetaFields does not allow null values. The CartHelper.CreateLineItem() method will not set a default value for custom MetaFields. In cases like this let the CartHelper class initialize default value for not nullable Fields
• Loading a lot of Entries (Object-API) in the CatalogEntryFull ResponseGroup will look for all child Relations in a recursive manner. If not narrowed down the loading operation can be rather expensive as the amount of Entries perhaps is large (all their properties are loaded too)
• The API expose a few more MetaField types than Commerce Manager does, not all types have a visual designer. For example, when creating MetaClasses with code some confusion can arise when using “int” instead of “integer” as the MetaField data type

The message to bring back at the development computer keyboard for a reader of this article is that several measures and paths are achievable for high performance data loading with EPiServer Commerce.

Comments

Gatis Bergšpics May 23, 2012 05:49 PM

what could be the reason if CatalogEntryFull is used, but RelationInfo is returned null?

using:
var catalogSearchParameters =
new CatalogSearchParameters
{
JoinType = "Inner Join",
JoinSourceTable = "CatalogEntry",
JoinTargetQuery = "(Select CatalogEntryId, CatalogNodeId, SortOrder From NodeEntryRelation) NodeEntryRelation",
JoinSourceTableKey = "CatalogEntryId",
JoinTargetTableKey = "NodeEntryRelation.CatalogEntryId",
SqlWhereClause = "NodeEntryRelation.CatalogNodeId=" + preferredCatalogNode.CatalogNodeId,
OrderByClause = "NodeEntryRelation.SortOrder DESC"
};
var catalogSearchOptions = new CatalogSearchOptions {RecordsToRetrieve = int.MaxValue};
var catalogEntryResponseGroup = new CatalogEntryResponseGroup(CatalogEntryResponseGroup.ResponseGroup.CatalogEntryFull);
var entries = CatalogContext.Current.FindItems(catalogSearchParameters, catalogSearchOptions, catalogEntryResponseGroup);

Roland Egedi Jul 26, 2013 03:18 PM

Only a note, hopefully it can help someone in the future (wish I had this tip an hour or two ago :) ):
Make sure the CurrentThread uses the UICulture you intend it for use... E.g. we have 'sv-SE' on a Commerce site, but CurrentUICulture was 'sv'... This caused some Attributes to have an empty value, and some worked as they should.

Eg: Thread.CurrentThread.CurrentUICulture = new CultureInfo(SiteContext.Current.LanguageName);

Sandeep Sahoo Nov 8, 2013 12:19 PM

Hi
This info is very helpful. I can see that you have provide the code for deserializing the data.
serialized = (MetaObjectSerialized)formatter.Deserialize
(new MemoryStream((byte[])catRow["SerializedData"]));

I am importing the catalogue using the code above. The default code suggest that we reset the serialized data using newEntryRow2.SetSerializedDataNull(); as otherwise the fields updated wont be shown on the site.

However I cannot seem to find the code to serialize the data again.

Could you please provide some help on this.

Kind Regards
Sandeep

Jonathan Roberts Feb 14, 2014 10:52 AM

Using CatalogEntryDto how do you set the Products DisplayName?