Loading...
Area: Episerver Profile Store

Exporting track events

Recommended reading 

Closed preview. This information is not publicly available, access is restricted. By using this feature you are committing to doing it under consultancy with your Episerver representative. You will be held liable if using the feature without sign-off.

This topic describes how to send a request to export track events based on a filter, check the status of the export request, and get the URLs of the exported BLOB files. 

In this topic

Steps for exporting track events

  1. ExportByQuery command follows Microsoft REST API Guidelines for supporting $filter and $select parameters. The optional $select parameter contains a comma-separated list of Profile fields to include in the export. If the field is not specified, all TrackEvent fields are included. 
  2. Send an export track events request using the {scope}/ExportByQuery?$filter endpoint. The RequestId is returned and used in next step.
  3. Send an export request with RequestId using ExportByQueryResult/{requestId} endpoint. Status is returned.
    • If Status is InProgress, ResultUrl returns an endpoint to continue checking the export process. Currently, it returns just the same endpoint with the requestId above.
    • When the export process is done, the response returns:
      • Status as Finished
      • ResultUrl, which is the URL of an index file (an index of all results).
      • Headers collection, which contains the authentication headers required to make a successful request to the index file.
    • When a successful request is made to the index file, it contains a list of the URLs of the exported BLOBs and their respective authentication headers. 
    • You can download each BLOB file using its URL accompanied by its corresponding authentication headers.

      Exporting-trackevents.PNG

Response data structure

  • Query. The OData filter is based on one or more TrackEvent fields to export track events. This is the original filter specified in export track events request. 
  • Select. The OData select option contains a comma-separated list of TrackEvent fields used to export track events. This is the original Select parameter specified in the export track events request. 
  • RequestId. Returned from ExportByQuery track events API.

    Note: A requestId expires after one day.

  • Status. The export request status:
    • SourceNotFound. The scope for which a user requested to export track events does not exist or was already deleted.
    • InProgress. The export track events request is still processing.
    • Error. The task for exporting track events encountered an error.
    • Expired. A request was made to retrieve task status for a request that exceeds the request's time limit, which is 1 day.
    • Finished. The export track events task completed successfully.
  • ResultUrl.
    • If Status is InProgress, ResultUrl returns an endpoint to continue checking the export process. Currently, it just returns the same endpoint with the requestId above.
    • If the export process has Finished, this field contains the URL of an index file if any track events matching the filter were found. The index file contains information about all exported BLOBs and their authentication headers.
    • If no matching track events were found, the ResultUrlfield does not appear. 
  • RequestedTime. The time when the export request was received by the TrackEvents API.
  • Headers. The authentication headers for the index file. This field appears after the export process completes, and one or more matching track events are found.
    • It appears only when the Status is Finished.
    • It contains 3 headers: x-ms-date, x-ms-version and Authorization; you need all 3 headers to authenticate with the ResultUrl and download the index file.

Send an export track events request with only a filter

Endpoint

GET /api/v1.0/TrackEvents/{scope}/ExportByQuery?$filter={filter}&select={fields}

Note: Based on a number in the ProfileAPI configuration, only the specified number of export requests are executed at the same time; others wait for ongoing export requests to finish. This may affect a request's execution time.

Example 1: The URL https://yourserver-profilesapi.azurewebsites.net/api/v1.0/TrackEvents/9e48c767-70fe-4ca3-8ec7-a115559b85df/ExportByQuery?$filter=CountryCode%20eq%US represents a request to export track events in the scope 9e48c767-70fe-4ca3-8ec7-a115559b85df, which have the CountryCode field set to equal US In this example, there are no $select parameters specified so all fields for the Profile object are included in the resulting BLOB files.

Example 2: The URL https://yourserver-profilesapi.azurewebsites.net/api/v1.0/TrackEvents/9e48c767-70fe-4ca3-8ec7-a115559b85df/ExportByQuery?$filter=CountryCode%20eq%US&%24select=DeviceId,EventType represents a request to export track eventsin the scope 9e48c767-70fe-4ca3-8ec7-a115559b85df, which has the CountryCode field set to equal US. Additionally the $select parameter specifies to included the fields DeviceId, and EventType in the resulting BLOB files.

Response

{
  "Query"         : "CountryCode eq US",
  "Select"        : "DeviceId,EventType",
  "RequestId"     : "cnVBUF5VQVsBRxR0SVNBXUhHAlxxfUdZQlJKLGxCI3cUXkpcFkZQEXRxRQ9LB0QXVhQmd0VRQlxDFwJAJBokGxIGGTNFFy4xAywLFR0ERy1zd0hdXlFBRB5GcmhCWkFUDjb3vi4",
  "ResultUrl"     : "https://yourserver-profilesapi.azurewebsites.net/api/v1.0/TrackEvents/ExportByQueryResult/cnVBUF5VQVsBRxR0SVNBXUhHAlxxfUdZQlJKLGxCI3cUXkpcFkZQEXRxRQ9LB0QXVhQmd0VRQlxDFwJAJBokGxIGGTNFFy4xAywLFR0ERy1zd0hdXlFBRB5GcmhCWkFUDjb3vi4",
  "Scope"         : "9e48c767-70fe-4ca3-8ec7-a115559b85df",
  "RequestedTime" : "2019-03-25T17:57:15.8689111Z",
  "Status"        : "Accepted"
}

The Status should be Accepted, which means the export request was accepted and will process soon.

Send an export result request with RequestId

Endpoint

GET /api/v1.0/TrackEvents/ExportByQueryResult/{requestId}

Usage

Response

{
  "Query"         : "CountryCode eq US",
  "Select"        : "DeviceId,EventType",
  "RequestId"     : "cnVBUF5VQVsBRxR0SVNBXUhHAlxxfUdZQlJKLGxCI3cUXkpcFkZQEXRxRQ9LB0QXVhQmd0VRQlxDFwJAJBokGxIGGTNFFy4xAywLFR0ERy1zd0hdXlFBRB5GcmhCWkFUDjb3vi4",
  "ResultUrl"     : "https://yourstorageserver.blob.core.windows.net/190325/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190325155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_index.json",
  "Headers"       : {
                      "x-ms-date"     : "Tue, 26 Mar 2019 19:35:44 GMT",
                      "x-ms-version"  : "2018-03-28",
                      "Authorization" : "SharedKey yourstorageserver:bNi08kvmE3p17bZ7WLqHmJwYcUBOwWVVYY0y/vnjDo0="
                    },
  "Scope"         : "9e48c767-70fe-4ca3-8ec7-a115559b85df",
  "RequestedTime" : "2019-03-25T19:28:11.1870178Z",
  "Status"        : "Finished"
}

If the Status is InProgress, allow more time for processing then send the request again until Status is Finished.

Accessing results using ResultUrl

After the export is done (Status is set to Finished), make a request to the ResultUrl using the  Headers in the response to download the index file. This file contains a list of URLs of the exported BLOB files and their authentication headers. It also contains a count of the total number of BLOBs generated.

An example JSON from an index file is shown below.

{
    "NumberOfBlobs"        : 3,
    "ResultUrlsAndHeaders" : {
"https://yourstorageserver.blob.core.windows.net/190325/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190325155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_1a1378cf-e40d-41e7-ac62-ac9d9ce34a73.json": 
                                {
                                 "x-ms-date"     : "Tue, 26 Mar 2019 18:05:17 GMT",
                                   "x-ms-version"  : "2018-03-28",
                                  "Authorization" : "SharedKey yourstorageserver:qYCu1ydyu3iy4hRwttx8roFa/yTmKlDZ091lpUoX/U8="
                                },
"https://yourstorageserver.blob.core.windows.net/190325/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190325155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_73a91e39-b9a0-4298-bc04-665fba64022d.json": 
                                {
                                  "x-ms-date"     : "Tue, 26 Mar 2019 18:05:17 GMT",
                                  "x-ms-version"  : "2018-03-28",
                                  "Authorization" : "SharedKey yourstorageserver:qSHtrbjdcnrUhLoM0aMy2JDRaBAJt8kDTtZlrl57Y28="
                                },
"https://yourstorageserver.blob.core.windows.net/190325/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190325155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_f7b8d95b-457c-40bd-9e9e-c9c30a68e487.json": 
                                {
                                  "x-ms-date"      : "Tue, 26 Mar 2019 18:05:17 GMT",
                                  "x-ms-version"   : "2018-03-28",
                                  "Authorization"  : "SharedKey yourstorageserver:Ym+tmZFs0PzT1lRL+Xt4VxEZi/KJNwmfuOPr8yPj3eU="
                                },
                              }
}
An example JSON from an exported BLOB file is shown below. You can download a BLOB file using its URL, accompanied with its authentication headers from the index file.
{
  "NumberOfItems" : 10000,
  "Items"         : [
                      {
                        "TrackId"       : null,
                        "DeviceId"      : "c10119ad-ae5d-4383-a296-f86b0d7b9b3a",
                        "EventType"     : "product",
                        "EventTime"     : "2018-10-05T09:49:27.894763Z",
                        "Value"         : "Viewed product with code P-2093.",
                        "Scope"         : "9e48c767-70fe-4ca3-8ec7-a115559b85df",
                        "CountryCode"   : "LK",
                        "PageUri"       : "http://lucie.org",
                        "PageTitle"     : null,
                        "RemoteAddress" : "244.105.10.148",
                        "Payload"       : {
                                            "RecContent" : "refCodeOnly",
                                            "Channel"    : "web",
                                            "Product"    : {
                                                             "RefCode": "P-2093"
                                                           },
                                            "Type"       : "product",
                                            "Ip"         : "41.185.112.160",
                                            "Session"    : "55d1588f-499c-4cb5-818b-ee7cd1c4abf2",
                                            "Cuid"       : "0000000007|FzyjXdaBqVnJxyYjChmTXZsW",
                                            "Lang"       : "en",
                                            "currentURI" : "https://localhost:50244/steel/cotton/mindshare",
                                            "UserAgent"  : "Mozilla/5.0 (Windows; U; Windows NT 5.2) AppleWebKit/535.0.0 (KHTML, like Gecko) Chrome/35.0.899.0 Safari/535.0.0"
                                          },
                        "User"          : {
                                            "Name"       : "Sheri Gerlach",
                                            "Email"      : "Sheri.Gerlach@yahoo.com"
                                          }
                      },
                    ....
                    ]
}

Each exported BLOB file contains a subset of the matching track events in the form of a list of serialized TrackEvent objects. It also contains a count of the total number of track events saved in this BLOB. You can find more information about the structure of a TrackEvent object here.

Additionally, when using the $select argument you can limit the number of fields returned in the exported BLOB file resulting in smaller files that contain only the information you need. Example: using a $select field where the columns are specified such as DeviceId, EventType the resulting BLOB file would contain items such as:

{
  "NumberOfItems" : 10000,
  "Items"         : [
                      {
                        "DeviceId"  : "c10119ad-ae5d-4383-a296-f86b0d7b9b3a",
                        "EventType" : "product"
                      },
                    ....
                    ]
}

Maximum number of track events in a blob

A single BLOB file can contain up to a preconfigured maximum number of track events (defaults to 10000). This configuration value indicates the maximum number of track events that can exist in a single BLOB, the actual number of items in a BLOB file may be less than this value depending on the data returned by elastic search while iterating through a large matching data set.

Note: TheExportByQuery API is intended for exporting large datasets of track events and not for filtering track events based on real time user requests. 

The following example shows a dataset containing around 1 million track events.

  • Total number of track events matching your query in the index= 3000110.
  • Configuration setting for maximum number of track events per BLOB = 10000.
  • Results: Total 302 BLOB files, with 299 files containing 10000 track events each, and the remaining three BLOB files containing 989, 902 and 8219 track events each.
Do you find this information helpful? Please log in to provide feedback.

Last updated: Apr 12, 2019

Recommended reading