Loading...
Area: Episerver Profile Store
Applies to versions: 1.3.0 and higher

Exporting profiles based on query

Recommended reading 

Closed preview. This information is not publicly available, access is restricted. By using this feature you are committing to doing it under consultancy with your Episerver representative. You will be held liable if using the feature without sign-off.

This topic describes how to send a request to export profiles based on a filter, check the status of the export request, and get the URLs of the exported blob files. 

In this topic

Steps for exporting profiles

  1. ExportByQuery command follows Microsoft REST API Guidelines for supporting $filter and $select parameters.  The optional $select parameter contains a comma separated list of Profile fields to include in the export. If the field is not specified, all Profile fields are included.
  2. Send an export Profiles request using the {scope}/ExportByQuery?$filter endpoint. The RequestId is returned and used in next step.
  3. Send an export request with RequestId using ExportByQueryResult/{requestId} endpoint. Status is returned.
    • If Status is InProgress, ResultUrl returns an endpoint to continue checking the export process. Currently, it returns just the same endpoint with the requestId above.
    • When the export process is done, the response returned indicates (Status as Finished). The response also returns the ResultUrl, which is the URL of an index file (an index of all results) and the Headers collection, which contains the authentication headers required to make a successful request to the index file. When a successful request is made to the index file, it contains a list of the URLs of the exported BLOBs and their respective authentication headers. 
    • You can download each BLOB file using its URL accompanied by its corresponding authentication headers.

Exporting-profiles.PNG

Response data structure

  • Query. The OData filter based on one or more Profile fields to export profiles. This is the original filter specified in export profiles request. 
  • Select. The OData select option contains a comma separated list of Profile fields used to export profiles. This is the original Select parameter specified in the export profiles request. 
  • RequestId. Returned from ExportByQuery Profiles API.

    Note: A requestId expires after one day.

  • Status. The export request status:
    • NotFoundResult. The requested export task was not found. For example, an incorrect RequestId was specified in the ExportByQueryResult API. 
    • SourceNotFound. The scope for which a user requested to export profiles does not exist or was already deleted.
    • InProgress. The export profiles request is still processing.
    • Error. The task for exporting profiles encountered an error.
    • Expired. A request was made to retrieve task status for a request that exceeds the request's time limit, which is 1 day.
    • Finished. The export profiles task completed successfully.
  • ResultUrl.
    • If Status is InProgress, ResultUrl returns an endpoint to continue checking the export process. Currently, it just returns the same endpoint with the requestId above.
    • If the export process has Finished, this field contains the URL of an index file if any profiles matching the filter were found. The index file contains information about all exported BLOBs and their authentication headers.
    • Otherwise, this field does not appear.
  • RequestedTime. The time when the export request was received by the Profiles API.
  • Headers. The authentication headers for the index file. This field appears after the export process completes, and one or more matching profiles are found.
    • It appears only when the Status is Finished.
    • It contains 3 headers: x-ms-date, x-ms-version and Authorization; you need all 3 headers to authenticate with the ResultUrl and download the index file.

Send an export profiles request with only a filter

Endpoint

GET /api/v1.0/Profiles/{scope}/ExportByQuery?$filter={filter}&$select={fields}

Note: Based on a number in the ProfileAPI configuration, only the specified number of export requests are executed at the same time; others wait for ongoing export requests to finish. This may affect a request's execution time.

Example 1: The URL https://yourserver-profilesapi.azurewebsites.net/api/v1.0/Profiles/9e48c767-70fe-4ca3-8ec7-a115559b85df/ExportByQuery?$filter=Visits%20gt%200 represents a request to export profiles in the scope 9e48c767-70fe-4ca3-8ec7-a115559b85df, which has the Visits field set to a value greater than 0. In this example, there are no $select parameters specified so all fields for the profiles will be included in the response file.

Example 2: The URL https://yourserver-profilesapi.azurewebsites.net/api/v1.0/Profiles/9e48c767-70fe-4ca3-8ec7-a115559b85df/ExportByQuery?$filter=Visits%20gt%200&%24select=ProfileId,Name,Visits represents a request to export profiles in the scope 9e48c767-70fe-4ca3-8ec7-a115559b85df, which has the Visits field set to a value greater than 0. Additionally the $select parameter specifies to included the fields ProfileId, Name, and Visits in the resulting BLOB files.

Response

{
    "Query"          : "Visits gt 0",
    "Select"         : "ProfileId,Name,Visits",
    "RequestId"      : "cXxAW0JcQ0MGQHNzQl1EOkIVARZ3fEkNQwYRQgdHJn0SXxIAFBABR3h0SVgSVEASbCIyKhYAHwABM0sCLzcENkoARk5QRXZyXV5DAxdbBxEhdl1RFgZFW1JDcXBFXEoHSkNXFDwTGRoaEQFWVBdgdX3liIA",
    "ResultUrl"      : "https://yourserver-profilesapi.azurewebsites.net/api/v1.0/Profiles/ExportByQueryResult/cXxAW0JcQ0MGQHNzQl1EOkIVARZ3fEkNQwYRQgdHJn0SXxIAFBABR3h0SVgSVEASbCIyKhYAHwABM0sCLzcENkoARk5QRXZyXV5DAxdbBxEhdl1RFgZFW1JDcXBFXEoHSkNXFDwTGRoaEQFWVBdgdX3liIA",
    "Scope"          : "9e48c767-70fe-4ca3-8ec7-a115559b85df",
    "RequestedTime"  : "2019-02-19T15:52:36.247173Z",
    "Status"         : "Accepted"
}

The Status should be Accepted, which means the export request was accepted and will process soon.

Send an export result request with RequestId

Endpoint

GET /api/v1.0/Profiles/ExportByQueryResult/{requestId}

Usage

Response

{
  "Query"         : "Visits gt 0",
  "Select"        : "ProfileId,Name,Visits",
  "RequestId"     : "cXxAW0JcQ0MGQHNzQl1EOkIVARZ3fEkNQwYRQgdHJn0SXxIAFBABR3h0SVgSVEASbCIyKhYAHwABM0sCLzcENkoARk5QRXZyXV5DAxdbBxEhdl1RFgZFW1JDcXBFXEoHSkNXFDwTGRoaEQFWVBdgdX3liIA",
  "ResultUrl"     : "https://yourstorageserver.blob.core.windows.net/190219/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190219155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_index.json",
  "Headers"       : {
                      "x-ms-date"     : "Wed, 20 Feb 2019 15:58:54 GMT",
                      "x-ms-version"  : "2018-03-28",
                      "Authorization" : "SharedKey yourstorageserver:3tVZZFK+KoBaLUltOs3zthAPRhVbhv0j0ATdfm7HW3Y="
                    },
  "Scope"         : "9e48c767-70fe-4ca3-8ec7-a115559b85df",
  "RequestedTime" : "2019-02-19T15:52:36.247173Z",
  "Status"        : "Finished"
}

If the Status is InProgress, allow more time for processing then send the request again until Status is Finished.

Accessing results using ResultUrl

After the export is done (Status is set to Finished), make a request to the ResultUrl using the  Headers in the response to download the index file. This file contains a list of URLs of the exported BLOB files and their authentication headers. It also contains a count of the total number of BLOBs generated.

An example JSON from an index file is shown below.

{
  "NumberOfBlobs"        : 3,
  "ResultUrlsAndHeaders" : {
"https://yourstorageserver.blob.core.windows.net/190219/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190219155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_1a1378cf-e40d-41e7-ac62-ac9d9ce34a73.json":
                             {
                               "x-ms-date"     : "Wed, 20 Feb 2019 15:58:54 GMT",
                               "x-ms-version"  : "2018-03-28",
                               "Authorization" : "SharedKey yourstorageserver:qYCu1ydyu3iy4hRwttx8roFa/yTmKlDZ091lpUoX/U8="
                             },
"https://yourstorageserver.blob.core.windows.net/190219/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190219155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_73a91e39-b9a0-4298-bc04-665fba64022d.json": 
                             {
                               "x-ms-date"     : "Wed, 20 Feb 2019 15:58:54 GMT",
                               "x-ms-version"  : "2018-03-28",
                               "Authorization" : "SharedKey yourstorageserver:qSHtrbjdcnrUhLoM0aMy2JDRaBAJt8kDTtZlrl57Y28="
                             },
"https://yourstorageserver.blob.core.windows.net/190219/LSQCHRoLXwVSBiggAkQDCh4XQRszaBMGHp13dOI/9e48c767-70fe-4ca3-8ec7-a115559b85df/190219155236247_9e48c767-70fe-4ca3-8ec7-a115559b85df_f7b8d95b-457c-40bd-9e9e-c9c30a68e487.json": 
                             {
                               "x-ms-date"     : "Wed, 20 Feb 2019 15:58:54 GMT",
                               "x-ms-version"  : "2018-03-28",
                               "Authorization" : "SharedKey yourstorageserver:Ym+tmZFs0PzT1lRL+Xt4VxEZi/KJNwmfuOPr8yPj3eU="
                             },
                           }
}
An example JSON from an exported BLOB file is shown below. You can download a BLOB file using its URL, accompanied with its authentication headers from the index file.
{
  "NumberOfProfiles" : 10000,
  "Profiles"         : [
                         {
                           "ProfileId"          : "6243d1c0-70dd-453e-9cc4-63c577cb25ad",
                           "Name"               : "John Smith",
                           "ProfileManager"     : June Sigman, 
                           "FirstSeen"          : "2019-03-04T23:28:35.4303651Z",
                           "LastSeen"           : "2019-03-04T23:28:35.4303651Z",
                           "Visits"             : 1150,
                           "UpdateTime"         : "2019-03-06T21:29:38.4088275Z",
                           "Info"               : {
                                                    "Picture"         : "http://lorempixel.com/640/480/people",
                                                    "Website"         : "rolando.biz",
                                                    "StreetAddress"   : "18695 Barton Vista, East Bernie, Monaco",
                                                    "Phone"           : "741-236-6647 x42285",
                                                    "Mobile"          : "1-830-513-3557 x3926",
                                                    "City"            : "South Damionton",
                                                    "State"           : "Alabama",
                                                    "ZipCode"         : "80014-2513",
                                                    "JobTitle"        : "District Accounts Specialist",
                                                    "Company"         : "ABC, Inc.",
                                                    "Country"         : "ZM",
                                                    "InferredCountry" : "MG",
                                                    "Email"           : "John@yahoo.com"
                                                  },
                           "ContactInformation" : [
                                                    "Callable",
                                                    "Mailable",
                                                    "Known"
                                                  ],
                           "DeviceIds"          : [
                                                    "9f4447e4-9f00-4fa8-ad30-a1a971d4fbf6"
                                                  ],
                           "Scope"              : "3284-432-42-2321",
                           "Payload"            : {
                                                    "customField"      : 14,
                                                    "otherCustomField" : "apple"
                                                  }
                         },
                         ....
                       ]
}

Each exported BLOB file contains a subset of the matching profiles in the form of a list of serialized Profile objects. It also contains a count of the total number of profiles saved in this BLOB. You can find more information about the structure of a Profile object here.

Additionally, when using the $select argument you can limit the number of fields returned in the exported BLOB file resulting in smaller files that contain only the information you need. Example: using a $select field where the columns are specified such as ProfileId, Name, and Visits the resulting BLOB file would contain items such as:

{
  "NumberOfProfiles" : 10000,
  "Profiles"         : [
                         {
                           "ProfileId"  : "6243d1c0-70dd-453e-9cc4-63c577cb25ad",
                           "Name"       : "John Smith",
                           "Visits"     : 1150,
                         },
                       ....
                       ]
}

Maximum number of profiles in a blob

A single BLOB file can contain up to a preconfigured maximum number of profiles (defaults to 10000). This configuration value indicates the maximum number of profiles that can exist in a single BLOB, the actual number of items in a BLOB file may be less than this value depending on the data returned by elastic search while iterating through a large matching data set.

Note: TheExportByQuery API is intended for exporting large datasets of profiles and not for filtering profiles based on real time user requests. 

The following example shows a dataset containing around 1 million profiles.

  • Total number of profiles matching your query in the index= 1092517.
  • Configuration setting for maximum number of profiles per BLOB = 10000.
  • Results: Total 111 BLOB files, with 108 files containing 10000 profiles each. And the remaining 12517 profiles distributed into the remaining three BLOB files (which contain less than 10000 items per file).
Do you find this information helpful? Please log in to provide feedback.

Last updated: Apr 12, 2019

Recommended reading