Custom content provider using FIND - really slow.

Vote:
 

Hello, I've built a custom content provider that fetches data from Find. It's reaaally slow, in edit mode it can take up to 3 minutes to fetch all data(all data = 400 objects...).

This does not seem right, 400 objects(and from FIND!) should load really fast.

Anyhow, my first thought was that I should split the objects in different subfolders so episerver does not load all data right away, but when I try to do that I get the following error message: "Deadlock risk detected. Trying to read a page from the same thread that is already trying to read the same page."

I will post my all methods in my Content provider below, would really appreciate if someone could look at it and maybe give me some pointers... :)

Questions:

  1. Why is it so slow?
  2. Why do I get a deadlock warning when executing GetFolderReference from 
public static ContentFolder GetEntryPoint(string name)
{
	var contentRepository = ServiceLocator.Current.GetInstance < IContentRepository > ();
	var folder = contentRepository.GetBySegment(ContentReference.RootPage, name, LanguageSelector.AutoDetect()) as ContentFolder;
	if (folder == null)
	{
		folder = contentRepository.GetDefault < ContentFolder > (ContentReference.RootPage);
		folder.Name = name;
		contentRepository.Save(folder, SaveAction.Publish, AccessLevel.NoAccess);
	}
	return folder;
}

protected override IContent LoadContent(ContentReference contentLink, ILanguageSelector languageSelector)
{
	var mappedItem = this.identityMappingService.Get(contentLink);
	if (mappedItem != null) {
		var id = mappedItem.ExternalIdentifier.ToString().Split('/').Last();
		var release = this.client.Get < ReleaseModel > (id);
		return ConvertToRelease(release);
	}
	return null;
}

protected override IList < GetChildrenReferenceResult > LoadChildrenReferencesAndTypes(ContentReference contentLink, string languageID, out bool languageSpecific)
{
	languageSpecific = false;
	var releases = GetReleasesFromFind();

	var result = releases.Select(p = >new GetChildrenReferenceResult
	{
		ContentLink = this.identityMappingService.Get(MappedIdentity.ConstructExternalIdentifier(ProviderKey, p.Id.ToString()), true).ContentLink,
		ModelType = typeof(Release)
	}).ToList();
	return result;
}

public Release ConvertToRelease(ReleaseModel releaseModel)
{
	var type = this.contentTypeRepository.Load(typeof(Release));
	var release = this.contentFactory.CreateContent(type, new BuildingContext(type)
	{
	    //Parent = GetFolderReference() this causes a deadlock warning.
		Parent = this.contentRepository.Get < IContent > (EntryPoint),
		
	}) as Release;

	release.Status = VersionStatus.Published;
	release.IsPendingPublish = false;
	release.StartPublish = releaseModel.PublishDate;

	var externalId = MappedIdentity.ConstructExternalIdentifier(ProviderKey, releaseModel.Id.ToString());
	var mappedContent = this.identityMappingService.Get(externalId, true);
	release.ContentLink = mappedContent.ContentLink;
	release.ContentGuid = mappedContent.ContentGuid;

	release.Title = releaseModel.Title;
	release.Name = releaseModel.Title;
	release.Body = releaseModel.Body;
	release.Complete = releaseModel.Complete;
	release.LogoUrl = releaseModel.LogoUrl;

	release.MakeReadOnly();
	return release;
}

// TODO change to query
public List < ReleaseModel > GetReleasesFromFind()
{
	var query = this.client.Search < ReleaseModel > ().Take(1000);
	var batch = query.GetResult();
	var releases = new List < ReleaseModel > ();
	releases.AddRange(batch);
	var totalNumberOfPages = batch.TotalMatching;

	var nextBatchFrom = 1000;
	while (nextBatchFrom < totalNumberOfPages) {
		batch = query.Skip(nextBatchFrom).GetResult();
		releases.AddRange(batch);
		nextBatchFrom += 1000;
	}

	return releases;
}

private IContent GetFolderReference(ReleaseModel releaseModel)
{
	var year = releaseModel.PublishDate.Year.ToString();
	var month = releaseModel.PublishDate.ToString("MMM");

	// check if year folder exists
	var yearFolder = this.contentRepository.GetBySegment(EntryPoint, year, LanguageSelector.AutoDetect()) as ContentFolder;
	var yearFolderReference = yearFolder ? .ContentLink;
	if (yearFolder == null) {
		yearFolderReference = CreateFolder(year, EntryPoint);
	}

	var monthFolder = this.contentRepository.GetBySegment(yearFolderReference, month, LanguageSelector.AutoDetect()) as ContentFolder;
	if (monthFolder != null) {
		return monthFolder;
	}

	var monthFolderReference = CreateFolder(month, yearFolderReference);
	return this.contentRepository.Get < ContentFolder > (monthFolderReference);
}

private ContentReference CreateFolder(string folderName, ContentReference parentReference)
{
	var contentFile = this.contentRepository.GetDefault < ContentFolder > (parentReference);
	contentFile.Name = folderName;
	return this.contentRepository.Save(contentFile, SaveAction.Publish, AccessLevel.NoAccess);
}

[Pasting files is not allowed]

#183658
Oct 19, 2017 9:38
Vote:
 

Try to run Fiddler (https://www.telerik.com/fiddler) to see what it is happening in the communication from the site to Episerver Find.

#183659
Oct 19, 2017 10:49
Vote:
 

The find query is really fast. The problem seems to be the calls to identityMappingService, also, I noticed that LoadChildrenReferencesAndTypes gets called for each item. I've added a return in that method now, if it's a child to the folder I just return a empty list. Not ideal, but it made it a lot faster.

#183660
Oct 19, 2017 10:51
Vote:
 

Ok. Maybe a lot of talking to the DDS is happening in that Service that makes it slow (I do not remember where it keeps the mappings but I guess in the DDS and that is not directly famous to be fast).

#183661
Oct 19, 2017 10:55
Vote:
 

Hey Josef, did you find any solution to this issue? Im getting the same dead lock message. Not using Find but Elastic, so pretty similar setup. 

#188605
Feb 27, 2018 17:16
Vote:
 

No, I ended up skipping the custom content provider altogether, had a lot of other problems as well, sorry..

#188644
Feb 28, 2018 12:47
Vote:
 

Ok, I got it working now. Problem was that I messed up the content folder structure I created in LoadChildrenReferencesAndTypes.

#188698
Mar 01, 2018 11:37
* You are NOT allowed to include any hyperlinks in the post because your account hasn't associated to your company. User profile should be updated.