Querying an external index

 

Hi,

I have two separate, but related, sites. One of the sites I would like to promote on the other. One of the ways I'm going to promote it is by showing search results from the smaller site on the primary site.

I have created a separete IClient instance with:

IClient mySecondClient = new Client(serviceUrl, indexName);

When trying to access the results with GetResult I get a 401 Unauthorized error.
What configuration options are needed in order to query an external index? I can't find anything in the documentation about this.

Edit: Quite embarrasing but the 401 error was because of an incorrect serviceUrl and indexName. I get no errors now, but no results either. I try to get the results with filterForPublicSearch set to false.

Thanks!

/Niklas

#190568 Edited, Apr 12, 2018 10:15
  • Member since: 2013
     

    Niklas, have you set up a demo Find index? -- https://find.episerver.com

    Are you attemping to search content outside of Episerver? If so you'll need to index it first: https://world.episerver.com/documentation/developer-guides/find/NET-Client-API/searching/connectors/

    Would you post the rest of your search code (all the way down to the GetResults or GetPageResults call)?

    #190594 Edited, Apr 12, 2018 14:52
  •  

    Thanks for your reply.

    I have a demo-index for my local devsite and a proper index for the other site.

    I am not trying to search for content outside of Episerver, I'm trying to search for content in an index from another Episerver site.

    I noticed now that I will get results if I put the serviceUrl and defaultIndex in my web.config but not if I have it in my SearchService.
    I do however need to use two differenent serviceUrls for the two different indexes.

    Here is my SearchService.

    public class SearchPageService : ISearchPageService
        {
            private readonly IClient _searchClient;
            private readonly IClient _secondClient;
            private readonly IContentLoader _contentLoader;
    
            public SearchPageService(IClient searchClient, IContentLoader contentLoader)
            {
                _searchClient = searchClient;
                _contentLoader = contentLoader;
                _secondClient = new Client("https://es-eu-api03.episerver.net/rA1notarealindexHcpkqhOU7T", "index-name");
            }
    
            public SearchResult ExecuteSearch(SearchParameters searchParameters)
            {          
                var query = ExecuteUnifiedSearch(_searchClient, searchParameters);
                var secondQuery = _secondClient.UnifiedSearchFor(searchParameters.QueryString);
    
    //some filtering for the local query here
                var filterSites = GetFilterSites();
                query = AddFilterFacetsForSites(query, filterSites);
                
                query = ApplyViewFilter(query, searchParameters.ViewFilter);
    
                var hitSpec = new HitSpecification
                {
                    ExcerptLength = searchParameters.ExcerptLength
                };
    
                var unifiedSearchResults = query.GetResult(hitSpec, false);
                var secondResults = secondQuery.GetResult(hitSpec, false);
    
            // Map results to a SearchResult object
                var result = MapSearchResult(searchParameters, filterSites, unifiedSearchResults, secondResults);
    
                return result;
        }
    }

    When I debug the secondResults object has 0 hits.
    If I instead replace the serviceUrl and indexName from the Client() constructor with the index for the local site in my web.config i get expected results. So something is not working with creating a new client.

    [Pasting files is not allowed]

    #190599 Edited, Apr 12, 2018 15:05
  • Member since: 2013
     

    Interesting. 

    Is that your API key? You might want to hide / obfuscate in case someone gets cute. Even if it's just a demo index. 

    Stupid question -- is "index-name" the actual name of your second index? 

    Taking a step back... Why are you using two Find indexes? 

    #190605 Apr 12, 2018 16:54
  •  

    I have obfuscated it some. The "index-name" is just me changing the proper name to something else.

    I am using two indexes because I want to show results from my secondary site on my primary site. The sites are not part of the same Epi-solution, therefore I can't use the same index (or at least that's what I thought). It just hit me that I maybe could put the same serviceUrl and indexName in the find-configuration in web.config of both sites, would that work?

    #190606 Apr 12, 2018 16:59
  • Member since: 2013
     

    Okay great. I've never encountered a use case for searching two separate Find indexes, but in theory this should be possible. 

    If you're getting zero results back, instead of a server error, then my guess is that one of your filters is to blame. I would try temporarily stripping the search down to the basics. Get rid of your filtering, sorting, faceting, etc. Call your unified search with a word that you know appears in both indexes. If the base query returns results from both indexes, then you can isolate down to something else you're chaining onto the search object. 

    For grins, I tried newing up two separate Find clients with different indexes and both returned results using the following query: 

    var results = client.Search<PageData>().For("test").GetPagesResult(); 

    I realize you're using unified search, but I would give that a try too. 

    #190611 Apr 12, 2018 20:24
  • Member since: 2007
     

    With zero result, it might be so that it search for another namespace and also siteid, look with fiddler how the elastic query is done

    #190615 Apr 13, 2018 7:06
  •  

    First off, thanks alot for taking your time in helping me with this.

    I have done some testing and found the following cases where it works/doesn't work.

    Case 1 - local index in web.config, external index in SearchPageService constructor.

    The index-details for my local dev-site is located in the webconfig like so:

    <episerver.find configSource="EpiserverFind.config" />

    The contents of EpiserverFind.config looks like this:

    <episerver.find
      serviceUrl="https://es-eu-dev-api01.episerver.net/localindexkey/"
      defaultIndex="my-local-index"/>

    The SearchPageService looks like this:

        public class SearchPageService : ISearchPageService
        {
            private readonly IClient _searchClient;
            private readonly IClient _secondClient;
            private readonly IContentLoader _contentLoader;
    
            public SearchPageService(IClient searchClient, IContentLoader contentLoader)
            {
                _searchClient = searchClient;
                _contentLoader = contentLoader;
                _secondClient = new Client("https://es-eu-api03.episerver.net/external-api-key", "external-index");
            }
    
            public SearchResult ExecuteSearch(SearchParameters searchParameters)
            {          
                var query = _searchClient.UnifiedSearchFor(searchParameters.QueryString);
                var secondQuery = _secondClient.UnifiedSearchFor(searchParameters.QueryString);
    
                var hitSpec = new HitSpecification
                {
                    ExcerptLength = searchParameters.ExcerptLength
                };
    
                var localResult = query.GetResult(hitSpec, false);
                var secondResult = secondQuery.GetResult(hitSpec, false);
    
                var result = new SearchResult
                {
                    Hits = new List<SearchResultItem>(),
                    FilterSelectItems = new List<SelectListItem>(),
                    SearchSiteName = string.Empty
                };
    
                return result;
            }

    When I'm checking the results I have just put a breakpoint on the return statement and inspected both the localResult (which uses the injected searchclient) and the secondResult which uses the manually created client.

    In this case the secondResult contains 0 hits. If I in my browser navigate to the site where this index is from and search for the same term I get 4 results.

    The localResult contains 8 hits, same as if I would search directly in the index in the Find configuration interface.

    Case 2 - External index in web.config, local index in SearchPageService constructor

    If I swap the contents of my EpiserverFind.config so that the EpiserverFind.config contains the external index and the new Client in the SearchPageService constructor contains the local index I get 4 hits (as expected) on the results from the external index but 0 results from the client that uses the local index.

    This shows that there is nothing wrong with the index itself or the filtering (since in both cases they use the exact same UnifiedSearchFor with no filtering).

    I cant seem to get a index to work if I don't put it in the web.config.

    The issue seems to lie in how/when the Client is created/instantiated.

    @Drew, where/when do you 'new up' your Clients? I have tried both in the SearchPageService constructor, as shown above, but this seems not to work. The SearchPageService function ExecuteSearch is called from the SearchPage Controllers action when the search-form is submitted. 

    I have also tried to use an interface that contains a IClient property like so:

        public interface ISecondClient
        {
            IClient Client { get; }
        }

    I then have a class that implements this interface like so:

        public class SecondClient: ISecondClient
        {
            public SecondClient()
            {
                Client = new Client("https://es-eu-api03.episerver.net/external-api-key", "external-index");
            }
    
            public IClient Client { get; }
        }

    In my ServiceRegistrationModule I then register the ISecondClient as a singleton to use with the SecondClient like so:

        [InitializableModule]
        public class ServiceRegistrationModule : IConfigurableModule
        {
            public static void ConfigureContainer(ConfigurationExpression container)
            {
                container.For<ContentAreaRenderer>().Use<CustomContentAreaRenderer>();
                container.ForSingletonOf<ISecondClient>().Use<SecondClient>();
                container.Scan(x =>
                {
                    x.AssembliesFromApplicationBaseDirectory(assembly => assembly.FullName.Contains(WebConstants.AssemblyName));
                    x.WithDefaultConventions();
                });
            }
    }

    Then I inject it into my SearchPageService like so:

            public SearchPageService(IClient searchClient, IContentLoader contentLoader, ISecondClient secondClient)
            {
                _searchClient = searchClient;
                _contentLoader = contentLoader;
                _secondClient= secondClient.Client;
            }

    But this gives the same result as Case 1 above, no results from the _secondClient but expected results from the _searchClient.

    I also have a FindInitializer that maybe can interfere in some way? It looks like this.

        [ModuleDependency(typeof(IndexingModule))]
        public class FindInitializer : IInitializableModule
        {
            public void Initialize(InitializationEngine context)
            {
                // Exclude everything. Assets folder, standalone blocks and some other items are by default indexed.
                ContentIndexer.Instance.Conventions.ForInstancesOf<IContent>().ShouldIndex(x => false);
    
                // Exclude deleted content
                ContentIndexer.Instance.Conventions.ForInstancesOf<IContent>().ShouldIndex(x => !x.IsDeleted);
    
                // include files/blobs/blocks
                ContentIndexer.Instance.Conventions.ForInstancesOf<IContentMedia>().ShouldIndex(x => true);
                ContentIndexer.Instance.Conventions.ForInstancesOf<BlockData>().ShouldIndex(x => true);
    
                // exclude some regular pages that we don't want/need indexed
                ContentIndexer.Instance.Conventions.ForInstancesOf<ContainerPageModel>().ShouldIndex(x => false);
                ContentIndexer.Instance.Conventions.ForInstancesOf<GlobalSettingsPageModel>().ShouldIndex(x => false);
                ContentIndexer.Instance.Conventions.ForInstancesOf<SearchPageModel>().ShouldIndex(x => false);
                ContentIndexer.Instance.Conventions.ForInstancesOf<SettingsPageModel>().ShouldIndex(x => false);
                ContentIndexer.Instance.Conventions.ForInstancesOf<TestPageModel>().ShouldIndex(x => false);
    
                // Exclude properties
                SearchClient.Instance.Conventions.ForInstancesOf<SitePageData>()
                    .ExcludeField(x => x.GlobalSettings)
                    .ExcludeField(x => x.Settings);
    
                // Conventions for unified search
                SearchClient.Instance.Conventions.UnifiedSearchRegistry.Add(typeof(PageData));
            }
    
            public void Uninitialize(InitializationEngine context)
            {
            }
        }

    Any ideas as to why it only works when the index-information is put in the web.config?

    #190617 Apr 13, 2018 7:38
  • Member since: 2013
     

    By "new up" I mean instantiate. You can do that in your search service constructor. If I needed to reuse that second client, I would consider putting the service URL and index name in my own custom app settings. And / or use a named DI type registration so you can inject it where ever you want. 

    But first you should just get it to work... 

    Looking at your code, I think you're missing a `UnifiedSearchRegistry.Add()` call for your second client.

    Try this -- put the following code in your search service, change the Find service URLs and index names (of course), set a breakpoint at the first line, and debug. Both results should have items in them. 

    var client1 = new Client("https://es-eu-dev-api01.episerver.net/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "index_1");
    client1.Conventions.UnifiedSearchRegistry.Add(typeof(PageData));
    var results1 = client1.UnifiedSearchFor("*test*").GetResult();
    
    var client2 = new Client("https://es-us-api01.episerver.com/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "index_2");
    client2.Conventions.UnifiedSearchRegistry.Add(typeof(PageData));
    var results2 = client2.UnifiedSearchFor("*test*").GetResult();

    Does that work? 

    #190651 Apr 13, 2018 14:48
  •  

    The `UnifiedSearchRegistry.Add()`call was what was missing! I'm getting results now!

    Thank you so much for your help!

    #190682 Apr 16, 2018 7:46
  • Member since: 2013
     

    Awesome -- of course it had to be something simple! Thanks for the follow up

    #190722 Apr 16, 2018 14:14