In tblBigTable, I have a huge amount of items marked with StoreName = 'IndexRequestQueueDataStore' (currently about 845 000 items, and it grows by roughly a hundred new items every hour depending on user activity in the communities.
What is responsible for handling these items, and what might cause it to stop processing the queue?
Did you check these settings for your app?
There are currently four load-balanced web servers (EPiServer CMS 6 R2 Enterprise) in the environment. One of the servers is configured to be the search server, another is a search client, and the other two aren't configured to use FTS yet.
This is the section from web.config for the server acting as an FTS search server:
<add name="clientIndexingService" baseUri="http://hostname-of-FTS-server/IndexingService/IndexingService.svc" accessKey="secret-key" />
<add name="secret-key" description="Search clients" ipAddress="ip-adresses-of-servers,127.0.0.1/8" allowLocal="true" readonly="false" />
<add name="default" directoryPath="\\fileserver\share\Indexes" readonly="false" />
The server configured as a search client has the same episerver.search section, but nothing configured in the episerver.search.indexingservice section (except the defaultIndex attribute).
I can see that something changed in the queue after configuring FTS, since new items are added with the new service name in the NamedIndexingService column, but new items never seems to be removed from the queue.
I am about to configure the other two servers so they also use these (FTS client) settings.
Using dotPeek I've been able to find out a little about what is responsible for handling the indexing request queue.
The indexing service itself (the server side) is responsible for processing the queue. Which is pretty obvious. :)
However, I have no idea if it actually does what it is supposed to be doing... If I open Windows Explorer and look in the indexing directory, I lots and lots of files being created and then deleted again. They are named something similar to _3h5c8.del, and have different file extensions, such as .cfs, .frq, och some other extensions.
However, is the index for Lucene stored on disk? I.e. is a file supposed to increase in size? The only files whose name doesn't change is segments.gen and write.lock, and they are 1 KB and 0 KB each.
EPiServer FTS is causing more problems on the server configured as an indexing server... For some reason it doesn't complete the requests, but leaves them "hanging". It doesn't take long before the maximum concurrent request limit is reached (5000 concurrent requests) and the site returns an error message instead. I think it's time to remove FTS from all the servers. :)