Hi Michael,

I'm not sure I fully understand your question. Are you thinking about using Find as the only storage for some objects (comments?) or are you contemplating different ways to build the forum using storage either as pages or as custom objects stored in a database or the DDS and want ideas about how to index these objects?

Either way, should users be able to search the forum for threads or individual comments/posts?

Because the majority of the content of the site would be user generated, I'm looking for the best data store for all user generated content. There will also be some content that will be generated by admin (CMS), for example a set list of "category" forum pages. I'm looking at all options for storage, and trying to determine the best method as either pages or as custom objects.

Obviously with Find, pages are automatically indexed, so to be able to have that functionality out-of-the-box is quite advantageous. With the new EPiServer 7 data architecture, I'm assuming that it would not even be a question, as this would be an excellent example of new data store.

All I want to use Find for is the search of forum pages, user profiles, and potentially going into comments as well.

Alright, so somewhat simplified you have these options:

1. Store user generated contet as pages/content in the CMS.
2. Use EPiServer Community/Relate or some third party community platform.
3. Build custom functionality using a custom data access layer, the DDS, some O/R-mapper or possibly Find.
4. Some sort of hybrid between the above, most likely storing some things, such as forum threads, as CMS content but other things, such as replies/comments on threads in the DDS or some other data store.

In order to build search/querying functionality with Find objects needs to be indexed. This involves:

• Ensuring that objects can be indexed (serialized) - This typically involves excluding properties that can cause circular references, handling types that have custom serialization logic (annotated with TypeConverterAttribute etc) and ensuring that objects created by some dynamic proxy library is indexed as their non-proxy types.
• Ensuring that the correct values are indexed - This usually involves creating extension methods for indexed types that return data that isn't exposed by their properties and ensuring that the return value of those methods are indexed. One example of this would be to create an extension method for forum threads that return all replies so that it's possible to search for threads rather than individual replies. This may also involve excluding properties that shouldn't be indexed, such as user passwords.
• Event driven indexing - Somehow hooking into when the objects are saved, updated and deleted to index them. This can either be done by listening for events, extending the functionality that handles these CRUD operations (such as modifying a potential ForumRepository's Save method), or simply modifying the functionality close to the UI that triggers such operations (such as modifying the event handler for the button the user presses when creating a new thread).
• Reindexing functionality - While event driven functionality can handle indexing on a real-time basis having functionality to do complete reindexing is almost a must as it may otherwise be tricky to handle changes to the indexed types. It's also needed when data that should be indexed exists prior to building the event driven indexing functionality.

Of course given that reindexing functionality has been built it's possible to skip having event driven indexing but then the search functionality won't be updated in real-time.

Given this, what does this mean for each of the data store alternatives?

If user generated content is stored as CMS content the first and last of the steps needed for indexing is taken care of. Some work may be needed to ensure that for instance comments/replies are indexed along with their threads meaning that some extension method may be needed for the thread page/content type and we'd have to ensure that threads are reindexed when a reply is added to them. The latter could be done using the "related pages" functionality in Find.

If user generated content is stored using EPiServer Relate/Community we'd have to exclude some properties that cause circular references from forum threads and replies. We'd also have to implement all of the other bullet points for indexing. However, doing so would probably be fairly easy as we wouldn't have to deal with all Community entity types but only a few specific ones and (as I recall) Community exposes the relevant events. Re-indexing functionality could be built using a fairly simple scheduled job.

If user generated content is stored in some sort of custom data layer we'd most likely wouldn't have any problems ensuring that the objects could be indexed as the types are likely to be fairly simple. If we were using some O/R-mapper like Entity Framework that uses dynamic proxy objects we'd have to instruct Find to index the objects as their non-proxy types however. We'd of course have to build the indexing functionality ourselves but as we're in control of the CRUD functionality this should be fairly easy. Reindexing functionality could, again, be built using a fairly simple scheduled job.

In a schenario where we're storing some objects as CMS content and some using a custom data layer we'd probably be able to utilize the existing reindexing functionality. We'd have to ensure that replies/comments are indexed with their threads.

So, storing user generated content as CMS content gives us a lot of indexing functionality for free but we'd probably have to do some work to ensure that replies/comments are indexed with their threads. The other alternatives involves more work as we'd have to build the indexing functionality ourselves. However, building this functionality doesn't have to take very long. For instance one project that I worked with stored comments in the DDS and we created both event driven indexing and re-indexing functionality in about an hour or two.

With that in mind my recommendation would be to choose the best solution for storing user generated content without regarding indexing as, no matter what alternative you choose building indexing functionality is likely to be a very small percentage of the work. You may want to consider if Find can help you solve some problems that you'd otherwise face using some of the storage alternatives. For instance, if you'd store the content as CMS pages/content you may have trouble building non-hierarchial listing and Find could certainly help you there.

I'm not sure wether my reply helps you. Please let me know if it doesn't :)

Thanks Joel, that is some awesome information, and gives me plenty to think about.

In terms of storage and mapping comments to threads, my idea would be that if I'm storing all content as pages (to keep the idea of easy to index going), I would have a field on a comment item that would be the reference map back to the original thread. That way I could store the comments in its own section of the tree, i.e. seperate from the threads. (Not sure if there's any benefit at this stage with that approach). 

Then when performing a search on returning all comments for a thread, I can just filter out the comments based on the "reference" field. The idea behind this is that it will not only be threads that would have comments. Consider that a user profile could have comments etc. 

I think you are right though in that putting indexing aside, I should use the best data store for the application, not for the index, as either way, there is not alot of work involved to do this.

Mentioning EPiServer 7 again, this would not pose as significant of a query/answer correct?

I'm afraid I don't follow how you envision searching for threads and comments completely, but keep in mind that what you search for with Find is documents. Out-of-the-box one object == one document. Meaning that if each comment is indexed on its own you can easily search for individual comments. You can then of course link from a found comment to its thread, but you won't be able to search for threads based on all of their comments. To handle that you can include the comments in each threads document using for instance an extension method that returns all comments. In other words, you can't do joins but you can denormalize your data.

I don't think EPiServer 7 will have any significant impact on this with the exception that you could store user generated data as a custom content type that doesn't show in the page tree.