Making EPiServer and Search Engines Work Together

With the dawn of search appliance everybody wants to rack up an external box in their server hall, leaving the gritty details of indexing and searching to companies who specialize in doing it best.

We often forget how picky these magic boxes are. They can not think for themselves and thrive on being provided exact directions on how to correctly index your information. That is where we more often than not fail, many times without even knowing we failed.

The Problem

When deleting or moving pages in EPiServer, there is no information stored about that specific page’s history. You either get 200 OK or 404 Not Found, taking no concern whether that page has actually been moved or even deleted. This confuses search engines, clutters your index and breaks links from external sources when restructuring your page tree.

By attaching to some of the events exposed by EPiServer’s DataFactory I will try and fix this problem with a custom module, so let’s implement IHttpModule and get running.

Wiring Up

First, let us take a look on initializing the module.

   1: public void Init(HttpApplication context)

   2: {

   3:     SetupEvents();

   4:  

   5:     context.EndRequest += new EventHandler(context_EndRequest);

   6: }

In SetupEvents() we need to listen to the relevant events.

   1: private Type _urlMapperType = typeof(SimpleSqlUrlMapper);

   2: private static readonly object _lockObject = new object();

   3: private static bool _initialized = false;

   4:  

   5: private IUrlMapper Mapper

   6: {

   7:     get

   8:     {

   9:         return Activator.CreateInstance(_urlMapperType) as IUrlMapper;

  10:     }

  11: }

  12:  

  13: private void SetupEvents()

  14: {

  15:     if (!_initialized)

  16:     {

  17:         try

  18:         {

  19:             if (Monitor.TryEnter(_lockObject))

  20:             {

  21:                 this.Mapper.GenerateSchema();

  22:  

  23:                 EPiServer.DataFactory.Instance.DeletedPage += 

  24:                     new EPiServer.PageEventHandler(Instance_DeletedPage);

  25:                 

  26:                 EPiServer.DataFactory.Instance.MovedPage += 

  27:                     new EPiServer.PageEventHandler(Instance_MovedPage);

  28:                 

  29:                 EPiServer.DataFactory.Instance.PublishedPage += 

  30:                     new EPiServer.PageEventHandler(Instance_PublishedPage);

  31:  

  32:                 _initialized = true;

  33:             }

  34:         }

  35:         finally

  36:         {

  37:             Monitor.Exit(_lockObject);

  38:         }

  39:     }

  40: }

Locking here is very important, since we do not want a possible race condition, resulting in hooking up the events multiple times. I generate a schema for the SimpleSqlUrlMapper. Then, we let a static boolean tell us whether the module has been initialized or not.

IUrlMapper is a simple interface with just a couple of methods. I was thinking about splitting responsibility for some things into another interface, but in the end I wanted something that’s just simple to set up.

   1: public interface IUrlMapper

   2: {

   3:     void GenerateSchema();

   4:     void Add301MovedPermanentlyStatus(string oldUrl, string newUrl);

   5:     void Add410GoneStatus(string url);

   6:     void RemoveMappings(string url);

   7:     bool Exists(string url);

   8:     string GetRedirectUrl(string oldUrl);

   9:     HttpStatusCode? GetStatusCode(string url);

  10: }

Now. let us see what is hidden in context_EndRequest()

   1: private void context_EndRequest(object sender, EventArgs e)

   2: {

   3:     HttpApplication app = sender as HttpApplication;

   4:     if (app != null && app.Context.Response.StatusCode == (int)HttpStatusCode.NotFound)

   5:     {

   6:         HttpStatusCode? statusCode = 

   7:             this.Mapper.GetStatusCode(app.Context.Request.Url.AbsolutePath);

   8:  

   9:         if (statusCode != null)

  10:         {

  11:             app.Context.Server.ClearError();

  12:             app.Context.Response.Clear();

  13:             app.Context.Response.StatusCode = (int)statusCode.Value;

  14:  

  15:             if (statusCode == HttpStatusCode.MovedPermanently)

  16:             {

  17:                 app.Context.Response.AddHeader("Location", 

  18:                     this.Mapper.GetRedirectUrl(app.Context.Request.Url.AbsolutePath));

  19:             }

  20:         }

  21:     }

  22: }

We need to attach to EndRequest, since the Error event will not work under IIS7 in Integrated Mode. The code is pretty self explanatory. If we get a 404, check if we have mapped this URL and respond with the correct status code.

I will not go into the implementation details of the UrlMapper, because that is not really relevant to the problem. Instead, let’s see what happens with this module set up!

Shuffling Pages Around

Let’s create a new page and fiddle around to see what happens!

Now, let’s move it to “News”.

And we end up with this in our table.

That is all fine and dandy. Now let’s see if our magic http module works by browsing to http://acme/en/test-page/ !

Hooray! We got a 301 and the browser redirected us. Let us move the page around a little more to see what happens.

This should make both /en/test-page/ and /en/News/test-page/ tell us that this page now resides under /en/Events/test-page/.

…but let’s say you decided you wanted it under the start page after all.

That would make us have to remove the old mapping aside from making the other locations point to the new one.

Now, let’s say you’re sick of this page and its infidelity with the page structure and you decide to remove it.

That should teach it. Let’s see what’s in the table now.

Let’s open up Firefox and open one one of the URLs.

Mission accomplished! But, what happens if we create another test-page under the root now?

The mapping should be gone. Let’s look.

It is.

Let’s see if we can change the URL segment of the page too.

Should be updated in the table now.

Disclaimer

This is the result of a late night hack, thus it is probably riddled with bugs. However, it is a good start towards building EPiServer sites with better support for search engines. There are a few known bugs or things I just neglected for now due to sleep deprivation, such as:

* Weird behaviour when moving pages out of waste basket
* No support for mapping descendants of moved pages yet

Go ahead and try it out, and please give me a shout if you make any improvements or just want to toss ideas at me.

Install the module by putting it in your bin folder and add this to your modules section in web.config:
<add name="StatusCodeModule" type="Nansen.StatusCodes.Modules.StatusCodeModule, Nansen.StatusCodes" />