Hi Andrew,

We are experiencing the exact same thing.

We also have a load balanced environment, and can reproduce it by setting the "Maximum Worker Processes" to more than 1 (simulate load balanced environment) in our IIS applicataion pool.

(We have modified our local host files, and set up an IIS site with application pool on our local developer machines)

We have not found a solution, but are talking to Episerver to figure this out.

Do you have any steps on how to reproduce this maybe with the Quicksilver project?

If anyone have experinced this it would be cool if you would reply to this so we can figure this out! :)

For anyone who wants to reproduce this issue (I also send this EPiServer and will post an update once I get an answer)

I managed to recreate the problem in Episervers Quicksilver project.

Here are the steps I did to recreate the problem:

1. Download the Quicksilver project https://github.com/episerver/Quicksilver
2. Load up the project in Visual Studio
3. Locate the file connectionString.config in the EPiServer.Reference.Commerce.Site project
4. Outcomment EPiServerAzureBlops and EPiServerAzureEvents lines (line 5 & 6)
5. Build the project (restoring packages requires episerver nuget feed)
6. Run the file SetupDatabases.cmd in the folder Setup.
7. Start the project in debug-mode and do the final migration step (should be automatic) (login is admin/store)

Then we must setup the project in IIS (Internet Information Services):

1. Open your localhost file located in C:\Windows\System32\drivers\etc\hosts
2. Add the line:
   127.0.0.1 dev.quicksilver.com
   at the end of the file.
3. Open IIS and create a new website.
4. Fill out the following:
   Site name: dev.quicksilver.com
   physical path: path to EPiServer.Reference.Commerce-Site folder
   Host name: dev.quicksilver.com
5. Now go to application pools in IIS – Right click the dev.quicksilver.com pool and go to advanced settings.
6. Change the value of the “Maximum Worker Processes” to for example 10 (2 should be fine but the error will happen more frequently if you change it to a high value)

Now we must start multiple instances of the site:

1. Start your browser and go to dev.quicksilver.com and the site will start up (styles might be missing but we don’t care about that)
2. Start an incognito browser and browse to dev.quicksilver.com – If the site takes 5-10 seconds to start it means you successfully started a new instance in the IIS.
3. Close the incognito browser and repeat step 2 until you have seen at least 4-5 long loads (meaning you have around 5 instances of the IIS running)
4. This can also be checked in the task manager – Check the processes “IIS Worker Process” if the use “a lot of memory” it means they are running instances of the site.

Now we need to capture the deadlocks in the SQL Profiler:

1. Start SQL Profiler.
2. Start a new trace using the template TSQL_Locks.
3. Go to “Events selection” and tick off (as in we do not care about these - keep the others "clicked") Lock:Cancel, SP:StmtCompleted, SP:StmtStarting, SQL:StmtCompleted & SQL:StmtStarting.

Now that the profiler is running is time to wait.

For me it took some time (30 minutes until the first deadlock happened)

Disable the scheduler on other servers from web.config, make sure there is only one server running the schedule job.