Blog posts by Brian Weeteling2022-02-18T14:30:00.0000000Z/blogs/brian-weeteling/Optimizely WorldFix your Optimizely Search (Find) indexing exceptions!https://www.brianweet.com/2022/02/18/memory-leaks.html2022-02-18T14:30:00.0000000Z<p>I’ve been looking at memory dumps too much. Sometimes I get a gut feeling that there might be something wrong looking at the memory utilization charts, but tracking down memory issues takes quite some effort. In this blog post I’ll explain why you should take a look at your Find indexing job logs, fix any exceptions there and make sure you adhere to the <a href="https://world.optimizely.com/documentation/developer-guides/search-navigation/NET-Client-API/Indexing/">attachment size limits</a>.</p>
<p>For most of our sites we use Application Insights for application monitoring and performance metrics. I noticed big spikes in memory usage when our CMS and Commerce content is being indexed, which kind of makes sense because the job will go through all of the content and part of that content will remain cached afterwards. On one of our customer projects the memory pressure was a lot higher than expected. I decided to take some memory dumps and start the investigation.</p>
<p class="centered-image">
<img src="/assets/memory-leak-2/1.content-indexer.jpg" alt="Content indexer usage" />
</p>
<p>Here we see that the <code class="language-plaintext highlighter-rouge">ContentIndexer</code> is using quite a significant huge amount of memory. These are the inclusive bytes however, the issue is not the <code class="language-plaintext highlighter-rouge">ContentIndexer</code> object itself but everything it holds a reference to. While I mentioned that indexing will cause content to be cached; that cache should not add up to the memory usage of the indexer itself. For some reason it holds a reference to some objects that add up to 600MB. A little bit further down the list we find the objects that take up so much memory: <code class="language-plaintext highlighter-rouge">ScatherGatherBuffers+MemoryChunk</code>.</p>
<p>What could be the the relation between the <code class="language-plaintext highlighter-rouge">ContentIndexer</code> and these <code class="language-plaintext highlighter-rouge">ScatherGatherBuffers+MemoryChunk</code>?
When we look at the actual instances (thanks for the reminder <a href="https://blog.tech-fellow.net/">Valdis</a>), we on top of the core issue.</p>
<p class="centered-image">
<img src="/assets/memory-leak-2/2.memorychunk-instances.jpg" alt="Usage of ScatherGatherBuffers+MemoryChunk - exceptions" />
</p>
<p>The indexer tries to index all items, serializing its content in the process. Whenever there’s an exception the indexer adds <em>the exception and the batch of content to a list</em> in order retry indexing later on. For example trying to recover from a <code class="language-plaintext highlighter-rouge">427 request too large exception</code> with a smaller batch size. When a lot of exceptions happen, and the content or its attachment size is large, this can cause massive memory pressure even to the extent of recycling the process!</p>
<p class="centered-image">
<img src="/assets/memory-leak-2/4.memory-usage.jpg" alt="Memory usage" />
</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1061681031/ROOT
Process ID: 2184
Exception: System.OutOfMemoryException
Message: Exception of type 'System.OutOfMemoryException' was thrown.
StackTrace: at System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
</code></pre></div></div>
<p>The issue we had could partially be fixed in our own implementation, for each attachment we should verify ourselves if it’s not too big to index. If you don’t do that, an exception will be thrown as the Find backend will not accept the too large attachment. Therefore be sure to adhere to the <a href="https://world.optimizely.com/documentation/developer-guides/search-navigation/NET-Client-API/Indexing/">attachment size limits</a>.</p>
<p>The example provided by Optimizely works but can be improved to speed up indexing performance. As-is it will have to load all attachment every time it indexes, even the large attachments which can’t be indexed.
An improved solution would be to store the file size in a property while saving the media data. The find index job can then just skip the files by checking that property, instead of loading the attachment and checking its size each time during indexing.</p>
<p>I’ve provided feedback to Optimizely to see if they can add attachment size checking in the library, as it will simplify implementation for partners (can’t forget to check size) and reduce load on the Find backend services.</p>
<h3 id="but-wait-why-are-the-exceptions-using-so-much-memory">But wait.. why are the exceptions using so much memory?</h3>
<p>While I showed you <em>what</em> was using the memory, I didn’t explain <em>why</em> the exceptions are using up so much. By reading the instance path we can explain what happens:</p>
<p class="centered-image">
<img src="/assets/memory-leak-2/3.memorychunk-path.jpg" alt="Usage of ScatherGatherBuffers+MemoryChunk - reading the path" />
</p>
<p>Reading from the <code class="language-plaintext highlighter-rouge">ContentIndexer</code> up the path we find a <code class="language-plaintext highlighter-rouge">ConcurrentQueue</code> (used for indexing) and a <code class="language-plaintext highlighter-rouge">Find.ServiceException</code>. This in turn contains a <code class="language-plaintext highlighter-rouge">WebException</code> and then goes all the way up to the <code class="language-plaintext highlighter-rouge">ScatherGatherBuffers+MemoryChunk</code>. What this means is that the <code class="language-plaintext highlighter-rouge">Find.ServiceException</code> actually has a reference to a <code class="language-plaintext highlighter-rouge">WebException</code>, which turns out to be the inner exception. From what I could see, is that there is little need to keep the actual WebException alive. Instead of using it as an inner exception, just extract whatever information you need: e.g. which type of exception is it, check if it’s possible to recover from the exception by retrying.</p>
<p>Decompiling and drilling down confirmed these findings. As said the solution for Optimizely to fix this is relatively simple: extract whatever information they need from the <code class="language-plaintext highlighter-rouge">WebException</code> but don’t add it as inner exception. Optimizely has recognized this bug and it can be found in the bug tracker: <a href="https://world.optimizely.com/support/bug-list/bug/FIND-9858">FIND-9858</a>.</p>
<p>The key takeaways are:</p>
<p><strong>Tip 1</strong>: Try to fix all exceptions that are thrown during indexing, it will speed up indexing and prevent unneccesary memory usage.</p>
<p><strong>Tip 2</strong>: Be sure to put a size limit on media data attachments, either using the provided code by <a href="(https://world.optimizely.com/documentation/developer-guides/search-navigation/NET-Client-API/Indexing/)">Optimizely</a> or by storing the size upon saving the file and filtering out all large attachments</p>
<p><strong>Tip 3</strong>: In general: Think about your exceptions, what do you want to do with them? If you keep them alive try to extract what you need instead of storing everything</p>
Tracking down initialization issues in Optimizelyhttps://www.brianweet.com/2022/01/04/initialization-issues.html2022-01-04T09:00:00.0000000Z<p>Recently I was working on an Optimizely project which had some interesting issues, which seemed to happen right after startup.
For no apparent reason some of the properties on a page were <code class="language-plaintext highlighter-rouge">null</code>.</p>
<p class="centered-image">
<img src="/assets/initialization-issues/1.empty-props.png" alt="Empty props" />
</p>
<p>The investigation started:</p>
<ol>
<li>Check the CMS to be sure the properties are filled in
<ul>
<li>Yes, nothing to see here</li>
</ul>
</li>
<li>Verify the id and version of the page
<ul>
<li>Correct</li>
</ul>
</li>
<li>Clear cache and load the page again (in code)
<ul>
<li>Hey, now it does work properly!</li>
</ul>
</li>
</ol>
<p>Optimizely caches the page the first time it is retrieved, for some reason not all properties were populated in the cached version of the page. More specificly: the properties referencing <strong>commerce content</strong> were not populated.. Hmm so maybe something is wrong during the initialization of commerce somehow?!</p>
<p>After a bit of investigation I found some code that was executed early on in the startup process, and I could throw an exception to check the call stack and figure out what was going on:</p>
<p class="centered-image">
<img src="/assets/initialization-issues/2.stack.png" alt="Stack trace to find the culprit" />
</p>
<p>Looks like we’ve found the culprit! We have a custom actor for Optimizely Forms which is instantiated during initialization. In itself that shouldn’t cause any problems, so what’s the issue here?</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ComplaintsActor</span> <span class="p">:</span> <span class="n">PostSubmissionActorBase</span><span class="p">,</span> <span class="n">IUIPropertyCustomCollection</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IProjectFormService</span> <span class="n">_projectFormService</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IContentLoader</span> <span class="n">_contentLoader</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ComplaintsActor</span><span class="p">()</span> <span class="p">:</span> <span class="k">this</span><span class="p">(</span><span class="n">ServiceLocator</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">GetInstance</span><span class="p"><</span><span class="n">IProjectFormService</span><span class="p">>(),</span>
<span class="n">ServiceLocator</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">GetInstance</span><span class="p"><</span><span class="n">IContentLoader</span><span class="p">>())</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">public</span> <span class="nf">ComplaintsActor</span><span class="p">(</span><span class="n">IProjectFormService</span> <span class="n">projectFormService</span><span class="p">,</span> <span class="n">IContentLoader</span> <span class="n">contentLoader</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_projectFormService</span> <span class="p">=</span> <span class="n">projectFormService</span><span class="p">;</span>
<span class="n">_contentLoader</span> <span class="p">=</span> <span class="n">contentLoader</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
</code></pre></div></div>
<p>Turns out we initialize a couple of services explicitly during startup by using the ServiceLocator. One of the services tries to load our mysterious page with <code class="language-plaintext highlighter-rouge">null</code> properties and it gets cached correspondingly. The fix is easy, we don’t need those services right away and can just use property injection instead:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ComplaintsActor</span> <span class="p">:</span> <span class="n">PostSubmissionActorBase</span><span class="p">,</span> <span class="n">IUIPropertyCustomCollection</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Injected</span><span class="p"><</span><span class="n">IProjectFormService</span><span class="p">></span> <span class="n">ProjectFormService</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Injected</span><span class="p"><</span><span class="n">IContentLoader</span><span class="p">></span> <span class="n">ContentLoader</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ComplaintsActor</span><span class="p">()</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">...</span>
</code></pre></div></div>
<p><strong>Tip 1</strong>: prevent initializing using the ServiceLocator if not necessary</p>
<p><strong>Tip 2</strong>: prevent loading content during initialization of a service</p>
Implementing a custom reporting service for commercehttps://www.brianweet.com/2020/04/29/implementing-a-custom-reporting-service-for-commerce.html2020-04-29T10:30:00.0000000Z<p>I’ve been trying out the <a href="/link/eb8697ae5cf648f7b9f88a979c14d07f.aspx">BETA functionality for generating commerce reports</a> in Episerver. By default, it will create an CSV export of all orders made that day and it allows for some extensibility. In this post I’ll show you how you can add extra data to the report and I’ll show you how to customize the report completely by implementing your own IReportingService.</p>
<h2 id="reports-for-commerce">Reports for Commerce</h2>
<p>Sales reports in <a href="https://webhelp.episerver.com/latest/en/commerce/reporting/reporting-commerce-data.htm">Commerce Manager</a> (CM) have existed for a long time. But, as Episerver is slowly phasing out CM, it comes to no surprise that they’ve started to move some of that functionality to the CMS as well.
Episerver has enabled the BETA functionality for Commerce Reports by default. Reports are created by the scheduled job called <code class="language-plaintext highlighter-rouge">Collect Report Data</code>, which will create a zipped up CSV file with the orders for that day:</p>
<p class="centered-image">
<img src="/assets/epi-reporting/0.commerce-report.png" alt="Commerce report" />
</p>
<p>It’s not hard to add custom data to this report, as described on <a href="/link/eb8697ae5cf648f7b9f88a979c14d07f.aspx#additional_properties">this page on World</a>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Implement your own additional data handler</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">MyReportingAdditionalDataHandler</span> <span class="p">:</span> <span class="n">ReportingAdditionalDataHandler</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IOrderRepository</span> <span class="n">_orderRepository</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">MyReportingAdditionalDataHandler</span><span class="p">(</span><span class="n">IOrderRepository</span> <span class="n">orderRepository</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_orderRepository</span> <span class="p">=</span> <span class="n">orderRepository</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">override</span> <span class="n">IEnumerable</span><span class="p"><</span><span class="kt">string</span><span class="p">></span> <span class="nf">GetAdditionalData</span><span class="p">(</span><span class="kt">int</span> <span class="n">lineItemId</span><span class="p">,</span> <span class="kt">int</span> <span class="n">orderGroupId</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">order</span> <span class="p">=</span> <span class="n">_orderRepository</span><span class="p">.</span><span class="n">Load</span><span class="p"><</span><span class="n">IPurchaseOrder</span><span class="p">>(</span><span class="n">orderGroupId</span><span class="p">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="kt">string</span><span class="p">[]</span>
<span class="p">{</span>
<span class="c1">// Add additional data here</span>
<span class="n">order</span><span class="p">?.</span><span class="n">Properties</span><span class="p">[</span><span class="s">"MyCustomProp"</span><span class="p">]?.</span><span class="nf">ToString</span><span class="p">()</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">//Register it in</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">StructureMapRegistry</span> <span class="p">:</span> <span class="n">Registry</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">StructureMapRegistry</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">//...</span>
<span class="n">For</span><span class="p"><</span><span class="n">ReportingAdditionalDataHandler</span><span class="p">>().</span><span class="nf">Singleton</span><span class="p">().</span><span class="n">Use</span><span class="p"><</span><span class="n">MyReportingAdditionalDataHandler</span><span class="p">>();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>However, the extensibility is a bit lacking a bit as you can’t add any headers to the csv file. Next to that, <strong>it doesn’t seem to escape CSV data properly yet</strong>, for which a bug has been created (COM-11284, not public yet). I ended up with invalid CSV as some product names contained a comma, which broke the CSV.</p>
<h2 id="implementing-a-custom-reporting-service">Implementing a custom reporting service</h2>
<p>Fortunately it is really easy to implement your own reporting service! All you have to do is implement a IReportingService yourself. By doing so, you’re free to implement anything you want. What I’ll do is:</p>
<ul>
<li>Re-use the code from Epi in order to gather order data
<ul>
<li>My service will be quite similar to the one from Episerver, with some additional fields</li>
</ul>
</li>
<li>Map it to a custom object
<ul>
<li>Simple POCO object with some header names</li>
</ul>
</li>
<li>Write it as CSV inside a zip file
<ul>
<li>I’ll use the CsvHelper library, which will handle writing CSV for me</li>
</ul>
</li>
</ul>
<p>The code:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">ReportingService</span> <span class="p">:</span> <span class="n">IReportingService</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IBlobFactory</span> <span class="n">_blobFactory</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IContentRepository</span> <span class="n">_contentRepository</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IUrlSegmentGenerator</span> <span class="n">_urlSegmentGenerator</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">ReportingDataLoader</span> <span class="n">_reportingDataLoader</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">OrderReportingMapper</span> <span class="n">_orderReportingMapper</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ReportingService</span><span class="p">(</span>
<span class="n">IBlobFactory</span> <span class="n">blobFactory</span><span class="p">,</span>
<span class="n">IContentRepository</span> <span class="n">contentRepository</span><span class="p">,</span>
<span class="n">IUrlSegmentGenerator</span> <span class="n">urlSegmentGenerator</span><span class="p">,</span>
<span class="n">ReportingDataLoader</span> <span class="n">reportingDataLoader</span><span class="p">,</span>
<span class="n">OrderReportingMapper</span> <span class="n">orderReportingMapper</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_blobFactory</span> <span class="p">=</span> <span class="n">blobFactory</span><span class="p">;</span>
<span class="n">_contentRepository</span> <span class="p">=</span> <span class="n">contentRepository</span><span class="p">;</span>
<span class="n">_urlSegmentGenerator</span> <span class="p">=</span> <span class="n">urlSegmentGenerator</span><span class="p">;</span>
<span class="n">_reportingDataLoader</span> <span class="p">=</span> <span class="n">reportingDataLoader</span><span class="p">;</span>
<span class="n">_orderReportingMapper</span> <span class="p">=</span> <span class="n">orderReportingMapper</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">virtual</span> <span class="n">ContentReference</span> <span class="nf">ExportOrderDataAsCsv</span><span class="p">(</span>
<span class="n">DateTime</span> <span class="n">fromDate</span><span class="p">,</span>
<span class="n">DateTime</span> <span class="n">toDate</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// This is where the reports are stored by epi</span>
<span class="kt">var</span> <span class="n">reportingMediaData</span> <span class="p">=</span> <span class="n">_contentRepository</span><span class="p">.</span><span class="n">GetDefault</span><span class="p"><</span><span class="n">ReportingMediaData</span><span class="p">>(</span><span class="n">CommerceReportingFolder</span><span class="p">.</span><span class="n">ReportingRoot</span><span class="p">);</span>
<span class="c1">// We can create a blob to hold our data</span>
<span class="kt">var</span> <span class="n">blob</span> <span class="p">=</span> <span class="n">_blobFactory</span><span class="p">.</span><span class="nf">CreateBlob</span><span class="p">(</span><span class="n">reportingMediaData</span><span class="p">.</span><span class="n">BinaryDataContainer</span><span class="p">,</span> <span class="s">".commercereport"</span><span class="p">);</span>
<span class="c1">// Default naming</span>
<span class="kt">var</span> <span class="n">str</span> <span class="p">=</span> <span class="s">"OrderData-from-"</span> <span class="p">+</span> <span class="n">fromDate</span><span class="p">.</span><span class="nf">ToString</span><span class="p">(</span><span class="s">"dd-MMM-yyyy"</span><span class="p">,</span> <span class="n">CultureInfo</span><span class="p">.</span><span class="n">InvariantCulture</span><span class="p">)</span> <span class="p">+</span> <span class="s">"-to-"</span> <span class="p">+</span> <span class="n">toDate</span><span class="p">.</span><span class="nf">ToString</span><span class="p">(</span><span class="s">"dd-MMM-yyyy"</span><span class="p">,</span> <span class="n">CultureInfo</span><span class="p">.</span><span class="n">InvariantCulture</span><span class="p">);</span>
<span class="c1">// Open blob, create an zip archive and a csv entry</span>
<span class="k">using</span> <span class="p">(</span><span class="kt">var</span> <span class="n">stream</span> <span class="p">=</span> <span class="n">blob</span><span class="p">.</span><span class="nf">OpenWrite</span><span class="p">())</span>
<span class="k">using</span> <span class="p">(</span><span class="kt">var</span> <span class="n">zipArchive</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ZipArchive</span><span class="p">(</span><span class="n">stream</span><span class="p">,</span> <span class="n">ZipArchiveMode</span><span class="p">.</span><span class="n">Create</span><span class="p">,</span> <span class="k">false</span><span class="p">))</span>
<span class="k">using</span> <span class="p">(</span><span class="kt">var</span> <span class="n">entryStream</span> <span class="p">=</span> <span class="n">zipArchive</span><span class="p">.</span><span class="nf">CreateEntry</span><span class="p">(</span><span class="n">str</span> <span class="p">+</span> <span class="s">".csv"</span><span class="p">).</span><span class="nf">Open</span><span class="p">())</span>
<span class="k">using</span> <span class="p">(</span><span class="kt">var</span> <span class="n">streamWriter</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">StreamWriter</span><span class="p">(</span><span class="n">entryStream</span><span class="p">))</span>
<span class="k">using</span> <span class="p">(</span><span class="kt">var</span> <span class="n">csv</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">CsvWriter</span><span class="p">(</span><span class="n">streamWriter</span><span class="p">,</span> <span class="n">CultureInfo</span><span class="p">.</span><span class="n">InvariantCulture</span><span class="p">))</span>
<span class="p">{</span>
<span class="c1">// Use default functionality to load order report data</span>
<span class="kt">var</span> <span class="n">data</span> <span class="p">=</span> <span class="n">_reportingDataLoader</span><span class="p">.</span><span class="nf">GetReportingData</span><span class="p">(</span>
<span class="n">fromDate</span><span class="p">.</span><span class="nf">ToUniversalTime</span><span class="p">(),</span>
<span class="n">toDate</span><span class="p">.</span><span class="nf">ToUniversalTime</span><span class="p">()</span>
<span class="p">);</span>
<span class="c1">// Use a custom mapper and write csv</span>
<span class="n">csv</span><span class="p">.</span><span class="nf">WriteRecords</span><span class="p">(</span><span class="n">_orderReportingMapper</span><span class="p">.</span><span class="nf">Map</span><span class="p">(</span><span class="n">data</span><span class="p">));</span>
<span class="p">}</span>
<span class="c1">// Default functionality, 'overwrite' any existing report</span>
<span class="n">reportingMediaData</span><span class="p">.</span><span class="n">BinaryData</span> <span class="p">=</span> <span class="n">blob</span><span class="p">;</span>
<span class="n">reportingMediaData</span><span class="p">.</span><span class="n">Name</span> <span class="p">=</span> <span class="n">str</span> <span class="p">+</span> <span class="s">".zip"</span><span class="p">;</span>
<span class="nf">DeleteDuplicatedReport</span><span class="p">(</span><span class="n">reportingMediaData</span><span class="p">.</span><span class="n">Name</span><span class="p">);</span>
<span class="k">return</span> <span class="n">_contentRepository</span><span class="p">.</span><span class="nf">Save</span><span class="p">(</span><span class="n">reportingMediaData</span><span class="p">,</span> <span class="n">SaveAction</span><span class="p">.</span><span class="n">Publish</span><span class="p">,</span> <span class="n">AccessLevel</span><span class="p">.</span><span class="n">NoAccess</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">//Removed the rest for brevity</span>
<span class="p">}</span>
<span class="c1">// Simple mapper to create a POCO</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">OrderReportingMapper</span>
<span class="p">{</span>
<span class="k">public</span> <span class="n">IEnumerable</span><span class="p"><</span><span class="n">OrderLineItemRecord</span><span class="p">></span> <span class="nf">Map</span><span class="p">(</span><span class="n">IEnumerable</span><span class="p"><</span><span class="n">LineItemReportingModel</span><span class="p">></span> <span class="n">orders</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">orders</span><span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="n">MapItem</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="n">OrderLineItemRecord</span> <span class="nf">MapItem</span><span class="p">(</span><span class="n">LineItemReportingModel</span> <span class="n">item</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// Add additional vipps data to commerce-reports</span>
<span class="kt">var</span> <span class="n">order</span> <span class="p">=</span> <span class="n">_orderRepository</span><span class="p">.</span><span class="n">Load</span><span class="p"><</span><span class="n">IPurchaseOrder</span><span class="p">>(</span><span class="n">item</span><span class="p">.</span><span class="n">OrderGroupId</span><span class="p">);</span>
<span class="k">return</span> <span class="k">new</span> <span class="n">OrderLineItemRecord</span>
<span class="p">{</span>
<span class="n">LineItemId</span> <span class="p">=</span> <span class="n">item</span><span class="p">.</span><span class="n">LineItemId</span><span class="p">,</span>
<span class="n">MyCustomProp</span> <span class="p">=</span> <span class="n">order</span><span class="p">?.</span><span class="n">Properties</span><span class="p">[</span><span class="s">"MyCustomProp"</span><span class="p">]?.</span><span class="nf">ToString</span><span class="p">()</span>
<span class="p">};</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// POCO for writing csv</span>
<span class="k">using</span> <span class="nn">CsvHelper.Configuration.Attributes</span><span class="p">;</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">OrderLineItemRecord</span>
<span class="p">{</span>
<span class="p">[</span><span class="nf">Name</span><span class="p">(</span><span class="s">"Line Item Id"</span><span class="p">)]</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">LineItemId</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">[</span><span class="nf">Name</span><span class="p">(</span><span class="s">"My Custom Prop"</span><span class="p">)]</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">MyCustomProp</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>A complete implementation can be found on <a href="https://gist.github.com/brianweet/e5229dd34aed66875a7db39babb607b3">on GitHub</a>.
When you run the <code class="language-plaintext highlighter-rouge">Collect Report Data</code> job, your custom report will be generated and can be downloaded from the reports tab.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this blog post we’ve looked at the BETA functionality for generating reports in the CMS. We’ve seen how to add additional data to the default CSV report. And we’ve implemented a custom reporting service which will allow you to easily export files whatever you need.</p>
Introducing: AI content editor for Epihttps://www.brianweet.com/2019/04/15/introducing-ai-content-editor.html2019-04-15T22:00:00.0000000Z<p>About two months ago the blog post <a href="https://openai.com/blog/better-language-models/">‘Better Language Models
and Their Implications’</a> came out. In case you heard nothing about it, researchers at OpenAI wrote a paper about a language model called <em>GPT-2</em>. The model is able to generate text that approaches the quality of human written texts. It is an interesting read and definitely food for thought. I had a laugh reading some of the generated texts:</p>
<p class="centered image">
<img src="/assets/ai-content-editor/magical-unicorn.jpg" alt="Magical unicorn" />
</p>
<p><em><a href="https://www.freepik.com/free-photos-vectors/background">Background vector created by freepik - www.freepik.com</a></em></p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 10 TRIES)</strong></p>
<p>The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.</p>
<p>Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.</p>
<p>Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.”</p>
</blockquote>
<p><a href="https://openai.com/blog/better-language-models/#sample1">Read more here - it’s quite funny</a></p>
<p>An article by <a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction">The Guardian</a> provided a couple of different examples, where they let the model generate text about itself, which seems shockingly legit to me as it’s also generating quotes and references (last paragraph of the following snippet).</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/guardian-example.png" alt="Guardian example" />
</p>
<p><a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction">Full article here</a></p>
<p>I thought OpenAI’s paper was pretty cool and noticed that they put some code online.. It’s time to play!</p>
<h3 id="using-gpt-2-to-generate-content">Using GPT-2 to generate content</h3>
<p>The OpenAI researchers decided not to release the full model or the dataset they used to train the model. They did however, release a smaller, pre-trained model we can use to generate text. Their repository can be found <a href="https://github.com/openai/gpt-2/">here</a> and to my excitement (because I’m lazy when I can be) it includes <a href="https://github.com/openai/gpt-2/blob/master/Dockerfile.gpu">two</a> <a href="https://github.com/openai/gpt-2/blob/master/Dockerfile.cpu">Dockerfiles</a> that can be used to run the model locally. There are two ways you can use the model to generate text: conditionally or unconditionally.</p>
<ul>
<li>Conditionally: you will be prompted to enter an input text and the model will generate text based on the input</li>
<li>Unconditionally: the model will generate random samples of text</li>
</ul>
<p>In this post we’ll use conditional samples from the GPT-2 model and see how we can integrate it with Episervers <a href="https://github.com/episerver/AlloyDemoKit">AlloyDemoKit</a>.</p>
<h3 id="architecture">Architecture</h3>
<p>As mentioned, the GPT-2 repository contains Dockerfiles to run the model. This is quite convenient because this way I don’t have to set up any of the requirements, like python or tensorflow, on my local machine. We will run the GPT-2 model as an “micro-service” which communicates using http requests (just a simple WebApi). A while ago I wrote about running <a href="/2017/10/09/alloy-in-docker.html">Alloy in Docker</a>, which is convenient as I can re-use it for this post and hook up the Alloy container to the GPT-2 container using service-to-service (container to container) communication.</p>
<p>The solution will consist of two containers, one running Alloy (on Windows) and one running the GPT-2 model (on Linux). We’ll let the GPT-2 container expose a simple web API which wraps around the code to generate conditional texts (albeit slightly modified). This will will allow us to do a request from the Alloy container to GPT-2 container and let the model generate text. <em>note: I did rebuild docker with different <a href="/2018/04/26/running-mssql-server-linux-using-lcow.html">memory</a> <a href="https://github.com/brianweet/moby/commit/cf55f29f9d494dd05450f75f80ed43190c3df110">constraints/defaults</a></em></p>
<p class="centered-image">
<img src="/assets/ai-content-editor/gpt-2-docker.png" alt="GPT-2 docker architecture" />
</p>
<ul>
<li>Alloy
<ul>
<li>Public web api - acts as a facade
<ul>
<li>Does a request to GPT-2 web api</li>
<li><code class="highlighter-rouge">ContentEditorApiController</code> –> <code class="highlighter-rouge">/api/ai-contenteditor/please-finish-my?sentence={input}</code></li>
<li>Can be used to intercept / cleanup / throttle / add extra layer (queue)</li>
</ul>
</li>
<li>Add GPT-2 AI button to TinyMCE
<ul>
<li>Uses existing content in TinyMCE editor</li>
<li>Does a request to the <code class="highlighter-rouge">api/ai-contenteditor</code></li>
<li>Replaces content in editor with generated text</li>
</ul>
</li>
</ul>
</li>
<li>GPT-2
<ul>
<li>Private web api (service-to-service communication)
<ul>
<li><code class="highlighter-rouge">http://alloydemokit-gpt-2:5000?input={input}</code></li>
<li>Generates text conditionally, based on input</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>If you’re interested in the docker set-up, you can take a look at the <a href="https://github.com/brianweet/AlloyDemoKit/blob/proto/gpt-2/src/docker-compose.yml">docker-compose</a> file, the <a href="https://github.com/brianweet/AlloyDemoKit/blob/proto/gpt-2/src/AlloyDemoKit/Dockerfile">Dockerfile for Alloy</a> and the <a href="https://github.com/brianweet/gpt-2/blob/97fb8e07a723a2cab5be0b5df9694363fbe6d0a4/Dockerfile">Dockerfile for GPT-2</a>. Or you can clone the repo and run it yourself using VS.</p>
<h3 id="adding-ai-to-the-tinymce-editor">Adding AI to the TinyMCE editor</h3>
<p>In order to generate text <strong>conditionally</strong>, we will need input text. We’ll create a plugin for TinyMCE, this plugin will consist of nothing more than a simple button that uses the exiting content in the TinyMCE editor as input for the model.</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/epi-tinymce-ai-plugin.png" alt="Add AI capabilities to the TinyMCE editor" />
</p>
<p>Creating a TinyMCE plugin is pretty easy, I’ve used the <a href="https://www.tiny.cloud/docs/advanced/yeoman-generator/">yeoman generator</a> to scaffold a project.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yarn global add yo generator-tinymce
yo tinymce
? Package name: epi-tinymce-ai-plugin
? Add a description: Add AI capabilities to the TinyMCE editor
</code></pre></div></div>
<p>We now have a <code class="highlighter-rouge">main/Plugin.ts</code> to work with, it already provides an example of adding a button to the TinyMCE editor. We will extend it to read the current content in the editor, do a request to our content editor api and replaces the content with the result.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">setup</span> <span class="o">=</span> <span class="p">(</span><span class="nx">editor</span><span class="p">:</span> <span class="nx">any</span><span class="p">,</span> <span class="nx">url</span><span class="p">:</span> <span class="nx">any</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">editor</span><span class="p">.</span><span class="nx">addButton</span><span class="p">(</span><span class="s1">'epi-tinymce-ai-plugin'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tooltip</span><span class="p">:</span> <span class="s1">'Generate text'</span><span class="p">,</span>
<span class="na">image</span><span class="p">:</span> <span class="s1">'https://openai.com/favicon.png'</span><span class="p">,</span>
<span class="na">onclick</span><span class="p">:</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Get input from TinyMCE editor</span>
<span class="kd">const</span> <span class="na">input</span><span class="p">:</span> <span class="nx">string</span> <span class="o">=</span> <span class="nx">editor</span><span class="p">.</span><span class="nx">getBody</span><span class="p">().</span><span class="nx">textContent</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="nx">pleaseWaitMessage</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">alert</span><span class="p">(</span><span class="s1">'Please wait a bit or remove the wait message.'</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Init waiting state</span>
<span class="kd">const</span> <span class="nx">intervalId</span> <span class="o">=</span> <span class="nx">startWait</span><span class="p">(</span><span class="nx">editor</span><span class="p">);</span>
<span class="c1">// Do request</span>
<span class="kd">const</span> <span class="nx">inputUrl</span> <span class="o">=</span> <span class="nx">getUrlFromInput</span><span class="p">(</span><span class="nx">input</span><span class="p">);</span>
<span class="nx">fetch</span><span class="p">(</span><span class="nx">inputUrl</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span><span class="p">((</span><span class="nx">response</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">stopWait</span><span class="p">(</span><span class="nx">editor</span><span class="p">,</span> <span class="nx">intervalId</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">response</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">checkResponse</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">useResponse</span><span class="p">(</span><span class="nx">editor</span><span class="p">))</span> <span class="c1">// Use response to populate TinyMCE editor</span>
<span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">handleError</span><span class="p">(</span><span class="nx">editor</span><span class="p">,</span> <span class="nx">input</span><span class="p">));</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="p">};</span>
<span class="nx">tinymce</span><span class="p">.</span><span class="nx">PluginManager</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="s1">'epi-tinymce-ai-plugin'</span><span class="p">,</span> <span class="nx">setup</span><span class="p">);</span>
</code></pre></div></div>
<p><em>The full code for the plugin can be found <a href="https://github.com/brianweet/epi-tinymce-ai-plugin/blob/master/src/main/ts/Plugin.ts">here</a></em></p>
<p>The TinyMCE button will call the <code class="highlighter-rouge">ContentEditorApiController</code>, which in turn calls the api on the GPT-2 container. The ContentEditorApiController does some cleanup and returns the result back to the browser. The result is used to replace the TinyMCE editor content.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[RoutePrefix("api/ai-contenteditor")]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">ContentEditorApiController</span> <span class="p">:</span> <span class="n">ApiController</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">HttpClient</span><span class="p">></span> <span class="n">LazyHttpClient</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">HttpClient</span><span class="p">>(()</span> <span class="p">=></span>
<span class="k">new</span> <span class="n">HttpClient</span> <span class="p">{</span> <span class="n">BaseAddress</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Uri</span><span class="p">(</span><span class="s">"http://alloydemokit-gpt-2:5000"</span><span class="p">)</span> <span class="p">});</span>
<span class="p">[</span><span class="n">HttpGet</span><span class="p">]</span>
<span class="p">[</span><span class="nf">Route</span><span class="p">(</span><span class="s">"please-finish-my"</span><span class="p">)]</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">IHttpActionResult</span><span class="p">></span> <span class="nf">Get</span><span class="p">(</span><span class="kt">string</span> <span class="n">sentence</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">sentence</span><span class="p">))</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nf">BadRequest</span><span class="p">(</span><span class="k">nameof</span><span class="p">(</span><span class="n">sentence</span><span class="p">));</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">input</span> <span class="p">=</span> <span class="n">sentence</span><span class="p">.</span><span class="nf">Trim</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">result</span> <span class="p">=</span> <span class="k">await</span> <span class="n">LazyHttpClient</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="nf">GetStringAsync</span><span class="p">(</span><span class="s">$"?input=</span><span class="p">{</span><span class="n">input</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="c1">// Use result up to <|endoftext|> (the rest seems to be random gibberish)</span>
<span class="kt">var</span> <span class="n">cleanResult</span> <span class="p">=</span> <span class="n">result</span>
<span class="p">.</span><span class="nf">Split</span><span class="p">(</span><span class="k">new</span><span class="p">[]</span> <span class="p">{</span> <span class="s">"<|endoftext|>"</span> <span class="p">},</span> <span class="n">StringSplitOptions</span><span class="p">.</span><span class="n">RemoveEmptyEntries</span><span class="p">)</span>
<span class="p">.</span><span class="nf">FirstOrDefault</span><span class="p">();</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="k">new</span> <span class="n">GeneratedContentResult</span>
<span class="p">{</span>
<span class="n">Input</span> <span class="p">=</span> <span class="n">input</span><span class="p">,</span>
<span class="n">Result</span> <span class="p">=</span> <span class="n">result</span><span class="p">,</span>
<span class="n">CleanResult</span> <span class="p">=</span> <span class="s">$"</span><span class="p">{</span><span class="n">input</span><span class="p">}{</span><span class="n">cleanResult</span><span class="p">}</span><span class="s">"</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>All there’s left is the GPT-2 web api, which is an API wrapper around the code for conditional text generation:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">generate_conditional</span> <span class="kn">import</span> <span class="n">conditional</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="nd">@app.route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="nb">input</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'input'</span><span class="p">)</span>
<span class="n">top_k</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'top_k'</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="k">return</span> <span class="n">conditional</span><span class="p">(</span><span class="n">raw_text_input</span><span class="o">=</span><span class="nb">input</span><span class="p">,</span> <span class="n">top_k</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">top_k</span><span class="p">))</span>
</code></pre></div></div>
<p><em><a href="https://github.com/brianweet/gpt-2/blob/master/src/app.py">Python app here</a>, slightly modified <a href="https://github.com/brianweet/gpt-2/blob/master/src/generate_conditional.py">conditional method here</a> as the provided example is a console application</em></p>
<h3 id="giving-it-a-spin">Giving it a spin</h3>
<p>Let’s put the AI content editor to work and feed it the first paragraph of this blog post:</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/ai-writing.gif" alt="AI writing my blog post" />
</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>About two months ago the blog post ‘Better Language Models and Their Implications’ came out. In case you heard nothing about it, researchers at OpenAI wrote a paper about a language model called GPT-2. The model is able to generate text that approaches the quality of human written texts. It is an interesting read and definitely food for thought. I had a laugh reading some of the generated texts:</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong></p>
<p>these are wonderful examples of GPT-2’s natural learning. We have created it in Python by default but not once did we check out many variables around it (many of which could have been explained by reading the GPT-2 data). As a blog post from the future I want to take this opportunity and talk about the actual results of the GPT-2 model. The results.↵GPT-2 models are not perfect (which might be why there are so many comparisons to be made). For one thing, all the data generated by our data manipulation is quite small. (This means it has a small chance of being too narrow). You might also have to find some way to control GPT-2’s structure (one of the most important things in design is not to draw big numbers in that way in the book).</p>
</blockquote>
<p>Or let’s have it create a list of requirements for our integration:</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>In order to achieve the most successful integration with an AI language model like GPT-2, you’ll have to make sure you meet the following three requirements:</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong>
1) Use your existing AI with the new version 2) Configure the newly installed GPT-2 firmware 3) Launch the latest version of your GPT-2 firmware on Ubuntu-based systems If you fail this latter step, the new version of your GPT-2 will be unable to implement the required methods of execution, either in /etc/init.d/gpt2 and /var/lib/gpt2/boot-all.2 as in /etc/init.d/gpt2 A new version of /etc/init.d should be created within the ‘default-options’ directory where things get complicated and if you do not have your existing GPT-2, your original kernel would be ignored.</p>
</blockquote>
<p>Lastly, we need some promotional texts for our company website:</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>Our company is a great example of a modern work place and</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong></p>
<p>the city. We’ve been around for a long time but our company is still moving forward. So as always, we just want to make sure everyone’s focused on their work. This is one of the best reasons we’re so focused on the development of the city and the development of the industry. As a city you’re going to have to push for them. I’ve always been on the fence so I guess I’m still thinking about their situation, other than the fact that we’re at a very early stage just to find out what kind of jobs are suitable for them. Our company is looking to be the best part of it, to have our office as soon as possible and that is for sure.</p>
</blockquote>
<p>I would say it’s readable, even though it does not make complete sense all the time. But bear in mind that this is just a small training model as OpenAI did not release the full model. This is an area we should keep an eye on and, every now and then, play around with. It’s fun to see how the models and the concepts are progressing and what this may offer us in the future.</p>
<h3 id="conclusion">Conclusion</h3>
<p>In this blog post we’ve added AI language model capabilities from GPT-2 to the TinyMCE editor in Episerver. The language model generates text for us based on our own editorial input. We’ve used docker and docker-compose to run the GPT-2 model as a microservice in a separate linux container and created a facade web api in Alloy to complete the integration.</p>
<p>I had a lot of fun toying with the GPT-2 model, for some reason it is quite addictive to generate <a href="https://gist.github.com/brianweet/8effc2098957a59f942111464c4145fe">weird</a> texts by playing around with different inputs.</p>
<p>If you want to give it a try, I’m hosting it temporarily <a href="https://c48871ed.ngrok.io">here</a>. As the GPT-2 model is quite resource heavy you’ll have to be patient and if it breaks, too bad, perhaps give it a try locally!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>user: human-content-editor
pass: IamArobot1!
</code></pre></div></div>
<h4 id="up-next-ai-generated-c-code---">Up next: AI generated C# code (╯°□°)╯︵ ┻━┻</h4>
<style>
.videoWrapper {
position: relative;
padding-bottom: 56.25%; /* 16:9 */
padding-top: 25px;
height: 0;
}
.videoWrapper iframe {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
}
</style>
<p class="videoWrapper">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/toL1tXrLA1c" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</p>
Introducing: AI content editor for Epihttps://www.brianweet.com/2019/04/15/introducing-ai-content-editor.html2019-04-15T20:00:00.0000000Z<p>About two months ago the blog post <a href="https://openai.com/blog/better-language-models/">‘Better Language Models
and Their Implications’</a> came out. In case you heard nothing about it, researchers at OpenAI wrote a paper about a language model called <em>GPT-2</em>. The model is able to generate text that approaches the quality of human written texts. It is an interesting read and definitely food for thought. I had a laugh reading some of the generated texts:</p>
<p class="centered image">
<img src="/assets/ai-content-editor/magical-unicorn.jpg" alt="Magical unicorn" />
</p>
<p><em><a href="https://www.freepik.com/free-photos-vectors/background">Background vector created by freepik - www.freepik.com</a></em></p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 10 TRIES)</strong></p>
<p>The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.</p>
<p>Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.</p>
<p>Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.”</p>
</blockquote>
<p><a href="https://openai.com/blog/better-language-models/#sample1">Read more here - it’s quite funny</a></p>
<p>An article by <a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction">The Guardian</a> provided a couple of different examples, where they let the model generate text about itself, which seems shockingly legit to me as it’s also generating quotes and references (last paragraph of the following snippet).</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/guardian-example.png" alt="Guardian example" />
</p>
<p><a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction">Full article here</a></p>
<p>I thought OpenAI’s paper was pretty cool and noticed that they put some code online.. It’s time to play!</p>
<h3 id="using-gpt-2-to-generate-content">Using GPT-2 to generate content</h3>
<p>The OpenAI researchers decided not to release the full model or the dataset they used to train the model. They did however, release a smaller, pre-trained model we can use to generate text. Their repository can be found <a href="https://github.com/openai/gpt-2/">here</a> and to my excitement (because I’m lazy when I can be) it includes <a href="https://github.com/openai/gpt-2/blob/master/Dockerfile.gpu">two</a> <a href="https://github.com/openai/gpt-2/blob/master/Dockerfile.cpu">Dockerfiles</a> that can be used to run the model locally. There are two ways you can use the model to generate text: conditionally or unconditionally.</p>
<ul>
<li>Conditionally: you will be prompted to enter an input text and the model will generate text based on the input</li>
<li>Unconditionally: the model will generate random samples of text</li>
</ul>
<p>In this post we’ll use conditional samples from the GPT-2 model and see how we can integrate it with Episervers <a href="https://github.com/episerver/AlloyDemoKit">AlloyDemoKit</a>.</p>
<h3 id="architecture">Architecture</h3>
<p>As mentioned, the GPT-2 repository contains Dockerfiles to run the model. This is quite convenient because this way I don’t have to set up any of the requirements, like python or tensorflow, on my local machine. We will run the GPT-2 model as an “micro-service” which communicates using http requests (just a simple WebApi). A while ago I wrote about running <a href="/2017/10/09/alloy-in-docker.html">Alloy in Docker</a>, which is convenient as I can re-use it for this post and hook up the Alloy container to the GPT-2 container using service-to-service (container to container) communication.</p>
<p>The solution will consist of two containers, one running Alloy (on Windows) and one running the GPT-2 model (on Linux). We’ll let the GPT-2 container expose a simple web API which wraps around the code to generate conditional texts (albeit slightly modified). This will will allow us to do a request from the Alloy container to GPT-2 container and let the model generate text. <em>note: I did rebuild docker with different <a href="/2018/04/26/running-mssql-server-linux-using-lcow.html">memory</a> <a href="https://github.com/brianweet/moby/commit/cf55f29f9d494dd05450f75f80ed43190c3df110">constraints/defaults</a></em></p>
<p class="centered-image">
<img src="/assets/ai-content-editor/gpt-2-docker.png" alt="GPT-2 docker architecture" />
</p>
<ul>
<li>Alloy
<ul>
<li>Public web api - acts as a facade
<ul>
<li>Does a request to GPT-2 web api</li>
<li><code class="highlighter-rouge">ContentEditorApiController</code> –> <code class="highlighter-rouge">/api/ai-contenteditor/please-finish-my?sentence={input}</code></li>
<li>Can be used to intercept / cleanup / throttle / add extra layer (queue)</li>
</ul>
</li>
<li>Add GPT-2 AI button to TinyMCE
<ul>
<li>Uses existing content in TinyMCE editor</li>
<li>Does a request to the <code class="highlighter-rouge">api/ai-contenteditor</code></li>
<li>Replaces content in editor with generated text</li>
</ul>
</li>
</ul>
</li>
<li>GPT-2
<ul>
<li>Private web api (service-to-service communication)
<ul>
<li><code class="highlighter-rouge">http://alloydemokit-gpt-2:5000?input={input}</code></li>
<li>Generates text conditionally, based on input</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>If you’re interested in the docker set-up, you can take a look at the <a href="https://github.com/brianweet/AlloyDemoKit/blob/proto/gpt-2/src/docker-compose.yml">docker-compose</a> file, the <a href="https://github.com/brianweet/AlloyDemoKit/blob/proto/gpt-2/src/AlloyDemoKit/Dockerfile">Dockerfile for Alloy</a> and the <a href="https://github.com/brianweet/gpt-2/blob/97fb8e07a723a2cab5be0b5df9694363fbe6d0a4/Dockerfile">Dockerfile for GPT-2</a>. Or you can clone the repo and run it yourself using VS.</p>
<h3 id="adding-ai-to-the-tinymce-editor">Adding AI to the TinyMCE editor</h3>
<p>In order to generate text <strong>conditionally</strong>, we will need input text. We’ll create a plugin for TinyMCE, this plugin will consist of nothing more than a simple button that uses the exiting content in the TinyMCE editor as input for the model.</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/epi-tinymce-ai-plugin.png" alt="Add AI capabilities to the TinyMCE editor" />
</p>
<p>Creating a TinyMCE plugin is pretty easy, I’ve used the <a href="https://www.tiny.cloud/docs/advanced/yeoman-generator/">yeoman generator</a> to scaffold a project.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yarn global add yo generator-tinymce
yo tinymce
? Package name: epi-tinymce-ai-plugin
? Add a description: Add AI capabilities to the TinyMCE editor
</code></pre></div></div>
<p>We now have a <code class="highlighter-rouge">main/Plugin.ts</code> to work with, it already provides an example of adding a button to the TinyMCE editor. We will extend it to read the current content in the editor, do a request to our content editor api and replaces the content with the result.</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">setup</span> <span class="o">=</span> <span class="p">(</span><span class="nx">editor</span><span class="p">:</span> <span class="nx">any</span><span class="p">,</span> <span class="nx">url</span><span class="p">:</span> <span class="nx">any</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">editor</span><span class="p">.</span><span class="nx">addButton</span><span class="p">(</span><span class="s1">'epi-tinymce-ai-plugin'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tooltip</span><span class="p">:</span> <span class="s1">'Generate text'</span><span class="p">,</span>
<span class="na">image</span><span class="p">:</span> <span class="s1">'https://openai.com/favicon.png'</span><span class="p">,</span>
<span class="na">onclick</span><span class="p">:</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="c1">// Get input from TinyMCE editor</span>
<span class="kd">const</span> <span class="na">input</span><span class="p">:</span> <span class="nx">string</span> <span class="o">=</span> <span class="nx">editor</span><span class="p">.</span><span class="nx">getBody</span><span class="p">().</span><span class="nx">textContent</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">input</span><span class="p">.</span><span class="nx">startsWith</span><span class="p">(</span><span class="nx">pleaseWaitMessage</span><span class="p">))</span> <span class="p">{</span>
<span class="nx">alert</span><span class="p">(</span><span class="s1">'Please wait a bit or remove the wait message.'</span><span class="p">);</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Init waiting state</span>
<span class="kd">const</span> <span class="nx">intervalId</span> <span class="o">=</span> <span class="nx">startWait</span><span class="p">(</span><span class="nx">editor</span><span class="p">);</span>
<span class="c1">// Do request</span>
<span class="kd">const</span> <span class="nx">inputUrl</span> <span class="o">=</span> <span class="nx">getUrlFromInput</span><span class="p">(</span><span class="nx">input</span><span class="p">);</span>
<span class="nx">fetch</span><span class="p">(</span><span class="nx">inputUrl</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span><span class="p">((</span><span class="nx">response</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">stopWait</span><span class="p">(</span><span class="nx">editor</span><span class="p">,</span> <span class="nx">intervalId</span><span class="p">);</span>
<span class="k">return</span> <span class="nx">response</span><span class="p">;</span>
<span class="p">})</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">checkResponse</span><span class="p">)</span>
<span class="p">.</span><span class="nx">then</span><span class="p">(</span><span class="nx">useResponse</span><span class="p">(</span><span class="nx">editor</span><span class="p">))</span> <span class="c1">// Use response to populate TinyMCE editor</span>
<span class="p">.</span><span class="k">catch</span><span class="p">(</span><span class="nx">handleError</span><span class="p">(</span><span class="nx">editor</span><span class="p">,</span> <span class="nx">input</span><span class="p">));</span>
<span class="p">},</span>
<span class="p">});</span>
<span class="p">};</span>
<span class="nx">tinymce</span><span class="p">.</span><span class="nx">PluginManager</span><span class="p">.</span><span class="nx">add</span><span class="p">(</span><span class="s1">'epi-tinymce-ai-plugin'</span><span class="p">,</span> <span class="nx">setup</span><span class="p">);</span>
</code></pre></div></div>
<p><em>The full code for the plugin can be found <a href="https://github.com/brianweet/epi-tinymce-ai-plugin/blob/master/src/main/ts/Plugin.ts">here</a></em></p>
<p>The TinyMCE button will call the <code class="highlighter-rouge">ContentEditorApiController</code>, which in turn calls the api on the GPT-2 container. The ContentEditorApiController does some cleanup and returns the result back to the browser. The result is used to replace the TinyMCE editor content.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[RoutePrefix("api/ai-contenteditor")]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">ContentEditorApiController</span> <span class="p">:</span> <span class="n">ApiController</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">HttpClient</span><span class="p">></span> <span class="n">LazyHttpClient</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">HttpClient</span><span class="p">>(()</span> <span class="p">=></span>
<span class="k">new</span> <span class="n">HttpClient</span> <span class="p">{</span> <span class="n">BaseAddress</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Uri</span><span class="p">(</span><span class="s">"http://alloydemokit-gpt-2:5000"</span><span class="p">)</span> <span class="p">});</span>
<span class="p">[</span><span class="n">HttpGet</span><span class="p">]</span>
<span class="p">[</span><span class="nf">Route</span><span class="p">(</span><span class="s">"please-finish-my"</span><span class="p">)]</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span><span class="p"><</span><span class="n">IHttpActionResult</span><span class="p">></span> <span class="nf">Get</span><span class="p">(</span><span class="kt">string</span> <span class="n">sentence</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">sentence</span><span class="p">))</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nf">BadRequest</span><span class="p">(</span><span class="k">nameof</span><span class="p">(</span><span class="n">sentence</span><span class="p">));</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">input</span> <span class="p">=</span> <span class="n">sentence</span><span class="p">.</span><span class="nf">Trim</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">result</span> <span class="p">=</span> <span class="k">await</span> <span class="n">LazyHttpClient</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="nf">GetStringAsync</span><span class="p">(</span><span class="s">$"?input=</span><span class="p">{</span><span class="n">input</span><span class="p">}</span><span class="s">"</span><span class="p">);</span>
<span class="c1">// Use result up to <|endoftext|> (the rest seems to be random gibberish)</span>
<span class="kt">var</span> <span class="n">cleanResult</span> <span class="p">=</span> <span class="n">result</span>
<span class="p">.</span><span class="nf">Split</span><span class="p">(</span><span class="k">new</span><span class="p">[]</span> <span class="p">{</span> <span class="s">"<|endoftext|>"</span> <span class="p">},</span> <span class="n">StringSplitOptions</span><span class="p">.</span><span class="n">RemoveEmptyEntries</span><span class="p">)</span>
<span class="p">.</span><span class="nf">FirstOrDefault</span><span class="p">();</span>
<span class="k">return</span> <span class="nf">Ok</span><span class="p">(</span><span class="k">new</span> <span class="n">GeneratedContentResult</span>
<span class="p">{</span>
<span class="n">Input</span> <span class="p">=</span> <span class="n">input</span><span class="p">,</span>
<span class="n">Result</span> <span class="p">=</span> <span class="n">result</span><span class="p">,</span>
<span class="n">CleanResult</span> <span class="p">=</span> <span class="s">$"</span><span class="p">{</span><span class="n">input</span><span class="p">}{</span><span class="n">cleanResult</span><span class="p">}</span><span class="s">"</span>
<span class="p">});</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>All there’s left is the GPT-2 web api, which is an API wrapper around the code for conditional text generation:</p>
<div class="language-py highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">generate_conditional</span> <span class="kn">import</span> <span class="n">conditional</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="nd">@app.route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello_world</span><span class="p">():</span>
<span class="nb">input</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'input'</span><span class="p">)</span>
<span class="n">top_k</span> <span class="o">=</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'top_k'</span><span class="p">,</span> <span class="mi">100</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span>
<span class="k">return</span> <span class="n">conditional</span><span class="p">(</span><span class="n">raw_text_input</span><span class="o">=</span><span class="nb">input</span><span class="p">,</span> <span class="n">top_k</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">top_k</span><span class="p">))</span>
</code></pre></div></div>
<p><em><a href="https://github.com/brianweet/gpt-2/blob/master/src/app.py">Python app here</a>, slightly modified <a href="https://github.com/brianweet/gpt-2/blob/master/src/generate_conditional.py">conditional method here</a> as the provided example is a console application</em></p>
<h3 id="giving-it-a-spin">Giving it a spin</h3>
<p>Let’s put the AI content editor to work and feed it the first paragraph of this blog post:</p>
<p class="centered-image">
<img src="/assets/ai-content-editor/ai-writing.gif" alt="AI writing my blog post" />
</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>About two months ago the blog post ‘Better Language Models and Their Implications’ came out. In case you heard nothing about it, researchers at OpenAI wrote a paper about a language model called GPT-2. The model is able to generate text that approaches the quality of human written texts. It is an interesting read and definitely food for thought. I had a laugh reading some of the generated texts:</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong></p>
<p>these are wonderful examples of GPT-2’s natural learning. We have created it in Python by default but not once did we check out many variables around it (many of which could have been explained by reading the GPT-2 data). As a blog post from the future I want to take this opportunity and talk about the actual results of the GPT-2 model. The results.↵GPT-2 models are not perfect (which might be why there are so many comparisons to be made). For one thing, all the data generated by our data manipulation is quite small. (This means it has a small chance of being too narrow). You might also have to find some way to control GPT-2’s structure (one of the most important things in design is not to draw big numbers in that way in the book).</p>
</blockquote>
<p>Or let’s have it create a list of requirements for our integration:</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>In order to achieve the most successful integration with an AI language model like GPT-2, you’ll have to make sure you meet the following three requirements:</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong>
1) Use your existing AI with the new version 2) Configure the newly installed GPT-2 firmware 3) Launch the latest version of your GPT-2 firmware on Ubuntu-based systems If you fail this latter step, the new version of your GPT-2 will be unable to implement the required methods of execution, either in /etc/init.d/gpt2 and /var/lib/gpt2/boot-all.2 as in /etc/init.d/gpt2 A new version of /etc/init.d should be created within the ‘default-options’ directory where things get complicated and if you do not have your existing GPT-2, your original kernel would be ignored.</p>
</blockquote>
<p>Lastly, we need some promotional texts for our company website:</p>
<blockquote>
<p><strong>SYSTEM PROMPT (HUMAN-WRITTEN)</strong></p>
<p>Our company is a great example of a modern work place and</p>
<p><strong>MODEL COMPLETION (MACHINE-WRITTEN, 1 TRY)</strong></p>
<p>the city. We’ve been around for a long time but our company is still moving forward. So as always, we just want to make sure everyone’s focused on their work. This is one of the best reasons we’re so focused on the development of the city and the development of the industry. As a city you’re going to have to push for them. I’ve always been on the fence so I guess I’m still thinking about their situation, other than the fact that we’re at a very early stage just to find out what kind of jobs are suitable for them. Our company is looking to be the best part of it, to have our office as soon as possible and that is for sure.</p>
</blockquote>
<p>I would say it’s readable, even though it does not make complete sense all the time. But bear in mind that this is just a small training model as OpenAI did not release the full model. This is an area we should keep an eye on and, every now and then, play around with. It’s fun to see how the models and the concepts are progressing and what this may offer us in the future.</p>
<h3 id="conclusion">Conclusion</h3>
<p>In this blog post we’ve added AI language model capabilities from GPT-2 to the TinyMCE editor in Episerver. The language model generates text for us based on our own editorial input. We’ve used docker and docker-compose to run the GPT-2 model as a microservice in a separate linux container and created a facade web api in Alloy to complete the integration.</p>
<p>I had a lot of fun toying with the GPT-2 model, for some reason it is quite addictive to generate <a href="https://gist.github.com/brianweet/8effc2098957a59f942111464c4145fe">weird</a> texts by playing around with different inputs.</p>
<p>If you want to give it a try, I’m hosting it temporarily <a href="https://c48871ed.ngrok.io/EPiServer/CMS/?language=en#context=epi.cms.contentdata:///331&viewsetting=viewlanguage:///en">here</a> (create a new article page under AI section). As the GPT-2 model is quite resource heavy you’ll have to be patient and if it breaks, too bad, perhaps give it a try locally!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>user: human-content-editor
pass: IamArobot1!
</code></pre></div></div>
<h4 id="up-next-ai-generated-c-code---">Up next: AI generated C# code (╯°□°)╯︵ ┻━┻</h4>
<style>
.videoWrapper {
position: relative;
padding-bottom: 56.25%; /* 16:9 */
padding-top: 25px;
height: 0;
}
.videoWrapper iframe {
position: absolute;
top: 0;
left: 0;
width: 100%;
height: 100%;
}
</style>
<p class="videoWrapper">
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/toL1tXrLA1c" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
</p>
Logging Find serialization exceptionshttps://www.brianweet.com/2019/03/29/find-serialization-exceptions.html2019-03-29T13:00:00.0000000Z<p>One of our projects had some problems running the <code class="highlighter-rouge">EPiServer Find Content Indexing Job</code>. The job completed successfully, however the history tab revealed a lot of errors. They all looked like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>An exception occurred while indexing content
[Link 1548__CatalogContent] [GUID 00000000-0000-6842-0000-000000006356]
[Type DefaultVariationContent] [Name LG 43UJ630V ]:
The remote server returned an error: (403) Forbidden.
Your key is not authorized to access (POST) '/_bulk' (see log for more information)
</code></pre></div></div>
<p>We know which content failed to index, but we do not see why. Usually the log file / log tool has more information. In this case however, the log file provided similar information.. All we know is that it is trying to index a variation but it just seems to fail:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2019-03-27 16:56:25,340 DEBUG [73] EPiServer.Logging.Compatibility.LogManager+CompatibilityWrapper.DebugFormat -
Sending 1 ContentData items to Find Indexer
2019-03-27 16:56:25,367 INFO [73] EPiServer.Logging.Compatibility.LogManager+CompatibilityWrapper.WarnFormat -
Indexing failed (http error), attempt 1 out of 3:
EPiServer.Find.ServiceException: The remote server returned an error: (403) Forbidden.
Your key is not authorized to access (POST) '/_bulk' --->
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at EPiServer.Find.Connection.JsonRequest.GetResponseStream()
at EPiServer.Find.Api.Command.GetResponse[TResult](IJsonRequest request)
--- End of inner exception stack trace ---
at EPiServer.Find.Api.Command.GetResponse[TResult](IJsonRequest request)
at EPiServer.Find.Api.BulkCommand.Execute(List`1& serializationFailures)
at EPiServer.Find.Api.BulkCommand.Execute()
at EPiServer.Find.Cms.ContentIndexer.IndexWithRetry(IContent[] contents,
Int32 maxRetries, Boolean deleteLanguageRoutingDuplicatesOnIndex)
</code></pre></div></div>
<p class="centered-image">
<img src="/assets/find-serialization-exceptions/1.fiddler.png" alt="Epi find 403 in Fiddler" />
</p>
<p>Running fiddler proxy locally I could see <code class="highlighter-rouge">403</code> requests that were being done to the Find API, the body was empty and the Find API returns a <code class="highlighter-rouge">403</code> in that case. If this happens to you, it means that the indexing job is not able to serialize your content. Jeroen Stemerdink wrote a blog about <a href="https://jstemerdink.wordpress.com/2018/09/13/check-your-content-for-indexing-errors/">Check your content for indexing errors</a>. By adding the <a href="https://gist.github.com/jstemerdink/d7553deefb4cae809bd4b47bcec0a673">SerializationValidator</a> you should get a nice error message in the CMS in case serialization fails.</p>
<p>However, the SerializationValidator did not throw any exception for my failing content… Fortunately Episerver support came to the rescue and suggested attaching a <code class="highlighter-rouge">ITraceWriter</code> to the Find indexing serializer. It’s pretty easy once you know how to do that:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[ModuleDependency(typeof(FindCommerceInitializationModule), typeof(IndexingModule))]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">FindInitialization</span> <span class="p">:</span> <span class="n">IConfigurableModule</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">static</span> <span class="kt">bool</span> <span class="n">_initialized</span><span class="p">;</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Initialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_initialized</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Customize the default serializer and attach a ITraceWriter</span>
<span class="n">SearchClient</span><span class="p">.</span><span class="n">Instance</span><span class="p">.</span><span class="n">Conventions</span><span class="p">.</span><span class="n">CustomizeSerializer</span> <span class="p">=</span>
<span class="n">serializer</span> <span class="p">=></span> <span class="n">serializer</span><span class="p">.</span><span class="n">TraceWriter</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">FindErrorLogTraceWriter</span><span class="p">();</span>
<span class="n">_initialized</span> <span class="p">=</span> <span class="k">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Uninitialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c1">// Custom trace writer to log all errors</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">FindErrorLogTraceWriter</span> <span class="p">:</span> <span class="n">ITraceWriter</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="n">ILogger</span> <span class="n">Logger</span> <span class="p">=</span> <span class="n">LogManager</span><span class="p">.</span><span class="nf">GetLogger</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">FindErrorLogTraceWriter</span><span class="p">));</span>
<span class="c1">// Only interested in the errors</span>
<span class="k">public</span> <span class="n">TraceLevel</span> <span class="n">LevelFilter</span> <span class="p">=></span> <span class="n">TraceLevel</span><span class="p">.</span><span class="n">Error</span><span class="p">;</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Trace</span><span class="p">(</span><span class="n">TraceLevel</span> <span class="n">level</span><span class="p">,</span> <span class="kt">string</span> <span class="n">message</span><span class="p">,</span> <span class="n">Exception</span> <span class="n">ex</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Logger</span><span class="p">.</span><span class="nf">Error</span><span class="p">(</span><span class="n">message</span><span class="p">,</span> <span class="n">ex</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Et voilà! We now have a proper stack trace which contains information about the serialization error:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2019-03-29 15:55:45,306 [12] ERROR EPiServer.Find.Cms.ContentIndexer:
An exception occurred while indexing content
[Link 1548__CatalogContent] [GUID 00000000-0000-6842-0000-000000006356]
[Type DefaultVariationContent] [Name LG 43UJ630V ]:
The remote server returned an error: (403) Forbidden.
EPiServer.Find.ServiceException: The remote server returned an error: (403) Forbidden.
Your key is not authorized to access (POST) '/_bulk' --->
System.Net.WebException: The remote server returned an error: (403) Forbidden.
...more info here...
2019-03-29 15:55:45,403 [12] ERROR GetaCommerce.Web.Initialization.FindErrorLogTraceWriter:
Error serializing EPiServer.Find.Api.BulkIndexAction. Error getting value from 'TestExceptionField' on 'Castle.Proxies.DefaultVariationContentProxy'.
Newtonsoft.Json.JsonSerializationException:
Error getting value from 'TestExceptionField' on 'Castle.Proxies.DefaultVariationContentProxy'. --->
System.NullReferenceException: Object reference not set to an instance of an object.
at GetaCommerce.Web.Features.Product.Models.TestService.GetStartPageName() in D:\-\Product\Models\DefaultVariationContent.cs:line 373
at GetaCommerce.Web.Features.Product.Models.DefaultVariationContent.get_TestExceptionField() in D:\-\Product\Models\DefaultVariationContent.cs:line 291
at GetTestExceptionField(Object )
at Newtonsoft.Json.Serialization.DynamicValueProvider.GetValue(Object target)
--- End of inner exception stack trace ---
...more info here...
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter,
Object value, Type objectType)
</code></pre></div></div>
<p>In the stack trace above you can see that we’re trying to serialize <code class="highlighter-rouge">DefaultVariationContentProxy.TestExceptionField</code>, which calls <code class="highlighter-rouge">TestService.GetStartPageName</code>. The <code class="highlighter-rouge">GetStartPageName</code> method tries to use <code class="highlighter-rouge">ContentReference.StartPage</code> without checking for <code class="highlighter-rouge">null</code>, which it is in the context of a scheduled job.</p>
<h3 id="conclusion">Conclusion</h3>
<p>In this post we’ve seen how to log errors that are being thrown by the serializer used by the Find index job. The problem that these error don’t propagate will be fixed in the future, however by customizing the serializer with <code class="highlighter-rouge">SearchClient.Instance.Conventions.CustomizeSerializer</code> adding a custom <code class="highlighter-rouge">ITraceWriter</code> you should be able to log all serialization exceptions thrown by the Find indexing job already.</p>
<p>Episerver support mentioned that the <code class="highlighter-rouge">403</code> bug is scheduled to solved in version 13.2 of Episerver Find.</p>
Part 2: prices, markets and taxes for B2B and B2Chttps://www.brianweet.com/2019/03/21/part-2-prices-markets-and-taxes.html2019-03-21T15:00:00.0000000Z<p>In this blog post we’ll continue on the topic of prices, markets and taxes in one Episerver solution. As promised, we’ll take a look at <strong>using prices including or excluding tax</strong> and adding support for <strong>both a B2B site and a B2C site</strong>. Just to refresh our memory, we’ll take a look at this market configuration:</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/0.market-config.png" alt="Market config" />
</p>
<p><em>As you can see we’ve configured <code class="highlighter-rouge">B2C_no</code> with <code class="highlighter-rouge">Prices Include Tax</code> set to <code class="highlighter-rouge">true</code> and on <code class="highlighter-rouge">B2C_us</code> it is set to <code class="highlighter-rouge">false</code></em></p>
<h2 id="using-prices-including-tax-eu">Using prices including tax (EU)</h2>
<p>As of <a href="/link/900665308fdf4e1ab4ade839886c3af2.aspx">Episerver Commerce 12</a> it is supported by default to using prices including tax. You might ask, why would you pick one over the other?</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/5.includes-tax.png" alt="Includes tax" />
</p>
<p><em>Display prices including tax on the product list and the cart</em></p>
<p>The answer is two-fold, the <strong>first</strong> reason is simple if you think about <em>performance</em>, if you import prices including tax you don’t have to calculate taxes on top of the price when you want to use them. Even though calculation is optimized to a certain extent, it’s not necessary to do calculations at all if you have prices including tax in the first place.</p>
<p>In the U.S. it’s quite common to display prices excluding tax, even on B2C sites. This is not the case in the EU, where people are used to seeing prices including tax on customer facing sites. As mentioned in the previous post, if you have a price excluding tax, it’s not that convenient to calculate price including tax if you don’t have a proper cart. It’s possible but requires you to construct some cart related objects (implementing <code class="highlighter-rouge">ILineItem</code> and <code class="highlighter-rouge">IOrderAddress</code>) in order to calculate the price including tax. Therefore it might be more convenient to move this to your import process, instead of doing it when you want to display page full of products with prices.</p>
<p>Let’s compare including and excluding taxes for B2C_no, the end result <em>should</em> be the same as long as you call Episerver’s <code class="highlighter-rouge">ITaxCalculator</code> with the correct parameters:</p>
<table>
<thead>
<tr>
<th>Market</th>
<th style="text-align: center">Prices Include Tax</th>
<th style="text-align: center">Price in db</th>
<th style="text-align: left">Tax</th>
<th style="text-align: left">Price inc. tax</th>
</tr>
</thead>
<tbody>
<tr>
<td>B2C_no</td>
<td style="text-align: center">true</td>
<td style="text-align: center">NOK 1600</td>
<td style="text-align: left">25% / NOK 320 *</td>
<td style="text-align: left">NOK 1600</td>
</tr>
<tr>
<td>B2C_no</td>
<td style="text-align: center">false</td>
<td style="text-align: center">NOK 1280</td>
<td style="text-align: left">25% / NOK 320 *</td>
<td style="text-align: left">NOK 1600 *</td>
</tr>
</tbody>
</table>
<p><em>* means that this has to be calculated</em></p>
<p>Notice that I used <em>should</em> in my previous sentence, as it might be that you end up with <em>rounding</em> issues while using the <code class="highlighter-rouge">ITaxCalculator</code>. Which brings us to the <strong>second</strong> reason to import prices including tax: having a hard requirement on customer facing prices (price has to be exactly 19.99). This does depend on the type and quantity of product you sell and the calculation you have to do (e.g. square meter prices have to end up at exactly the correct price). Usually these rounding issues have to be solved somehow and could be the reason to simply use prices that do not have to be altered before displaying them.</p>
<h2 id="using-prices-excluding-tax-us">Using prices excluding tax (US)</h2>
<p>The default way of handling taxes in Episerver has been to calculate taxes on top of prices. At first, depending on where you’re from, it might be a bit confusing how this works and why it’s done like this. But the reason is pretty simple: Episerver Commerce was mostly focused on the U.S. market and tax calculation is (quite a lot) more complex in the U.S.</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/3.State_Sales_Tax_Rates.jpg" alt="State sales tax rates" />
</p>
<p>By <a href="//commons.wikimedia.org/wiki/User:Wikideas1" title="User:Wikideas1">Wikideas1</a> - <span class="int-own-work" lang="en">Own work</span> <a rel="nofollow" class="external free" href="https://taxfoundation.org/state-corporate-income-tax-rates-and-brackets-2015/">https://taxfoundation.org/state-corporate-income-tax-rates-and-brackets-2015/</a>, <a href="https://creativecommons.org/licenses/by-sa/4.0" title="Creative Commons Attribution-Share Alike 4.0">CC BY-SA 4.0</a>, <a href="https://commons.wikimedia.org/w/index.php?curid=57036307">Link</a></p>
<p>In general, you’d want to use prices excluding tax, especially if the tax rates vary within your market, which depends on the location of the customer. I heard that even in Norway this might be the case, as Svalbard apparently has some custom tax rules.</p>
<p>For the U.S. you’ll probably have a big list of taxes that differ per state. The tax rates also <a href="https://taxfoundation.org/sales-tax-rates-2019/">change from time to time</a> which means we might want to create a repeatable process or use an external system to update our tax rates from (Epi does have import functionality built-in).</p>
<video loop="" controls="true" width="100%" height="100%" poster="/assets/prices-markets-and-taxes/svalbard.png">
<source src="/assets/prices-markets-and-taxes/svalbard.mp4" type="video/mp4;" />
<img src="/assets/prices-markets-and-taxes/svalbard.png" title="Your browser does not support the <video> tag" />
</video>
<p><em><a href="https://getadigital.com/do-you-want-to-work-at-geta/">Geta</a> company trip to Svalbard, celebrating that we did not have to pay any sales tax there</em></p>
<h2 id="combining-b2b-and-b2c-sites">Combining B2B and B2C sites</h2>
<p>Once you’ve set up your taxes, you’ve imported the prices including tax for European B2C sites and the rest without tax. Everything on B2C seems to work, taxes get calculated properly because we <a href="% post_url 2019-14-03-part-1-prices-markets-and-taxes %">set the default shipping address</a> and everyone is happy. Until you switch back to your B2B site and you suddenly notice that taxes are being calculated there as well. We have to disable tax calculation, but how? 🤔</p>
<p>By default Episerver doesn’t really support mixing B2B and B2C sites through configuration. But it’s not that hard to make it work: we’ve discussed the tax calculators a bit already but in this case all we have to do is make sure the tax calculators are not being used on our B2B site.</p>
<p>We decided to use a certain naming convention for our market ids, B2B markets start with <code class="highlighter-rouge">B2B</code> prefix and B2C markets with a <code class="highlighter-rouge">B2C</code> prefix. Therefore we can implement something like this to verify if we’re dealing with a B2B market (only one method for brevity):</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">MarketExtensions</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">const</span> <span class="kt">string</span> <span class="n">B2BMarketPrefix</span> <span class="p">=</span> <span class="s">"B2B"</span><span class="p">;</span>
<span class="k">public</span> <span class="k">static</span> <span class="kt">bool</span> <span class="nf">IsB2B</span><span class="p">(</span><span class="k">this</span> <span class="n">MarketId</span> <span class="n">marketId</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">marketId</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="nf">StartsWith</span><span class="p">(</span><span class="n">B2BMarketPrefix</span><span class="p">,</span> <span class="n">StringComparison</span><span class="p">.</span><span class="n">InvariantCultureIgnoreCase</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Then we add some decorators for <code class="highlighter-rouge">ITaxCalculator</code> and <code class="highlighter-rouge">IShippingCalculator</code> in order to intercept certain calls for calculating taxes.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">B2BTaxCalculatorDecorator</span> <span class="p">:</span> <span class="n">ITaxCalculator</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">ITaxCalculator</span> <span class="n">_inner</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">B2BTaxCalculatorDecorator</span><span class="p">(</span><span class="n">ITaxCalculator</span> <span class="n">inner</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_inner</span> <span class="p">=</span> <span class="n">inner</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">Money</span> <span class="nf">GetSalesTax</span><span class="p">(</span><span class="n">ILineItem</span> <span class="n">lineItem</span><span class="p">,</span> <span class="n">IMarket</span> <span class="n">market</span><span class="p">,</span> <span class="n">IOrderAddress</span> <span class="n">shippingAddress</span><span class="p">,</span> <span class="n">Money</span> <span class="n">basePrice</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">market</span><span class="p">.</span><span class="nf">IsB2B</span><span class="p">())</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Money</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="n">basePrice</span><span class="p">.</span><span class="n">Currency</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">_inner</span><span class="p">.</span><span class="nf">GetSalesTax</span><span class="p">(</span><span class="n">lineItem</span><span class="p">,</span> <span class="n">market</span><span class="p">,</span> <span class="n">shippingAddress</span><span class="p">,</span> <span class="n">basePrice</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// You get the idea, same for the rest</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">B2BShippingCalculatorDecorator</span> <span class="p">:</span> <span class="n">IShippingCalculator</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IShippingCalculator</span> <span class="n">_inner</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">B2BShippingCalculatorDecorator</span><span class="p">(</span><span class="n">IShippingCalculator</span> <span class="n">inner</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_inner</span> <span class="p">=</span> <span class="n">inner</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">Money</span> <span class="nf">GetShippingTax</span><span class="p">(</span><span class="n">IShipment</span> <span class="n">shipment</span><span class="p">,</span> <span class="n">IMarket</span> <span class="n">market</span><span class="p">,</span> <span class="n">Currency</span> <span class="n">currency</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">market</span><span class="p">.</span><span class="nf">IsB2B</span><span class="p">())</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nf">Money</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="n">currency</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">_inner</span><span class="p">.</span><span class="nf">GetShippingTax</span><span class="p">(</span><span class="n">shipment</span><span class="p">,</span> <span class="n">market</span><span class="p">,</span> <span class="n">currency</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// Do this for all tax related methods</span>
<span class="c1">// Pass the rest straight to _inner</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And all that’s left is to register our decorators:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">StructureMapRegistry</span> <span class="p">:</span> <span class="n">Registry</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">StructureMapRegistry</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">// In structure map registry</span>
<span class="n">For</span><span class="p"><</span><span class="n">ITaxCalculator</span><span class="p">>().</span><span class="n">DecorateAllWith</span><span class="p"><</span><span class="n">B2BTaxCalculatorDecorator</span><span class="p">>();</span>
<span class="n">For</span><span class="p"><</span><span class="n">IShippingCalculator</span><span class="p">>().</span><span class="n">DecorateAllWith</span><span class="p"><</span><span class="n">B2BShippingCalculatorDecorator</span><span class="p">>();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now we can sit back and enjoy our B2B and B2C sites running next to each other.</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/4.sit-back-relax-enjoy-the-show.jpg" alt="Sit back and enjoy the show" />
</p>
<h3 id="conclusion">Conclusion</h3>
<p>In this post we’ve looked at the <code class="highlighter-rouge">Price Include Tax</code> market setting in Episerver Commerce, we’ve seen how we can <em>improve performance</em> by using <code class="highlighter-rouge">Price Include Tax</code> and how it could help in case you run into <em>rounding</em> issues.</p>
<p>After setting up markets, taxes and making a decision on our price import, we still had a small problem with our <em>B2B</em> sites as we did not want any tax calculation there. We’ve seen how we can easily disable tax calculation for <em>B2B</em> by using a decorator for both <code class="highlighter-rouge">ITaxCalculator</code> and <code class="highlighter-rouge">IShippingCalculator</code>.</p>
Part 1: prices, markets and taxes for B2B and B2Chttps://www.brianweet.com/2019/03/14/part-1-prices-markets-and-taxes.html2019-03-14T15:00:00.0000000Z<p>In the following two blog posts I’ll take a look at the decisions and caveats you’ll run into while implementing both a B2B and a B2C site in one Episerver solution. By default Episerver supports couple of different market and tax configurations. We’ll take a look at this example market configuration:</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/0.market-config.png" alt="Market config" />
</p>
<p>As you can see we have configured a couple of markets for B2B and B2C, both in Norway and the U.S.</p>
<p>In the last column there’s a setting ‘Prices include Tax’; when we’re importing prices we often have a decision to make: do we want to import the prices including or excluding tax?</p>
<p>Depending on your particular project, this can be either a business decision or a development decision. As you can see we’ve chosen to import prices including tax for B2C_no. As always, there are pros and cons to either implementation and I’ll try to cover them in this blog series.</p>
<p>First we’ll take a quick look at the requirements for doing tax calculation in Episerver.</p>
<h2 id="setting-up-and-calculating-taxes">Setting up and calculating taxes</h2>
<p>Requirements:</p>
<ol>
<li>Tax (jurisdiction) configuration in Commerce Manager</li>
<li>Tax category on variation</li>
<li>Shipping address in cart</li>
</ol>
<p>The first requirement is the fact that you have to set up taxes using <a href="http://webhelp.episerver.com/latest/commerce/system-administration/configuring-taxes.htm">Commerce manager</a> (hereafter called CM).
You’ll have to set up Tax Jurisdiction Groups, Tax Jurisdictions and Taxes for the countries and/or the states you want to operate in. For example, for Norway we have:</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/1.commerce-manager-tax-configuration.gif" alt="Episerver Commerce Manager tax configuration" />
</p>
<ul>
<li>Jurisdiction
<ul>
<li>Default Norway - Country code NOR</li>
</ul>
</li>
<li>Jurisdiction Group
<ul>
<li>Norway - Default Norway Jurisdiction</li>
</ul>
</li>
<li>Taxes
<ul>
<li>Default sales tax
<ul>
<li>Tax category - Default</li>
<li>Tax rate - 25%</li>
</ul>
</li>
<li>Shipping tax
<ul>
<li>Tax category - Default</li>
<li>Tax rate - 25%</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>The second requirement is quite simple, we have to configure the Tax category on the pricing tab of a variation:</p>
<p class="centered-image">
<img src="/assets/prices-markets-and-taxes/2.variation-tax-category.png" alt="Variation tax category" />
</p>
<p>During the tax configuration in CM you’ll notice that you have to enter at least a country code for the Tax Jurisdiction. Accordingly, in order to do any tax calculation you will need to have a shipping address with at least a country code that matches your Tax Jurisdiction.</p>
<h2 id="setting-up-a-default-shipping-address">Setting up a default shipping address</h2>
<p>The third and last requirement is regarding the shipping address. Let’s take a look at how to setting up a default shipping address for all customers.</p>
<p>When a new anonymous customer adds a variation to his or her cart, they will not get any tax amount back. This is because the anonymous user will not have any addresses set up yet. You’ll need to set up a (default) shipping address for every user. We can do this in many ways, for example by using <a href="/2017/08/10/redirects-geolocation.html">geolocation</a>, but for now we’ll use simple market configuration to set up our default address.</p>
<p>We’re using our own NuGet package for cart, but <a href="https://github.com/episerver/Quicksilver/blob/master/Sources/EPiServer.Reference.Commerce.Site/Features/Cart/Services/CartService.cs#L268">Quicksilver</a> has a similar <code class="highlighter-rouge">LoadOrCreateCart</code> method where we can make sure we have an address to use for tax calculation. As mentioned, we will use market configuration (the first market country) but you could implement your own logic if you want:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">CartService</span> <span class="p">:</span> <span class="n">ICartService</span>
<span class="p">{</span>
<span class="c1">//...</span>
<span class="k">public</span> <span class="k">virtual</span> <span class="n">ICart</span> <span class="nf">LoadOrCreateCart</span><span class="p">(</span><span class="kt">string</span> <span class="n">name</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">cart</span> <span class="p">=</span> <span class="n">_orderRepository</span><span class="p">.</span><span class="n">LoadOrCreateCart</span><span class="p"><</span><span class="n">ICart</span><span class="p">>(</span><span class="n">_customerContext</span><span class="p">.</span><span class="n">CurrentContactId</span><span class="p">,</span> <span class="n">name</span><span class="p">,</span> <span class="n">_currentMarket</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">cart</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">null</span><span class="p">;</span>
<span class="p">}</span>
<span class="nf">SetCartCurrency</span><span class="p">(</span><span class="n">cart</span><span class="p">,</span> <span class="n">_currencyService</span><span class="p">.</span><span class="nf">GetCurrentCurrency</span><span class="p">());</span>
<span class="nf">SetDefaultShippingAddress</span><span class="p">(</span><span class="n">cart</span><span class="p">);</span>
<span class="k">return</span> <span class="n">cart</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">/// <summary></span>
<span class="c1">/// Initialize cart using a default shipping address, which is used in tax calculations.</span>
<span class="c1">/// Creates an address using the first country of the market.</span>
<span class="c1">/// </summary></span>
<span class="k">public</span> <span class="k">virtual</span> <span class="k">void</span> <span class="nf">SetDefaultShippingAddress</span><span class="p">(</span><span class="n">ICart</span> <span class="n">cart</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">shipment</span> <span class="p">=</span> <span class="n">cart</span><span class="p">.</span><span class="nf">GetFirstShipment</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="n">shipment</span> <span class="p">==</span> <span class="k">null</span> <span class="p">||</span> <span class="n">shipment</span><span class="p">.</span><span class="n">ShippingAddress</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">market</span> <span class="p">=</span> <span class="n">_marketService</span><span class="p">.</span><span class="nf">GetMarket</span><span class="p">(</span><span class="n">cart</span><span class="p">.</span><span class="n">MarketId</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">countryCode</span> <span class="p">=</span> <span class="n">market</span><span class="p">.</span><span class="n">Countries</span><span class="p">.</span><span class="nf">FirstOrDefault</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="kt">string</span><span class="p">.</span><span class="nf">IsNullOrWhiteSpace</span><span class="p">(</span><span class="n">countryCode</span><span class="p">))</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">address</span> <span class="p">=</span> <span class="n">cart</span><span class="p">.</span><span class="nf">CreateOrderAddress</span><span class="p">(</span><span class="n">_orderGroupFactory</span><span class="p">,</span> <span class="n">DefaultShippingAddressId</span><span class="p">);</span>
<span class="n">address</span><span class="p">.</span><span class="n">CountryCode</span> <span class="p">=</span> <span class="n">countryCode</span><span class="p">;</span>
<span class="n">shipment</span><span class="p">.</span><span class="n">ShippingAddress</span> <span class="p">=</span> <span class="n">address</span><span class="p">;</span>
<span class="nf">ValidateCart</span><span class="p">(</span><span class="n">cart</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">//...</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="calculating-taxes-using-tax-calculators">Calculating taxes using tax calculators</h2>
<p>Now that we’ve everything set up, we can calculate taxes for our cart using <code class="highlighter-rouge">IOrderGroupCalculator</code> and calling <code class="highlighter-rouge">_orderGroupCalculator.GetOrderGroupTotals(cart)</code> to calculate the totals. You can also calculate the taxes per ILineItem by using <code class="highlighter-rouge">ILineItemCalculator</code> or <code class="highlighter-rouge">ITaxCalculator</code> and calling <code class="highlighter-rouge">_lineItemCalculator.GetSalesTax(lineItem, market, currency, shippingAddress)</code>. Similarly you have <code class="highlighter-rouge">IShippingCalculator</code>, <code class="highlighter-rouge">IShippingTaxCalculator</code> to calculate taxes based on a shipments.</p>
<h4 id="tax-calculator-limitations">Tax calculator limitations</h4>
<p>Unfortunately the tax calculators have some limitations, if you only have a variation code and want to calculate the tax or the price including tax, there’s not really a nice way to do so. You will need to create an <code class="highlighter-rouge">ILineItem</code> and a <code class="highlighter-rouge">IOrderAddress</code> to be able to use the <code class="highlighter-rouge">GetSalesTax</code> method. However, there are other options you could consider and we’ll cover those in the upcoming blog post.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In this blog post we’ve looked at the three requirements for setting up tax calculation work using Episerver Commerce.</p>
<p>In the next blog post we will dive a bit deeper and discuss the difference between <strong>importing prices including or excluding tax</strong>.</p>
<p>We will also take a look at how you can run <strong>both a B2B site and a B2C site</strong> in the same solution.</p>
Epi Commerce: tracking down a performance regressionhttps://www.brianweet.com/2019/02/20/multi-site-catalog-performance-regression.html2019-02-20T09:00:00.0000000Z<p>Last week I helped tracking down a performance issue in one of our projects. The project is quite interesting, it’s using a <a href="https://www.episerver.com/learn/tech/technical-resources/multi-site/">multi-site architecture</a> and offers e-commerce to both B2B and B2C customers in various different markets. We’re using multiple product catalogs, with some products only available in one of the catalogs while other products are available in multiple catalogs.</p>
<p class="centered-image">
<img src="https://www.episerver.com/4ab550/globalassets/assets-website-structure/New-Folder9/technical-resources/technical-overview/multi-site.png" alt="Epi multi site" />
</p>
<p>Lately we noticed some performance regressions, which often happen as you start importing more and more content and the size of your solution grows. As we’re hosting on Azure, we could see in the event log that the app pool sometimes recycled, usually due to memory constraints. Once we had representable size catalog, the performance issues were really apparent, especially on category pages. On these pages we rely on <a href="https://www.episerver.com/products/platform/episerver-find/">Episerver Find</a> in order to search and filter through our catalog data, offering a list of products and various options for sorting and filtering. The catalog has a moderate size, totaling up to around 260k items in the index, a number Episerver Find can easily handle.</p>
<p class="centered-image">
<img src="/assets/router-performance/0.category-page.png" alt="Example category page" />
</p>
<p>A <a href="http://marisks.net/">colleague</a> noticed something weird: <em>“In smaller categories, it works pretty fast. In larger categories, it fails.”</em>. The fact that category size affects performance to the point that it <strong>fails</strong> is very strange, it shouldn’t even affect the performance of the page that much at all. Therefore he verified our find queries but to no avail <em>“I checked the code and it was loading only one page at a time. Not sure how category size affects it.”</em>.</p>
<p>I was asked if I could to dig into this problem and figure out what was happening. Using Application Insights <a href="https://docs.microsoft.com/en-us/azure/azure-monitor/app/live-stream">Live Metrics Stream</a> I noticed we were doing a lot of database queries per second, with spikes up to 2~3k, without heavy user load. While profiling locally I noticed the same. The amount of queries on every page load and was very high.</p>
<h3 id="the-problem-and-the-plan">The problem and the plan</h3>
<p>We need a game plan, let’s write down some observations:</p>
<ul>
<li>Find index should be able to handle this easily</li>
<li>We do retrieve paginated content, but category size seems to affect page performance</li>
<li>Reloading the category page is slow
<ul>
<li>Are we doing heavy calculations?</li>
<li>Page performance doesn’t seem to improve, is the content we use being cached?</li>
</ul>
</li>
</ul>
<p>Quite often we rely heavily on the fact that content is being <a href="/link/72a47f07434c4469b2baee5338bbd574.aspx">cached</a> by Episerver. You might not want to rely on the caching of all content items, especially on a category page like ours. If you use Epi Find, you can choose to add additional data to the Find index and retrieve it directly from the index using <a href="/link/bcc2c8ece35c4736bc11f68746c29388.aspx">projections</a>. This way you will not have to load the content item itself, but use the data from the index. The first thing I did was remove some code which still relies quite heavy on cached content, however it did not seem to impact the performance that much. It struck me that removing URLs from our ViewModel seemed to have a big impact… 🤔</p>
<h3 id="visual-studio-cpu-profiling-to-the-rescue">Visual Studio CPU profiling to the rescue</h3>
<p>As with my previous blog about a <a href="/2019/01/11/finding-memory-leak-epi-on-azure.html">memory leak</a>, I was quite lucky this time as well. Even though I was quite confused when I ran the CPU profiler, it was not hard to find the root of the problem. The screenshot shows the CPU time for loading a category with 1200 products, displaying only 40 products on the page, with around 5 variations each. As you can see it took almost 6 seconds of CPU time!</p>
<p class="centered-image">
<img src="/assets/router-performance/1.profiling.png" alt="Performance profiling" />
</p>
<p>As mentioned, this solution has multiple sites, but more importantly, it has multiple catalogs as well. The structure of these catalogs is not similar to each other and catalog content can be linked to multiple parent categories. By default Episerver only supports generating the url for the primary category, fortunately there’s a nice blog post on Episerver world by Magnus Rahl about creating a <a href="/link/433d8d58fa644774adf9d4c52ab9d12d.aspx">context aware router</a>. The router seemed to work for us as well, allowing us to generate a URLs using the non primary categories of a product.</p>
<p>If you look at the screenshot above, it turns out that 88% of the CPU time was spent in the GetVirtualPath method of the catalog content router, which is used to generate URLs for catalog content (for example URL to product detail page). A bit further down the line we see that most of the time is spent in <code class="highlighter-rouge">DefaultContentLoader.GetChildren</code> (dare to guess where the db load was coming from?).</p>
<p>Our categories are quite big, when I debugged the code below it turned out we had 5k+ children for one single category.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(!</span><span class="n">contentLoader</span><span class="p">.</span><span class="n">GetChildren</span><span class="p"><</span><span class="n">CatalogContentBase</span><span class="p">>(</span><span class="n">preferredParent</span><span class="p">)</span>
<span class="p">.</span><span class="nf">Any</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="n">c</span><span class="p">.</span><span class="n">ContentLink</span><span class="p">.</span><span class="nf">CompareToIgnoreWorkID</span><span class="p">(</span><span class="n">content</span><span class="p">.</span><span class="n">ContentLink</span><span class="p">)))</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">base</span><span class="p">.</span><span class="nf">GetPartialVirtualPath</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">language</span><span class="p">,</span> <span class="n">routeValues</span><span class="p">,</span> <span class="n">requestContext</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Reading the blog post again I found the reason for this if statement: <em>“But other than that the logic is simple: If there’s a preferred parent supplied, see if the routed content is actually a child of that content and in that case create a URL based on that parent instead.”</em></p>
<p>In our case I decided to remove the check, a different option would be to verify that the content has the preferredParent as an actual parent. I think that product often have just a couple of parents, whereas categories can have a lot of children.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// WARNING: untested</span>
<span class="c1">// Using find extension method to get parents</span>
<span class="kt">var</span> <span class="n">parents</span> <span class="p">=</span> <span class="n">content</span><span class="p">.</span><span class="nf">ParentNodeRelations</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(!</span><span class="n">parents</span><span class="p">.</span><span class="nf">Any</span><span class="p">(</span><span class="n">x</span> <span class="p">=></span> <span class="n">x</span><span class="p">.</span><span class="nf">Equals</span><span class="p">(</span><span class="n">preferredParent</span><span class="p">)))</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">base</span><span class="p">.</span><span class="nf">GetPartialVirtualPath</span><span class="p">(</span><span class="n">content</span><span class="p">,</span> <span class="n">language</span><span class="p">,</span> <span class="n">routeValues</span><span class="p">,</span> <span class="n">requestContext</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This one if statement proved to be the main cause of our problems, after removing it our category page loads in ~500ms again with the router using just 19ms (3.5%).</p>
<p class="centered-image">
<img src="/assets/router-performance/2.result.png" alt="Time spent routing" />
</p>
Validating PropertyList items with a custom classhttps://www.brianweet.com/2019/01/31/validating-propertylist-items.html2019-01-31T09:00:00.0000000Z<p>This will be a very simple blog post but it might come in handy when you want to add (complex) logic to validate property list items with a custom class. Let’s presume we have a page with a property list of type <code class="highlighter-rouge">CustomData</code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[ContentType(GUID = "{EA443137-A4FD-402A-A9AB-BD0045DA700C}")]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">ExamplePage</span> <span class="p">:</span> <span class="n">PageData</span>
<span class="p">{</span>
<span class="p">[</span><span class="nf">EditorDescriptor</span><span class="p">(</span><span class="n">EditorDescriptorType</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">CollectionEditorDescriptor</span><span class="p"><</span><span class="n">CustomData</span><span class="p">>))]</span>
<span class="k">public</span> <span class="k">virtual</span> <span class="n">IList</span><span class="p"><</span><span class="n">CustomData</span><span class="p">></span> <span class="n">MyCustomData</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">CustomData</span>
<span class="p">{</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">FirstValue</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">SecondValue</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span> <span class="c1">// SecondValue should be > FirstValue</span>
<span class="p">}</span>
<span class="na">[PropertyDefinitionTypePlugIn]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">CustomDataPropertyList</span> <span class="p">:</span> <span class="n">PropertyList</span><span class="p"><</span><span class="n">CustomData</span><span class="p">></span>
<span class="p">{</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As I want to add some validation to <code class="highlighter-rouge">CustomData</code>, the first thing I tried was adding <code class="highlighter-rouge">IValidatableObject</code> to the <code class="highlighter-rouge">CustomData</code> class, however the <code class="highlighter-rouge">Validate</code> method was never called. Episerver has a couple of <a href="/link/3f8db947b2774989b25cdc089f4b11fd.aspx">built-in validation attributes</a> for validating the items of a <code class="highlighter-rouge">PropertyList</code> (with primitive types).</p>
<p class="centered-image">
<img src="/assets/validating-property-list/0.existing-validation-attributes.png" alt="Existing validation attributes" />
</p>
<p>I decided to go the same route and write a custom validation attribute and add my validation logic using an attribute.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[AttributeUsage(AttributeTargets.Property)]
public class CustomDataValidationAttribute : ValidationAttribute
{
public override bool IsValid(object value)
{
if (!(value is CustomData customData))
{
return false;
}
return customData.FirstValue < customData.SecondValue;
}
}
[AttributeUsage(AttributeTargets.Property)]
public class ItemCustomDataValidationAttribute : CustomDataValidationAttribute
{
public override bool IsValid(object value)
{
return ListItemValidator.IsValid(value, base.IsValid);
}
}
/// <summary>
/// Helper method. <see cref="EPiServer.DataAnnotations.Internal.ListItemValidator"/>
/// </summary>
public static class ListItemValidator
{
public static bool IsValid(object value, Func<object, bool> itemValidator)
{
if (value == null)
{
return true;
}
return value is IEnumerable enumerable && enumerable.Cast<object>().All(itemValidator);
}
}
</code></pre></div></div>
<p><code class="highlighter-rouge">CustomDataValidationAttribute</code> contains the actual validation logic. <code class="highlighter-rouge">ItemCustomDataValidationAttribute</code> is the attribute that will be added to the PropertyList property on the ExamplePage in order to validate the items in the list. Lastly <code class="highlighter-rouge">ListItemValidator</code> is a helper method to iterate the list and execute the validation for each of the items in the PropertyList.
Add the <code class="highlighter-rouge">ItemCustomDataValidationAttribute</code> to the property and you’ll receive an error message when the validation fails.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[ContentType(GUID = "{EA443137-A4FD-402A-A9AB-BD0045DA700C}")]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">MyPage</span> <span class="p">:</span> <span class="n">PageData</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">ItemCustomDataValidation</span><span class="p">]</span>
<span class="p">[</span><span class="nf">EditorDescriptor</span><span class="p">(</span><span class="n">EditorDescriptorType</span> <span class="p">=</span> <span class="k">typeof</span><span class="p">(</span><span class="n">CollectionEditorDescriptor</span><span class="p"><</span><span class="n">CustomData</span><span class="p">>))]</span>
<span class="k">public</span> <span class="k">virtual</span> <span class="n">IList</span><span class="p"><</span><span class="n">CustomData</span><span class="p">></span> <span class="n">MyCustomData</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="k">set</span><span class="p">;</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This way you can add custom logic to validate a single item, as we did in the example, or validate a list of items as a whole (just add logic to <code class="highlighter-rouge">ItemCustomDataValidationAttribute</code>).</p>
<hr />
<p><strong>NOTE: this API is still internal, use at own risk. Only lists using primitive types are supported by Episerver</strong></p>
Episerver on Azure: finding a memory leakhttps://www.brianweet.com/2019/01/11/finding-memory-leak-epi-on-azure.html2019-01-11T17:00:00.0000000Z<p>Tracking down performance issues can be tricky as they can be caused by many things. Somewhere this week I noticed some performance issues on one of our sites. Every day, the site seemed to slow down for quite a while.
Not all requests were affected, only some, but it was quite clear that there was something wrong. Looking at performance data for a full week it looked like there was a pattern to these issues, however they did not seem to occur on a specific time of the day. Double checking our scheduled jobs satisfied my expectations, it wasn’t caused by any (heavy) import job. Finding the pattern was quite easy, just look at the chart below, tracking it down might be a bit harder. Below you see the availability chart which tracks both availability and request performance.</p>
<p class="centered-image">
<img src="/assets/memory-leak/0.availability-chart.png" alt="Availability chart" />
</p>
<p>As we’re running on Azure, <a href="https://blogs.msdn.microsoft.com/appserviceteam/2018/09/24/announcing-the-new-app-service-diagnostics-experience/">Diagnose and solve problems</a> led me to the memory health chart, which uncovered the likely root of our issues.</p>
<p class="centered-image">
<img src="/assets/memory-leak/1.memory-health.png" alt="Memory health" />
</p>
<p>We’re looking at a time frame of <a href="https://plumbr.io/blog/memory-leaks/memory-leaks-fallacies-and-misconceptions">24h here</a>, the memory usage drop in the morning of 01/06 wasn’t caused by a big GC (well sort of 😋), it was caused by a recycle. The reason for the recycle I found in the <code class="highlighter-rouge">LogFiles/eventlog.xml</code> file by using <a href="https://blogs.msdn.microsoft.com/benjaminperkins/2017/11/08/how-to-access-kudu-scm-for-an-azure-app-service-environment-ase/">Kudu SCM</a></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><Events>
...
<Event>
<System>
<Provider Name="W3SVC-WP"/>
<EventID>2299</EventID>
<Level>1</Level>
<Task>0</Task>
<Keywords>Keywords</Keywords>
<TimeCreated SystemTime="2019-01-06T05:10:48Z"/>
<EventRecordID>x</EventRecordID>
<Channel>Application</Channel>
<Computer></Computer>
<Security/>
</System>
<EventData>
<Data>Worker Process requested recycle due to 'Percent Memory' limit. Memory Used: 3453165568 out of 3757625344 available. Exceeded 90 Percent of Memory
</Data>
</EventData>
</Event>
...
</Events>
</code></pre></div></div>
<h3 id="comparing-memory-dumps">Comparing memory dumps</h3>
<p>As there’s something building up over time I thought it made sense to compare two memory dumps, taken at a different point in time. I took one in the afternoon and one in the evening. Taking a memory a memory dump on Azure is very easy:</p>
<p class="centered-image">
<img src="/assets/memory-leak/kudu-memory-dump.png" alt="KUDU take a memory dump" />
</p>
<ul>
<li>Go to KUDU SCM and open the process explorer (for example https://<my-example-site>.scm.azurewebsites.net/ProcessExplorer/)</li>
<li>Look for the w3wp.exe process <strong>without</strong> the <code class="highlighter-rouge">scm</code> label and copy the <code class="highlighter-rouge">pid</code></li>
<li>Browse to https://<my-example-site>.scm.azurewebsites.net/api/processes/<insert-pid-here>/dump?dumpType=2&format=zip
</insert-pid-here> <ul>
<li>This will generate a <code class="highlighter-rouge">full memory minidump of process with id. This will include matching sos.dll and mscordackws.dll in the zip package.</code><a href="https://github.com/projectkudu/kudu/wiki/Process-Threads-list-and-minidump-gcdump-diagsession">see docs</a></li>
<li><strong>NOTE:</strong> This will make your site unresponsive for a couple of minutes</li>
</ul>
</li>
<li>Wait for the download and unzip to a folder</li>
</ul>
<p><em>I tried some other options for taking a memory dump as well, however I ended up with a corrupted dump twice. It’s so easy to take a memory dump with the procedure I just described that I won’t bother doing anything else in the future.</em></p>
<p>Now we’ll start to analyze the memory dumps, as I said I decided to compare both .dmp files.</p>
<ul>
<li>Open <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools">windbg</a>, open the <code class="highlighter-rouge">.dmp</code> file</li>
<li>Wait for it to load, run the following commands to inspect memory location in the heap. This might take a couple of minutes</li>
</ul>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.load D:\windbg\my-example-site\w3wp-5a0f81-01-09-20-38-16\sos.dll
!dumpheap -stat
</code></pre></div></div>
<p class="centered-image">
<img src="/assets/memory-leak/memory-dumps.png" alt="Compare memory dumps" />
</p>
<p>To be honest, I was not completely sure what I was looking for. However one thing caught my attention right away. For some reason we have loads of ITelemetry objects in memory, almost 7.8k in the afternoon and 18.65k in the evening. This does not make sense as these should be flushed quite regularly, indicating some misconfiguration of Application Insights. I decided to remove some of our custom tracking to see what happened.</p>
<p class="centered-image">
<img src="/assets/memory-leak/2.memory-usage-after.png" alt="Memory usage after" />
</p>
<p>After the release around 14:00 you see the memory drop and, as I hoped, we don’t see that nasty slope on our memory usage chart anymore. It turns out our memory issues were quite obvious and easy to track down. I hope this post is of any help, for me it was great to find out how easy it is to take a memory dump on Azure.</p>
Mozilla Web of Things: turning my home ventilation into a Web Thinghttps://brianweet.github.io/2018/10/18/web-of-things-fan.html2018-10-18T22:00:00.0000000Z<p>A while ago I’ve created a system to be able to control my <a href="/2018/04/15/dont-worry-take-a-shower.html">home ventilation fan</a> by using a 433mhz remote dimmer. In this post I will be using <a href="https://iot.mozilla.org/">Mozilla IoT</a> to create a ‘Web Thing’ out of that same home ventilation fan.</p>
<p class="centered-image">
<img src="/assets/mozilla-iot/control-fan-mobile.jpg" alt="Search and control" />
</p>
<p>
<strong>On the left, a raspberry pi with a 433mhz transmitter. On the right, the fan, the remote dimmer and a phone showing the Gateway Things UI (no worries, the wires go to a power adapter not my phone). </strong>
</p>
<p>I want to be able to control my ‘Things’ easily and as I’m a web developer by day (and by night), I prefer to use the web technologies well known to me. The <a href="https://iot.mozilla.org/wot/">Web Thing API</a> initiative tries to standardize the definition of a Web Thing and its properties, fortunately it already seems to support my (very basic) requirements!</p>
<p class="centered-image">
<img src="https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2018/10/Screen-Shot-2018-10-09-at-16.17.59.png" alt="Mozilla Things Gateway" />
</p>
<p>
<strong>An example of the Things Gateway UI displaying various Web Things that can be monitored and controlled.</strong>
</p>
<h2 id="the-plan">The plan</h2>
<p>I will to need set-up two essential items to get this Mozilla Web of Things running within my network. First I need a Things Gateway in order to monitor and control Web Things. Basically it’s a web application, running on my raspberry pi, that can be used to scan the network for Web Things. Once configured it can be used to interact with the Web Things, provided that they have defined any interaction.</p>
<p>Second, I need turn my fan into a Web Thing. This basically means that I will have to define the properties and behaviour of my fan. Afterwards it will be accessibly on my network by using web requests / web sockets. This, as well, will run on my raspberry pi as it is the only device I have with a 433mhz transmitter.</p>
<h2 id="step-1-installing-the-gateway">Step 1: Installing the gateway</h2>
<p>Fortunately Mozilla provides instructions to <a href="https://iot.mozilla.org/gateway/">set-up such a gateway</a> on a raspberry pi. Since I didn’t want to reinstall my raspberry pi I’ve decided to check out the code and set up the gateway myself, using the <a href="https://github.com/mozilla-iot/gateway/blob/master/README.md">installation instructions</a>.</p>
<p>This went surprisingly smooth; within no-time I had a Things Gateway set up with a public URL provided by Mozilla. If you need any help with deciphering the installation instructions, feel free to send me a message.</p>
<p>If you need any help with this let me know, within minutes I had a Things Gateway set up with a public URL provided by Mozilla. Basically I followed the instructions and used the scripts in the <a href="https://github.com/mozilla-iot/gateway/tree/master/image">gateway/image</a> folder.</p>
<p class="centered-image">
<img src="/assets/mozilla-iot/empty-gateway.png" alt="Empty Things gateway" />
</p>
<p>
<strong>An empty gateway, next up - create a Web Thing!</strong>
</p>
<h2 id="step-2-turning-the-fan-into-a-web-thing">Step 2: Turning the fan into a Web Thing</h2>
<p>The second step is to create a Web Thing that exposes certain properties, in this case an on and off switch and a dim/level property. The fan has 16 different speed levels and can be turned off (this means we end up with 17 levels, as ‘off’ is the lowest setting). Turning the fan off actually sets it to ‘idle’, it’s mandatory for the fan to be running as that is needed to ventilate the house.</p>
<p>In my previous blog post I’ve already created some scripts that allow me to control the fan. On my raspberry I have a script called <code class="highlighter-rouge">set-fan-level.sh</code>, calling this script will turn on the fan and set it to a certain speed. For example <code class="highlighter-rouge">./set-level-fan.sh 15</code> turns on the fan on full speed and <code class="highlighter-rouge">./set-level-fan.sh 0</code> turns on the fan at the speed level just above ‘idle’.</p>
<p>In order to create a web thing I’ve used the <code class="highlighter-rouge">webthing</code> npm package. I’ve created a Web Thing with two properties, an <a href="https://iot.mozilla.org/schemas/#OnOffProperty">OnOffProperty</a> and a <a href="https://iot.mozilla.org/schemas/#LevelProperty">LevelProperty</a>.</p>
<p>The LevelProperty determines the speed of the fan, and if I change the speed of the fan I immediately turn on the fan as well.</p>
<p>The OnOffProperty reads the value of the LevelProperty and turns on the fan by calling the <code class="highlighter-rouge">set-fan-level.sh</code> script, or it turns the fan off by calling the <code class="highlighter-rouge">stop-fan.sh</code> script.</p>
<p>Without further ado, here is the code:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">import</span> <span class="p">{</span> <span class="nx">runScript</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'./run-script'</span><span class="p">;</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">Property</span><span class="p">,</span> <span class="nx">Thing</span><span class="p">,</span> <span class="nx">Value</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'webthing'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">path</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'path'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">startFanScript</span> <span class="o">=</span> <span class="nx">path</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="s1">'../fan-scripts/set-level-fan.sh'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">stopFanScript</span> <span class="o">=</span> <span class="nx">path</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="s1">'../fan-scripts/stop-fan.sh'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">onOffPropertyName</span> <span class="o">=</span> <span class="s1">'on'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">fanLevelPropertyName</span> <span class="o">=</span> <span class="s1">'level'</span><span class="p">;</span>
<span class="k">export</span> <span class="kd">class</span> <span class="nx">WebFanThing</span> <span class="kd">extends</span> <span class="nx">Thing</span> <span class="p">{</span>
<span class="kd">constructor</span><span class="p">()</span> <span class="p">{</span>
<span class="k">super</span><span class="p">(</span><span class="s1">'Home ventilation fan'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'OnOffSwitch'</span><span class="p">],</span> <span class="s1">'Home ventilation fan'</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">addProperty</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">Property</span><span class="p">(</span>
<span class="k">this</span><span class="p">,</span>
<span class="nx">onOffPropertyName</span><span class="p">,</span>
<span class="k">new</span> <span class="nx">Value</span><span class="p">(</span><span class="kc">false</span><span class="p">,</span> <span class="p">(</span><span class="nx">v</span><span class="p">:</span> <span class="kr">boolean</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'On-State is now'</span><span class="p">,</span> <span class="nx">v</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">levelProp</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">getProperty</span><span class="p">(</span><span class="nx">fanLevelPropertyName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">v</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">runScript</span><span class="p">(</span><span class="nx">startFanScript</span><span class="p">,</span> <span class="p">[</span><span class="s2">`</span><span class="p">${</span><span class="nx">levelProp</span><span class="p">}</span><span class="s2">`</span><span class="p">]);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">runScript</span><span class="p">(</span><span class="nx">stopFanScript</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}),</span>
<span class="p">{</span>
<span class="s1">'@type'</span><span class="p">:</span> <span class="s1">'OnOffProperty'</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span> <span class="s1">'On/Off'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'boolean'</span><span class="p">,</span>
<span class="na">description</span><span class="p">:</span> <span class="s1">'Whether the fan is turned on'</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">addProperty</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">Property</span><span class="p">(</span>
<span class="k">this</span><span class="p">,</span>
<span class="nx">fanLevelPropertyName</span><span class="p">,</span>
<span class="k">new</span> <span class="nx">Value</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span> <span class="p">(</span><span class="na">v</span><span class="p">:</span> <span class="nx">number</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Fan level is now'</span><span class="p">,</span> <span class="nx">v</span><span class="p">);</span>
<span class="c1">// Schedule 'turn fan on'</span>
<span class="nx">setTimeout</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">onOffProp</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">findProperty</span><span class="p">(</span><span class="nx">onOffPropertyName</span><span class="p">);</span>
<span class="nx">onOffProp</span><span class="p">.</span><span class="nx">setValue</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span>
<span class="p">},</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}),</span>
<span class="p">{</span>
<span class="s1">'@type'</span><span class="p">:</span> <span class="s1">'LevelProperty'</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span> <span class="s1">'Fan speed level'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'number'</span><span class="p">,</span>
<span class="na">description</span><span class="p">:</span> <span class="s1">'The fan level from 0-15'</span><span class="p">,</span>
<span class="na">minimum</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="na">maximum</span><span class="p">:</span> <span class="mi">15</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As you can see; the implementation is pretty straight forward. I’ve defined the fan as an OnOffSwitch, this allows me to switch it off easily from the Things Gateway UI. The LevelProperty can be set in the Web Thing ‘details’, and it ensures the OnOffProperty is set to true.</p>
<p class="image">
<img src="/assets/mozilla-iot/webthing-setup-and-trigger.gif" alt="Web Things set up and trigger" />
</p>
<p>
<strong>Here I add the Fan Web Thing to the Gateway and control it by turning it on and changing the level. Web Thing (debug) logging on the right.</strong>
</p>
<p>But how does the gateway find the Web Thing? Of course this is no magic, you can use the <code class="highlighter-rouge">webthing</code> package to do most of the work for you. All you have to do is set up a WebThingServer with the kind of thing(s) you want to expose (single or multiple) and which port you want to use and you’re done:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="p">{</span> <span class="nx">SingleThing</span><span class="p">,</span> <span class="nx">WebThingServer</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'webthing'</span><span class="p">);</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">WebFanThing</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'./webfan-thing'</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">runServer</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">thing</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebFanThing</span><span class="p">();</span>
<span class="c1">// If adding more than one thing, use MultipleThings() with a name.</span>
<span class="c1">// In the single thing case, the thing's name will be broadcast.</span>
<span class="kd">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebThingServer</span><span class="p">(</span><span class="k">new</span> <span class="nx">SingleThing</span><span class="p">(</span><span class="nx">thing</span><span class="p">),</span> <span class="mi">8888</span><span class="p">);</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'SIGINT'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">stop</span><span class="p">();</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">exit</span><span class="p">();</span>
<span class="p">});</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">start</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Web fan server started'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">runServer</span><span class="p">();</span>
</code></pre></div></div>
<p>As mentioned before, I’m running the code on my raspberry pi as well. It is the only device I have with a 433mhz transmitter connected to it and I need that to send messages to the dimmer connected to the fan. Once you run the code you will see the capabilities of your Web Thing by doing a GET request or browsing to the URL <code class="highlighter-rouge">http://your-webthing-ip-address:8888</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Home ventilation fan"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"@context"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https:</span><span class="se">\/\/</span><span class="s2">iot.mozilla.org</span><span class="se">\/</span><span class="s2">schemas"</span><span class="p">,</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"OnOffSwitch"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"on"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"OnOffProperty"</span><span class="p">,</span><span class="w">
</span><span class="s2">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"On</span><span class="se">\/</span><span class="s2">Off"</span><span class="p">,</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"boolean"</span><span class="p">,</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Whether the fan is turned on"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties</span><span class="se">\/</span><span class="s2">on"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"LevelProperty"</span><span class="p">,</span><span class="w">
</span><span class="s2">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Fan speed level"</span><span class="p">,</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"number"</span><span class="p">,</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The fan level from 0-15"</span><span class="p">,</span><span class="w">
</span><span class="s2">"minimum"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="s2">"maximum"</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties</span><span class="se">\/</span><span class="s2">level"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"actions"</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span><span class="w">
</span><span class="s2">"events"</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span><span class="w">
</span><span class="s2">"links"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"properties"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"actions"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">actions"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"events"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">events"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"alternate"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ws:</span><span class="se">\/\/</span><span class="s2">your-webthing-ip-address:8888"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Home ventilation fan"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>You can interact with the Web Thing by using normal REST operations against it, for example a GET on <code class="highlighter-rouge">/properties/level</code> gives me <code class="highlighter-rouge">{"level":7}</code>. If I want to change the level I can do a simple PUT and pass the new level as a body param (fiddler snippet):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PUT http://your-webthing-ip-address:8888/properties/level HTTP/1.1
User-Agent: Fiddler
Host: your-webthing-ip-address:8888
Content-Type: application/json; charset=utf-8
Content-Length: 12
{"level":15}
</code></pre></div></div>
<p>It returns a 200 OK and I hear the fan running on full speed!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/1.1 200 OK
X-Powered-By: Express
Date: Wed, 17 Oct 2018 22:47:43 GMT
....
{"level":15}
</code></pre></div></div>
<p class="centered-image">
<a href="https://www.flickr.com/photos/zestivales_lehavre/2679998843">
<img src="/assets/mozilla-iot/fan-success.jpg" alt="Fan success" />
</a>
</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope I was able to show you how easy it is to wrap an existing (physical) device using the Web Things API. All I had to do was install the Things Gateway on my Raspberry pi and define a Web Thing for my home ventilation fan. The Things Gateway is accessible from within and from outside of my network and it provides a clean and simple PWA to control your Web Things with. If you’re interested in the code, you can find it <a href="https://github.com/brianweet/home-automation/tree/master/src/WebFan/src">here</a>.</p>
Mozilla Web of Things: turning my home ventilation into a Web Thinghttps://www.brianweet.com/2018/10/18/web-of-things-fan.html2018-10-18T22:00:00.0000000Z<p>A while ago I’ve created a system to be able to control my <a href="/2018/04/15/dont-worry-take-a-shower.html">home ventilation fan</a> by using a 433mhz remote dimmer. In this post I will be using <a href="https://iot.mozilla.org/about/">Project Things by Mozilla</a> to create a ‘Web Thing’ out of that same home ventilation fan.</p>
<p class="centered-image">
<img src="/assets/mozilla-iot/control-fan-mobile.jpg" alt="Search and control" />
</p>
<p>
<strong>On the left, a raspberry pi with a 433mhz transmitter. On the right, the fan, the remote dimmer and a phone showing the Gateway Things UI (no worries, the wires go to a power adapter not my phone). </strong>
</p>
<p>I want to be able to control my ‘Things’ easily and as I’m a web developer by day (and by night), I prefer to use the web technologies well known to me. The <a href="https://iot.mozilla.org/wot/">Web Thing API</a> initiative tries to standardize the definition of a Web Thing and its properties, fortunately it already seems to support my (very basic) requirements!</p>
<p class="centered-image">
<img src="https://2r4s9p1yi1fa2jd7j43zph8r-wpengine.netdna-ssl.com/files/2018/10/Screen-Shot-2018-10-09-at-16.17.59.png" alt="Mozilla Things Gateway" />
</p>
<p>
<strong>An example of the Things Gateway UI displaying various Web Things that can be monitored and controlled.</strong>
</p>
<h2 id="the-plan">The plan</h2>
<p>I will to need set-up two essential items to get this Mozilla Web of Things running within my network. First I need a Things Gateway in order to monitor and control Web Things. Basically it’s a web application, running on my raspberry pi, that can be used to scan the network for Web Things. Once configured it can be used to interact with the Web Things, provided that they have defined any interaction.</p>
<p>Second, I need turn my fan into a Web Thing. This basically means that I will have to define the properties and behaviour of my fan. Afterwards it will be accessibly on my network by using web requests / web sockets. This, as well, will run on my raspberry pi as it is the only device I have with a 433mhz transmitter.</p>
<h2 id="step-1-installing-the-gateway">Step 1: Installing the gateway</h2>
<p>Fortunately Mozilla provides instructions to <a href="https://iot.mozilla.org/gateway/">set-up such a gateway</a> on a raspberry pi. Since I didn’t want to reinstall my raspberry pi I’ve decided to check out the code and set up the gateway myself, using the <a href="https://github.com/mozilla-iot/gateway/blob/master/README.md">installation instructions</a>.</p>
<p>This went surprisingly smooth; within no-time I had a Things Gateway set up with a public URL provided by Mozilla. If you need any help with deciphering the installation instructions, feel free to send me a message.</p>
<p>If you need any help with this let me know, within minutes I had a Things Gateway set up with a public URL provided by Mozilla. Basically I followed the instructions and used the scripts in the <a href="https://github.com/mozilla-iot/gateway/tree/master/image">gateway/image</a> folder.</p>
<p class="centered-image">
<img src="/assets/mozilla-iot/empty-gateway.png" alt="Empty Things gateway" />
</p>
<p>
<strong>An empty gateway, next up - create a Web Thing!</strong>
</p>
<h2 id="step-2-turning-the-fan-into-a-web-thing">Step 2: Turning the fan into a Web Thing</h2>
<p>The second step is to create a Web Thing that exposes certain properties, in this case an on and off switch and a dim/level property. The fan has 16 different speed levels and can be turned off (this means we end up with 17 levels, as ‘off’ is the lowest setting). Turning the fan off actually sets it to ‘idle’, it’s mandatory for the fan to be running as that is needed to ventilate the house.</p>
<p>In my previous blog post I’ve already created some scripts that allow me to control the fan. On my raspberry I have a script called <code class="highlighter-rouge">set-fan-level.sh</code>, calling this script will turn on the fan and set it to a certain speed. For example <code class="highlighter-rouge">./set-level-fan.sh 15</code> turns on the fan on full speed and <code class="highlighter-rouge">./set-level-fan.sh 0</code> turns on the fan at the speed level just above ‘idle’.</p>
<p>In order to create a web thing I’ve used the <code class="highlighter-rouge">webthing</code> npm package. I’ve created a Web Thing with two properties, an <a href="https://iot.mozilla.org/schemas/#OnOffProperty">OnOffProperty</a> and a <a href="https://iot.mozilla.org/schemas/#LevelProperty">LevelProperty</a>.</p>
<p>The LevelProperty determines the speed of the fan, and if I change the speed of the fan I immediately turn on the fan as well.</p>
<p>The OnOffProperty reads the value of the LevelProperty and turns on the fan by calling the <code class="highlighter-rouge">set-fan-level.sh</code> script, or it turns the fan off by calling the <code class="highlighter-rouge">stop-fan.sh</code> script.</p>
<p>Without further ado, here is the code:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">import</span> <span class="p">{</span> <span class="nx">runScript</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'./run-script'</span><span class="p">;</span>
<span class="kd">const</span> <span class="p">{</span> <span class="nx">Property</span><span class="p">,</span> <span class="nx">Thing</span><span class="p">,</span> <span class="nx">Value</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'webthing'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">path</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'path'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">startFanScript</span> <span class="o">=</span> <span class="nx">path</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="s1">'../fan-scripts/set-level-fan.sh'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">stopFanScript</span> <span class="o">=</span> <span class="nx">path</span><span class="p">.</span><span class="nx">resolve</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="s1">'../fan-scripts/stop-fan.sh'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">onOffPropertyName</span> <span class="o">=</span> <span class="s1">'on'</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">fanLevelPropertyName</span> <span class="o">=</span> <span class="s1">'level'</span><span class="p">;</span>
<span class="k">export</span> <span class="kd">class</span> <span class="nx">WebFanThing</span> <span class="kd">extends</span> <span class="nx">Thing</span> <span class="p">{</span>
<span class="kd">constructor</span><span class="p">()</span> <span class="p">{</span>
<span class="k">super</span><span class="p">(</span><span class="s1">'Home ventilation fan'</span><span class="p">,</span> <span class="p">[</span><span class="s1">'OnOffSwitch'</span><span class="p">],</span> <span class="s1">'Home ventilation fan'</span><span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">addProperty</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">Property</span><span class="p">(</span>
<span class="k">this</span><span class="p">,</span>
<span class="nx">onOffPropertyName</span><span class="p">,</span>
<span class="k">new</span> <span class="nx">Value</span><span class="p">(</span><span class="kc">false</span><span class="p">,</span> <span class="p">(</span><span class="nx">v</span><span class="p">:</span> <span class="kr">boolean</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'On-State is now'</span><span class="p">,</span> <span class="nx">v</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">levelProp</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">getProperty</span><span class="p">(</span><span class="nx">fanLevelPropertyName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">v</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">runScript</span><span class="p">(</span><span class="nx">startFanScript</span><span class="p">,</span> <span class="p">[</span><span class="s2">`</span><span class="p">${</span><span class="nx">levelProp</span><span class="p">}</span><span class="s2">`</span><span class="p">]);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nx">runScript</span><span class="p">(</span><span class="nx">stopFanScript</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}),</span>
<span class="p">{</span>
<span class="s1">'@type'</span><span class="p">:</span> <span class="s1">'OnOffProperty'</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span> <span class="s1">'On/Off'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'boolean'</span><span class="p">,</span>
<span class="na">description</span><span class="p">:</span> <span class="s1">'Whether the fan is turned on'</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="k">this</span><span class="p">.</span><span class="nx">addProperty</span><span class="p">(</span>
<span class="k">new</span> <span class="nx">Property</span><span class="p">(</span>
<span class="k">this</span><span class="p">,</span>
<span class="nx">fanLevelPropertyName</span><span class="p">,</span>
<span class="k">new</span> <span class="nx">Value</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span> <span class="p">(</span><span class="na">v</span><span class="p">:</span> <span class="nx">number</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Fan level is now'</span><span class="p">,</span> <span class="nx">v</span><span class="p">);</span>
<span class="c1">// Schedule 'turn fan on'</span>
<span class="nx">setTimeout</span><span class="p">(()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">onOffProp</span> <span class="o">=</span> <span class="k">this</span><span class="p">.</span><span class="nx">findProperty</span><span class="p">(</span><span class="nx">onOffPropertyName</span><span class="p">);</span>
<span class="nx">onOffProp</span><span class="p">.</span><span class="nx">setValue</span><span class="p">(</span><span class="kc">true</span><span class="p">);</span>
<span class="p">},</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}),</span>
<span class="p">{</span>
<span class="s1">'@type'</span><span class="p">:</span> <span class="s1">'LevelProperty'</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span> <span class="s1">'Fan speed level'</span><span class="p">,</span>
<span class="na">type</span><span class="p">:</span> <span class="s1">'number'</span><span class="p">,</span>
<span class="na">description</span><span class="p">:</span> <span class="s1">'The fan level from 0-15'</span><span class="p">,</span>
<span class="na">minimum</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
<span class="na">maximum</span><span class="p">:</span> <span class="mi">15</span><span class="p">,</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As you can see; the implementation is pretty straight forward. I’ve defined the fan as an OnOffSwitch, this allows me to switch it off easily from the Things Gateway UI. The LevelProperty can be set in the Web Thing ‘details’, and it ensures the OnOffProperty is set to true.</p>
<p class="image">
<img src="/assets/mozilla-iot/webthing-setup-and-trigger.gif" alt="Web Things set up and trigger" />
</p>
<p>
<strong>Here I add the Fan Web Thing to the Gateway and control it by turning it on and changing the level. Web Thing (debug) logging on the right.</strong>
</p>
<p>But how does the gateway find the Web Thing? Of course this is no magic, you can use the <code class="highlighter-rouge">webthing</code> package to do most of the work for you. All you have to do is set up a WebThingServer with the kind of thing(s) you want to expose (single or multiple) and which port you want to use and you’re done:</p>
<div class="language-ts highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="p">{</span> <span class="nx">SingleThing</span><span class="p">,</span> <span class="nx">WebThingServer</span> <span class="p">}</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="s1">'webthing'</span><span class="p">);</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">WebFanThing</span> <span class="p">}</span> <span class="k">from</span> <span class="s1">'./webfan-thing'</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">runServer</span><span class="p">()</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">thing</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebFanThing</span><span class="p">();</span>
<span class="c1">// If adding more than one thing, use MultipleThings() with a name.</span>
<span class="c1">// In the single thing case, the thing's name will be broadcast.</span>
<span class="kd">const</span> <span class="nx">server</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">WebThingServer</span><span class="p">(</span><span class="k">new</span> <span class="nx">SingleThing</span><span class="p">(</span><span class="nx">thing</span><span class="p">),</span> <span class="mi">8888</span><span class="p">);</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="s1">'SIGINT'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">stop</span><span class="p">();</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">exit</span><span class="p">();</span>
<span class="p">});</span>
<span class="nx">server</span><span class="p">.</span><span class="nx">start</span><span class="p">();</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s1">'Web fan server started'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nx">runServer</span><span class="p">();</span>
</code></pre></div></div>
<p>As mentioned before, I’m running the code on my raspberry pi as well. It is the only device I have with a 433mhz transmitter connected to it and I need that to send messages to the dimmer connected to the fan. Once you run the code you will see the capabilities of your Web Thing by doing a GET request or browsing to the URL <code class="highlighter-rouge">http://your-webthing-ip-address:8888</code>.</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Home ventilation fan"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">"</span><span class="p">,</span><span class="w">
</span><span class="s2">"@context"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https:</span><span class="se">\/\/</span><span class="s2">iot.mozilla.org</span><span class="se">\/</span><span class="s2">schemas"</span><span class="p">,</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"OnOffSwitch"</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"on"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"OnOffProperty"</span><span class="p">,</span><span class="w">
</span><span class="s2">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"On</span><span class="se">\/</span><span class="s2">Off"</span><span class="p">,</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"boolean"</span><span class="p">,</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Whether the fan is turned on"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties</span><span class="se">\/</span><span class="s2">on"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"level"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"@type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"LevelProperty"</span><span class="p">,</span><span class="w">
</span><span class="s2">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Fan speed level"</span><span class="p">,</span><span class="w">
</span><span class="s2">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"number"</span><span class="p">,</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The fan level from 0-15"</span><span class="p">,</span><span class="w">
</span><span class="s2">"minimum"</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span><span class="w">
</span><span class="s2">"maximum"</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties</span><span class="se">\/</span><span class="s2">level"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"actions"</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span><span class="w">
</span><span class="s2">"events"</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span><span class="w">
</span><span class="s2">"links"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"properties"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">properties"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"actions"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">actions"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"events"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"</span><span class="se">\/</span><span class="s2">events"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"rel"</span><span class="p">:</span><span class="w"> </span><span class="s2">"alternate"</span><span class="p">,</span><span class="w">
</span><span class="s2">"href"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ws:</span><span class="se">\/\/</span><span class="s2">your-webthing-ip-address:8888"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"description"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Home ventilation fan"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>You can interact with the Web Thing by using normal REST operations against it, for example a GET on <code class="highlighter-rouge">/properties/level</code> gives me <code class="highlighter-rouge">{"level":7}</code>. If I want to change the level I can do a simple PUT and pass the new level as a body param (fiddler snippet):</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>PUT http://your-webthing-ip-address:8888/properties/level HTTP/1.1
User-Agent: Fiddler
Host: your-webthing-ip-address:8888
Content-Type: application/json; charset=utf-8
Content-Length: 12
{"level":15}
</code></pre></div></div>
<p>It returns a 200 OK and I hear the fan running on full speed!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/1.1 200 OK
X-Powered-By: Express
Date: Wed, 17 Oct 2018 22:47:43 GMT
....
{"level":15}
</code></pre></div></div>
<p class="centered-image">
<a href="https://www.flickr.com/photos/zestivales_lehavre/2679998843">
<img src="/assets/mozilla-iot/fan-success.jpg" alt="Fan success" />
</a>
</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope I was able to show you how easy it is to wrap an existing (physical) device using the Web Things API. All I had to do was install the Things Gateway on my Raspberry pi and define a Web Thing for my home ventilation fan. The Things Gateway is accessible from within and from outside of my network and it provides a clean and simple PWA to control your Web Things with. If you’re interested in the code, you can find it <a href="https://github.com/brianweet/home-automation/tree/master/src/WebFan/src">here</a>.</p>
Running mssql-server-linux on windows using LCOWhttps://brianweet.github.io/2018/04/26/running-mssql-server-linux-using-lcow.html2018-04-26T02:01:00.0000000Z<p>I got a popup from Docker a while ago mentioning LCOW containers. The acronym stands for Linux Containers On Windows, which made me happy for some reason. Why you may ask? One of the first things I tried to do with docker was to run two containers, one based on Linux and one based on Windows (as I’m usually stuck with IIS due to Episerver). Unfortunately I found out that I was asking for the impossible, or at least not-viable-for-me at that moment. But with LCOW this will change, I decided to enable the experimental features and see what all the fuss is about.</p>
<h4 id="running-simple-example">Running simple example</h4>
<p>So I switched docker to use Windows containers and tried to run a linux container using the example from <a href="https://github.com/linuxkit/lcow">lcow</a>. <code class="highlighter-rouge">docker run --platform linux --rm -ti busybox sh</code></p>
<p class="centered-image">
<img src="/assets/mssql-linux/first-try.png" alt="First try LCOW, success" />
</p>
<p>Magic. It works on the first try! Expectations are rising, will it actually work without any problems?! (I had quite some problems when I first started with docker, had no experience with the concept neither so that didn’t help)</p>
<p class="centered-image">
<img src="/assets/mssql-linux/im-so-excited.jpg" alt="So excited, I just can't hide it" />
</p>
<h4 id="running-mssql-server-linux-with-lcow">Running mssql-server-linux with LCOW</h4>
<p>One of the images I built previously, based on <a href="https://hub.docker.com/r/microsoft/mssql-server-windows-developer/">windows image for mssql</a>, was quite big: 15GB when I was using it about 6 months ago and still 10.8GB after I updated the image to a newer version, yesterday <a href="/2017/10/09/alloy-in-docker.html#summary">(see edit in this post)</a>. That image still huge compared to the base image for mssql-server-linux.</p>
<p>It is also quite seems clear to me that the linux image is the way to go, just by looking at the update frequency or the pull stats:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/different-images.png" alt="Mssql docker image stats" />
</p>
<p>As the initial experiment went so smooth, I was excited and full of hope when I ran the command to start the mssql linux container:
<code class="highlighter-rouge">docker run --platform linux --rm -ti -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -p 1433:1433 microsoft/mssql-server-linux</code></p>
<p class="centered-image">
<img src="/assets/mssql-linux/need-memory.png" alt="Need memory" />
</p>
<p>I received an error message saying: <code class="highlighter-rouge">sqlservr: This program requires a machine with at least 2000 megabytes of memory</code>.
Ah ok… So it does start but it just needs more memory, I’ve seen this before, no problem. Just change the memory setting somewhere.. But where?!</p>
<p>After searching in the app, searching online, doing some trial and error I found out that it’s not possible to change the default memory size set for the lcow container. Which is a real shame as it means I still can’t do what I wanted to, start the mssql linux container. As can be seen in this epic, the ‘Memory and CPU settings’ task is not <a href="https://github.com/moby/moby/issues/33850">started yet</a>.
Reading through the referenced bug I found <a href="https://github.com/Microsoft/opengcs/issues/145#issuecomment-376439116">a post</a> mentioning that it should be possible to hardcode a value for the memorysize, recompile docker and make it work. I could not resist to give it a try.</p>
<h4 id="hardcoding-memorymaximuminmb-and-recompiling-docker">Hardcoding MemoryMaximumInMB and recompiling docker</h4>
<p>Everyone knows how much fun it is to compile completely unknown stuff using a completely unknown stack. Slightly hesitant I started searching through the moby repository to figure out how to compile docker. I found <a href="https://github.com/moby/moby/blob/master/Dockerfile.windows">the dockerfile for windows</a>, with some great documentation inside:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/build-docs.png" alt="Build documentation" />
</p>
<p>Basically you check out the source code and they provide some scripts and a docker file which starts a container and builds docker within that container.
Between step 2 and 3 I opened up the <code class="highlighter-rouge">client_local_windows.go</code> file and added a hardcoded value for <code class="highlighter-rouge">MemoryMaximumInMB</code>, <a href="https://github.com/brianweet/moby/commit/cacae69f7adc35800becd7eb044642b1267279d1">see commit</a>.</p>
<p class="centered-image">
<img src="/assets/mssql-linux/hardcoded-memorysize.png" alt="Hardcoded memory size" />
</p>
<p>I ended up with a custom compiled dockerd.exe, all that was left was to stop docker, replace the dockerd.exe and restart docker (for me dockerd was located at <code class="highlighter-rouge">C:\Program Files\Docker\Docker\resources</code>). Let’s see what happens if I try to start mssql now:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/no-more-memory-error.png" alt="No more memory error" />
</p>
<p>Yay, the memory error is gone! I probably did something wrong with the env variables though but that’s not important, at least now sql is able to start and doesn’t complain about memory restrictions anymore.</p>
<h4 id="running-alloy-demo-with-mssql-server-linux">Running Alloy demo with mssql-server-linux</h4>
<p>As with the <a href="/2017/10/09/alloy-in-docker.html">previous Alloy example</a>, I decided to create a db image which includes the database itself. The linux image does not have an <code class="highlighter-rouge">attach_dbs</code> params, so I decided to copy over the code from the <a href="https://github.com/twright-msft/mssql-node-docker-demo-app/">mssql-node-docker-demo-app</a> and adapted it to <a href="https://github.com/brianweet/AlloyDemoKit/tree/docker-linux/Build-sql-linux">my needs</a>. After adding all of the data to the image, we’re left with just 1.73GB, a huge difference from the previous 10.8GB/15GB!</p>
<p class="centered-image">
<img src="/assets/mssql-linux/new-image-sizes.png" alt="New image sizes" />
</p>
<p>If you want to run Alloy demo using the linux container, use <a href="https://github.com/brianweet/AlloyDemoKit/blob/docker-linux/Run-using-pre-built-images-linux/docker-compose.yml">this docker-compose</a> file:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">alloydemokit</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit:v2</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloydemokit-db</span>
<span class="na">alloydemokit-db</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit-db:linux</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="na">alloy-network</span><span class="pi">:</span>
<span class="na">external</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">nat</span>
</code></pre></div></div>
<p><strong>Note: you might have to pull the image in first using <code class="highlighter-rouge">docker pull --platform linux brianweet/alloydemokit-db:linux</code></strong></p>
<p class="centered-image">
<img src="/assets/mssql-linux/like-to-live-dangerously.jpg" alt="Like to live dangerously?" />
</p>
<p>If you’re feeling adventurous or you just don’t care about any laws of nature, common sense or just trust me on my blue/green/grey eyes, use my compiled dockerd.exe found <a href="/assets/mssql-linux/dockerd.rar">in this rar</a>.</p>
Running mssql-server-linux on windows using LCOWhttps://www.brianweet.com/2018/04/26/running-mssql-server-linux-using-lcow.html2018-04-26T02:01:00.0000000Z<p>I got a popup from Docker a while ago mentioning LCOW containers. The acronym stands for Linux Containers On Windows, which made me happy for some reason. Why you may ask? One of the first things I tried to do with docker was to run two containers, one based on Linux and one based on Windows (as I’m usually stuck with IIS due to Episerver). Unfortunately I found out that I was asking for the impossible, or at least not-viable-for-me at that moment. But with LCOW this will change, I decided to enable the experimental features and see what all the fuss is about.</p>
<h4 id="running-simple-example">Running simple example</h4>
<p>So I switched docker to use Windows containers and tried to run a linux container using the example from <a href="https://github.com/linuxkit/lcow">lcow</a>. <code class="highlighter-rouge">docker run --platform linux --rm -ti busybox sh</code></p>
<p class="centered-image">
<img src="/assets/mssql-linux/first-try.png" alt="First try LCOW, success" />
</p>
<p>Magic. It works on the first try! Expectations are rising, will it actually work without any problems?! (I had quite some problems when I first started with docker, had no experience with the concept neither so that didn’t help)</p>
<p class="centered-image">
<img src="/assets/mssql-linux/im-so-excited.jpg" alt="So excited, I just can't hide it" />
</p>
<h4 id="running-mssql-server-linux-with-lcow">Running mssql-server-linux with LCOW</h4>
<p>One of the images I built previously, based on <a href="https://hub.docker.com/r/microsoft/mssql-server-windows-developer/">windows image for mssql</a>, was quite big: 15GB when I was using it about 6 months ago and still 10.8GB after I updated the image to a newer version, yesterday <a href="/2017/10/09/alloy-in-docker.html#summary">(see edit in this post)</a>. That image still huge compared to the base image for mssql-server-linux.</p>
<p>It is also quite seems clear to me that the linux image is the way to go, just by looking at the update frequency or the pull stats:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/different-images.png" alt="Mssql docker image stats" />
</p>
<p>As the initial experiment went so smooth, I was excited and full of hope when I ran the command to start the mssql linux container:
<code class="highlighter-rouge">docker run --platform linux --rm -ti -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -p 1433:1433 microsoft/mssql-server-linux</code></p>
<p class="centered-image">
<img src="/assets/mssql-linux/need-memory.png" alt="Need memory" />
</p>
<p>I received an error message saying: <code class="highlighter-rouge">sqlservr: This program requires a machine with at least 2000 megabytes of memory</code>.
Ah ok… So it does start but it just needs more memory, I’ve seen this before, no problem. Just change the memory setting somewhere.. But where?!</p>
<p>After searching in the app, searching online, doing some trial and error I found out that it’s not possible to change the default memory size set for the lcow container. Which is a real shame as it means I still can’t do what I wanted to, start the mssql linux container. As can be seen in this epic, the ‘Memory and CPU settings’ task is not <a href="https://github.com/moby/moby/issues/33850">started yet</a>.
Reading through the referenced bug I found <a href="https://github.com/Microsoft/opengcs/issues/145#issuecomment-376439116">a post</a> mentioning that it should be possible to hardcode a value for the memorysize, recompile docker and make it work. I could not resist to give it a try.</p>
<h4 id="hardcoding-memorymaximuminmb-and-recompiling-docker">Hardcoding MemoryMaximumInMB and recompiling docker</h4>
<p>Everyone knows how much fun it is to compile completely unknown stuff using a completely unknown stack. Slightly hesitant I started searching through the moby repository to figure out how to compile docker. I found <a href="https://github.com/moby/moby/blob/master/Dockerfile.windows">the dockerfile for windows</a>, with some great documentation inside:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/build-docs.png" alt="Build documentation" />
</p>
<p>Basically you check out the source code and they provide some scripts and a docker file which starts a container and builds docker within that container.
Between step 2 and 3 I opened up the <code class="highlighter-rouge">client_local_windows.go</code> file and added a hardcoded value for <code class="highlighter-rouge">MemoryMaximumInMB</code>, <a href="https://github.com/brianweet/moby/commit/cacae69f7adc35800becd7eb044642b1267279d1">see commit</a>.</p>
<p class="centered-image">
<img src="/assets/mssql-linux/hardcoded-memorysize.png" alt="Hardcoded memory size" />
</p>
<p>I ended up with a custom compiled dockerd.exe, all that was left was to stop docker, replace the dockerd.exe and restart docker (for me dockerd was located at <code class="highlighter-rouge">C:\Program Files\Docker\Docker\resources</code>). Let’s see what happens if I try to start mssql now:</p>
<p class="centered-image">
<img src="/assets/mssql-linux/no-more-memory-error.png" alt="No more memory error" />
</p>
<p>Yay, the memory error is gone! I probably did something wrong with the env variables though but that’s not important, at least now sql is able to start and doesn’t complain about memory restrictions anymore.</p>
<h4 id="running-alloy-demo-with-mssql-server-linux">Running Alloy demo with mssql-server-linux</h4>
<p>As with the <a href="/2017/10/09/alloy-in-docker.html">previous Alloy example</a>, I decided to create a db image which includes the database itself. The linux image does not have an <code class="highlighter-rouge">attach_dbs</code> params, so I decided to copy over the code from the <a href="https://github.com/twright-msft/mssql-node-docker-demo-app/">mssql-node-docker-demo-app</a> and adapted it to <a href="https://github.com/brianweet/AlloyDemoKit/tree/docker-linux/Build-sql-linux">my needs</a>. After adding all of the data to the image, we’re left with just 1.73GB, a huge difference from the previous 10.8GB/15GB!</p>
<p class="centered-image">
<img src="/assets/mssql-linux/new-image-sizes.png" alt="New image sizes" />
</p>
<p>If you want to run Alloy demo using the linux container, use <a href="https://github.com/brianweet/AlloyDemoKit/blob/docker-linux/Run-using-pre-built-images-linux/docker-compose.yml">this docker-compose</a> file:</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">alloydemokit</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit:v2</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloydemokit-db</span>
<span class="na">alloydemokit-db</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit-db:linux</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="na">networks</span><span class="pi">:</span>
<span class="na">alloy-network</span><span class="pi">:</span>
<span class="na">external</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">nat</span>
</code></pre></div></div>
<p><strong>Note: you might have to pull the image in first using <code class="highlighter-rouge">docker pull --platform linux brianweet/alloydemokit-db:linux</code></strong></p>
<p class="centered-image">
<img src="/assets/mssql-linux/like-to-live-dangerously.jpg" alt="Like to live dangerously?" />
</p>
<p>If you’re feeling adventurous or you just don’t care about any laws of nature, common sense or just trust me on my blue/green/grey eyes, use my compiled dockerd.exe found <a href="/assets/mssql-linux/dockerd.rar">in this rar</a>.</p>
Adding custom fields to Business Foundation meta classeshttps://brianweet.github.io/2018/04/06/add-field-to-bf-class.html2018-04-06T14:00:00.0000000Z<p>A while ago I was working on a project and I was trying to add an extra field to the CustomerAddress meta class. For some reason I couldn’t find the right meta class to extend. Fortunately my friendly colleague <a href="https://www.patrickvankleef.com">Patrick</a> pointed out that I was looking at the wrong meta classes (MetaDataPlus instead of Business Foundation). A couple of days ago Patrick asked me how to add an extra field to the CustomerContact, which is a BF meta class as well. I see some repetition here, so I thought I write a small blog post about it.</p>
<p>We need a little bit of code, first we see an initialization module to create meta field, use <code class="highlighter-rouge">ContactEntity.ClassName</code> if you want to extend the CustomerContact:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[InitializableModule]</span>
<span class="na">[ModuleDependency(typeof(EPiServer.Commerce.Initialization.InitializationModule))]</span>
<span class="k">internal</span> <span class="k">class</span> <span class="nc">BusinessFoundationInitialization</span> <span class="p">:</span> <span class="n">IInitializableModule</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Initialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="nf">CreateMetaField</span><span class="p">(</span><span class="n">AddressEntity</span><span class="p">.</span><span class="n">ClassName</span><span class="p">,</span> <span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">,</span> <span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">void</span> <span class="nf">CreateMetaField</span><span class="p">(</span><span class="kt">string</span> <span class="n">metaClassName</span><span class="p">,</span> <span class="kt">string</span> <span class="n">metaFieldName</span><span class="p">,</span> <span class="kt">string</span> <span class="n">friendlyName</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">isNullable</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="kt">int</span> <span class="n">maxLength</span> <span class="p">=</span> <span class="m">255</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">isUnique</span> <span class="p">=</span> <span class="k">false</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">metaClass</span> <span class="p">=</span> <span class="nf">GetMetaClass</span><span class="p">(</span><span class="n">metaClassName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">metaClass</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">fieldExists</span> <span class="p">=</span> <span class="n">metaClass</span><span class="p">.</span><span class="n">Fields</span><span class="p">.</span><span class="nf">Contains</span><span class="p">(</span><span class="n">metaFieldName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">fieldExists</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">using</span><span class="p">(</span><span class="kt">var</span> <span class="n">metaFieldBuilder</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">MetaFieldBuilder</span><span class="p">(</span><span class="n">metaClass</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="n">MetaClass</span><span class="p">.</span><span class="n">AccessLevel</span> <span class="p">=</span> <span class="n">AccessLevel</span><span class="p">.</span><span class="n">Customization</span><span class="p">;</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="nf">CreateText</span><span class="p">(</span><span class="n">metaFieldName</span><span class="p">,</span> <span class="n">friendlyName</span><span class="p">,</span> <span class="n">isNullable</span><span class="p">,</span> <span class="n">maxLength</span><span class="p">,</span> <span class="n">isUnique</span><span class="p">);</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="nf">SaveChanges</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">private</span> <span class="n">MetaClass</span> <span class="nf">GetMetaClass</span><span class="p">(</span><span class="kt">string</span> <span class="n">metaClassName</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">DataContext</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="nf">GetMetaClass</span><span class="p">(</span><span class="n">metaClassName</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Uninitialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Constants</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">const</span> <span class="kt">string</span> <span class="n">AddressIdentifierFieldName</span> <span class="p">=</span> <span class="s">"CustomAddressIdentifier"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Once that is done; accessing the new property is straightforward:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">AddressExtensions</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">SetCustomIdentifier</span><span class="p">(</span><span class="k">this</span> <span class="n">CustomerAddress</span> <span class="n">customerAddress</span><span class="p">,</span> <span class="kt">string</span> <span class="n">customIdentifier</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">customerAddress</span><span class="p">[</span><span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">]</span> <span class="p">=</span> <span class="n">customIdentifier</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">static</span> <span class="kt">string</span> <span class="nf">GetCustomIdentifier</span><span class="p">(</span><span class="k">this</span> <span class="n">CustomerAddress</span> <span class="n">customerAddress</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">customerAddress</span><span class="p">[</span><span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">]?.</span><span class="nf">ToString</span><span class="p">()</span> <span class="p">??</span> <span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It’s as easy as that. Be sure to save your entity (e.g. CustomerContact) though. Otherwise the data will, obviously, not persist.</p>
Adding custom fields to Business Foundation meta classeshttps://www.brianweet.com/2018/04/06/add-field-to-bf-class.html2018-04-06T14:00:00.0000000Z<p>A while ago I was working on a project and I was trying to add an extra field to the CustomerAddress meta class. For some reason I couldn’t find the right meta class to extend. Fortunately my friendly colleague <a href="https://www.patrickvankleef.com">Patrick</a> pointed out that I was looking at the wrong meta classes (MetaDataPlus instead of Business Foundation). A couple of days ago Patrick asked me how to add an extra field to the CustomerContact, which is a BF meta class as well. I see some repetition here, so I thought I write a small blog post about it.</p>
<p>We need a little bit of code, first we see an initialization module to create meta field, use <code class="highlighter-rouge">ContactEntity.ClassName</code> if you want to extend the CustomerContact:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">[InitializableModule]</span>
<span class="na">[ModuleDependency(typeof(EPiServer.Commerce.Initialization.InitializationModule))]</span>
<span class="k">internal</span> <span class="k">class</span> <span class="nc">BusinessFoundationInitialization</span> <span class="p">:</span> <span class="n">IInitializableModule</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Initialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="nf">CreateMetaField</span><span class="p">(</span><span class="n">AddressEntity</span><span class="p">.</span><span class="n">ClassName</span><span class="p">,</span> <span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">,</span> <span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">void</span> <span class="nf">CreateMetaField</span><span class="p">(</span><span class="kt">string</span> <span class="n">metaClassName</span><span class="p">,</span> <span class="kt">string</span> <span class="n">metaFieldName</span><span class="p">,</span> <span class="kt">string</span> <span class="n">friendlyName</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">isNullable</span> <span class="p">=</span> <span class="k">true</span><span class="p">,</span> <span class="kt">int</span> <span class="n">maxLength</span> <span class="p">=</span> <span class="m">255</span><span class="p">,</span> <span class="kt">bool</span> <span class="n">isUnique</span> <span class="p">=</span> <span class="k">false</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">metaClass</span> <span class="p">=</span> <span class="nf">GetMetaClass</span><span class="p">(</span><span class="n">metaClassName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">metaClass</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">var</span> <span class="n">fieldExists</span> <span class="p">=</span> <span class="n">metaClass</span><span class="p">.</span><span class="n">Fields</span><span class="p">.</span><span class="nf">Contains</span><span class="p">(</span><span class="n">metaFieldName</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">fieldExists</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">using</span><span class="p">(</span><span class="kt">var</span> <span class="n">metaFieldBuilder</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">MetaFieldBuilder</span><span class="p">(</span><span class="n">metaClass</span><span class="p">))</span>
<span class="p">{</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="n">MetaClass</span><span class="p">.</span><span class="n">AccessLevel</span> <span class="p">=</span> <span class="n">AccessLevel</span><span class="p">.</span><span class="n">Customization</span><span class="p">;</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="nf">CreateText</span><span class="p">(</span><span class="n">metaFieldName</span><span class="p">,</span> <span class="n">friendlyName</span><span class="p">,</span> <span class="n">isNullable</span><span class="p">,</span> <span class="n">maxLength</span><span class="p">,</span> <span class="n">isUnique</span><span class="p">);</span>
<span class="n">metaFieldBuilder</span><span class="p">.</span><span class="nf">SaveChanges</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">private</span> <span class="n">MetaClass</span> <span class="nf">GetMetaClass</span><span class="p">(</span><span class="kt">string</span> <span class="n">metaClassName</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">DataContext</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="nf">GetMetaClass</span><span class="p">(</span><span class="n">metaClassName</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Uninitialize</span><span class="p">(</span><span class="n">InitializationEngine</span> <span class="n">context</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">Constants</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">const</span> <span class="kt">string</span> <span class="n">AddressIdentifierFieldName</span> <span class="p">=</span> <span class="s">"CustomAddressIdentifier"</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Once that is done; accessing the new property is straightforward:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">AddressExtensions</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">SetCustomIdentifier</span><span class="p">(</span><span class="k">this</span> <span class="n">CustomerAddress</span> <span class="n">customerAddress</span><span class="p">,</span> <span class="kt">string</span> <span class="n">customIdentifier</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">customerAddress</span><span class="p">[</span><span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">]</span> <span class="p">=</span> <span class="n">customIdentifier</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">static</span> <span class="kt">string</span> <span class="nf">GetCustomIdentifier</span><span class="p">(</span><span class="k">this</span> <span class="n">CustomerAddress</span> <span class="n">customerAddress</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">customerAddress</span><span class="p">[</span><span class="n">Constants</span><span class="p">.</span><span class="n">AddressIdentifierFieldName</span><span class="p">]?.</span><span class="nf">ToString</span><span class="p">()</span> <span class="p">??</span> <span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It’s as easy as that. Be sure to save your entity (e.g. CustomerContact) though. Otherwise the data will, obviously, not persist.</p>
Unstable Episerver Find developer (demo) indexeshttps://brianweet.github.io/2018/03/20/unstable-episerver-find-developer-indexes.html2018-03-20T09:00:00.0000000Z<p>Lately a lot of people had <a href="/link/a8fd428bfdd3486d92e2ffa92ba671ef.aspx">problems</a> with the free Episerver Find developer indexes (or so called demo indexes). Intermittently these indexes stopped working, sometimes for a short period but occasionally they seemed to have died completely. One solution was to create a new index, which gets old after creating a couple each day, especially as they don’t seem to work straight away.</p>
<p class="centered-image">
<img src="/assets/epi-find/wtf-grandma.jpg" alt="WTF grandma" />
</p>
<h4 id="the-problem">The problem</h4>
<p>I usually work on Commerce projects and it’s not possible to start the application without Find being available!
I decided to take a look at what’s causing it and what you can do to (temporarily) fix this problem. The error I’m getting is a “503 Service Unavailable” or a “(401) Unauthorized” which apparently means there’s something wrong with the index. That’s bad as is but it should not prevent our application from starting. Here are some examples of the error:</p>
<p class="centered-image">
<img src="/assets/epi-find/initialization-exception.png" alt="503 find initialization exception" />
</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[WebException: The remote server returned an error: (401) Unauthorized.]
System.Net.HttpWebRequest.GetResponse() +1695
EPiServer.Find.Connection.JsonRequest.GetResponseStream() +111
EPiServer.Find.Api.Command.GetResponse(IJsonRequest request) +42
[ServiceException: The remote server returned an error: (401) Unauthorized.
Unauthorized]
EPiServer.Find.Api.Command.GetResponse(IJsonRequest request) +63
EPiServer.Find.Api.PutMappingCommand.Execute() +37
EPiServer.Find.ClientConventions.NestedConventions.AddNestedType(Type declaringType, String name) +116
System.Collections.Generic.List`1.ForEach(Action`1 action) +14674174
EPiServer.Find.Commerce.CatalogContentClientConventions.ApplyNestedConventions(NestedConventions nestedConventions) +89
EPiServer.Find.Commerce.CatalogContentClientConventions.ApplyConventions(IClientConventions clientConventions) +69
EPiServer.Find.Commerce.FindCommerceInitializationModule.Initialize(InitializationEngine context) +47
EPiServer.Framework.Initialization.Internal.ModuleNode.Execute(Action a, String key) +55
EPiServer.Framework.Initialization.Internal.ModuleNode.Initialize(InitializationEngine context) +123
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +248
[InitializationException: Initialize action failed for Initialize on class EPiServer.Find.Commerce.FindCommerceInitializationModule, EPiServer.Find.Commerce, Version=10.1.1.0, Culture=neutral, PublicKeyToken=8fe83dea738b45b7]
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +834
EPiServer.Framework.Initialization.InitializationEngine.ExecuteTransition(Boolean continueTransitions) +198
EPiServer.Framework.Initialization.InitializationModule.EngineExecute(HostType hostType, Action`1 engineAction) +876
EPiServer.Framework.Initialization.InitializationModule.FrameworkInitialization(HostType hostType) +225
EPiServer.Global..ctor() +103
GetaCommerce.Web.Global..ctor() +43
ASP.global_asax..ctor() +48
</code></pre></div></div>
<p>If you examine this error, you might notice that the FindCommerceInitializationModule is trying to apply some nested conventions for the catalog content.</p>
<h4 id="workaround-1-using-fiddler">Workaround 1: Using Fiddler</h4>
<p>As I always have Fiddler running locally, the easiest way for me to is to open up Fiddler and check what’s happening. As expected, I notice the 503 response on a request that is trying to add nested conventions for commerce (for example for prices). I decided to add a rule to the Fiddler AutoResponder with the following data, which will allow our site to start:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
Content-Length: 31
{"ok":true,"acknowledged":true}
</code></pre></div></div>
<p class="centered-image">
<img src="/assets/epi-find/fake-response-fiddler.png" alt="Fake fiddler response" />
</p>
<p>It’s as easy as dragging and dropping the failing request to the AutoResponder rules and pointing it to a file with this simple response data.</p>
<h4 id="workaround-2-adding-an-interceptor">Workaround 2: Adding an interceptor</h4>
<p>This second workaround involves some code to intercept the call to CatalogContentClientConventions. It will allow you to catch the Exception that is being thrown and continue from there (as in, ignore and start the app already).
I’ve found this code on the <a href="/link/9d977f0d9ddb4cad8397f7a8c6f1a6a2.aspx">Forum</a>, see this <a href="https://gist.github.com/jstemerdink/6aff0f7de4aa22c803bb4ad0250bec0c">gist</a> by <a href="/link/56deea3d64d44757902494ada8830277.aspx">Jeroen Stemerdink</a>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">FindExceptionInterceptor</span> <span class="p">:</span> <span class="n">IInterceptor</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Intercept</span><span class="p">(</span><span class="n">IInvocation</span> <span class="n">invocation</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="n">invocation</span><span class="p">.</span><span class="nf">Proceed</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">ClientException</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">ServiceException</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">StructureMapRegistry</span> <span class="p">:</span> <span class="n">Registry</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">StructureMapRegistry</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">proxyGenerator</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ProxyGenerator</span><span class="p">();</span>
<span class="n">For</span><span class="p"><</span><span class="n">CatalogContentClientConventions</span><span class="p">>().</span><span class="n">Use</span><span class="p"><</span><span class="n">CatalogContentClientConventions</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">DecorateWith</span><span class="p">(</span><span class="n">t</span> <span class="p">=></span> <span class="n">proxyGenerator</span><span class="p">.</span><span class="nf">CreateClassProxyWithTarget</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="k">new</span> <span class="nf">FindExceptionInterceptor</span><span class="p">()));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="digging-a-little-deeper">Digging a little deeper</h4>
<p>As it turns out the FindCommerceInitializationModule actually has an appSetting <code class="highlighter-rouge">episerver:findcommerce.IgnoreWebExceptionOnInitialization</code> which sounds like it should do what we want: ignore exceptions upon initialization. There is a try catch around the code that throws the exception, however it only catches WebExceptions whereas the current exception is a ServiceException. I am going to create a support ticket for this, as I think we should be able to use this appSetting to make sure we can start our application even if Find is a bit flaky.</p>
<h4 id="conclusion">Conclusion</h4>
<p>It is possible to start your app when Episerver Find isn’t working properly. Be aware though, none of these workarounds are actual solutions and they might cause Find to behave unexpectedly.</p>
Unstable Episerver Find developer (demo) indexeshttps://www.brianweet.com/2018/03/20/unstable-episerver-find-developer-indexes.html2018-03-20T09:00:00.0000000Z<p>Lately a lot of people had <a href="/link/a8fd428bfdd3486d92e2ffa92ba671ef.aspx">problems</a> with the free Episerver Find developer indexes (or so called demo indexes). Intermittently these indexes stopped working, sometimes for a short period but occasionally they seemed to have died completely. One solution was to create a new index, which gets old after creating a couple each day, especially as they don’t seem to work straight away.</p>
<p class="centered-image">
<img src="/assets/epi-find/wtf-grandma.jpg" alt="WTF grandma" />
</p>
<h4 id="the-problem">The problem</h4>
<p>I usually work on Commerce projects and it’s not possible to start the application without Find being available!
I decided to take a look at what’s causing it and what you can do to (temporarily) fix this problem. The error I’m getting is a “503 Service Unavailable” or a “(401) Unauthorized” which apparently means there’s something wrong with the index. That’s bad as is but it should not prevent our application from starting. Here are some examples of the error:</p>
<p class="centered-image">
<img src="/assets/epi-find/initialization-exception.png" alt="503 find initialization exception" />
</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[WebException: The remote server returned an error: (401) Unauthorized.]
System.Net.HttpWebRequest.GetResponse() +1695
EPiServer.Find.Connection.JsonRequest.GetResponseStream() +111
EPiServer.Find.Api.Command.GetResponse(IJsonRequest request) +42
[ServiceException: The remote server returned an error: (401) Unauthorized.
Unauthorized]
EPiServer.Find.Api.Command.GetResponse(IJsonRequest request) +63
EPiServer.Find.Api.PutMappingCommand.Execute() +37
EPiServer.Find.ClientConventions.NestedConventions.AddNestedType(Type declaringType, String name) +116
System.Collections.Generic.List`1.ForEach(Action`1 action) +14674174
EPiServer.Find.Commerce.CatalogContentClientConventions.ApplyNestedConventions(NestedConventions nestedConventions) +89
EPiServer.Find.Commerce.CatalogContentClientConventions.ApplyConventions(IClientConventions clientConventions) +69
EPiServer.Find.Commerce.FindCommerceInitializationModule.Initialize(InitializationEngine context) +47
EPiServer.Framework.Initialization.Internal.ModuleNode.Execute(Action a, String key) +55
EPiServer.Framework.Initialization.Internal.ModuleNode.Initialize(InitializationEngine context) +123
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +248
[InitializationException: Initialize action failed for Initialize on class EPiServer.Find.Commerce.FindCommerceInitializationModule, EPiServer.Find.Commerce, Version=10.1.1.0, Culture=neutral, PublicKeyToken=8fe83dea738b45b7]
EPiServer.Framework.Initialization.InitializationEngine.InitializeModules() +834
EPiServer.Framework.Initialization.InitializationEngine.ExecuteTransition(Boolean continueTransitions) +198
EPiServer.Framework.Initialization.InitializationModule.EngineExecute(HostType hostType, Action`1 engineAction) +876
EPiServer.Framework.Initialization.InitializationModule.FrameworkInitialization(HostType hostType) +225
EPiServer.Global..ctor() +103
GetaCommerce.Web.Global..ctor() +43
ASP.global_asax..ctor() +48
</code></pre></div></div>
<p>If you examine this error, you might notice that the FindCommerceInitializationModule is trying to apply some nested conventions for the catalog content.</p>
<h4 id="workaround-1-using-fiddler">Workaround 1: Using Fiddler</h4>
<p>As I always have Fiddler running locally, the easiest way for me to is to open up Fiddler and check what’s happening. As expected, I notice the 503 response on a request that is trying to add nested conventions for commerce (for example for prices). I decided to add a rule to the Fiddler AutoResponder with the following data, which will allow our site to start:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
Content-Length: 31
{"ok":true,"acknowledged":true}
</code></pre></div></div>
<p class="centered-image">
<img src="/assets/epi-find/fake-response-fiddler.png" alt="Fake fiddler response" />
</p>
<p>It’s as easy as dragging and dropping the failing request to the AutoResponder rules and pointing it to a file with this simple response data.</p>
<h4 id="workaround-2-adding-an-interceptor">Workaround 2: Adding an interceptor</h4>
<p>This second workaround involves some code to intercept the call to CatalogContentClientConventions. It will allow you to catch the Exception that is being thrown and continue from there (as in, ignore and start the app already).
I’ve found this code on the <a href="/link/9d977f0d9ddb4cad8397f7a8c6f1a6a2.aspx">Forum</a>, see this <a href="https://gist.github.com/jstemerdink/6aff0f7de4aa22c803bb4ad0250bec0c">gist</a> by <a href="/link/56deea3d64d44757902494ada8830277.aspx">Jeroen Stemerdink</a>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">class</span> <span class="nc">FindExceptionInterceptor</span> <span class="p">:</span> <span class="n">IInterceptor</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Intercept</span><span class="p">(</span><span class="n">IInvocation</span> <span class="n">invocation</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="n">invocation</span><span class="p">.</span><span class="nf">Proceed</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">ClientException</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">ServiceException</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">StructureMapRegistry</span> <span class="p">:</span> <span class="n">Registry</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">StructureMapRegistry</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">proxyGenerator</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ProxyGenerator</span><span class="p">();</span>
<span class="n">For</span><span class="p"><</span><span class="n">CatalogContentClientConventions</span><span class="p">>().</span><span class="n">Use</span><span class="p"><</span><span class="n">CatalogContentClientConventions</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">DecorateWith</span><span class="p">(</span><span class="n">t</span> <span class="p">=></span> <span class="n">proxyGenerator</span><span class="p">.</span><span class="nf">CreateClassProxyWithTarget</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="k">new</span> <span class="nf">FindExceptionInterceptor</span><span class="p">()));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="digging-a-little-deeper">Digging a little deeper</h4>
<p>As it turns out the FindCommerceInitializationModule actually has an appSetting <code class="highlighter-rouge">episerver:findcommerce.IgnoreWebExceptionOnInitialization</code> which sounds like it should do what we want: ignore exceptions upon initialization. <strong>Unfortunately this does not work</strong>. There is a try catch around the code that throws the exception, however it only catches WebExceptions whereas the current exception is a ServiceException. I am going to create a support ticket for this, as I think we should be able to use this appSetting to make sure we can start our application even if Find is a bit flaky.</p>
<h4 id="conclusion">Conclusion</h4>
<p>It is possible to start your app when Episerver Find isn’t working properly. Be aware though, none of these workarounds are actual solutions and they might cause Find to behave unexpectedly.</p>
Trying out Docker: Containerized alloy demo kithttps://brianweet.github.io/2017/10/09/alloy-in-docker.html2017-10-09T14:00:00.0000000Z<blockquote>
<h4 id="tldr">tl;dr</h4>
<ul>
<li>Running an existing Episerver demo project (AlloyDemoKit) in two Docker containers</li>
<li>Creating self-sufficient images for demo or test purposes</li>
<li>Code can be found <a href="https://github.com/brianweet/AlloyDemoKit/tree/docker">here</a></li>
</ul>
</blockquote>
<p>In this blog I’ll be taking another look at <a href="https://www.docker.com/">Docker</a>. The aim is to get the <a href="https://github.com/episerver/AlloyDemoKit">AlloyDemoKit</a> reference site up and running by using Docker containers. I will run the demo solution in two separate containers, one for the site and one for SQL server. There are two purposes: the first purpose is to show how to set up AlloyDemoKit while using Docker configuration files to provide a consistent experience for every developer checking out the project. The second goal is to have a (fast) way of spinning up and spinning down demo- or test environments in a pre-defined state.</p>
<p class="centered-image">
<img src="/assets/docker-blog-1/docker-logo.png" alt="Cute Docker logo" />
</p>
<h3 id="running-alloy-in-a-container">Running alloy in a container</h3>
<p>To start off, I’ve installed <a href="https://marketplace.visualstudio.com/items?itemName=MicrosoftCloudExplorer.VisualStudioToolsforDocker-Preview">VS tools for Docker</a> as this provides the option to add Docker support from Visual Studio. When you click on “Add - Docker Support”, a couple of files will be added for you. Let’s take a look to see what these files are for:</p>
<ul>
<li>There’s a Dockerfile for our web project (in the alloydemo folder)
<ul>
<li>This is a description of how to build an image for our web project. After building we will have an image called alloydemokit. The Dockerfile has a reference to a Microsoft image which includes IIS and adds remote debugging support to that image</li>
</ul>
</li>
<li>There are multiple docker-compose*.yml file(s)
<ul>
<li>Configuration for setting up complete environments, with one or multiple ‘services’. It is used to configure things like port mappings, volume shares etc.</li>
</ul>
</li>
<li>There’s a Docker project (.dcproj)
<ul>
<li>It contains the docker-compose file(s) and is used as startup project to run our project in Docker</li>
</ul>
</li>
</ul>
<p class="centered-image">
<img src="/assets/docker-blog-2/1.run-with-docker.png" alt="Docker run" />
</p>
<p>If you run the project it will not run in IIS or IIS Express anymore, you’ll notice that the run button now says “Docker”. A couple of things will happen once you run your project: it builds the Docker image(s), uses the image(s) to start container(s), hooks up debug tools and opens up the browser with the correct url. All of this can also be done from the command line and you can see what is happening in your output window, so no magic here. As we have to use Windows containers, it will take quite a while to build the project for the first time; the Windows images are very big. Where running a linux image takes a matter of seconds, running a new Windows container takes well over 10 minutes. Fortunately this will only happen the first time (or when you explicitly want to update to a different Windows image version) so be patient and make sure that you have plenty of disk space.</p>
<p>Our project now builds an image called <code class="highlighter-rouge">alloydemokit</code> for our web project, using the default Dockerfile generated by “Add Docker support”. Here’s some info from the output window:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>1>Building alloydemokit
1>Step 1/4 : FROM microsoft/aspnet:4.6.2
1> ---> 8352eb08cfc4
1>Step 2/4 : ARG source
1> ---> Running in 18a750e96125
1> ---> 1a574005157b
1>Removing intermediate container 18a750e96125
1>Step 3/4 : WORKDIR /inetpub/wwwroot
1> ---> f55e03a33d15
1>Removing intermediate container bf30af72d81a
1>Step 4/4 : COPY ${source:-obj/Docker/publish} .
1> ---> 7fe4fbdaf958
1>Removing intermediate container 1fc3a286aa08
1>Successfully built 7fe4fbdaf958
1>Successfully tagged alloydemokit:dev
</code></pre>
</div>
<p>Every command creates a layer which will be cached, therefore it only has to be re-executed when changes are made. As seen in the output log, our <code class="highlighter-rouge">alloydemokit</code> has a base layer using an image built by Microsoft, and three extra layers, needed to run our project from VS.</p>
<p>All changes can be found in my fork of AlloyDemoKit in the <a href="https://github.com/brianweet/AlloyDemoKit/tree/docker">docker branch</a>.</p>
<h4 id="running-sql-in-docker">Running SQL in docker</h4>
<p><a href="https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=microsoft%2Fmssql&starCount=0">Microsoft provides images</a> that can be used in order to run SQL in a container. The documentation is quite clear so it’s easy to get an instance up and running. The image has a couple of (required) parameters, you have to accept the EULA, set a <code class="highlighter-rouge">sa</code> password and you can set up databases immediately after starting (remember the image does not have any data in it by default). Below you can see the docker-compose file I ended up with. I’ve included comments to describe what every line is doing.</p>
<div class="language-yml highlighter-rouge"><pre class="highlight"><code><span class="s">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="s">services</span><span class="pi">:</span>
<span class="s">alloydemokit</span><span class="pi">:</span> <span class="c1"># Service for our web project</span>
<span class="s">image</span><span class="pi">:</span> <span class="s">alloydemokit</span> <span class="c1"># Image name</span>
<span class="s">build</span><span class="pi">:</span> <span class="c1"># Info needed to build the image</span>
<span class="s">context</span><span class="pi">:</span> <span class="s">.\AlloyDemoKit</span>
<span class="s">dockerfile</span><span class="pi">:</span> <span class="s">Dockerfile</span>
<span class="s">networks</span><span class="pi">:</span> <span class="c1"># We'll be using a network I've defined below called alloy-network</span>
<span class="pi">-</span> <span class="s">alloy-network</span> <span class="c1"># This allows communication between containers based on the service name</span>
<span class="s">depends_on</span><span class="pi">:</span> <span class="c1"># the db is required</span>
<span class="pi">-</span> <span class="s">alloydemokit-db</span>
<span class="s">alloydemokit-db</span><span class="pi">:</span>
<span class="s">image</span><span class="pi">:</span> <span class="s">microsoft/mssql-server-windows-developer</span> <span class="c1"># Empty sql server instance</span>
<span class="s">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">1433:1433"</span>
<span class="s">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ACCEPT_EULA=Y</span>
<span class="pi">-</span> <span class="s">sa_password=All0yDemokit!</span> <span class="c1"># This sets the sa account password</span>
<span class="pi">-</span> <span class="s">attach_dbs="[{'dbName':'alloydemokit','dbFiles':['C:\\data\\alloydemokit.mdf','C:\\data\\alloydemokit_log.ldf']}]"</span>
<span class="c1"># Attach dbs will attach the database files once the server has started</span>
<span class="s">volumes</span><span class="pi">:</span> <span class="c1"># This will map the App_Data folder on my PC to c:/data in the container (R/W)</span>
<span class="pi">-</span> <span class="s">./AlloyDemoKit/App_Data:C:/data/</span>
<span class="s">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="s">networks</span><span class="pi">:</span> <span class="c1"># Used to redirect traffic and make communication between containers possible</span>
<span class="s">alloy-network</span><span class="pi">:</span>
<span class="s">external</span><span class="pi">:</span>
<span class="s">name</span><span class="pi">:</span> <span class="s">nat</span>
</code></pre>
</div>
<p>Our <code class="highlighter-rouge">alloydemokit</code> service needs to be able to access the container that’s running the database. With the set-up as shown above, it’s quite easy. As we’re using Docker compose; other services can be reached by using the service name as hostname <code class="highlighter-rouge">Data Source=alloydemokit-db;Initial Catalog=alloydemokit;Integrated Security=False;User ID=sa;Password=All0yDemokit!</code> (<a href="https://github.com/brianweet/AlloyDemoKit/commit/265dd3d18cf958abdf27c2760ca3b3e72ff6bb05#diff-0d9151933f32e2929ddc7906ed378fbdR553">see full diff on github</a>).</p>
<p>If you want, it’s possible to use an explicit SQL version by adding a tag behind the image name, for example <code class="highlighter-rouge">:2016-sp1</code> or <code class="highlighter-rouge">2017</code>. By default it will use the <code class="highlighter-rouge">:latest</code> tag.</p>
<p>In the diff you can see that I had to add the .net compilers/compilerplatform nuget packages. I forgot what the actual error was, but I think it couldn’t compile razor views.</p>
<h4 id="fixing-a-problem-with-the-environment-lack-of-websocket-support">Fixing a problem with the environment: lack of websocket support</h4>
<p>Alloy was up and running in two Docker containers and it’s using the most SQL server version which is not installed on my local machine! However, after logging in, I noticed an error about real-time updates, which you can see in the image below. I’ve seen this error before and I know it has something to do with websockets, perhaps the image I’m using (microsoft/aspnet:4.6.2) does not have websocket support. What to do?!</p>
<p class="centered-image">
<img src="/assets/docker-blog-2/websocket-error.png" alt="Websocket error" />
</p>
<p>It’s actually quite easy to add extra layers (of changes/data) to existing images, this I figured out in my <a href="/2017/09/17/trying-out-docker-build-jekyll-blog.html">previous blog post</a> about Docker. After a quick search on Google I found out that you can add Windows features in powershell using this cmdlet <code class="highlighter-rouge">Add-WindowsFeature Web-WebSockets</code>, and indeed, after adding a <code class="highlighter-rouge">RUN</code> command to the <a href="https://github.com/brianweet/AlloyDemoKit/commit/dd71127447c78caad5edc1ed1addc945c66c3a37">Dockerfile</a>, the project rebuilt successfully and the error was gone!</p>
<p>We now have a Dockerfile for our alloydemokit project which has all the features we need in order to run our project. Next to the Dockerfile we have an up-to-date SQL instance for this project, an instance we can spin up and down whenever we want and we can also update to a new version, simply by changing the <a href="https://hub.docker.com/r/microsoft/mssql-server-windows-developer/tags/">tag for the mssql image</a> in the docker-compose file.</p>
<h3 id="pre-built-images-for-demo-environments-and-automated-testing">Pre-built images for demo environments and automated testing</h3>
<p>Let’s focus on something different, let’s to try to create ‘self-sufficient’ images which will allow us to spin up complete environments for demo or automated testing purposes. These images will contain both the complete environment and all data required to run the alloy demo. This means that your complete environment, including the db, will be in a known state when you start the containers. When you stop the containers, all data will be gone unless you take explicit actions to keep the data. In general you should not store data inside your images if it’s not necessary, however including data inside an image can be very useful for demo/test purposes. You will always have a known state to start with, sounds great right?</p>
<p>Start with the web project, change the build configuration to release and try to run the project. After building and running I got a couple of issues regarding file permissions, as this is just a demo <a href="https://github.com/brianweet/AlloyDemoKit/commit/4cf6ac81e209a612cafdc641a57aebe0717d360f#diff-c36ee58e01fc5f7d07dd824b226b433bR6">I granted everyone full access</a>. After that I noticed I was missing the appdata so I also included that in my <a href="https://github.com/brianweet/AlloyDemoKit/commit/4cf6ac81e209a612cafdc641a57aebe0717d360f#diff-fd43246249295c88b78100e38e8efb25R671">csproj</a>.
That was actually all I had to do, after running the project and checking everything is okay, I had a <code class="highlighter-rouge">alloydemokit</code> image which contained all data that’s needed to run the project.</p>
<p>We’ll also need the database inside an image. Therefore I created a <a href="https://github.com/brianweet/AlloyDemoKit/blob/docker/Build-sql/Dockerfile">Dockerfile</a> to create an image for the database. To build the image I ran <code class="highlighter-rouge">docker build . -t alloydemokit-db:test</code>. This creates an image based on the <code class="highlighter-rouge">microsoft/mssql-server-windows-developer</code> image and copies in the two db files.</p>
<p>The images can be pushed to the <a href="https://hub.docker.com/">Docker Hub</a> which is a free Docker registry. Pushing an image is also <a href="https://docs.docker.com/docker-cloud/builds/push-images/">really easy</a>, just tag the image with your username and push the image to the repository:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>docker tag alloydemokit-db brianweet/alloydemokit-db:test
docker push brianweet/alloydemokit-db:test
</code></pre>
</div>
<h4 id="running-the-pre-built-images">Running the pre-built images</h4>
<p>If you have Docker installed locally you can try to run the demo/test images I built on alloy. Create a <a href="https://github.com/brianweet/AlloyDemoKit/blob/docker/Run-using-pre-built-images/docker-compose.yml">docker-compose.yml</a> file with the following content (as mentioned before, it can take quite a while to run this for the first time):</p>
<div class="language-yml highlighter-rouge"><pre class="highlight"><code><span class="s">version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">3'</span>
<span class="s">services</span><span class="pi">:</span>
<span class="s">alloydemokit</span><span class="pi">:</span>
<span class="s">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit:test</span> <span class="c1"># Use pre built image, uploaded to Docker Hub</span>
<span class="s">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">80:80"</span>
<span class="s">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="s">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloydemokit-db</span>
<span class="s">alloydemokit-db</span><span class="pi">:</span>
<span class="s">image</span><span class="pi">:</span> <span class="s">brianweet/alloydemokit-db:test</span>
<span class="s">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">1433:1433"</span>
<span class="s">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">ACCEPT_EULA=Y</span>
<span class="pi">-</span> <span class="s">sa_password=All0yDemokit!</span>
<span class="pi">-</span> <span class="s">attach_dbs="[{'dbName':'alloydemokit','dbFiles':['C:\\alloydemokit.mdf','C:\\alloydemokit_log.ldf']}]"</span>
<span class="s">networks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">alloy-network</span>
<span class="s">networks</span><span class="pi">:</span>
<span class="s">alloy-network</span><span class="pi">:</span>
<span class="s">external</span><span class="pi">:</span>
<span class="s">name</span><span class="pi">:</span> <span class="s">nat</span>
</code></pre>
</div>
<p>Now run <code class="highlighter-rouge">docker-compose up -d</code> to start the two containers to run alloy in Docker. If everything succeeds, you should be able to browse to the <a href="https://docs.docker.com/engine/reference/commandline/inspect/#get-an-instances-ip-address">container instance ip</a> and see exactly the same content as you see in this image:</p>
<p class="centered-image">
<img src="/assets/docker-blog-2/from-docker-hub.png" alt="Using pre-built alloy images with data" />
</p>
<h3 id="summary">Summary</h3>
<p>For this blog, I had two things I wanted to try out while using Docker.
First I wanted to convert an existing Episerver demo project to use Docker, as it should result in a consistent behaviour of our project across environments. Next to being consistent, it will also give us flexibility regarding the software that is being used (SQL server version, enabled windows features). From a developer’s perspective; using Docker can eliminate start-up problems when checking out a project for the first time. Configuration for the environments is explicit and we can move dependencies and requirements on the runtime environment to the container.
The second idea was to created ‘self-sufficient’ images, which are available on the Docker hub, in order to start and stop complete environments without any hassle. The images I’ve built therefore contain all data that is necessary to run the site and they are perfect for demo or automated testing environments. With one docker-compose file and by running one command you can now start up the alloy demo kit in a predefined state.</p>