Can you try running this code against your solution. Anywhere will work.

var storeDefinition = StoreDefinition.Get(DynamicDataStoreFactory.Instance.GetStoreNameForType(typeof(ComponentData)));
storeDefinition.Remap(typeof(ComponentData));
storeDefinition.CommitChanges();

Here is what I ran with no luck....

var errors = new List<string>();
            foreach (var storeDefinition in StoreDefinition.GetAll())
            {
                try
                {
                    var type = TypeResolver.GetType(storeDefinition.StoreName);
                    if (type != null)
                    {
                        storeDefinition.Remap(type);
                        storeDefinition.CommitChanges();
                    }
                    else
                    {
                        var provider = _dataStoreProviderFactory.Create();
                        provider.SaveStoreDefinition(storeDefinition);
                    }
                }
                catch (Exception ex)
                {
                    errors.Add($"Error remapping '{storeDefinition.StoreName}': {ex.Message} {ex.StackTrace}");
                }
            }

Also, we see the variants in Commerce Manager UI, just not the UI in the screenshot above

Scott, what should I try adding the web.config?

Where am I adding <episerver.dataStore> to web.config?

Configuration - episerver.dataStore Section | Optimizely Develope

episerver.dataStore should be at the same level as episerver.web etc. However I'm not quite sure about your problem. Where do you see that exception? If it happens it usually means your site has YSOD, not just "variants not showing" 

So we only saw the error once in our logs, however, you're correct, Quan.  You'd expect to see a YSOD or at least the error being logged all the time.  It was only logged once.  Maybe the error is not related to this problem.  The problem still remains.  No variants are showing up in the new catalog UI, but do under the legacy Commerce Manager UI.

Any tips as to where to look would be much appreciated.  If not, we're about to open a ticket with support.

Could you see any problem in your browser console?

No nothing, all XHR requests return 200OK and no js errors.  I did not see any request when switching to the Variants tab, im guessing everything is loaded on catalog node click.  

I see requests for catalog content like the ones below... is there anything I should look for in particular?

/AMEPiServer/cms/Stores/contentstructure/48746__CatalogContent?dojo.preventCache=1628259017876

/AMEPiServer/cms/Stores/contentstructure/?references=1073755763__CatalogContent&query=getpagedchildren&allLanguages=true&typeIdentifiers=episerver.commerce.catalog.contenttypes.productcontent&typeIdentifiers=episerver.commerce.catalog.contenttypes.variationcontent&typeIdentifiers=episerver.commerce.catalog.contenttypes.packagecontent&typeIdentifiers=episerver.commerce.catalog.contenttypes.bundlecontent&sort()&dojo.preventCache=1628259017675

You should be able to get the data from

<your-site>/AMEPiServer/EPiServer.Commerce.Shell/Stores/relation/?referenceId=<your-product-content-link>&relationTypes=4

I don't see any request to that endpoint.... here are all the requests i see when clicking on a product.  I did not include query strings.

/AMEPiServer/cms/Stores/contentstructure

/AMEPiServer/shell/Stores/context

/AMEPiServer/cms/Stores/contentdata/62530_85995964_CatalogContent

/AMEPiServer/cms/Stores/contentversion/62530_85995964_CatalogContent

/AMEPiServer/EPiServer.Marketing.Testing/Stores/ABTestConfigStore

/AMEPiServer/shell/Stores/metadata/EPiServer.Core.ContentData

/AMEPiServer/cms/Stores/project-item

/AMEPiServer/cms/Stores/language

/AMEPiServer/cms/Stores/contentstructure (it looks like this has variants in its response, however the UI must not be using it)

/AMEPiServer/cms/Stores/contentstructure/62530__CatalogContent

I guess that is the problem. Unless we can closely investigate your site it is hard to say why. Did you contact developer support service?