Include/IncludeOn/Exclude/ExcludeOn: a simple explanation

When I come across this question https://world.episerver.com/forum/developer-forum/-Episerver-75-CMS/Thread-Container/2020/3/trouble-with-availablecontenttypesattribute-excludeonincludeon/ I was rather confused by the properties of AvailableContentTypesAttribute (admittedly I don’t use them that often!). Looking at the code that defined them, or the XML documentation does not really help. I only come to an understanding when I look into how they are used, and I guess many other developers, especially beginners, might have same confusion, so here’s a simple explanation.

Include : defines content types that can be created as children of a content of this type (being decorated by the attribute)

IncludeOn: defines content types that can be parent of a content of this type

Exclude: defines content types that can not be created as children of a content of this type

ExcludeOn: defines content types that can not be parent of a content of this type.

If there is a conflict between those properties, for example content type A has Include with content type B, and content type B has ExcludeOn with content type A, then Exclude and ExcludeOn take priority (i.e. they will override Include and IncludeOn. In the example above then content type B will not be able to be children of content type A)

While AvailableContentTypesAttribute is extremely helpful, the property naming is not the best – they are short and symmetric, but they are not easy to understand and remember. An “improved” example might be

CanBeParentOf

CanBeChildrenOf

CannotBeParentOf

CannotBeChildrenOf

Yes they are more verbose, but they are unambiguous and you will not have to check the document (or this blog post) when you use them.

This is not the first time we have something that rather confusing in our API. One notable example is the old (now removed) ILinksRepository with the Source and Target properties in Relation . For quite some time I always had to check the code to know what to use, and then had the documentation updated, and eventually, changed to Parent and Child. No API is created perfect, but we can improve over time.

Disabling Catalog Dto cache: maybe, don’t?

Recently (as recent as this morning) I was asked to look into a case when the Find indexing performance was subpar. Upon investigation – looking at a properly captured trace from dotTrace – it was clear that at least 30% percent of time was spending in loading the CatalogDto

This is something that should not happen, as the CatalogDto should have been cached. Also, a normal site should have very few catalogs, so the cache should be very effective. However, data does not lie – it has been hitting database a lot, and a quick check on the site settings revealed that the entire DTO cache has been indeed, disabled

 <Cache enabled="false" collectionTimeout="0:0:0" entryTimeout="0:0:0" nodeTimeout="0:0:0" schemaTimeout="0:0:0" /> 

By setting these timeout settings to 0, the cache is immediately invalidated, rendering them useless. The CatalogDto, therefore, is loaded everytime from database, causing the bottleneck.

The reason for setting those timeout to 0 was probably – I guess – to reduce the memory footprint of the site. However, Catalog DTOs are fairly small in size, and since Commerce 11, it has been smart enough to skip caching the DTOs if there is cache on a higher (content ) level, thanks to my colleague Magnus Rahl. So DTOs should not be of any concerns, if you are not actively using them (and in most of the case, you should not). By re-enabling the cache, the indexing time can be cut, at least 30%, according to the aforementioned trace.

As you might wonder, Catalog content provider still uses the DTOs internally, therefore it would load those for data.

Moral of the story:

  • The cache settings are there, but because you can, does not mean you should. I personally think cache settings should be as hidden as possible from accidental changes. Disabling cache, and in a lesser extend, changing default cache timeout, can have unforeseeable consequences. Only do so if you have strong reasons to do so. Or better, let us know why you need to do that, and we can figure out a compromise.

Hide certain tabs in Catalog UI

It has been a while since I write something in my blog – have been “fairly” busy making Commerce even faster for a while. But I should take a break from time to time and share things that will benefit community as a whole – and this is one of that break.

Today I come across this question on World https://world.episerver.com/forum/developer-forum/Episerver-Commerce/Thread-Container/2019/10/remove-item-from-tab-in-content-editor/ . Basically, how to hide a specific tab in the Catalog UI when you open All Properties view of a catalog content.

The original poster has found a solution from https://world.episerver.com/forum/legacy-forums/Episerver-7-CMS/Thread-Container/2013/10/Is-there-any-way-to-hide-the-settings-tab/ . While it works, I think it is not the easiest or simple way to do it. Is there a simpler way? Yes.

The Related Entries tab is generated for content with implements IAssociating interface. Bad news is EntryContentBase implements that interface, so each and every entry type you have, has that tab. But good news is we can override the implementation – by just override the property defined by IAssociating.

How?

Simple as this

        /// <inheritdoc />
        [IgnoreMetaDataPlusSynchronization]
        [Display(AutoGenerateField = false)]
        public override Associations Associations { get; set; }

We are overriding the Associations property, and the change the Display attribute to have AutoGenerateField = false. Just try to build it and see

No Related Views! But is it the end of the story. Not yet, Related Views can still be accessed by the menu

A complete solution is to also disable that view. How? By using the same technique here https://world.episerver.com/blogs/Quan-Mai/Dates/2019/8/enable-sticky-mode-for-catalog-content/ i.e. using `UIDescriptor`. You can disable certain views by adding this to your constructor

AddDisabledView(CommerceViewConfiguration.RelatedEntriesEditViewName);

A few notes:

  • This only affects the type you add the property, so for example you can hide the tab for Products, but still show it for Variants.
  • Related Entries is not the only tab you can hide. By applying the same technique you can have a lot of control over what you can hide, and what you show. I will leave that to you for exploration!

Listing permissions per user/group

This week I came cross this question on Episerver World forum https://world.episerver.com/forum/developer-forum/Episerver-Commerce/Thread-Container/2019/5/get-rolepermission-data/ , and while it is not Commerce-related. it is quite interesting to solve. Perhaps this short post will help the original poster, as well future visitors.

As in the thread, I replied the first piece to solve the puzzle:


You can use PermissionTypeRepository to get the registered PermissionTypes, then PermissionRepository to figure out which groups/users have a specific permission 

If you want to list permissions granted to a specific role or user, it is just a simple reversion using a dictionary:

            var rolePermissionMap = new Dictionary<string, HashSet<PermissionType>>(StringComparer.OrdinalIgnoreCase);
            var permissionTypes = _permissionTypeRepository.List();
            foreach (var permissionType in permissionTypes)
            {
                var securityEntities = _permissionRepository.GetPermissions(permissionType);
                foreach (var securityEntity in securityEntities)
                {
                    if (rolePermissionMap.ContainsKey(securityEntity.Name))
                    {
                        rolePermissionMap[securityEntity.Name].Add(permissionType);
                    }
                    else
                    {
                        rolePermissionMap[securityEntity.Name] = new HashSet<PermissionType>() { permissionType };
                    }
                }
            }

As suggested above, we use
PermissionTypeRepository to list the registered PermissionType(s) , and then for each PermissionType we get the list of SecurityEntity it is granted for. A SecurityEntity can be an user, a group, or a virtual role, and is identified by the name. For purpose of demonstration, we only use names: For each SecurityEntity granted a permission, we check if it is in our dictionary already, if yes, then add the permission to the list, otherwise add a new entry.

Simple, eh?

Unless if you are assigning/un-assigning permissions a lot, it is probably a good idea to keep this Dictionary in cache for some time, because it is not exactly cheap to build.

IContentLoader.Get(contentLink) is considered harmful for catalog content.

A while ago I wrote about how you should be aware of IContentLoader.GetChildren<T>(contentLink) here. However, that is only half of story.

IContentLoader.Get<T>(contentLink) is also considered harmful. Not in terms of it causes damage to your site (we would never, ever let that happen), nor it is slow (not unless you abuse it), but because it can behave very unexpectedly.

As you might already know, catalog content fully supports language versions, which means a catalog might have multiple languages enabled, and each and every catalog item in that catalog (node/category, and entry) will be available in those languages. However, those languages are not equal, (only) one is master language. What’s the difference then?

One of very important characteristics of that is how it affects the properties. Properties with [CultureSpecific] attribute decorated will be different in each language, and therefore, can be edited in each language. Properties without [CultureSpecific] attribute decorated will be the same in all languages, and can only be edited in master language. In Catalog UI, if you switch to non master languages, those properties will be grayed out, indicating they can’t be edited.

Now, why IContentLoader.Get<T>(contentLink) is considered harmful? Because you don’t supply a CultureInfo to let it know which version you want, it relies on the current preferred language to load the content. And if you have a catalog which has master language that is different from the current preferred language, you are loading a non-master language version. And then if you try to edit a non [CultureSpecific] property, then save it, the changes will not be saved, without error or warning.

It then will be very confusing because it sometimes works (someone changes the current preferred language that matches the catalog master language, and sometimes it doesn’t.

Which can cost you hours, if not days, to figure out what is wrong with your code.

Same thing applies to IContentLoader.TryGet<T>(contentLink)

Solution? Always use the overload that takes a CultureInfo or a LoaderOptions parameter, even if you just want to read the content. That creates a “good” habit and you can quickly spot code that might be problematic.

Use this to load master language version, if you wish to update some non CultureSpecific property.

 new LoaderOptions() { LanguageLoaderOption.MasterLanguage() }

Later versions of Commerce will log a warning if you are trying to save a non master language version with one or more changed non [CultureSpecific]properties.

Control the thousand separator for Money in Episerver Commerce

If you are selling goods in multiple markets which same currency but with different languages, such as EuroZone, you might notice that while everything looks quite good, except that the thousand separator might be off from time to time: it is always the same and does not change to match with the language, so sometimes it’s correct, sometimes it’s not.

Let’s take a step back to see how to properly show the thousand delimiter 

In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space.

https://docs.microsoft.com/en-us/globalization/locale/number-formatting

You might ask why the problem happens with Episerver Commerce. In Commerce, each currency has an attached NumberFormatInfo which let the framework knows how to format the currency. During startup, the system will loop through the available CultureInfo and assign its .NumberFormat to the currency.

The problem is there might be multiple CultureInfo that can handle same currency, for example, EUR which is used across Eurozone, can be handled by multiple (20? ) cultures. However, the first matching CultureInfo to handle the format of the currency will be used. In most of the cases, it will be br-FR (because the CultureInfo(s) are sorted by name, and this CultureInfo is the first in the list to handle EUR)

br-FR does not have a thousand separator, but a whitespace. That’s why even if your language is de-DE, the amount in EUR will not be properly formatted as 1.234,45 but 1 234,45

How to fix that problem?

Luckily, we can set the NumberFormatInfo attached for each currency. If you are only selling in Germany, you can make sure that EUR is always formatted in German style, by adding this to one of your initialization modules:

var culture = CultureInfo.GetCultureInfo("de-DE");
Currency.SetFormat("EUR", culture.NumberFormat);

But if you have multiple languages for one currency, this will simply not work (because it’s static, so it will affect all customer). Your only option is to avoid using Money.ToString(), but to use Money.ToString(IFormatProvider), for example

money.ToString(CultureInfo.CurrentUICulture);

Assuming CultureInfo.CurrentUiCulture is set to correct one.

This, however, does not resolve the problem with merchandisers using Commerce Manager. They might have to work with orders from multiple markets, and for example, if your site is selling good stuffs in Europe, there are chances that merchandisers see the prices without correct thousand separator. Most of places in Commerce Manager uses Money.ToString(), and there is a reason for that: it’s too risky to use Money.ToString(CultureInfo.CurrentUICulture), because if a merchandiser uses English, he or she is likely gonna see money formatted as “$” instead of “€”, and that is a much bigger problem of itself.

Moral of the story: localization is hard, and sometimes a compromise is needed.

Speed up catalog routing if you have multiple children under catalog

A normal catalog structure is like this: you have a few high level categories under the catalog, then each high level category has a few lower level categories under it, then each lower level category has their children, so on and so forth until you reach the leaves – catalog entries.

However it is not uncommon that you have multiple children (categories and entries) directly under catalog. Even though that is not something you should do, it happens. 

But that is not without drawbacks. You might notice it is slow to route to a product. It might not be visible to naked eyes, but if you use some decent profilers (which I personally recommend dotTrace), it can be fairly obvious that your site is not routing optimally.

Why?

To route to a specific catalog content, for example http://commerceref/en/fashion/mens/mens-shirts/p-39101253/, the default router have to figure out which content is mapped to an url segment. So with default registration where the catalog root is the default routing root, we will start with the catalog which maps to the first part of route (fashion ). How do it figure out which content to route for the next part (mens ) ? 

Until recently, what it does it to call GetChildren on the catalog ContentReference . Now you can see the problem. Even with a cached result, that is still too much – GetChildren with a big number of children is definitely expensive.

We noticed this behavior, thanks to Erik Norberg. An improvement have been made in Commerce 12.10 to make sure even with a number of children directly under Catalog, the router should perform adequately efficient.

If you can’t upgrade to 12.10 or later (you should!), then you might have a workaround that improve the performance. By adding your own implementation of HierarchicalCatalogPartialRouter, you can override how you would get the children content – by using a more lightweight method (GetBySegment)

    public class CustomHierarchicalCatalogPartialRouter : HierarchicalCatalogPartialRouter
    {
        private readonly IContentLoader _contentLoader;

        public CustomHierarchicalCatalogPartialRouter(Func<ContentReference> routeStartingPoint, CatalogContentBase commerceRoot, bool enableOutgoingSeoUri) : base(routeStartingPoint, commerceRoot, enableOutgoingSeoUri)
        {
        }

        public CustomHierarchicalCatalogPartialRouter(Func<ContentReference> routeStartingPoint, CatalogContentBase commerceRoot, bool supportSeoUri, IContentLoader contentLoader, IRoutingSegmentLoader routingSegmentLoader, IContentVersionRepository contentVersionRepository, IUrlSegmentRouter urlSegmentRouter, IContentLanguageSettingsHandler contentLanguageSettingsHandler, ServiceAccessor<HttpContextBase> httpContextAccessor) : base(routeStartingPoint, commerceRoot, supportSeoUri, contentLoader, routingSegmentLoader, contentVersionRepository, urlSegmentRouter, contentLanguageSettingsHandler, httpContextAccessor)
        {
            _contentLoader = contentLoader;
        }

        protected override CatalogContentBase FindNextContentInSegmentPair(CatalogContentBase catalogContent, SegmentPair segmentPair, SegmentContext segmentContext, CultureInfo cultureInfo)
        {
            return _contentLoader.GetBySegment(catalogContent.ContentLink, segmentPair.Next, cultureInfo) as CatalogContentBase;
        }
    }

And then instead of using CatalogRouteHelper.MapDefaultHierarchialRouter , you register your router directly

 var referenceConverter = ServiceLocator.Current.GetInstance<ReferenceConverter>();
            var contentLoader = ServiceLocator.Current.GetInstance<IContentLoader>();
            var commerceRootContent = contentLoader.Get<CatalogContentBase>(referenceConverter.GetRootLink());
            routes.RegisterPartialRouter(new HierarchicalCatalogPartialRouter(startingPoint, commerceRootContent, enableOutgoingSeoUri));

(ServiceLocator is just to make it easier to understand the code. You should do this in an IInitializationModule, so use context.Locate.Advanced instead.

This is applicable from 9.2.0 and newer versions. 

Moral of the story:

  • Catalog structure can play a big role when it comes to performance.
  • You should do profiling whenever you can
  • We do that too, and we make sure to include improvements in later versions, so keeping your website up to date is a good way to tune performance.

Refactoring Commerce catalog code, a story

It is not a secret that I am a fan of refactoring. Clean. shorter, simpler code is always better. It’s always a pleasure to delete some code while keeping all functionalities: less code means less possible bugs, and less places to change when you have to change.

However, while refactoring can bring a lot of enjoying to the one who actually does it, it’s very hard to share the experience: most of the cases it’s very specific and the problem itself is not that interesting to the outside world. This story is an exception because it might be helpful/useful for other Commerce developer.

Continue reading “Refactoring Commerce catalog code, a story”

Commerce batching performance – part 2: Loading prices and inventories

UPDATE: When looked into it, I realize that I have a lazy loading collection of entry codes, so each test had to spent time to resolve the entry code(s) from the content links. That actually costs quite a lot of time, and therefore causing the performance tests to return incorrect results. That was corrected and the results are now updated.

In previous post we talked about how loading orders in batch can actually improve your website performance, and we came to a conclusion that 1000-3000 orders per batch probably yields the best performance result.

But orders are not the only thing you would need to load on your website. A more common scenario is to load prices and inventories for product. So If you are displaying a product listing page, it’s quite common to load prices and inventories for all products in that page. How should it be loaded?

Continue reading “Commerce batching performance – part 2: Loading prices and inventories”

Speed up your catalog indexing performance – part 2

Almost two years ago I wrote part 1 here: https://vimvq1987.com/speed-catalog-entries-indexing/ on how to speed up your catalog indexing performance. If you have a fairly big catalog with frequent changes, it might take longer time than necessary to build the index incrementally. (Rebuild index, in other hands, just delete everything and rebuild from scratch, so it is not affected by the long queue in ApplicationLog). I have seen some cases where rebuilding the entire index, is actually faster than waiting for it to build incrementally.

The tip in previous blog post should work very well if you are using anything lower than Commerce 11.6, but that is no longer the case!

Continue reading “Speed up your catalog indexing performance – part 2”