Before we even start, I would reiterate that manipulating data directly should be avoided unless absolutely necessary, it should be used as the last resort, and should be proceeded with cautions – always back up first and test your queries on development database first before running it in production. And if the situation dictates that you have to run the query, better do it with the 4 eyes principle – having a colleague double check it for you. When it comes to production database, nothing is too careful.
Now back to the question, if you absolutely have to delete a content, you should do like this
It is basically what Content Clouds (i.e. CMS) does under the hood, without the cache validation on the application layer of course.
So the moral of the story – do everything with API if you can. If you absolutely have to, use the built-in stored procedures – they are tested vigorously and should have minimal issues/bugs, and should take care of everything, data-wise for you. Only write your own query if there is no SP that can be used.
Update: Initially I mentioned Tomas’ post in this, and that gave impression his way is incorrect. I should have written better. My apologies to Tomas
One of the questions I have received, from time to time, is that how to store a lot of prices per SKU in Optimizely (B2C) Commerce Cloud. While this is usually a perfect candidate for Optimizely B2B Commerce, there are many customers invested in B2C and want to make the best out of it. Is it possible?
It’s important to understand the pricing system of Optimizely Commerce (which is, written in detail in my book – shameless plug). But in short:
There are two price systems, IPriceService and IPriceDetailService
One is handling prices in batch – i.e. prices per SKU (IPriceService), and one is handling prices per individual price (IPriceDetailService)
Both are cached in latest version (cache for IPriceDetailService was added in late 13.x version)
With that in mind, it would be very problematic if you use IPriceService for such high number of prices per SKU, because each time you save a price, you save a lot of prices at once (same as loading prices). This is how the defaultIPriceService implementation saves prices of a SKU
create procedure dbo.ecf_Pricing_SetCatalogEntryPrices
@CatalogKeys udttCatalogKey readonly,
@PriceValues udttCatalogEntryPrice readonly
as
begin
begin try
declare @initialTranCount int = @@TRANCOUNT
if @initialTranCount = 0 begin transaction
delete pv
from @CatalogKeys ck
join dbo.PriceGroup pg on ck.CatalogEntryCode = pg.CatalogEntryCode
join dbo.PriceValue pv on pg.PriceGroupId = pv.PriceGroupId
merge into dbo.PriceGroup tgt
using (select distinct CatalogEntryCode, MarketId, CurrencyCode, PriceTypeId, PriceCode from @PriceValues) src
on ( tgt.CatalogEntryCode = src.CatalogEntryCode
and tgt.MarketId = src.MarketId
and tgt.CurrencyCode = src.CurrencyCode
and tgt.PriceTypeId = src.PriceTypeId
and tgt.PriceCode = src.PriceCode)
when matched then update set Modified = GETUTCDATE()
when not matched then insert (Created, Modified, CatalogEntryCode, MarketId, CurrencyCode, PriceTypeId, PriceCode)
values (GETUTCDATE(), GETUTCDATE(), src.CatalogEntryCode, src.MarketId, src.CurrencyCode, src.PriceTypeId, src.PriceCode);
insert into dbo.PriceValue (PriceGroupId, ValidFrom, ValidUntil, MinQuantity, MaxQuantity, UnitPrice)
select pg.PriceGroupId, src.ValidFrom, src.ValidUntil, src.MinQuantity, src.MaxQuantity, src.UnitPrice
from @PriceValues src
left outer join PriceGroup pg
on src.CatalogEntryCode = pg.CatalogEntryCode
and src.MarketId = pg.MarketId
and src.CurrencyCode = pg.CurrencyCode
and src.PriceTypeId = pg.PriceTypeId
and src.PriceCode = pg.PriceCode
delete tgt
from dbo.PriceGroup tgt
join @CatalogKeys ck on tgt.CatalogEntryCode = ck.CatalogEntryCode
left join dbo.PriceValue pv on pv.PriceGroupId = tgt.PriceGroupId
where pv.PriceGroupId is null
if @initialTranCount = 0 commit transaction
end try
begin catch
declare @msg nvarchar(4000), @severity int, @state int
select @msg = ERROR_MESSAGE(), @severity = ERROR_SEVERITY(), @state = ERROR_STATE()
if @initialTranCount = 0 rollback transaction
raiserror(@msg, @severity, @state)
end catch
end
If you have experience with SQL (which you probably should), you will see that it’s a deletion of rows in PriceValue that have CatalogEntryCode same as , then a merge, then a deletion of left over rows. To make matters worse, IPriceService system stores data on 3 tables: PriceValue, PriceGroup and PriceType. Imagine doing that with a few dozen of thousands rows.
Even if you change just one price, all prices of that specific SKU will be touched. It’d be fine if you have like ten prices, but if you have ten thousands prices, it’ll be a huge waste.
Not just that. To save one price, you would still need to load all prices of that specific SKU. That’s two layers of waste: the read operations at database layer, and then on application, a lot of price objects will need to be constructed, and then you need to recreate a datatable to send all the data back to the database to do the expensiveoperation above.
And wait, because the prices saved to IPriceService needs to be synchronized to IPriceDetailService (however, you can disable this). Prices that were changed (which is, all of them) need to be replicated to another table.
So in short, IPriceService was not designed to handle many prices per SKU. If you have less than a few hundred prices per SKU (on average), it’s fine. But if you have more than 1000 prices per SKU, it’s time to look at other options.
No, I do not mean that big, big data (in size of terabytes or more). It’s big collection, like when you have a List<string> and it has more than a few hundreds of items. Where to store it?
Naturally, you would want to store that data as a property of a content. it’s convenient and it just works, so you definitely can. But the actual question is: should you?
It’s as simple as this
public virtual IList<String> MyBigProperty {get;set;}
But under the hood, it’s more than just … that. Let’s ignore UI for a moment (rendering such long list is a bad UX no matters how you look at it, but you can simply ignore rendering that property by appropriate attributes), and focus on the backend aspects of it.
List<T> properties are serialized as a long strings, and save at once to database. If you have a big property in your content, this will happen every time you load your content:
The data must be read from database, then transferred through the network
The data must be parsed to create an array (the underlying data structure of List<T>. The original string is tossed away.
Now you have a big array that you might not use every time. it’s just there taking your previous LOH (as with the original string)
Same thing happens when you actually save that property
The data must be serialized as string, the List<T> is now tossed away
The data then must be transferred through the network
The data then saved to database. Even though it is a very long string and you changed, maybe 10 characters, it’s completely rewritten. Due to its size, there might be multiple page writes needed.
As you can see, it can create a lot of waste, especially if you rarely use that property. To make the matter worse, due to the size of the property, it means they are taking up space in LOH (large objects heap).
And imagine if you have such properties in each and every of your content. The waste is multiplied, and your site is now at risk of some frequent Gen 2 Garbage collection. Nobody likes visiting a website that freezes (if not crashes) once every 30 minutes.
Then when to store such big collection data?
The obvious answer is … somewhere else. Without other inputs, it’s hard to give you some concrete suggestions, but how’s about a normalized custom table? You have the key as the content reference, and the other column is each value of the list. Just an idea. Then you only load the data when you absolutely need it. More work, yes, but it’s the better way to do it.
Just a reminder that whatever you do, just stay away from DDS – Dynamic Data Store. It’s the worst option of all. Just, don’t 🙂
I got the question here Content Events and Service API | Optimizely Developer Community , and CHKN contacted Optimizely developer support service which is the good/right thing to do. However it could be beneficial for a wider audience, hence this blogpost.
If you are developing new Optimizely Commerce Cloud now, you should be using the latest version (CMS 12/ Commerce 14), or at least Commerce 13.31 if you have to stay with .NET 4.8. You then could use the IContentEvents to listen to any events that might be fired from Service API.
However, if you are using older versions, you might be limited to lower level, non content event through EventContext. It works, but with a catch: there is no EntryDeleting event in EventContext. At this point I’m not entirely sure why, probably just an overlook. But it’s not impossible to work around that issue.
As I suggested in the post, EntryUpdating is like a blanket event – every change to the entries goes through there. The sender is a CatalogEntryDto, which should contain information about the being deleted entries.
private void Instance_EntryUpdating(object sender, Mediachase.Commerce.Catalog.Events.EntryEventArgs e)
{
var dto = sender as CatalogEntryDto;
if (dto != null)
{
var rows = dto.CatalogEntry.Where(x => x.RowState == DataRowState.Deleted);
var deletingCodes = rows.Select(x => (string)x["Code", DataRowVersion.Original]);
//do stuffs with codes being deleted.
}
}
And then to listen to the event in one of your IInitializationModule
However, there is a caveat here: as EntryUpdating happens before the entry deletion, it is possible that the change did not go through (i.e. the changes were to be reverted). It is unlikely, but it’s a possibility. You might accept that, or you can:
Store the id and code in a “Deleting entry” dictionary
Listen to EntryDeleted event and match from DeletedEntryEventArgs parameter (which contains EntryId ) to get the deleted Code, and continue from there.
It’s not a secret that cache is one of the most, if not the most, important factors to their website performance. Yes cache is great, and if you are using Optimizely Content/Commerce Cloud, you should be using ISynchronizedObjectInstanceCache to cache your objects whenever possible.
But caching is not easy. Or rather, the cache invalidation is not easy.
To ensure that you have effective caching strategy, it’s important that you have cache dependencies, i.e. In principles, there are two types of dependencies:
Master keys. This is to control a entire cache “segment”. For example, you could have one master key for the prices. If you needs to invalidate the entire price cache, just remove the master key and you’re done.
Dependency keys. This is to tell cache system that your cache item depends on this or that object. If this or that object is invalidated, your cache item will be invalidated automatically. This is particularly useful if you do not control this or that object.
ISynchronizedObjectInstanceCache allows you to control the cache dependencies by CacheEvictionPolicy . There are a few ways to construct an instance of CacheEvictionPolicy, from if the cache expiration will be absolute (i.e. it will be invalidated after a fixed amount of time), or sliding (i.e. if it is accessed, its expiration will be renewed), to if your cache will be dependent on one or more master keys, and/or one or more dependency keys, like this
/// <summary>
/// Initializes a new instance of the <see cref="CacheEvictionPolicy"/> class.
/// </summary>
/// <param name="cacheKeys">The dependencies to other cached items, idetified by their keys.</param>
public CacheEvictionPolicy(IEnumerable<string> cacheKeys)
/// <summary>
/// Initializes a new instance of the <see cref="CacheEvictionPolicy"/> class.
/// </summary>
/// <param name="cacheKeys">The dependencies to other cached items, idetified by their keys.</param>
/// <param name="masterKeys">The master keys that we depend upon.</param>
public CacheEvictionPolicy(IEnumerable<string> cacheKeys, IEnumerable<string> masterKeys)
The constructors that takes master keys and dependency keys look pretty the same, but there is an important difference/caveat here: if there is no dependency key already existing in cache, the cache item you are inserting will be invalidated (i.e. removed from cache) immediately. (For master keys, the framework will automatically add an empty object (if none existed) for you.)
That will be some unpleasant surprise – everything seems to be working fine, no error whatsoever. But, if you look closely, your code seems to be hitting database more than it should. But other than that, your website performance is silently suffering (sometimes, not “silently”)
This is an easy mistake to make – I did once myself (albeit long ago) in an important catalog content cache path. And I saw some very experienced developers made the same mistake as well (At this point you might wonder if the API itself is to blame)
Take away:
Make sure you are using the right constructor when you construct an instance of CacheEvictionPolicy . Are you sure that the cache keys you are going to depend on, actually exist?
In newer version of CMS, there would be a warning in log if the cache is invalidated immediately, however, it could be missed, unless you are actively looking for it.
Note that this behavior is the same with ISynchronizedObjectInstanceCache as it extends IObjectInstanceCache.
From time to time, I have to dig into some customers’ profiler traces to figure out why their site is slow (yeah, if you follow me, you’d know that’s kind of my main job). There are multiple issues that can eat your website performance for breakfast, from loading too much content, to unmaintained database indexes. While my blog does not cover everything, I think you can get a good grasp of what mistakes to avoid.
The library we’ll be investigating today would be Maxmind.db. To be honest, I’ve never used it my own, but it seems to be a very popular choice to geography-map the visitors. It’s usually used by Optimizely sites for that purpose, using VisitorGroup (which is why it came under my radar).
For several sites that use it, it seems more often than not stuck in this stack
It’s essentially to think that CreateActivator is doing something heavy here (evidently with the LambdaCompiler.Compile part. A peek from decompiling actually shows that yes, it’s heavy. I’m not quite sure I can post the decompiled code here without violating any agreement (I did, in fact, accepted no agreement at this point), but it’s quite straightforward code: TypeActivatorCreator uses reflection to get the constructors of the Type passed to it, to sees if there is any constructor decorated with MaxMind.Db.Constructor attribute, then prepares some parameters, and creates an LambdaExpression that would create an instance of that Type, using found constructor (which is a good thing because a compiled expression would perform much better than just a reflection call).
(I’m using Mindmax.db 2.0.0, for the record)
The code is straightforward, but it is also slow – as any code which involves reflection and lambda compilation would be. The essential step would be to cache any result of this. This is actually a very good place to cache. The number of types are fixed during runtime (except for very edge cases where you dynamically create new types), so you won’t have to worry about cache invalidation. The cache would significantly improve the performance of above code.
And in TypeActivatorCreator there is a cache for it. It is a simple ConcurrentDictionary<Type, TypeActivator> , which would return an TypeActivator if the Type was requested before, or create one and cache it, it it hasn’t been. As I said, this is a very good place to add cache to this.
There is a cache for that, which is good. However, the very important tidbit here is that the dictionary is not static. That means, the cache only works, if the class is registered as Singleton (by itself, or by another class down the dependency chain), meaning, only one of the instance is created and shared between thread (which is why the ConcurrentDictionary part is important).
But except it’s not.
When I look at a memory dump that collected for a customer that is using Maxmind.db, this is what I got:
0:000> !dumpheap -stat -type TypeAcivatorCreator Statistics: MT Count TotalSize Class Name 00007ffa920f67e0 1 24 MaxMind.Db.TypeAcivatorCreator+<>c 00007ffa920f6500 147 3528 MaxMind.Db.TypeAcivatorCreator Total 148 objects
So there were 147 instances of TypeAcivatorCreator. Note that this is only the number of existing instances. There might be other instances that were disposed and garbaged by CLR.
Now it’s clear why it has been performing bad. For supposedly every request, a new instance of TypeActivatorCreator is created, and therefore its internal cache is simply empty (it is just newly created, too). Therefore each of request will go through the expensive path of CreateActivator, and performance suffers.
The obvious fix here is to make the dictionary static, or making the TypeActivatorCreator class Singleton. I don’t have the full source code of Mindmax.Db to determine which is better, but I’d be leaning toward the former.
Moral of the story:
Caching is very, very important, especially when you are dealing with reflection and lambda compilation
You can get it right 99%, but the 1% left could still destroy performance.
Update:
I reached out to Maxmind.db regarding this issue on November 9th, 2021
About 6h later they replied with this
I was at first confused, then somewhat disappointed. It is a small thing to fix to improve overall performance, rather than relying on/expecting customers to do what you say in documentation. But well, let’s just say we have different opinions.
It has been 5 years since I started Pro Episerver Commerce back in early 2016. The book was a success, not as big as I hoped for, but definitely bigger than I expected. Tackling a niche market, it was fairly popular within the community, and it gave me a lot of happiness (and some pocket changes) to see that it helped many developers to understand and use the framework – which I help created, and love – better.
So much has changed in the last 5 years.
I have my first kid, and a second one. I left Commerce development team, to work on my own, then have a small team. Episerver bought Optimizely, then rebrand.
And so much more has happened with Episerver Commerce, more than just being renamed to Optimizely Commerce Cloud.
It deserves a new book!
To celebrate my 10th anniversary with Episerver (now Optimizely), I am proud, and excited to announce the second edition of Pro Episerver Commerce – Pro Optimizely Commerce Cloud. Most of the content written in Pro Episerver Commerce is still very much applicable, but I feel there is a need to refocus and expand on important parts.
There are two big types of relations in Episerver (Optimizely) B2C Commerce: relations between entries and nodes, and between nodes. As you will most likely have to touch one or another, or both, sooner or later, this post aims to give you some understanding on how they are structured/work.
Node-Entry relation
When you add a product (or variant, or package, or bundle) to a category, you are creating a NodeEntryRelation. And there are a two types of NodeEntryRelation
Primary NodeEntryRelation, which means the category is counted as true parent of the entry. Each entry can only have at most one primary NodeEntryRelation (Which means it can have no primary NodeEntryRelation at all).
Secondary NodeEntryRelation, which means the entry is linkedto the category. You do that when it makes sense for the product to be in additional categories. for example, a GoPro can be in Camera (primary category), but can also be in Sport Gears (linked). An entry can have no, or multiple secondary NodeEntryRelation.
The concept of primary NodeEntryRelation was added to Commerce 11. Before that, it’s a bit more of a guess work – the “main” category is determined by the sort order – the relation with lowest sort order is considered “main relation”. That introduces some inconsistency, which prompted the rework on that.
What is the main different between those two things? For one thing, access rights. For Commerce, you can set access rights down to categories, but not entries. The entries will inherit access rights from their true parents (i.e. primary nodes). An entry without primary node-entry relation is considered the direct children of a catalog, and inherits its access right settings.
Another smaller difference is that if you delete a category, true children entries will be deleted. Linked entries will only be detached from that category.
NodeEntryRelation can be managed fully by IRelationRepository, using the NodeEntryRelation type, and you can use a few extension methods to make it easier – for example EntryContentBase.GetCategories().
How are your actions in Catalog UI reflected on a data level:
When you create a new entry (product/SKU/etc.) in a category, you create a primary node-entry relation for them
When you move (cut then paste) an entry to a new category, you are creating a new primary node-entry relation. If the entry already has a primary node-entry relation, the new one will take over.
When you link/detach an entry to/from a new category, you are creating/removing a non-primary node-entry relation
Node-Node Relation
Like Node-Entry relation, a node can be a true parent of a node, or just be a “linked” parent.
Unlike Node-Entry relation, Node-Node relation is quite different that it’s separated in two places.
Linked nodes are represented by NodeRelation(s) (it might be interesting to know that NodeRelation is the parent class of NodeEntryRelation. The interpretation is that a NodeRelation is – a relation of a node, which can be with another node, or an entry)
There is no primary NodeRelation, the true parent node is identified by a property of the category itself. When you have a NodeContent content, then the ParentLink property points to the true parent.
For that reason, a node will always have a true parent, either a catalog, or a node. You can’t use IRelationRepository (and therefore, ServiceAPI) to manage (delete or update) a true parent of a node , you would have to:
Set its ParentLink to something else
Use IContentRepository.Move to move it to a new parent.
Note that this is the limitation of the Relations API in ServiceAPI. You can technically change the ParentLink of a node and update via POST. It’s just more work and not as intuitive as the Relations API.
Why the disparity, you might ask? Well, a lot of design decisions in Commerce comes from historical reasons, and after that, constrained resources (time/man power) and priority. While the disparity is probably not the best thing you want, it still works fairly well, and if you understand the quirk then it is all well.
If you are using Episerver Commerce (or should I say, Optimizely B2C Commerce), you will, at some point, need to get the contact by an email address. That sounds like an easy enough task, until you realize that the class to manage customer contacts –CustomerContext has no such method. You will need to find another way, and this is one way you can do it
Of course this is not the optimal – avoid it if you can. First of all it loads a lot of contact just to find one. Also while it looks like you are getting all contacts (which is of course something to avoid), you are only get the first 1000 contacts by default, so the code above would return inaccurate result.
Is there a better way?
Yes of course.
Contact was built on “Business Foundation” – think of it as an ORM with extensions. Business Foundation allows great flexibility, with a few caveats. This is how you can find contact by email address:
BusinessManager does not have a Get method, so we are use List instead. Note that Email is supposed to be unique, so this should not have any down side with performance (see note below).
This is not limited to email, or to contact. You can find contacts by other properties, or get an Organization with same technique.
It is very important to note, however, this need a proper index on Contact tables, to make sure you are not killing your database.
HttpContext is not designed to be thread-safe, which means you can run into nasty things if you share a HttpContext between thread. For example, this is one of the things that can happen: One (or more) thread is stuck in this:
If you don’t know why this is scary, this is any infinite loop, meaning your CPU will be spent 100% in to this Dictionary.FindEntry, unable to do anything else. The only way to solve this problem is to restart the instance.
That is caused by unsafe accessing of a Dictionary – if you have a thread that is enumerating it, and another thread trying to write to it, it is possible to run into a dead end like this.
And HttpContext just happens to have many Dictionary properties and sub-properties. HttpContext.Request.RequestContext.RouteData.DataTokens is one of them (And a reason the code above ended in a disaster), making it vulnerable for this kind of problem. Which is exactly why it is not recommended to share a HttpContext between threads.
Sometimes, you can just set HttpContext.Current to null. Sometimes, you need to take a step back and ask yourself that do you really need to run things in parallel?