No this is not really “art” – I’m just trying to have a more clickbait title. It’s more about understanding what you have at your disposal and use them for your benefits – in this case – how new SQL statement can drastically improve your performance.
In this blogpost we will look into paging feature of SQL Server. in Commerce we usually work with large set of data – millions of rows are fairly common, and it’s natural to load data by page. There is no point loading thousands, or even millions of rows in one go. First it’s not practical to display all of them. Second you’ll likely end up with an timeout exception and/or an out of memory exception. Even if you are lucky enough to get through, it’s still able to take your SQL Server instance to a knee, and transferring that much data over network will be another bottleneck for your system. So my friends, the best practice for loading data is to do it by batches, and to not load everything at once.
It’s quite common when a customer browses your site without logging in – either she/he intentionally does that, or just forget about logging in. The customer might add some items to carts and even checks out, but then is asked to log in or remember to log in. What would happen?
By default, Episerver Commerce will do as following:
Attach all orders made by that section to logged in customer. (I once placed an order in a famous retailer in Sweden without logging in, and then I asked their customer service to link that order to my account so I can track it easier. To my surprise, it cannot be done! That’s why I personally appreciate this feature.)
Merge all carts to existing carts, by name and market. So if I’m currently in US market and I added a item to that cart, and the US-cart linked to my account already have 2 other items, then when I log in, my US-cart will contain 3 items.
Merge all wishlist to existing wishlists, also per market. (The name for wishlist, by default, is fixed to “Wishlist”)
One of the last things you want to get from your Commerce site is that the order data is gone. What can be more confused than if your log shows that the cart has been converted into a purchase order, you even got the PO number, but after that, the order disappears? It’s nowhere to be found, even if you look into database. It’s kind of magic, but not the kind of magic you would want to have.
But everything happens for a reason. And actually it’s with a good reason: data consistency.
Episerver Commerce has the concept of TransactionScope. Simply put, it allows two or more database operations to be done as atomic: Either all of them succeed, or all of them will revert back. If a TransactionScope contains 3 operations A, B, C, then even if A, B succeeded, but C is yet to commit, and something goes wrong, then A and B would be reverted.
Recently I worked on two support cases from our customers as they see SQL Server errors, such as “System.Data.SqlClient.SqlException (0x80131904): The INSERT statement conflicted with the FOREIGN KEY constraint “FK_ShipmentEx_Shipment”. The conflict occurred in database “dbCommerce”, table “dbo.Shipment”, column ‘ShipmentId’“, or “System.Data.SqlClient.SqlException (0x80131904): The MERGE statement attempted to UPDATE or DELETE the same row more than once. This happens when a target row matches more than one source row. A MERGE statement cannot UPDATE/DELETE the same row of the target table multiple times. Refine the ON clause to ensure a target row matches at most one source row, or use the GROUP BY clause to group the source rows.”
These errors happened randomly, during the high load times – it seems to be affected by the concurrency level.
What was wrong? and why?
It took me a good amount of time, and good amount of hairs, too. The actual error is another one, and the one above is just the “by product”.
The cart system in Episerver Commerce suffers from a design flaw: it shares (almost) everything with the purchase orders. ShoppingCart is just another metaclass extended OrderGroup, so it’ll use the same OrderGroup, OrderForm, Shipment, LineItem and OrderAddress tables in the database, like PurchaseOrder and PaymentPlan. At first, it seems to be reasonable approach. But when you have hundreds, or thousands of customers visiting your website (and you would be happy to see that ;)) – problems start to appear.
For a good long amount of time, workflows have been an essential part of Episerver Commerce (and even before that, Mediachase eCF). Once you get into order system, you just can’t escape workflows – because you need them. They handle many – if not all things, from validating items, checking inventories (to make sure that you are not selling something out-of-stock, or just discontinued (think of Galaxy Note 7, poor little shiny phone)), applying promotions, calculating taxes, process payments, and finally adjust inventories (yay, a customer places an order, let’s ship to him as soon as possible, firstly allocate the goods for him). And workflows are even required by later processing – when you complete the shipments (another happy customer!), or when you issue a return or an exchange (well, let’s keep the customer still happy).