Speed up your Catalog incremental indexing

As your products are being constantly updated, you would naturally want them to be properly (and timely) indexed – as that’s crucial to have the search results that would influence your customers into buying stuffs. For example, if you just drop the prices of your products , you would want those products to appear in new price segment as soon as possible.

This should be very easy with Find.Commerce – so if you are using Find (which you should) – stop reading, nothing for you here. Things, however, can be more complicated if you are using the more “traditional” SearchProvider.

There are three ways to make your index up-to-date:

  • By event-driven indexing. This ensures that whenever an entry is updated (including cases when its associations or relations are updated), the entry will be automatically re-indexed. This sounds great in theory, but it does not work that great in practice. I’m going to be honest with you – sometimes the index processor stops working, and nobody really knows why, and the only solution is to restart the site. The chance of running into it is fairly small, but the impact can be quite big for your business.
  • By the incremental catalog indexing. It is supposed to index only the changed products, so it’s our preferred choice. But in older versions, incremental catalog indexing requires a fairly complicated query to get the “changed” entries, which I explained here . Even with this index added, it’s still not lightning fast. If you find on World, there have been several complains regarding the performance of this job if you have a very big history of changes.
  • By full-catalog indexing. This can be done by the ”
    Full Search Index Job”. Of course this will be a heavy task, and if you updated just 10 products, you would not want to reindex your entire 200 thousands products, right?

Ehh, what is your solution then? Are we doomed?

Of course not. We as a framework always think about solution which works better than the previous. And I have good news for you – in Commerce 11.4 this problem is essentially solved. We introduced a new indexing queue system which has very low overhead. No matter how big your history is, the time to get list of changed entries will remain (almost) constant. Pretty cool, huh?

Now your option to keep your catalog index up to date is simple. Disable the event-driven index processor, and turn on the incremental indexing job for 5 minutes (or any values of your choice). Well, your index is not truly real-time, but 5 minutes delay should be as good as you can get.

If you upgrade to 10.6 or later, it is even better, by adding this setting to your appSettings

And you can optimize it further.

Morals of the story:

  • We care about performance, a lot. And we promise to look into any problems you might find.
  • Upgrading to latest versions solve a lot of problems for you, for free.

Leave a Reply