I was asked this question: we have about 3TB of assets, any way to clean it up.
These days, storage is cheap, but still not free. and big storage means you need space for back up. and with that, bandwidth and time.
Is there away to clean up things you no longer need?
Yes!
Optimizely Content already has a scheduled job named Remove Abandoned BLOBs, but this job only removes the blobs that have no content associated. I.e. the content is deleted by IContentRepository.Delete but the blob was left behind. The job uses the log to find out which content were deleted, then find those blobs.
How’s about the assets that still have contents associated with them, but not used anywhere? Time to get your hands dirty!
Due to the nature of this task, it is best to make it a scheduled job.
All of the assets are children under the global asset root. By iterating over them, we can check if each of them is being used by another content. If not, we will add them to a list for later delete. Before deleting the content, we will find the blob and then delete it as well. Easy, right?
To get the content recursively we use this little piece of code
public virtual IEnumerable<T> GetAssetRecursive<T>(ContentReference parentLink, CultureInfo defaultCulture) where T : MediaData
{
foreach (var folder in LoadChildrenBatched<ContentFolder>(parentLink, defaultCulture))
{
foreach (var entry in GetAssetRecursive<T>(folder.ContentLink, defaultCulture))
{
yield return entry;
}
}
foreach (var entry in LoadChildrenBatched<T>(parentLink, defaultCulture))
{
yield return entry;
}
}
private IEnumerable<T> LoadChildrenBatched<T>(ContentReference parentLink, CultureInfo defaultCulture) where T : IContent
{
var start = 0;
while (!_isStopped)
{
var batch = _contentRepository.GetChildren<T>(parentLink, defaultCulture, start, 50);
if (!batch.Any())
{
yield break;
}
foreach (var content in batch)
{
// Don't include linked products to avoid including them multiple times when traversing the catalog
if (!parentLink.CompareToIgnoreWorkID(content.ParentLink))
{
continue;
}
yield return content;
}
start += 50;
}
}And we will start from SiteDefinition.Current.GlobalAssetsRoot, and use IContentRepository.GetReferencesToContent to see if it is used in any content (both CMS and Catalog). If not, we add it to a list. Later, we use IPermanentLinkMapper to see if it has any blob associated, and delete that as well
foreach (var asset in GetAssetRecursive<MediaData>(SiteDefinition.Current.GlobalAssetsRoot, CultureInfo.InvariantCulture))
{
totalAsset++;
if (!_contentRepository.GetReferencesToContent(asset.ContentLink, false).Any())
{
toDelete.Add(asset.ContentLink.ToReferenceWithoutVersion());
}
if (toDelete.Count % 50 == 0)
{
var maps = _permanentLinkMapper.Find(toDelete);
foreach (var map in maps)
{
deletedAsset++;
_contentRepository.Delete(map.ContentReference, true, EPiServer.Security.AccessLevel.NoAccess);
var container = Blob.GetContainerIdentifier(map.Guid);
//Probably redundency, can just delete directly
var blob = _blobFactory.GetBlob(container);
if (blob != null)
{
_blobFactory.Delete(container);
}
OnStatusChanged($"Deleting asset with id {map.ContentReference}");
}
toDelete.Clear();
}
}We need another round of delete after the while loop to clean up the left over (or if we have less than 50 abandoned assets)
And we’re done!
Testing this job is simple – uploading a few assets to your cms and do not use it anywhere, then run the job. it should delete those assets.
Things to improve: we might want to make sure only assets that created more than a certain number of days ago are deleted. This allows editors to upload assets for later uses without having to use them immediately.
The code has been open sourced at vimvq1987/DeleteUnusedAssets: Delete unused assets from an Optimizely/Episerver site (github.com) , and I have uploaded a nuget package to Packages (optimizely.com) to be reviewed.
