Sitefinity 200k products importing very slow - Ecommerce - Ecommerce - Progress Community

Sitefinity 200k products importing very slow

 Ecommerce

Sitefinity 200k products importing very slow

  • Sitefinity 200k products importing very slow
  • Hi Team,

     I have a task of migrating data from a CSV to Sitefinity ecommerce. The tedious part is importing 200k products from CSV. Every single product savechanges call takes time. I tweaked the code as below to run in batches of 10. Still the performance is really sad. To import 1,000 products it takes around 10 minutes. For 1 lakh proudcts it is taking more than 10 hours. Please advise on this as this is really very slow and I am stuck with a migration task to reduce the time to 1 hour.

     int skip = 0;
                    int take = _batchSize == 0 ? 10 : _batchSize;
                    int currentCount = 0;

                    int totalCount = productImportCsvs.Count();
                    while (currentCount < totalCount)
                   
                        string skusNotImported = string.Empty;
                        try
                       

                            IEnumerable<ProductImportCSV> chunkedProducts = productImportCsvs.Skip(skip).Take(take);
                            skip = skip + take;
                            currentCount = skip;

                            foreach (ProductImportCSV product in chunkedProducts)
                           
                                if (string.IsNullOrEmpty(product.SKU))
                               
                                    continue;
                               

                           
                                ImportSingleProduct(dynamicModuleManager, scheduledJob, catMgr, liveProducts, masterProducts, ref successfullyImportedProducts,
                                    ref errorImportProducts, product, deleteExistingProducts);
                           

                               
                            catMgr.SaveChanges();
                       

  • Hi Praneeth,

    You can check the Product import widget sample.

    More information and installation instructions is available in that article.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
     
  • HiSvetoslav ,

     I had already tried it even before writing the post :-(. The performance was not satisfactory and hence writing this post. Actually some part of implementation I had taken from the code itself. As mentioned 200k records even with that code took almost 10 to 15 hours. We need something which can do the work atleast within 2 hours.

     

    Do you have some sort of SQLMapper class. I am planning to write SQLBulkCopy and finish the job, which I am completely against :-(

     Thanks

    Praneeth

  • 7653f59a-594b-430f-8ad2-b2d772b0b15d_ProducstImport.zip
    Hello Praneeth,

    In order to speedup the product creation, you can try to use manager.Lifecycle.Publish(item). Also, SaveChanges() should be called in batches for example for every 50 or 100 items created.

    You can check the attached code sample. This should speedup the creation.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items
     
  • Tried this code already. The methodImportSingleProduct what I mentioned does the same :-(. Still it is slow. Any other suggestions.

     

    Thanks

    Praneeth

  • Hello Praneeth,

    What is the time you have experienced for 200k products using the latest example provided? During the tests we have 20k products have been created for about 18 min.

    In addition, would you please upload your entire code sample in order to inspect and test it.

    Regards,
    Svetoslav Manchev
    Telerik
     
    Do you want to have your say in the Sitefinity development roadmap? Do you want to know when a feature you requested is added or when a bug fixed? Explore the Telerik Sitefinity CMS Ideas&Feedback Portal and vote to affect the priority of the items