Batch job for performing an import is taking a long time to process - Forum - Rollbase - Progress Community

Batch job for performing an import is taking a long time to process

 Forum

Batch job for performing an import is taking a long time to process

  • I have a csv file that has roughly 320,000 records with 9 fields on each record.  I have a batch process setup to pull this file from a remote ftp server and perform an update/create nightly.  Upon creation or update of the record, 3 triggers are executed.  2 to attach related records and one to update a field on the object in Rollbase.  I kicked off the process manually and it too almost 26 hours to complete the process.  Obviously, that will make it difficult to update daily.  We've had other import processes that didn't take nearly that long to complete.  Is there anything I can do to make this process speed up?  I have screen shots of what I am seeing in the document below.

    Long import processing.docx

  • Hello,

    Did you try testing it partially and disabling the triggers just to see how much time the import is taking without running any triggers. Let me know how much time an import is taking by uploading a sample record of 50,000. Another cause of this is if there are other processes running in the system jobs section in the admin settings page.

    Thanks,
    Gian
  • This particular process normally has about 50K records.  It ran again this morning with 52K and took just under 6 hours.  That was with the triggers being fired as well. 

    I can't test it right now as the customer is working in there today.  I can fire off a test this evening running in manually as bulk and see what the time comes back as.  I know we've run other much larger imports that haven't take as long.

    I believe this was the only thing running at the time because it was over night.  I have other batch jobs scheduled that were in the queue, but that was all that was running at that time according to the job log file.

  • Hello,

    Thanks for the information you provided. I just calculated that approximately, 200 records are create every minute since the process lasted 26 hours. The 52,000 record ran slower with an estimate of 144 records per minute. What process are you using for importing those records in the batch job?

    Thanks,
    Gian
  • Speed of import may be reduced if lots of relationships have to be resolved in process. Records without relationships typically can be created much faster.

  • I am using the schedled ftp import.  Here is how I have it set up.

  • I understand. This only has a few relationships and is pretty basic.  We have to be able to have this many and most likely more going forward.  

  • Hello,

    Our imports also take a long time to process. Especially in our main employee object which has 140 relationships and about 10 triggers. It takes about two seconds to process one record. Which means that it takes 2 hours to import 3600 records and we have an import batch job running every night on each of our customer zones. (We are using import maps with about 30 to 50 mapped fields).

    We tried with disabling every trigger and it actually kinda helps (With no trigger being activated, we are able to have about 3 times faster processing of imports).

    But still, we would like it to be way faster because we'll soon have too much customers which means the import batch jobs are getting delayed and are now running during the day (They are planned to be processed at 8pm but are so long to process that it goes almost all day long (20hours non stop of imports)).

    Could the "bulk" option be added to import batch jobs? Right now, it only is possible when doing manual imports but could be very useful for importing large amount of data using import batch jobs.

    If we aren't able to improve performance of imports, we'll soon need a second STORAGE server in our private cloud instance to divide import processing of customers into two servers, but we would like to avoid that if possible.

    Kind regards,

    Romain.

  • Hello,

    I will ask the developers if adding the bulk functionality is possible in the future. However, I believe that using the bulk method will not run any triggers so you will have to manually run those triggers,

    Thank you,
    Gian
  • Hello,

    May I know how many fields you have in the object you are trying to import with? I just did a 50,000 import that only took minutes to finish. However this doesn't have any triggers and I only imported a single column value to the object. The create pick list value is also unchecked when I tested it.

    Thank you,
    Gian
  • There are 9 fields, two attach related triggers and the pick list is set to create.

  • Hello

    Can you uncheck the picklist create and then disable all running triggers? It should import the records rather quickly.

    Let me know how it goes,

    Thank you,
    Gian
  • Are you just looking for a benchmark time on the performance?  We have active users in here, so it isn't something I can just do during the day.  I also need the picklist values to be created and the relationship triggers to fire nightly.

  • Hello,

    I see. Yes, this is just to test the performance of your import process. As I mentioned previously, I imported 50k records within minutes (Using my local machine). I used a newly created object with a single import field. If you have a test environment with the same application that would greatly help us debug why the import is so slow.

    Let me know how it goes.

    Thank you.

  • I tried to clone the object and then run an import to test this.  I got an error that the object ID didn't exist.  I've had this happen in the past too.  If I look at my new object, it shows that the ID exists.  I wanted to get an exact copy of this, to make sure the test was accurate.  I guess I'll have to create a new object from scratch and try that.