I have a csv file that has roughly 320,000 records with 9 fields on each record. I have a batch process setup to pull this file from a remote ftp server and perform an update/create nightly. Upon creation or update of the record, 3 triggers are executed. 2 to attach related records and one to update a field on the object in Rollbase. I kicked off the process manually and it too almost 26 hours to complete the process. Obviously, that will make it difficult to update daily. We've had other import processes that didn't take nearly that long to complete. Is there anything I can do to make this process speed up? I have screen shots of what I am seeing in the document below.
Long import processing.docx
This particular process normally has about 50K records. It ran again this morning with 52K and took just under 6 hours. That was with the triggers being fired as well.
I can't test it right now as the customer is working in there today. I can fire off a test this evening running in manually as bulk and see what the time comes back as. I know we've run other much larger imports that haven't take as long.
I believe this was the only thing running at the time because it was over night. I have other batch jobs scheduled that were in the queue, but that was all that was running at that time according to the job log file.
Speed of import may be reduced if lots of relationships have to be resolved in process. Records without relationships typically can be created much faster.
I am using the schedled ftp import. Here is how I have it set up.
I understand. This only has a few relationships and is pretty basic. We have to be able to have this many and most likely more going forward.
Our imports also take a long time to process. Especially in our main employee object which has 140 relationships and about 10 triggers. It takes about two seconds to process one record. Which means that it takes 2 hours to import 3600 records and we have an import batch job running every night on each of our customer zones. (We are using import maps with about 30 to 50 mapped fields).
We tried with disabling every trigger and it actually kinda helps (With no trigger being activated, we are able to have about 3 times faster processing of imports).
But still, we would like it to be way faster because we'll soon have too much customers which means the import batch jobs are getting delayed and are now running during the day (They are planned to be processed at 8pm but are so long to process that it goes almost all day long (20hours non stop of imports)).
Could the "bulk" option be added to import batch jobs? Right now, it only is possible when doing manual imports but could be very useful for importing large amount of data using import batch jobs.
If we aren't able to improve performance of imports, we'll soon need a second STORAGE server in our private cloud instance to divide import processing of customers into two servers, but we would like to avoid that if possible.
There are 9 fields, two attach related triggers and the pick list is set to create.
Are you just looking for a benchmark time on the performance? We have active users in here, so it isn't something I can just do during the day. I also need the picklist values to be created and the relationship triggers to fire nightly.
I tried to clone the object and then run an import to test this. I got an error that the object ID didn't exist. I've had this happen in the past too. If I look at my new object, it shows that the ID exists. I wanted to get an exact copy of this, to make sure the test was accurate. I guess I'll have to create a new object from scratch and try that.