Mapping files into SQL Server databases

Tags:
SQL Server Database File Mapping
SQL Server databases
Unix
Unix Split Function
I have used the split function utility in UNIX to split large files into multiple files of small length. However, the time it takes to map these files into my SQL Server databases has been reduced performance-wise but not tremendously well. To load 50 milion records, it is taking seven hours of time. Is there anything else I can do?

Answer Wiki

Thanks. We'll let you know when a new response is added.

How are you loading these files into the SQL Server?

For something like this it is recommended to use the BCP command line tool. When using BCP use the -b flag to have it process the records in smaller batches. This way it will commit the records in small groups instead of trying to commit all the records in the file at once.

I usually use batches of 1000. Using this method you should be able to load in the 50M records within a couple of hours maybe.

If there are any indexes on the table, you can try removing them as this will speed up the insert process.

Discuss This Question:  

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Thanks! We'll email you when relevant content is added and updated.

Following