Mapping files into SQL Server databases
I have used the split function utility in UNIX to split large files into multiple files of small length. However, the time it takes to map these files into my SQL Server databases has been reduced performance-wise but not tremendously well. To load 50 milion records, it is taking seven hours of time. Is there anything else I can do?
Software/Hardware used:
Software/Hardware used:
ASKED:
February 25, 2009 8:44 PM
UPDATED:
February 25, 2009 11:53 PM
Answer Wiki:
How are you loading these files into the SQL Server?
For something like this it is recommended to use the BCP command line tool. When using BCP use the -b flag to have it process the records in smaller batches. This way it will commit the records in small groups instead of trying to commit all the records in the file at once.
I usually use batches of 1000. Using this method you should be able to load in the 50M records within a couple of hours maybe.
If there are any indexes on the table, you can try removing them as this will speed up the insert process.
To see all answers submitted to the Answer Wiki: View Answer History.
RSS - Discussions Help
Discuss This Question:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
