How to sort 500 GB text file in Linux

1152130 pts.
Tags:
Big Data
Linux
We have a 500 GB text file that has roughly 10 billion rows that need to be sorted in alphabetical order. What's the best way to do this? A algorithm? For now, we've been using this command:
LANG=C
sort -k2,2 --field-separator=',' --buffer-size=(80% RAM) --temporary-directory=/volatile BigFile
And we're running this in AWS EC2 with a 120 GB RAM and 16 cores virtual machine. It takes most of the day to do. Could anyone help us out?
0

Answer Wiki

Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

Discuss This Question:  

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

Thanks! We'll email you when relevant content is added and updated.

Following

Share this item with your network: