How to use Python to parse a 12 GB CSV file

1131585 pts.
Tags:
Big Data
CSV file
Python
We currently have a 12 GB CSV file. We're now trying to extract some columns from this data and then write a new CSV file that would load into R for data analysis. But we keep getting this error when we're loading the list before writing the new file. Is there a way we can parse the data row by row and create a CSV output. Here's our code:
import pandas

colnames = ['contributor name', 'recipient name', 'recipient party', 'contributor cfscore', 'candidate cfscore', 'amount']

DATA = pandas.read_csv('pathnamefilename.csv', names=colnames)
DATA.to_csv(''pathnamefilename.csv', cols = colnames)
Thank you!
1

Answer Wiki

Thanks. We'll let you know when a new response is added.

Hi
ITKE,

By the
using of this CSV module line-by-line

import csv

infname = "csv.csv"

outfname = "csv_stripped.csv"

cols = ["col1", "col2", "col3"]

with open(infname) as inf, open(outfname, 'w+') as outf:

    reader = csv.DictReader(inf)

    writer = csv.DictWriter(outf, cols, extrasaction='ignore')

    writer.writeheader()

    for line in reader:

        writer.writerow(line)

To
know about them please click here:
https://docs.python.org/2/library/csv.html

I hope
it will assist you in resolving issues.

Thanks
& Regards

Clark
Kent

Discuss This Question:  

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

Thanks! We'll email you when relevant content is added and updated.

Following

Share this item with your network: