Writing to block device is *slower* than writing to the filesystem?

344855 pts.
Tags:
Cache Memory
RAID
Storage
Hi all, We have a new machine with 3ware 9650SE controllers and I am testing hardware RAID and linux software MD raid performances. For now I am on hardware RAID. I have setup a raid-0 with 14 drives. If I create an xfs filesystem on it (whole device, no partitioning, aligned stripes during mkfs, etc) then I write to a file with dd (or with bonnie++) like this: sync ; echo 3 > /proc/sys/vm/drop_caches ; dd if=/dev/zero of=/mnt/tmp/ddtry bs=1M count=6000 conv=fsync ; time sync about 540MB/sec come out (last sync takes 0 seconds). This is similar to 3ware-declared performances of 561MB/sec http://www.3ware.com/KB/Article.aspx?id=15300 however, if instead I write directly to the block device like this sync ; echo 3 > /proc/sys/vm/drop_caches ; dd if=/dev/zero of=/dev/sdc bs=1M count=6000 conv=fsync ; time sync performance is 260MB/sec!?!? (last sync takes 0 seconds) I tried many times and this is the absolute fastest I could obtain. I tweaked the bs, the count, I removed the conv=fsync... i ensured 3ware caches are ON on the block device, I set anticipatory scheduler... No way. I am positive that creating the xfs filesystem and writing on it is definitely faster than writing to the block device directly. How could that be!? Anyone knows what's happening? Please note that the machine is absolutely clean and there is no other workload. I am running kernel 2.6.31 (ubuntu 9.10 alpha live). Thank you

Answer Wiki

Thanks. We'll let you know when a new response is added.

When you write to the FS on RAID, the writes are getting distributed across 14 drives, but when you write directly to a single drive you are limited by the speed of that drive.

One thing i’m not sure about is how you managed to get 260MB/s on a single drive. Are you using an SSD?

Discuss This Question:  

 
There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when members answer or reply to this question.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

To follow this tag...

There was an error processing your information. Please try again later.

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Thanks! We'll email you when relevant content is added and updated.

Following