Infiniband

Infiniband adapters arrived so I inserted one to the NAS and started testing the speeds.


First of all, CIFS/SMB write traffic even through IB when sync=always looks bad. It gives around 80MB/s writes and that is not acceptable. When sync=standard it doesn't go through ZIL and the RAIDZ2 can write 100-140MB/s. Still not acceptable.... waiting to figure something out.

Reading on the other hand works perfectly, I transferred around 3GB file from cache and it was around 600MB/s, that is more than acceptable speed. Transfer directly from RAIDZ2 was around 200MB/s, this is also enough.

Below is the test I did with iperf from NAS to my other host:


[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.20 GBytes  7.05 Gbits/sec

So to conclude this time, the connectivity between hosts work perfectly and is fast enough.. the only problem is CIFS/SMB writing. (NFS was not tested yet, since I need an IB Switch to connect more hosts)

Performance 2

I have been testing a lot with the performance tuning and decided that since I need a reliable data storage to keep my stuff safe, I will keep the RAIDZ2 array with mirrored ZIL and striped L2ARC. Below are two charts showing the performance I get with direct writes and reads to the filesystem.

So what comes to writing, we can see that the bigger the blocksize, the faster the speed (says Captain Obvious) and the reading from the fs is somewhat inconsistent, but the speeds are pretty acceptable.

After these tests I turned off atime from the fs and the speeds are a bit better with each blocksize. (zfs set atime=off <fs>)

more later...

Performance

Performance testing was done with NFS & SMB copying from Windows 7 computer. Which was a mistake... NFS on Windows 7 is not a very working solution. I was getting around 35MB/s transfer speeds with NFS and SSD as ZIL device. I almost decided to dump the whole thing.. but then..

I changed to use a  Linux box and voila, speed was 110MB/s constantly with NFS, but then the SMB was still not so good, around 120MB/s at start and it went down to 45MB/s in the end of the transfer on Windows 7 machine. So I decided to change the whole fs to always use sync, so that it would go through ZIL. This gave it a constant speed of 80MB/s which isn't still very good, but manageable.

Direct writing to the fs was tested with both, striped and mirrored ZIL, with sync always on. Speeds were 333MB/s with mirrored ZIL and 512MB/s with striped ZIL, which would be something we are looking for.

Reading speed has not been tested yet, I am waiting for another Infiniband card, since this one is not supported by the OS.

So as a conclusion:

• Do not use Windows 7 as NFS client
• Keep all writes synchronized (zfs set sync=always <fs>)
• Max write speed depends on your ZIL speed (my setup, 333MB/s - 512MB/s)
• SMB writes are not as fast as I hoped, but we'll see after changing to Infiniband

Some more later..

OS Installation

OS installation was done from one USB to another USB

• Openindiana Build 151a5 Server 64bit
• Napp-IT (installed using ->  wget -O - www.napp-it.org/nappit | perl)

Configured a pool named "data" with all four 2TB disks as RAIDZ2 and partitioned SSD's:

(We need good data integrity and redundancy on this box, so if this doesn't give enough speed, we go to mirroring)

• 60% for ZIL mirror
• 39% for L2ARC Stripe

To add them to the pool ZIL and L2ARC devices:

• zpool add data log mirror c5t0d0p1 c5t1d0p1
• zpool add data cache c5t0d0p2 c5t1d0p2

That's the basic disk configuration, now let's see the performance.....

Parts arrived: part 2

Now I finally got all the parts, the following additions arrived today

• 2x  OCZ Agility 3 60GB 2.5" SSD SATA III
• 1x  Eolize SVD-NC11-4 Mini-ITX NAS
• 1x 4GB USB memory for OS

Now it's finally time to start building the machine....

Oh well, the case is a bit small for the motherboard&CPU&FAN combo, I had to "customize" the FAN a bit to make it fit in the case. No worries, it fits now.

This is how I did it (cut one of the four FAN holders away)

See the cut "holder"

Otherwise everything fit in "ok"

The final product looks like this

A black box...

Next we start the OS installation, this will be in the next post..

Parts arrived: part 1

Oh goodie, some of the parts arrived today, and the rest are coming tomorrow. Here's what I have so far:

• 1x Asus P8H77-I Intel H77 LGA1155 MiniITX - Motherboard
• 1x Intel Core i5 3450S 2.8 GHz LGA1155 - CPU
• 1x G.Skill RipjawsX DDR3 1600 Mhz 16 GB (2 x 8GB)
• 4x Seagate Barracuda 2 TB 64 MB 7200 RPM 3.5" SATA
• 1x Infiniband 10gbit adapter

The case and SSD's are still missing, so after I'll get those, I can start building the system. Even though it looks to have pretty basic hardware, I am expecting a very good performance with caching SSD's. 

Fevered nerd from the bed, now back to sleep....

Why, what, how much does it cost, will it work?

To start the blog for a DIY NAS I post this first post to say few things.

Why?
NAS solutions on the market cost a lot, if you want enterprise speeds and require very powerful hardware. I am an IT Architect and want to test if doing this cheaply is possible or not. NAS will be used by couple of VMWare virtual hosts and host my personal data.

What?
DIY NAS, with better than 1gbit performance, and good IOPS. Idea is to use 10gbit Infiniband to connect other devices to the NAS. Target would be around 500MB/s - 800MB/s

How much does it cost?
At the moment the costs are around 1100 € including VAT (847 € excluding VAT). The specific hardware info will be provided later.

Will it work?
I don't know, hopefully it will. We'll see. Wish me luck!

Nasman